LLM 工程师工具箱：120+大模型库全攻略！

作者

 Datawhale推荐 

开源项目：llm-engineer-toolkit

编辑：Coggle数据科学团队

在大语言模型（LLM）迅速发展的今天，开发者们面临着海量的资源和工具选择。如何高效地筛选和利用这些资源，成为了每一个 LLM 开发者的关键任务。 今天，我们要介绍的 GitHub 仓库——LLM Engineer Toolkit，或许能成为你的得力助手！

https://github.com/KalyanKS-NLP/llm-engineer-toolkit

这个由 KalyanKS-NLP 创建的仓库，精心整理了超过 120 个 LLM 相关的库，并按照类别进行了分类。无论是训练、推理、应用开发，还是数据提取、安全评估，你都能在这里找到对应的工具。


unsetunset大模型工具划分unsetunset

  • 🚀 LLM Training：专注于 LLM 训练和微调的工具，帮助你更快、更高效地优化模型。
  • 🧱 LLM Application Development：从框架到多 API 接入，再到缓存和低代码开发，为应用开发提供全方位支持。
  • 🩸 LLM RAG：Retrieval-Augmented Generation（检索增强生成）相关的库，提升模型的知识检索能力。
  • 🟩 LLM Inference：推理加速和优化工具，让模型运行更流畅。
  • 🚧 LLM Serving：模型部署和推理服务的解决方案。
  • 📤 LLM Data Extraction：数据提取工具，帮助你从各种来源获取高质量数据。
  • 🌠 LLM Data Generation：生成合成数据，丰富你的训练集。
  • 💎 LLM Agents：构建智能代理，实现自动化任务和多代理协作。
  • ⚖️ LLM Evaluation：评估工具，确保模型性能达到预期。
  • 🔍 LLM Monitoring：监控模型运行状态，及时发现并解决问题。
  • 📅 LLM Prompts：优化和管理提示词，提升模型输出质量。
  • 📝 LLM Structured Outputs：生成结构化输出，让模型结果更易用。
  • 🛑 LLM Safety and Security：保障模型的安全性和可靠性。
  • 💠 LLM Embedding Models：提供先进的文本嵌入模型。
  • ❇️ Others：其他实用工具，涵盖更多开发场景。

unsetunsetLLM Training and Fine-Tuningunsetunset

Library
Description
unsloth
Fine-tune LLMs faster with less memory.
PEFT
State-of-the-art Parameter-Efficient Fine-Tuning library.
TRL
Train transformer language models with reinforcement learning.
Transformers
Transformers provides thousands of pretrained models to perform tasks on different modalities such as text, vision, and audio.
Axolotl
Tool designed to streamline post-training for various AI models.
LLMBox
A comprehensive library for implementing LLMs, including a unified training pipeline and comprehensive model evaluation.
LitGPT
Train and fine-tune LLM lightning fast.
Mergoo
A library for easily merging multiple LLM experts, and efficiently train the merged LLM.
Llama-Factory
Easy and efficient LLM fine-tuning.
Ludwig
Low-code framework for building custom LLMs, neural networks, and other AI models.
Txtinstruct
A framework for training instruction-tuned models.
Lamini
An integrated LLM inference and tuning platform.
XTuring
xTuring provides fast, efficient and simple fine-tuning of open-source LLMs, such as Mistral, LLaMA, GPT-J, and more.
RL4LMs
A modular RL library to fine-tune language models to human preferences.
DeepSpeed
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
torchtune
A PyTorch-native library specifically designed for fine-tuning LLMs.
PyTorch Lightning
A library that offers a high-level interface for pretraining and fine-tuning LLMs.

unsetunsetLLM Application Developmentunsetunset

Frameworks

Library
Description
LangChain
LangChain is a framework for developing applications powered by large language models (LLMs).
Llama Index
LlamaIndex is a data framework for your LLM applications.
HayStack
Haystack is an end-to-end LLM framework that allows you to build applications powered by LLMs, Transformer models, vector search and more.
Prompt flow
A suite of development tools designed to streamline the end-to-end development cycle of LLM-based AI applications.
Griptape
A modular Python framework for building AI-powered applications.
Weave
Weave is a toolkit for developing Generative AI applications.
Llama Stack
Build Llama Apps.

Data Preparation

Library
Description
Data Prep Kit
Data Prep Kit accelerates unstructured data preparation for LLM app developers. Developers can use Data Prep Kit to cleanse, transform, and enrich use case-specific unstructured data to pre-train LLMs, fine-tune LLMs, instruct-tune LLMs, or build RAG applications.

Multi API Access

Library
Description
LiteLLM
Library to call 100+ LLM APIs in OpenAI format.
AI Gateway
A Blazing Fast AI Gateway with integrated Guardrails. Route to 200+ LLMs, 50+ AI Guardrails with 1 fast & friendly API.

Routers

Library
Description
RouteLLM
Framework for serving and evaluating LLM routers – save LLM costs without compromising quality. Drop-in replacement for OpenAI’s client to route simpler queries to cheaper models.

Memory

Library
Description
mem0
The Memory layer for your AI apps.
Memoripy
An AI memory layer with short- and long-term storage, semantic clustering, and optional memory decay for context-aware applications.
Letta (MemGPT)
An open-source framework for building stateful LLM applications with advanced reasoning capabilities and transparent long-term memory
Memobase
A user profile-based memory system designed to bring long-term user memory to your Generative AI applications.

Interface

Library
Description
Streamlit
A faster way to build and share data apps. Streamlit lets you transform Python scripts into interactive web apps in minutes
Gradio
Build and share delightful machine learning apps, all in Python.
AI SDK UI
Build chat and generative user interfaces.
AI-Gradio
Create AI apps powered by various AI providers.
Simpleaichat
Python package for easily interfacing with chat apps, with robust features and minimal code complexity.
Chainlit
Build production-ready Conversational AI applications in minutes.

Low Code

Library
Description
LangFlow
LangFlow is a low-code app builder for RAG and multi-agent AI applications. It’s Python-based and agnostic to any model, API, or database.

Cache

Library
Description
GPTCache
A Library for Creating Semantic Cache for LLM Queries. Slash Your LLM API Costs by 10x 💰, Boost Speed by 100x. Fully integrated with LangChain and LlamaIndex.

unsetunsetLLM RAGunsetunset

Library
Description
FastGraph RAG
Streamlined and promptable Fast GraphRAG framework designed for interpretable, high-precision, agent-driven retrieval workflows.
Chonkie
RAG chunking library that is lightweight, lightning-fast, and easy to use.
RAGChecker
A Fine-grained Framework For Diagnosing RAG.
RAG to Riches
Build, scale, and deploy state-of-the-art Retrieval-Augmented Generation applications.
BeyondLLM
Beyond LLM offers an all-in-one toolkit for experimentation, evaluation, and deployment of Retrieval-Augmented Generation (RAG) systems.
SQLite-Vec
A vector search SQLite extension that runs anywhere!
fastRAG
fastRAG is a research framework for efficient and optimized retrieval-augmented generative pipelines, incorporating state-of-the-art LLMs and Information Retrieval.
FlashRAG
A Python Toolkit for Efficient RAG Research.
Llmware
Unified framework for building enterprise RAG pipelines with small, specialized models.
Rerankers
A lightweight unified API for various reranking models.
Vectara
Build Agentic RAG applications.

unsetunsetLLM Inferenceunsetunset

Library
Description
LLM Compressor
Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment.
LightLLM
Python-based LLM inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.
vLLM
High-throughput and memory-efficient inference and serving engine for LLMs.
torchchat
Run PyTorch LLMs locally on servers, desktop, and mobile.
TensorRT-LLM
TensorRT-LLM is a library for optimizing Large Language Model (LLM) inference.
WebLLM
High-performance In-browser LLM Inference Engine.

unsetunsetLLM Servingunsetunset

Library
Description
Langcorn
Serving LangChain LLM apps and agents automagically with FastAPI.
LitServe
Lightning-fast serving engine for any AI model of any size. It augments FastAPI with features like batching, streaming, and GPU autoscaling.

unsetunsetLLM Data Extractionunsetunset

Library
Description
Crawl4AI
Open-source LLM Friendly Web Crawler & Scraper.
ScrapeGraphAI
A web scraping Python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown, etc.).
Docling
Docling parses documents and exports them to the desired format with ease and speed.
Llama Parse
GenAI-native document parser that can parse complex document data for any downstream LLM use case (RAG, agents).
PyMuPDF4LLM
PyMuPDF4LLM library makes it easier to extract PDF content in the format you need for LLM & RAG environments.
Crawlee
A web scraping and browser automation library.
MegaParse
Parser for every type of document.
ExtractThinker
Document Intelligence library for LLMs.

unsetunsetLLM Data Generationunsetunset

Library
Description
DataDreamer
DataDreamer is a powerful open-source Python library for prompting, synthetic data generation, and training workflows.
fabricator
A flexible open-source framework to generate datasets with large language models.
Promptwright
Synthetic Dataset Generation Library.
EasyInstruct
An Easy-to-use Instruction Processing Framework for Large Language Models.

unsetunsetLLM Agentsunsetunset

Library
Description
CrewAI
Framework for orchestrating role-playing, autonomous AI agents.
LangGraph
Build resilient language agents as graphs.
Agno
Build AI Agents with memory, knowledge, tools, and reasoning. Chat with them using a beautiful Agent UI.
Agents SDK
Build agentic apps using LLMs with context, tools, hand off to other specialized agents.
AutoGen
An open-source framework for building AI agent systems.
Smolagents
Library to build powerful agents in a few lines of code.
Pydantic AI
Python agent framework to build production grade applications with Generative AI.
BeeAI
Build production-ready multi-agent systems in Python.
gradio-tools
A Python library for converting Gradio apps into tools that can be leveraged by an LLM-based agent to complete its task.
Composio
Production Ready Toolset for AI Agents.
Atomic Agents
Building AI agents, atomically.
Memary
Open Source Memory Layer For Autonomous Agents.
Browser Use
Make websites accessible for AI agents.
OpenWebAgent
An Open Toolkit to Enable Web Agents on Large Language Models.
Lagent
A lightweight framework for building LLM-based agents.
LazyLLM
A Low-code Development Tool For Building Multi-agent LLMs Applications.
Swarms
The Enterprise-Grade Production-Ready Multi-Agent Orchestration Framework.
ChatArena
ChatArena is a library that provides multi-agent language game environments and facilitates research about autonomous LLM agents and their social interactions.
Swarm
Educational framework exploring ergonomic, lightweight multi-agent orchestration.
AgentStack
The fastest way to build robust AI agents.
Archgw
Intelligent gateway for Agents.
Flow
A lightweight task engine for building AI agents.
AgentOps
Python SDK for AI agent monitoring.
Langroid
Multi-Agent framework.
Agentarium
Framework for creating and managing simulations populated with AI-powered agents.
Upsonic
Reliable AI agent framework that supports MCP.

unsetunsetLLM Evaluationunsetunset

Library
Description
Ragas
Ragas is your ultimate toolkit for evaluating and optimizing Large Language Model (LLM) applications.
Giskard
Open-Source Evaluation & Testing for ML & LLM systems.
DeepEval
LLM Evaluation Framework
Lighteval
All-in-one toolkit for evaluating LLMs.
Trulens
Evaluation and Tracking for LLM Experiments
PromptBench
A unified evaluation framework for large language models.
LangTest
Deliver Safe & Effective Language Models. 60+ Test Types for Comparing LLM & NLP Models on Accuracy, Bias, Fairness, Robustness & More.
EvalPlus
A rigorous evaluation framework for LLM4Code.
FastChat
An open platform for training, serving, and evaluating large language model-based chatbots.
judges
A small library of LLM judges.
Evals
Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
AgentEvals
Evaluators and utilities for evaluating the performance of your agents.
LLMBox
A comprehensive library for implementing LLMs, including a unified training pipeline and comprehensive model evaluation.
Opik
An open-source end-to-end LLM Development Platform which also includes LLM evaluation.

unsetunsetLLM Monitoringunsetunset

Library
Description
MLflow
An open-source end-to-end MLOps/LLMOps Platform for tracking, evaluating, and monitoring LLM applications.
Opik
An open-source end-to-end LLM Development Platform which also includes LLM monitoring.
LangSmith
Provides tools for logging, monitoring, and improving your LLM applications.
Weights & Biases (W&B)
W&B provides features for tracking LLM performance.
Helicone
Open source LLM-Observability Platform for Developers. One-line integration for monitoring, metrics, evals, agent tracing, prompt management, playground, etc.
Evidently
An open-source ML and LLM observability framework.
Phoenix
An open-source AI observability platform designed for experimentation, evaluation, and troubleshooting.
Observers
A Lightweight Library for AI Observability.

unsetunsetLLM Promptsunsetunset

Library
Description
PCToolkit
A Unified Plug-and-Play Prompt Compression Toolkit of Large Language Models.
Selective Context
Selective Context compresses your prompt and context to allow LLMs (such as ChatGPT) to process 2x more content.
LLMLingua
Library for compressing prompts to accelerate LLM inference.
betterprompt
Test suite for LLM prompts before pushing them to production.
Promptify
Solve NLP Problems with LLMs & easily generate different NLP Task prompts for popular generative models like GPT, PaLM, and more with Promptify.
PromptSource
PromptSource is a toolkit for creating, sharing, and using natural language prompts.
DSPy
DSPy is the open-source framework for programming—rather than prompting—language models.
Py-priompt
Prompt design library.
Promptimizer
Prompt optimization library.

unsetunsetLLM Structured Outputsunsetunset

Library
Description
Instructor
Python library for working with structured outputs from large language models (LLMs). Built on top of Pydantic, it provides a simple, transparent, and user-friendly API.
XGrammar
An open-source library for efficient, flexible, and portable structured generation.
Outlines
Robust (structured) text generation
Guidance
Guidance is an efficient programming paradigm for steering language models.
LMQL
A language for constraint-guided and efficient LLM programming.
Jsonformer
A Bulletproof Way to Generate Structured JSON from Language Models.

unsetunsetLLM Safety and Securityunsetunset

Library
Description
JailbreakEval
A collection of automated evaluators for assessing jailbreak attempts.
EasyJailbreak
An easy-to-use Python framework to generate adversarial jailbreak prompts.
Guardrails
Adding guardrails to large language models.
LLM Guard
The Security Toolkit for LLM Interactions.
AuditNLG
AuditNLG is an open-source library that can help reduce the risks associated with using generative AI systems for language.
NeMo Guardrails
NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems.
Garak
LLM vulnerability scanner
DeepTeam
The LLM Red Teaming Framework

unsetunsetLLM Embedding Modelsunsetunset

Library
Description
Sentence-Transformers
State-of-the-Art Text Embeddings
Model2Vec
Fast State-of-the-Art Static Embeddings
Text Embedding Inference
A blazing fast inference solution for text embeddings models. TEI enables high-performance extraction for the most popular models, including FlagEmbedding, Ember, GTE and E5.

unsetunsetOthersunsetunset

Library
Description
Text Machina
A modular and extensible Python framework, designed to aid in the creation of high-quality, unbiased datasets to build robust models for MGT-related tasks such as detection, attribution, and boundary detection.
LLM Reasoners
A library for advanced large language model reasoning.
EasyEdit
An Easy-to-use Knowledge Editing Framework for Large Language Models.
CodeTF
CodeTF: One-stop Transformer Library for State-of-the-art Code LLM.
spacy-llm
This package integrates Large Language Models (LLMs) into spaCy, featuring a modular system for fast prototyping and prompting, and turning unstructured responses into robust outputs for various NLP tasks.
pandas-ai
Chat with your database (SQL, CSV, pandas, polars, MongoDB, NoSQL, etc.).
LLM Transparency Tool
An open-source interactive toolkit for analyzing internal workings of Transformer-based language models.
Vanna
Chat with your SQL database. Accurate Text-to-SQL Generation via LLMs using RAG.
mergekit
Tools for merging pretrained large language models.
MarkLLM
An Open-Source Toolkit for LLM Watermarking.
LLMSanitize
An open-source library for contamination detection in NLP datasets and Large Language Models (LLMs).
Annotateai
Automatically annotate papers using LLMs.
LLM Reasoner
Make any LLM think like OpenAI o1 and DeepSeek R1.

一起“点赞”三连↓

（文：Datawhale）

欢迎分享

发表评论