Home › Inference Platforms › ai-gateway

ai-gateway - AI Inference Platforms Tool

Overview

ai-gateway (LangDB AI Gateway) is an open-source API gateway for orchestrating requests to many LLM providers through a single, OpenAI-compatible API. It is built in Rust for throughput and low latency and is designed to run self-hosted (or as a managed service) so teams can route traffic to OpenAI, Anthropic, Google/Gemini, local inference endpoints (Ollama, vLLM) and other providers while keeping governance, observability and cost controls centralized. ([lib.rs](https://lib.rs/crates/ai-gateway?utm_source=openai)) The gateway exposes standard OpenAI-style endpoints (chat completions, embeddings, images) and adds production features such as dynamic routing/auto-router (route by cost, quality, availability), rate limiting and rolling cost controls, content guardrails/safety filtering, request/response tracing with spans (OpenTelemetry-compatible), and integrations for analytics and MCP (Model Context Protocol) servers. LangDB’s documentation and SDKs show examples of integrating the gateway into agent frameworks (LangChain, CrewAI, Langroid) and connecting self-hosted models, enabling unified observability across agent runs. ([docs.langdb.ai](https://docs.langdb.ai/?utm_source=openai))

Installation

Install via docker:

git clone https://github.com/langdb/ai-gateway && cd ai-gateway  # repository referenced in LangDB docs.
Create a config file (e.g., config.yaml or Config.toml) with provider API keys and model registry.  See docs for example fields.
docker build -t langdb/ai-gateway:latest .  # build the image from source (Rust project).
docker run -p 8080:8080 -v $(pwd)/config.yaml:/app/config.yaml langdb/ai-gateway:latest  # start gateway with mounted config.

Key Features

  • OpenAI-compatible HTTP endpoints: chat, embeddings, models, and image generation endpoints.
  • Multi-provider routing: unify OpenAI, Anthropic, Google/Gemini, Ollama, vLLM and other providers.
  • Cost control & rate limiting: per-project/tenant controls and rolling cost enforcement.
  • Automatic routing/auto-router: choose providers by cost, latency, availability, or quality rules.
  • Tracing & spans (OpenTelemetry-compatible): end-to-end traces of LLM calls and agent workflows.
  • Guardrails & content filtering: configurable safety rules and system-prompt injection.
  • MCP support & OSS model integration: connect Model Context Protocol servers and local inference targets.

Community

LangDB’s AI Gateway is published as an open-source Rust project (Apache-2.0) with a public crate/manifest and active releases; lib.rs shows multiple releases and crate metadata (e.g., 0.2.x series and project metrics). The project is documented at docs.langdb.ai and promoted on langdb.ai, and ecosystem integrations (Langroid, CrewAI and other agent/tooling) show growing adoption and examples. There is a registry of MCP servers and shared community MCPs on LangDB’s platform for common tooling and data connectors. For code-level metrics, lib.rs reports approximately 17K SLoC and package download statistics; docs show the gateway provides access to 250+ LLMs in the registry and enterprise-grade observability features. Community resources include the official docs, example repositories/samples and the LangDB app pages listing MCP servers and integrations. ([lib.rs](https://lib.rs/crates/ai-gateway?utm_source=openai))

Last Refreshed: 2026-01-09

Key Information

  • Category: Inference Platforms
  • Type: AI Inference Platforms Tool
Visit Official Website

Related Tools

Replicate

Platform to run and fine‑tune AI models with an API; host and deploy state-of-the-art models at scale.
HUGS

Optimized, zero‐configuration inference microservices from Hugging Face designed to simplify and accelerate the deployment of open AI models via an OpenAI‐compatible API.
OpenVINO Toolkit

Open-source toolkit to optimize and deploy AI inference across vision, speech, NLP, and generative models.
Dynamic Speculation

A novel method developed by Intel labs and Hugging Face that accelerates text generation by up to 2.7x using dynamic speculation lookahead in language models, integrated into the Transformers library.
LocalAI

Open‑source, OpenAI‑compatible local inference server for LLMs, TTS, ASR, and diffusion with CPU/GPU backends and container images.
Inference Endpoints by Hugging Face

A fully managed inference deployment service that allows users to easily deploy models (such as Transformers and Diffusers) from the Hugging Face Hub on secure, compliant, and scalable infrastructure. It offers pay-as-you-go pricing and supports a variety of tasks including text generation, speech recognition, image generation, and more.