croit AI Services
AI Consulting & Training for Self-Hosted LLMs
Own your AI stack
Large language models are becoming core infrastructure. But routing your prompts and data through a third-party API is not always an option, whether for compliance, cost, latency, or control. croit helps you run modern LLMs on your own hardware or private cloud, with the same reliability and operational rigor we bring to Ceph and software-defined storage. Our advice is grounded in real engineering: we build and maintain the open-source croit LLM Gateway, an authenticated, OpenAI-API-compatible reverse proxy that routes requests across multiple model backends, with OIDC login, per-user API tokens, role-based access control, server-side tools, and a built-in chat UI.
Inside the croit LLM Gateway
OpenAI-compatible API
Drop-in /v1/chat/completions, /v1/embeddings, and /v1/audio/transcriptions endpoints. Point any OpenAI SDK at your own infrastructure with no code changes.
Multi-backend routing
Named upstream pools load-balance across your GPU backends with health probes and live model discovery. Add capacity or load a new model with no config change.
OIDC login and RBAC
Browser sign-in against your identity provider, revocable per-user API tokens hashed at rest, and roles that gate which models and tools each user can reach.
Server-side tools
The gateway runs tools mid-completion: web search, URL fetch, document rendering, RAG, and network lookups. The client just receives one finished answer.
How we help
AI Consulting
We take you from idea to a running, secure LLM platform. Engagements cover architecture and GPU sizing for self-hosted inference, deploying and hardening the LLM Gateway with OIDC, RBAC, token policy, and tool governance, model selection and backend pooling, RAG pipelines over your internal knowledge base, safe integration of server-side tools and MCP servers, and clear analysis of cost, latency, and data-sovereignty trade-offs.
AI Training
We upskill your team to operate AI infrastructure with confidence. Courses cover the fundamentals of self-hosted LLMs and OpenAI-compatible APIs, hands-on workshops with the LLM Gateway, access control and security with OIDC, roles, tokens, and tool permissions, building and maintaining RAG collections, and day-two operations such as monitoring, troubleshooting, and scaling. We tailor the curriculum for engineers, platform teams, and decision-makers.