AI basics
Why Self-Host Your AI Applications on a Server
Most of an AI application is ordinary server work: documents, vector search, automation and logs. Run it on your own servers and call the model as an API.
1. The model is one part of the stack
Hosting AI is often pictured as a rack of GPUs running a large model. For many businesses that is neither necessary nor the right place to start. The model can come from OpenAI, Anthropic, Google, Mistral or another provider, while the documents, databases, embeddings, automation, APIs, logs and customer data stay on servers you control.
Take an internal assistant asked about the cancellation terms in enterprise contracts. Answering it involves these steps:
- Authenticate the employee and apply their permissions.
- Search the document store and retrieve the relevant contract paragraphs from a vector database.
- Build a prompt with those paragraphs and call the LLM.
- Stream the answer back, record the interaction and track latency, token usage and cost.
Only the model call needs a large AI model. Everything else runs well on a conventional VM or dedicated server.
2. Run your own RAG platform
Retrieval-augmented generation (RAG) retrieves relevant information from your own sources first and passes it to the model as context, so the model does not need to have been trained on your data. A self-hosted RAG platform usually contains PostgreSQL for users, permissions and conversations; pgvector or Qdrant for embeddings; object storage for PDFs, documents and transcripts; workers for text extraction, chunking and indexing; an LLM gateway; and a web interface.
Your documents do not have to move into a third-party SaaS knowledge base. You decide where they are stored, how they are indexed, how long logs are kept and which systems can reach them. Because the model sits behind a gateway, you can change provider without rebuilding the rest of the platform.
3. Vector databases and semantic search
A vector database finds content by meaning. A search for cheap family holiday near Athens can match an article about affordable destinations for children even when those words never appear in it. pgvector, Qdrant, OpenSearch and Redis all offer vector search.
A small deployment can share a VM with the application. As a knowledge base grows to millions of chunks, move the vector database to a server with more RAM and NVMe storage. Each query may search a large index before the model receives anything, so memory and predictable disk latency have a direct effect on response time.
4. Automate with self-hosted n8n and agents
AI is more useful when it can start actions. A self-hosted automation tool such as n8n can connect a model to your existing systems. An incoming email can be classified, matched against the knowledge base, given a suggested reply and turned into a CRM ticket for the right team. A newly uploaded document can be extracted, chunked, embedded and made searchable within minutes.
AI agents follow the same pattern: a model decides the next step, and most of the execution happens in ordinary code that queries databases, processes files and calls APIs. A production setup typically runs workers pulling jobs from a queue, Redis for coordination, PostgreSQL for state and object storage for files. You add workers as usage grows, as with any distributed system.
5. Put an AI gateway in front of every provider
When each team connects straight to a different provider, API keys, model choice, logging and spending end up spread across many applications. An internal gateway at an address such as ai.company.com gives applications one OpenAI-compatible endpoint and moves those decisions to one place. LiteLLM is one open-source option. Through the gateway you can manage:
- Authentication, per-application API keys and access policies.
- Rate limits, usage limits and cost tracking per team.
- Model routing, provider failover and switching models without changing application code.
- Central request logging.
The same approach works for the user interface. Open WebUI can be self-hosted as a company chat portal connected to commercial APIs, your RAG services and local models, with sign-in and network access under your control.
6. Observe what the AI is doing
Server monitoring tells you whether a host is up. An AI application also needs to show which model answered, what prompt was sent, how long retrieval and inference took, how many tokens were used, what the request cost, which documents were retrieved and whether the user found the answer useful. Langfuse is an LLM observability platform that can itself be self-hosted.
Prompts and responses add up quickly. At meaningful volume this becomes a question of database sizing, storage and retention, and those logs may contain personal data, so set retention and access rules from the start.
7. Pipelines and media archives
Knowledge bases need continuous ingestion: files are converted, OCR-processed where needed, chunked, embedded, indexed and refreshed. These pipelines benefit from many CPU cores, plenty of RAM, fast NVMe, high sequential throughput and reliable background processing. Embeddings can come from an external API or from a smaller model run locally, which is often practical on CPU.
Media companies can use the same approach on photo, video and audio archives. AI services can tag people, places, objects, speech, scenes, logos and on-screen text, and the resulting metadata and embeddings stay on the broadcaster’s own infrastructure. Editors can then search for football player celebrating in front of supporters instead of guessing filenames. The hard part is storing, indexing and serving a very large dataset efficiently.
8. What the infrastructure has to provide
AI applications produce more data than most teams expect: source files, embeddings, indexes, conversation history, logs, workflow state, evaluation sets and backups. A few requirements follow from that.
| Need | Why it matters | Typical approach |
|---|---|---|
| Storage | Databases and indexes are latency-sensitive; source files and media are large | Local NVMe for databases and indexes, object storage for files, separate backup storage |
| Bandwidth | The model returns kilobytes of text, but ingestion moves archives, video and backups | Enough transit and peering for bulk transfers alongside production traffic |
| Latency | A model call already takes seconds; spreading components across regions adds more | Keep the application, databases, vector search, workers and gateway in one environment |
| Availability | A demo on one VM is fine; a service people depend on needs redundancy | Multiple app nodes, database replication, load balancing, monitoring and tested failover |
9. Start small and scale the parts that matter
You do not need a large platform on day one. One VM running Docker, n8n, PostgreSQL with pgvector, Open WebUI, a gateway and a small internal application is enough for a prototype or an early internal tool. Separate components as load appears:
- Stage 1: one VM for prototypes, internal tools and early production.
- Stage 2: separate database and application servers, with more RAM and faster NVMe for the database.
- Stage 3: several application and worker nodes behind a load balancer, with Redis, replicated databases and a dedicated vector store.
- Stage 4: high availability across physical servers with redundant networking and storage, possibly in more than one location.
When do you need GPUs?
GPUs matter when you want to run large models locally, fine-tune or train models, or serve heavy inference yourself. That is a separate category of infrastructure with its own costs and planning.
You can self-host almost the entire application around the model and still consume the model as an API. For many organisations this is the more practical design: the business logic and data stay under their control, and they do not have to operate GPU clusters to get there.