Generative AI doesn't have to run on a public cloud for every task. Today, small and quantized language models can be used on high-performance laptops, Macs, PCs, or your own servers. This opens up new possibilities—especially when processing source code, protocols, contracts, customer documents, or internal knowledge.

The four-hour workshop was therefore not about a blanket “on-premises instead of the cloud” approach. The focus was on making better architectural decisions: What data is being processed? What level of quality is required? Who operates the system? And which solution makes economic and organizational sense for the specific use case?

Cloud or on-premises—it's not a matter of faith

Cloud models offer high performance, a quick start, and a steady stream of new features. On the other hand, on-premises models can enhance data sovereignty, offline capability, cost control, and independence. At the same time, they create new responsibilities related to hardware, updates, security, monitoring, and quality assurance.

Local operation is not an automatic guarantee of data protection. It is a technical option that becomes reliable only when backed by a sound authorization, operation, and deletion framework.

The statement “Data does not leave the company” also applies only if the entire process is designed accordingly—that is, without unintended telemetry, external plugins, cloud integrations, or connected web services.

From a Model to a Functional Workspace

The first practical section covered everything from installation to the first local conversation with a language model. Tools like Ollama or LM Studio make it easy to get started, while Open WebUI can provide a familiar interface for different models, users, and knowledge sources.

The participants were able to understand how model size, quantization, speed, and available hardware influence one another. It’s not necessarily the case that the largest model automatically wins. For many clearly defined tasks, a smaller, faster model is the more sensible choice.

Make Your Own Documents Available Using RAG

One key focus was Retrieval-Augmented Generation, or RAG for short. In this approach, a language model receives relevant excerpts from approved company documents for each query. The model is not retrained using these documents; instead, it receives the context needed specifically for the task at hand.

A well-defined RAG process typically includes:

select and clean up the appropriate documents,
Break content down into searchable sections,
Identify passages that relate to the question,
Display the answer and sources together,
Check quality using prepared test questions.

For example, this can lead to the creation of an internal knowledge base covering products, processes, or technical documentation. It remains crucial to use up-to-date sources, maintain a record of accesses, and ensure that answers are reviewed by subject matter experts.

Specific Areas of Application in IT Companies

The workshop examined numerous tasks where local AI offers a practical starting point:

Explain and document source code, and analyze it as an additional verification tool,
Presort support tickets and prepare draft responses,
Organize log files and highlight anomalies for further analysis,
Transcribe meetings locally and convert them into assignments or minutes,
conduct targeted searches of internal documentation and knowledge bases,
as well as prepare recurring drafts of proposals or technical specifications.

An AI system does not automatically assume professional responsibility in this context. It speeds up preparatory work and makes information more easily accessible. Approval, safety-critical decisions, and binding statements remain the responsibility of humans.

Safety Remains a Management and Operational Responsibility

Even a locally operated model can hallucinate, react to manipulated content, or carry out far-reaching actions via connected tools. That is why clear guidelines must be included in the very first prototype:

minimal permissions based on the principle of least privilege,
human approval prior to external or irreversible actions,
Logging and traceable versions,
separate test and production environments,
a fixed set of realistic test questions and negative tests,
as well as a clear classification of model and software licenses.

The most important recommendation: Start small and measure your results

The most sensible way to get started is rarely a company-wide AI platform. It’s better to start with a clearly defined use case that involves known data, a designated expert, and a measurable goal.

The solution should only be expanded once quality, speed, operational costs, and actual benefits have been evaluated. Local and cloud-based AI can coexist. What matters is not the model’s location alone, but a robust overall process.

Would you like to see if local AI, RAG, or a hybrid architecture is right for your business? Let's discuss your specific use case →

A human-free representation of a local AI infrastructure with its own hardware, document workflow, and security checks. — Editorial stock image illustrating local language models, RAG, and controlled AI infrastructure within a company.

Workshop Recap · June 16, 2026

Local AI Power: Data-Secure AI Solutions for IT Companies

On Tuesday, June 16, 2026, I led the “Local AI Power” workshop at WIFI from 9:00 a.m. to 1:00 p.m. Entrepreneurs, CEOs, and senior executives from the information technology sector explored how generative AI can be run on their own hardware and effectively integrated into business processes.