Hosting & Securing AI Infrastructure

Anyone running agents learns quickly that the model is the easy part. This paper describes the work that starts afterwards: hosting models and agents on infrastructure you control, isolating them at the network layer, hardening them, and running them around the clock, the way LSA runs Linux infrastructure.

1. The thesis

Language models can be downloaded today, and agent runtimes are open software; procurement is a solved problem. What is not solved is the state afterwards: an agent that works in multi-step loops depends on the environment it runs on. It needs compute that does not drop out in continuous operation, a network path that keeps it bounded, logs that evidence what it actually did, and a recovery path that still works when nobody is sitting next to it.

Our thesis: agents are production environments, and they deserve the same care Linux teams have invested for years in hardening, monitoring, and backups. For self-hosters and DevOps practitioners this is not a new discipline; it is the next tier of their own. Anyone operating servers today, who knows patch cycles and has rehearsed restores, already owns the foundation an agent stack runs on. This is not a rethink, it is an extension of what exists.

The difference between a well-run agent stack and a risk emerges therefore not at model selection, but in hosting, isolation, and operations. That is exactly where the rest of this paper lives.

What this paper is, and what it is not

This paper is not a tutorial, not a model benchmark, and not legal advice. It is an operations view: the perspective of a team that administers Linux servers, hardens environments, and keeps operations open around the clock. Regulatory facts appear where they directly touch operations; they do not replace legal review for the individual case. The chapters follow the order operations asks in: where it runs (Section 3), how it is bounded (Section 4), how it stays healthy in daily business (Section 5).

2. Agents are operational loads

From chatbot to production environment

A chatbot answers when asked. An agent works in a loop: it plans steps, invokes tools, checks intermediate results, and continues, as often as the task requires. From an infrastructure perspective the difference is concrete:

1. Continuous operation: agents run in 24/7 mode; the environment needs the same standards as any production service.
2. Tool invocations: every loop generates calls on APIs, services, and data sources; each one must be logged and bounded.
3. Resource draw: model inference is a spiky compute load; capacity you plan for, not capacity that simply happens to be left over.

On top of that load comes one that rarely shows in pilots: chaining. As soon as one agent's output feeds the next, the access profile multiplies: every link in the chain is a path that has to be bounded, logged, and cut when needed. What was a single loop in testing is a small federation in operation, with as many paths as if you had introduced several services at once.

What the market numbers say about operations

The market reports outline the situation, and for operators it sounds operational: McKinsey's State of AI survey of March 2025 reports that more than three-quarters of respondents say their organizations use AI in at least one business function. Gartner, in its press release of 25 June 2025, predicts that by 2028 around 33% of enterprise software applications will include agentic AI (up from less than 1% in 2024) and that at least 15% of day-to-day work decisions will be made autonomously through agentic AI by 2028.

The second half of the same prediction belongs to an honest account: more than 40% of agentic AI projects will be canceled by the end of 2027, according to Gartner, because of escalating costs, unclear business value, or inadequate risk controls. Both statements hold at once: usage is broad, the dropout quota is high. For infrastructure teams the meaning is clear: the cancellation reasons are operational, and that is precisely why they are our own responsibilities. Cost control means plannable capacity (Section 3); value control means measurable workflows and an owner per agent (Section 6); risk control means isolation, logs, and recoverability (Sections 4 and 5). Every cancellation reason the market names is, from that angle, an architecture piece you can put in place beforehand.

Agent frameworks are already in active operation for several of our customers. In the environments that stand and carry weight, the difference is rarely the model; it is the foundation those frameworks run on. Fittingly, the same Gartner text carries the advice: pursue agents only where a clear value or ROI is visible. We do not read that as a sales ban but as a planning rule, and it applies just as much to the infrastructure you build for them.

Where agents genuinely carry weight today, everyday work decides: repetitive, well-bounded, fault-tolerant tasks: routine diagnostics, log reading, incident triage, ticket upkeep. In our blog post on the role of AI in Linux server administration (AI in Linux server administration, EN) we describe the operations side of the same topic: the models that can now run locally, and the monitoring that goes with them.

3. Hosting: on-premise, colocation, cloud

Where your models live: you decide

There are three sensible places for agent stacks, and none of them asks for an act of faith: on-premise on your hardware, colocation in a data center you choose, or a cloud region you control. The difference is not the model; it is the data path and the accountability: what is processed on your machines stays inside your perimeter; what runs in a cloud region falls under that provider's operating agreement. Both routes are legitimate as long as logging, access, and recovery stay under control.

We ourselves serve environments in all three forms (on-premise in Munich, colocated in Frankfurt, cloud-native in a cloud region), and we recommend the form by workload and data-protection needs, not as a matter of principle. Cloud builds and migrations are named in our services; EU cloud regions are the default case wherever data-protection needs call for them.

Dedicated GPU hosting, at its core, means this: the capacity behind your inference workloads lives under the same operational discipline as the rest of your stack: dimensioned with you, patched, monitored, and restartable, wherever the GPUs sit. The concrete form is set in binding terms by that decision: dedicated GPU servers in our own data center, plus operating customer-owned GPU hardware.

Local models: the operating reality

Local models have one undeniable advantage: the data path stays short and inside your perimeter, and the stack scales with your dimensioning, not with usage. The unvarnished second half belongs to it: local does not mean secure by default. A local stack also writes logs, holds context in memory, and needs a rights pattern, and architecture decides whether that surface stays small. What disappears locally is the provider share of the data path; what appears locally is your own operations share.

Dimensioning is craft, not secret knowledge. Labelled example calculation: Meta's Llama 3.1 in the 8B variant has a 128k context window per the model card; the quantized Q4_K_M build in the Ollama library comes in at roughly 4.9 GB model size (8.03B parameters). Models like that run on hardware self-hosters already operate, but practice needs far more headroom in the calculation than the bare model size: RAM for context and caches, storage for logs and artifacts, the CPU-GPU balance per task type. Knowing the size of the models is what lets you plan the rest: model updates arrive in a cycle, like other software updates, and a rollback to a tested version is not an extra, it is part of the package.

Larger models in a controlled cloud region remain the complementary pole for tasks with a high quality bar; the mix emerges per task, not as a one-time configuration. Vector stores and retrieval components run, in that picture, like any other service: versioned, backed up, monitored; none of it is “just an embedding”.

Model serving and agent runtimes

The long-term operation of an agent stack splits into two operating components:

1. Model serving: models run containerized (Docker, Podman, Kubernetes), versioned and restartable; model updates follow the same pattern as any other software update: tested, logged, reversible. Rollback paths are planned, not improvised.
2. Agent runtimes: agents get their own lane, sandboxes with least-privilege credentials instead of admin keys; open agent frameworks (such as OpenClaw or Hermes) as the orchestration, reaching your systems only through defined paths. One identity per agent, no shared keys: what an agent may invoke is defined at the access layer, not in the orchestration.

Both components share the same principle: the runtime decides what an agent does; the rights and access layer decides what it may do. Only there are rights auditable, only there revocable.

4. Securing: network isolation and hardening

Segments, allowlists, VPN access

An agent stack belongs in its own network segments, separated from your production traffic, your back office, and anything that does not need to touch it. The patterns are standard DevOps craft; for agents they are duty, not option. The reason sits in the calling pattern: agents cross trust boundaries by design, they pull context from one system to act on another. Without segments and allowlist rules, every one of those boundary crossings is silently approved:

• ▸Dedicated network segments and firewall rules for agent workloads, agreed with you
• ▸Agents reach your systems exclusively through explicit allowlist rules; no open paths
• ▸Administrative access via VPN only; agent administration separated from other admin traffic

This structure gives you two things that are otherwise hard to get: a small, explainable attack surface, and the ability to reconstruct afterward what an agent actually reached. With segmentation, an incident changes from “somewhere in the estate” to “in the agent segment”: bounded, traceable, and re-openable once cleared.

Secrets and credentials

Credentials have no place in an agent's working context: no keys in prompts, no shared admin access, no hard-coded paths inside the orchestration. Secrets are managed as secrets and injected at runtime, never stored in agent context. Rotation belongs to it: credentials that change regularly keep even the awkward case manageable (an agent behaving suspiciously) because revoking its rights is then an administrative act, not a teardown project.

Hardening as a working standard

Agent environments deserve no exotic tooling, but the same hardening standard as every production server, with an addition: they host a system that invokes itself. What has proven itself in our daily work:

• ▸CIS benchmark alignment, CVE patching, and SSH hardening as a fixed standard
• ▸Rate and resource limits: a runaway agent cannot take its neighbours down
• ▸Full action logging per agent, feeding your audit log review, including who granted or revoked which rights

The log trail is more than evidence: it is the raw material of every later analysis (debugging, audit, incident reports). Demand it from day one and you operate a system; introduce it only after the first incident and everything before that was an experiment. Log retention is part of the trail: you set how long action logs stay retrievable; we arrange storage to match, so follow-up questions do not run into nothing.

Keep it small, keep it stoppable

Two questions settle the security side: how little may the agent do? And: how fast can you stop it? The answer to the first is least privilege: per-agent credentials, bound to the concrete task, revocable, expiring. The answer to the second is a control layer: irreversible actions run through confirmed steps; every agent process is stoppable; emergency procedures get rehearsed, not just documented. An agent nobody observes is not an operation; it is an experiment waiting for its incident. And because rights are granted and revoked, that history belongs in the log too: who granted which agent which permission, and when, becomes relevant the moment anyone asks follow-up questions after an incident.

5. Running it: monitoring, backup, recovery

Monitoring that sees the loop

Agents do not sleep, and the infrastructure behind them cannot either. We wire GPU hosts, model serving, and agent runtimes into monitoring (Grafana and Prometheus, your instance or ours) with metrics for runtime, error rates, tool invocations, and capacity. An agent stack carries signals above the classic server metrics that have to be instrumented to be seen: a tool call taking longer than usual, retries piling up, a loop slowing down, a queue growing at the GPU; all states you spot when you measure them, and miss when you do not. Alerts go into the channels you already use, instead of founding a new island. Proactive monitoring and alerting are part of every engagement, not an add-on.

Backups with tested recovery

Model artifacts, agent state, and configurations need deliberate backup strategies, and recovery processes that have actually been tested, not a “probably works”. The difference from classic backups lies in the systems' tempo: agent state changes while they run; a restore therefore has to answer which consistency you need: the last confirmed action, not the last written state. A restore deployed untested is not a restore; it is a search.

Operating under reporting and transparency duties (no legal advice)

We are not lawyers, and nothing in this section is legal advice. But the regulatory situation is an operating situation, and it touches agent stacks concretely: under the NIS2 directive (Directive (EU) 2022/2555) the requirements have applied since 18 October 2024, and for significant incidents Article 23 requires a chain of an early warning (without undue delay, in any event within 24 hours), an incident notification (within 72 hours), and a final report (no later than one month after the notification). Within those windows you can only report what your own systems evidence: who was involved, what changed, which data was affected. An agent stack without full action logging creates exactly the gap that makes those windows practically unfilled.

The same principle, a second layer of rules: the majority of the EU AI Act applies since 2 August 2026, including the transparency rules of Article 50 (timeline: European Commission, AI Act Service Desk). Traceability becomes a design requirement: the log structure you need for incidents is the same one transparency demands. Two clocks, one foundation: keep logs fit to answer incidents inside the NIS2 windows and you have already built what the transparency duties ask for, without running two separate log worlds.

For an assessment of your obligations in the individual case, take legal advice. Our part is the operation that makes those requirements achievable: logs that answer the questions, and procedures that hold in an incident.

The operation behind this paper carries the same commitments we publish for the rest of our stack: availability 24/7, an uptime target of 99.9%, and a response time of < 1hr. These are not aspirations drafted for this paper; they are the figures our Linux operations already run on, carried over to agent stacks and GPU hosts unchanged, because the same team, the same monitoring, and the same escalation paths stand behind both. Continuous operation around the clock is the baseline this section has described throughout (monitoring, tested recovery, logged actions), and it is the baseline your agent stack inherits.

6. From paper to collaboration

A paper stays paper until someone applies it to your environment. Working with LSA therefore does not start with building a stack: it starts with an inventory, and each step is followed by one of your decisions:

1. 1

   First contact

   Tell us briefly where agents should (or already do) work in your workflows, by form, email, or WhatsApp; we get back to you, and it is decided there whether the inventory is the right entry.

2. 2

   Inventory

   We map what runs (servers, networks, access paths, data flows, existing automation) and work out, from live operations rather than a form-filling exercise, the workflows where an agent can genuinely take over multi-step work. Customers we already serve have a head start here: environment, history, and security level are partly already known; the inventory then does not start at zero.

3. 3

   Recommendation & roadmap

   The stack recommendation, prioritized use cases, and the security frame stand in writing, including an open answer where agents do not (yet) carry weight. You get the reasoning walked through in a briefing; approval is yours, not ours.

4. 4

   Build & operate

   After your approval we build (GPU capacity, model serving, isolated runtimes, monitoring, and backup strategies), all documented so your team, or any later operator, can run it safely, and operate it continuously afterwards. Your team keeps full access at all times.

We hold ourselves to the same standard Gartner applies in the same prediction: pursue agents only where clear value or ROI is visible. The inventory can therefore end with the answer that agents bring no extra value in your environment today. That answer is a result, not a failure, and the cheapest possible point to know it.

The flow is deliberately built so that every step leaves a written intermediate result that belongs to you: pause, defer, or continue after each one. What never happens anywhere in this flow is a stack appearing without your written approval; the roadmap is the decision document, not a standing mandate.

7. Closing

Agents move the real work to the front of the project: before the first model starts, the foundation decides the hosting form (on-premise, colocation, or cloud, matched to workload and data-protection needs), the isolation (segments, allowlists, separate runtimes), and the operation (monitoring, logged actions, tested recovery). The discipline agent stacks need already exists in well-run Linux environments, completed by isolation, action logging, and rights administration. None of it restarts your practice; it is the same practice, extended by a new building block.

If you want to find out where agents carry weight in your workflows, and what your foundation already delivers for them: talk to us about your environment. The infrastructure briefing is free, and the decision stays yours.

Sources

1. Gartner, Inc.: "Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027" (press release, 2025-06-25) — "Over 40% of agentic AI projects will be canceled by the end of 2027, due to escalating costs, unclear business value or inadequate risk controls"; within it: "33% of enterprise software applications will include agentic AI by 2028, up from less than 1% in 2024"; "at least 15% of day-to-day work decisions will be made autonomously through agentic AI by 2028, up from 0% in 2024"; "Gartner recommends agentic AI only be pursued where it delivers clear value or ROI". Direct fetch blocked; verified via a Wayback snapshot (20250926185809, web.archive.org, accessed 2026-11-01).https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027
2. McKinsey & Company: "The state of AI: How organizations are rewiring to capture value" (2025-03-12) — "More than three-quarters of respondents now say that their organizations use AI in at least one business function." Direct fetch bot-bound (timeout); verified via a Wayback snapshot (20251229181420, web.archive.org, accessed 2026-11-01); the exact percentages (78/72/55) do not appear in the archived text — deliberately restated as "more than three-quarters".https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai-how-organizations-are-rewiring-to-capture-value
3. Directive (EU) 2022/2555 (NIS2), official EUR-Lex text (CELEX 32022L2555) — Article 41(1): "By 17 October 2024, Member States shall adopt and publish the measures necessary to comply with this Directive. … They shall apply those measures from 18 October 2024."; Article 23: early warning "within 24 hours of becoming aware of the significant incident", notification "within 72 hours", final report "not later than one month after the incident notification". Direct fetch blocked; verified via the Wayback snapshot of the ELI permalink (20261005173546, web.archive.org, accessed 2026-11-01).https://eur-lex.europa.eu/eli/dir/2022/2555/oj/eng
4. European Commission: "Timeline for the Implementation of the EU AI Act" (AI Act Service Desk) — milestone "02 Aug 2026 — The majority of rules of the AI Act come into force and enforcement starts for applicable rules. Transparency rules (Article 50) start to apply." (accessed live 2026-11-01).https://ai-act-service-desk.ec.europa.eu/en/ai-act/timeline/timeline-implementation-eu-ai-act
5. Meta: Llama 3.1 model card (meta-llama/Llama-3.1-8B-Instruct, Hugging Face) — 8B/70B/405B, 128k context window (accessed live 2026-11-01). · Ollama library: llama3.1:8b-instruct-q4_K_M — "4.9GB", "model arch llama · parameters 8.03B · quantization Q4_K_M" (accessed live 2026-11-01).https://huggingface.co/meta-llama/Llama-3.1-8B-Instructhttps://ollama.com/library/llama3.1:8b-instruct-q4_K_M
6. LSA live assets (own fetches 2026-11-01): environment line on all three homepages (EN: "Whether your servers are on-premise in Munich, colocated in Frankfurt or cloud-native in [provider region — redacted per copy rule (d), 2026-11-01], we adapt to your stack and your working language."); services lines on .de /service-support ("Sicherheitshärtung & Audits CIS-Benchmarks, CVE-Patching, SSH-Härtung, Audit-Log-Review"; "Netzwerk & Firewall Firewall-Konfiguration, VPN-Setups"); tech stack line ("Linux · Nginx · systemd · Docker/Podman · Kubernetes · Ansible · Terraform · [provider name — rule (d)] · Grafana · Prometheus"); cloud card ("[provider names — rule (d)]"); the AI blog post live on all three domains (.com slug ai-transforming-linux-server-admin; .de same slug; .cz jak-ai-meni-spravu-linux-serveru) — .com quote: "you can now run capable models locally — on the same server or a small GPU box — without sending sensitive logs to a third-party API".