Running open-source large language models on hospital-owned infrastructure is the deployment pattern for organizations that cannot use cloud-hosted AI — IT governance restrictions, payer-required data isolation, state-level privacy laws, prior-breach experience that hardened the data-residency posture, or contractual data-residency clauses with academic affiliations. The 2026 production stack includes Llama 3 70B as the default for high-capability deployments, Mistral and Mixtral for high-volume inference workloads, Phi-3 for resource-constrained deployments, and Qwen for multilingual use cases — all served via vLLM (the production-default inference engine), with Triton Inference Server, llama.cpp, and Ollama as alternatives for specific deployment patterns. The compliance perimeter shrinks back to the hospital’s existing audited perimeter; there is no model-provider BAA question because there is no model provider in the loop. The engineering work shifts to model serving infrastructure, monitoring, fine-tuning, hardware sizing, and lifecycle management — all under the customer’s existing security posture.
A meaningful share of US hospitals cannot use cloud-hosted LLMs at all. In 2024, this segment of the buyer market was effectively excluded from generative AI deployment. In 2026, that exclusion has ended: open-source models running on hospital-owned GPU infrastructure or single-tenant private cloud infrastructure the hospital controls have reached capability sufficient for most clinical use cases. Capability sits roughly one to two model generations behind frontier closed models. For clinical documentation, summarization, intake triage, prior-authorization letter generation, and most copilot patterns, that gap is operationally irrelevant.
This guide is the engineering reference Taction Software® uses on on-prem LLM engagements with hospitals.
What “On-Prem” Actually Means in 2026
Three deployment patterns fall under the on-prem umbrella in 2026, with substantially different operational implications.
Pattern 1 — True On-Prem on Hospital-Owned Hardware
The hospital owns the GPU servers running model inference. The hardware sits in the hospital’s data center under the hospital’s existing physical-security and operational controls. No external cloud is involved.
Strengths. Maximum data control. The compliance perimeter is the hospital’s existing audited perimeter. No external-vendor BAAs needed for inference. Latency is minimal because data never leaves the hospital network.
Trade-offs. Hospital owns the operational complexity (GPU procurement, hardware lifecycle, on-prem MLOps, capacity planning). Capital expense is substantial; ongoing operational cost is not negligible. Capability is bounded by what can run on the hospital’s hardware.
When to choose. Hospitals with existing data-center operations, strict data-residency policies, or specialty data-handling requirements that exclude any cloud option.
Pattern 2 — Single-Tenant Private Cloud the Hospital Controls
The model runs on infrastructure dedicated to the hospital, operated by the hospital or a hospital-controlled vendor. The cloud provider supplies the hardware; the hospital controls the deployment, the data flows, and the operational keys.
Strengths. Operational complexity is shared with the cloud provider. Capital expense shifts to operating expense. Capability scales more easily than true on-prem. Data-residency and compliance posture remains under the hospital’s control.
Trade-offs. Cloud-provider BAA still required. Some on-prem-only policies exclude this pattern; others permit it because data isolation is operationally tight.
When to choose. Hospitals with on-prem-strong policies that permit single-tenant private cloud. Often the operational sweet spot for hospitals that want data control without the full operational burden of true on-prem.
Pattern 3 — VPC-Isolated Cloud with Customer-Managed Keys
The model runs in the hospital’s cloud account (AWS, Azure, GCP) within a VPC isolated from external networks, with customer-managed encryption keys controlling all data access. The hospital’s existing cloud BAA covers the deployment.
Strengths. Operational simplicity. Cloud-provider’s GPU infrastructure scales naturally. Standard cloud-native MLOps tools apply.
Trade-offs. This is technically not “on-prem” — data resides in the cloud provider’s infrastructure. Whether this satisfies the hospital’s policy depends on the policy specifics. Some hospitals’ policies exclude it; some permit it under specific configurations.
When to choose. Hospitals whose policies permit cloud deployment but require customer-controlled encryption and isolation. Operationally easier than true on-prem; legally distinct from cloud-hosted LLM provider patterns.
The right pattern is institution-specific. Many engagements scope all three before settling on the one that matches the customer’s specific policy and operational posture.
The Open-Source Model Landscape in 2026
The four open-source model families most production deployments use.
Llama 3 70B (and Llama 3.1, 3.2, 3.3, 4 as released)
The default for high-capability deployments. Strong instruction-following, well-supported tooling ecosystem (vLLM, Ollama, llama.cpp, TGI), generous license terms.
When to choose. Most clinical-quality use cases — clinical documentation, summarization, intake processing, prior-authorization letter generation, copilot patterns where capability matters more than throughput.
Hardware footprint. ~140GB at full precision (BF16); ~80GB at INT8 quantization; ~40GB at INT4. Production typical: 2–4× H100 80GB or 4–8× A100 80GB depending on quantization and concurrency.
Mistral and Mixtral
Mistral 7B for high-throughput resource-constrained workloads; Mixtral 8x7B and Mixtral 8x22B for mixture-of-experts architectures that produce high effective capability per dollar at high inference volume.
When to choose. High-volume inference where the per-inference cost matters substantially; or when the institutional infrastructure can’t support the largest Llama variants.
Hardware footprint. Mistral 7B fits on a single GPU (24GB) at full precision. Mixtral 8x7B needs 90GB+ at full precision; quantizes well to fit on 2× A100 40GB or 1× H100 80GB.
Phi-3 (and Phi-4)
Microsoft’s small-but-capable models. Phi-3 medium (14B) and Phi-3 small (3.8B) deliver capability competitive with much larger models at a fraction of the inference footprint.
When to choose. Resource-constrained deployments — smaller hospital infrastructure, edge use cases, on-device patterns. Specific tasks (classification, simple structured generation, routing) where smaller models match larger-model performance.
Hardware footprint. Phi-3 medium fits comfortably on a single 24GB GPU at full precision; Phi-3 small fits on consumer-grade hardware.
Qwen
Alibaba’s Qwen family. Strong multilingual capability — particularly Chinese, Korean, Japanese, and Spanish. Capable across the parameter scale (7B, 14B, 72B variants).
When to choose. Multilingual deployments where the patient population requires it. Use cases where Qwen’s specific capabilities (long context, multilingual reasoning) match the use case.
The Production Inference Stack
The inference engine determines latency, throughput, and operational complexity. The 2026 production landscape.
vLLM (the production default)
vLLM is the dominant production inference engine for open-source LLMs in 2026. Strong throughput, supports continuous batching for high-concurrency workloads, supports the major model families, and integrates with standard observability tooling.
When to choose. Most production deployments. The default unless specific requirements push elsewhere.
Operational characteristics. OpenAI-compatible API surface, so application code written against OpenAI can switch to vLLM with minimal changes. Supports tensor parallelism for multi-GPU deployments. Supports quantization (INT8, INT4, AWQ) for memory-constrained scenarios.
NVIDIA Triton Inference Server
Triton is NVIDIA’s production inference server. Strong for multi-model deployments (running multiple models on the same GPU fleet), supports broader model types (vision, audio, multi-modal alongside LLMs).
When to choose. Mixed-model deployments where LLMs share infrastructure with imaging models, ASR, and other AI components.
llama.cpp
llama.cpp is the high-efficiency inference engine for CPU-based and edge deployments. Runs Llama, Mistral, Phi, and other models with aggressive quantization (INT2 through INT8).
When to choose. Edge deployments, resource-constrained environments, or deployments where CPU-only inference is acceptable for the use case’s latency tolerance.
Ollama
Ollama is a developer-friendly inference layer wrapping llama.cpp and similar engines. Easy to deploy; less production-grade than vLLM or Triton; good for development environments and smaller production workloads.
When to choose. Development and prototyping; smaller production deployments where operational simplicity matters more than peak throughput.
The Engineering Architecture
The reference architecture for production on-prem LLM deployments.
Layer 1 — Inference engine. vLLM serving the chosen model on GPU hardware. Multiple GPU servers behind a load balancer for production-grade availability.
Layer 2 — Inference gateway. Same architecture as cloud-hosted deployments. Single internal service through which all model calls flow. Adds audit logging, RBAC enforcement, schema validation, content-safety filtering, and routing logic.
Layer 3 — Application layer. Clinical AI features call the inference gateway. Application code is largely unchanged from cloud deployments because the gateway exposes an OpenAI-compatible interface.
Layer 4 — Monitoring and observability. Standard MLOps tooling — model performance monitoring, GPU utilization, inference latency, error rates, drift detection. Integrates with the hospital’s existing observability stack where possible.
Layer 5 — Lifecycle management. Model version management, deployment rollouts (typically blue/green), rollback capability, periodic re-evaluation, and security patching of the underlying infrastructure.
Hardware Sizing Reality
Hardware sizing is where most on-prem engagements get into operational trouble. Three dimensions drive the calculation.
Dimension 1 — Model size. A 7B-parameter model fits on a single consumer-grade GPU. A 70B-parameter model needs 4–8 enterprise GPUs at full precision, or 2× H100 80GB at INT8 quantization. The model choice determines the floor.
Dimension 2 — Concurrency. Throughput per GPU depends on model size, quantization, batch configuration, and inference framework. vLLM’s continuous batching delivers substantially better throughput than naive sequential serving — often 5–10× more requests/second on the same hardware.
Dimension 3 — Latency targets. Some clinical use cases tolerate seconds of latency (batch summarization, asynchronous note drafting); interactive copilots and ambient documentation require sub-second response, which pushes toward smaller models or more aggressive quantization.
Sizing ranges across our engagements.
- $80,000 for a single-server deployment of Llama 3 8B for a small hospital (one to two clinical use cases, modest volume)
- $150,000–$250,000 for a multi-GPU server running Llama 3 70B for a mid-size hospital
- $400,000+ for a multi-server cluster sized for a multi-thousand-clinician health system
Hardware costs are separate from engineering costs. The hardware refresh cycle (every 3–4 years for GPU infrastructure) is part of ongoing operating expense, not a one-time cost.
Pricing and Engagement Structure
| Engagement | Duration | Price Range | Scope |
| Discovery + Architecture | 4–6 weeks | $45,000 | On-prem readiness assessment, hardware sizing, model selection, deployment-pattern decision, ROI projection |
| Production-grade Deployment | 12–16 weeks | $110,000–$160,000 | vLLM serving, inference gateway, monitoring, audit logging, single-use-case integration |
| Multi-Use-Case Deployment | 16–32 weeks | $200,000–$400,000 | Multi-use-case shared infrastructure, fine-tuning pipeline (where applicable), MLOps maturation, operational handoff |
| Hardware (separate) | — | $80,000–$400,000+ | GPU servers, networking, storage, sized to model and concurrency requirements |
Total on-prem LLM deployment typically runs $300,000–$800,000 across engineering and hardware, with substantial variance depending on hospital scale and use-case scope.
Closing
On-prem LLM deployment in 2026 has moved from “the segment excluded from generative AI” to “a viable path with capability sufficient for most clinical use cases.” The model landscape is mature, the inference stack is production-grade, and the engineering patterns are well-defined. Hospitals with data-control postures that exclude cloud-hosted AI now have a path forward.
If you are scoping an on-prem LLM deployment for your hospital, book a 60-minute scoping call. Taction Software has shipped 785+ healthcare implementations since 2013, with 200+ EHR integrations across Epic, Cerner-Oracle, Athena, and Allscripts, zero HIPAA findings on shipped software, and active BAA paper trails with every major AI provider. Our healthcare engineering team builds production on-prem LLM deployments with the architecture described above as default scope. Our verified case studies cover the production deployments behind these patterns. For the engineering scope behind the engagement, see our healthcare software development practice and our hospital and health-system practice for the operational context. For the data integration patterns this work depends on, see our healthcare data integration practice. For an estimate against your specific use case, see the healthcare engineering cost calculator. For deeper context, see our broader generative AI healthcare applications work.