The short answer: A regulated firm can use large language models without sending sensitive data to a public model. The model itself isn't what decides that. What decides it is where your data is allowed to go, who can see what the model sees, and whether you can prove both to an auditor. Pick the least complex deployment that keeps regulated data inside a boundary you control. Rent the model, and own your data, workflows and evaluation.
Why regulated firms stall on generative AI
Most of the executives I talk to in healthcare, finance and real estate aren't asking whether generative AI is useful. They already know it is. The questions that hold them back are harder. Where does patient or client data go once someone pastes it into a prompt? Who is accountable when a model gets something wrong? What will we show an examiner or an auditor when they ask?
Those are the right questions. When firms stall, it's usually because they try to answer them all at once, for every use case. My job as Chief Technology & Innovation Officer at Leon Capital Group is to break them apart. The technology, cybersecurity and engineering teams I lead need a clear line between what is safe to run on a public service and what has to stay private.
What "private LLM" actually means
"Private LLM" isn't one thing. It's a range of options, and each step along it gives you more control in exchange for more work.
| Option | Where the model runs | Where your data goes | What you own | Typical fit |
|---|---|---|---|---|
| Public API under enterprise terms | The vendor's cloud | The vendor processes it under contract | Prompts, workflows | Low-sensitivity work and early pilots |
| Managed model inside your cloud (VPC) | Your cloud tenancy, run by the provider | Stays in your cloud boundary | Network, access, logs | Most regulated workloads |
| Self-hosted open-weight model | Your own infrastructure, on-prem or in your VPC | Never leaves your environment | The whole stack | The most sensitive data, or strict residency rules |
| Retrieval (RAG) or fine-tuning on top of any of these | Wherever the base model runs | Your documents stay in your own index | The data layer and the evaluation | Almost every serious use case |
The last row is the important one. Retrieval and fine-tuning aren't a separate option. They're the layer you build on top of whichever model you choose, and it's where most of the value ends up.
Buy the commodity, build the differentiator
At DFW Startup Week this August, I summed up my approach in one line: "Buy the commodity, build the differentiator." I treat foundation models as a rented utility. The part worth owning is your data, your workflows and your orchestration.
For a regulated firm, that gives you a clear split. Buy the models, the GPUs and the managed platform, because those keep getting better and cheaper and you won't out-engineer the vendors on them. Build the things nobody can sell you: clean and permissioned data, the retrieval layer that knows which documents a given user may see, the workflow that puts a human at the right step, and the evaluation set that tells you whether the system is getting better or worse. If a vendor changes its pricing or a better model comes out, you swap the commodity and keep everything you built.
A decision framework
For each use case, not for the firm as a whole, I work through five questions.
- How sensitive is the data? Public, internal, confidential, or regulated (health information, personal financial data, material non-public information). The most sensitive data in the workflow decides where it can run.
- What do latency and cost allow? A chat assistant for a few hundred staff and a batch job over millions of documents need very different setups. Price it per task, not per model.
- What skills do you have in-house? Self-hosting a model means you patch it, scale it and monitor it. If you don't have that team, a managed deployment inside your cloud is usually the more honest choice.
- How exposed are you to vendor lock-in? Keep prompts, retrieval and evaluation portable, so you can change models without rebuilding.
- What's the regulatory surface? In healthcare, that means rules like HIPAA. In finance, it means SEC and FINRA recordkeeping and supervision. Across all of them, frameworks like SOC 2 shape what your customers will ask of you. I'm not giving legal advice here, and your counsel and compliance team should own the reading. But the architecture has to make their answers provable.
A reference architecture
The design I keep coming back to has five layers.
- Data boundary. Regulated data stays in an environment you control. Network rules, encryption and contract terms make sure of it.
- Retrieval layer. Documents are indexed with their access permissions, so the model only sees what the person asking is allowed to see.
- Guardrails. Input and output filters for sensitive data, blocked topics and policy checks, placed before and after the model.
- Logging and audit. Every prompt, retrieved source, response and user action is recorded and kept according to your retention policy.
- Human in the loop. Anything that touches a client, a patient, a trade or a filing passes a named reviewer before it takes effect.
Picture it as a box drawn around your data. The model sits inside the box or behind a private connection to it, and nothing crosses the edge without being logged.
The security side
As someone who also heads cybersecurity at Leon, I treat an LLM as a new attack surface, not just a new tool. Most of the trouble comes from weak boundaries around the model, not from missing features. The risks I plan for are these:
- Prompt injection. Instructions hidden in a document or email that try to take over the model. Treat everything retrieved as untrusted input.
- Data exfiltration. A model that can read sensitive records and also send data out is a leak waiting to happen. Separate those two abilities.
- Access control. The assistant should never see more than the user it's serving. Pass the user's identity all the way through to retrieval.
- Red-teaming. Attack your own system before it launches, and again after every major change.
Five questions a board should ask
- Which of our data is this system allowed to touch, and how do we know?
- Who is accountable for its outputs, and where does a human sign off?
- How do we measure whether it's accurate, and who reviews the results?
- What would an examiner or auditor see if they asked tomorrow?
- If we had to change vendors next quarter, what would we lose?
If management can answer all five plainly, the rollout is ready for a pilot.
Industry snapshots
Healthcare. Health information sets the boundary. Back-office work like intake, document handling and revenue-cycle tasks is usually a better first step than anything clinical. I've written more about the technology challenges in healthcare investment.
Capital markets. The hard requirements are recordkeeping and supervision. Every AI-assisted communication or research summary has to be captured and reviewable. That's covered in how AI is changing capital markets in 2026.
Real estate. Leases, property data and investor documents are full of sensitive terms. Retrieval with tight permissions does most of the work here. More in real estate technology and AI automation.
Where to start
Pick one use case with clear value and data that's sensitive but well understood. Put it inside your own boundary, log everything, and measure it against a real evaluation set before you scale. Keep the commodity swappable and invest in what's yours.
I've spent more than 24 years in technology, from solution architecture at MAF Holdings in Dubai to the CTIO role at Leon, and the firms that do this well treat AI as an operating discipline, not a demo. If you're working through these decisions, you can read more about how I approach enterprise AI strategy in Dallas, and I cover this ground in my talks on practical AI. The best way to reach me is a message on LinkedIn.
FAQ
Is a private LLM HIPAA compliant?
Not on its own. Compliance depends on the controls around the model: where the data lives, who can access it, the contracts in place, and how activity is logged. A private deployment makes those controls easier to put in place and to prove.
Do we need to self-host an open-weight model to keep data private?
Usually not. For many regulated workloads, a managed model running inside your own cloud keeps data within your boundary without your team having to run model infrastructure. Self-hosting makes sense for the most sensitive data or strict residency rules.
Should we fine-tune a model or use retrieval?
Start with retrieval. It keeps your documents in your own index, respects permissions, and updates as soon as the source data changes. Fine-tune only when you need a consistent style or format that retrieval can't give you.
What's the first step for a regulated firm?
Classify your data, pick one well-scoped use case, and define how you'll measure accuracy and keep an audit trail before you choose a model.