If you've been looking at AI tools for therapy notes, you've probably seen the phrase “local AI” or “on-device AI” and wondered what it actually means. It sounds like a technical distinction. It isn't. It's a fundamental difference in what happens to your clients' information, and for therapists specifically, it matters more than for almost any other profession.
The difference in plain terms
Every AI note tool works by taking your session audio, turning it into a transcript, and using that transcript to generate a structured progress note. The question is where that processing happens.
With a cloud-based tool, it happens on a server: a computer owned and operated by the software company, located somewhere else, accessed over the internet. Your session recording travels from your device to their server, gets processed there, and the note comes back to you. The vendor now has a copy of your client's words on their infrastructure.
With a local AI tool, all of that processing happens on your computer. The AI model is downloaded and runs on your device. The transcript is generated on your device. The note is drafted on your device. Nothing leaves your machine. There is no server. There is no third party.
That's the whole distinction. It's simple, but the downstream implications for therapy are significant.
Why this became possible recently
Until a few years ago, running a capable AI model on a laptop wasn't realistic. The models that could actually produce useful clinical notes were large and computationally demanding, and they needed the kind of processing power that only existed in data centers. Cloud-based tools weren't a privacy trade-off. They were the only viable option.
Two things changed. Consumer hardware got significantly more powerful, with modern laptops now carrying the kind of processing capability that only existed in servers a few years ago. And the AI models themselves got smaller and more efficient, with researchers figuring out how to get high-quality results from models that could actually fit and run on a personal computer. Those two trends converged, and local AI went from impractical to genuinely usable.
Whisper, the speech recognition model developed by OpenAI and released publicly in 2022, was an early signal. It runs on-device quickly enough to be practical for clinical use. The language models used for note drafting have followed the same trajectory: smaller, faster, and good enough that the quality difference from cloud models has largely disappeared for structured tasks like progress notes. Local AI for therapy notes is possible now in a way it simply wasn't before.
Why it matters for therapists specifically
A lot of professions are adopting AI tools that process data in the cloud, and for many of them the risks are manageable. For therapists, the stakes are different.
What clients say in therapy is often the most sensitive information in their lives. Conversations about trauma, mental health diagnoses, relationship dynamics, past behaviors, and things they haven't told anyone else. The therapeutic relationship depends on clients feeling genuinely safe. Not just legally safe, but actually safe. When a client learns that their session was uploaded to a server operated by a company they've never heard of, that feeling of safety is harder to maintain. For some clients, it changes what they're willing to say.
There are also concrete practical risks. Cloud data gets breached. Companies get acquired. Terms of service change. De-identified session transcripts get used to train models. These aren't hypothetical risks. They're things that happen in the normal course of running a software company. With a cloud-based note tool, your clients' information is subject to all of it. With a local tool, it isn't. There is no server to breach. There is no policy change that affects what happens to your data. The vendor could go out of business tomorrow and nothing about your clients' privacy would change.
What it means practically
No BAA required
A Business Associate Agreement is the HIPAA contract you need with any vendor that handles your clients' protected health information. With a local AI tool, there's no vendor handling PHI. The data never leaves your control, so there's nothing to contract around. This simplifies your compliance posture considerably, though you still need to handle your device and recordings responsibly.
A simpler, more honest consent conversation
Informed consent for AI-assisted documentation is required before you use any tool with clients. With a cloud-based tool, that conversation involves explaining servers, third-party vendors, data retention policies, and HIPAA protections that clients can't easily verify. With a local tool, the conversation is shorter and more honest: your session is recorded on my laptop, an AI on my laptop helps me draft the note, the recording is automatically deleted after a few days, and nothing goes anywhere else. Most clients can genuinely get behind that.
Your data is independent of the vendor
With a cloud-based tool, your clients' privacy depends on what the vendor is doing with their data right now, what they might do after the next funding round, and what happens if they get acquired or shut down. With a local tool, the vendor's decisions don't affect your clients' information, because their information was never on the vendor's systems to begin with. You can switch tools, stop using the tool, or have the company cease to exist. None of it changes anything about what happened to your clients' data.
How to tell if a tool is actually local
Some tools use “local” loosely, or have hybrid architectures where some processing is on-device and some isn't. Before assuming a tool is fully local, ask:
- Does session audio ever leave my device? Under any circumstances?
- Is the AI model downloaded to my machine, or does it run on your servers?
- Is transcription done on-device or in the cloud?
- Where is the note draft generated?
- Does the app require an internet connection to function?
A genuinely local tool can answer all of these directly. If a vendor is vague about where processing happens, assume it's in the cloud.
You can verify it yourself
You don't have to take any software company's word for what their tool does with your data. There's a free utility called Charles Proxy that lets you see every network request leaving your computer in real time. It's the same tool developers use to debug APIs, and you can use it to watch exactly what a piece of software is sending over the internet.
Open Charles Proxy, then run a session in a local AI note tool. What you'll see is a brief flicker of activity at the start: a check for software updates, a license validation to confirm your copy is registered. After that, the network log goes quiet. Record a session, generate a note, review it. Nothing.
Now do the same thing with a cloud-based note tool. You'll see a constant back-and-forth throughout the session: audio being uploaded, transcription requests going out, responses coming in. The tool is doing exactly what it says it does, which is to say, your session is leaving your computer the entire time.
This is what “local” means in practice. Not a privacy policy. Not a reassuring marketing page. A network log that goes silent when you're doing clinical work.
The real trade-off
Local AI isn't without limitations, and it's worth being honest about them.
It requires a capable machine. Modern computers handle local AI well. Older hardware may not have enough processing power to run these models at a practical speed, so if your work computer is several years old, it's worth checking whether local AI is supported before committing to a tool.
Local AI also isn't connected to the internet, so it can't look things up or update itself automatically. The model does what it was trained to do, and that's it. For note drafting, this is fine: the task is contained and doesn't require real-time information. But it's a difference worth knowing about.
Confidant is built on this model. The AI runs entirely on your computer: transcription, note drafting, everything. Session audio never touches a server. It was built this way because for therapy specifically, the privacy architecture shouldn't be an afterthought. For more on the compliance side, this post covers what HIPAA actually requires for AI note tools. And for the informed consent conversation with clients, this post has sample scripts and what to put in writing.