Everyone is talking about “agents.” If you spend five minutes on LinkedIn, you’ll see a dozen demos of AI agents supposedly “revolutionizing” everything from pizza delivery to hedge fund management. But in the world of high-stakes healthcare, a “cool demo” is a liability, not an asset.
A few months ago, a CTO of a mid-sized regional health system sat across from us. Their problem wasn’t a lack of tech; it was a surplus of friction. Clinicians were spending 40% of their day on documentation. Triage desks were overwhelmed. Patient data was trapped in legacy EHR (Electronic Health Record) systems that felt like they were built in 1995.
They didn’t want a chatbot. They wanted an agent that could actually do things: read patient histories, flag risks, and draft clinical notes that didn’t sound like a robot wrote them.
We said yes. This is the unvarnished story of what happened next, the parts people usually leave out of the case studies.
Chapter 1: The Discovery (The Mess Behind the Curtain)
We started where every project starts: the “Discovery” phase. In healthcare, discovery is less about brainstorming and more about forensic data archaeology.
We quickly realized that “clean data” is a myth. The client’s data was a chaotic mix of structured EHR fields, scanned PDFs of handwritten notes, and fragmented HL7 messages. To build an agent that could actually assist in triage, we couldn’t just “point it at the database.”
The agent needed to understand context. If a patient wrote “SOB” in an intake form, the agent needed to know that in a clinical context, that’s “Shortness of Breath,” not an insult.
The Strategy
We decided to build a bespoke AI agent centered on a Retrieval-Augmented Generation (RAG) architecture. But instead of just searching documents, this agent needed “tools”, specifically, the ability to query the EHR via FHIR (Fast Healthcare Interoperability Resources) APIs and perform reasoning over the retrieved data before presenting a summary to a clinician.
Chapter 2: The Architecture (Beyond the Chatbot)
Calling something an “agent” implies autonomy. In healthcare, “autonomous” is a terrifying word for a CTO. We had to build a system that was agentic in its processing but strictly supervised in its output.
We used a “Chain-of-Thought” reasoning loop. When a patient submitted a triage request, the agent would:
- Analyze the input: Is this an emergency? (If yes, bypass AI and alert a human immediately).
- Fetch context: Query the patient’s longitudinal record. Are there pre-existing conditions like asthma or heart disease that make “Shortness of Breath” more critical?
- Draft the Note: Using the clinical context, draft a SOAP note (Subjective, Objective, Assessment, Plan) for the clinician to review.
The technical stack was a blend of Python-based orchestration and specialized vector databases. We didn’t use a single “black box” model. We used a tiered approach: a smaller, faster model for initial classification and a larger, more capable model for the complex reasoning and documentation drafting.
Chapter 3: The “Holy Sh*t” Moments
About three weeks into the build, we hit the first wall.
In a laboratory setting, AI is perfect. In a clinical setting, AI encounters “The Human Element.” We found that clinicians all have different styles of note-taking. One doctor’s “Assessment” is another doctor’s “Plan.” If the agent drafted a note that didn’t match the specific doctor’s style, they didn’t just edit it: they ignored it entirely.
We had to build a style-transfer layer. We indexed historical notes from specific departments so the agent could mimic the local clinical vernacular. This wasn’t “fluff”; it was the difference between adoption and shelfware.
Then came the HIPAA hurdle. You can’t just send PHI (Protected Health Information) to any old API. We had to ensure every single touchpoint: from the vector database to the inference endpoint: was within a BA (Business Associate) agreement-covered environment. We spent more time on VPC (Virtual Private Cloud) configurations and encryption-at-rest than we did on the actual LLM prompts.
Chapter 4: The Integration (The Last Mile)
This is where 90% of AI projects die. Integration.
The client’s EHR didn’t have a “Plug and Play” button for AI agents. We had to work with legacy APIs that were temperamental at best. We spent weeks writing middleware to handle the “handshake” between our AI agent and the system of record.
We also had to design the Human-in-the-Loop (HITL) interface. We learned quickly that if you give a doctor a finished note, they are prone to “automation bias”: they might skim it and miss a subtle error. We redesigned the interface to highlight the specific parts of the patient record the agent used to reach its conclusion. We turned the agent into an “Evidence-Based Assistant” rather than a “Magic Box.”
Chapter 5: The Results (The Real Numbers)
After six weeks of piloting in a controlled environment, we looked at the data.
- Documentation Time: Decreased by 32%. Clinicians were able to “finalize” notes in minutes rather than hours at the end of their shifts.
- Triage Accuracy: The agent’s risk-flagging matched the senior triage nurse’s assessment in 94% of cases. In the remaining 6%, the agent was actually more conservative, flagging potential issues the nurse (due to high volume) had initially overlooked.
- Clinician Burnout: While harder to quantify, the qualitative feedback was clear: “I don’t have to stay until 8:00 PM doing paperwork anymore.”
But it wasn’t all sunshine. We found that the agent still struggled with extremely rare pediatric conditions: edge cases where there simply wasn’t enough historical data to form a reliable pattern. Our solution? We programmed the agent to “know what it doesn’t know.” If it encountered a high-uncertainty case, it simply stopped and handed it off to a specialist with a “High Uncertainty” flag.
The Unvarnished Truth
Building an AI agent isn’t about the AI. It’s about the plumbing.
If you’re a CTO or a Head of Innovation looking at AI, don’t ask “Which model should we use?” Ask:
- “Where is our data trapped?”
- “Who has to approve the ‘Last Mile’ of this decision?”
- “Is our infrastructure ready for the security requirements of PHI?”
At Newaiv, we don’t believe in off-the-shelf “solutions” for complex industries. Healthcare doesn’t need more “wrappers”; it needs bespoke systems that respect the reality of the clinic.
We built an agent. It was hard, it was messy, and it required more engineering than “prompting.” But for that client, the result wasn’t just a tech upgrade: it was a return to what they actually care about: treating patients instead of filing paperwork.
If you’re ready to stop watching demos and start building something that works, let’s talk.
