Building Trustworthy AI for Healthcare: Lessons from Heidi's Journey

In the rapidly evolving landscape of healthcare technology, artificial intelligence (AI) is poised to revolutionize patient care and administrative processes. However, creating AI-driven solutions that are both effective and compliant with stringent regulations presents a significant challenge. Australian-founded AI Care Partner, Heidi, has successfully navigated these complexities to deliver its flagship product, Heidi Scribe, which automates much of the administrative burden on clinicians. With a presence in over 190 countries and approximately 2.7 million patient interactions weekly, Heidi's journey illustrates the importance of robust infrastructure in supporting AI applications in healthcare.

Yu Liu, co-founder and chief technology officer at Heidi, emphasizes that in healthcare, even a small error rate can lead to severe consequences. This reality necessitates an architectural approach that prioritizes safety and compliance from the outset. As organizations strive to modernize their data infrastructures to support AI, Heidi's experience provides valuable insights into how to effectively balance innovation with regulatory requirements.

The Unique Challenges of AI Deployment in Healthcare

Deploying AI in healthcare differs substantially from other sectors due to the regulatory landscape that governs patient data and clinical processes. For companies like Heidi, a deep understanding of these regulations is paramount. Different countries have varying privacy laws, such as the Australian Privacy Principles, General Data Protection Regulation (GDPR), Act on the Protection of Personal Information (APPI), and Health Insurance Portability and Accountability Act (HIPAA). These regulations dictate that patient data must reside within the geographic boundaries of the country where the clinician operates, complicating data management strategies.

Heidi addresses these challenges by implementing fully isolated production deployments around the globe, ensuring compliance with local laws through architectural design rather than contractual promises. This approach not only enhances data security but also instills confidence in healthcare providers and patients alike.

Architectural Choices for AI Workflows

One of the critical decisions for Heidi was selecting the right database technology to manage the diverse and dynamic medical data that fuels its AI workflows. The company faced the challenge of consolidating a variety of data types, including forms, clinician notes, and referral documents, into a coherent and accessible format. Traditional relational databases, with their rigid structure, were ill-suited to accommodate the continuous changes inherent in healthcare data.

Heidi opted for a document-oriented database, specifically MongoDB, which provides the flexibility needed to adapt to the evolving demands of AI applications. This technology allows sessions to encapsulate various data artifacts, such as transcripts, structured notes, and patient context, without requiring constant database migrations. Liu states, "The model is maybe 20% of the system, and the data architecture is what determines whether the other 80% holds up under real clinical load." This flexibility is critical for maintaining operational efficiency and ensuring high performance as the system scales.

Building a Reliable Retrieval-Augmented Generation (RAG) System

As AI applications grow in complexity, the ability to retrieve and utilize relevant information becomes paramount, especially in healthcare settings where accurate clinical guidance is essential. Liu highlights that retrieval in healthcare is fundamentally a data architecture problem. Unlike consumer applications that may rely on open web data, healthcare organizations must source information from licensed clinical knowledge bases that adhere to compliance standards.

Heidi’s RAG system retrieves from reputable sources, such as BMJ Best Practice and NICE CKS, ensuring that clinicians receive guidance specific to their region. This jurisdiction-aware retrieval system protects against the risks of providing inaccurate information based on regional differences in medical practice. By embedding and indexing data within MongoDB Vector Search, Heidi ensures that compliance with residency laws is maintained, as all data retrieval operations occur within the same securely isolated environments.

Scaling Globally with Regional Isolation

Heidi’s architectural strategy involves deploying isolated systems across various regions, each with its own MongoDB Atlas clusters, computing resources, and security keys. This design not only facilitates compliance but also allows the company to scale efficiently across international markets. Liu notes, "Running multiple isolated regions with a lean team only works because the database layer is managed and consistent." This architecture has enabled Heidi to seamlessly enter new markets, such as the U.S., where the company quickly established a compliant deployment to support healthcare systems.

The successful integration of Heidi’s AI solution at institutions like Beth Israel Lahey Health demonstrates the effectiveness of this approach. Following a pilot program, 74% of clinicians reported reduced after-hours documentation burdens, illustrating the potential for AI to enhance clinician productivity and improve patient care.

Lessons Learned and Future Directions

Heidi’s experience offers several key lessons for organizations aiming to build data-heavy AI products. One crucial takeaway is the importance of making foundational decisions about data partitioning and architectural scaling early in the development process. Liu emphasizes that choosing a shard key is a design decision that can significantly impact the system's performance and scalability down the line.

Looking ahead, Heidi plans to expand its capabilities beyond consult notes to encompass the entire clinical workflow, facilitating processes from pre-visit context gathering to post-visit documentation and referral management. By leveraging its partnership with MongoDB and integrating large language models, Heidi aims to create an ecosystem that enhances clinical workflows while maintaining the trust of healthcare providers.

In the realm of healthcare AI, Liu asserts, "Reliability engineering is trust engineering." Maintaining clinician trust is paramount, as any downtime, latency issues, or data inconsistencies can rapidly erode confidence in the system. Therefore, Heidi invests heavily in invisible engineering practices, such as canary releases with automatic rollback, continuous integration gates on database changes, and cross-region consistency checks, all aimed at ensuring a stable and reliable platform for clinicians.

Key Takeaways

• Compliance is Crucial: Healthcare AI must prioritize regulatory compliance through architectural design.
• Data Architecture Matters: The choice of database technology impacts AI performance and scalability.
• Reliable Retrieval is Key: AI systems must retrieve information from compliant, licensed sources.
• Regional Isolation Enhances Trust: Isolated deployments ensure data residency and compliance across different markets.
• Trust Requires Investment: Reliability engineering practices are essential for maintaining clinician trust in AI systems.

Frequently Asked Questions

What is the significance of data residency in healthcare AI?

Data residency refers to the physical location where data is stored and processed. In healthcare, it is crucial because regulations such as HIPAA and GDPR dictate that patient data must remain within specific geographic boundaries. This ensures compliance with local laws and protects patient privacy. Companies like Heidi implement regionally isolated deployments to guarantee that data residency is enforced by infrastructure rather than contractual obligations.

How does Heidi ensure the reliability of its AI systems?

Heidi employs a variety of engineering practices designed to enhance the reliability of its AI systems. This includes implementing continuous integration gates on database changes, canary releases with automatic rollback mechanisms, and cross-region consistency checks. These practices help to prevent downtime and data inconsistencies, which are critical for maintaining clinician trust and ensuring patient safety in healthcare settings.

What role does MongoDB play in Heidi's architecture?

MongoDB serves as the backbone of Heidi’s data architecture, providing a document-oriented database solution that allows for flexibility in handling diverse medical data. This technology enables Heidi to consolidate various data types seamlessly and adapt to changing requirements without the need for constant migrations. Furthermore, MongoDB’s features, such as Vector Search, enhance the system's ability to perform real-time analytics and semantic searches, crucial for clinical workflows.

What are the future plans for Heidi in the healthcare space?

Heidi aims to extend its capabilities beyond just consult notes to support the entire clinical workflow, including pre-visit and post-visit documentation and workflow automation. The company is also exploring integrations with large language models and other technologies to create a comprehensive ecosystem that enhances clinical efficiency while ensuring compliance and reliability in patient care.