AI at Woebot Health – Our Core Principles


August 2023
By: Joe Gallagher, Chief Product Officer and Sharanya Srinivasan, Senior AI Product Manager

With recent advances in AI, it’s more relevant than ever to ask ourselves what it means to responsibly and effectively use AI, especially Large Language Models (LLMs), in a healthcare setting. Deploying AI to vulnerable populations comes with the potential to inadvertently cause harm. To responsibly deploy any form of AI in our products, we must deeply and intentionally consider the impact that the technology has on users. We’ve outlined below how we think about the management and mitigation of the risks that accompany the use of AI in the mental healthcare field.

We recently announced the launch of our first clinical trial exploring LLM technology*. The goal of this study is to help us understand how LLMs can be applied to accelerate the delivery of safe, engaging, and potent digital solutions for mental health. As these technologies mature, we expect to better understand what constitutes safe and effective development practices, alongside regulatory guidance. It is also important that companies in this space proactively adopt and share effective practices.

Given the broad capabilities of LLMs, we anticipate a spectrum of possible use cases within our products, each requiring due consideration to ensure clinical appropriateness, safety and augmentation to the overall experience. We broadly categorize these into (a) using LLMs to interpret user input and route to content that has been written by a human—essentially using the LLMs only for understanding; and (b) using LLMs to generate responses that will be shown directly to users. 

While generating responses is a use case that requires additional scoping and safety considerations, we believe that using LLMs for understanding intent combined with human-composed responses provides for improved conversational quality. We currently are learning more about the value and risks of response generation, and we therefore only use generative capabilities in IRB-regulated study settings.

All of this is to say that using LLMs will augment our ability to understand natural language from users and better enable us to meet them where they are. As we begin to explore the usage of LLMs in our product and think about how to apply those learnings in future versions of Woebot, we continue to hold ourselves to high standards of safety, rigor and transparency. The following outlines our core principles around responsibly deploying AI, including LLM-based technology, to our users. 

Safety

  • User safety is our top priority when deploying any form of AI. When LLM-based technology is used in conversation, we have several guardrails in place to ensure safety. This set of safety features detailed below has allowed us to use our own well-validated NLP and our strong conversational design principles as added layers to bolster and stabilize applications of LLMs. More details on our approach to safety can be found on our safety page.
  • Users never interact directly with LLMs. A number of input and output checks and validations are performed on both user messages and model output when invoking LLMs. Our proprietary Concerning Language Detection algorithm runs on user input before it is passed to an LLM. We use a proprietary prompt architecture that was designed to prevent prompt injection attacks. In order to minimize the risks of unwanted LLM behaviors like hallucination, we have also implemented other guardrails such as off-topic identification, maximum turn enforcement, and output validation to keep our interactions with the models targeted and succinct.
  • Where LLMs are used for understanding intent, the user never sees any AI-generated content. The most frequent use case of LLMs is to classify user input into one of a given set of categories and use that classification to route the user to content that has been written by a human, mitigating any risk that would stem from unexpected model outputs. All content written at Woebot Health is designed in collaboration with conversational design, clinical, and translational science experts.
  • Woebot behaves in alignment with our core design principles. When generative capabilities of LLMs are being studied prior to inclusion in commercial products—for example, to provide personalized empathy in response to a user input—our prompts are designed to ensure alignment with our core design principles: for example, we do not diagnose, use language that could be considered offensive, or deliver medical advice.
  • The user is in charge of the conversation. Separate from our integration with LLM technology, we have several conversational principles that further reduce the risk of safety issues. Woebot was designed to not make assumptions. Where AI is used to interpret user intent, users retain agency in terms of what the next step in their conversation is.

Rigor​

  • AI is developed in collaboration with clinical and translational science experts. We only infuse AI into Woebot after thorough research and decision-making in partnership with our team of clinical and translational science experts. We will never deploy any form of AI just for the sake of using it; we test and use AI carefully, and only in service of better outcomes for users. That principle applies to any tools, including LLMs.
  • We test rigorously and continuously. Before we deploy any AI, it is thoroughly evaluated by our team of engineering and clinical experts. We perform a combination of manual and automated tests to validate performance and safety. LLM-based technology is tested especially rigorously with an emphasis on evaluating performance across a diverse set of user personas. Once a model is deployed, regularly evaluating its performance and drift and retraining when necessary allows us to maintain confidence in the reliability of our work.
  • We prioritize learning and keeping Science in the Loop. What we learn from the LLM study will help us develop and validate safe ways to deliver more potent and engaging interactions for users, and identify where LLMs could help us scale and study future versions of our products. That study is our 18th RCT, and it continues a long history of generating scientific evidence.  

Transparency​

  • We are committed to driving transparency in our design and implementation of machine learning technology. When using any model, whether it is an LLM or not, it is important to know where exactly it is used, what outcomes are driven by its output, and what fail-safes are in place. We can stay accountable to our users and to ourselves by thoroughly documenting and versioning our models. 
  • We are transparent about how user data is processed, used and stored. This is especially relevant when we use data from users to train our self-hosted models. We follow the principle of least privilege when accessing user data internally. Finally, we always process user data in accordance with our Privacy Policy and Terms of Service. Where LLMs are used, user data is shared with 3rd parties in order to provide our service, but the 3rd party is prohibited from using the data outside of the agreed-upon service they are providing to us. The 3rd party cannot use user data for its own purposes or to train its own models. This allows us to provide the service without compromising users’ data privacy rights.

We are excited and optimistic about the potential of LLMs and are increasingly confident in our ability to build the safest approach to applying AI in healthcare. We will accomplish this by doing what we do best – taking emerging technology and applying rigorous and continuous user research, engineering, clinical and translational science oversight to create more precise and personalized products.

*W-DISC-MVP is an investigational medical device. It has not been evaluated, cleared, or approved by the FDA.

Woebot Health Platform is the foundational development platform where components are used for multiple types of products in different stages of development and enforced under different regulatory guidelines.

Woebot does not provide crisis counseling and is not a suicide prevention or crisis intervention service. Concerning language and escalation data is not reviewed or assessed internally at Woebot Health in real-time for intervention and such data is not used for managing potential crises or any acute or non-acute patient safety issue. Discomfort may be experienced when answering sensitive questions. Temporary upset may occur as a result of discontinued access to Woebot Health Platform.

Note, August 2023: Our research into large language models (LLMs) is exploratory. In our commercially or publicly available products, we do not use LLMs to generate responses to users; all text is developed by our conversational writers, and always with clinical oversight.