Electronic Health Records are only one of the core technology solutions a healthcare organization uses. Whether it’s scheduling, billing, or an AI solution, provider organizations might have anywhere from 2 to 200 additional digital health solutions, depending on whether you run a private practice or a health system. Choosing a vendor is often a time consuming process, but one that is worth it when you get it right.

At the beginning of 2024, Healthie’s recently launched marketplace called the Harbor, had its team lead an initiative to review AI solutions. Here’s a breakdown of how we formed a strategy to analyze our customer’s main needs and evaluate the market.

First, identify exactly what you’re looking for.

When we started the project, the team set out to evaluate the “AI Copilot” tools that exploded onto the market in recent years. But,  rather than simply evaluate  a category of tools, the evaluation process needed to be focused on the pain point we needed to solve for our customers. 

As a former consultant, our Director of Marketplace has seen firsthand that healthcare organizations can get caught up in the category of tools and lose track of what admins and clinicians are actually asking for. Unfortunately, iIt’s often at the end of the process that end users are brought in for feedback. For example, the category of tool being purchased may not always be the best tool to solve the pain point. For example, patient self-scheduling tools aren’t necessarily the answer to reduce call center volume related to rescheduling - an appointment reminder system that more easily gives the patient the ability to self-reschedule can sometimes show better results and be more cost effective.  In order to identify the exact pain points to address,, we reviewed a number of product feedback insights we received directly from customers and held discovery calls with a variety of customers and stakeholders  to understand their pain points.

This research helped us narrow our focus: 

“What’s the best tool that can help clinicians write their chart notes?”

Second, identify who is the best team to make this decision?

If possible,  bring in a variety of team members when making vendor decisions. At Healthie, we’re evaluating which AI solution to embed into our platform to help create an ideal experience for our customers. In order to evaluate the solutions from multiple points of views, expertise, and experience, we included team members from the following departments:

• The Harbor team (experience in integrations and partnerships)
• Product (how will this tool fit into our product experience)
• Design (how will this tool impact UI and UX)
• Engineering (how well can we integrate this solution)
• Customer Success (brings the voice of the customer)

To gather more clinical input, as mentioned we interviewed a number of customers from various specialties to learn more about the solutions they’ve evaluated on their own, decision criteria, and more. By including a variety of team members, you’ll have a better chance of removing bias and accounting for factors that an individual or smaller team may not be able to address alone.

After determining your goal and forming a team,  create a timeline for the project and kickoff. Unless you’re evaluating vendors to replace an existing vendor or meet another hard deadline, it may not be critical to wrap up the evaluation process by a hard deadline; however, there are several downsides to not setting hard dates throughout the process. As each step relies on the step before it, RFPs can drag out notoriously long. Considering that healthtech solutions can take months to implement, it’s important that a decision is made well ahead of the need to implement the solution. An easy way to pick a deadline is by reversing engineering your schedule from the data of expiration of your existing contract. 

Third, research the market for participants and create your selection criteria.

Prior to starting  product demos and thorough product investigations, create a standardized way to evaluate all the companies. Without a standard set of evaluation criteria, it can be easy to forget to ask a question during a demo or to evaluate the companies based on a consistent set of criteria overall. It’s also important to evaluate the companies on a similar timeline. If you’re able to see demos of all the solutions within one or two weeks, you’ll be more easily able to compare them than if participants enter the process slowly.

At Healthie, we evaluated the market and found about a dozen participants we wanted to evaluate based on the following criteria:

• Company Background
• Security and Data Privacy
• Commercial (Partnership Details)
• The Note (Note quality, customization, the LLMs and AI that go into creating the note, and more)
• Workflow Impact
• Integration Capability
• Additional Considerations

Additional considerations encompassed anything else that we specifically cared about, from accessibility to contacting references to pilot programs.

Fourth, let the evaluation process begin!

Before involving each company in the process, conduct a quick discovery call with the potential partner to evaluate whether they’re a good fit for the process. At Healthie, we discovered that two well-known companies in the space would not be a good fit. There were simple explanations for this, and it saved a lot of time on both sides. For example, we learned that one of the companies was focused on a certain type of healthcare organization where the majority of their visits occurred in an inpatient setting. At Healthie, our customers are focused on outpatient and virtual care settings, so partnering with a tool focused on inpatient care settings wouldn’t make sense.

How will you evaluate each company? We asked each willing company to conduct a one-hour demo and complete a small sheet of questions categorized by the selection criteria mentioned above. During each demo, it can be helpful to have each member of your selection committee use the RFP as a guide for evaluating the demo. It can be helpful for your team to grade the demos live rather than relying on memory.

It’s important that each selection committee member grades from their own document and is blinded from other members’ grades until the process is completed. You want to remove bias. It’s better to have someone NOT score a criterion they feel unqualified to than have them assign a grade based on other team members’ grades.

Throughout the process, we noted a couple of key areas that separated AI companies:

• Where are they primarily being used? Inpatient vs. Outpatient
• What specialties are using them?
• Are they primarily focused on complex medical specialties like neurology and cardiology, or on mental health visits being used in hour-long therapy sessions?
• Product considerations for different specialties

Healthie has a variety of specialties on the platform, one of the largest is mental health. We identified companies that severely struggled to listen to appointments for longer than 40-45 minutes.

It was also clear which companies had extensive use in mental health and who did not. For example, many  AI tools are trained to focus on medically relevant audio from the visit and they’ll ignore miscellaneous small chat. For example, if I start talking about how my best friend made me upset last week during a podiatry visit, it’s rather irrelevant. However, in a mental health context, it’s keenly important. It’s important the AI team can adapt for your specialty or is a tool specifically designed for your specialty.

Workflow Considerations:

• How does the AI tool capture audio from the visit? Some capture audio from the browser, others from Zoom, and still others are done by opening an app on your phone to record the audio. In an integrated scenario, you may be able to bypass some of these workflows; however, if you can’t, some can be disruptive or just awkward.
• In-person workflows don’t always work well in virtual care settings. In an in-person appointment, having your phone record the audio to write the note for you would be straightforward; however, if you’re conducting 6 therapy sessions in a day, the phone would be listening to your audio and then listening to your laptop’s speakers to hear the patient. These workflow adjustments can make a big difference in training providers.

Finally, come together as a committee to review how you graded each participant.

At Healthie, we graded each partner on the criteria and sub-criteria on a scale of 1 to 5, with 1 being the worst and 5 being excellent/beyond expectations. This helps create a scoring system that, when tallied up, can provide a tangible score to compare across participants. In the scoring criteria, we only allowed one 5 to be given per item being graded. For example, if we’re evaluating each participant on the customization of the chart note, only one company could receive a 5. Other well-performing participants would have to settle for a 4.

Before selecting a vendor, put together a real-world test. This could involve using their sandbox environments to test the promised integration capabilities or a production environment to test out the product with a small number of pilot users. These tests should be performed with a shortlist of vendors. At Healthie, we narrowed our list from about a dozen down to 3.

To recap: 

• Identify the exact reason for the project: What is the pain point you're trying to solve?
• Identify the best team to execute the project.
• Identify the companies you’re interested in and establish your selection criteria.
• Evaluate the companies, both via live demos and written responses.
• Shortlist your vendors down to the top few selections to perform testing or a pilot program. 

Going through an RFP process is an incredibly complex process to get right and we couldn’t include every detail here. If you have questions about evaluating a tool for your organization, check in with our Harbor team at marketplace@gethealthie.com