AI Product Manager Interview Question 2

Prepare for AI PM Interviews

We will keep on discussing AI PM Interview Questions asked recently → If you want to stay updated, you can consider Subscribing

You can also visit our previous post on

Question

You are a Product Manager and have been tasked to improve the discovery of available doctors in your hospital.

The underlying problem is that many users come on the hospitals website, browse through the directory of doctors but are not making an appointment.

Data for your Observation

1. The number of unique visitors on your website is steadily increasing
2. Doctor search queries per user are also increasing
3. The number of appointments per visited user is declining sharply

Understanding the Problem

Looking at this data, it is very clear that users are also coming to the platform and showing intent to search.

Appointments are not happening because the website is not able to provide accurate results for search queries [Assumption — The appointment booking workflow is working fine]

Deep Dive & Possible Approaches -

Upon doing the query analysis, you were able to find that a large chunk of the user queries are coming in the form of natural language, such as:

• Recurring pain in the legs
• Chest pain due to non-heart-related issues

There are two ways to solve the search problem -

Keyword-Based Search

This is a very traditional search method wherein keywords extracted from your search queries are matched with a catalogue of available doctors in your hospital. The underlying issues are -

The doctor’s catalogue will talk about the specialisation of the doctor, but the user queries talk about the symptoms

For example,

1. Doctor Specialisation — Obstructive Sleep Apnoea
2. User Query — Repeated pauses in breathing during sleep

Though the user query & the doctor specialisation have the exact match, but the keyword search will not help in answering the user query

RAG enabled LLM-powered search

This is a modern approach where we will utilise the power of LLMs, i.e. superior language skills, reasoning abilities, while restricting the knowledge set to our latest doctor’s directory.

This modern approach will help us in searching effectively and solving the following aspects of our problem

1. Understand the natural language of the user
2. Match the user-provided query to the doctor’s catalogue
3. Restrict the responses to your doctor’s directory ( An LLM without RAG can provide you a generic answer, more about this later in this article)
4. Hence, in the end, provide meaningful search results, taking your appointment bookings order up

Understanding of RAG-Based LLM Search

In order to achieve the stated goal using RAG following steps need to be followed.

Creation of dataset embeddings

As you might be aware, LLMs are trained on the world’s knowledge and not necessarily on the doctor’s directory of your hospital. Even if they were trained in your doctor’s directory, their understanding of it might not be the latest.

Hence, if we were to rely on generic LLMs, there are good chances that your revamped search will be worse off than the keyword match search.

Converting the user query in embeddings -

Finding the answer from the database using semantic search -

So once you have done that, then we need to measure the Search Relevance, how many people who are trying to search end up booking the appointment.

If you want to understand Embeddings in detail, you can read

Resources

Course - Crack PM Interview Like top 1% ( Using First Principle Thinking )

1. What is Agentic AI and future Strategy?
2. Why is Perplexity trying to own the gateway to the internet?
3. How is LLM changing the Search Landscape?
4. Why Google / Microsoft are putting a strategic bet on Quantum Computing
5. What is Apple AI Play? Are they running delayed?

About Me

Hey, I’m Shailesh Sharma! I help PMs and business leaders excel in Product, Strategy, and AI using First Principles Thinking. For more, check out my Live cohort course, PM Interview Mastery Course, Cracking Strategy, and other Resources