MIS 752 · Lab 5 · James Seegel
Speak the Patient's Language: semantic search vs keyword search
A measured comparison of semantic and keyword search using a course-provided teaching corpus.
My takeaways
Sentences within the same topic averaged 0.235 cosine similarity, compared with 0.129 across topics, showing some separation between subjects. Keyword search handled “my refill is not ready at the pharmacy” because “refill,” “ready,” and “pharmacy” appear in the matching sentence. For queries without shared vocabulary, keyword MRR was 0.000 versus 0.800 for semantic search. An empty search page could leave patients without useful information even when the right material exists. Overall, semantic MRR was 0.917, compared with 0.500 for keyword search. However, semantic search missed “the reading on my meter jumped since supper,” returning car5, dia5, and car2 instead of the relevant diabetes sentences. I think “meter” was ambiguous, and the model did not connect the meal reference with blood sugar. In an IT help desk, semantic search could connect “it keeps kicking me out” with “session timeout.” I would test similar phrases requiring different solutions before relying on the results.
Measured results
Model: sentence-transformers/all-MiniLM-L6-v2
Sentences: 20
Embedding dimensions: 384
Evaluation queries: 12
Results requested per query: 3
Mean cosine within a topic: 0.235
Mean cosine across topics: 0.129
Difference: 0.106
precision@k recall@k MRR
method
keyword (word overlap) 0.375 0.319 0.500
semantic (embeddings) 0.417 0.542 0.917
Similarity heatmap
Search comparison
Example query
chest hurts when i climb stairs
rank id topic voice score text
1 car1 cardiac patient 0.7874 My chest feels heavy and tight when I walk up the stairs.
2 car2 cardiac clinical 0.3919 Patient reports exertional chest pressure radiating to the left arm.
3 car4 cardiac patient 0.3740 I get out of breath and my heart pounds for no reason at all.
Per-query results
query query_type method retrieved p_at_k recall_at_k rr first_rank
chest hurts when i climb stairs patient guessed our words keyword (word overlap) car1,car2 1.000 0.667 1.0 1.0
chest hurts when i climb stairs patient guessed our words semantic (embeddings) car1,car2,car4 0.667 0.667 1.0 1.0
my refill is not ready at the pharmacy patient guessed our words keyword (word overlap) med5 1.000 0.500 1.0 1.0
my refill is not ready at the pharmacy patient guessed our words semantic (embeddings) med5,dia4,car5 0.333 0.500 1.0 1.0
is it safe to take ibuprofen with warfarin patient guessed our words keyword (word overlap) med1,dia1,med2 0.667 1.000 1.0 1.0
is it safe to take ibuprofen with warfarin patient guessed our words semantic (embeddings) med1,med2,med5 0.667 1.000 1.0 1.0
i feel low and cannot sleep patient guessed our words keyword (word overlap) men1,men3 0.500 0.333 1.0 1.0
i feel low and cannot sleep patient guessed our words semantic (embeddings) men1,dia3,dia5 0.333 0.333 1.0 1.0
it aches to climb to the second floor patient's own words keyword (word overlap) (nothing) 0.000 0.000 0.0 NaN
it aches to climb to the second floor patient's own words semantic (embeddings) car1,dia5,men1 0.333 0.333 1.0 1.0
the reading on my meter jumped since supper patient's own words keyword (word overlap) (nothing) 0.000 0.000 0.0 NaN
the reading on my meter jumped since supper patient's own words semantic (embeddings) car5,dia5,car2 0.000 0.000 0.0 NaN
would the anti-inflammatory clash with what thins my blood patient guessed our words keyword (word overlap) car5,med1 0.500 0.500 0.5 2.0
would the anti-inflammatory clash with what thins my blood patient guessed our words semantic (embeddings) med1,med2,men3 0.667 1.000 1.0 1.0
i have been down and wide awake at 3am for weeks patient guessed our words keyword (word overlap) med5,men1,men5 0.333 0.333 0.5 2.0
i have been down and wide awake at 3am for weeks patient guessed our words semantic (embeddings) men1,dia3,men2 0.667 0.667 1.0 1.0
i cannot pay for the name brand any more patient's own words keyword (word overlap) (nothing) 0.000 0.000 0.0 NaN
i cannot pay for the name brand any more patient's own words semantic (embeddings) med3,med5,car5 0.333 0.500 1.0 1.0
my feet feel like pins and needles patient guessed our words keyword (word overlap) dia5,men3 0.500 0.500 1.0 1.0
my feet feel like pins and needles patient guessed our words semantic (embeddings) dia5,car1,car5 0.333 0.500 1.0 1.0
the drug store says come back next week patient's own words keyword (word overlap) (nothing) 0.000 0.000 0.0 NaN
the drug store says come back next week patient's own words semantic (embeddings) med5,men5,dia4 0.333 0.500 1.0 1.0
i get winded carrying the groceries in patient's own words keyword (word overlap) (nothing) 0.000 0.000 0.0 NaN
i get winded carrying the groceries in patient's own words semantic (embeddings) car4,car1,dia1 0.333 0.500 1.0 1.0
Method and limitations
The 20 teaching sentences and 12 evaluation queries were provided in Dr. Richard Young's MIS 752 lab notebook. I used sentence-transformers/all-MiniLM-L6-v2 to create normalized embeddings, then ranked results by cosine similarity. The keyword baseline counts shared content words after removing stopwords.
Precision uses relevant results divided by the number actually returned; recall uses relevant results divided by all relevant items in the answer key. MRR uses the first relevant result within the top 3, with zero when none is found.
These results describe this small teaching corpus and its answer key. BM25 and reranking were not evaluated. The examples are course materials and do not validate a clinical search system.