Course Wiki
Search Parameter Tuning — LLM Zoomcamp Module 4
--- video_url: 'https://www.youtube.com/watch?v=rSBSS_kCYN0&list=PL3MmuxUbc_hLZFNgSad56pDBKK8KO0XIv' --- # Search Parameter Tuning
Related Pages
LLM Zoomcamp › Module 4: Evaluation › Search Parameter Tuning
Notes
video_url: “https://www.youtube.com/watch?v=rSBSS_kCYN0&list=PL3MmuxUbc_hLZFNgSad56pDBKK8KO0XIv”
Search Parameter Tuning
In the previous lesson, we defined Hit Rate, MRR, and the evaluate
function. Now we can use them to tune search parameters.
Instead of guessing which settings are better, we measure them on the ground truth dataset.
So far we’ve boosted question to 3.0. The idea was that a query should
match the FAQ question. That kind of match should count for more than
matching the answer text. It sounds reasonable. But it’s a guess, and now
we can check it against data instead of trusting it.
This is the main benefit of offline evaluation. We change one parameter, run the same questions again, and see whether the metric moves. The dataset stays fixed, so the comparison is fair.
Trying different boosts
Start with a search function where the question boost is configurable:
def search_boost(query, question_boost):
boost_dict = {"question": question_boost, "section": 0.5}
return index.search(
query,
num_results=5,
boost_dict=boost_dict,
)
Evaluate several boost values:
for boost in [0.5, 1.0, 3.0, 5.0, 10.0]:
result = evaluate(
ground_truth,
lambda query, boost=boost: search_boost(query, boost)
)
print(f"boost={boost}: {result}")
For the data we prepared on May 29, 2026, this gives:
boost=0.5: {'hit_rate': 0.9113924050632911, 'mrr': 0.800548523206751}
boost=1.0: {'hit_rate': 0.9240506329113924, 'mrr': 0.8139240506329113}
boost=3.0: {'hit_rate': 0.8987341772151899, 'mrr': 0.7693248945147676}
boost=5.0: {'hit_rate': 0.8708860759493671, 'mrr': 0.7401265822784809}
boost=10.0: {'hit_rate': 0.8582278481012658, 'mrr': 0.7122362869198313}
Increasing the question boost makes the metrics worse, not better. The
best value here is 1.0, no boost at all. That’s already the opposite of
what the intuition predicted.
But this is only one parameter. We can also tune answer and section
together with question.
Define a search function with all three boosts:
def search_boosts(query, question_boost, answer_boost, section_boost):
boost_dict = {
"question": question_boost,
"section": section_boost,
"answer": answer_boost,
}
return index.search(
query,
num_results=5,
boost_dict=boost_dict,
)
Now do a small grid search:
results = []
for question_boost in [1.0, 2.0, 5.0]:
for answer_boost in [1.0, 2.0, 4.0, 10.0]:
for section_boost in [0.1, 0.2, 0.5]:
print(
f"Evaluating question_boost={question_boost},"
f" answer_boost={answer_boost},"
f" section_boost={section_boost}..."
)
result = evaluate(
ground_truth,
lambda query, question_boost=question_boost, answer_boost=answer_boost, section_boost=section_boost: search_boosts(
query,
question_boost,
answer_boost,
section_boost
)
)
results.append({
"question": question_boost,
"answer": answer_boost,
"section": section_boost,
"hit_rate": result["hit_rate"],
"mrr": result["mrr"],
})
Sort by MRR:
df_results = pd.DataFrame(results)
df_results.sort_values("mrr", ascending=False).head(10)
For the same data, the best rows are:
question answer section hit_rate mrr
1.0 2.0 0.1 0.975 0.885
2.0 4.0 0.2 0.975 0.885
5.0 10.0 0.5 0.975 0.885
5.0 10.0 0.2 0.975 0.884
5.0 10.0 0.1 0.975 0.884
2.0 4.0 0.1 0.975 0.884
2.0 4.0 0.5 0.977 0.884
1.0 2.0 0.2 0.977 0.884
1.0 2.0
## Key concepts
- [RAG](/course-wiki/rag/)
- [Trading Strategy](/course-wiki/trading-strategy/)
- [Pandas](/course-wiki/pandas/)
- [Prompt Engineering](/course-wiki/prompt-engineering/)
- [LLM Evaluation](/course-wiki/llm-evaluation/)
## Related notes
- [llmz-m01-agents](/course-wiki/llmz-m01-agents/)
- [llmz-m01-building-the-prompt](/course-wiki/llmz-m01-building-the-prompt/)
- [llmz-m01-data-ingestion](/course-wiki/llmz-m01-data-ingestion/)
- [llmz-m01-function-calling](/course-wiki/llmz-m01-function-calling/)
- [llmz-m01-introduction](/course-wiki/llmz-m01-introduction/)
- [llmz-m01-other-frameworks](/course-wiki/llmz-m01-other-frameworks/)
- [llmz-m01-quick-rag-revision-optional](/course-wiki/llmz-m01-quick-rag-revision-optional/)
- [llmz-m01-rag](/course-wiki/llmz-m01-rag/)
- [llmz-m01-rag-helper](/course-wiki/llmz-m01-rag-helper/)
- [llmz-m01-search](/course-wiki/llmz-m01-search/)
- [llmz-m01-the-agentic-loop](/course-wiki/llmz-m01-the-agentic-loop/)
- [llmz-m01-the-course-faq-dataset](/course-wiki/llmz-m01-the-course-faq-dataset/)
- [llmz-m01-the-llm](/course-wiki/llmz-m01-the-llm/)
- [llmz-m01-toyaikit](/course-wiki/llmz-m01-toyaikit/)
- [llmz-m01-wrap-up-of-part-1](/course-wiki/llmz-m01-wrap-up-of-part-1/)
- [llmz-m02-embeddings](/course-wiki/llmz-m02-embeddings/)
- [llmz-m02-next-steps](/course-wiki/llmz-m02-next-steps/)
- [llmz-m02-rag-with-vector-search](/course-wiki/llmz-m02-rag-with-vector-search/)
- [llmz-m02-vector-search](/course-wiki/llmz-m02-vector-search/)
- [llmz-m02-vector-search-with-minsearch](/course-wiki/llmz-m02-vector-search-with-minsearch/)
- [llmz-m02-vector-search-with-pgvector](/course-wiki/llmz-m02-vector-search-with-pgvector/)
- [llmz-m02-vector-search-with-sqlitesearch](/course-wiki/llmz-m02-vector-search-with-sqlitesearch/)
- [llmz-m03-ai-agents](/course-wiki/llmz-m03-ai-agents/)
- [llmz-m03-ai-copilot](/course-wiki/llmz-m03-ai-copilot/)
- [llmz-m03-ai-orchestration](/course-wiki/llmz-m03-ai-orchestration/)
- [llmz-m03-best-practices](/course-wiki/llmz-m03-best-practices/)
- [llmz-m03-context-engineering](/course-wiki/llmz-m03-context-engineering/)
- [llmz-m03-multi-agent-systems](/course-wiki/llmz-m03-multi-agent-systems/)
- [llmz-m03-next-steps](/course-wiki/llmz-m03-next-steps/)
- [llmz-m03-retrieval-augmented-generation](/course-wiki/llmz-m03-retrieval-augmented-generation/)
- [llmz-m04-agent-evaluation](/course-wiki/llmz-m04-agent-evaluation/)
- [llmz-m04-evaluation](/course-wiki/llmz-m04-evaluation/)
- [llmz-m04-generating-ground-truth-data](/course-wiki/llmz-m04-generating-ground-truth-data/)
- [llmz-m04-generating-ground-truth-for-all-documents](/course-wiki/llmz-m04-generating-ground-truth-for-all-documents/)
- [llmz-m04-generating-rag-answers](/course-wiki/llmz-m04-generating-rag-answers/)
- [llmz-m04-llm-as-a-judge](/course-wiki/llmz-m04-llm-as-a-judge/)
- [llmz-m04-next-steps](/course-wiki/llmz-m04-next-steps/)
- [llmz-m04-rag-and-agent-evaluation](/course-wiki/llmz-m04-rag-and-agent-evaluation/)
- [llmz-m04-search-evaluation](/course-wiki/llmz-m04-search-evaluation/)
- [llmz-m04-search-evaluation-metrics](/course-wiki/llmz-m04-search-evaluation-metrics/)
- [llmz-m05-assistant](/course-wiki/llmz-m05-assistant/)
- [llmz-m05-built-in-judge](/course-wiki/llmz-m05-built-in-judge/)
- [llmz-m05-capturing-metrics](/course-wiki/llmz-m05-capturing-metrics/)
- [llmz-m05-chat-app](/course-wiki/llmz-m05-chat-app/)
- [llmz-m05-docker-compose](/course-wiki/llmz-m05-docker-compose/)
- [llmz-m05-monitoring](/course-wiki/llmz-m05-monitoring/)
- [llmz-m05-next-steps](/course-wiki/llmz-m05-next-steps/)
- [llmz-m05-querying-data](/course-wiki/llmz-m05-querying-data/)
- [llmz-m05-storing-data-in-postgresql](/course-wiki/llmz-m05-storing-data-in-postgresql/)
- [llmz-m05-streamlit-dashboard](/course-wiki/llmz-m05-streamlit-dashboard/)
- [llmz-m05-synthetic-data-generation](/course-wiki/llmz-m05-synthetic-data-generation/)
- [llmz-m05-user-feedback](/course-wiki/llmz-m05-user-feedback/)
- [llmz-m06-best-practices-for-rag](/course-wiki/llmz-m06-best-practices-for-rag/)
- [llmz-m06-document-reranking](/course-wiki/llmz-m06-document-reranking/)
- [llmz-m06-next-steps](/course-wiki/llmz-m06-next-steps/)
- [llmz-m07-chunking-for-longer-texts](/course-wiki/llmz-m07-chunking-for-longer-texts/)
- [llmz-m07-content-processing-cases-and-steps](/course-wiki/llmz-m07-content-processing-cases-and-steps/)
- [llmz-m07-end-to-end-project-example](/course-wiki/llmz-m07-end-to-end-project-example/)
- [llmz-m07-evaluating-rag](/course-wiki/llmz-m07-evaluating-rag/)
- [llmz-m07-evaluating-retrieval](/course-wiki/llmz-m07-evaluating-retrieval/)
- [llmz-m07-interface-and-ingestion-pipeline](/course-wiki/llmz-m07-interface-and-ingestion-pipeline/)
- [llmz-m07-monitoring-and-containerization](/course-wiki/llmz-m07-monitoring-and-containerization/)
- [llmz-m07-summary-and-closing-remarks](/course-wiki/llmz-m07-summary-and-closing-remarks/)
## Sources
- [Video](https://www.youtube.com/watch?v=rSBSS_kCYN0&list=PL3MmuxUbc_hLZFNgSad56pDBKK8KO0XIv)
- [Lesson file](https://github.com/llm-zoomcamp/blob/main/llm-zoomcamp/cohorts/2026/04-evaluation/06-search-tuning.md)