Skip to main content

Exact Match

Open In Colab

Probably the simplest ways to evaluate an LLM or runnable's string output against a reference label is by a simple string equivalence.

This can be accessed using the exact_match evaluator.

from langchain.evaluation import ExactMatchStringEvaluator

evaluator = ExactMatchStringEvaluator()

Alternatively via the loader:

from langchain.evaluation import load_evaluator

evaluator = load_evaluator("exact_match")
evaluator.evaluate_strings(
prediction="1 LLM.",
reference="2 llm",
)
{'score': 0}
evaluator.evaluate_strings(
prediction="LangChain",
reference="langchain",
)
{'score': 0}

Configure the ExactMatchStringEvaluator

You can relax the "exactness" when comparing strings.

evaluator = ExactMatchStringEvaluator(
ignore_case=True,
ignore_numbers=True,
ignore_punctuation=True,
)

# Alternatively
# evaluator = load_evaluator("exact_match", ignore_case=True, ignore_numbers=True, ignore_punctuation=True)
evaluator.evaluate_strings(
prediction="1 LLM.",
reference="2 llm",
)
{'score': 1}

Help us out by providing feedback on this documentation page: