Exact Match
Probably the simplest ways to evaluate an LLM or runnable's string output against a reference label is by a simple string equivalence.
This can be accessed using the exact_match
evaluator.
from langchain.evaluation import ExactMatchStringEvaluator
evaluator = ExactMatchStringEvaluator()
Alternatively via the loader:
from langchain.evaluation import load_evaluator
evaluator = load_evaluator("exact_match")
evaluator.evaluate_strings(
prediction="1 LLM.",
reference="2 llm",
)
{'score': 0}
evaluator.evaluate_strings(
prediction="LangChain",
reference="langchain",
)
{'score': 0}
Configure the ExactMatchStringEvaluator
You can relax the "exactness" when comparing strings.
evaluator = ExactMatchStringEvaluator(
ignore_case=True,
ignore_numbers=True,
ignore_punctuation=True,
)
# Alternatively
# evaluator = load_evaluator("exact_match", ignore_case=True, ignore_numbers=True, ignore_punctuation=True)
evaluator.evaluate_strings(
prediction="1 LLM.",
reference="2 llm",
)
{'score': 1}