JSON Evaluators
Evaluating extraction and function calling applications often comes down to validation that the LLM's string output can be parsed correctly and how it compares to a reference object. The following JSON
validators provide functionality to check your model's output consistently.
JsonValidityEvaluator
The JsonValidityEvaluator
is designed to check the validity of a JSON
string prediction.
Overview:
- Requires Input?: No
- Requires Reference?: No
from langchain.evaluation import JsonValidityEvaluator
evaluator = JsonValidityEvaluator()
# Equivalently
# evaluator = load_evaluator("json_validity")
prediction = '{"name": "John", "age": 30, "city": "New York"}'
result = evaluator.evaluate_strings(prediction=prediction)
print(result)
{'score': 1}
prediction = '{"name": "John", "age": 30, "city": "New York",}'
result = evaluator.evaluate_strings(prediction=prediction)
print(result)
{'score': 0, 'reasoning': 'Expecting property name enclosed in double quotes: line 1 column 48 (char 47)'}
JsonEqualityEvaluator
The JsonEqualityEvaluator
assesses whether a JSON prediction matches a given reference after both are parsed.
Overview:
- Requires Input?: No
- Requires Reference?: Yes
from langchain.evaluation import JsonEqualityEvaluator
evaluator = JsonEqualityEvaluator()
# Equivalently
# evaluator = load_evaluator("json_equality")
result = evaluator.evaluate_strings(prediction='{"a": 1}', reference='{"a": 1}')
print(result)
{'score': True}
result = evaluator.evaluate_strings(prediction='{"a": 1}', reference='{"a": 2}')
print(result)
{'score': False}
The evaluator also by default lets you provide a dictionary directly
result = evaluator.evaluate_strings(prediction={"a": 1}, reference={"a": 2})
print(result)
{'score': False}
JsonEditDistanceEvaluator
The JsonEditDistanceEvaluator
computes a normalized Damerau-Levenshtein distance between two "canonicalized" JSON strings.
Overview:
- Requires Input?: No
- Requires Reference?: Yes
- Distance Function: Damerau-Levenshtein (by default)
Note: Ensure that rapidfuzz
is installed or provide an alternative string_distance
function to avoid an ImportError.
from langchain.evaluation import JsonEditDistanceEvaluator
evaluator = JsonEditDistanceEvaluator()
# Equivalently
# evaluator = load_evaluator("json_edit_distance")
result = evaluator.evaluate_strings(
prediction='{"a": 1, "b": 2}', reference='{"a": 1, "b": 3}'
)
print(result)
{'score': 0.07692307692307693}
# The values are canonicalized prior to comparison
result = evaluator.evaluate_strings(
prediction="""
{
"b": 3,
"a": 1
}""",
reference='{"a": 1, "b": 3}',
)
print(result)
{'score': 0.0}
# Lists maintain their order, however
result = evaluator.evaluate_strings(
prediction='{"a": [1, 2]}', reference='{"a": [2, 1]}'
)
print(result)
{'score': 0.18181818181818182}
# You can also pass in objects directly
result = evaluator.evaluate_strings(prediction={"a": 1}, reference={"a": 2})
print(result)
{'score': 0.14285714285714285}
JsonSchemaEvaluator
The JsonSchemaEvaluator
validates a JSON prediction against a provided JSON schema. If the prediction conforms to the schema, it returns a score of True (indicating no errors). Otherwise, it returns a score of 0 (indicating an error).
Overview:
- Requires Input?: Yes
- Requires Reference?: Yes (A JSON schema)
- Score: True (No errors) or False (Error occurred)
from langchain.evaluation import JsonSchemaEvaluator
evaluator = JsonSchemaEvaluator()
# Equivalently
# evaluator = load_evaluator("json_schema_validation")
result = evaluator.evaluate_strings(
prediction='{"name": "John", "age": 30}',
reference={
"type": "object",
"properties": {"name": {"type": "string"}, "age": {"type": "integer"}},
},
)
print(result)
{'score': True}
result = evaluator.evaluate_strings(
prediction='{"name": "John", "age": 30}',
reference='{"type": "object", "properties": {"name": {"type": "string"}, "age": {"type": "integer"}}}',
)
print(result)
{'score': True}
result = evaluator.evaluate_strings(
prediction='{"name": "John", "age": 30}',
reference='{"type": "object", "properties": {"name": {"type": "string"},'
'"age": {"type": "integer", "minimum": 66}}}',
)
print(result)
{'score': False, 'reasoning': "<ValidationError: '30 is less than the minimum of 66'>"}