Tuneable fuzzy matching for language identification evaluation.

Evaluating language identification (LID) entails deciding whether a predicted language label matches the gold. However, this task is not as straightforward as it first seems: models and test sets may use different labelling schemes and cover different sets of languages, and which language varieties are matches depends on the application.

langid-eval is an open-source Python package for simplifying LID evaluation. It builds on the langcodes library and the Unicode CLDR Project to reconcile language labels automatically. It allows the user to define the granularity of match that they need, from as narrow as the exact variety and script to as broad as the same macrolanguage. It also calculates key summary statistics in a reproducible way, either for all languages in the test set or only for those covered by the tested model.

langid-eval can be installed from this repository, directly or using tools like pip or uv. It will be added to PyPI after publication!

Requires Python ≥3.14.

from langid_eval import LangIdEval

# similar labels match without preprocessing
golds = ["en", "en", "es", "ar", "zh", "fr"]
preds = ["en-GB", "fr", "es-MX", "arb", "cmn", "fr"]

lideval = LangIdEval(golds, preds)
lideval.precision()        # macro average over all classes by default
lideval.recall(average="weighted")
lideval.f1(by_class=True)  # {'en': 0.667, 'es': 1.0, ...}

lideval = LangIdEval(golds, preds, supported_languages=["en", "fr"])
lideval.fpr(supported_languages_only=True)
print(lideval.classification_report(digits=2))

Given two equal-length files of gold and predicted labels (one per line), print an sklearn-style classification report.

langid-eval golds.txt preds.txt [-s supported_langs.txt]

Three constructor arguments to LangIdEval affect how matches are defined:

• ignore_script (default: True). Ignore script subtags when matching.
• max_tag_distance (default: 5). The maximum tag distance (as defined by langcodes) at which two language tags will count as a match.
• supported_languages (default: ()). The subset of languages to evaluate on, intended to be the languages supported by the model under testing. Setting the method argument supported_languages_only=True will filter the test set only to the gold labels which are within max_tag_distance of one of the languages in this sequence prior to calculating statistics. The intended use is to evaluate models only on the languages they claim to cover.

langid-eval uses the tag_distance function from the langcodes package to adjust the strictness of the match. It returns a number from 0 to 134 indicating the distance between the language the user desires and a supported language (predicted and gold labels for langid-eval). This distance is intended as a measure of likely language acceptability; that is, given a user understands one language, how likely is it that the other will be acceptable to them? The distance data comes from CLDR v38.1 and involves a lot of judgment calls made by the Unicode Consortium. For this reason, it is not a true metric, but rather should be taken as indicative. Note that distances are not necessarily symmetric: for example, tag_distance("arz", "ar") = 10, whereas tag_distance("ar", "arz") = 80.

We copy the table describing tag distances from the documentation for langcodes below. See the docstring of tag_distance for more explanation and examples.

Since langid-eval is intended to match similar language varieties, we recommend that max_tag_distance is set to 20 at the very most. As a rule of thumb, a value of 0 matches equivalent codes after normalisation, 5 (the default) allows for regional differences, and 10-15 matches individual languages to their macrolanguage. In case of unexpected behaviour, please raise an issue in this repository.

Full parameter details are in the docstrings, e.g. help(LangIdEval.f1).

After publication! Please see the paper for further details and analysis of the package.

langid-eval is released under the MIT license.