Language Tools

Resiliparse language tools API documentation.

resiliparse.parse.lang.detect_fast(text, cutoff=1200, n_results=1, langs=None)

Perform a very fast (linear-time) language detection on the input string.

The output is a tuple of the detected language name and the calculated out-of-place rank, which indicates how far the given text is from the closest-matching language profile. The higher the rank, the less accurate the detection is. Values above 1200 are usually false results.

The given Unicode string should be in composed normal form (NFC) for the best results.

Parameters:

  • text (str) – input text

  • cutoff (int) – OOP rank cutoff after which to return "unknown"

  • n_results (int) – if this is greater than one, a list of the n_results best matches will be returned

  • langs (list[str]) – restrict detection to these languages

Returns:

tuple of the detected language (or "unknown") and its out-of-place rank

Return type:

(str, int) | list[(str, int)]

resiliparse.parse.lang.supported_langs()

Get a list of all languages that are supported by the fast language detector.

Returns:

list of supported languages

Return type:

list[str]

resiliparse.parse.lang.train_language_examples(examples, vec_len=256)

Train a language vector for fast language detection on a list of example texts.

Parameters:

  • examples (t.Iterable[str]) – list of example texts for this language

  • vec_len (int) – output vector length

Returns:

vector of trained values

Return type:

list[int]