WOLFRAM

Wolfram Language & System Documentation Center

"Language" (Built-in Classifier)

Identify which natural language the text is in.

Classes

Details

  • This classifier assumes the text input is written in a unique language. The probabilities reflect the belief in which language the text is written, not the proportion of languages.
  • For large texts, the classifier randomly selects snippets to determine the overall language.
  • In the current version, all texts must be written in one of their official alphabets.

Examples

open all close all

Basic Examples  (2)

Determine the languages of a list of examples:

Wolfram Language code: Classify["Language", {"the house is blue", "la maison est bleue", "la casa es azul", "das Haus ist blau", "房子是蓝色的", "المنزل باللون الأزرق", "будинок синій"}]

Obtain the probabilities for the most likely languages:

Wolfram Language code: Classify["Language", "ella", "TopProbabilities" -> 3]

Obtain a ClassifierFunction for this classifier:

Wolfram Language code: c = Classify["Language"]

Apply the classifier to a list of texts:

Wolfram Language code: c[{"the house is blue", "la maison est bleue", "la casa es azul", "das Haus ist blau", "در زندگی زخم‌های هست", "房子是蓝色的", "المنزل باللون الأزرق", "будинок синій"}]

Scope  (1)

Load the ClassifierFunction corresponding to the built-in classifier:

Wolfram Language code: c = Classify["Language"]

Obtain the possible classes:

Wolfram Language code: Information[c, "Classes"]

Options  (3)

ClassPriors  (1)

Use a custom ClassPriors to restrict the possible outputs:

Wolfram Language code: Classify["Language", {"What is this language?", "¿Qué idioma es ese?", "Quelle est cette langue?"}, ClassPriors -> <|Entity["Language", "English"] -> 0.5, Entity["Language", "Spanish"] -> 0.5|>]

IndeterminateThreshold  (1)

Use a custom IndeterminateThreshold:

Wolfram Language code: Classify["Language", "المنزل", IndeterminateThreshold -> 0.9]

UtilityFunction  (1)

Obtain the utility function of the classifier:

Wolfram Language code: utility = Information[Classify["Language"], "UtilityFunction"];

Modify this utility function to penalize being misclassified as :

Wolfram Language code: utility[Entity["Language", "Persian"], Entity["Language", "Arabic"]] = -1.;

Classify the text using this new utility:

Wolfram Language code: Classify["Language", "منزل", UtilityFunction -> utility]

Compare to the result with the default utility:

Wolfram Language code: Classify["Language", "منزل"]

Possible Issues  (1)

In some cases, non-language texts are still recognized as one language:

Wolfram Language code: Classify["Language", "poivreibflkqwhlqhrgw"]

Neat Examples  (1)

Obtain different possible languages with their corresponding probabilities:

Wolfram Language code: possiblelanguages = Classify["Language", "Zu meiner Familie gehören außer mir noch drei weitere Personen", "Probabilities"];

Visualize the result using WordCloud:

Wolfram Language code: WordCloud[possiblelanguages]