WOLFRAM

Wolfram Language & System Documentation Center

"LanguageExtended" (Built-in Classifier)

Identify which natural language the text is in.

Classes

Details

  • This classifier assumes the text input is written in a unique language. The probabilities reflect the belief in which language the text is written, not the proportion of languages.
  • In the current version, all texts must be written in one of their official alphabets.

Examples

open all close all

Basic Examples  (2)

Determine the languages of a list of examples:

Wolfram Language code: Classify["LanguageExtended", {"the house is blue", "la maison est bleue", "la casa es azul", "das Haus ist blau", "號寫正嗰一部長篇小說", "المنزل باللون الأزرق", "будинок синій", "ตรชายของอั", "ܝܐ ܕܩܐܡܦ ܢ", "גלעד. אור ", "մութիւններ", "ားကို အထူး", "и пщым и у", "성적 욕망에 대한 ", "лакьысра"}]

Obtain the probabilities for the most likely languages:

Wolfram Language code: Classify["LanguageExtended", "منزل ", "TopProbabilities" -> 3]

Obtain a ClassifierFunction for this classifier:

Wolfram Language code: c = Classify["LanguageExtended"]

Apply the classifier to a list of texts:

Wolfram Language code: c[{"the house is blue", "la maison est bleue", "la casa es azul", "das Haus ist blau", "در زندگی زخم‌های هست", "房子是蓝色的", "المنزل باللون الأزرق", "будинок синій"}]

Scope  (2)

Load the ClassifierFunction corresponding to the built-in classifier:

Wolfram Language code: c = Classify["LanguageExtended"]

Obtain the possible classes:

Wolfram Language code: Short[Information[c, "Classes"], 1]

Load the ClassifierFunction corresponding to the built-in classifier:

Wolfram Language code: c = Classify["LanguageExtended"]

Apply the classifier to a list of text samples:

Wolfram Language code: textsamples = {"the house is blue", "la maison est bleue", "la casa es azul", "das Haus ist blau", "號寫正嗰一部長篇小說", "المنزل باللون الأزرق", "будинок синій", "ตรชายของอั", "גלעד. אור ", "մութիւններ", "и пщым и у", "성적 욕망에 대한 "}; c[textsamples]

Transliterate sample texts and apply the classifier to the transliterated texts:

Wolfram Language code: trsamples = Transliterate[textsamples] c[trsamples]

Options  (3)

ClassPriors  (1)

Use a custom ClassPriors to restrict the possible outputs:

Wolfram Language code: Classify["LanguageExtended", {"What is this language?", "Qué idioma es ese?", "Quelle est cette langue?"}, ClassPriors -> <|Entity["Language", "English::385w8"] -> 0.5, Entity["Language", "Spanish::77gfp"] -> 0.5|>]

IndeterminateThreshold  (1)

Use a custom IndeterminateThreshold:

Wolfram Language code: Classify["LanguageExtended", "منزل", IndeterminateThreshold -> 0.9]

UtilityFunction  (1)

Obtain the utility function of the classifier:

Wolfram Language code: utility = Information[Classify["LanguageExtended"], "UtilityFunction"];

Modify this utility function to penalize being misclassified as:

Wolfram Language code: utility[Entity["Language", "Persian::48q57"], Entity["Language", "Bakhtiari::tw2cx"]] = -4;

Classify the text using this new utility:

Wolfram Language code: Classify["LanguageExtended", "مردم", UtilityFunction -> utility]

Compare to the result with the default utility:

Wolfram Language code: Classify["LanguageExtended", "مردم"]

Possible Issues  (1)

In some cases, non-language texts are still recognized as one language:

Wolfram Language code: Classify["LanguageExtended", "poivreibflkqwhlqhrgw"]

Neat Examples  (1)

Obtain different possible languages with their corresponding probabilities:

Wolfram Language code: possiblelanguages = Classify["LanguageExtended", "Zu meiner Familie gehören außer mir noch drei weitere Personen", "Probabilities"];

Visualize the result using WordCloud:

Wolfram Language code: WordCloud[possiblelanguages]