Identifying the language of a piece of text is a fairly broad topic, and I am not going to go into the algorithms behind it in detail here. Instead I will look at some methods that can be used off the shelf for the semantic analysis of text in order to extract its language. Since I am working on a project, a small part of which is extracting information about the language of a given text, I had to do a short piece of research on the subject. The methods I tried and the conclusions I reached are described here.
First attempt: Microsoft Extended Linguistic Services
Before trying to make any web requests to APIs and burn through traffic (and you know that every MB costs money, especially in the cloud), I decided to dig around and see what good old .NET and the good old Windows APIs have to offer.
It turned out that .NET has no built-in classes for the semantic analysis of text, but there is a Windows API called Extended Linguistic Services (ELS), which tries to suggest the possible languages a given text might be in. From Windows 7 onwards, ELS is installed automatically together with the operating system, which gives good hope that it will be built into Windows 8 Server too.
Of course there was also a .NET wrapper for it, which incidentally included quite a few useful things related to the Windows Shell, DirectX, the Windows 7 taskbar and so on. For the semantic analysis involved in recognising text, all you actually need out of the whole "Windows API Code Pack" is "Microsoft.WindowsAPICodePack.ExtendedLinguisticServices.dll".
Using it is a little odd, and looks like this:
MappingService serv = new MappingService(
MappingAvailableServices.LanguageDetection);
var res = serv.RecognizeText(@"
Your cruel device
your blood, like ice
One look, could kill
My pain, your thrill
Your mouth, so hot
Your web, I'm caught
Your skin, so wet
Black lace, on sweat", null);
var langs = res.FormatData(new StringArrayFormatter());
foreach (var lang in langs[0])
{
Console.Write("{0} ", lang);
}
Console.WriteLine();
Its big drawback, however, is that it offers a great many candidate languages. For the text above it is obvious that the language is English, yet ELS gave me 11 suggestions (Serbian among them): en, af, de, sk, sl, ca, et, fi, hr, sr-Latn, tn. For one verse of the Bulgarian poem "Az sam balgarche" ("Аз съм българче. Обичам наште планини зелени, българин да се наричам - първа радост е за мене.") ELS suggested the following languages: bg, mk, ru, sr-Cyrl, be, uk, kk.
A small digression. Another of the useful functions of Extended Linguistic Services is transliterating text from Cyrillic to Latin. Naturally, that does not work correctly either. Why? I have a theory that Microsoft have two kinds of products: the polished ones that work almost perfectly (SQL Server, Visual Studio, the .NET Framework and so on) and the ones thrown together and full of bugs. ELS is of the second kind. Here is how you can transliterate text from Cyrillic to Latin with the .NET Windows API Code Pack:
MappingService serv = new MappingService(
MappingAvailableServices.TransliterationCyrillicToLatin);
var res = serv.RecognizeText(@"
Аз съм българче. Обичам
наште планини зелени,
българин да се наричам -
първа радост е за мене.", null);
var langs = res.FormatData(new StringFormatter());
Console.WriteLine(langs[0]);
The result of the code above is "Az s?m b?lgarce. Obicam naste planini zeleni, b?lgarin da se naricam - p?rva radost e za mene.". The question marks are apparently meant to stand in for the letters that do not exist in Russian. Only Microsoft know what they wrote there. :D
Second attempt: AlchemyAPI
AlchemyAPI is a web service offering all kinds of semantic analysis: language identification, name extraction, tag (keyword) extraction, text categorisation, positive/negative sentiment detection, extraction of meta tags and microformats, and quite a lot more.
It handles language identification perfectly (it supports 97 languages), but unfortunately most of its other functions do not work with Bulgarian (categorisation and sentiment analysis, for example). It has an API wrapper for .NET and using it is fairly easy (when compiling, use build.bat and set your paths to csc.exe, or run the .bat file from the Visual Studio Command Prompt). Here is a short example of using it (for it to work you must replace _YOUR_API_KEY_ with the API key you receive on registration):
string text = @"Аз съм българче.";
AlchemyAPI.AlchemyAPI alchemyObj = new AlchemyAPI.AlchemyAPI();
alchemyObj.SetAPIKey("_YOUR_API_KEY_");
var xmlData = alchemyObj.TextGetLanguage(text);
Console.WriteLine(xmlData);
For the three Bulgarian words above, AlchemyAPI identified Bulgarian without hesitation. The main drawback of AlchemyAPI is that it is paid, and the prices are not published anywhere. For a price you have to write to their sales department. There is also a free tier supporting 1,000 API calls per day once you register, but for large projects that is nowhere near enough.
Third attempt: Google Translate API
Good old Google Translate works perfectly as a web tool and probably produces some of the most accurate results, but Google want fairly serious money for the use of its APIs. For every 1 million characters (about 1 MB) they charge 20 dollars, both for translation and for language identification. Using the Google Translate API for language identification really would be an expensive pleasure.
Fourth attempt: NTextCat
NTextCat is an open-source library for text classification. Its main purpose is identifying the language of a given text. It supports different language models (the model extracted from Wikipedia supports over 280 languages). I took a look at the library and it seemed to me that it does its analysis based on statistical information about how often letters occur in a given language. A few months ago I wrote a blog post with statistical information from Wikipedia, which anyone interested can read.
Using NTextCat turned out to be considerably easier than I expected. You reference its libraries, add a "using IvanAkcheurov.NTextCat.Lib.Legacy;" and you can start analysing text. Here is a short demo analysing our favourite verse:
LanguageIdentifier id = new LanguageIdentifier(
@"N:\...\NTextCat 0.1.6\LanguageModels\Wikipedia-MostCommon-Utf8");
LanguageIdentifier.LanguageIdentifierSettings sett =
new LanguageIdentifier.LanguageIdentifierSettings();
var res = id.ClassifyText(@"Аз съм българче. Обичам наште планини зелени,
българин да се наричам - първа радост е за мене.", sett);
foreach (var item in res)
{
Console.Write("{0}; ", item);
}
The result is "(bg, 94883); (mk, 97529);". Apparently NTextCat thinks the poem could also be in Macedonian. :D According to the project's authors, at least 50 words are needed for this library to determine the language of a text accurately.
I ran a few more experiments with NTextCat. First I tried a long Bulgarian text (800 letters) with the "Wikipedia-MostCommon-Utf8" language model, and the result was roughly the same: Bulgarian beats Macedonian by a small margin. When I ran the analysis on Macedonian text, however, NTextCat decided without hesitation that the language was Macedonian. With the other language models generated from Wikipedia the result is almost the same, but the analysis takes considerably longer. When I tried the library's default language model it did not identify the language at all, because that model does not support UTF-8 encoding.
On the whole NTextCat (with the "Wikipedia-MostCommon-Utf8" model) can be trusted. It gives good results, even if it sometimes hesitates between Bulgarian and Macedonian for Bulgarian text. A big advantage, though, is that it can sometimes work out the encoding of a sequence of bytes as well as the language. Overall I recommend it as an offline option for language identification.
Conclusion
There are certainly other ways of identifying the language of a text, but these were the four I looked at. None of them managed to recognise "shlyokavitsa" - Bulgarian written in Latin letters ("Az sam bulgarche obicham..."). In the end I settled on NTextCat. The reasons are clear: it works correctly and reasonably fast, it is open source, it is stable, it is easy to use and it does not require additional resources (internet traffic, for example). And since I found no problem or reason not to use NTextCat, I will happily go on using it. ;) I hope the article was useful, and if you know a better option for identifying the language of a text I would be glad to hear from you. :)