Europarl

A corpus of parallel text in 21 European languages from the proceedings of the European Parliament.

The Europarl parallel corpus is extracted from the proceedings of the European Parliament (1996-2011). It includes versions in 21 European languages: Romanic (French, Italian, Spanish, Portuguese, Romanian), Germanic (English, Dutch, German, Danish, Swedish), Slavik (Bulgarian, Czech, Polish, Slovak, Slovene), Finni-Ugric (Finnish, Hungarian, Estonian), Baltic (Latvian, Lithuanian), and Greek. Parallel sentence counts are in the range 400K-2M, depending on the language combination.

The goal of the extraction and processing was to generate sentence aligned text for statistical machine translation systems. For this purpose we extracted matching items and labeled them with corresponding document IDs. Using a preprocessor we identified sentence boundaries. We sentence aligned the data using a tool based on the Church and Gale algorithm.

The Europarl corpus was collected mainly to aid research in statistical machine translation (training, evaluation), but it has been used for many other natural language problems: word sense disambiguation, anaphora resolution, information extraction, etc.

Monolingual datasets are also available for 9 languages. These are supersets of the parallel versions. Monolingual word counts are in the range 7M-54M, depending on the language.

Test Sets: Several test sets have been released for the Europarl corpus. In general, the Q4/2000 portion of the data (2000-10 to 2000-12) should be reserved for testing. All released test sets have been selected from this quarter.

Source: Europarl website