Experiments to evaluate multi-lingual capability of Gemini

Pranav Mani

Many parts of the world are multi-lingual

Humans are incredibly efficient at processing “long contexts” of multilingual text. They can detect languages, effectively reason and recall information with extreme accuracy.

Question

Is the language detection capability of Gemini 1.5 robust to handle large amount of multilingual text?

​

​

​

Collecting dataset

I scraped random articles per language from Wikipedia upto 100k words. The script used for scraping articles is also shared.

Why random order? Switching topics randomly from one article to another effectively “stresses” the long context capabilities of Gemini.

The datasets are classified per language. Multi-lingual datasets�are generated using single language datasets by concatenating�them.

Experiments and results

Without Context Caching:

• Experiment 1: Five datasets. One language per dataset. Prompt Gemini to detect the languages in the text.
• 100% detection, context caching was not required
• Experiment 2: Multi-language dataset. Concatenated dataset from 1 to 5 languages. Ask Gemini to detect languages per dataset.
• As the number of languages increases, deterioration in language detection capabilities is observed. I see improved results with “split and sample” approach. With context caching, ⅘ languages are detected.
• Experiment 3: Multi-language dataset with random alphanumeric strings of length 25 added at random spots in the dataset. Ask Gemini to detect these strings.
• Gemini fails to detect the strings both with and without context caching.

Add on experiment: Simulating a bilingual conversation with 2 Gemini APIs.

�

Takeaways

• Cost per run of notebook > 5 USD. Be aware of costs. Use context caching whenever possible.
• Human creativity is needed to solve problems that pop up with with multi-language datasets.
• API documentation was simple and easy to use as a first time user of Gemini APIs.
• These experiments can also be extended with image based datasets.