📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.

Italy’s Minerva-3B, a large language model trained entirely from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored only 4.9% on the INVALSI Italian school-exam benchmark, a result that questions the effectiveness of current scaling approaches for native-language AI models.

The Minerva project, led by Sapienza University of Rome and funded through Italy’s national AI strategy, built a 7-billion-parameter model with open weights, training on a massive dataset that included roughly half Italian data. Despite impressive technical achievements and public sharing of data and code, the model’s performance on the INVALSI benchmark was near chance, indicating that large-scale native-language training alone may not suffice for complex language tasks. Researchers emphasized that dataset size and parameter count are critical, but even with extensive Italian data, the model struggled with academic content tests. This finding complicates the narrative that larger, native-language training automatically yields deeper country-specific knowledge and suggests a need to reconsider scaling strategies in European sovereign-LLM initiatives.

Implications of Minerva’s Benchmark Performance for European Sovereign-LLMs

The poor performance of Minerva-3B on the INVALSI benchmark demonstrates that simply increasing data and parameters may not produce the desired depth of country-specific knowledge in large language models. This has broad implications for European AI strategies, which often emphasize large-scale, native-language training. The results suggest that the European sovereign-LLM movement may need to re-evaluate its assumptions about the scale of investment required to develop truly effective, localized AI models, potentially impacting future funding, research directions, and policy decisions.

Scaling Challenges in Sovereign-Language Model Development

Italy’s Minerva project represents a significant effort to develop a European sovereign LLM from scratch, utilizing a dataset of 2.5 trillion tokens, with half Italian content, and leveraging Italy’s national supercomputing infrastructure. Despite these investments, the model’s performance on complex academic benchmarks was notably poor. This follows broader debates within the European AI community about whether continuation training or large-scale from-scratch approaches are more effective. Prior efforts, such as Portugal’s AMÁLIA, layered specialization onto multilingual models, but Italy’s approach prioritized native-language training at scale. The results from Minerva challenge the assumption that larger data and parameters automatically translate into deeper country-specific knowledge, highlighting a critical technical and strategic dilemma for European AI sovereignty efforts.

Unresolved Questions About Scaling and Model Effectiveness

It remains unclear whether further scaling, additional fine-tuning, or alternative training methodologies could improve Minerva’s performance on complex academic and country-specific benchmarks. The ongoing iteration of the project may address some of these issues, but the fundamental question of how much native-language investment is necessary for deep country knowledge remains open. Additionally, the generalizability of these findings to other languages and models is still under investigation.

Next Steps for European Sovereign-Language AI Projects

The Minerva team is continuing to refine their models and methodologies, including upcoming experiments with continual training and larger datasets. Policymakers and research institutions are likely to reassess their strategies based on these findings, potentially shifting focus toward more targeted, quality-focused approaches rather than scale alone. Future benchmarks and evaluations will clarify whether these efforts can overcome the current limitations and achieve deeper country-specific AI capabilities.

Key Questions

Why did Minerva-3B perform poorly on the Italian exam benchmark?

The model’s limited performance suggests that even large-scale native-language training may not be enough to develop deep, complex language understanding without additional targeted strategies.

Does this mean European sovereign-LLMs are ineffective?

Not necessarily. It indicates that current approaches may need to be scaled or refined further; the results highlight challenges but do not render the entire strategy invalid.

Will increasing model size improve results?

It is uncertain. While larger models can improve performance, the Minerva findings suggest that size alone is insufficient without considering data quality and training methodology.

What does this imply for future European AI investments?

Policymakers may need to prioritize targeted, high-quality data and innovative training techniques over simply increasing scale to achieve meaningful country-specific AI capabilities.

Source: ThorstenMeyerAI.com