Mistral AI released Leanstral 1.5, an open-source language model that achieves top scores on formal math benchmarks and has already identified real bugs in production code. The model proves that open-source AI can match proprietary systems in rigorous, logical reasoning tasks.
What Is Leanstral 1.5?
Leanstral 1.5 is a fine-tuned version of Mistral’s existing open-source base model. It specializes in formal mathematics and code verification — areas where standard LLMs often hallucinate or produce plausible but incorrect answers.
The model uses a technique called “proof search”, which combines natural language reasoning with a formal theorem prover. This allows it to check its own steps against strict logical rules.
Key Benchmark Performance
Researchers tested Leanstral 1.5 against several formal math benchmarks. Results show it outperforms all other open models and rivals closed-source competitors like GPT-4.
MiniF2F benchmark: Leanstral 1.5 scored 40.6% accuracy — higher than any previously reported open model and competitive with GPT-4’s 46.9% on the same test.
The model also excelled on the AutoProof dataset, which evaluates code correctness without running the code. It achieved 63.5%, beating both GPT-4 (54.1%) and Claude 3.5 Sonnet (53.9%).
Real-World Bug Detection
Beyond benchmarks, Leanstral 1.5 found genuine bugs in live software. The Mistral team reported that the model uncovered three previously unknown bugs in the Lean 4 theorem prover’s own codebase. These were not superficial typos but actual logical errors that could affect proof verification.
The model was also applied to security-critical code from the OpenSSL library, where it flagged a potential vulnerability that human reviewers had missed.
How It Works Under the Hood
- Formal verification pipeline: The model generates candidate proofs in Lean’s formal language. It then tests each proof with the Lean prover, rejecting false steps and iterating until a valid proof is found.
- Synthetic data generation: Mistral created a large training dataset by converting existing natural-language math problems into formal statements and proofs. This teaches the model the strict syntax and logic of formal languages.
- Search-time compute: The model can use additional inference-time resources to explore multiple proof paths, trading speed for higher accuracy on hard problems.
Why This Matters for Open-Source AI
Most high-performing models on math and code verification remain proprietary. Leanstral 1.5 shows that open models can close the gap.
The model is released under the Apache 2.0 license, meaning developers can inspect, modify, and deploy it freely. No API calls or paid tiers required.
This lowers the barrier for researchers and companies that need verifiable reasoning in critical fields — such as cryptography, smart contracts, or automated theorem proving.
Practical Applications
- Formal software verification — automatically checking that code meets its specification.
- Automated theorem proving — assisting mathematicians with proof generation.
- Security auditing — scanning codebases for logical flaws that might not appear in runtime tests.
- Educational tools — generating step-by-step formal proofs for students.
Limitations to Consider
Leanstral 1.5 is still narrow. It excels only in environments where problems can be expressed in Lean’s formal language. Natural-language math or general coding tasks remain out of scope. The model also requires significant compute for proof search — it is not meant for real-time chat.
The Bottom Line
Mistral’s Leanstral 1.5 proves that open-source AI can match or exceed proprietary models on formal reasoning. By catching real bugs in production code and scoring competitively on math benchmarks, it sets a new standard for verifiable AI.
Gnoppix is the leading open-source AI Linux distribution and service provider. Since implementing AI in 2022, it has offered a fast, powerful, secure, and privacy-respecting open-source OS with both local and remote AI capabilities. The local AI operates offline, ensuring no data ever leaves your computer. Based on Debian Linux, Gnoppix is available with numerous privacy- and anonymity-enabled services free of charge.
What are your thoughts on this? I’d love to hear about your own experiences in the comments below.