Major AI Models Face Compliance Challenges Ahead of EU’s AI Act
A new AI compliance tool developed by LatticeFlow AI has raised alarms regarding the readiness of leading artificial intelligence models from major tech companies to meet the European Union’s evolving AI Act standards. Early assessments indicate that models from Meta, OpenAI, and Alibaba may struggle in critical areas such as cybersecurity and discriminatory output, revealing significant regulatory gaps that could have serious implications for these companies.
The Compliance Tool: LLM Checker
The compliance checker, known as the “Large Language Model (LLM) Checker,” was developed by Swiss startup LatticeFlow AI in collaboration with researchers from ETH Zurich and INSAIT in Bulgaria. This innovative tool is designed to evaluate AI models against the EU AI Act, which is set to gradually come into force over the next two years. The LLM Checker assesses models across various categories, including technical robustness, safety, and cybersecurity resilience, assigning scores ranging from 0 to 1. A score below 0.75 indicates potential weaknesses in specific regulatory areas, signaling a need for companies to address these gaps to avoid significant financial penalties.
A Mixed Bag of Results for AI Leaders
The results from the LLM Checker, published by LatticeFlow, reveal a mixed performance among some of the most prominent AI models. For instance, OpenAI’s widely used “GPT-3.5 Turbo” model received a concerning score of 0.46 for its ability to prevent discriminatory output, highlighting ongoing challenges in mitigating bias within AI systems. Alibaba’s “Qwen1.5 72B Chat” model fared even worse, scoring only 0.37 in the same category.
Cybersecurity vulnerabilities were also a significant concern. Meta’s “Llama 2 13B Chat” model scored just 0.42 for its ability to defend against prompt hijacking—a type of cyberattack where malicious actors disguise harmful prompts to extract sensitive data. Similarly, French AI startup Mistral’s model “8x7B Instruct” scored poorly, with a score of 0.38 in the same area.
In contrast, Anthropic’s “Claude 3 Opus” model, backed by Google, emerged as the top performer with an impressive overall score of 0.89, indicating stronger compliance readiness. However, the varying performance across models underscores the urgent need for further fine-tuning to meet the stringent requirements of the forthcoming AI Act.
Regulatory Compliance: A Growing Priority for Big Tech
The EU’s AI Act represents one of the most comprehensive regulatory frameworks globally, aimed at curbing the risks posed by artificial intelligence technologies while promoting innovation. Companies that fail to comply with the Act’s provisions could face fines as high as €35 million or 7% of their global annual turnover.
Petar Tsankov, CEO of LatticeFlow, emphasized that the tool provides companies with a roadmap to adjust their models in line with evolving EU standards. “The EU is still working out all the compliance benchmarks, but we can already see some gaps in the models,” Tsankov noted. “With a greater focus on optimizing for compliance, we believe model providers can be well-prepared to meet regulatory requirements.”
While the European Commission has not yet officially endorsed the LLM Checker, it has been closely monitoring its development. A spokesperson for the Commission described the tool as a “first step” towards translating the AI Act’s legal requirements into technical guidelines that companies can follow.
Looking Ahead: A Compliance Challenge
As the EU moves forward with its AI Act, tech companies must prioritize compliance or risk facing steep penalties. The mixed results from the LLM Checker provide early insights into the regulatory challenges that lie ahead. While some AI developers are ahead of the curve, others must make significant improvements in critical areas like cybersecurity and bias mitigation to align with the forthcoming laws.
With the clock ticking towards the full enforcement of the AI Act, the demand for compliance tools like LatticeFlow’s LLM Checker will only increase. These tools offer Big Tech a clear path to regulatory adherence in an increasingly scrutinized AI landscape.
As Tsankov concluded, “This is an opportunity for AI developers to proactively address these issues, rather than reacting when the regulations are already in force.” For AI companies, the next few years will be pivotal in shaping their compliance strategies and ensuring their technologies meet Europe’s stringent new standards.
In summary, the findings from LatticeFlow’s LLM Checker serve as a wake-up call for major AI players. With the EU AI Act on the horizon, the time for action is now. Companies must take proactive steps to ensure their models not only comply with regulatory requirements but also contribute positively to the broader societal discourse on the ethical use of artificial intelligence.