Navigating the Compliance Landscape: AI Models and European Regulations

As artificial intelligence (AI) continues to evolve and permeate various sectors, the need for robust regulatory frameworks has become increasingly evident. The European Union (EU) has taken significant strides in this direction, particularly with the introduction of the AI Act, which aims to ensure that AI technologies are developed and deployed responsibly. However, recent findings indicate that some of the most prominent AI models are falling short of these regulations, particularly in areas such as cybersecurity resilience and discriminatory output.

The EU’s AI Act: A Framework for Responsible AI

The EU’s AI Act has been a topic of extensive debate and discussion, especially following the public release of OpenAI’s ChatGPT in late 2022. The model’s unprecedented popularity raised concerns about the potential risks associated with generative AI technologies. In response, EU lawmakers began drafting specific regulations aimed at governing "general-purpose" AI (GPAI) to mitigate these risks.

The AI Act is designed to establish a comprehensive framework for AI technologies, focusing on safety, transparency, and accountability. It categorizes AI systems based on their risk levels—ranging from minimal to high—and sets forth requirements for compliance. Companies that fail to adhere to these regulations face severe penalties, including fines of up to 35 million euros or 7% of their global annual turnover.

LatticeFlow’s LLM Checker: A New Compliance Tool

In a bid to assess compliance with the AI Act, Swiss startup LatticeFlow, in collaboration with researchers from ETH Zurich and INSAIT, has developed a tool known as the "Large Language Model (LLM) Checker." This innovative tool evaluates generative AI models from major tech companies, including Meta, OpenAI, and Alibaba, across various categories aligned with the AI Act’s stipulations.

The LLM Checker assigns scores between 0 and 1 to each model, providing a clear indication of their compliance levels. Recently published results revealed that while models from Alibaba, Anthropic, OpenAI, Meta, and Mistral scored an average of 0.75 or above, significant gaps remain in critical areas.

Key Findings: Areas of Concern

The LLM Checker’s findings highlight several areas where AI models are struggling to meet regulatory standards:

1. Discriminatory Output

One of the most pressing issues in AI development is the tendency for models to produce biased or discriminatory outputs. This reflects underlying human biases related to gender, race, and other factors. In testing for discriminatory output, OpenAI’s "GPT-3.5 Turbo" received a score of 0.46, while Alibaba Cloud’s "Qwen1.5 72B Chat" scored even lower at 0.37. These scores underscore the urgent need for companies to address bias in their models to comply with the AI Act.

2. Cybersecurity Resilience

Cybersecurity is another critical area where AI models are falling short. The LLM Checker assessed models for vulnerabilities, including susceptibility to "prompt hijacking," a cyberattack method where malicious prompts are disguised as legitimate requests. Meta’s "Llama 2 13B Chat" scored 0.42 in this category, while Mistral’s "8x7B Instruct" model received a score of 0.38. These results indicate that companies must prioritize enhancing the cybersecurity resilience of their AI systems.

3. Overall Compliance Readiness

Despite the mixed results, the LLM Checker provides a roadmap for companies to improve their models in line with the AI Act. Petar Tsankov, CEO of LatticeFlow, emphasized that the test results offer valuable insights into compliance gaps, allowing companies to focus their resources effectively. The EU is still in the process of finalizing the enforcement of the AI Act, with a code of practice expected to be established by spring 2025.

The Path Forward: Preparing for Compliance

As the EU continues to refine its regulatory framework, tech companies must proactively address the compliance challenges highlighted by the LLM Checker. This involves not only improving model performance in areas like discriminatory output and cybersecurity resilience but also fostering a culture of accountability and transparency in AI development.

The LLM Checker will be made freely available for developers, enabling them to assess their models’ compliance and make necessary adjustments. While the European Commission cannot verify external tools, it has acknowledged the LLM Checker as a "first step" in translating the AI Act into actionable technical requirements.

Conclusion

The landscape of artificial intelligence is rapidly changing, and with it, the regulatory environment. As the EU’s AI Act comes into effect, companies must navigate the complexities of compliance to avoid significant penalties and ensure the responsible use of AI technologies. The findings from LatticeFlow’s LLM Checker serve as a crucial reminder of the work that lies ahead, urging tech companies to prioritize ethical considerations and compliance readiness in their AI development efforts. By doing so, they can contribute to a safer and more equitable future for AI.