EU AI Act analysis

Summary analysis of AI Act

The EU AI Act introduces the notion of a high-risk AI system that is a subject to assessment before they enter the market as well as to the registration obligation (article 6). 

High-risk AI systems include (annex III), among others,  

In connection to Education and vocational training — those:

• intended to be used to determine access or admission to institutions of all levels
• intended to be used to evaluate learning outcomes
• intended to be used for the purpose of assessing the level of education in the context of educational and vocational training institutions at all levels

In connection to Employment, workers’ management and access to self-employment — those:

• intended to be used for the recruitment or selection of natural persons, in particular to place targeted job advertisements, to analyse and filter job applications, and to evaluate candidates;
• intended to be used to make decisions affecting terms of work-related relationships, the promotion or termination of work-related contractual relationships, to allocate tasks based on individual behaviour or personal traits or characteristics or to monitor and evaluate the performance and behaviour of persons in such relationships.

In connection to Credit, Life & Health Insurance, etc (essential private and public services and benefits) — those:

• intended to be used by public authorities or on behalf of public authorities to evaluate the eligibility of natural persons for healthcare services;
• intended to be used to evaluate the creditworthiness of natural persons or establish their credit score;
• intended to be used for risk assessment and pricing in relation to natural persons in the case of life and health insurance;

In connection to Law enforcement, Migration, Administration of justice and Democratic processes (elections and voting).

Apart from High-risk AI models, the Act recognizes general-purpose AI models and imposes obligations to them as well. Providers of general-purpose AI models must maintain technical documentation and make it available to authorities and providers of AI systems who intend to integrate them into their AI systems (article 53).

Additionally, the Act defines criteria for classifying general-purpose AI models as having systemic risk. Models are considered systemic risk if they have high impact capabilities, as evaluated by technical tools and methodologies (article 51). Their providers must evaluate models using standardised protocols, assess and mitigate systemic risks, report serious incidents, and ensure cybersecurity. They can adhere to codes of practice (until a harmonised standard is published) or demonstrate alternative means of compliance. 

The document is addressed towards two main groups of subjects: providers (manufacturers) of the AI systems and their deployers. 

Providers bear the strictest part of obligations, including quality management, corrective actions in case of suspected breach and the duty of information, etc. Yet, each participant of the so-called AI value chain — distributors, importers, deployers, or third parties — become providers of high-risk AI systems if they rebrand, substantially modify, or change the intended purpose of an AI system, making it high-risk (article 25).  

On the part of providers,—

High risk AI systems shall be made subject to continuous risk management (article 9), data governance for training and other purposes (article 10), their technical documentation shall be kept and made available to public bodies (article 11). High-risk AI systems shall technically allow for the automatic recording of events (logs) over the lifetime of the system (article 12) as well as have a quality management system in place (articles 16 , 17).

As for providers (developers) of AI systems, they must ensure their operation is sufficiently transparent, allowing deployers to understand and appropriately use their outputs. Instructions should also detail human oversight measures, computational and hardware requirements, maintenance needs, and logging mechanisms to ensure proper functioning and compliance (article 13).

Deployers of high-risk AI systems must ensure they use the systems according to instructions, with competent human oversight, and monitor their operation. They must manage input data, keep logs for at least six months, and inform providers and authorities of any risks or incidents (article 26).   

On the part of deployers,—

High-risk AI systems shall be designed and developed in such a way that they can be effectively overseen by natural persons. To do so the high-risk AI system shall be provided to the deployer in such a way that natural persons to whom human oversight is assigned may (article 14): 

• properly understand its relevant capacities and limitations;
• remain aware of the possible tendency of automatically over-relying on their output (automation bias);
• correctly interpret the high-risk AI system’s output;
• intervene in the operations — stop, override or reverse the output of the high-risk AI system.

Before deploying certain high-risk AI systems, specific private entities must perform an assessment of the impact on fundamental rights, describing the usage, affected groups, risks, human oversight, and mitigation measures. Deployers must notify authorities of the results using a provided template (article 27). 

On the part of providers and deployers alike,—

The act imposes the transparency obligations on providers and deployers of certain AI systems (article 50):

• AI-generated or -manipulated content must be clearly marked and detectable as artificially generated.
• AI-generated text must be disclosed as artificially generated.
• Information must be provided clearly, distinguishably and in an accessible way
• The Commission shall develop guidelines on the practical implementation of transparency obligations.

To exercise proper governance the EU Commission established the AI Office (article 64).  

What do these requirements by EU AI Act mean in practice?

This may mean that in certain areas including Education and Training, Employment and work relations as well as financial services connected to credit and insurance, the use of AI models in business operations becomes highly regulated. This opens a door to legal prosecutions of those failing to comply, which poses a considerable financial risk to companies that run what are called high risk AI systems.

For example, private businesses that run financial facilitation or brokerage services may not use AI (as before) in order to simulate or attempt at determining the credit score or availability and price of financial or insurance services. 

Also, multiple mobile applications and websites that would use the AI to assess a person's readiness to any kind of educational levels or language levels may not lawfully proceed doing so without falling into the scope of this Act, thus, satisfying the conformity requirements.

Importantly, a huge market of employment and worker’s management also fall under the regulation. First, use of AI to screen and filter CVs becomes regulated — companies must assess their impact, register these systems as well as adhere to competent human oversight and detailed documentation of their activity. The same concerns usage on AI models to allocate tasks at work and/or evaluate employees' performance.

Potentially, in case of damage to a private person as a result of operation of non-compliant business, it will be considerably easier to prove their fault — simply by pointing to the lack of evidence of compliance. 

This brings us to the important conclusions — in the wake of this Act, organizations working with high-risk AI systems and general-purpose AI models as having systemic risk must adopt internal risk management, quality management and governance, etc procedures that shall be evidenced to their clients — on site and in app. One of such obligations — rising from use of general-purpose AI models as having systemic risk, including ChatGPT, Claude, etc — is to clearly mark all content generated or manipulated by AI, i.e. drafted, edited, proofread, etc.