Written and maintained by CASRAI Editorial Board
Last updated
This page is not legal advice. It is a plain-language account of two United States enforcement actions in which a regulator did something unusual: it wrote down, control by control, what an organisation deploying an automated system should have had in place and did not. Nothing below establishes a legal standard, and neither document binds anyone but its respondents. In NIKOLAI, CASRAI’s independent frontier-AI-safety dictionary, the closest concept is the Safeguard element on track N6 — a technical or procedural measure intended to reduce risk, identified by type, by the risk domain it targets, and by the deployment scope it applies to. Both cases below are, in effect, safeguard registers written backwards from the harm.
Last verified: 25 September 2026, against the FTC’s press release of 19 December 2023, the stipulated order filed in FTC v. Rite Aid Corporation (E.D. Pa., Case 2:23-cv-05023), the separate statement of Commissioner Alvaro M. Bedoya, and the EEOC’s announcement of 11 September 2023 in EEOC v. iTutorGroup, Inc. (E.D.N.Y., No. 1:22-cv-02565). Paragraph citations to the FTC complaint below are taken from Commissioner Bedoya’s footnotes.
Why these two cases are worth more than their penalties
Measured in money, neither case is large. iTutorGroup paid $365,000. Rite Aid paid no civil penalty at all on the facial-recognition counts. If you rank United States AI enforcement by dollars recovered, both cases disappear.
Rank them instead by specificity, and they sit at the top. Most AI governance writing describes controls in the abstract: do a risk assessment, test for bias, train your people, monitor in production. These two documents are the rare instances where a regulator looked at a system that had already hurt people, and then named the missing controls one at a time, in operative language, with deadlines attached. The FTC’s Rite Aid order is the longer and more useful of the two — Commissioner Bedoya described it in his concurring statement as “a baseline for what a comprehensive algorithmic fairness program should look like,” which is about as close as a sitting regulator comes to publishing a control framework.
That is why this page treats them as a checklist rather than as case notes. The question is not “what did these two companies do wrong” but “what does an enforcement agency consider the irreducible minimum for a deployed automated system that can substantially injure a person.”
Case one: FTC v. Rite Aid
What was deployed
Between 2012 and 2020, Rite Aid ran facial recognition across hundreds of its stores to flag customers it had previously judged likely to shoplift or commit other offences. When the system reported a match against its gallery of stored images, it alerted store staff, who then acted — searching people, calling the police, expelling them from the shop.
The FTC’s complaint alleges the system produced thousands of incorrect matches. Bedoya’s statement pulls out two of them. A Rite Aid employee stopped and searched an eleven-year-old girl because of a false match; her mother reported missing work because the child was so distressed. In another incident, employees called the police on a customer after an alert fired against an image later described as depicting “a white lady with blonde hair.” The customer was Black.
The controls the FTC said were missing
The complaint’s allegations are unusually concrete, and each one reads as the absence of a specific control rather than as generalised carelessness:
- No risk identification worth the name. The complaint alleges Rite Aid gave an internal presentation on expanding the programme that identified exactly one risk: “[m]edia attention and customer acceptance.” Reputational risk to the company was on the register. Injury to a misidentified customer was not.
- No input-quality standard. The gallery was built from low-quality images “unsuitable for automated analysis,” including frames grabbed from closed-circuit television and pictures taken from media reports.
- Incentives pointed the wrong way. Store employees were trained to push for as many enrolments as possible — a volume target on a database whose false-positive rate nobody was measuring.
- No uncertainty signal at the point of decision. The system typically did not surface confidence intervals, so the shop-floor employee receiving an alert had nothing to help them judge whether it was likely to be wrong.
- No demographic testing, against a known failure mode. Face recognition has been documented for years to perform less well on darker skin and on women; the FTC alleges Rite Aid was nonetheless more likely to deploy the technology in stores located in plurality-non-White areas.
- No accuracy testing before or after deployment, and no adequate training on the possibility of a false positive.
There is a separate and older strand to the case. Rite Aid had been under a 2010 FTC data-security order, and the Commission alleged it had failed to implement the comprehensive information security programme that order required — specifically around oversight of third-party service providers, where assessments were conducted orally, without backup documentation, even for vendors classed as high risk. A control that exists only as a conversation is, for enforcement purposes, a control that does not exist.
The order as a control list
The stipulated order runs to thirteen numbered provisions. Provision I bans Rite Aid from using facial recognition or analysis systems for security or surveillance for five years. Provision II requires deletion of the biometric information collected, together with any data, models or algorithms derived from it, and requires third parties to be directed to do the same. The order itself terminates twenty years from issuance.
The substance for everyone else sits in Provision III, the “Mandated Automated Biometric Security or Surveillance System Monitoring Program.” Rite Aid may not operate any such system again unless it first establishes and maintains a programme that identifies and addresses the risk that the system causes “physical, financial, or reputational harm to consumers, stigma, or severe emotional distress” — including risks that those harms fall disproportionately on people by race, ethnicity, gender, sex, age or disability, “alone or in combination.” The required elements:
| Provision | Control required | Cadence |
|---|---|---|
| III.A | Document the content, implementation and maintenance of the programme in writing | Continuous |
| III.B | Designate a named, qualified employee accountable for the programme | Continuous |
| III.C | Written System Assessment of risks to consumers, covering twelve enumerated matters | Before deployment (or within 90 days for a system already running), then at least every 12 months |
| III.D | Safeguards sized to the severity and likelihood of each identified risk | Continuous |
| III.D.1 | Select service providers capable of meeting the programme, bind them contractually, and require them to hand over what the assessment needs | On selection and ongoing |
| III.D.2 | Documented training for every Operator | At least annually |
| III.E | Evaluate and adjust the programme in light of assessment results and all monitoring and testing; remediate identified risks substantially and in time | At least every 12 months |
| III.F | Give the written System Assessment and the programme to the board of directors or governing body — or, absent one, to a responsible senior officer | At least every 12 months |
| III.G | Do not deploy, or discontinue deployment, where accuracy is not substantiated or identified risks are not eliminated in time | Trigger-based |
| IV | Tell people they have been enrolled, tell them when action is taken against them on that basis, and tell them how to contest both; answer complaints promptly | Event-driven |
| V–VI | Retention limits on biometric information; clear disclosure that such a system is in use | Continuous |
| IX–XI | Third-party information security assessments; cooperation with the assessor; annual certification | Initial, then biennial; certification annually |
| XII | Report covered incidents to the Commission within 10 days of notifying any US federal, state or local authority | Event-driven |
Three details in the order that most frameworks miss
Testing has to replicate the deployment environment. The System Assessment must review whether testing was “conducted using reliable methodologies and under conditions similar to those in which the [system] will operate,” and Bedoya summarises the annual testing requirement as running “under conditions that materially replicate conditions in which the system is deployed.” That is a validity requirement, not a coverage requirement — it is precisely the gap NIKOLAI’s N5 track treats as an evaluation-validity threat: a benchmark result obtained under laboratory conditions tells you little about a system running on shop CCTV.
Training has to cover the human, not just the machine. Provision III.D.2 requires annual Operator training on four things, and the third of them is remarkable: “an overview of the types of human cognitive bias, such as automation bias and confirmation bias, that could foreseeably affect Operators’ interpretations of the Outputs.” The FTC identified the real failure point as the interface between a probabilistic output and a person disposed to believe it. As Bedoya put it, “a computer is telling a person that the customer is suspicious. And people trust computers.”
There is a stop condition with an evidentiary bar. Provision III.G forbids deployment unless the company holds “competent and reliable scientific evidence” — defined in the order as tests or studies conducted and evaluated objectively by qualified persons and generally accepted in the profession — sufficient to substantiate that the outputs are likely to be accurate. The default is off. Evidence turns it on.
Case two: EEOC v. iTutorGroup
What was deployed
The mechanism here was almost banal by comparison. In 2020, iTutorGroup — which recruited United States-based English-language tutors to teach students in China — programmed its tutor application software to automatically reject female applicants aged 55 or older and male applicants aged 60 or older. More than 200 qualified United States-based applicants were rejected on that basis.
No model was involved. No training data, no drift, no opaque embedding. A date-of-birth field, a threshold, and a rejection path. It is worth holding that next to Rite Aid, because the two cases bracket the range: one is a statistical system failing unevenly, the other is a deterministic rule encoding the discrimination directly. Both were enforced under law written long before either technique existed — the FTC used its ordinary unfairness authority under Section 5, and the EEOC used the Age Discrimination in Employment Act of 1967.
EEOC Chair Charlotte A. Burrows put the principle in one line: “Age discrimination is unjust and unlawful. Even when technology automates the discrimination, the employer is still responsible.”
What the consent decree required
The EEOC filed suit on 5 May 2022 in the Eastern District of New York after conciliation failed; the parties filed a joint notice of settlement on 9 August 2023, and the agency announced the terms on 11 September 2023. The decree provides $365,000 for distribution to the rejected applicants, and imposes:
- Injunctions against age-based and sex-based hiring discrimination.
- A new anti-discrimination policy.
- Training for all personnel involved in tutor recruitment.
- A prohibition on requesting applicants’ dates of birth — the cleanest control on the list, because it removes the input the rule ran on rather than trying to govern the rule.
- EEOC monitoring for a minimum of five years.
- If United States operations resume, notification to the previously rejected applicants and an opportunity to be considered.
The date-of-birth prohibition is the part worth copying. Most proposed remedies for discriminatory screening are downstream: audit the outputs, measure impact ratios, document the justification. The decree instead cuts the protected attribute out of the pipeline. That is a design control, and design controls are cheaper to verify than statistical ones.
How each failure was actually found
NIKOLAI’s Discovery method element on track N7 proposes recording, for any incident, the detection channel, the detecting party, and the latency between occurrence and detection. Run these two cases through it and the result is uncomfortable.
| Rite Aid | iTutorGroup | |
|---|---|---|
| Detection channel | External — consumer complaints and regulator investigation, after years of operation | External — a single rejected applicant testing the system |
| Detecting party | The FTC, not the operator | The charging party, not the employer |
| Latency | Years; the system ran from 2012 to 2020 | Immediate, but only because the applicant happened to run the experiment |
| Internal channel that should have caught it | Employee “bad match” reporting existed, but staff were not adequately trained on false positives | None described |
The iTutorGroup discovery is the more instructive. The applicant submitted her application with her real date of birth and was rejected. She then reapplied with a later date of birth — and was offered an interview. That is a two-sample A/B test with a sample size of one, conducted by the person harmed, at no cost, and it was decisive. The employer could have run the identical test at any point in the development of the software and did not.
Neither organisation discovered its own failure. In both cases the discovery method was an outsider. Any governance programme whose detection story depends on somebody outside the organisation noticing is not a detection story.
What survives a change of administration, and what does not
There is a postscript that matters for anyone trying to work out how durable this material is.
The EEOC’s non-binding AI technical-assistance documents — the guidance on adverse impact in algorithmic selection procedures and on the Americans with Disabilities Act and AI, published in 2023 under its Artificial Intelligence and Algorithmic Fairness Initiative — were removed from eeoc.gov in late January 2025, following executive orders rescinding the previous administration’s AI policy. The removal is documented in contemporaneous law-firm reporting rather than in an agency press release. It did not change anyone’s obligations: Title VII, the ADA and the ADEA apply to automated hiring decisions exactly as before, because those documents only ever explained existing law.
The iTutorGroup consent decree was unaffected, because it is not guidance. It is a court order in a live case with a five-year monitoring term.
Rite Aid, meanwhile, filed for Chapter 11 in October 2023 — which is why the FTC’s order required bankruptcy court approval before it could take effect — and filed a second Chapter 11 in May 2025, selling prescription files for more than a thousand pharmacies and moving to close its remaining retail locations.
So the honest summary is this: the most detailed algorithmic-fairness control list any United States regulator has published binds a company that is winding down its stores, and the clearest federal explanation of how discrimination law applies to hiring software has been taken off the agency’s website. The underlying statutes are untouched, and the two orders remain the best available evidence of what an enforcement agency thinks adequate controls look like. But nobody should mistake either for a standard. They are settlements, negotiated against particular evidentiary records, and they bind two respondents.
Turning the orders into something you can use
If you strip both documents down to what an organisation deploying a consequential automated system would have to be able to show, you get roughly eleven items. They are not a compliance obligation for anyone who is not Rite Aid or iTutorGroup. They are a set of questions with a known provenance — a regulator has already said, at least once, that their absence contributed to harm.
- Does your risk register list harms to the people the system acts on, or only harms to you? The single-entry “media attention and customer acceptance” register is the failure mode to check for first.
- Is there a named person accountable for the programme, by role, in writing?
- Was there a written assessment before go-live, and is it refreshed at a fixed interval rather than on request?
- Was the system tested under conditions resembling deployment — the actual cameras, the actual applicant pool, the actual latency — rather than only on a clean benchmark?
- Was it tested for differential error rates by race, ethnicity, gender, sex, age and disability, including in combination?
- Is there an input-quality standard, and does anything enforce it?
- Do the people acting on outputs see a confidence signal, and have they been trained on automation bias and confirmation bias by name?
- Are vendors contractually bound to the same programme, and obliged to supply what your assessment needs? See our guide to assessing third-party AI vendor risk for how that flow-down is usually drafted.
- Do affected people get told that they are in the system and that an action was taken on that basis, with a route to contest it?
- Does the written assessment reach the board or a responsible senior officer on a schedule? Provision III.F is a reporting line, and the duty it presupposes is the one discussed in our guide to board oversight of frontier AI under Caremark.
- Is there a stop condition with a stated evidentiary bar — and has anyone rehearsed using it?
For the hiring case specifically, add the iTutorGroup control: do not collect the protected attribute before the offer stage unless something genuinely requires it. If you are operating automated employment decision tools in New York City or California, that sits alongside a live set of statutory obligations — see our comparison of the NYC Local Law 144 and California FEHA rules on automated decision systems. For the wider pattern of what the FTC has actually charged in AI-adjacent matters, see our record of SEC and FTC AI-claims enforcement.
The research-administration angle
Universities and academic medical centres sit squarely inside both fact patterns, and often do not realise it.
On the hiring side, the ADEA, Title VII and the ADA apply to a university exactly as they applied to iTutorGroup. Institutions screen very large applicant pools for postdoctoral, research-staff and technician posts, frequently through an applicant-tracking system with configurable knockout rules that a departmental administrator can edit without any governance review. The iTutorGroup decree’s control — do not request a date of birth before an offer — is a five-minute configuration check in most systems, and nobody needs an AI governance committee to authorise it.
On the surveillance side, campus security, animal facilities, high-containment laboratories and controlled-access research spaces increasingly use biometric entry systems, sometimes procured through facilities rather than through IT. A false non-match at a laboratory door is an inconvenience; a false positive that routes a person to campus police is the Rite Aid fact pattern with a different badge. Provision III.D.1’s vendor flow-down and Provision III.F’s reporting line are the two that institutions tend to be missing, because the system was bought as building infrastructure rather than as an information system.
Research security and export-control offices are a third case: personnel screening against restricted-party and sensitive-affiliation lists is automated matching on names, and false positives there attach to an individual researcher’s career. The same eleven questions apply, and the name-matching literature has the same demographic error structure.
Frequently asked questions
Is the FTC’s Rite Aid order a standard that applies to other companies?
No. It is a stipulated order binding two respondents, entered in a single federal case. It does not create obligations for anyone else and it is not a rule. Commissioner Bedoya described it as a baseline that industry should understand, which is a signal about enforcement posture, not a legal duty. Treat it as the clearest available evidence of what one regulator considers adequate — and as an indication of what a future order against a different company might contain.
Did Rite Aid pay a fine?
Not on the facial-recognition counts. The relief is injunctive: a five-year ban on facial recognition for security or surveillance, deletion of the biometric data and of models and algorithms derived from it, and a mandated monitoring programme if any such system is deployed again. The order runs for twenty years from issuance.
Was iTutorGroup an “AI” case?
Only loosely, and the distinction matters. The software applied a fixed age cut-off to a date-of-birth field. There was no model and no learned behaviour. It is routinely described as the EEOC’s first AI-related settlement, and it is a landmark for automated employment decisions, but the mechanism was a hard-coded rule. That is arguably what makes it useful: the legal analysis did not depend on the sophistication of the tool.
Does the removal of the EEOC’s AI guidance mean automated hiring tools are no longer regulated federally?
No. The documents removed from eeoc.gov in January 2025 were non-binding technical assistance explaining how existing statutes apply. Title VII, the ADA and the ADEA were not amended and continue to apply to automated hiring decisions. What changed is the availability of the agency’s published interpretation, not the underlying law. State-level obligations, such as New York City’s Local Law 144, are separate and unaffected.
What is the single most transferable control from the two orders?
Probably Provision III.D.2’s cognitive-bias training, because almost nobody has it. Organisations commonly test the model and then hand its output to a human whom they treat as an independent check. The FTC’s order treats that human as part of the system, with a known failure mode — automation bias — that has to be trained against explicitly. If you audit only one line item from this page, audit whether your operators have ever been told, in a session you can evidence, that the system will sometimes be confidently wrong.
How do these cases relate to NIKOLAI?
NIKOLAI is CASRAI’s own independent frontier-AI-safety dictionary; it is unendorsed, and its crosswalk rows are shadow mappings unless an organisation has filed a Mapping Declaration. Neither the FTC nor the EEOC has filed one, so nothing on this page is a NIKOLAI mapping. The relevance is conceptual: Provision III is a register of Safeguards in the N6 sense, and the way both failures came to light is a textbook illustration of why the N7 Discovery method element records the detecting party separately from the detection channel.
