Who Is Liable for Third-Party Facial Recognition Errors?
Who pays when rented facial recognition causes a false arrest? See NIST error data, the Workday ruling, settlements, and the SaaS contract terms that shift risk.
Who is liable when rented facial recognition gets it wrong?
The buyer who rents the software is usually the one held responsible, though courts are starting to disagree. Third-party software liability risk is simple to state and hard to escape: you rent a vendor's algorithm to make a consequential decision, you can't inspect how it works, and you still get held responsible when it's wrong. Facial recognition is the clearest proof we have, and the bill is coming due in courtrooms across the country.
TL;DR
- Renting facial recognition software leaves buyers legally liable for errors they cannot independently audit.
- NIST testing found many commercial algorithms were 10 to 100 times more likely to misidentify Black or East Asian faces, with false positive rates highest for Black women.
- A July 2024 federal ruling held that Workday could be treated as its customers' agent, a sign that courts are starting to reach AI vendors directly for discriminatory outcomes.
- Vendor contracts push the other way, since 88% of AI vendors cap their own liability, often at the subscription fee.
- Settlements are already landing, including Detroit's rules after the Robert Williams case and a $200,000 payout by Jefferson Parish in the Randal Reid case.
- Owning the model, the data pipeline, and the decision logic, or having real audit rights, is the only real audit trail for high-stakes decisions.
What is the pattern behind a wrong match and a real arrest?
Robert Williams was handcuffed in front of his family and detained for nearly 30 hours after Detroit police ran a photo through facial recognition software purchased from DataWorks Plus, which matched him to a shoplifting suspect he had nothing to do with. He was the first publicly known American wrongfully arrested this way, but he was not the last. At least a dozen documented cases now span Detroit, Louisiana, Texas, New York, and North Dakota, each one a different vendor, a different jurisdiction, and the same underlying failure: a police department trusted a match it had no way to independently verify.
This isn't a story about one bad algorithm. It's about what happens when the entity making the arrest doesn't actually understand, or control, the software producing the evidence.
Why is this a software-ownership problem, not just a policing problem?
It's tempting to file these stories under policing reform and move on. That misses the structural issue: a police department, an HR team, or a fraud-detection unit that licenses a facial-recognition or screening tool from a vendor is buying a decision, not just a piece of software. The vendor's training data, error rates, and demographic performance gaps are usually proprietary. The buyer can't audit the model, retrain it on better data, or even fully explain why it produced a given output.
That's the core problem with renting black-box decision software: you inherit the consequences of a system you can't open. When something goes wrong, you can't point to a log, a version history, or a documented reason. You're left explaining a mistake you never had the tools to catch.
What does NIST's testing show about the black box?
The error rates behind these arrests aren't a mystery. They're measured. The National Institute of Standards and Technology tested dozens of commercial facial-recognition algorithms and found many were 10 to 100 times more likely to misidentify a Black or East Asian face than a white one, with false positive rates highest for Black women. That's not a rounding error. It's a documented performance gap that varies enormously from vendor to vendor.
The catch: this data lives in a federal testing report, not the sales deck. Most police departments and companies buying these tools never see vendor-specific error rates broken out by demographic group before they sign a contract. They learn about the gap after someone is wrongfully arrested.
Who actually holds the liability?
Vendor contracts are written on the assumption that the buyer holds the risk. Courts are starting to disagree.
In July 2024, a federal judge ruled that Workday, an HR software vendor, could be treated as an agent of its customers and held directly liable under anti-discrimination law for outcomes produced by its AI screening tool, because it was performing a function the employer would otherwise have done itself. That ruling reaches far beyond hiring software. It suggests that when a vendor's tool performs a delegated business function, like matching a face to a suspect or screening a resume, the vendor doesn't get to hide behind the fact that a customer clicked "buy."
Meanwhile, the paper buyers actually sign points the other way. Vendor contracts routinely cap the vendor's own liability, often at the value of the subscription fee, while offering few guarantees the software will perform as advertised. The result is a liability squeeze: courts are extending responsibility toward the people who built the system, while contracts try to shift responsibility toward the people who merely rented it. Whoever loses that argument in court is the one who pays.
What do the settlements and suits show, and what do they cost?
This is already showing up on balance sheets, not just in law review articles.
- Detroit's settlement with Robert Williams didn't just include damages. It forced the police department to adopt new rules the ACLU calls a potential national model, including a ban on using facial-recognition matches in photo lineups without independent corroborating evidence, plus an audit of every facial-recognition case back to 2017.
- Randal "Quran" Reid spent six days in a Louisiana jail on a warrant built from a Clearview AI match he had never been near. The warrant cited only a "credible source," hiding the fact that an unverified algorithm was the real basis for his arrest. Jefferson Parish settled the resulting federal civil rights suit for $200,000 in 2025 without admitting fault.
- Clearview AI, the company whose faceprint database underlies many of these law enforcement matches, agreed to a $51.75 million equity-based class settlement, approved in March 2025, along with a separate permanent ban on selling its faceprint database to most private businesses under an Illinois biometric privacy settlement.
Third-party involvement is a recognized cost multiplier in security incidents generally, not just in facial-recognition cases. Vendor and supply-chain breaches average roughly $4.91 million, and IBM's breach research has repeatedly flagged third-party involvement as one of the top factors that drives costs higher. Renting critical software doesn't just create legal exposure. It creates a more expensive failure mode across the board.
What's buried in the SaaS contract fine print?
Read a typical AI vendor contract closely and the imbalance is stark. Research from Stanford Law's CodeX group found that 92% of AI vendor contracts claim broad rights to use customer data beyond what's needed to deliver the service, while only 33% offer indemnification for third-party intellectual property claims, both far more vendor-favorable than standard SaaS norms. Only 17% of AI vendors explicitly commit to complying with applicable law, compared to 36% for traditional SaaS agreements. Combined with the finding that 88% of AI vendors cap their own liability, often at the subscription fee itself, the contract you sign is designed to leave you holding the risk for a system you can't inspect.
The hidden costs of SaaS lock-in usually get discussed in terms of price increases and data-retrieval fees. Legal exposure belongs on that same ledger. "We bought it from a vendor" is not a legal shield, and increasingly, it isn't even a good defense.
Why is ownership the only real audit trail?
Here's the affirmative case. If you own the model, the data pipeline, and the decision logic, or you have contractual audit rights that actually let you inspect them, you can catch a demographic performance gap before it produces a wrongful arrest or a discrimination suit. You can retrain on better data. You can document exactly why a system produced a given output, which is the single hardest thing to do with a rented black box after the fact.
This isn't an argument for building everything from scratch. It's an argument that for surveillance-adjacent or high-stakes decision software, the ability to see inside the system isn't optional. Owning the stack behind a consequential decision is what lets you actually stand behind it when someone asks why it went wrong.
| Audit Rights | Buyer's Liability Exposure | Upfront Cost | Documented Data Provenance | Control Over Human-in-the-Loop Rules | |
|---|---|---|---|---|---|
| Rent a vendor's black-box toollow-stakes, low-cost deployment | Low | High | Low | No | Low |
| RecommendedBuild or own the decision layerhigh-stakes decisions about a person's liberty, job, or benefits | High | Low | High | Yes | High |
The same logic applies to the code your own employees ship. If facial-recognition vendors deserve scrutiny for black-box decisions, so does the automation an employee quietly built on top of an AI coding tool and never had reviewed. The audit gap is the same shape either way.
What belongs on a build-vs-buy checklist for high-stakes decisions?
Before you license a vendor tool to make or support a consequential decision about a person, whether that's an arrest, a hire, a loan, or a benefits denial, run it through a short list:
- Audit rights. Can you actually see error rates broken out by demographic group, not just an aggregate accuracy number?
- Mutual liability. Does the contract cap the vendor's liability far below what a wrongful decision could cost you, while leaving your exposure uncapped?
- Human-in-the-loop requirements. Is there a documented rule, like Detroit's post-settlement policy, that a match or score alone can never be the sole basis for action?
- Data provenance. Do you know what data trained the model, and can you get evidence of that in a legal proceeding if you need it?
- Ownership threshold. For the highest-stakes use cases, is the smarter move to build or own the decision layer in-house rather than rent it, even at higher upfront cost?
None of this eliminates risk. But it puts the questions on the table before a settlement check gets written, not after.
The pattern across Detroit, Louisiana, and Clearview's own legal history isn't that facial recognition is uniquely broken. It's that renting a black-box decision engine means renting its mistakes too, and those mistakes eventually land on a real person, in handcuffs, waiting to find out what algorithm decided their fate.
Frequently Asked Questions
Who is liable when a third-party SaaS tool causes a wrongful arrest or discriminatory decision?
It depends on the facts, but liability is no longer automatically limited to the buyer. In Mobley v. Workday, a federal judge ruled a vendor could be treated as an agent of its customer and held directly liable for outcomes from software performing a delegated business function. At the same time, most vendor contracts try to cap the vendor's own liability, often at the subscription price, which can leave the buyer exposed in practice even when courts are willing to reach the vendor.
What is the NIST facial recognition test and why does it matter?
NIST's Face Recognition Vendor Test (FRVT) Part 3 measured how commercial facial-recognition algorithms perform across demographic groups. It found many algorithms were 10 to 100 times more likely to misidentify Black or East Asian faces than white faces, with the highest false-positive rates for Black women. It matters because these error rates are documented and public, yet rarely disclosed to the police departments and businesses actually buying the software.
Has anyone actually been paid damages for a facial-recognition wrongful arrest?
Yes. Detroit reached a settlement with Robert Williams that included policy reforms the ACLU calls a potential national model. Jefferson Parish, Louisiana paid Randal Reid $200,000 after a Clearview AI misidentification led to six days in jail. Clearview AI separately settled a nationwide class action for $51.75 million in equity.
Why can't buyers just audit the vendor's algorithm themselves?
Most vendor contracts don't grant that kind of access. Contract-data research found 92% of AI vendor agreements claim broad rights over customer data while offering limited transparency into the model itself, and only 17% commit to regulatory compliance warranties. Without audit rights built into the contract, buyers are relying on the vendor's word for how the system performs.
Is owning software instead of renting it a realistic fix for this problem?
For the highest-stakes decisions, yes, at least partially. Owning or having real audit rights over the model, data, and decision logic lets an organization catch demographic performance gaps and document its reasoning before a mistake becomes a lawsuit. It's not a fit for every tool, but for surveillance-adjacent or consequential decision software, it changes what an organization can actually defend in court.
How we source, write, and correct our articles. Editorial standards
