Many critics of police facial recognition use warn of the potential for racial bias that leads to false arrests, and unfortunately that appears to have happened. The ACLU has filed a complaint against Detroit police for the wrongful arrest of Robert Williams when a DataWorks Plus facial recognition system incorrectly matched security footage against Williams’ driver’s license, marking him as a suspect. Officers showed the match to an offsite security consultant who identified Williams as the culprit, but this person never saw the perpetrator first-hand.

The ACLU argued that the DataWorks system “can’t tell Black people apart” and that the whole system was “tainted” by officers’ assumptions that the facial recognition system produced the right suspect. In a Washington Post opinion piece, Williams added that he was concerned about the tech even if it was completely accurate — he didn’t want his daughters’ faces to go into a database and prompt future police questioning when they’re spotted at a “protest the government didn’t like.”