Guide
What counts as a "significant model change"?
Updated June 16, 2026 · By Max Langley, Bias Audit USA
NYC Local Law 144 requires a fresh bias audit not just annually, but after any significant change to the tool. A significant change generally means anything that could meaningfully alter the tool's outputs: retraining on new data, swapping the underlying model, or materially changing the inputs or how they are weighted. When a change is borderline, the conservative path is to re-audit.
Why the standard is intentionally vague
The law does not list specific triggers, and that is on purpose. A rigid checklist would be easy to engineer around, so the standard focuses on substance: did the change affect how the tool evaluates people. That flexibility protects candidates, but it puts the judgment, and the risk, on the employer using the tool. Teams that retrain models regularly are the most exposed, because a routine update can quietly reset the audit clock.
Changes that likely trigger a re-audit
- · Retraining the model on new or refreshed data.
- · Swapping the underlying model or algorithm.
- · Adding, removing, or re-weighting the inputs the tool scores on.
- · Changing scoring thresholds or how candidates are ranked.
Changes that usually do not
- · Interface or design updates that do not touch scoring.
- · Bug fixes and performance work that leave outputs unchanged.
- · Reporting or export changes downstream of the decision.
The test is whether the change could affect the outcomes the tool produces, not whether you touched the codebase.
How to operationalize it
Build a checkpoint into your model-change process. Before any retrain or model swap goes live for NYC roles, flag it for a quick significance review with counsel, and schedule the re-audit so the tool is never in use without a current audit behind it. A standing relationship with an independent auditor makes that turnaround fast, which matters when a release is waiting. Used without a current audit, a changed tool carries penalties up to $1,500 per day.
What is a "significant model change" under Local Law 144?
There is no single bright line. In practice a significant change means anything that could meaningfully alter the tool’s outputs: retraining on new data, switching the underlying model, or materially changing the inputs or how they are weighted. Any of those can require a fresh bias audit before you keep using the tool.
Does retraining on new data trigger a new audit?
Usually yes. Retraining can shift how the tool scores and ranks candidates, which is exactly what the audit measures. If you retrain on a new dataset, treat it as a likely trigger and confirm with counsel before continuing to use the tool for NYC roles.
What changes usually do not trigger a re-audit?
Cosmetic or non-scoring changes typically do not: UI updates, bug fixes that do not affect scoring, performance optimizations, or reporting tweaks. The test is whether the change could affect the outcomes the tool produces, not whether you touched the software.
Who decides whether a change is significant?
You do, with your counsel. The standard is deliberately broad and the burden is on the employer using the tool. When a change is borderline, the conservative path is to re-audit, because using a materially changed tool without a current audit carries daily penalties.
How do I avoid an accidental compliance gap?
Put a checkpoint in your model-change process. Before any retrain or model swap goes live for NYC roles, flag it for a significance review, and line up the re-audit so the tool is never in use without a current audit behind it.
Sources
- NYC Department of Consumer and Worker Protection, Automated Employment Decision Tools, nyc.gov.
- NYC DCWP final rules on AEDTs, rules.cityofnewyork.us.
Changed your model? Get a fast re-audit.
We keep a standing relationship with clients so a retrain or model swap does not leave a compliance gap. Tell us what changed.
This guide is general information, not legal advice. Confirm your obligations with qualified counsel.