Preparing for the NAIC AI Systems Evaluation Tool: An Insurer’s Readiness Guide

Key Takeaways

• Adoption is close. The pilot is done; the tool goes to the NAIC Fall National Meeting on November 16, 2026.
• It makes the AI Model Bulletin testable. Four exhibits cover AI use, governance, high-risk models and data.
• Evidence beats description. Examiners want a dated record for a specific model, not a policy.
• Some evidence can’t be rebuilt later. Bias tests, monitoring logs and approvals must be created as you go.
• Vendor AI is your responsibility. Third-party models need the same controls as your own.

The NAIC AI Systems Evaluation Tool turns the principles of the 2023 AI Model Bulletin into questions an examiner can put to a specific carrier, about a specific model, on a specific date. After a six-month pilot in twelve states, the revised version is on the agenda for the NAIC Fall National Meeting in Dallas (November 14-17, 2026). Once adopted, any state insurance department can draw on it in market conduct reviews, financial analysis and financial examinations.

This guide is written for chief risk officers, compliance and market conduct leaders, actuaries and AI governance owners at carriers and payers. It is not a news recap. It walks through each part of the tool and sets out three things: what the examiner is likely to ask, what a strong answer looks like, and the artifact that supplies it. It closes with a gap assessment to run now and repeat.

A note on the name. In the August 2026 redraft, the working group retitled the instrument the AI Risk Evaluation Supplement. The pilot states worked from version 4.0, still labeled the AI Systems Evaluation Tool, and most of the industry still uses that name. This guide does too, and flags where later drafts changed the questions.

What Is the NAIC AI Systems Evaluation Tool?

The NAIC AI Systems Evaluation Toll is a set of optional supplemental exhibits that help state regulators measure how much a carrier relies on AI and whether its governance is likely to hold up.

NAIC’s Innovation, Cybersecurity and Technology (H) Committee charged the Big Data and Artificial Intelligence (H) Working Group with building it. The tool supplements, rather than replaces, existing market conduct, financial analysis and financial examination procedures. The working group proposed the tool in summer 2025, worked through several drafts and comment rounds, and held a half-day line-by-line session at the December 2025 Fall National Meeting.

Which States Piloted the NAIC AI Evaluation Tool?

Twelve: California, Colorado, Connecticut, Florida, Iowa, Louisiana, Maryland, Pennsylvania, Rhode Island, Vermont, Virginia and Wisconsin.

The pilot ran from March to September 2026, with each state choosing anywhere from one to more than ten domestic insurers to test it on. States used it in market conduct exams, financial exams, financial analysis and general regulatory inquiries, and met monthly to compare notes and avoid duplicate requests.

Two pilot design choices matter for everyone else:

• Proportionality. Examiners were told to spend their time on high-risk AI systems that could cause serious consumer or financial harm, and less on low-risk back-office tools.
• State discretion. Each jurisdiction could modify the tool, and participation could be mandatory depending on the state’s authority and circumstances. A request you receive may not match any public draft line for line.

Where It Goes From Here

The sequence the working group set out has largely held:

When	Step	Status as of Nov 10, 2026
Mar – Sep 2026	Pilot in 12 states, version 4.0	Complete
Aug 31 – Sep 29	Version 5.0, renamed AI Risk Evaluation Supplement, exposed for 30-day comment	Closed
Oct 16 – Oct 30	Version 6.0 exposed for 14-day comment	Closed
Nov 2	Public working group call on the supplement	Held
Nov 16	Working group session at the Fall National Meeting, Dallas	Upcoming

Source: NAIC Big Data and AI (H) Working Group

Two cautions. First, the Fall Meeting item is consideration for adoption; a slip to 2027 is possible. Second, adoption does not make the exhibits mandatory. The working group has said states would use the tool on a voluntary basis from 2027, with the force of any request coming from the exam or other authority the regulator cites. In practice, “voluntary for the state” still means “answer when asked” for the carrier.

How the Tool Sits Alongside the NAIC AI Model Bulletin

The Bulletin sets the expectation; the tool tests it. The NAIC adopted the Model Bulletin on the Use of AI Systems by Insurers in December 2023. Roughly two dozen states plus the District of Columbia have since adopted it, with a handful of others issuing their own AI rules.

The Bulletin asks each insurer to maintain a written AI Systems Program (AIS Program): the controls and processes it uses to ensure responsible use of AI. It is principle-based and says little about format. The evaluation tool supplies the format. Its exhibits ask a carrier to count its AI, describe its program, document its high-risk models and map its data, in a structure an examiner can compare across companies and states.

Three practical consequences follow:

• Bulletin states are first in line, but not the only ones. A carrier domiciled in a non-Bulletin state can still receive the exhibits. The tool rides on existing exam authority, and its questions apply unfair trade practice and unfair discrimination law that every state already has.
• The tool does not create new governance requirements. The drafts say so explicitly. But a missing answer reads as a gap regardless of what the law strictly requires.
• Vendor AI is your AI. The Bulletin expects the AIS Program to cover third-party systems, and the later drafts add a checklist item on how the company governs, monitors and tests vendor-developed AI. “We bought it” is not a defense.

The Failure Mode That Matters

Most carriers that struggle with this tool will not lack governance. They will lack evidence. They can describe their AI committee, their model risk policy and their fairness standards fluently in a meeting, and then cannot produce a dated record showing that a specific underwriting model was tested for bias before it went live, who approved it, and what monitoring has flagged since.

The tool is built to surface exactly that gap. Its exhibits move from the program level down to the individual model, and the later drafts ask companies to cite the document name and page where each governance element lives. An element nobody can point to reads as missing, however well the practice actually works.

The exhibits also check each other. The Exhibit A inventory, the Exhibit B program description, the Exhibit C model files and the Exhibit D data map describe one program from four angles. When the counts in one disagree with the list in another, or the risk tier in the inventory differs from the tier in the model file, the examiner has found a thread to pull. A polished narrative fails the moment another department produces a record that contradicts it.

The test for every section below is therefore the same: could you hand over the dated artifact for any named model within a day?

What Insurance Regulators Will Ask About AI, Section by Section

The tool gathers four kinds of information, one per exhibit: the extent and use of AI across operations (A), governance and risk mitigation (B), detail on high-risk models (C) and the data used as inputs (D). Vendor oversight and post-deployment monitoring cut across all four, so they get their own sections here.

Area	Exhibit	Core artifact
AI inventory and risk tiering	A	Live AI inventory with use-case classification and risk tier
AI systems program	B	Written AIS Program with board and senior management accountability
High-risk model documentation	C	Per-model file with dated bias and performance testing
Data inputs	D	Data lineage and quality evidence per model
Third-party AI	B, C, D	Vendor due diligence file and contract terms
Post-deployment monitoring	B, C	Monitoring logs, alerts, and remediation records

1. AI Inventory and Risk Tiering (Exhibit A)

What the examiner asks. How many AI systems are in use, and how many were implemented recently, by operational area: marketing, underwriting, rating, claims, fraud, utilization management, reserving and reinsurance. Which ones augment or automate decisions with direct consumer impact or material financial impact. Later drafts split counts by model type, including GLMs and generative or agentic AI, and request the model inventory itself.

What a strong answer looks like. One inventory that produces the counts on demand. Each entry records owner, business purpose, operational area, build-versus-buy status, degree of autonomy, consumer and financial impact, and an inherent risk tier set by a documented method. GLMs are in scope; leaving them out undercounts. The materiality threshold behind “material financial impact” is written down, since it runs through Exhibits A, B and C.

The artifact. A system-of-record AI inventory with use-case classification, risk tier and the date each field was last confirmed. An existing inventory can replace Exhibit A only if it holds the fields the exhibit asks for.

2. The AI Systems Program (Exhibit B)

What the examiner asks. Whether a written AIS Program exists, when it was adopted, how often it is reviewed, who maintains it and what the board or senior management sees. How AI risk enters ERM and ORSA. For consumer-facing uses: human in the loop, how errors are caught and fixed, and staffing changes at go-live. How explainability is ensured. The checklist version asks where the program covers each of about sixteen elements, by document and page.

What a strong answer looks like. A program document naming accountable executives, a board committee with a reporting cadence, and a decision path for approving, changing and retiring models. Minutes prove the cadence; each checklist element maps to a real document section.

The artifact. The written AIS Program, a governance charter, board and committee reporting packs, and a dated approval history. Approvals scattered across email and spreadsheets are the weakest point.

3. High-Risk Model Documentation (Exhibit C)

What the examiner asks. For each in-production high-risk model: name, version, purpose, model type, implementation date, who developed it and whether you can modify it. Its risk classification, known limitations, and whether it automates, augments or supports decisions. Then the evidence: testing for drift, accuracy, unfair discrimination and performance degradation; date of last test; validation and monitoring results; effect on financial reporting; and compliance review against state and federal law.

What a strong answer looks like. Written high-risk criteria, applied consistently with the inventory tier. Each high-risk model has fairness and performance results across policyholder segments, with test date, method, thresholds and the response to any breach.

The artifact. A model file per high-risk system: model card, validation report, pre-deployment bias and fairness assessment, performance by segment, legal sign-off and dated retest history.

4. Data Inputs, Lineage and Quality (Exhibit D)

What the examiner asks. Which of about 25 data categories each model uses, from aerial imagery, demographics and credit-based data to telematics, voice analysis and weather, plus non-traditional data the regulator names. How each is used, whether it is sourced in-house, and which vendor supplies it if not. Regulators may tailor this by line of business and ask for a data dictionary.

What a strong answer looks like. A data map tied to named models, showing source, transformation and use, with documented quality checks and proxy-discrimination analysis for sensitive and external data. Exhibit D maps sources rather than asking about proxy testing directly, but its answers point straight back to the Exhibit C testing evidence.

The artifact. Data lineage records and data quality reports per model, a data dictionary, and the external consumer data and vendor list.

5. Third-Party AI Vendor Oversight

What the examiner asks. How vendor-built models and data are selected, governed, tested and monitored, whether the model is a true third-party product or built for you, and whether you can modify it. The checklist adds a dedicated item on vendor-developed AI, and the narrative asks about professional service providers such as actuarial, claims, MGA and audit firms.

What a strong answer looks like. Vendor AI goes through the same inventory, tiering, testing and monitoring as internal models. Contracts give you the right to documentation, testing evidence and audit, and oblige the vendor to notify you of material model changes and incidents within a set time.

The artifact. A due diligence file per AI vendor: questionnaire, documentation received, your own validation results, and the executed contract clauses on documentation, audit rights, change notification and incident notification.

6. Post-Deployment Monitoring

What the examiner asks. When each model was last tested, what monitoring is in place for drift, accuracy and fairness, how exceptions are caught and resolved, and what remediation followed.

What a strong answer looks like. Monitoring that runs continuously with defined thresholds, alerts that route to named owners, and a log showing each alert, the investigation and the outcome. Quiet periods are documented too: a record that monitoring ran and found nothing is evidence.

The artifact. Monitoring dashboards and logs, alert history, issue tickets and remediation records, with timestamps.

Evidence You Can Assemble Later vs. Evidence You Can’t

Some of what the tool asks for can be written under deadline. A program document can be drafted, an inventory can be compiled, a data dictionary can be built. Much of the evidence cannot. A bias test that was never run before deployment cannot be backdated. Monitoring that never logged cannot be reconstructed. An approval that happened by hallway conversation has no date.

Can be assembled under exam pressure	Has to be generated continuously
Written AIS Program and governance charter	Pre-deployment bias and performance test results, dated
Inventory compiled from interviews	Inventory changes over time (who added, retired, re-tiered what, and when)
Data dictionary and data map	Ongoing data quality checks and drift results
Vendor questionnaire responses	Vendor change and incident notifications received over time
Narrative description of monitoring	Monitoring logs, alerts, investigations, and remediation
Description of the approval process	Approval records: who signed, on what evidence, on what date

The right column is where carriers lose credibility. An examiner who asks for the last fairness test on your auto underwriting model is asking a question with a date in the answer. If that evidence is produced as a by-product of how models are built, approved and run, the exam response is an export. If it is not, it is a reconstruction, and examiners can usually tell.

NAIC AI Exam Readiness: A Gap Assessment to Run Every Quarter

Run this as a mock exam, not a self-attestation. Pick two models you would classify as high-risk, one built in-house and one from a vendor, and give a team one business day to produce each item below. Anything that takes longer, or arrives without a date, is a gap.

Foundation

• Settle one materiality threshold and write down why; use it consistently in the inventory, program and model files.
• Write down your high-risk criteria and confirm the inventory applies them.
• Pull Exhibit A-style counts by operational area and model type straight from the inventory. Do the numbers come out without manual work?
• Confirm GLMs, generative AI and vendor models are all in the inventory.

Program and accountability

• Map each Exhibit B checklist element to a document and page.
• Produce the last 12 months of board or committee reporting on AI.
• Produce the approval record for both test models: approver, date and the evidence they reviewed.

Model evidence

• Produce the pre-deployment bias and performance assessment for both models, broken out by policyholder segment.
• Produce the date and results of the most recent retest.
• Produce the legal and compliance review for each model.

Data and vendors

• For each model, list the Exhibit D data categories used, their sources and their vendors.
• Produce data lineage and quality evidence for one external data source.
• Check the vendor contract for documentation, audit, change-notification and incident-notification terms. Note what is missing for renewal.

Monitoring

• Produce monitoring logs and alert history for both models for the last two quarters.
• Trace one alert from trigger to resolution.

Cross-checks

• Compare the risk tier for each model across the inventory, model file and program documents. Do they match?
• Compare vendor and data details between the model file and the data map.

Then sort every gap into the two columns of the previous section. Assembly gaps can be closed with a sprint. Continuous-evidence gaps need a change to how models are built, approved and run, and every week without that change is evidence you will never have. If you write business in a pilot state, assume the request could arrive before the national version is final.

Make it recurring. Treat the first run as a baseline. After that, repeat it every quarter to line up with board risk reporting. Also run it whenever something changes: a new high-risk model goes live, a vendor model or contract changes, the NAIC adopts a new version of the tool, or a data call or exam notice arrives. Keep each run’s results so you can compare them over time. A list of slow or undated items that gets shorter from run to run is evidence in its own right, because it shows examiners the program is improving, not only that it exists.

How Lumenova AI Helps Carriers Get Exam-Ready

Lumenova AI captures the evidence examiners ask for while your teams build, approve and run models, so it is already dated and in one place when a request arrives.

What the examiner asks for	How Lumenova AI supplies it
AI inventory and risk tiering (Exhibit A)	AI Inventory & Registry: every use case, internal or vendor, registered with an owner, approver and risk tier, tracked across its lifecycle
A documented AIS Program mapped to requirements (Exhibit B)	Pre-built insurance controls and framework coverage for the NAIC Model Bulletin, Colorado insurers, NIST AI RMF and ISO/IEC 42001, managed in AI Risk Management
Dated approval history	AI Governance: a register, approve, enforce, record and review flow that replaces spreadsheet-and-email approvals, with decision records and GRC export
Bias and performance testing for high-risk models (Exhibit C)	AI Evaluations: fairness and accuracy testing before launch and in production, with results kept with each use case
Post-deployment monitoring evidence	AI Observability: monitoring for drift and unfair outcomes, with alerts routed to your ITSM tools
Help preparing the response itself	The Forward Deploy Team, embedded with your governance, compliance, and data science staff to run gap assessments and prepare for exams

The goal is simple: when the examiner names a model and asks for the evidence, you hand over a dated record in a day. Talk to Lumenova AI about running the gap assessment above against your own models.

Frequently Asked Questions

---

Related topics: AI FairnessInsuranceNAIC