Quick Intro: Who’s in the Room (so we don’t waste anyone’s time)

• What's your primary role on pricing, underwriting, or analytics decisions? Options: Chief Actuary, Head of Pricing, CUO/Chief Underwriting Officer, VP of Personal Lines, VP of Commercial Lines, CIO/Head of Data & Engineering, Other
• Who are the decision-makers who must sign off on model approach, regulatory filing, and deployment? (select all that apply) Options: Chief Actuary, Head of Pricing, CUO, CISO/IT Security, Head of Data Engineering, Head of Underwriting, General Counsel, Board/Executive Committee, Other
• Who will be the day-to-day sponsor and primary point of contact for this engagement? Please include name, title, and preferred channel.
• Have you partnered with external pricing/modeling vendors before? If yes, what worked and what didn’t? Options: Yes — very productive, Yes — mixed results, Yes — not successful, No, this would be our first time
• Tell us briefly about the one recent pricing decision or filing that felt especially painful or slow.

If We Keep Doing What We’re Doing, What Will Break First?

• Is it possible your current pricing is quietly losing profitable customers or mispricing risk in ways you haven’t fully measured? Options: Yes — we suspect material leakage, Some — we see pockets of under/overpricing, Unclear — data gaps prevent us from knowing, No — we believe current approach is stable
• Which of these consequences worries you most if pricing doesn’t improve? Options: Loss of profitable customers to competitors, Deteriorating combined ratio/loss ratio, Regulatory pushback or DOI scrutiny, Agent pushback and decreased sales, Underwriter rejection of model outputs, Other
• Can you estimate the annual financial impact or range where you think pricing errors currently sit (e.g., premium leakage, loss ratio drift)? If you don’t have a number, describe how you’d like us to help quantify it. Options: > 5% of premium, 2–5% of premium, < 2% of premium, Unknown — needs analysis
• When pricing has gone wrong in the past, what internal reactions have followed (e.g., model rewrites, emergency rate increases, tightened underwriting)?
• How does pricing uncertainty make you feel as an executive — frustrated, cautious, exposed to regulatory risk, or something else? Options: Frustrated, Cautious, Exposed to regulatory risk, Energetic to change, Other

Behind the Curtain: How Your Pricing Actually Runs Today

• If your pricing stack were a machine, which parts feel most brittle or kludged right now? Options: Data ingestion/pipelines, Feature engineering/variable refresh, Model training/experiments, Rating engine integration, Underwriting rules layer, Regulatory filing artifacts, Other
• Which modeling approaches are currently used to set rates for the line(s) we’ll focus on? Options: GLMs / actuarial models, Rule-based / manual relativities, Gradient-boosted trees / ML, Bayesian hierarchical models, Proprietary vendor scoring, No formal model (expert judgment)
• Which data sources feed your pricing today? Select all that apply and note the best/worst quality ones in the next question. Options: Policy system, Claims system, Exposure/units-of-risk data, Third-party data (credit, demographics), Telematics/IoT, Underwriting notes, Agent data, Other
• Describe the most common data quality or availability problems we should expect (missing fields, inconsistent identifiers, lag, small cohorts, etc.).
• How mature is your engineering and MLOps capability to retrain, validate, and deploy a model on an automated cadence? Options: Production-ready MLOps and CI/CD, Basic pipelines, semi-automated, Ad-hoc scripts and manual handoffs, No engineering support — acturial-only workflows
• Where does your pricing model ultimately live or integrate (rating engine, underwriting system, batch export to policy admin)? Options: Rating engine (real-time), Batch score export to policy admin, Underwriting portal only, BI/dashboard only, Not yet integrated

What’s Getting in the Way of Better Pricing — Let’s Call It Out

• If I told you we could improve segmentation by 10–15%, what would your internal barriers be to adopting it? Options: Regulatory approval risk, Underwriter resistance, Agent/market pushback, IT/integration cost, Model explainability concerns, Procurement/budget timing
• How constrained are you by regulatory transparency requirements versus your willingness to use black‑box models? Options: Must be fully transparent (GLMs preferred), Prefer interpretable models but open to hybrid approaches, Comfortable with opaque models if risk-managed, Unsure — need guidance
• What has been the most surprising cause of delay or failure in past model deployments? Options: Data readiness, IT integration, Stakeholder sign-off, Regulatory questions, Underwriter adoption, Budget/contractual issues, Other
• Which internal process (pricing governance, actuarial review, IT change control, underwriting training) typically takes the longest before a new rate can go live? Options: Pricing governance review, Actuarial peer review, IT change control and testing, Regulatory filing prep and response, Underwriter/agent enablement
• How would you describe your organization’s appetite for controlled risk during a pilot — conservative, balanced, or aggressive? Options: Conservative, Balanced, Aggressive

What Would Winning Actually Look Like (so we can aim precisely)?

• If a new pricing model met your goals, what specific metric improvement would you want to see first (pick one primary outcome)? Options: Reduction in loss ratio, Increase in premium adequacy (rate level), Improved Gini/segmentation, Higher retention of profitable customers, Faster time-to-file/deploy
• Beyond a headline metric, what three signals or leading indicators will convince you the model is working (e.g., lift curves, conversion by segment, claims frequency by cohort)?
• How soon after project start would you expect a credible pilot or filing-ready artifact? Options: 8–12 weeks, 3–6 months, 6–9 months, Longer than 9 months
• Who should own which KPIs post-deployment (actuarial, pricing, underwriting, data engineering, product)? Please map roles to outcomes.
• What level of model explainability or documentation do you expect for actuarial peer review and DOI examination? Options: Full GLM-style documentation, Hybrid: interpretable summaries + technical appendix, Technical model internals with explainability tools, Only high-level business logic

Trade-offs: Are You Willing to Sacrifice Explainability for Lift?

• Would you accept a small loss in interpretability if it meaningfully improved segmentation and profitability? Options: Yes, if controls and explainers exist, Maybe — depends on regulator tolerance, No — interpretability is non-negotiable
• Which explainability artifacts are required for you to accept a complex model? Options: Variable importance and partial dependence, SHAP/LIME explanations on cohorts, Surrogate GLM approximations, Case-level explainability for appeals, Full technical appendix
• What level of adverse selection monitoring and rollback control would make you comfortable in the first 6–12 months post-deployment? Options: Strict guardrails + immediate rollback, Monitored with phased premium caps, Standard monitoring with thresholds, Minimal monitoring — confident in model
• What internal approvals or committees must sign off before a model can be filed or released to production? Options: Actuarial peer review, Pricing governance committee, Risk committee, IT change control, Legal/Compliance, Other
• If we propose an ML approach you’re wary of, what evidence would most persuade you to proceed? Options: Back-test results on holdouts, Pilot with limited lines/regions, Regulatory-simulated disclosure, Underwriter acceptance testing, Cost/benefit analysis

Regulators, Underwriters, and the Market — Who Will Push Back, and How?

• How have state DOIs historically responded to model/filing changes from your company or peers? Options: Supportive and constructive, Skeptical but workable, Challenging and slow, Mixed by state
• Do you foresee specific regulatory constraints (e.g., prohibited variables, required disclosures) that we must design around? Options: Yes — explicit prohibitions, Yes — strong disclosure needs, Some variability by state, No major constraints anticipated
• How important is having actuarial-facing documentation versus consumer-facing explanations for this project? Options: Both equally important, Actuarial-facing is primary, Consumer-facing is critical for agents/DOI, Unsure — need guidance
• How much involvement do underwriters and agents need during model testing to ensure adoption? Options: High — hands-on testing essential, Medium — demos and targeted workshops, Low — post-deployment training only
• Describe one regulatory or market interaction you’d like us to prepare materials for (e.g., state filing, DOI Q&A, agent town hall).

Decision, Budget, and Timing — Where Are You Ready to Commit?

• What is your target date for a go/no‑go decision on a pilot or full engagement? Options: Within 2 weeks, 1 month, 2–3 months, 3–6 months, Undetermined
• How does your procurement/budget cycle affect timing for starting this work? Options: Funds available now, Needs next quarter approval, Annual budget cycle — long lead, Ad hoc approvals possible
• Which commercial model do you prefer for an engagement like this? Options: Fixed-fee pilot + success fee, Time & materials, Outcome-based with shared uplift, Not sure — want options
• What are the non-negotiable acceptance criteria for moving from pilot to full deployment? Options: Statistical lift thresholds, Regulatory filing approval, Operational readiness tests, Underwriter sign-off, IT integration validation
• Who holds final sign-off authority for commercial terms and for technical acceptance (list names/titles)?

Practical Next Steps — Small Bets That Build Trust

• Which low-risk pilot would you find acceptable to demonstrate value quickly? Options: Geographic holdout pilot, Product-line subgroup pilot, Feature-lift comparison vs baseline GLM, Backtest-only proof of concept
• What data artifacts can you reasonably deliver within two weeks to get started (e.g., 12–36 months of de-identified policy and claims data)? Options: Full de-identified policy + claims, Policy-only with limited claims, Sample cohorts by product, None ready — need to prepare
• What governance cadence would help you feel in control during the pilot—weekly checkpoints, biweekly demos, or monthly steering? Options: Weekly checkpoints, Biweekly demos, Monthly steering meetings, Ad-hoc updates only
• Who should attend a 30–45 minute kick-off working session from your side (list roles/titles)?
• Would you be willing to share a brief sample dataset or a redacted filing example now to accelerate scoping? Options: Yes — sample dataset available, Yes — redacted filing available, Not now — can provide after NDA, No

Quick Snapshot: Tell Us About Your Current Pricing Landscape

• Which line(s) of business are we discussing for this project? Options: Personal Auto, Homeowners, Small Commercial, Large Commercial, Specialty Lines, Other
• Which pricing model families are in active use today (select all that apply)? Options: Generalized Linear Models (GLM), Gradient-Boosted Trees / XGBoost/LightGBM, Neural Networks, Decision Trees / Random Forests, Rule-based / Manual Relativities, Vendor-provided Scorecards, No formal model — spreadsheet/heuristic
• Who currently owns model development and who owns deployment (list roles or teams)? Options: Actuarial, Pricing team, Data science/analytics, IT/engineering, Third-party vendor, Shared ownership
• Briefly summarize the last time you changed a rating plan (what triggered it and how long did implementation take):
• Do you maintain model version history and changelogs that let you reproduce prior scoring? Options: Yes — full versioning and lineage, Partial tracking — some artifacts, No formal versioning

If Your Pricing Were Speaking, What Would It Confess?

• What’s one assumption baked into your pricing stack that you suspect is wrong but have avoided changing?
• Which portfolio segments most consistently underperform or outperform expectations (be specific by product, geography, or cohort)?
• Which metrics do you monitor to detect model degradation, and which of those have proved least reliable? Options: Population Stability Index (PSI), AUC / ROC, Lift / decile analysis, Claim frequency/severity drift, Loss ratio vs expected, Manual spot checks, We don’t have consistent metrics
• How often do you observe selection effects or unexpected shifts in customer mix after rate changes? Options: Almost always, Often (multiple times per year), Occasionally (yearly), Rarely, Never / Not tracked
• Tell us about a time a model change produced an unexpected business or regulatory reaction—what happened and what did you learn?

Where Does Your Data Really Come From?

• Which of these data sources feed pricing models today? Options: Policy administration system, Claims system, Exposure / telematics, Credit and underwriting bureau data, Vehicle / VIN data, Third‑party vendor enrichments, Public records / geospatial / census, Agent-provided fields, Other
• Which sources have automated ingestion pipelines versus manual uploads? Options: Mostly automated, Mixed (some automated, some manual), Mostly manual uploads
• Highlight the one or two data sources that cause the most headaches (missing fields, late delivery, unreliable joins) and explain why:
• Do you have stable unique identifiers (e.g., policy ID, claimant ID) that allow deterministic joins across policy, claims, and exposure? Options: Yes — reliable across systems, Partially — some links missing, No — joins are imperfect or manual
• Do you snapshot or version training datasets so a model build can be reproduced exactly later? Options: Yes — snapshots and metadata, Partially — ad-hoc snapshots, No — cannot reproduce exactly

Engineering Confidence: How Production-Ready Are Your Models?

• Where do model runs currently execute: exploratory environments or production pipelines? Options: Exploratory notebooks only, Scheduled batch pipelines, Streaming / near-real-time pipelines, Vendor-hosted scoring, Not currently running models
• Describe your typical deployment workflow for a pricing model (code repo → tests → approvals → deploy):
• What test coverage do you have for data validation, feature stability, and scoring logic? Options: Comprehensive automated tests, Partial automated tests + manual checks, Manual testing only, No formal tests
• How long does a deployment typically take from code freeze to scores available for rate generation? Options: Under 1 week, 1–4 weeks, 1–3 months, 3+ months, We don’t have a predictable timeline
• Who is responsible for day-to-day maintenance of model code and pipelines? Options: Actuarial team, Data science/ML engineers, IT/DevOps, Third-party vendor, Shared responsibility

How Do Models Plug Into Your Pricing Engine?

• Which integration points must be completed for a model to produce final, billable rates? Options: Rating engine (rate tables), Policy administration system, Underwriting rules engine, Billing system, Agent portals / quote engines, Regulatory filing toolchain, Analytics dashboards
• How are rate outputs currently delivered to your rating engine (flat files, APIs, manual edits, other)? Options: Flat files (CSV/Excel), APIs / service calls, Direct DB updates, Manual rating tool edits, Other
• What limitations in your rating engine force you to simplify models (for example: no interactions, limited bucket counts, monotonic requirements)?
• How do you reconcile automated model outputs with existing manual relativities or business rules during release?
• How often do integration or deployment errors occur during rollouts and what are the typical root causes? Options: Never, Rarely (<1/year), Occasionally (1–3/year), Often (>3/year)

Failure Modes We Need to Map Before They Surprise You

• When a model has failed in production, which root causes have you observed most often? Options: Data quality issues, Code / pipeline bugs, Flawed assumptions or target leakage, Integration errors, Governance/approval breakdowns, Multiple simultaneous causes
• Please list recent incidents or near-misses (briefly: what happened, business impact, and how it was remediated):
• Do you have documented rollback procedures and the ability to revert rates quickly if adverse selection or other issues appear? Options: Yes — automated rollback, Yes — manual but quick, Partial — slow rollback, No rollback plan
• How do you quantify the financial impact of a model failure (premium erosion, loss ratio change, customer churn)? Options: Regularly quantified with metrics, Estimated after event, Ad-hoc gut estimates, Not measured
• Who is responsible for external communications (agents, regulators, customers) when pricing issues surface? Options: Chief Actuary, Head of Pricing, Compliance / Legal, Communications / PR, Underwriting leadership, Other

Regulatory & Explainability: Can You Tell the Story?

• If a regulator asked you to justify a major rate change tomorrow, how confident are you that you could present a clear, data-backed narrative? Options: Very confident, Somewhat confident, Not confident, We haven’t been asked before
• Which explainability artifacts do you produce today alongside model outputs? Options: GLM tables and coefficient documentation, Variable importance / PDPs, Local explanations (SHAP/LIME), Complete model documentation & model cards, None
• Have you ever had a state DOI or regulator request model documentation that delayed approval? Options: Yes — caused a major delay, Yes — minor interactions/delays, No, Not sure
• What interpretability or documentation standards must a model meet for actuarial and compliance sign-off?
• How do you run tests for fairness or disparate impact before deploying a model (if at all)? Options: Regular automated fairness testing, Occasional manual checks, Not tested, Not applicable / unsure

People, Process, and Decision Rhythm: Who Moves the Needle?

• Where does the final pricing decision typically reside (actuarial sign-off, pricing committee, executive approval, delegated authority)? Options: Chief Actuary sign-off, Pricing committee, Executive sign-off (CUO/CFO), Delegated underwriter / product owner, IT/CIO approval, Other
• Who are the must‑involve stakeholders we need for a modeling engagement, and who historically slows approvals the most?
• What is your usual timeline and approval gates from modeling proposal to regulatory filing? Options: Highly variable, 1–2 months, 3–6 months, 6–12 months, 12+ months
• How are modeling projects prioritized against competing demands like product launches, regulatory deadlines, or reinsurance changes?
• If we partnered together, who would be the day-to-day contact and which roles would require executive-level updates? Options: Actuary, Head of Pricing, Data Scientist/Analyst, CTO/CIO, Program Manager, Compliance/Legal

Minimum Viable Change: What Low-Risk Steps Could Prove Value Quickly?

• What is one small, low-risk modeling change you believe could meaningfully improve outcomes within a single filing cycle?
• Which portfolio or segment would you prefer for a pilot (high volume, geographically concentrated, stable claims pattern, or high variance)? Options: Personal Auto - high volume, Personal Auto - telematics subset, Homeowners - targeted geography, Small commercial - specific class, Specialty line - niche segments, Other
• Which success metrics would you use to evaluate a pilot (select up to three)? Options: Increase in predictive lift, Improved loss ratio, Successful regulatory filing, No increase in complaints / cancellations, Faster time-to-deploy, Operational cost reduction
• What resources could you realistically commit to a 6–8 week pilot (people, data access, environments)?
• What would be the go/no-go milestone after a pilot (statistical validation, regulatory pre-approval, business sign-off, operational readiness)? Options: Statistical validation achieved, Regulatory pre-approval, Positive business sign-off (actuarial/pricing), Operational readiness and monitoring in place, Time-based decision point

Quick Check: Who's in the Room?

• Who are the decision-makers and approvers for taking a pricing model from staging to production? Options: Chief Actuary, Head of Pricing, CUO, CIO/CTO, Head of Underwriting, Legal/Compliance, Product Owner, Other
• Who will be our primary day‑to‑day contact for technical coordination and approvals?
• What is your target timeline for filing, deployment, or first market roll‑out? Options: ASAP (within 30 days), 1–3 months, 3–6 months, 6–12 months, No firm timeline
• What would “success” look like for this deployment in the first 90 days? Be specific (metrics, behaviors, decisions).
• What level of short‑term financial risk are you willing to accept during rollout? Options: Very conservative — avoid visible premium shifts, Moderate — limited segment repricing, Aggressive — full segmentation to optimize portfolio, Undecided / need to discuss

What Keeps You Up at Night About Going Live?

• If we flipped the switch tomorrow, what's the single most likely thing to derail deployment?
• How worried are you about adverse selection once the model hits market? Options: Extremely worried, Concerned but manageable, Minimal concern, Unsure
• Have you experienced customer or regulator pushback on pricing changes in the past 3 years? Tell us what happened.
• If the model caused a noticeable loss ratio deterioration, who in the org would drive the response? Options: Actuarial, Underwriting, Ops/Claims, Executive leadership, Legal/Compliance, Other
• Which stakeholder concerns feel under‑addressed right now (interpretability, speed, governance, commercial impact, other)? Options: Interpretability / explainability, Adverse selection mitigation, Integration complexity, Time to file, Operational capacity, Other

Where Does Your Data Really Live—and How Trustworthy Is It?

• How confident are you that the data feeding this model accurately reflects the risks you must price? Options: Very confident, Mostly confident with known gaps, Significant doubts, We haven’t audited it yet
• Which primary data sources will feed the model (select all that apply)? Options: Policy administration system, Claims system, Exposure / exposure units, Third‑party data (credit, MVR, demographics), Telematics / IoT, Underwriting rules / manual fields, Other
• How frequently are those source systems refreshed and available to production scoring? Options: Real‑time / streaming, Hourly, Daily, Weekly, Ad‑hoc
• Are there known ETL transforms, manual overrides, or business rules applied to those feeds we should be aware of? Describe them.
• Who owns the pipelines end‑to‑end (team or vendor)? Options: Internal Data Engineering, Actuarial / Analytics, IT/Platform, Managed services / vendor, Shared ownership, Not sure
• What recurring data quality issues have you observed recently (missing fields, drift, deduping, delayed records, other)?

Can Your Systems Actually Run This Model Without Breaking?

• If the model produced a new rate table today, how would your rating engine accept and apply it? Options: Direct API integration, Batch flat files / SFTP, Manual upload via UI, Requires vendor support, Unsure
• Do you have a test environment that mirrors production (data, logic, integrations) for full end‑to‑end validation? Options: Yes — full mirror, Partial mirror, Only unit/test staging, No test environment
• Can your system compute model features at quote time (online scoring) or do features need to be precomputed? Options: Online scoring available, Hybrid (some precompute), All precomputed / batch only, Not sure
• Who will own the technical deployment tasks (integration, scheduling, monitoring) and what are their contact details?
• What is the acceptable latency tolerance for scoring during quote flow or batch runs? Options: <100ms, <500ms, <2s, Minutes acceptable, No strict requirement defined

How Will Regulators & Audit Teams React?

• What’s the toughest question a regulator or auditor could ask about this model that we need to be ready to answer?
• Which jurisdictions require filings or additional disclosures for this product? Options: Single state — specify in comments, Multi‑state — regional, Nationwide, Commercial lines / different rules, We don’t have the list yet
• Do you have standard DOI‑ready documentation templates (model spec, variable descriptions, validation results, consumer impact analysis)? Options: Yes — fully formed, Partial templates exist, No, we need support
• Have regulators previously questioned algorithmic or ML components in filings? What was the outcome? Options: Yes — required changes, Yes — accepted after review, No prior questions, Not applicable / unsure
• Who owns regulatory filings and DOI communications within your organization? Options: Legal/Compliance, Pricing/Actuarial, Product, External consultant/vendor, Other

Who's Going to Monitor It — And What Will They Actually Do When It Strays?

• If the model's loss ratio worsens materially tomorrow, how quickly would someone detect and begin remediation? Options: Within 24 hours, 1–7 days, Weeks, We’d likely miss it for months, Not sure
• Which operational KPIs should trigger alerts (select all that apply)? Options: Loss ratio by decile/segment, Quote conversion rate, Hit ratio / shopping behavior, Average premium movement, Claim frequency changes, Complaint volume, Other
• What alert thresholds or statistical rules would reasonably trigger investigation (give examples or ranges)?
• Do you have runbooks or playbooks for model incidents (investigation steps, rollback, communication)? Options: Complete runbooks, High‑level playbook, No formal runbook
• Who is the escalation path for incidents (names/roles) and who approves emergency rollbacks?

If We Had to Pause or Roll Back, What Would That Look Like?

• What rollback strategy would cause the least operational pain: full stop, selective segment pause, or targeted scoring adjustments? Options: Full stop (remove model), Pause specific segments/products, Adjust thresholds / offset factors, Soft launch reductions only, Undecided
• Can you run phased / canary rollouts (percent traffic, by geography, by broker) and how granular can we get? Options: Yes — fine‑grained, Yes — coarse (by region/product), No phased rollout capability, Unsure
• Do you retain prior rate engine versions and can you revert reliably to a previous state? Options: Yes — versioned with rollback, Partial capability, No — not versioned
• How would you communicate a pause or rollback to internal stakeholders, regulators, distribution partners, and customers?
• Are there customer remediation or pricing correction policies already in place if a pricing error materially affects policyholders? Options: Yes — formal policy, Informal practice, No policy

Imagine We're 30 Days Post‑Launch — What's Different?

• If this launch is a clear success, what single outcome will your executive team highlight?
• Which secondary indicators would you track to prove the model is working (select up to 4)? Options: Improved loss ratio by segment, Higher quote‑to‑bind conversion, Reduced underwriting manual overrides, Improved hit ratios vs competition, Stable complaint/regulatory volumes, Improved retention in profitable segments
• How will you fold learnings from production back into pricing governance and model updates (cadence, owners, tooling)?
• What ongoing budget or headcount are you willing to commit for monitoring, retraining, and regulatory upkeep? Options: Dedicated ongoing budget and team, Ad hoc budget per need, Limited budget — expect vendor to cover, Undecided
• How would you prefer to log, triage, and track post‑launch issues and enhancement requests (tools and cadence)? Options: Shared ticketing system (Jira, ServiceNow), Real‑time chat (Slack/MS Teams) + board, Regular weekly review calls, Other

Opening — Quick Check-In Before We Dive In

• To start, which statement best describes where your model(s) currently sit in the lifecycle? Options: Data exploration / feature work, Model prototyping, Internal validation in progress, Ready for final validation/back-testing, Deployed in production (monitoring only), Other
• Who will be the day-to-day owner(s) for the final validation and the regulatory filing from your team? (select all that apply) Options: Chief Actuary, Head of Pricing, CUO / Head Underwriting, CIO / Head of Engineering, Compliance / Legal, Model Risk / Validation Lead, Other
• Briefly describe the model(s) we're validating: line of business, target (frequency/severity/pure premium), and model family (GLM, GBM, NN, scorecard, ensemble, etc.).
• What is your target timeline for deployment if validation meets acceptance? Options: Next 4 weeks, 1–3 months, 3–6 months, 6–12 months, No firm timeline / exploratory
• Which stakeholders must sign off before you can file or put rates into the engine? (select all that apply) Options: Actuarial leadership, Pricing committee, Underwriting leadership, C-suite (CUO/CFO), Compliance / Legal, IT/Engineering / Data Ops, Board / Rate committee, Other

Are We Sure the Model Won’t Do Something Dangerous?

• What concrete evidence would convince you this model will not produce adverse selection or unexpected deterioration once rates are live?
• Do you maintain a pre-defined holdout and back-test strategy that mirrors the exact rollout population (geography, channel, product)? Options: Yes — standard industry holdout, Yes — custom holdout mirroring rollout, Partial — some back-tests but gaps remain, No — holdout not defined
• Which validation metrics do you prioritize when deciding ‘go / no-go’? (select up to 3) Options: Expected vs actual loss ratio, Lift / segmentation improvement, Gini / AUC, Calibration by score band, KS statistic, Population Stability Index (PSI), Prediction bias by cohort
• Tell us about a past time a back-test missed a real-world issue—what did you learn and what changed in your process?
• Do you hold out time-based out-of-time datasets (e.g., last 12–24 months) separate from random validation? Options: Yes — time-based OOT sets in place, Partial — some time splits used, No

Where the Numbers Clash with Real Agent / Customer Behavior

• Which model assumptions do you suspect are most fragile when exposed to live market behavior (pricing elasticity, selection effects, geographic mix, etc.)?
• Have you stress‑tested the model against plausible shifts—mix changes, underwriting rule changes, or macro shocks? Options: Yes — extensive scenario testing, Yes — limited scenarios, No — we haven't stress-tested
• Which scenario types have you simulated in stress tests? (select all that apply) Options: Geographic concentration shifts, Underwriting selection changes, Rate compression or market competition, Macro economic shock (e.g., unemployment), Catastrophe frequency/severity spike, Feature drift / data missingness
• Describe one test scenario where model performance materially degraded—what happened and how did you adjust?
• Do you have pre-defined quantitative thresholds that define ‘material degradation’ for rollout or rollback? Options: Yes — numeric thresholds (please attach), Yes — qualitative thresholds, No — thresholds not defined

Can We Explain This Clearly to Regulators and Stakeholders?

• If a state DOI asked you to justify a 10% increase in a named segment, what’s the simplest non-technical explanation you would give today?
• Do you have documentation packages tailored for regulators (non-technical summary, data lineage, validation report, fairness analysis)? Options: Complete and regulator-ready, Draft documentation exists, Only technical internal docs, No documentation yet
• Which model explainability techniques do you rely on for external communications? (select all that apply) Options: SHAP / feature attribution, Partial dependence plots, Surrogate (interpretable) models, Traditional GLM factor tables, Rule extraction / decision boundaries, Other
• Who on your team will lead DOI/Regulatory Q&A and what would they need from us to feel ready?
• Have you rehearsed regulator-style Q&A or an independent peer review of the validation materials? Options: Yes — rehearsed with external reviewers, Yes — internal rehearsal, No — not rehearsed

Who Bears the Risk If Validation Is Weak?

• Which internal stakeholder is most likely to block deployment if validation is insufficient — and why would they push back? Options: Chief Actuary, Head of Pricing, CUO / Chief Underwriting Officer, CIO / Head of Engineering, Compliance / Legal, Commercial / Distribution Leadership, Other
• How concerned are your distribution and retention teams about selection effects or customer churn from new rates? Options: Very concerned, Somewhat concerned, Neutral, Optimistic about impact
• Are there contractual, regulatory, or market constraints that could prevent or delay filing/deployment even if validation is successful? Options: Yes — explicit constraints (describe below), Possibly — depends on DOI / contract, No known constraints
• Which governance controls are mandatory for your sign-off? (select all that apply) Options: Independent validation report, Governance committee approval, Monitoring & SLA plan, Rollback & contingency plan, Regulatory filing package ready, Third‑party audit
• What contingency or mitigation plans do you have prepared if monitoring shows adverse selection in early life?

Are the Data Pipes and Tests Truly Production-Grade?

• How confident are you that production pipelines will produce the exact features and data quality the model expects every day? Options: Very confident, Somewhat confident, Doubting — gaps exist, Not confident
• What level of automated testing exists for feature calculations and data schema checks? Options: Full unit & integration tests, Partial automated tests, Manual checks primarily, No tests
• Which operational controls do you have for data quality and drift detection? (select all that apply) Options: Automated drift alerts, Schema validation & versioning, Backfill / repair procedures, Manual QA gates, Data catalog with lineage, None
• When was the last time a pipeline change caused a downstream model failure? Describe what happened and the recovery timeline.
• Who will own operational monitoring and incident response post-deployment? (select all that apply) Options: Data Ops / Engineering, Actuarial team, Pricing team, Third-party vendor, Model Risk / Validation team

How Will We Know It’s Working — And When to Stop?

• What single, concrete signal in the first 90 days would make you stop the rollout immediately?
• Which KPIs must be continuously monitored during the pilot and first 6 months? (select all that apply) Options: Loss ratio by segment, Claim frequency by cohort, Take / hit rates / bind ratios, Score distribution shifts, Selection indicators (adverse movement), Profitability by cell, Customer complaints / NPS
• How often would you like monitoring reports and who should receive them? Options: Daily automated alerts + weekly report, Weekly summary, Monthly executive digest, Event-driven only
• Describe the acceptance tests you expect during a pilot rollout (minimum sample size, key metrics, duration).
• Which rollback mechanisms are acceptable to you if early signals trigger a pause? (select all that apply) Options: Full rollback to prior rates, Selective rollback by cohort, Throttle new business / quota limits, Manual underwriting overrides, Immediate regulatory pause

Commercial Reality Check — If Validation Clears, What Then?

• If validation proves the promised lift and risk controls, what minimum commercial commitment would you need to move from pilot to full deployment?
• What contractual or governance terms are non-negotiable for you before signing for deployment services? (select all that apply) Options: Acceptance criteria tied to metrics, Milestones & payment schedule, Liability / indemnity clauses, Data access and IP rights, Support & maintenance SLAs, Regulatory assistance clause
• What is the current budget posture for model deployments this year? Options: Budget approved & allocated, Budget allocated pending approval, Contingent on pilot results, No budget allocated
• Who must be in the final mutual-commit meeting from your side (names/titles and decision authority)?
• What would you like the next concrete step to be after completing this checklist? Options: Schedule validation kickoff, Share current validation artifacts, Commission independent validation, Define pilot parameters, Other

Final Readiness Snapshot — Quick Checklist

• Which of the following are already complete on your side? (select all that apply) Options: Holdout and back-test plan, Regulator-facing documentation, Operational data pipelines with tests, Monitoring & alerting in place, Governance sign-off criteria defined, Rollback & contingency plan
• Are there any unresolved issues, internal politics, or external constraints we should know about that could affect validation or filing?
• Is there any supporting artifact you'd be willing to share now to speed validation (validation reports, sample data schemas, model card)? Options: Yes — validation reports, Yes — sample data schema, Yes — model card / documentation, No — not ready to share
• Anything else you want us to know that would change how we approach the final validation and filing?