Why you're starting this evaluation

• Tell me the main decision or deadline that prompted you to seek an external evaluation now.
• Who on your team will coordinate with external evaluators day to day? Options: Program director, Monitoring & evaluation lead, Data manager, Grants manager, Other
• Which program, cohort, or grant cycle should be included in scope?
• How many staff or sites are directly involved in delivering the program? Options: 1-5, 6-20, 21-50, 51-200, 200+
• Describe the primary audiences who must find the final report credible and useful.
• Estimate the funding or renewal amount tied to this evaluation decision. Options: Under $250k, $250k–$1M, $1M–$5M, Over $5M, Unsure
• If the evaluation concludes before your next board meeting, would that change how you would use the findings? Options: Yes, we would fast-track decisions, No, timing is not critical, Maybe, depends on results, Unsure

Where the program stands today

• Imagine a skeptical board member asked you to prove the program works today, what pieces of evidence would you show first?
• List the routine metrics you already collect on participation and outcomes. Options: Enrollment, Attendance, Completion, Short-term outcomes, Long-term outcomes, Financial metrics, Other
• Who currently owns the data and where is it stored, in a database, spreadsheets, or elsewhere? Options: Central database with API, Multiple databases, Spreadsheets, Paper records, External partner holds data
• On a scale from 1 to 5, how confident are you in the quality of those outcome measures? Options: 1 - Not confident, 2, 3, 4, 5 - Very confident
• Give a brief example of a recent report or dashboard you'd share with funders.
• Which single data gap would prevent an external evaluator from producing credible results?

What's getting in the way of clear answers

• Walk me through the last time a data or approval issue delayed a decision, what broke and what was the consequence?
• In your experience, which approvals or gatekeepers most commonly set the timeline back? Options: Legal counsel, Data owners, IRB or ethics board, Funders, Site leadership, Other
• To whom do requests for sensitive participant data typically go first?
• Do past vendor contracts or confidentiality clauses limit what external teams can access? Options: Yes, significant limits, Some limits but negotiable, No, access is generally granted, Unsure
• Give an example of a political or donor sensitivity that shaped what you could publish previously.
• Name the single approval or constraint that, if not satisfied, would stop this evaluation from starting.

Who the evaluation must speak to

• If equity were the primary test of success, where do you expect your program to struggle most?
• List the participant subgroups and geographic contexts where outcomes may differ.
• In stakeholder conversations, what language is used to describe service gaps or unfairness? Options: Access, Quality, Cultural fit, Cost barriers, Other
• Name the community partners or oversight roles whose buy-in will make the findings credible.
• What kinds of harms or unintended effects are you most worried an evaluation could uncover?
• Should the evaluation indicate widening disparities, who has the authority to change program policy immediately? Options: Executive director, Program board, Funders, Government partner, Other

How we'll design a study that people will trust

• Imagine an external reviewer will judge the study design, which design decision would most affect their verdict?
• Identify the three outcomes your leadership considers non-negotiable for drawing conclusions.
• Estimate the minimum sample size or number of sites you believe will convince funders. Options: Case study only, 5–10 sites, 11–30 sites, 30+ sites, Unsure
• Do your stakeholders prefer randomized, quasi-experimental, or pre-post designs, or are they flexible? Options: Prefer randomized, Prefer quasi-experimental, Prefer pre-post, Flexible, Unsure
• Select the deliverable format your board prefers, a 2-page brief, an executive memo, a full technical report, or a data appendix. Options: 2-page brief, Executive memo, Full technical report, Data appendix, Combination
• Tell us the deadline or milestone that would force you to stop the study if we could not meet it.

The other options you're weighing

• Suppose your current provider gave you the same findings at no extra cost, what would still be missing?
• Provide the external vendors, internal teams, or academic partners you have already spoken with.
• What would have to be true about your current approach for you to keep it rather than hire an outside evaluator?
• Has anyone on your team proposed doing this evaluation internally instead of contracting an external firm? Options: Yes, ready to do internally, Yes, but lacks capacity, No, not proposed, Unsure
• Rank the alternatives you are considering by likelihood you'll pick them, with 1 being most likely. Options: External vendor A, External vendor B, Internal team, Academic partner, Other
• Explain the one advantage the incumbent must demonstrably have for you to stay with them.

Practical readiness and gating issues

• Identify the single infrastructure or data access gap that would prevent the evaluation from proceeding on schedule.
• Does your data live in a central system with APIs, in multiple databases, or primarily in spreadsheets? Options: Central system with APIs, Multiple databases, Primarily spreadsheets, Paper records, External partner stores data
• Point to the role that will provide credentials and technical contacts for system access. Options: IT director, Data manager, Program manager, Vendor contact, Other
• Provide your best estimate of the percentage of records that are incomplete or inconsistent today. Options: 0–5%, 6–15%, 16–30%, 31–50%, 50%+
• Are there IRB, ethics, or legal reviews required, and if so, what is the expected approval timeline? Options: No approvals required, IRB pending (weeks), Legal review pending (weeks), Already approved
• Please indicate whether your team has capacity to support interviews, recruitment, and data pulls, or whether you expect the evaluator to provide that capacity. Options: We have full capacity, We have partial capacity, We need evaluator to provide capacity, Unsure
• Should key data access not be available within 6 weeks, will you postpone, rescope, or cancel the project? Options: Postpone, Rescope, Cancel, Negotiate alternate access

How you'll know this evaluation succeeded and what comes next

• Explain the single finding that would cause leadership to change funding or scale the program immediately.
• To whom should final deliverables be addressed and who will lead the acceptance review?
• On a scale from 'sign now' to 'request more analysis', where would an ideal final brief land for your board? Options: Sign now, Mostly satisfied, minor edits, Request more analysis, Ask for additional evidence
• Within how many weeks would you want the evaluator to provide a draft for internal review after fieldwork ends? Options: 1–2 weeks, 3–4 weeks, 5–8 weeks, More than 8 weeks
• Select from the following follow-up supports that would increase adoption, and add any others. Options: Implementation plan, Staff training, Interactive dashboard, Ongoing monitoring plan, Short executive brief
• Are you prepared to commit funding to a scale-up within the next quarter if the pilot meets the agreed success criteria? Options: Yes, Maybe, pending board vote, No, Unsure

• Administer standardized outcome surveys to participants
• Collect and clean administrative and program data
• Link participant records across datasets
• Conduct in-depth participant interviews
• Facilitate stakeholder focus groups
• Implement randomized assignment and trial procedures
• Perform quasi-experimental matching and comparative analysis
• Run statistical outcome modeling and subgroup tests
• Disaggregate outcomes by race, income, and geography
• Produce executive findings brief with specific recommendations
• Develop accessible data visualizations and dashboards
• Manage IRB submission and human-subjects compliance

Administer standardized outcome surveys to participants

• Do you plan to administer surveys in person, by phone, by web, or a mix of modes? Options: In person, Phone, Web/online, Mixed modes
• Which standardized instrument(s) do you intend to use for primary outcomes (for example PHQ-9 for depression, PROMIS for health, validated employment outcomes)? Options: PHQ-9 (depression), PROMIS (health/function), Validated employment measure (hours/employment status), Use a custom outcome instrument, Not yet chosen
• How many survey waves do you require (for example baseline, endline at 12 months, 6-month follow-up)? Options: One (cross-sectional), Baseline + Endline, Baseline + Endline + 1 follow-up, Other cadence
• What target sample size do you expect per wave for the survey (approximate counts are fine)? Options: Less than 200, 200-499, 500-1,000, More than 1,000
• Which language(s) must the instrument and consent materials be available in (list specific languages used by participants)? Options: English, Spanish, Other languages
• Who will provide the participant contact list and existing consent status for outreach to complete the surveys? Options: You will provide a fully consented contact list, You will provide a list and we assist with consent collection, We will recruit and collect consent directly, Mixed approach
• Describe the expected nonresponse or attrition risk between waves and any strategies you require (for example phone reminders, incentives).
• What incentive strategy is acceptable to encourage survey completion (for example $25 gift card, raffle, no incentive)? Options: Fixed incentive per respondent, Raffle/prize, No incentive, Other
• Which data security measures must be applied during survey collection (for example SSL/TLS for web surveys, encrypted recorder for phone interviews)? Options: SSL/TLS for web, Encrypted audio recordings, Secure upload portal for files, Other
• How should we handle consent documentation for minors or legally protected populations if included? Options: Parent/guardian consent required, Assent plus guardian notification, Not applicable

Collect and clean administrative and program data

• What administrative sources will be available for extraction (for example case management database, LMS export, payroll files, enrollment rosters)?
• Which file formats will you provide for administrative exports (for example CSV, Excel, SQL dump)? Options: CSV, Excel (.xlsx), Database export (SQL), Other
• Who will be the technical contact with access to the source systems and data dictionary for each administrative source?
• What minimum data quality thresholds should trigger remediation prior to analysis (for example >=95% completeness on primary outcome fields, <2% duplicate IDs)?
• What acceptance criteria will confirm the cleaned analytic dataset is ready for linkage and analysis (give measurable checks such as variable-level dictionary, completeness rate, resolved mismatches)?
• Are there regulatory or contractual restrictions on sharing raw administrative files (for example data-use agreements, state restrictions)? Options: Yes — restrictions apply, No known restrictions, Not sure — need to confirm
• How do you prefer we document cleaning steps and changes to the source fields (for example annotated data dictionary, cleaning script with version control)? Options: Annotated data dictionary, Cleaning scripts with comments (R/Python), Both, Other
• Which timeline do you require for initial data delivery and the cleaned deliverable (for example raw files within 2 weeks, cleaned analytic file within 6 weeks)?
• What out-of-scope activities should be noted for the engagement (for example ongoing operational support, full ETL pipeline maintenance, implementation of recommendations)?
• Are there agreed performance SLAs for turnaround on urgent data requests during analysis (for example 48-hour turnaround on ad-hoc extracts)? Options: 48 hours, 72 hours, One week, No SLA required

Link participant records across datasets

• Which unique identifiers are present across datasets to support deterministic linkage (for example participant ID, email, phone, SSN last four)?
• What privacy constraints apply to identifiers (for example SSN cannot be transferred offsite, hashed IDs only)? Options: No transfer of direct identifiers, Hashed/encrypted identifiers allowed, Full identifiers can be shared under DUA, Other
• Which fuzzy-match fields would you allow for probabilistic linkage if exact IDs are missing (for example first name, last name, date of birth, ZIP code)?
• How will you validate linkage quality (for example clerical review sample, match rate thresholds, false match checks)? Options: Clerical review sample, Minimum match-rate threshold, Manual review on mismatches, Other
• Who will own the master linkage key and how should it be named and versioned in the analytic files?
• How should we handle duplicate or conflicting records across sources (for example preference order, most recent record wins)? Options: Source priority list provided by you, Most recent record wins, Combine fields where possible, Other rule — specify
• Which governance approvals are required before attempting cross-system linkage (for example signed data-use agreement, IT firewall exception)? Options: Signed data-use agreement, Site-level IT approval, No additional approvals, Other
• How should provenance of linked records be tracked in deliverables (for example source system column, match confidence score)? Options: Source system column, Match confidence score, Both

Conduct in-depth participant interviews

• Which participant subgroups should be prioritized for in-depth interviews (for example non-responders, high-benefit, marginalized groups)?
• What is the target number of interviews and the expected duration for each interview (for example 30–60 minutes)? Options: 10-20 interviews, 21-40 interviews, 41-80 interviews, 80+
• Which interview language(s) and interpreter arrangements are required for interviews? Options: English, Spanish, Other — please list
• How should interview participants be recruited and consented (for example phone outreach by your staff, we recruit using provided contacts, consent at time of interview)? Options: You recruit and confirm consent, We recruit and obtain consent, Mixed approach
• What recording and transcription standards do you require (for example verbatim transcript, time-stamped, translated transcripts)? Options: Verbatim transcript, Time-stamped transcript, Translated transcript, Summary notes only
• Are there sensitive topics requiring special protocols for trauma-informed interviewing or referral resources? Options: Yes — trauma-informed protocols required, No special protocols, Unsure — need guidance
• Which topic guides or domains must interviews cover based on your logic model or theory of change (for example barriers to participation, perceived program benefits)?
• Do you require verbatim quotations linked to participant demographic metadata for reporting (for example quote + age bracket + program dosage)? Options: Yes — include linked metadata, No — quotes de-identified only

Facilitate stakeholder focus groups

• Which stakeholder types should be convened for focus groups (for example frontline staff, program participants, partner agencies)? Options: Frontline staff, Program participants, Partner agencies, Funders/board members, Other
• How many focus groups do you anticipate and what is the ideal group size per session? Options: 1-2 groups (6-8 people), 3-4 groups (6-8 people), 5+ groups
• What facilitation style and deliverables do you prefer (for example recorded + transcript, facilitator summary with verbatim quotes, live synthesis session)? Options: Transcript + recordings, Facilitator summary with quotes, Live synthesis session, Other
• Which accessibility accommodations are required for participants (for example childcare stipend, remote access, ASL interpreters)?
• Who will coordinate participant recruitment logistics such as incentives, scheduling, and venue or virtual platform access? Options: You coordinate recruitment and logistics, We coordinate with your point of contact, We handle end-to-end recruitment
• Are there confidentiality or power-dynamics concerns we should plan for when mixing stakeholder types in the same group? Options: Yes — concerns exist, No concerns, Not sure — need consultation
• Which outputs do you expect from each focus group session (for example facilitator synthesis memo, thematic codebook, verbatim excerpts)? Options: Facilitator synthesis memo, Thematic codebook, Verbatim excerpts, All of the above

Implement randomized assignment and trial procedures

• What is the planned unit of randomization (for example individual participant, site/center, cohort)? Options: Individual, Site/center, Cohort, Other
• Which stratification or blocking factors should the randomization include (for example baseline risk score, region, ZIP code)?
• How should treatment assignment be documented and auditable (for example randomization script with seed, treatment assignment log, timestamped exports)? Options: Randomization script + seed, Assignment log with timestamps, Both
• What minimum balance checks do you require post-randomization (for example p>0.10 for covariate differences, standardized mean differences <0.1)?
• Which monitoring or integrity safeguards should be in place to prevent contamination or treatment crossover (for example fidelity checks, site training, monitoring visits)?
• Who will be the authorized signatory for protocol deviations and emergency unblinding if needed?
• What timeline milestones are tied to randomization (for example baseline window closes then randomize within 7 days)?

Perform quasi-experimental matching and comparative analysis

• Which comparison group sources are available for matching (for example waitlist applicants, administrative records from similar sites, external cohort)?
• What baseline covariates should be prioritized for matching (for example prior outcomes, age, baseline score, ZIP code-level poverty rate)?
• Which matching method do you prefer or allow (for example propensity score matching, coarsened exact matching, nearest neighbor)? Options: Propensity score matching, Coarsened exact matching, Nearest neighbor matching, Other/No preference
• How should imbalance after matching be evaluated and reported (for example standardized differences table, visual diagnostics)? Options: Standardized differences table, Visual diagnostics (Love plot), Both
• Are there exclusion criteria for comparison units you want enforced before matching (for example missing baseline outcomes, prior participation in pilot)? Options: Yes — list exclusions, No exclusions, Undecided
• What level of documentation do you require for matching code and reproducibility (for example commented scripts, reproducible notebook, versioned code repository)? Options: Commented scripts, Reproducible notebook, Versioned repository, Other
• Are linked covariate data available at the individual or site level to support matching (for example individual prior outcomes, site-level baseline rates)? Options: Individual-level covariates available, Only site-level covariates, Mixed availability, Not available

Run statistical outcome modeling and subgroup tests

• Which primary outcome measures and time windows should models target (for example employment status at 6 months, test score change at endline)?
• What modeling approaches are acceptable (for example linear mixed effects, logistic regression, survival analysis)? Options: Linear mixed effects, Logistic regression, Survival analysis / Cox, Other/No preference
• What statistical significance threshold or confidence level should we use for hypothesis tests (for example alpha = 0.05, 95% confidence intervals)? Options: Alpha = 0.05 (95% CI), Alpha = 0.01 (99% CI), Report both 95% and 99%

• Non-Disclosure Agreement (NDA)
• Master Services Agreement (MSA)
• Statement of Work (SOW)
• Fee Schedule & Payment Terms
• Data Sharing & Processing Agreement (DSA/DPA)
• IRB / Ethics Approval Confirmation
• Regulatory Compliance Addendum (conditional)

• Findings Acceptance Check (weeks 1-4)
• Roadmap Prioritization Workshop (weeks 4-10)
• Implementation Readiness Check (around day 60-90)
• Quarterly Follow-up and Uptake Review (ongoing quarterly)

• Client-side owners to update their initiative status in the shared tracker at least 5 business days before the next quarterly review.
• Compile and circulate a single tracker of identified implementation blockers with proposed resolution steps and deadlines.
• Client to confirm owners and update any timeline constraints within 7 business days.
• Status report on each prioritized initiative
• Confirm the percentage of implementation blockers resolved and decide whether readiness targets recorded in Engagement Scope are met.
• Agree remediation timelines for any remaining blockers that prevent initiative start or scale-up.
• Update the roadmap tracker with blocker resolution status and revised start/completion dates where applicable.
• Publish a short decision memo stating whether readiness targets recorded in Engagement Scope are met and listing conditional remediation items if applicable.
• Client to confirm internal resource commitments and finalize any required internal approvals within the agreed remediation timelines.
• Quarterly metric review
• Maintain or improve the implementation blockers resolved rate quarter over quarter and keep the prioritized action count on schedule.
• Ensure each initiative retains a named client-side owner who provides a status update before the next quarterly review.
• Circulate the quarterly status dashboard showing prioritized initiatives, blockers resolved, and the findings acceptance rate trend.
• Create a short issue tracker for any newly surfaced blockers with proposed resolution steps and target dates.
• Recap final deliverables and acceptance criteria
• Achieve a documented findings acceptance rate and record which report sections are accepted or disputed.
• Produce a discrete list of unresolved evidence gaps with remediation actions and deadlines.
• Distribute an annotated copy of the final deliverable marking accepted sections and disputed items for formal sign-off.
• Provide a dated list of remediation actions and the evidence required to raise the findings acceptance rate to target levels recorded in Engagement Scope.
• Client to submit formal acceptance status (accepted / conditionally accepted / not accepted) and any final comments within 5 business days.
• Review recommended initiatives and priority criteria
• Produce a ranked roadmap listing the prioritized action count with a named client-side owner assigned to each initiative.
• Capture the initial set of implementation blockers and a plan to resolve the highest-risk items before the readiness check.
• Publish the prioritized roadmap with assigned owners, tentative dates, and the scoring rationale to the shared workspace.
• Review acceptance feedback and audit findings acceptance rate
• Review implementation blockers resolved vs open
• Review status updates from client-side owners
• Score and rank initiatives
• Surface and triage new blockers
• Assign client-side owners and tentative timelines
• List unresolved evidence gaps or clarifying questions
• Root-cause diagnosis for stalled initiatives
• Adjust follow-up frequency or format if needed
• Decision on readiness to proceed
• Identify known implementation blockers
• Agree next steps to resolve open items
• Document acceptance status outcome
• Agree handoff and documentation for implementation