Technology Enterprise Software & IT Data Platforms & Analytics

Data Lakehouse Platforms

Platform decisions with deep integration complexity, organizational change, and long-term data stakes.

Example organizations in this space: Databricks Snowflake Microsoft Fabric Dremio

This interactive experience is the shipped product itself — the same application code customers run in production, mounted read-only in your browser over a real sample journey. Not a video, not a mockup: because the demo and the product are one codebase, it can never drift from the real thing.

Inside this journey
  1. Outcome Discovery

    Align stakeholders, baseline cloud spend and recurring workloads, and define measurable success criteria, timelines, and decision roles.

    Discovery Questions

    Opening the conversation: scale, spend, and near-term targets

    • How many terabytes or petabytes of analytics data does your team actively store and query today? Options: <10 TB, 10–100 TB, 100 TB–1 PB, 1–5 PB, >5 PB, Unsure
    • When you look at monthly analytics usage, how many distinct teams run recurring queries, feature engineering, or operational reporting? Options: 1 team, 2–5 teams, 6–15 teams, 16–50 teams, More than 50 teams
    • Describe your current annual spend on cloud data storage plus the separate data warehouse compute bill for analytics and machine learning
    • Estimate the year over year percentage increase in your combined storage and compute spend over the last 12 months Options: Declined, 0–10%, 11–25%, 26–50%, >50%, Unsure
    • If finance required a 25 percent reduction in the combined analytics infrastructure bill within 12 months, what would prevent you from committing to a scoped pilot?

    Where duplicate copies are adding cost and creating governance gaps

    • Which datasets are copied most frequently across teams and environments, and why do those copies exist?
    • List the typical number of copies for a high-value table such as order history, customer master, or metrics table Options: Single canonical copy, 2 copies, 3–5 copies, 6–10 copies, More than 10 copies, Unsure
    • Who owns the source-of-truth version of those datasets and who routinely grants access to copies? Options: Data engineering, Data platform team, Data science, Analytics/business intelligence, Product, Other
    • Describe the manual handoffs, ETL jobs, or tooling that create and maintain those copies
    • Would keeping the current duplication pattern be acceptable to your chief data officer if it meant avoiding near-term migration costs, and why or why not? Options: Yes, acceptable, No, not acceptable, Only temporarily, Conditional on further savings

    How you will validate performance, ML, and cost claims

    • Is matching your incumbent's performance on the most expensive recurring query enough, or do you require additional ML and ingestion benchmarks to consider switching? Options: Query performance only, Query plus ingestion, Query plus ML training, All three: query, ingestion, ML
    • Name the top three query patterns, model training workflows, or ingestion rates that define acceptance for your team
    • Provide the current SLA targets for latency, throughput, and concurrency for those workloads
    • Identify one sample dataset and one representative query or model training job we should use for benchmarks to be convincing
    • When would a successful benchmark move budget and executive approval to a pilot within your organization? Options: Immediately, Within 30 days, Within 60–90 days, Longer than 90 days, Not sure
    • Rate how much lower total cost versus your baseline would need to be for procurement to sign within the quarter, from 1 meaning 10 percent lower to 5 meaning 40 percent lower Options: 1 - 10%, 2 - 15%, 3 - 20%, 4 - 30%, 5 - 40%+

    Governance and compliance: controls you cannot compromise

    • Who in your organization is accountable for data governance, and would they approve unified fine-grained access controls across previously separate systems? Options: CDO, VP data engineering, Security/compliance, Legal, Shared accountability, Other
    • Which regulatory or compliance regimes, for example data residency, PCI, HIPAA, GDPR, constrain where analytics data can reside or be processed? Options: GDPR, HIPAA, PCI, SOC2, Data residency/sovereignty, Other, None of the above
    • Is current identity and access management federated in a way that lets you apply the same permissions across storage and query engines? Options: Yes, fully federated, Partially federated, No, separate systems, Planning to federate
    • Tell us about any audit, lineage, or certification reports you must produce that the platform must support
    • Would unresolved governance or residency constraints be a deal breaker for moving the workload in the next 90 days? Options: Yes, deal breaker, No, manageable, Depends on remediation plan

    Operational readiness: people, environments, and integration points

    • Assuming we plan a 6 to 8 week technical proof, do you have a named owner, allocated engineering time, and a sandbox environment ready to support data sample extraction and benchmarks? Options: All three ready, Partial readiness, No, not ready
    • List the external systems, APIs, or connectors that must integrate for ingestion and query, and indicate who owns each
    • Do you have service accounts and permissions that allow the seller to access representative data samples without exposing sensitive PII? Options: Yes, scoped access ready, Needs redaction or masking, No, requires legal review
    • How much dedicated engineering FTE can you commit weekly during the pilot phase? Options: <4 hours/week, 4–10 hours/week, 10–20 hours/week, 20+ hours/week
    • Identify any legal, procurement, or vendor approval steps that must be completed before a pilot starts and the typical time each takes

    Obstacles that have stopped projects in the past

    • Why have previous consolidation or migration initiatives stalled or failed in your organization?
    • What single technical or organizational risk would force you to halt a migration mid-execution?
    • Name the internal stakeholders or committees that require sign-off at each governance checkpoint
    • Rate your confidence that current monitoring, rollback, and incident playbooks are sufficient to handle a production migration, from 1 low to 5 high Options: 1, 2, 3, 4, 5
    • Tell us the one escalation path and expected response time that would be used if the initial workload regression exceeds acceptable thresholds

    Alternatives on the table and what would keep you with them

    • Provide the alternatives you are actively evaluating, including your incumbent data warehouse, internal rework, or other vendors Options: Incumbent cloud data warehouse, Internal re-architecture, Other vendor platforms, Proof of concept with another vendor, No alternative
    • If you were to stay on your current stack, what metric projection would have to be true about its cost or performance for leadership to require no change?
    • Have internal teams proposed solving this duplication without an external vendor, and if so which teams and what timeline did they propose? Options: Yes, data engineering, Yes, platform team, No internal proposal, Other
    • What is the minimum financial or performance improvement that would justify moving away from the incumbent today? Options: 10% total cost reduction, 20% total cost reduction, 25% with governance benefits, 30%+, Performance parity required
    • Assuming a pilot proves a 30 percent TCO reduction within 6 months, who would have final approval to convert that pilot into a signed annual consumption contract? Options: CFO, VP data engineering, CDO, Procurement, Multiple signatories

    Defining success, timeline, and the next binding decision

    • Estimate the earliest timeline, in calendar weeks, in which a validated pilot could deliver measurable cost savings against your baseline Options: 2–4 weeks, 4–8 weeks, 8–12 weeks, 12+ weeks, Unsure
    • Do you have a procurement or contracting cadence that prefers multi-year consumption models, or do you require quarter-by-quarter commitments? Options: Prefer multi-year, Prefer quarter-by-quarter, Flexible, Unsure
    • Outline the three acceptance gates you expect the pilot to meet for performance, governance, and cost
    • Prepare a short list of named owners who will approve moving from pilot to full migration and the approximate budget owner for each
    • Finally, what single decision or piece of evidence would make your CFO sign off within 30 days after a successful pilot?
  2. Technical Proof & Benchmarks

    Run representative query, storage, and native-ML benchmarks on buyer data samples to validate performance, open-table format compatibility, governance, and acceptance criteria.

    • desired_state
    • current_state
    • stakeholders
    • gaps
    • success_criteria
    • decision_readiness
    • desired_state
    • decision_readiness
    • current_state
    • success_criteria
    • gaps
    • stakeholders
    • desired_state
    • success_criteria
    • stakeholders
    • gaps
    • current_state
    • decision_readiness
    • decision_readiness
    • decision_readiness
    • decision_readiness
    • decision_readiness
  3. Solution Scope

    Define the migration boundary for the initial workload, required modules, responsibilities, performance targets, access controls, and three-year TCO inputs.

    Scope Configuration

    • Migrate high-value analytical workload to platform
    • Consolidate data into open-table format storage
    • Deploy streaming ingestion pipelines into platform
    • Deploy and tune SQL query engine for workload
    • Provision native ML runtime and training pipelines
    • Deploy feature store and feature pipelines
    • Configure unified fine-grained access controls
    • Consolidate historical storage and remove duplicate copies
    • Migrate recurring ETL/ELT pipelines to platform
    • Perform cutover and redirect production queries
    • Configure consumption metering and cost reporting
    • Export open-format datasets for portability

    Scope Questions

    Migrate high-value analytical workload to platform

    • In your environment, which recurring query pattern or scheduled report consumes the most monthly compute credits? Provide the query ID or report name.
    • How many distinct users in your organization execute that query or report each week? Options: 1-10, 11-50, 51-200, 200+
    • For that workload, which source tables and average daily scan gigabytes does it read (provide table name and GB/day)?
    • Do you require transactional consistency (for example snapshot isolation) during cutover for this workload? Options: Yes, No
    • Provide the baseline P95 query latency and aggregate CPU-seconds for the representative query on your current data warehouse.
    • Define the acceptance criteria you want us to use to validate the migrated workload (for example: P95 latency <= baseline, identical row counts by partition, and <1% model/feature drift).

    Consolidate data into open-table format storage

    • Specify the storage locations in your estate (bucket path or container prefix) that hold datasets you plan to convert to open-table format; list up to five prefixes.
    • Enumerate the current file formats in those locations (for example Parquet, Avro, ORC, CSV). Options: Parquet, Avro, ORC, CSV, Other
    • Estimate the total storage size in terabytes across the datasets you plan to convert. Options: <1 TB, 1-10 TB, 10-100 TB, 100+ TB
    • Indicate whether you need to preserve your existing partitioning scheme (for example partition by ingestion_date) during conversion. Options: Yes, No, Partial
    • Name the governance tags or metadata fields that must be copied into converted tables (for example PII_flag, data_owner, sensitivity_level).
    • Assign who in your organization will own verification of converted tables and provide the verification report (role or team).

    Deploy streaming ingestion pipelines into platform

    • Report which streaming sources and protocol you plan to ingest from (for example Kafka topic name, Kinesis stream, or HTTP ingestion endpoint).
    • Quantify the peak events per second for each streaming source. Options: <100 eps, 100-1k eps, 1k-10k eps, 10k+ eps
    • Choose the delivery semantics you require for each stream in your use case: at-least-once or exactly-once. Options: At-least-once, Exactly-once
    • Describe the ingest-time transformations you require (enrichment, schema evolution, deduplication) and the processors or DAGs currently implementing them.
    • Target the destination table names and partitioning strategy the stream should write to (provide table name and partition key).
    • Set the monitoring alerts and SLO thresholds for ingestion lag or failure that matter to you (for example lag > 5 minutes, error rate > 0.1%). Options: Lag > 1 min, Lag > 5 min, Error rate > 0.1%, Custom

    Deploy and tune SQL query engine for workload

    • State which compute family or instance class and vCPU count your representative queries currently run on.
    • Set the target SLAs for query latency and concurrency we should tune to (for example P95 < 500ms, sustained concurrency 50 users).
    • Choose the sizing approach you prefer for partitions and compute: single shared cluster, per-query autoscale, or per-team pools. Options: Single shared cluster, Per-query autoscale, Per-team pools, Other
    • Describe any user-defined functions or external libraries your queries depend on and provide package names or install requirements.
    • Supply the sample SQL queries (SQL IDs or SQL text) that must be included in the tuning and benchmark run.
    • State the performance targets that will signify acceptance of the tuned SQL engine against your baseline (for example: <=10% slower than baseline on the provided SQL set and sustained concurrency of N).

    Provision native ML runtime and training pipelines

    • Identify the model training jobs (job names or DAG IDs) you plan to run natively on the platform during year one.
    • Quantify the training dataset sizes in gigabytes and typical epoch counts for each job. Options: <1 GB, 1-10 GB, 10-100 GB, 100+ GB
    • Indicate whether your training pipelines require GPU acceleration or can use CPU-only runtimes. Options: No GPU, GPU required, Unsure
    • Explain how you currently version models and store trained artifacts (artifact registry path, model IDs, or S3 prefix).
    • Identify who in your organization will own retraining schedules and drift-detection alerts (role or email alias).
    • Define the acceptance evidence you expect to confirm a native training pipeline is production-ready (for example end-to-end run time, model accuracy within X% of baseline, and artifact lineage present).

    Deploy feature store and feature pipelines

    • List the feature tables and feature definitions (feature name, key, and description) required for the initial workload.
    • State the online and offline serving latency targets you need for these features (for example online <100ms, offline refresh daily).
    • Choose how you want feature freshness measured and indicate the acceptable staleness threshold (seconds, minutes, hours, or days). Options: Seconds, Minutes, Hours, Daily
    • Indicate whether features should be updated via deterministic recomputation, streaming updates, or a hybrid approach. Options: Deterministic recompute, Streaming updates, Hybrid
    • Provide the entity key(s) and cardinality estimates used by these feature tables (for example customer_id ~10M unique).
    • Appoint who will own feature schema changes and who approves backward-incompatible changes (role).

    Configure unified fine-grained access controls

    • List the identity providers and role naming conventions you currently use (for example SAML/OIDC provider and role prefix).
    • Enumerate the tables and specific columns that must enforce row-level security or column-level masking (provide table and column names).
    • Select the access-control model you require: role-based access control, attribute-based access control, or both. Options: RBAC, ABAC, Both, None
    • Specify the audit and access-log retention period you must meet (for example 90 days, 1 year, 3 years). Options: 90 days, 1 year, 3 years, Custom
    • List the service accounts or automation identities that must be mapped to platform connectors and provide their IDs.
    • Appoint who will be the approver for access requests and exception approvals (role or email alias).

    Consolidate historical storage and remove duplicate copies

    • Indicate the storage prefixes (bucket paths or container prefixes) that contain duplicate copies targeted for consolidation; list top prefixes by size.
    • Estimate the number of duplicate copies for the top dataset and specify the preferred canonical copy path you want to keep.
    • Enumerate any retention or legal-hold policies that prevent immediate deletion of older copies (policy ID or retention window).
    • Do you require validation after consolidation such as checksum match by partition or schema-drift checks? Options: Yes, No
    • List which teams must retain read access or be notified after consolidation for audit and compliance purposes (team names or roles).
    • Provide the historical storage inputs we should use for the three-year TCO model (current storage GB and $/GB-month per tier).

    Migrate recurring ETL/ELT pipelines to platform

    • Name the ETL/ELT DAGs or job IDs and their scheduling cadence that you want migrated.
    • Enumerate the connector types required for your pipelines (relational databases, APIs, on-prem file shares, message queues). Options: Relational DBs, APIs, File shares, Message queues, Other
    • State the maximum acceptable migration window per pipeline in minutes of downtime (or zero-downtime requirement). Options: Zero downtime, <15 minutes, 15-60 minutes, >60 minutes
    • Flag whether any pipelines depend on stored procedures or external UDFs that must be migrated or reimplemented. Options: Yes, No
    • Appoint who will be responsible for updating downstream consumers of transformed datasets after migration (role).
    • Define the verification steps you require to confirm parity after migration (for example row-level reconciliation, aggregation checks, SLA run-time comparison).

    Perform cutover and redirect production queries

    • List the production consumers (BI dashboards, scheduled reports, API endpoints) that must be redirected at cutover.
    • Propose the cutover window you prefer and identify any blackout periods that must be avoided.
    • Appoint the official cutover owner and who approves rollback decisions (role or 24/7 contact).
    • Specify the rollback trigger conditions that will cause an automated or manual revert (for example query error rate >5% or latency >2x baseline). Options: Error rate threshold, Latency threshold, Data mismatch, Manual rollback only
    • List the DNS entries, connection strings, or environment variables that must be updated at cutover and who has permission to change them.
    • Describe how you will validate that production queries are running on the platform after cutover (for example query ID mapping, monitoring alerts, or user confirmation).
  4. Mutual Commit

    Finalize commercial and legal terms, consumption pricing, milestones, and acceptance gates tied to performance and TCO milestones.

    Agreement Modules

    • Subscription Agreement
    • Order Form (Consumption Pricing)
    • Service Level Agreement (SLA)
    • Performance & TCO Acceptance Schedule
    • Data Processing Agreement (DPA)
    • Regulatory & Compliance Addendum
    • Change Order Agreement
    • Renewal & Termination Schedule
  5. Deployment

    Lock readiness facts and configuration values before execution begins.

    1. Pre-Deployment Readiness

      Confirm concrete readiness facts — environments, data locations, named owners, access permissions, migration windows, and rollback plans.

      Pre-Deployment Questions

      Environment and site access

      • Which environments will be involved in the migration? Select all that apply (we use this to build per-environment tasks). Options: Production, Staging / pre-production, Development, Data-science sandbox, Other (please specify)
      • For each environment selected above, confirm readiness status or provide the target date when the environment will be network-accessible to the platform (so we can schedule cutover and validation).

      Data and configuration

      • Which source system categories contain the datasets for the initial workload? Select all that apply (this helps size connectors and ingestion work). Options: Object storage (S3-compatible), Source data warehouse, Relational operational databases, Message streaming topics, File shares / NAS, Other (please specify)
      • Has the scope of datasets to migrate for the initial workload been finalized and assigned an owner? If yes, list dataset identifiers and the owning team/person; if not, state the date by which scope and owners will be finalized.

      People and ownership

      • Provide the named owner (name and email) for each role: deployment lead, data owner, security/compliance approver, network/infra owner, and finance/commercial contact (these names are used in the runbook and approval gates).
      • Are the required integration/service accounts and role-based access provisioned in the source and target systems for the platform's integration endpoint? Options: All required accounts and roles provisioned, Partially provisioned — some accounts/roles missing, Not provisioned — target date required, No — we need the seller to coordinate provisioning
      • If provisioning is partial or not done, list which accounts/roles are missing and the target date for provisioning (so we can sequence tasks).

      Timing and constraints

      • List approved migration windows and any blackout periods for each environment or site (include time zone) so we can plan cutover and validation runs.
      • Is an executable rollback plan documented and approved for the initial workload, and is an owner assigned to execute it if needed? Options: Yes — documented and owner assigned, Yes — documented but owner unassigned, No — plan required by a target date, No — we need the seller to draft the rollback plan
      • Are there data residency, regulatory, or encryption constraints that will affect where data can be processed or stored? If yes, name the constraint and the compliance owner who will approve handling. Options: No constraints, Yes — residency limits (specify in response), Yes — specific compliance controls required (specify in response), Other (specify)
    2. Configuration Details

      Lock exact configuration values the deployment requires — connectors, credentials, ingestion pipelines, partitioning/compute sizing, and monitoring thresholds.

      Configuration Details

      Environments & Endpoints

      • Enter the canonical environment name for the initial deployment (format: short, alphanumeric, no spaces — e.g., 'prod-analytics-1')
      • Enter the storage root path the platform will use for the initial workload (format: s3://bucket/path or gs://bucket/path or abfss://[email protected]/path)
      • Select the primary cloud region for the deployment (Default: AWS - us-east-1) Options: AWS - us-east-1, AWS - us-west-2, GCP - us-central1, GCP - europe-west1, Azure - eastus, Azure - westeurope, Other (enter region below)

      Connectors & Credential Identifiers

      • Select the source system type for the initial workload the connector will target Options: Cloud data warehouse (query endpoint), Object storage (raw files), Streaming platform (e.g., Kafka/event hub), Relational DB (OLTP), NoSQL store, Other (specify)
      • Enter the non-secret connector identifier the integration will use (format: integration client ID or service-principal name as recorded in your secrets manager — do NOT paste secrets)

      Ingestion & Pipelines

      • Select the ingestion pattern for the initial workload Options: Scheduled batch (daily/hourly), Micro-batch (minutes), Real-time streaming, Ad-hoc/manual loads
      • Enter the source sample data path that benchmarks will read (format: s3://... or gs://... or abfss://...)
      • Enter expected average daily ingest volume in GB (numeric — Default: 100)

      Partitioning, Compute & Monitoring Targets

      • Enter the preferred partitioning key for the initial workload (single field name, e.g., 'event_date' or 'customer_id')
      • Alert threshold: percent deviation from the agreed monthly cost baseline that should trigger finance notification (numeric percent — Default: 10)
    3. Rollout Execution

      Execute migration of the initial workload, validate query and ML performance, apply unified access controls, and measure cost savings versus the agreed baseline.

  6. Success

    Validate outcomes against success metrics (including TCO reduction), capture learnings, and track issues and enhancement requests to support expansion.

    Success Reviews

    • Go-live Health Check (weeks 1-4)
    • First Measurement Review (weeks 4-10)
    • Acceptance Gate and Incumbent Wind-down (around day 90)
    • Quarterly Realization Review (ongoing quarterly)
    • Annual Success Validation (12-month review)

    Issues & Enhancements

    • Publish the quarter's realization dashboard with the metric sources and any adjustments to the baseline calculation.
    • Publish the acceptance decision record that lists pass/fail per criterion and the named signatory.
    • Execute the incumbent decommission checklist or, if retained read-only, publish access controls, archive locations, and contract termination steps.
    • Document remaining remediation work with verification criteria and a re-check date for final acceptance where required.
    • Trend review of core metrics
    • Confirm whether the quarter's TCO reduction percentage remains on the trajectory toward the 12-month target recorded in Outcome Discovery.
    • Ensure median query latency for the recurring workload remains within the performance band recorded in Technical Proof & Benchmarks or document corrective actions if not.
    • Maintain a prioritized list of operational enhancements and a clear owner/timeline for each high-priority item.
    • Reconfirm success criteria and owners
    • Close or reassign operational tickets older than the agreed SLA and document root-cause fixes taken.
    • Add approved enhancement requests to the backlog with target delivery quarters and verification criteria.
    • Validate cumulative outcomes
    • Confirm whether the 12-month cumulative TCO reduction target recorded in Outcome Discovery was met and document the verified dollar or percentage impact.
    • Validate that storage footprint reduction in terabytes meets the consolidation targets recorded in Solution Scope and document any variance.
    • Produce a lessons-learned summary and an outstanding-issues tracker with owners and resolution dates.
    • Publish the annual success report with verified metrics, methodology, and lessons learned for internal stakeholder review.
    • Deliver the outstanding-issues register with owners and target resolution dates and schedule verification checkpoints.
    • Archive the acceptance artifacts and retention/decommission documentation for the incumbent system in the project repository.
    • Confirm the initial workload runs end-to-end in the production environment as described in Solution Scope.
    • Identify and assign owners for all high-severity issues blocking normal operations.
    • Agree a date for the first measurement meeting and the data sources to be used for metrics.
    • Publish a one-page deployment validation summary listing environment status, connectors verified, and open issues.
    • Resolve and document access or data-path blockers with target resolution dates before the first measurement meeting.
    • Confirm the telemetry and billing sources that will be used to report spend and performance at the first measurement checkpoint.
    • Present first measurement data
    • Determine whether monthly analytics infrastructure spend is moving toward the baseline target recorded in Outcome Discovery.
    • Confirm whether average and 95th percentile query latency for the representative workload meet the performance targets recorded in Technical Proof & Benchmarks.
    • Agree a concrete remediation plan with dates that puts the engagement on track for the acceptance gate at day 90.
    • Produce a remediation tracker listing each corrective action, expected impact on the named metrics, and target completion dates.
    • Run and share follow-up benchmark runs for the representative workload after remediation is applied.
    • Confirm the data sources and formulas used to calculate monthly analytics spend so subsequent reports are comparable to the baseline.
    • Restate numeric acceptance criteria
    • Produce a documented acceptance decision for each numeric criterion recorded in Outcome Discovery and Technical Proof & Benchmarks.
    • Confirm the incumbent system is either decommissioned or retained in read-only state and that data archival or migration is complete.
    • Agree remediation items with timelines for any conditional acceptance and schedule the verification checkpoint.
    • Persistent issues and ticket burn-down
    • Governance and access control review
    • Present outcome data against each criterion
    • Compare results to targets
    • Deployment and migration validation
    • Enhancement requests and backlog prioritization
    • Document pass or fail per criterion and decision
    • Capture learnings and process improvements
    • Root-cause diagnosis for any gaps
    • Early adoption signals and usage patterns
    • Outstanding issues and enhancement register
    • Open issues and blockers
    • Agree corrective actions and timeline to acceptance gate
    • Short operational actions and verification
    • Incumbent system wind-down confirmation
    • Agree immediate remediation actions
    • Agree remediation items and resolution timeline
First-Party AI

1-2 minutes please — Your AI agent is working

First-Party AI™ can make mistakes. Always check important information.