Definity Embeds Agents in Spark Pipelines to Prevent AI System
Definity, a Chicago-based startup, secured $12M in Series A funding to advance its unique data pipeline reliability solution. By embedding agents directly within Spark pipelines, Definity proactively identifies and prevents failures, bad data, and inefficiencies during execution, crucial for the integrity of agentic AI systems.
Definity, a Chicago-based data pipeline operations startup, announced on Wednesday, April 29, 2026, it has secured $12 million in Series A funding. The investment, led by GreatPoint Ventures with participation from Dynatrace, StageOne Ventures, and Hyde Park Venture Partners, will fuel Definity's mission to revolutionize data pipeline reliability. The company's innovative approach embeds intelligent agents directly within Spark and DBT pipelines, proactively catching and preventing failures, bad data, and inefficiencies during execution—a critical advancement for ensuring the integrity of data feeding increasingly vital agentic AI systems.
Why Existing Pipeline Monitoring Falls Short
Traditional data pipeline monitoring tools typically operate from outside the execution layer, gathering metrics only after a job has completed. Solutions from companies like Datadog (which acquired Metaplane), Databricks system tables, Unravel Data, and Acceldata provide valuable insights, but often after the damage is done. According to Roy Daniel, CEO and co-founder of Definity, this "after-the-fact" approach means that by the time a problem is identified, the pipeline has already run, potentially propagating bad data downstream, wasting compute resources, and ultimately breaking AI systems reliant on timely, clean input. This reactive posture is no longer sufficient for the demands of modern, AI-driven enterprises where data quality and availability are paramount.
Definity's In-Execution Intelligence
Definity differentiates itself by integrating its proprietary agents directly into the pipeline's execution layer. This is achieved through inline instrumentation, where a JVM agent is installed with a single line of code, operating below the platform layer to pull real-time execution data directly from Spark.
These agents capture a comprehensive range of critical metrics as the pipeline runs, including query execution behavior, memory pressure, data skew, shuffle patterns, and infrastructure utilization. Crucially, the system dynamically infers data lineage between pipelines and tables without requiring a predefined data catalog, providing a full-stack, real-time, and production-aware context.
Beyond mere observation, Definity's agents can actively intervene during a pipeline run. This includes modifying resource allocation dynamically, stopping a job before corrupt data can propagate further, or preempting a pipeline based on detected upstream data conditions. Daniel cited an instance where an agent prevented a downstream pipeline from starting because an upstream job had been preempted, leading to stale input data. While detection and prevention occur in real-time, comprehensive root cause analysis and optimization recommendations are generated on-demand when an engineer queries the assistant, utilizing the already-assembled execution context. The agent's overhead is minimal, adding approximately one second of compute to an hour-long run, and supports full on-premises deployment for sensitive environments by only transmitting metadata externally.
Real-World Impact at Nexxen
Nexxen, an ad tech platform that manages large-scale, on-premises Spark pipelines for mission-critical advertising workloads, has already experienced the tangible benefits of Definity's platform. Dennis Meyer, Director of Data Engineering at Nexxen, explained that their primary challenge wasn't frequent pipeline failures, but rather the cumulative cost of inefficiencies within a non-elastic, on-premises environment where waste directly impacts costs.
Existing monitoring tools provided fragmented visibility, making systematic optimization difficult. Upon deploying Definity without requiring any pipeline code changes, Nexxen quickly gained full-stack visibility. Meyer reported that his team identified 33% of its optimization opportunities within the first week, leading to a remarkable 70% reduction in engineering effort spent on troubleshooting and optimization. This operational efficiency freed up infrastructure capacity, enabling Nexxen to support increasing workload demands without additional hardware investments. Meyer underscored the shift: "The key shift was moving from reactive troubleshooting to proactive, continuous optimization. At scale, the biggest gap often isn't tooling — it's actionable visibility."
Implications for Enterprise Data Teams
Definity's approach signifies a crucial evolution for enterprise data teams, particularly those operating production Spark environments. As data pipelines increasingly underpin agentic AI workloads with direct business dependencies, the consequences of failures escalate from mere inconvenience to blocking critical AI delivery. This transformation elevates pipeline operations into a fundamental AI infrastructure challenge.
The proven ability to significantly reduce troubleshooting and optimization effort, as demonstrated by Nexxen's 70% reduction, highlights a substantial recoverable cost. For lean data engineering teams, reclaiming this time to focus on strategic roadmap initiatives presents a compelling immediate case for evaluating in-execution intelligence solutions like Definity. This paradigm shift from reactive post-mortem analysis to proactive, in-run intervention is set to redefine data reliability and operational efficiency in the era of pervasive AI.
FAQ
Q: How does Definity's approach differ from traditional data pipeline monitoring tools?
A: Traditional tools typically monitor pipelines externally and report issues after a job has completed. Definity embeds intelligent agents inside the pipeline's execution layer (via a JVM agent), allowing for real-time capture of execution data and proactive intervention, such as stopping a job or modifying resources, before failures or bad data propagate downstream.
Q: What specific benefits have early Definity users, like Nexxen, reported?
A: Nexxen, an ad tech platform, identified 33% of its optimization opportunities within the first week of deployment. They also saw a 70% reduction in engineering effort dedicated to troubleshooting and optimization, significantly freed up infrastructure capacity, and could support workload growth without additional hardware investment. Definity also claims customers resolve complex Spark issues up to 10x faster.
Q: Why is Definity's solution particularly important for agentic AI systems?
A: Agentic AI systems critically depend on a continuous supply of clean, accurate, and timely data. A data pipeline that delivers stale or faulty data, or fails silently, directly impairs or breaks the AI system relying on it. Definity's ability to prevent these issues in real-time ensures the foundational data integrity required for reliable and effective AI operations.
Related articles
Meta's AI Engine Propels New App Development Surge
Meta is leveraging AI, especially large language models (LLMs), to rapidly develop and launch new consumer applications, marking a strategic pivot. CEO Mark Zuckerberg announced more apps are coming soon, following recent launches for Facebook Groups, Marketplace, and Instagram. This AI-driven acceleration helps Meta test ideas faster and has significantly boosted apps like Threads.
Mastering Agentic AI: Building Autonomous Workflows with LangGraph
The software development landscape is evolving beyond single-prompt LLMs to autonomous AI agents capable of complex, multi-step workflows. LangChain, with its extension LangGraph, provides the essential tools to build these sophisticated systems, enabling stateful, cyclical agent behaviors. Developers can implement advanced features like Human-in-the-Loop, RAG, and streaming responses, and deploy these agents using industry best practices.
startups: AI agents are about to run the enterprise. Onyx raised
Onyx Security, an Israeli startup, has secured a $113 million Series B funding round, valuing it at $640 million. The company's "secure AI control plane" sits between enterprise AI agents and critical systems, inspecting and blocking unauthorized actions to keep humans in control. This investment addresses the growing need for AI agent accountability as autonomous AI rapidly takes over enterprise operations, a market already seeing significant investment and activity.
Venture Capitalist's Stirring Speech: A Rallying Cry for Seattle
AI House co-founder Jacob Colker delivered a passionate 'rallying cry' for Seattle at a recent tech event, urging residents to overcome a 'confidence problem' and recognize the city's immense potential. He emphasized the need for greater community and connection to fully leverage Seattle's role in the future of technology.
Zuckerberg Outlines Meta's Bold Vision for Personal AI Agents
Meta CEO Mark Zuckerberg has announced ambitious plans for a significant push into personal AI agents, a strategy he unveiled during the company's Q2 2026 earnings call on Wednesday, July 29, 2026. This initiative aims
TechCrunch Disrupt 2026: AI Stage Tackles SaaS Reckoning, Security
TechCrunch Disrupt 2026, held Oct 13-15 in San Francisco, features an AI Stage presented by Google for Startups. It will explore how AI is reshaping business models, creating security gaps like the 'agent security gap,' and pioneering new job categories like the 'GTM Engineer.' Industry leaders will share insights on topics from enterprise AI security to the future of video intelligence.