Event-driven workflows with Apache Kafka and Apache Airflow

Most enterprise workflows still run on a timetable: export at midnight, load at one, report at six. The business, meanwhile, happens continuously. Orders, payments, sensor readings and customer actions arrive every second, and a nightly batch means decisions are always a day late.

Event-driven workflows close that gap. Instead of asking “what time is it?”, the system asks “what just happened?” and starts the right work immediately.

Two jobs, two tools

Event-driven systems need to do two different things well: move events reliably, and coordinate the multi-step work those events trigger. We use Apache Kafka for the first and Apache Airflow for the second.

• Kafka: durable, ordered, high-throughput event streams that decouple producers from consumers and let you replay history.
• Airflow: dependencies, retries, timeouts, approvals, backfills and a full run history for every workflow an event starts.

The reference architecture

• Capture: applications and change-data-capture connectors publish business events to Kafka topics, with schemas registered and versioned.
• React: Airflow’s event-driven scheduling watches for those events and starts the matching workflow, rather than polling on a timer.
• Orchestrate: the workflow runs the steps the event requires — enrichment, validation, AI inference, writes to downstream systems — in order, with retries and timeouts.
• Decide: where judgment is needed, an approval task pauses the workflow and notifies the right person.
• Publish: results are written back to the warehouse, operational systems and outbound Kafka topics so other services can react in turn.

Design rules we follow

• Data contracts first: every topic has an owner and a registered schema, and breaking changes fail in CI, not in production.
• Idempotent consumers: every step can safely process the same event twice, because at-least-once delivery means it eventually will.
• Dead-letter handling: events that cannot be processed go to a dead-letter topic with enough context to fix and replay them.
• Back-pressure awareness: workflows are sized for peak event rates, with limits that protect downstream systems.
• End-to-end tracing: an event ID follows the work through Kafka, Airflow and every system it touches.

Where it pays off

The strongest cases we see are fraud and payment checks that must run before fulfilment, inventory and order flows that span several systems, real-time data products fed by change-data-capture, and AI workflows — such as refreshing embeddings or re-scoring a customer — that should run the moment new data arrives.

Getting there incrementally

You do not need to re-platform everything at once. We usually start by identifying two or three batch workflows where latency costs money, publishing their source events to Kafka and moving them to event-driven triggers in Airflow. The rest of the estate keeps running on schedules until it, too, earns the move.