- Market
- Saudi Arabia
- Written for
- CTO, CIO
- Decision
- Adopt event streaming where it is justified and design it so it stays operable
Information security for software delivery

ISO/IEC 27001
ISO/IEC 27001 certified
The first decision
Where event streaming beats batch, and where batch remains the right answer
The comparison most Kafka adoptions skip. Neither column is the modern one and neither is the legacy one; they answer different questions, and forcing one to do the other’s job is where the operational pain comes from.
| Option | The question it answers | What it demands of you | Where it goes wrong | Best for |
|---|---|---|---|---|
| Event streaming | "What is happening right now, and who else needs to know?" — fraud signals, inventory movements, telemetry, anything multiple systems consume as it occurs, or replay later. | Real operational investment: a platform to run or buy, schema governance, and teams comfortable debugging distributed consumers. | Adopted for one point-to-point feed a queue would carry, importing distributed-systems complexity with no second consumer to pay for it. | Events with several consumers, a replay need, or a real-time action attached. |
| Batch processing | "What is the complete, settled picture as of a point in time?" — reporting, reconciliation, billing runs, model training: work that wants completeness over immediacy. | Far less: a scheduler, idempotent jobs and a place to rerun yesterday when something fails — operational muscles most teams already have. | Stretched toward real time by running ever more often, until the "batch" is a fragile imitation of streaming with none of its guarantees. | Settled-picture work on a cadence the business genuinely accepts. |
| A message queue | "Hand this task to exactly one worker and forget it" — order processing, notifications, background jobs where the message is consumed once and gone. | The least of the three: mature brokers, simple semantics, well-understood failure handling. | Outgrown silently when a second consumer wants the same messages — which is the honest signal it is time to consider a log like Kafka. | Task distribution where one consumer per message is the design, not a limitation. |
The real work
Topics and schemas are the API your whole estate will call
Standing up a Kafka cluster is the small half of the project. The half that determines whether the platform is still loved in three years is the design of what flows through it: how topics map to business meaning, how they are partitioned, and what the events themselves look like. A topic is a published contract — every consumer that subscribes is a dependency on its shape — and partitioning decides both ordering and parallelism, because Kafka orders events only within a partition. Choose the partition key casually and you either lose the ordering a consumer silently relies on, or funnel the busiest entity through one partition and cap your own throughput.
Schemas are where the discipline either exists or does not. An event stream without governed schemas is a promise that every consumer must parse whatever every producer felt like sending, forever. A schema registry with compatibility rules turns that anarchy into a contract: a producer may evolve its events only in ways existing consumers can absorb, and the check runs at build time, not at the 2 a.m. moment a malformed event stalls a consumer group. This is unglamorous work, and it is the difference between an event backbone and a message swamp.
Exactly-once deserves an honest sentence, because it sells the platform and disappoints the unwary. Within a Kafka-to-Kafka pipeline, transactions and idempotent producers genuinely deliver exactly-once processing of a record’s effects. The moment the pipeline touches the outside world — a database write, an email, a payment call — that guarantee stops at the boundary, and the receiving side must be idempotent for the end-to-end behaviour to hold. Designing those boundaries is part of the work; believing a configuration flag removed the problem is how duplicates reach production with a clean conscience.
The shape of a flow
From a business event to every system that needs it, through one governed log
The value of the log is on the right of the diagram: many consumers, each reading at its own pace, none of them known to the producer. That decoupling is what you are buying with Kafka’s operational cost.
01
Producers
Order systems, sensors, applications — each publishes events against a registered schema and knows nothing about who consumes them.
- Apps
- Sensors
02
Topics, partitioned
Events land in topics whose partition keys preserve the ordering that matters — per order, per device — while spreading load across the cluster.
- Kafka
03
Schema registry
Every event conforms to a versioned schema, and compatibility rules stop a producer publishing a change its consumers cannot read.
- Registry
04
Consumer groups
Fraud checks read in milliseconds, the warehouse loads in minutes, a new service replays last month — the same events, independent paces, no coordination.
- Analytics
- Services
The dead-letter path
An event a consumer cannot process is parked, with its error, where an operator can inspect and replay it — instead of either stalling the partition behind it or being dropped in silence.
Consumer lag — how far each group trails the head of the log — is the one metric that tells you whether all of this is actually working.
Running it
Operating Kafka yourself versus paying someone to, as categories
The run-it-yourself route means owning brokers, storage, upgrades, rebalancing and capacity — a real platform-engineering commitment that makes sense where you have the team for it, where data residency or network isolation demands it, or where the workload is large and steady enough that the operational investment amortises. The managed category — cloud provider streaming services and vendor-hosted Kafka offerings — trades a recurring bill for most of that burden, and suits teams whose engineers should be building consumers, not tuning brokers. The middle path, running the open platform on cloud infrastructure with automation you own, keeps flexibility and keeps more of the burden than teams expect.
Whichever category you choose, the observability obligation is yours and it centres on consumer lag. Broker health tells you the platform is up; lag per consumer group tells you the business is keeping up. A fraud-check consumer drifting minutes behind the log has quietly stopped doing its job while every dashboard glows green. Lag belongs on the same alerting footing as error rates: a threshold per group, set by how stale that consumer’s work is allowed to be, paging someone when it is crossed.
Our role
What we design on Kafka and what your platform team runs
- 01
The honest case first
We assess each candidate flow against what streaming actually buys — multiple consumers, replay, real-time action — and we will say batch or a queue when that is the better answer, because a platform adopted for one flow is a cost centre.
- 02
Contracts before clusters
We design the topics, partition keys and schemas with compatibility rules enforced in the registry and in the build, so the event backbone evolves without producers and consumers breaking each other.
- 03
Observability from the first consumer
Lag dashboards, per-group thresholds and dead-letter handling ship with the first flow, not after the first silent failure — and they land in whatever monitoring your operation already watches.
Whether to run the platform yourselves or buy it managed is a staffing and residency decision that stays with you; we make the architecture work identically in either category, and leave contracts and runbooks your team can carry either way.
Common questions
What buyers ask
Is Kafka’s exactly-once delivery real or marketing?+
Real, within a boundary that matters. Inside a Kafka-to-Kafka pipeline — consume, transform, produce — transactions and idempotent producers genuinely give you exactly-once processing of a record’s effects, and that is an engineering achievement, not a slogan. The honesty comes at the edges: when a consumer writes to a database, calls a payment API or sends a message to a person, Kafka’s guarantee ends where that system begins, and a retry after an ambiguous failure can repeat the external effect. End-to-end correctness therefore needs idempotent boundaries — an upsert keyed on the event, a deduplication check, an idempotency key on the API call. We design those explicitly rather than letting a configuration setting stand in for them.
Do we need Kafka, or would a message queue serve us fine?+
Count your consumers and your replay needs. If each message is a task for exactly one worker and history has no value after processing, a message queue is simpler to run, simpler to reason about, and the better choice — using Kafka there is buying a distributed log to do a queue’s job. Kafka starts paying when several systems need the same events independently, when a new consumer must be able to replay the past, or when the stream feeds real-time decisions. The honest trajectory many organisations follow is queue first, Kafka when the second and third consumers of the same events appear — and making that move deliberately beats making it because a diagram looked better with a log in the middle.
How should we monitor Kafka consumer lag, and what does it actually tell us?+
Lag is the distance, in messages or time, between the head of the log and what a consumer group has processed — and it is the single best health signal the platform gives you, because it measures outcomes rather than infrastructure. Brokers can be healthy while a consumer falls hours behind, which means the business function that consumer performs has effectively stopped. Monitor lag per consumer group, set thresholds from the freshness each function requires — a fraud check tolerates seconds, a warehouse load tolerates an hour — and alert when a group trends past its threshold rather than only when it crosses it, because sustained growth means the consumer cannot keep up and needs scaling or fixing. Treat a lag alert with the seriousness of an error-rate alert; it is the same news arriving earlier.
In this cluster
Next step
Test the case for streaming
An assessment that lists your candidate event flows, says for each whether streaming is justified, and drafts the topic and schema design for the ones that are.
