Version: 2.4

Kafka Task

A Kafka Task enables asynchronous communication between the tSM Process Engine and external systems through Apache Kafka. It publishes a request and suspends the process until a correlated response arrives. An External Task is also a durable wait state, but its worker discovers and completes work through the REST fetchAndLock protocol.

When to Use a Kafka Task

Use a Kafka Task when:

the operation is long-running and you don't want to hold a worker thread
you need fire-and-wait semantics with asynchronous response
the consumer is a system that natively integrates with Kafka
you want to leverage Kafka's built-in durability, ordering, and scalability

For REST-native pull workers, use External Tasks instead.

How It Works

A Kafka Task is a standard BPMN Receive Task with the #{tsmKafkaTaskExecutor} execution listener attached to its start event:

The process reaches the Kafka Task
tSM publishes a message to the configured request topic with the task payload
tSM automatically adds correlation and context headers to the message (see below)
The process instance suspends and waits for a response
An external consumer processes the message and publishes a response to the response topic
tSM correlates the response using the correlationId header and continues the process

Process ──► Request Topic  ──► External Consumer
                                      │
Process ◄── Response Topic ◄──────────┘

The publish waits for the broker acknowledgement inside the Camunda command, so the process never commits into a waiting state after a known send failure.

Delivery is at-least-once

If Kafka acknowledges the request but the following database commit fails, the request has already been published and will be published again when the command is retried. Consumers must deduplicate requests by correlationId.

Task Templates and Kafka Tasks

Just like External Tasks, Kafka consumers expect a specific message contract — topic, payload structure, and headers. You cannot send arbitrary data and expect the consumer to understand it.

For this reason, Kafka Task consumers are typically delivered together with a Task Template that preconfigures the correct topic, payload structure, and defaults. When a template is available, prefer using it over manual configuration.

Configuring a Kafka Task

BPMN Element

Use a Receive Task with a messageRef, and attach the #{tsmKafkaTaskExecutor} delegate as an execution listener on the start event. There is no special task type to select — the Receive Task provides the waiting semantics and the listener provides the Kafka integration.

Always model camunda:asyncBefore="true"

With an async-before job, a Kafka publish failure is handled by standard Camunda job retry and failed-job incident handling, which is visible in the cockpit and retryable. Without it, the failure surfaces on whatever triggered the activity.

Task Fields

All fields are optional. A field value is either a literal or an expression, and an expression has to be the whole value — #{...} and ${...} are evaluated in the process expression context, while a concatenation such as order-#{orderId} is published verbatim as a literal, with no error. #{...} is evaluated as tSM SpEL (so #{#variables.orderId} resolves), ${...} as legacy JUEL; the two syntaxes are not interchangeable.

Field	Description	When omitted
`requestTopic`	The Kafka topic to publish the request message to	the configured default (see Topics)
`responseTopic`	The Kafka topic the consumer should reply to	the configured default
`key`	The message key (used for partitioning)	the process business key
`payload`	The message body	an empty JSON object `{}`

A payload that is already a JSON string is published verbatim; any other value is serialized to JSON.

caution

A payload expression that dereferences a process variable which is not set will fail the activity. Either guarantee the variable exists, or omit the field.

Topics

Topic names are resolved in this order, highest priority first:

the requestTopic / responseTopic field on the task,
tsm.process.kafka-task.request-topic / .default-response-topic,
tsm.kafka.topics.tsmProcessRequest / .tsmProcessResponse,
${tsm.kafka.prefix}-process-request / ${tsm.kafka.prefix}-process-response.

The prefix is part of the default on purpose: environments that share a Kafka cluster must not publish into each other's topics.

Every response topic must be listed in configuration

The listener topology is fixed at application startup. A task advertising a responseTopic that this engine does not consume is rejected at publish time rather than left waiting forever, so each custom response topic has to appear in tsm.process.kafka-task.response-topics — and the application has to be restarted afterwards. The default response topic (and the Connector Worker topic, when enabled) are included automatically.

Kafka Task does not create topics. Provision custom topics through the standard environment Kafka configuration before enabling them.

Engine Properties

The feature is opt-in. Enable it only in services that actually deploy Kafka Tasks — otherwise every Camunda database would consume and retry responses belonging to other services.

tsm:
  process:
    kafka-task:
      enabled: true
      # request-topic: custom-request
      # default-response-topic: custom-response
      response-topics:
        - custom-response
      connector-worker-compatibility-enabled: false
      # connector-worker-response-topic: connector-response
      # consumer-group: service-specific-group
      retry-attempts: 5
      retry-backoff-ms: 500

Property	Description	Default
`enabled`	Whether publishing and the response listener are active in this service	`false`
`request-topic`	Overrides the default request topic	prefix-derived
`default-response-topic`	Overrides the default response topic	prefix-derived
`response-topics`	Additional response topics to subscribe to	empty
`connector-worker-compatibility-enabled`	Consume the Connector Worker response topic with the same listener	`false`
`connector-worker-response-topic`	Overrides the Connector Worker response topic	`${tsm.kafka.prefix}-tsm-receive-task`
`consumer-group`	Overrides the consumer group	`${tsm.kafka.prefix}-${spring.application.name}-process-kafka-task`
`retry-attempts`	Redeliveries of a transient failure before the response is given up on	`5`
`retry-backoff-ms`	Backoff between redeliveries, in milliseconds	`500`

The default consumer group is shared by replicas of one service but differs between services with separate Camunda databases, so replicas balance partitions normally while unrelated engines never compete for the same response.

The first start replays the response topics

The consumer group is new the first time the feature is enabled in a service, and the listener reads from the earliest offset. Every response still retained on the configured response topics is therefore replayed once. A replayed response no longer matches a waiting execution, so each one burns the full retry budget (retry-attempts × retry-backoff-ms, blocking that partition) before it is acknowledged and logged as late. Enable the feature outside peak traffic, or seed the new consumer group's offsets to latest beforehand — most importantly with connector-worker-compatibility-enabled, where the shared Connector Worker topic can hold a lot of history.

Automatic Headers

tSM transparently adds the following headers to every outgoing Kafka message. You do not need to configure these — they are injected automatically:

Header	Description
`correlationId`	Opaque activation id — the responder must copy it back unchanged
`executionId`	Compatibility alias; do not use for new integrations
`processInstanceId`	ID of the current process instance
`responseTopic`	Topic the responder should publish to
`messageName`	Expected BPMN message name
`messageType`	Legacy alias for `messageName`
`ownerType`	Optional; from the `ownerType` process variable, falling back to `entityType`
`ownerId`	Optional; the process business key
`traceId`, `userId`, `tenantId`	Added by the shared Kafka sender when available

note

correlationId identifies one activation of one task — not the process instance and not the execution. Treat it as an opaque string: do not parse it, derive from it, or reuse it across messages. This is what lets tSM reject a late response that belongs to an earlier activation of the same task. Its internal shape also differs between an async-before job and a synchronous activation — one more reason not to parse it.

Response and Correlation

Publish to the advertised responseTopic and copy correlationId back unchanged. That is all the correlation needs: for a Kafka Task the engine stores the expected BPMN message name together with the correlation state, so it never depends on the response to tell it which message to deliver — a boundary message event on the same Receive Task does not confuse it. Copying messageName back is still recommended, because it is what resolves the message for producers that reply with a bare executionId (the Connector Worker contract), and it keeps the logs on both sides readable.

The listener also accepts the legacy executionId and messageType headers, so a producer written against the Connector Worker contract works unchanged.

Propagating processInstanceId, ownerType, ownerId and traceId is recommended for end-to-end traceability and consistent logging on both sides.

Response Payload

The response body is turned into process variables:

a JSON object is flattened into top-level process variables, and the whole payload is additionally stored under the BPMN message name;
a JSON array or scalar is stored only under the BPMN message name;
an empty body sets no variables.

For a Receive Task whose message is named kafka and this response:

{
  "convertedAmount": 2450.50,
  "exchangeRate": 24.505
}

the process continues with convertedAmount, exchangeRate and kafka (holding the whole object) available as variables.

Connector Worker ProcessingResult values are accepted the same way, for both successful and failed worker records.

Error Handling

Situation	Behaviour
Request cannot be published	The activity fails. With `camunda:asyncBefore="true"` this becomes a standard Camunda job retry and, once retries are exhausted, a failed-job incident.
Task advertises a response topic this engine does not consume	Publishing is refused with a configuration error — the process is not left waiting for a reply nobody will deliver.
Response body is not readable JSON, or the correlation header is missing or malformed	A contract violation: routed to the DLQ without retry.
Transient failure while correlating	Redelivered up to `retry-attempts` times with the configured backoff.
Response still unmatched after the retry budget	Acknowledged and logged as unknown, duplicate, or late. Deliberately not DLQ-ed — retrying could never succeed, and keeping these would bury real failures.
Duplicate or stale response	Dropped. The first response wins; a response belonging to an earlier activation cannot complete a later one.

The bounded retry is what makes an "early" response safe: a responder fast enough to answer before the waiting state is visible gets retried rather than lost.

Timeouts

The engine deliberately adds no timeout of its own — a Kafka Task waits as long as the Receive Task waits. Model the timeout in BPMN with an interrupting boundary timer:

<bpmn:boundaryEvent id="kafkaTimeout" attachedToRef="kafkaIntegrationCall"
                    cancelActivity="true">
  <bpmn:timerEventDefinition>
    <bpmn:timeDuration>PT5M</bpmn:timeDuration>
  </bpmn:timerEventDefinition>
</bpmn:boundaryEvent>

When the timer fires, the activity is cancelled together with its correlation state, so a response arriving afterwards is simply dropped — it cannot revive or duplicate the instance.

Relationship to the Connector Worker

Connector Worker responses use the same contract and are handled by the same listener — the Connector Worker is one producer implementation of the general Kafka Task contract.

This is a separate opt-in (connector-worker-compatibility-enabled), because that topic can carry high traffic shared by many processes. Connector Worker can also select a device-specific response topic, so every tsm.devices.*.kafka-response-topic actually in use must be listed in tsm.process.kafka-task.response-topics.

Example

BPMN Configuration

A Kafka Task uses a standard Receive Task — a BPMN element that naturally represents "wait for an external message". The Kafka-specific logic (publishing the request and setting up correlation) is attached as an execution listener on the start event. This is a clean fit:

The Receive Task handles the waiting semantics — the process suspends until a correlated message arrives.
The execution listener fires when the task starts, publishes the Kafka message, and lets the Receive Task take over.

This separation means the BPMN model stays standard-compliant, while the Kafka integration is handled transparently by the listener delegate.

The snippets below assume tsm.kafka.prefix=datalite, so the default topics resolve to datalite-process-request and datalite-process-response. The task leaves both topic fields out and relies on those defaults — prefer that over writing a topic name into the model, because a name taken from another environment is either rejected at publish time (responseTopic) or published where nobody listens (requestTopic).

<bpmn:receiveTask id="kafkaIntegrationCall" name="Call Integration via Kafka"
                  messageRef="Message_KafkaIntegration"
                  camunda:asyncBefore="true">
  <bpmn:extensionElements>
    <camunda:executionListener event="start"
                              delegateExpression="#{tsmKafkaTaskExecutor}">
      <camunda:field name="key" stringValue="#{execution.processBusinessKey}"/>
      <camunda:field name="payload"
                     stringValue="#{#variables.requestPayload}"/>
    </camunda:executionListener>
  </bpmn:extensionElements>
  <bpmn:incoming>Flow_1</bpmn:incoming>
  <bpmn:outgoing>Flow_2</bpmn:outgoing>
</bpmn:receiveTask>

Consumer Perspective

The consumer receives a Kafka message on datalite-process-request with:

Headers:

correlationId:     kt2|a1b2c3d4-e5f6-7890-abcd-ef1234567890|job:9d1c7b64-8f2a-11f0-9d3e-0242ac120003:activity:kafkaIntegrationCall
processInstanceId: a1b2c3d4-e5f6-7890-abcd-ef1234567890
responseTopic:     datalite-process-response
messageName:       kafka
ownerType:         Ticket
ownerId:           TICKET-1234
traceId:           6a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d
tenantId:          datalite

Payload:

{
  "action": "convertCurrency",
  "amount": 100,
  "sourceCurrency": "USD",
  "targetCurrency": "CZK"
}

The consumer processes the request and publishes to datalite-process-response, copying correlationId unchanged:

Headers:

correlationId:     kt2|a1b2c3d4-e5f6-7890-abcd-ef1234567890|job:9d1c7b64-8f2a-11f0-9d3e-0242ac120003:activity:kafkaIntegrationCall
messageName:       kafka
processInstanceId: a1b2c3d4-e5f6-7890-abcd-ef1234567890
ownerType:         Ticket
ownerId:           TICKET-1234
traceId:           6a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d

Payload:

{
  "convertedAmount": 2450.50,
  "exchangeRate": 24.505
}

tSM correlates the response via correlationId and the process continues.

Comparison: Kafka Task vs External Task

Aspect	External Task	Kafka Task
Communication	REST (fetchAndLock)	Kafka (publish/subscribe)
Pattern	Durable fetch-and-lock polling	Asynchronous request/result messaging
Worker model	Worker pulls tasks	Consumer receives messages
Durability	Engine-managed	Kafka-managed
Best for	Short-lived operations, REST-native workers	Long-running operations, event-driven systems
BPMN element	Service Task (type: external)	Receive Task with the `#{tsmKafkaTaskExecutor}` listener

Best Practices

Use Task Templates whenever available — they ensure the correct topic and payload structure
Model camunda:asyncBefore="true" so a publish failure becomes a retryable Camunda job
Deduplicate requests by correlationId on the consumer side — delivery is at-least-once
Copy correlationId back unchanged and never parse or derive from it; copy messageName too — it is what resolves the message for legacy executionId correlation
Model a boundary timer on any Kafka Task whose consumer might not answer
List every custom response topic in tsm.process.kafka-task.response-topics and restart the service
Enable the feature only in services that deploy Kafka Tasks, and enable Connector Worker compatibility only where those responses are actually awaited
Keep message payloads compact and structured
Leave requestTopic / responseTopic out of the model unless a consumer really needs its own topic — the defaults follow tsm.kafka.prefix and therefore differ per environment
Use the ownerType and ownerId headers, when present, for consumer-side routing and logging
Use the traceId header to participate in distributed tracing

When to Use a Kafka Task​

How It Works​

Task Templates and Kafka Tasks​

Configuring a Kafka Task​

BPMN Element​

Task Fields​

Topics​

Engine Properties​

Automatic Headers​

Response and Correlation​

Response Payload​

Error Handling​

Timeouts​

Relationship to the Connector Worker​

Example​

BPMN Configuration​

Consumer Perspective​

Comparison: Kafka Task vs External Task​

Best Practices​