Kafka Task

A Kafka Task enables asynchronous communication between the tSM Process Engine and external systems via Apache Kafka. Unlike External Tasks, which use synchronous REST-based polling, a Kafka Task sends a message to a Kafka topic and waits asynchronously for a response — the process instance is suspended until a correlated response message arrives.

---

When to Use a Kafka Task

Use a Kafka Task when:

• the operation is long-running and you don't want to hold a worker thread
• you need fire-and-wait semantics with asynchronous response
• the consumer is a system that natively integrates with Kafka
• you want to leverage Kafka's built-in durability, ordering, and scalability

For synchronous request/response patterns, consider External Tasks instead.

---

How It Works

A Kafka Task is implemented as a special type of Receive Task with its message listener set to KAFKA:

1. The process reaches the Kafka Task
2. tSM publishes a message to the configured request topic (default: tsm-process-request) with the task payload
3. tSM automatically adds correlation and context headers to the message (see below)
4. The process instance suspends and waits for a response
5. An external consumer processes the message and publishes a response to the response topic (default: tsm-process-response)
6. tSM correlates the response using the correlationId header and continues the process

Process ──► Request Topic  ──► External Consumer
                                      │
Process ◄── Response Topic ◄──────────┘

Both the request and response topics are configurable per task, with sensible defaults (tsm-process-request / tsm-process-response).

---

Task Templates and Kafka Tasks

Just like External Tasks, Kafka consumers expect a specific message contract — topic, payload structure, and headers. You cannot send arbitrary data and expect the consumer to understand it.

For this reason, Kafka Task consumers are typically delivered together with a Task Template that preconfigures the correct topic, payload structure, and defaults. When a template is available, prefer using it over manual configuration.

---

Configuring a Kafka Task

BPMN Element

Use a Receive Task in the BPMN model and configure its message listener type to KAFKA.

Kafka Producer Properties

The following standard Kafka properties are configured on the task:

Property	Description	Default	Example
Request Topic	The Kafka topic to publish the request message to	tsm-process-request	my-integration-request
Response Topic	The Kafka topic to listen on for the response	tsm-process-response	my-integration-response
Key	The message key (used for partitioning)	—	#{execution.processBusinessKey}
Payload	The message body (JSON)	—	#{#variables.requestPayload}

Automatic Headers

tSM transparently adds the following headers to every outgoing Kafka message. You do not need to configure these — they are injected automatically:

Header	Description	Example Value
correlationId	Unique ID for correlating the response back to the waiting process execution (UUID)	d4f7a8b1-3c2e-4f5a-9b8d-1e2f3a4b5c6d
processInstanceId	ID of the current process instance (UUID)	a1b2c3d4-e5f6-7890-abcd-ef1234567890
ownerType	Type of the owning entity, determined by the process type	Ticket, Order, Customer
ownerId	ID of the owning entity (the business key of the process, UUID)	f8e7d6c5-b4a3-2190-8765-432109876543
traceId	Distributed tracing ID propagated from OpenTracing / OpenTelemetry	6a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d

note

Internally, the correlationId contains the current executionId of the process engine. This is an implementation detail — consumers should treat it as an opaque identifier and simply copy it to the response.

These headers allow the consumer to:

• identify which process and entity the message belongs to
• route or filter messages based on owner type
• correlate the response back to the correct waiting execution
• participate in distributed tracing

---

Response and Correlation

Response Topic

The default response topic is tsm-process-response. When a consumer finishes processing, it publishes a response message to the configured response topic. tSM listens on the response topic, correlates the message using the correlationId header, and continues the suspended process.

The response message must include:

• the correlationId header (copied from the request) — this is how tSM matches the response to the waiting execution

The response should also carry the following headers (copied from the request) for logging and distributed tracing:

• processInstanceId
• ownerType and ownerId
• traceId

While only correlationId is strictly required for correlation, propagating all headers ensures end-to-end traceability and consistent logging on both sides.

Custom Response Topics

The response topic is configurable per task. If different consumers respond on different topics, you can override the default tsm-process-response on each Kafka Task. Each response topic used must be configured in the tSM process engine so that tSM knows to listen on it.

Response Payload

The response message body is deserialized and made available as process variables. Example:

{
  "status": "completed",
  "result": {
    "convertedAmount": 2450.50,
    "exchangeRate": 24.505
  }
}

---

Example

BPMN Configuration

A Kafka Task uses a standard Receive Task — a BPMN element that naturally represents "wait for an external message". The Kafka-specific logic (publishing the request and setting up correlation) is attached as an execution listener on the start event. This is a clean fit:

• The Receive Task handles the waiting semantics — the process suspends until a correlated message arrives.
• The execution listener fires when the task starts, publishes the Kafka message, and lets the Receive Task take over.

This separation means the BPMN model stays standard-compliant, while the Kafka integration is handled transparently by the listener delegate.

<bpmn:receiveTask id="kafkaIntegrationCall" name="Call Integration via Kafka"
                  messageRef="Message_KafkaIntegration">
  <bpmn:extensionElements>
    <camunda:executionListener event="start"
                              delegateExpression="#{tsmKafkaTaskExecutor}">
      <camunda:field name="requestTopic" stringValue="tsm-process-request"/>
      <camunda:field name="responseTopic" stringValue="tsm-process-response"/>
      <camunda:field name="key" stringValue="#{execution.processBusinessKey}"/>
      <camunda:field name="payload"
                     stringValue="#{#variables.requestPayload}"/>
    </camunda:executionListener>
  </bpmn:extensionElements>
  <bpmn:incoming>Flow_1</bpmn:incoming>
  <bpmn:outgoing>Flow_2</bpmn:outgoing>
</bpmn:receiveTask>

Consumer Perspective

The consumer receives a Kafka message on tsm-process-request with:

Headers:

correlationId:     d4f7a8b1-3c2e-4f5a-9b8d-1e2f3a4b5c6d
processInstanceId: a1b2c3d4-e5f6-7890-abcd-ef1234567890
ownerType:         Ticket
ownerId:           f8e7d6c5-b4a3-2190-8765-432109876543
traceId:           6a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d

Payload:

{
  "action": "convertCurrency",
  "amount": 100,
  "sourceCurrency": "USD",
  "targetCurrency": "CZK"
}

The consumer processes the request and publishes to tsm-process-response:

Headers:

correlationId: d4f7a8b1-3c2e-4f5a-9b8d-1e2f3a4b5c6d
processInstanceId: a1b2c3d4-e5f6-7890-abcd-ef1234567890
ownerType:         Ticket
ownerId:           f8e7d6c5-b4a3-2190-8765-432109876543
traceId:           6a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d

Payload:

{
  "convertedAmount": 2450.50,
  "exchangeRate": 24.505
}

tSM correlates the response via correlationId and the process continues.

---

Comparison: Kafka Task vs External Task

Aspect	External Task	Kafka Task
Communication	REST (fetchAndLock)	Kafka (publish/subscribe)
Pattern	Synchronous polling	Asynchronous messaging
Worker model	Worker pulls tasks	Consumer receives messages
Durability	Engine-managed	Kafka-managed
Best for	Short-lived operations, REST-native workers	Long-running operations, event-driven systems
BPMN element	Service Task (type: external)	Receive Task (listener: KAFKA)

---

Best Practices

• Use Task Templates whenever available — they ensure the correct topic and payload structure
• Keep message payloads compact and structured
• Always let tSM inject correlation headers — do not override them manually
• Ensure consumers copy the correlationId header to their response messages — treat it as an opaque value
• Override request/response topics only when needed; the defaults (tsm-process-request / tsm-process-response) work for most cases
• Use the ownerType and ownerId headers for consumer-side routing and logging
• Use the traceId header to participate in distributed tracing