Skip to main content

Create Pipe

POST /public/v1/pipes​

Creates a pipe and starts it. The pipe reads the source topic with a topic view and writes the result to the target topic.

The topic view is either a view saved in Kadeck (viewId) or a view configuration sent with the request (view).


Request​

Body Parameters​

FieldTypeDescriptionRequired
namestringName of the pipe, up to 255 charactersYes
descriptionstringWhat the pipe is forOptional
ownerstringWho is responsible for the pipe. Defaults to the user of the API key.Optional
modestringONCE or CONTINUOUS. Defaults to ONCE.Optional
sourceobject (Source)Where the pipe reads and with which topic viewYes
targetobject (Target)Where the pipe writesYes
configurationobject (PipeConfiguration)The settings of the pipeYes

Source Structure​

FieldTypeDescriptionRequired
serverIdintegerID of the connection to read fromYes
topicstringTopic to read fromYes, unless viewId
viewIdintegerID of a saved topic view. The topic is taken from the view. The view must be visible to the API key.Either this or view
viewobject (View)A topic view sent with the requestEither this or viewId

The id of a saved view is the last part of its link in the Data Explorer: /topicbrowser/<connection>/<topic>/<viewId>.

View Structure​

FieldTypeDescriptionRequired
namestringName shown for the view of the pipeOptional
configurationobjectThe configuration of the view, in the format Kadeck saves views in. An empty object reads the topic as it is.Yes

The most relevant fields of a view configuration:

FieldTypeDescription
KeyDecoderstringCodec to read keys with, for example String. Detected automatically when omitted.
ValueDecoderstringCodec to read values with, for example Json or Avro. Detected automatically when omitted.
QuickProcessorobjectThe JavaScript Processor: id, name, description, value, code, active and type (js). All fields are required. code is a function named execute that returns the record rec, or false to drop it. The key, value and headers of the returned record are what the pipe writes.
KVFiltersarrayAttribute filters: id, key, value, operator (as in the Data Explorer, for example =, !=, >, <), isKey, active
FilterobjectSearch in key and value: key, value, key_active, value_active
partitionoffsetsobjectPer partition: partition, active, offset. Without an active partition all partitions are read.
limitintegerMaximum number of results per partition. 50 when omitted.
limitTimeWindowintegerTime window in seconds, counted back from the start of the pipe
referenceTimeStartlongStart of a time range, Unix timestamp in milliseconds
referenceTimeEndlongEnd of a time range, Unix timestamp in milliseconds

Target Structure​

FieldTypeDescriptionRequired
serverIdintegerID of the connection to write toYes
topicstringTopic to write to. It must exist and cannot be the topic the pipe reads from.Yes

PipeConfiguration Structure​

FieldTypeDescriptionRequired
targetobject (Codecs)The codecs the records are written withYes
partsobject (Parts)The parts of the view the pipe applies. All parts when omitted.Optional
startWhenNoOffsetsstringearliest or latest: where the pipe starts if the applied parts do not say. Defaults to earliest.Optional
applyDataProtectionbooleanApply the data protection policies of the author. Defaults to true; always true if the system setting enforces it.Optional
lineageHeaderbooleanAdd the header kadeck.pipe with the name of the pipe to every record. Defaults to false.Optional
errorModestringDROP or DLQ: what happens to records that fail. Defaults to DROP.Optional
dlqobject (Dlq)The dead letter topic. Required with errorMode DLQ.Optional
alertsobject (Alerts)Who is notified about the pipeOptional

Codecs Structure​

FieldTypeDescriptionRequired
valueCodecstringCodec to write values with, for example Json, Avro or StringYes
keyCodecstringCodec to write keys with. Without it, records are written without a key.Optional
valueCodecConfigobjectSettings of the value codec: schemaId, messageType, schemaContext, as stringsOptional
keyCodecConfigobjectSettings of the key codecOptional

Naming a schemaId needs the right SchemaRegistryView for the target connection.

Parts Structure​

FieldTypeDescriptionDefault
processorbooleanApply the JavaScript Processor of the viewtrue
filtersbooleanApply the searches and attribute filters of the viewtrue
offsetsbooleanRead the partitions of the view, from its offsetstrue
endConditionbooleanApply the limit, time window or time range of the viewtrue

Dlq Structure​

FieldTypeDescriptionRequired
serverIdintegerID of the connection of the dead letter topicYes
topicstringTopic for records that fail. It cannot be the topic the pipe reads from.Yes

Alerts Structure​

FieldTypeDescriptionDefault
channelsstring[]IDs of the alert integrations to notifynone
onFailedbooleanNotify when the pipe failsfalse
onPausedbooleanNotify when the pipe is pausedfalse
lagThresholdlongNotify when the lag exceeds this number of records. null for never.null
cooldownMinutesintegerMinutes to wait before the lag alert is sent again15

Example — From a Saved View​

Writes what the saved view 42 shows to another topic, once.

curl --location 'https://your-instance.com/public/v1/pipes' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <your-api-key>' \
--data-raw '{
"name": "Fix account ids",
"owner": "Payments team",
"mode": "ONCE",
"source": { "serverId": 1, "viewId": 42 },
"target": { "serverId": 1, "topic": "payments" },
"configuration": {
"target": { "keyCodec": "String", "valueCodec": "Json" },
"parts": { "processor": true, "filters": true, "offsets": true, "endCondition": false },
"lineageHeader": true
}
}'

Example — With a View in the Request​

Keeps writing the orders above a value to a topic of their own, and routes records that fail to a dead letter topic.

curl --location 'https://your-instance.com/public/v1/pipes' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <your-api-key>' \
--data-raw '{
"name": "High value orders",
"mode": "CONTINUOUS",
"source": {
"serverId": 1,
"topic": "orders",
"view": {
"name": "High value",
"configuration": {
"KeyDecoder": "String",
"ValueDecoder": "Json",
"KVFilters": [
{ "id": "f1", "key": "total", "value": "10000", "operator": ">", "isKey": false, "active": true }
],
"QuickProcessor": {
"id": "p1",
"name": "Flag",
"description": "",
"value": "",
"code": "function execute(){\nrec.value.highValue = true;\nreturn rec;\n};",
"active": true,
"type": "js"
}
}
}
},
"target": { "serverId": 2, "topic": "orders.high-value" },
"configuration": {
"target": { "keyCodec": "String", "valueCodec": "Json" },
"parts": { "processor": true, "filters": true, "offsets": false, "endCondition": false },
"startWhenNoOffsets": "latest",
"errorMode": "DLQ",
"dlq": { "serverId": 1, "topic": "orders.pipe-errors" },
"alerts": { "channels": ["8c1f7b0e-5a55-4a3e-9d57-0a4f0f0c9b21"], "onFailed": true, "lagThreshold": 10000 }
}
}'

Response​

HTTP Status Code: 200 OK
Response Body: the created pipe object with the state RUNNING.

Keep the id of the pipe: it is needed to read its status and to control it.


Possible Errors​

Status CodeDescriptionSolution
400A required field is missing, source and target are the same topic, or a connection is not a Kafka connectionCheck the request body
401UnauthorizedCheck your API key
403The API key lacks a right for the source, target or dead letter topic or the Schema Registry, or the limit of active pipes of the edition is reachedCheck the rights of the API key
404A connection or the saved view does not exist or is not visible to the API keyCheck the IDs

A pipe that is created but cannot run, for example because the target topic does not exist, is answered with 200 and changes to the state FAILED shortly after. Use Test Pipe to find such problems before you create the pipe.


Best Practices​

  • Send the same body to Test Pipe first.
  • Set endCondition to false to process a whole topic. Otherwise the pipe writes at most limit records per partition, 50 by default.
  • Use errorMode DLQ for pipes that keep running, so that no record is dropped unnoticed.
  • Records are delivered at least once. Make sure the consumers of the target topic can handle a record that arrives twice.