Create Pipe
POST /public/v1/pipes
Creates a pipe and starts it. The pipe reads the source topic with a topic view and writes the result to the target topic.
The topic view is either a view saved in Kadeck (viewId) or a view configuration sent with the request (view).
Request
Body Parameters
| Field | Type | Description | Required |
|---|---|---|---|
name | string | Name of the pipe, up to 255 characters | Yes |
description | string | What the pipe is for | Optional |
owner | string | Who is responsible for the pipe. Defaults to the user of the API key. | Optional |
mode | string | ONCE or CONTINUOUS. Defaults to ONCE. | Optional |
source | object (Source) | Where the pipe reads and with which topic view | Yes |
target | object (Target) | Where the pipe writes | Yes |
configuration | object (PipeConfiguration) | The settings of the pipe | Yes |
Source Structure
| Field | Type | Description | Required |
|---|---|---|---|
serverId | integer | ID of the connection to read from | Yes |
topic | string | Topic to read from | Yes, unless viewId |
viewId | integer | ID of a saved topic view. The topic is taken from the view. The view must be visible to the API key. | Either this or view |
view | object (View) | A topic view sent with the request | Either this or viewId |
The id of a saved view is the last part of its link in the Data Explorer: /topicbrowser/<connection>/<topic>/<viewId>.
View Structure
| Field | Type | Description | Required |
|---|---|---|---|
name | string | Name shown for the view of the pipe | Optional |
configuration | object | The configuration of the view, in the format Kadeck saves views in. An empty object reads the topic as it is. | Yes |
The most relevant fields of a view configuration:
| Field | Type | Description |
|---|---|---|
KeyDecoder | string | Codec to read keys with, for example String. Detected automatically when omitted. |
ValueDecoder | string | Codec to read values with, for example Json or Avro. Detected automatically when omitted. |
QuickProcessor | object | The JavaScript Processor: id, name, description, value, code, active and type (js). All fields are required. code is a function named execute that returns the record rec, or false to drop it. The key, value and headers of the returned record are what the pipe writes. |
KVFilters | array | Attribute filters: id, key, value, operator (as in the Data Explorer, for example =, !=, >, <), isKey, active |
Filter | object | Search in key and value: key, value, key_active, value_active |
partitionoffsets | object | Per partition: partition, active, offset. Without an active partition all partitions are read. |
limit | integer | Maximum number of results per partition. 50 when omitted. |
limitTimeWindow | integer | Time window in seconds, counted back from the start of the pipe |
referenceTimeStart | long | Start of a time range, Unix timestamp in milliseconds |
referenceTimeEnd | long | End of a time range, Unix timestamp in milliseconds |
Target Structure
| Field | Type | Description | Required |
|---|---|---|---|
serverId | integer | ID of the connection to write to | Yes |
topic | string | Topic to write to. It must exist and cannot be the topic the pipe reads from. | Yes |
PipeConfiguration Structure
| Field | Type | Description | Required |
|---|---|---|---|
target | object (Codecs) | The codecs the records are written with | Yes |
parts | object (Parts) | The parts of the view the pipe applies. All parts when omitted. | Optional |
startWhenNoOffsets | string | earliest or latest: where the pipe starts if the applied parts do not say. Defaults to earliest. | Optional |
applyDataProtection | boolean | Apply the data protection policies of the author. Defaults to true; always true if the system setting enforces it. | Optional |
lineageHeader | boolean | Add the header kadeck.pipe with the name of the pipe to every record. Defaults to false. | Optional |
errorMode | string | DROP or DLQ: what happens to records that fail. Defaults to DROP. | Optional |
dlq | object (Dlq) | The dead letter topic. Required with errorMode DLQ. | Optional |
alerts | object (Alerts) | Who is notified about the pipe | Optional |
Codecs Structure
| Field | Type | Description | Required |
|---|---|---|---|
valueCodec | string | Codec to write values with, for example Json, Avro or String | Yes |
keyCodec | string | Codec to write keys with. Without it, records are written without a key. | Optional |
valueCodecConfig | object | Settings of the value codec: schemaId, messageType, schemaContext, as strings | Optional |
keyCodecConfig | object | Settings of the key codec | Optional |
Naming a schemaId needs the right SchemaRegistryView for the target connection.
Parts Structure
| Field | Type | Description | Default |
|---|---|---|---|
processor | boolean | Apply the JavaScript Processor of the view | true |
filters | boolean | Apply the searches and attribute filters of the view | true |
offsets | boolean | Read the partitions of the view, from its offsets | true |
endCondition | boolean | Apply the limit, time window or time range of the view | true |
Dlq Structure
| Field | Type | Description | Required |
|---|---|---|---|
serverId | integer | ID of the connection of the dead letter topic | Yes |
topic | string | Topic for records that fail. It cannot be the topic the pipe reads from. | Yes |
Alerts Structure
| Field | Type | Description | Default |
|---|---|---|---|
channels | string[] | IDs of the alert integrations to notify | none |
onFailed | boolean | Notify when the pipe fails | false |
onPaused | boolean | Notify when the pipe is paused | false |
lagThreshold | long | Notify when the lag exceeds this number of records. null for never. | null |
cooldownMinutes | integer | Minutes to wait before the lag alert is sent again | 15 |
Example — From a Saved View
Writes what the saved view 42 shows to another topic, once.
curl --location 'https://your-instance.com/public/v1/pipes' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <your-api-key>' \
--data-raw '{
"name": "Fix account ids",
"owner": "Payments team",
"mode": "ONCE",
"source": { "serverId": 1, "viewId": 42 },
"target": { "serverId": 1, "topic": "payments" },
"configuration": {
"target": { "keyCodec": "String", "valueCodec": "Json" },
"parts": { "processor": true, "filters": true, "offsets": true, "endCondition": false },
"lineageHeader": true
}
}'
Example — With a View in the Request
Keeps writing the orders above a value to a topic of their own, and routes records that fail to a dead letter topic.
curl --location 'https://your-instance.com/public/v1/pipes' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <your-api-key>' \
--data-raw '{
"name": "High value orders",
"mode": "CONTINUOUS",
"source": {
"serverId": 1,
"topic": "orders",
"view": {
"name": "High value",
"configuration": {
"KeyDecoder": "String",
"ValueDecoder": "Json",
"KVFilters": [
{ "id": "f1", "key": "total", "value": "10000", "operator": ">", "isKey": false, "active": true }
],
"QuickProcessor": {
"id": "p1",
"name": "Flag",
"description": "",
"value": "",
"code": "function execute(){\nrec.value.highValue = true;\nreturn rec;\n};",
"active": true,
"type": "js"
}
}
}
},
"target": { "serverId": 2, "topic": "orders.high-value" },
"configuration": {
"target": { "keyCodec": "String", "valueCodec": "Json" },
"parts": { "processor": true, "filters": true, "offsets": false, "endCondition": false },
"startWhenNoOffsets": "latest",
"errorMode": "DLQ",
"dlq": { "serverId": 1, "topic": "orders.pipe-errors" },
"alerts": { "channels": ["8c1f7b0e-5a55-4a3e-9d57-0a4f0f0c9b21"], "onFailed": true, "lagThreshold": 10000 }
}
}'
Response
HTTP Status Code: 200 OK
Response Body: the created pipe object with the state RUNNING.
Keep the id of the pipe: it is needed to read its status and to control it.
Possible Errors
| Status Code | Description | Solution |
|---|---|---|
| 400 | A required field is missing, source and target are the same topic, or a connection is not a Kafka connection | Check the request body |
| 401 | Unauthorized | Check your API key |
| 403 | The API key lacks a right for the source, target or dead letter topic or the Schema Registry, or the limit of active pipes of the edition is reached | Check the rights of the API key |
| 404 | A connection or the saved view does not exist or is not visible to the API key | Check the IDs |
A pipe that is created but cannot run, for example because the target topic does not exist, is answered with 200 and changes to the state FAILED shortly after. Use Test Pipe to find such problems before you create the pipe.
Best Practices
- Send the same body to Test Pipe first.
- Set
endConditiontofalseto process a whole topic. Otherwise the pipe writes at mostlimitrecords per partition,50by default. - Use
errorModeDLQfor pipes that keep running, so that no record is dropped unnoticed. - Records are delivered at least once. Make sure the consumers of the target topic can handle a record that arrives twice.