Skip to main content

Overview

The Pipes API creates, controls and monitors Kadeck Pipes: jobs that write the records of a source topic, processed the way a topic view shows them, to a target topic.

Use it to start pipes from a deployment pipeline, to pause and resume them around a maintenance window, or to read their status into your own monitoring.


Endpoints​

MethodEndpointDescription
GET/public/v1/pipesList pipes and their status
GET/public/v1/pipes/{id}Get one pipe, including its topic view
POST/public/v1/pipesCreate a pipe and start it
POST/public/v1/pipes/testTest a pipe with one record without writing
POST/public/v1/pipes/{id}/pausePause a running pipe
POST/public/v1/pipes/{id}/resumeResume a paused or failed pipe
POST/public/v1/pipes/{id}/stopStop a pipe for good
DELETE/public/v1/pipes/{id}Delete a pipe

Permissions​

The API key decides what a request may do. It needs the same rights as a user in the Kadeck UI:

ActionRequired rights
List and get pipesTopicAccessRead for the source topic, or TopicAccessRead or TopicAccessWrite for the target topic
Create, testTopicAccessRead for the source topic and TopicAccessWrite for the target topic and, if used, the dead letter topic
Pause, resume, stop, deleteTopicAccessRead for the source topic and TopicAccessWrite for the target topic

Pipes the key may not see are left out of lists and answered with 404.

The user of the API key becomes the author of the pipes it creates. The data protection policies of the author apply to the records a pipe writes.


The Pipe Object​

Every endpoint that returns a pipe returns this object.

FieldTypeDescription
idstring (UUID)Identifier of the pipe
namestringName of the pipe
descriptionstringDescription
authorstringUser id of the creator
ownerstringWho is responsible for the pipe
sourceServerintegerID of the connection the pipe reads from
sourceServerNamestringName of that connection
sourceTopicstringTopic the pipe reads from
targetServerintegerID of the connection the pipe writes to
targetServerNamestringName of that connection
targetTopicstringTopic the pipe writes to
modestringONCE or CONTINUOUS
statestringRUNNING, PAUSED, FAILED, STOPPED or COMPLETED
stateDetailstringWhy the pipe is in its state: who paused or stopped it, or the reason of a failure followed by its stack trace
consumerGroupstringConsumer group on the source cluster the pipe commits its position to
configurationobject (PipeConfiguration)The settings of the pipe, see Create Pipe
runtimeobject (PipeRuntime)Figures of the pipe
canManagebooleanWhether the API key may pause, resume, stop and delete the pipe
createdAtlongUnix timestamp in milliseconds
startedAtlongWhen the pipe was last started or resumed
updatedAtlongWhen the state last changed
finishedAtlongWhen the pipe was stopped or completed, 0 otherwise
viewobjectThe topic view the pipe applies: name and configuration. Only returned by Get a pipe, create and control requests

PipeRuntime Structure​

FieldTypeDescription
readlongRecords read from the source topic
writtenlongRecords written to the target topic
filteredlongRecords the view filtered out
droppedlongRecords that failed and were dropped
deadLetteredlongRecords that failed and were written to the dead letter topic
throughputnumberRecords read per second
laglongRecords the pipe still has to read, -1 if unknown. Only up to date while the pipe is running
rangeobjectPer partition: start and end offset of what the pipe reads. end is -1 for a pipe that keeps running
positionsobjectPer partition: the next offset the pipe reads
lastErrorobjectThe last record that failed: stage, message, partition, offset, at

States​

StateMeaningPossible requests
RUNNINGThe pipe reads, processes and writespause, stop, delete
PAUSEDPaused by a user; keeps its positionresume, stop, delete
FAILEDCannot go on; keeps its position. stateDetail holds the reasonresume, stop, delete
COMPLETEDA pipe with mode ONCE reached its enddelete
STOPPEDEnded by a userdelete

Errors​

Errors are answered with a plain text body that names the problem, for example:

BadRequest: a pipe cannot write to the topic it reads from
Status CodeDescription
400The request is not valid, or not possible in the current state of the pipe
401The API key is missing or not valid
403The API key lacks a right, or the limit of the edition is reached
404The pipe, a connection or the topic view does not exist or is not visible to the API key

Important Notes​