Overview
The Pipes API creates, controls and monitors Kadeck Pipes: jobs that write the records of a source topic, processed the way a topic view shows them, to a target topic.
Use it to start pipes from a deployment pipeline, to pause and resume them around a maintenance window, or to read their status into your own monitoring.
Endpoints
| Method | Endpoint | Description |
|---|---|---|
| GET | /public/v1/pipes | List pipes and their status |
| GET | /public/v1/pipes/{id} | Get one pipe, including its topic view |
| POST | /public/v1/pipes | Create a pipe and start it |
| POST | /public/v1/pipes/test | Test a pipe with one record without writing |
| POST | /public/v1/pipes/{id}/pause | Pause a running pipe |
| POST | /public/v1/pipes/{id}/resume | Resume a paused or failed pipe |
| POST | /public/v1/pipes/{id}/stop | Stop a pipe for good |
| DELETE | /public/v1/pipes/{id} | Delete a pipe |
Permissions
The API key decides what a request may do. It needs the same rights as a user in the Kadeck UI:
| Action | Required rights |
|---|---|
| List and get pipes | TopicAccessRead for the source topic, or TopicAccessRead or TopicAccessWrite for the target topic |
| Create, test | TopicAccessRead for the source topic and TopicAccessWrite for the target topic and, if used, the dead letter topic |
| Pause, resume, stop, delete | TopicAccessRead for the source topic and TopicAccessWrite for the target topic |
Pipes the key may not see are left out of lists and answered with 404.
The user of the API key becomes the author of the pipes it creates. The data protection policies of the author apply to the records a pipe writes.
The Pipe Object
Every endpoint that returns a pipe returns this object.
| Field | Type | Description |
|---|---|---|
id | string (UUID) | Identifier of the pipe |
name | string | Name of the pipe |
description | string | Description |
author | string | User id of the creator |
owner | string | Who is responsible for the pipe |
sourceServer | integer | ID of the connection the pipe reads from |
sourceServerName | string | Name of that connection |
sourceTopic | string | Topic the pipe reads from |
targetServer | integer | ID of the connection the pipe writes to |
targetServerName | string | Name of that connection |
targetTopic | string | Topic the pipe writes to |
mode | string | ONCE or CONTINUOUS |
state | string | RUNNING, PAUSED, FAILED, STOPPED or COMPLETED |
stateDetail | string | Why the pipe is in its state: who paused or stopped it, or the reason of a failure followed by its stack trace |
consumerGroup | string | Consumer group on the source cluster the pipe commits its position to |
configuration | object (PipeConfiguration) | The settings of the pipe, see Create Pipe |
runtime | object (PipeRuntime) | Figures of the pipe |
canManage | boolean | Whether the API key may pause, resume, stop and delete the pipe |
createdAt | long | Unix timestamp in milliseconds |
startedAt | long | When the pipe was last started or resumed |
updatedAt | long | When the state last changed |
finishedAt | long | When the pipe was stopped or completed, 0 otherwise |
view | object | The topic view the pipe applies: name and configuration. Only returned by Get a pipe, create and control requests |
PipeRuntime Structure
| Field | Type | Description |
|---|---|---|
read | long | Records read from the source topic |
written | long | Records written to the target topic |
filtered | long | Records the view filtered out |
dropped | long | Records that failed and were dropped |
deadLettered | long | Records that failed and were written to the dead letter topic |
throughput | number | Records read per second |
lag | long | Records the pipe still has to read, -1 if unknown. Only up to date while the pipe is running |
range | object | Per partition: start and end offset of what the pipe reads. end is -1 for a pipe that keeps running |
positions | object | Per partition: the next offset the pipe reads |
lastError | object | The last record that failed: stage, message, partition, offset, at |
States
| State | Meaning | Possible requests |
|---|---|---|
RUNNING | The pipe reads, processes and writes | pause, stop, delete |
PAUSED | Paused by a user; keeps its position | resume, stop, delete |
FAILED | Cannot go on; keeps its position. stateDetail holds the reason | resume, stop, delete |
COMPLETED | A pipe with mode ONCE reached its end | delete |
STOPPED | Ended by a user | delete |
Errors
Errors are answered with a plain text body that names the problem, for example:
BadRequest: a pipe cannot write to the topic it reads from
| Status Code | Description |
|---|---|
| 400 | The request is not valid, or not possible in the current state of the pipe |
| 401 | The API key is missing or not valid |
| 403 | The API key lacks a right, or the limit of the edition is reached |
| 404 | The pipe, a connection or the topic view does not exist or is not visible to the API key |
Important Notes
- A pipe cannot be changed. To change it, stop it and create a new one.
- A pipe starts as soon as it is created.
- Records are delivered at least once. See Configuration, guarantees and permissions.