Skip to main content

Configuration, guarantees and permissions

This article describes the settings of a Kadeck Pipe, what a pipe guarantees about the records it writes, and which rights are needed to work with pipes.

How a pipe processes a record​

The steps a Kadeck Pipe takes for every record: read, decode, mask, process, filter, encode and write

A pipe runs inside Kadeck. For every record of the source topic it takes the same steps the Data Explorer takes to show the record, and then writes the result:

  1. Read the record from the source topic.
  2. Decode key and value with the codecs of the view.
  3. Mask the attributes that are covered by data protection policies.
  4. Process the record with the JavaScript Processor of the view, which can change its key, value and headers or drop it.
  5. Filter by the searches, attribute filters and the end condition of the view.
  6. Encode key and value with the codecs chosen for the target.
  7. Write the record to the target topic.

The position of the pipe is committed only after the records up to it are written and acknowledged by the target cluster.

Settings​

All settings are made in the Convert to Kadeck Pipe dialog. A pipe cannot be changed afterwards: to change a setting, stop the pipe and duplicate it.

Target​

SettingDescription
ClusterThe connection the pipe writes to. It is the cluster of the source topic unless you choose another one.
TopicThe topic the pipe writes to. It must exist, and it cannot be the topic the pipe reads from.

Pipe​

SettingDescription
NameThe name of the pipe. It is shown in lists, in the flow and in the lineage header.
OwnerWho is responsible for the pipe: a person or a team. Free text.
DescriptionWhat the pipe is for.

The user who creates the pipe is its author. The author cannot be changed and decides which data protection policies apply.

Apply​

Entire view applies everything the view does. Selected parts lets you choose:

PartWhat the pipe takes from the viewWithout it
JavaScript processorThe processor of the view.Records are written as they are read.
FiltersSearches in key and value and attribute filters that are switched on.No record is filtered out.
Offsets and partitionsThe partitions the view reads and the offsets it starts at.All partitions are read.
End conditionThe maximum number of results per partition, the time window or the time range.The pipe reads to the end of the topic.

Important: A view always has a maximum number of results per partition, 50 by default. With Entire view, or with End condition selected, a pipe writes at most that many records per partition, counted after the filters. Switch End condition off to process a whole topic.

If the selected parts do not say where the pipe starts, Start at appears:

  • Earliest: from the beginning of the topic.
  • Latest: only records that arrive from now on. Available for pipes that keep running.

Run​

ModeStarts atEnds
Run onceWhere the applied parts of the view start.At the end condition, or at the end of the topic as it was when the pipe started.
Keep runningWhere the applied parts of the view start.When a user stops it. The end condition only decides where the pipe starts.

A time window that is relative to now, such as the last hour, is fixed at the moment the pipe starts.

Write as​

SettingDescription
Key codec, Value codecThe codecs the records are written with. They are filled in with the codecs of the view, as far as these can write. None writes no key.
Schema idFor codecs with schemas: the id of the schema to write with. With -1, the codec chooses, as it does when you produce a record in the Explorer.
Message typeFor Protobuf: the fully qualified name of the message type.
Schema contextThe context of the Schema Registry to use.

Schemas are looked up in the Schema Registry of the target cluster. Source and target can use different registries.

A key or value that the view did not change is written byte for byte as it was read, as long as source and target are on the same cluster, the codec stays the same and no schema is named. Everything else, including every key or value the JavaScript Processor changed, is encoded with the codecs chosen here.

On error​

A record can fail because it cannot be decoded, because the processor throws an error or does not finish, because the result does not fit the codec or schema of the target, or because the target cluster rejects it, for example as too large. Such a record does not stop the pipe:

  • Drop leaves the record out and counts it as Dropped.
  • Dead letter topic writes the record, as it was read, to a topic of your choice and counts it as Dead-lettered. The topic can be on any cluster. A record the target cluster rejected is written the way the pipe tried to write it.

A record in the dead letter topic carries these headers:

HeaderContent
kadeck.pipeThe name of the pipe
kadeck.pipe.error.stageWhere it failed: DECODE, PROCESS, ENCODE or PRODUCE
kadeck.pipe.error.messageWhat went wrong
kadeck.pipe.source.topicThe topic the record was read from
kadeck.pipe.source.partitionIts partition
kadeck.pipe.source.offsetIts offset

The last record that failed is shown in the details of the pipe.

The JavaScript Processor of a pipe may work on a single record for up to 5 seconds. A processor that takes longer is interrupted, and the record is treated as failed.

Governance​

SettingDescription
Data maskingApplies the data protection policies of the author to the records before they are processed and written. See Data protection.
Lineage headerAdds the header kadeck.pipe with the name of the pipe to every record, so that consumers can tell where a record came from.

Alerts​

A pipe can notify one or more alert integrations:

  • when the pipe fails,
  • when the pipe is paused,
  • when its lag exceeds a number of records.

The cooldown sets how long Kadeck waits before it sends the lag alert again.

Guarantees​

Delivery​

After a restart a pipe goes on from its last committed position, so records since then are written a second time and none is left out

A pipe delivers at least once. It commits its position only after the records up to that position are written, about once per second. When a pipe is paused, fails, or Kadeck is restarted, it goes on from the last committed position:

  • No record is left out.
  • Records can be written twice: the records between the last commit and an unexpected stop are written again. A pipe that is paused or stopped by a user commits first and does not repeat records.

If consumers of the target topic cannot handle duplicates, make the records identifiable, for example by a key or by an id in the value.

Order​

Records of one source partition are processed in their order. They are distributed over the partitions of the target topic by the key they are written with, so records with the same key keep their order. A key the JavaScript Processor changes therefore also changes the partition a record goes to. The target topic may have a different number of partitions than the source. Records without a key are distributed evenly.

What is kept​

Part of the recordWritten to the target
KeyAs the view shows it. A key the JavaScript Processor sets is the key of the written record; null writes the record without a key.
ValueAs the view shows it, with the changes of the JavaScript Processor.
HeadersAll headers, including repeated ones, unless the processor or data masking changes them.
TimestampThe timestamp of the source record.
PartitionChosen by the key that is written. The partition of the source record is not kept.

Changing the key​

The record a JavaScript Processor returns is the record the pipe writes, including its key:

rec.key = "New Key";
return rec;
  • The new key is encoded with the key codec of the pipe and decides the partition in the target topic.
  • A key the processor does not touch is written as it was read.
  • rec.key = null writes the record without a key.

A key or value that was not changed is written byte for byte as it was read where source and target are on the same cluster and use the same codec. This only concerns unchanged keys and values: a changed one is always encoded anew.

Restarts and failures​

  • Kadeck resumes all running pipes after a restart. No action is needed.
  • In a deployment with several Kadeck nodes, every pipe runs on exactly one node. If that node is no longer available, another node takes the pipe over after about 30 seconds.
  • A pipe that cannot go on, for example because a cluster is not reachable or a topic was deleted, changes to Failed and shows the reason. Fix the cause and Resume the pipe.

Position​

A pipe commits its position to the consumer group kadeck-pipe-<id> on the source cluster. The group is listed under Consumers and is deleted together with the pipe.

Important: The user Kadeck connects with needs the right to commit offsets for consumer groups with the prefix kadeck-pipe- on the source cluster, and the right to write to the target topic on the target cluster. Without them the pipe fails when it starts.

Data protection​

Data protection policies decide which attributes a user sees masked. A pipe must not become a way around them: it applies the policies of its author and writes the masked values.

  • By default, the system setting Always apply the author's data masking and protection policies to pipes under Administration → Settings is switched on. The Data masking setting of a pipe is then always on.
  • If an administrator switches the setting off, the author decides per pipe.
  • The policies are evaluated while the pipe runs. A change to a policy, or to the groups of the author, takes effect within a minute.
  • A policy that prohibits a codec for a topic also prohibits it for a pipe: the pipe fails with the reason.
  • When masking applies, records in the dead letter topic are masked as well. A record that could not be decoded, and therefore not masked, is written there without its key and value.
  • Every pipe that reads a topic covered by a policy is recorded in the audit log.

Important: A pipe keeps running when its author later loses access to the source or target topic, or is removed. Review the pipes of a user when you change their rights, and stop the pipes that should no longer run.

Permissions​

The rights needed for a Kadeck Pipe: read the source topic, write the target and dead letter topics, and view the Schema Registry when a schema is named

Pipes use the existing topic rights. There is no separate right for pipes.

ActionRequired rights
See a pipe and its figuresTopicAccessRead for the source topic, or TopicAccessRead or TopicAccessWrite for the target topic
Pause, resume, stop, delete, duplicateTopicAccessRead for the source topic and TopicAccessWrite for the target topic
Create a pipe, test runAs above, plus TopicAccessWrite for the dead letter topic if one is used
Name a schema idSchemaRegistryView for the cluster whose Schema Registry holds the schema

Rights are checked when a user creates or manages a pipe. The pipe itself runs with the connection of the cluster, not with the rights of a user.

A user who can see a pipe can open its details, including the JavaScript Processor and the filters of its view. The records of a topic are only shown to users with TopicAccessRead for that topic.

Limits​

  • One active pipe at a time in Kadeck Platform Free. Stopped and completed pipes do not count.
  • A pipe reads one topic and writes to one topic.
  • Apache Kafka only.