Skip to main content

Writing processors

This article describes what the code of a JavaScript Processor can use: the record, the values it can return, the state it can keep, and helpers for headers. It closes with examples you can copy.

The rec object of a JavaScript Processor: key, value and headers can be changed, the other attributes can be read

The record​

The record is available as rec.

AttributeDescriptionCan be changed
rec.keyThe key, decoded with the codec of the viewYes
rec.valueThe value, decoded with the codec of the viewYes
rec.headersThe headers as a list of { key, value }. See Headers.Yes
rec.topicThe topic of the recordNo
rec.partitionThe partitionNo
rec.offsetThe offset in the partitionNo
rec.timestampThe timestamp in milliseconds since 1970No
rec.timestampType.nameThe kind of timestamp, for example CreateTimeNo
rec.keyDecoderThe codec the key was read with, for example StringNo
rec.valueDecoderThe codec the value was read with, for example Json or AvroNo

Changes to attributes that cannot be changed are ignored.

With structured codecs such as JSON, Avro or Protobuf, rec.value is an object. Use the dot notation to reach its attributes, for example rec.value.customer.address.city. With the String codec, rec.value is a text.

Tip: To find the path of an attribute, select a record in the record list and look at its value in the detail view. The attribute names you see there are the names to use after rec.value..

Attributes that are covered by a data protection policy arrive masked. A processor cannot read the original value.

Return values​

Your code must end with a return.

ReturnResult
return rec;The record is shown with the key, value and headers it has now.
return true;The record is shown as it is. Changes you made to rec are not applied.
return false;The record is left out.

Any other return value, or no return at all, is an error.

A condition is a short way to write a filter:

return rec.value.total > 1000;

Make sure a filter returns true or false and not the value of an attribute, which may be missing. return !!rec.value.express; turns any value into true or false.

Changing key and value​

Change rec and return it:

// change an attribute
rec.value.status = rec.value.status.toUpperCase();

// add an attribute
rec.value.gross = rec.value.total * 1.19;

// remove an attribute
delete rec.value.internalNote;

// set a new key
rec.key = rec.value.customerId;

return rec;

You can also replace the whole value:

rec.value = {
id: rec.value.id,
name: rec.value.firstName + " " + rec.value.lastName
};
return rec;

State​

A processor restores its state, updates it and stores it again for every record

The processor is called once per record and forgets its variables between two records. To carry something over, put it into the state:

FunctionDescription
store(data)Saves data as the state. It replaces what was stored.
restore()Returns what was stored.
hasStore()true if something was stored already.

The state can be a number, a text, a list or an object.

// number the records
var count = hasStore() ? restore() : 0;
count = count + 1;
store(count);

rec.value.count = count;
return rec;

What you need to know about the state:

  • It starts empty every time the records are read: with every Fetch, when Live is started, and when a pipe is started or resumed.
  • It is kept per partition while past records are fetched. Kadeck reads the partitions side by side, each with a state of its own. A counter therefore counts per partition. In Live mode and in a Kadeck Pipe, there is one state for all partitions.
  • Keep it small. An object that grows with every record uses more and more memory. Remove what you no longer need.
  • Use the state, not global variables. Variables you declare outside of your code do not exist.

Headers​

rec.headers is a list of headers. Each header has a key and a value; the value is the list of the bytes of the header, written as text, for example "[97, 98, 99]".

FunctionDescription
headerToString(value)Turns the value of a header into text.
headerFromString(text)Turns text into the value of a header.

Read a header

var trace = rec.headers.find(h => h.key === "trace");
rec.value.trace = trace ? headerToString(trace.value) : null;
return rec;

Read a number from a header

Numbers are stored as bytes. Read them with a buffer:

var header = rec.headers.find(h => h.key === "retries");
var bytes = Buffer.from(JSON.parse(header.value));

rec.value.retries = bytes.readInt32BE(0); // 4 bytes
// Number(bytes.readBigInt64BE(0)) for 8 bytes (long)
// bytes.readFloatBE(0), bytes.readDoubleBE(0) for float and double

return rec;

Filter by a header

var source = rec.headers.find(h => h.key === "source");
return !!source && headerToString(source.value) === "crm";

Add or remove a header

rec.headers.push({ key: "checked-by", value: headerFromString("kadeck") });
rec.headers = rec.headers.filter(h => h.key !== "debug");
return rec;

A record keeps at least one header: removing all headers of a record has no effect.

What else is available​

  • Processors are written in modern JavaScript: let and const, arrow functions, template strings, JSON, Math, Date and regular expressions work as usual.
  • btoa(text) and atob(text) encode and decode Base64.
  • console.log() writes to the log of the Kadeck server, not to your browser.

Errors​

When the code fails, the error is shown below the editor.

ErrorCause
SyntaxError: …The code is not valid JavaScript.
TypeError: Cannot read properties of undefined (reading '…')A record does not have the attribute you read from. Check it first: if (rec.value.customer) … or rec.value.customer?.name.
ReferenceError: … is not definedA variable is used that was never declared, for example one from an earlier record. Use the state.
Unexpected error: …The code returned something other than rec, true or false, or nothing at all.

Records are not all alike. Code that works for the first record can fail for a later one with a missing attribute, a null value or a different structure, so check before you read.

Examples​

Filter by a date​

Show the orders that are delivered at a weekend:

var day = new Date(rec.value.deliveryDate).getDay();
return day === 0 || day === 6;

Compare two attributes​

return rec.value.shippedQuantity < rec.value.orderedQuantity;

Flatten a nested value​

var v = rec.value;
rec.value = {
orderId: v.id,
customer: v.customer.name,
city: v.customer.address.city,
items: v.items.length
};
return rec;

Moving average​

var s = hasStore() ? restore() : { sum: 0, n: 0 };
s.sum += rec.value.temperature;
s.n += 1;
store(s);

rec.value.average = s.sum / s.n;
return rec;

Show every key once​

var seen = hasStore() ? restore() : {};
if (seen[rec.key]) return false;

seen[rec.key] = true;
store(seen);
return true;

Show a timestamp as a date​

rec.value.producedAt = new Date(rec.timestamp).toISOString();
return rec;