Idempotency

In the me-delivery we have an idempotency part which shall provide duplicate sends. Actually we allow to use 2 different strategies for idempotency:

  • "At Most Once" (default): We send either 1 or no message at all to a client. This causes issues when the idempotency is set at the beginning of the processing of a message and the worker is then shut down before it can send the message or report the success/failure of a sending.

  • "At Least Once": This new approach theoretically allows that a push notification is sent multiple times to the same client. So in case that

    • the last idempotency step (sealing) cannot be executed AND

    • a duplicate will be received for processing, the duplicate message will be sent to a client.

At Most Once Strategy

With this strategy we ensure that never more than one message is sent to a client. The disadvantage is, that it can happen that a message is never sent, because a worker is interrupted after it registered the key but before it was able to send the message to the client. In such a case there is also no entry in sent/not sent tables in data platform and we do not really know about the success of a delivery.

Enabling & Configuration

  • TTL_IDEMPOTENCY_PER_CUSTOMER_SEC: The time in seconds a key remains in Redis per customer. If no customer related entry is available, then the "default" will be applied which is programmatically set to 6 hours.

At Least Once Strategy

As mentioned above, this strategy should mitigate issues which cause that we do not send to a contact at all, just because a worker (pod) got restarted between the setting of the idempotency key and the execution of the sending itself. In very rare cases it can happen that we send one duplicate if the last step (the sealing of a key) cannot be executed because of a restart in connection with a duplicate delivery of the same message.

States of a Redis Key

A Redis key can have 3 different states in case of the "at least once" strategy

  • "Unlocked": The key is not present at all in Redis

  • "Locked": They key is present in Redis and has assigned the value '0' (Locked) and a TTL of TTL_LOCK_TIME_LIMIT_MILLIS which is 2 minutes by default. If there is no update on the key within the lock time, then the key will be removed by Redis and so it is "Unlocked" again. If a transient exception occurs during processing of a message the key will be "unlocked" which means it will be programmatically removed from Redis. If any "non transient" issue occurs, then the "Locked" key will be marked as "Sealed".

  • "Sealed": They key is present in Redis and has assigned the value '1' (Sealed) and a TTL of either TTL_IDEMPOTENCY_PER_CUSTOMER_SEC or 6 hours if no customer related entry can be found. A key is "sealed" after the sending was completely processed by updating the value and TTL. If a key is "sealed" and there is the attempt to "lock" it, the attempt will be reported as "filtered by idempotency". If the TTL of a "sealed" key expires it will be removed by Redis and so again be "Unlocked".

Enabling & Configuration

There are 3 configuration settings which are relevant for the "at least once" strategy

  • AT_LEAST_ONCE_SENDING_CUSTOMERS: A flipper which contains a comma separated list of customer IDs or "*" if it is enabled for all customers. If enabled then the strategy will be applied.

  • TTL_LOCK_TIME_LIMIT_MS: The time a specific idempotency key stays "locked" before it is removed by Redis from its DB.

  • TTL_IDEMPOTENCY_PER_CUSTOMER_SEC: The time a sealed key stays in Redis till it is removed. This is an object with the customer ID as the key and the TTL (in seconds) as the value. If there is no setting for a specific customer available then the default is used, which is actually 6 hours.