Task Creator Worker

The Task Creator is a simple Pub/Sub consumer that consumes from a configurable subscription. It checks the validity of the received message and then assembles a cloud task which it then forwards to the Google Cloud Tasks Service.

If there are any issues with the received messages, the message will be published to a configurable error topic.

Artefacts

The service

generates logs in laas-kibana with the name:"me-pubsub-delayer"
sends a task with a given delay to the Google Cloud Tasks Service
publishes to the configured error topic in case of errors.

The service receives messages to be delayed via a Pub/Sub subscription. The message which shall be delayed must contain the following attributes:

me-pubsub-delayer/delay: A string indicating the desired delay. For example: 1d12h, 12000ms, 500s, 1m32s
me-pubsub-delayer/topic: The topic which shall be used to republish the contained message after the delay.

Queue Configuration

The service supports flexible queue routing through a YAML configuration file. This allows different topics to be routed to different Cloud Task queues based on pattern matching.

Configuration File

The queue mapping configuration is provided via a YAML file that must be mounted as a ConfigMap in Kubernetes.

Default location: /etc/pubsub-delayer/config/queue-mapping.yaml

Configuration Format

# Default queue for topics without specific mapping
default: pubsub-delayer

# Topic-to-queue mappings (evaluated in order, first match wins)
mappings:
  # Exact match
  client-changes: pubsub-delayer-client-changes

  # Pattern matching with wildcards
  sending-*: pubsub-delayer-sending-chain
  *-dg02: pubsub-delayer-dg02

Pattern Matching Rules

The service uses glob-style pattern matching to route messages:

Exact match (highest priority): client-changes matches exactly client-changes
Wildcard suffix: analytics-* matches analytics-events, analytics-logs, etc.
Wildcard prefix: *-events matches user-events, system-events, etc.
Wildcard middle: client-*-prod matches client-manager-prod, client-api-prod, etc.
Default fallback: If no pattern matches, the default queue is used

First match wins: Patterns are evaluated in the order they appear. The first matching pattern determines the queue.

Queue Validation

At startup, the service validates that all queues referenced in the configuration exist in Google Cloud Tasks. If any queue is missing, the service will fail to start with an error message.

Validation timeout: 5 seconds per queue

Environment Variables

Variable Description

Variable	Description
`GCLOUD_PROJECT_ID`	The ID of the GCLOUD project (mandatory)
`GCLOUD_PRIVATE_KEY`	The GCLOUD private key encoded in Base64 which belongs to the used service account (mandatory)
`GCLOUD_CLIENT_EMAIL`	The email which is used to identify the service account (mandatory).
`SERVICE_ACCOUNT_EMAIL`	The email address of the service account used for Cloud Tasks authentication (mandatory).
`ERROR_TOPIC`	The pub/sub topic which shall be used to publish erroneous messages. If not specified the default `pubsub-delayer-errors` is used.
`DELAYED_MESSAGE_SUBSCRIPTION`	The subscription which contains the messages for the `Task Creator`. If not specified the default `pubsub-delayer-sub` will be used.
`CLOUD_TASK_LOCATION_ID`	The location identifier of the Cloud Task service (default: `europe-west3`).
`QUEUE_MAPPING_FILE`	Path to the YAML configuration file that defines topic-to-queue mappings (default: `/etc/pubsub-delayer/config/queue-mapping.yaml`). This configuration is mandatory.
`MAX_OUTSTANDING_MESSAGES`	The number of Pub/Sub messages which are allowed to be processed in parallel. If not specified, the default `1000` will be used. `-1` indicates no limit (not recommended).

GCLOUD_PROJECT_ID

The ID of the GCLOUD project (mandatory)

GCLOUD_PRIVATE_KEY

The GCLOUD private key encoded in Base64 which belongs to the used service account (mandatory)

GCLOUD_CLIENT_EMAIL

The email which is used to identify the service account (mandatory).

SERVICE_ACCOUNT_EMAIL

The email address of the service account used for Cloud Tasks authentication (mandatory).

ERROR_TOPIC

The pub/sub topic which shall be used to publish erroneous messages. If not specified the default pubsub-delayer-errors is used.

DELAYED_MESSAGE_SUBSCRIPTION

The subscription which contains the messages for the Task Creator. If not specified the default pubsub-delayer-sub will be used.

CLOUD_TASK_LOCATION_ID

The location identifier of the Cloud Task service (default: europe-west3).

QUEUE_MAPPING_FILE

Path to the YAML configuration file that defines topic-to-queue mappings (default: /etc/pubsub-delayer/config/queue-mapping.yaml). This configuration is mandatory.

MAX_OUTSTANDING_MESSAGES

The number of Pub/Sub messages which are allowed to be processed in parallel. If not specified, the default 1000 will be used. -1 indicates no limit (not recommended).