Chunker

The Chunker is a logical unit composed of three sub-components that work together to process batch campaigns:

Slicer

The Slicer receives batch campaign requests and creates smaller tasks for the Fetcher:

  1. Fetches campaign information from me-push

  2. Gets the contact list size via the Suite API

  3. Creates fetch tasks, each containing a start offset and count (default slice size: 10,000 contacts)

  4. Publishes tasks to sending-batch-fetcher-<dg> (max 400 publishes per batch)

  5. For throttled campaigns (with sending limits), queues a Launcher job on BullMQ instead

Fetcher

The Fetcher receives fetch tasks and retrieves contacts:

  1. Fetches the given number of contacts from the Suite contact list API using the offset

  2. Creates and publishes per-contact messages to the appropriate output topic:

    • Push, personalized → event-pers.push.request-publisher.batch.<dg>

    • Push, non-personalized → sending-batch-devicemapper-<dg>

    • Embedded message, personalized → event-pers.push.request-publisher.batch.<dg>

    • Embedded message, non-personalized → sending-batch-em-<dg>

Launcher

The Launcher is a BullMQ worker that handles throttled batch campaigns with rate-limited sending:

  1. Receives jobs from the launches-<dg> BullMQ queue

  2. Fetches a batch of contacts (sized by sendingLimit) from the campaign_audience table

  3. Publishes per-contact messages to the same output topics as the Fetcher

  4. Marks contacts as utilized in the database

  5. Re-enqueues itself with ~1 minute delay until all contacts are sent

  6. Checks abort status via Redis (AbortedCampaignsInfoService) — aborted campaigns publish notSent events

  7. Monitors total throttled campaign runtime and cancels if exceeded

BullMQ Configuration

  • Queue name: launches-<dg> (per distribution group)

  • Job attempts: ~2880 (covering 2 days)

  • Backoff: fixed 60 second delay

  • Worker lock duration: 120 seconds

  • Redis is used for idempotency and queue management

The starting, finishing, and aborting of chunking is registered with the campaign monitoring.

Dependencies

  • Pub/Sub:

    • Slicer receives from subscription sending-batch-<dg>-sub

    • Publishes fetch tasks to sending-batch-fetcher-<dg>

    • Publishes per-contact messages to personalization or device mapper topics

    • Transient errors use Pub/Sub Delay Service

    • Fatal errors go to sending-mg-errors

  • me-push: Campaign information

  • Suite: Contact list API for fetching contact IDs

  • Redis: BullMQ queues, abort status, idempotency

  • PostgreSQL: Campaign audience tables (for Launcher)

Chunker Components & Dependencies

Error Handling

  • Transient errors: Pub/Sub delay service with exponential backoff. When retries are exhausted, the message is dropped and the chunking action is reported as aborted.

  • Fatal errors: Published to sending-mg-errors.

See Error Handling for the full retry policy.

Code Location

src/components/chunker/