Operations

Generally there are 2 things which might come up in context of operations for the scheduler

  • investigations about jobs which are contained in BullMQ queues or

  • investigations about errors reported for segmentation.

So in case you need to investigate

  • jobs in the queue a bit more than what is provided in LaaS logs you can find cli tools for that in the tools folder of the project. The cli tools are described in the following sections.

  • segmentation issues refer to the Segmentation Operations page.

Tools: The Bull Console

In the tools folder, there is a console folder which contains a console for investigations on the BullMQ queues. To use the queue on your local machine, all you have to do is to have the REDIS_URL defined in your .env file. The supported commands are described in the following sub chapters.

When you are connected to one of the application pods the command line tool is invoked by calling node tools/console <command> [param]…​. To invoke the console on your local machine simply type the following command when you are located in the project’s root folder: node ./src/tools/console/index.js <command> [param]…​. When omitting the command or passing --help the help page will be displayed.

bull-status

This command can be used to get an overview of the queue regarding the states the entries have. You have to specify the name of the queue you are interested in. The returned result will look like the following:

$ node ./src/tools/console/index.js bull-status launches
Status of queue "launches": {
  "wait": 0,
  "active": 0,
  "completed": 0,
  "failed": 6,
  "delayed": 3
}

So the queue contains 9 entries in total, 6 are in status failed and 3 are currently delayed.

bull-dump

This command is used to dump all jobs in a specific state of a specific job queue (so the name of the queue as well as the state has to be defined). Following you see an example of the displayed result:

$ node ./src/tools/console/index.js bull-dump launches delayed
Jobs with status "delayed" in queue "launches": [
  {
    "id": "785",
    "data": {
      "source": "segment",
      "customerId": 218524530,
      "segmentId": 1000500707,
      "campaignId": 1250,
      "applicationId": 7,
      "scheduledAt": "2019-01-08T00:00:00.000Z",
      "runId": "1546966927619821084086",
      "runCounter": 1
    },
    "opts": {
      "backoff": {
        "type": "exponential",
        "delay": 1000
      },
      "attempts": "24",
      "jobId": "785"
    },
    "progress": 0,
    "delay": 0,
    "timestamp": 1546966927632,
    "attempts": 24,
    "attemptsMade": 22,
    "failedReason": "Segment is being prepared.",
    "stacktrace": null,
    "returnvalue": null
  },
...
]

bull-find

This command can be used to find specific entries in a queue regardless of the state but filtered by the customer ID and/or campaign ID. The result will be an object that contains all possible states and an array which contains the jobs which matched the filter criteria. Following you can see an example:

$ node ./src/tools/console/index.js bull-find launches --customer-id 218524530 --campaign-id 1250
Applying filter: customerId =  218524530
Applying filter: campaignId =  1250
Jobs in queue "launches": {
  "active": [],
  "waiting": [],
  "delayed": [
    {
      "id": "785",
      "data": {
        "source": "segment",
        "customerId": 218524530,
        "segmentId": 1000500707,
        "campaignId": 1250,
        "applicationId": 7,
        "scheduledAt": "2019-01-08T00:00:00.000Z",
        "runId": "1546966927619821084086",
        "runCounter": 1
      },
      "opts": {
        "backoff": {
          "type": "exponential",
          "delay": 1000
        },
        "attempts": "24",
        "jobId": "785"
      },
      "progress": 0,
      "delay": 0,
      "timestamp": 1546966927632,
      "attempts": 24,
      "attemptsMade": 22,
      "failedReason": "Segment is being prepared.",
      "stacktrace": null,
      "returnvalue": null
    }
  ],
  "completed": [],
  "failed": [
    {
      "id": "1939",
      "data": {
        "source": "segment",
        "customerId": 218524530,
        "segmentId": 1000500707,
        "campaignId": 1250,
        "applicationId": 7,
        "scheduledAt": "2019-01-08T00:00:00.000Z",
        "runId": "1546966927619821084086",
        "runCounter": 2
      },
      "opts": {
        "backoff": {
          "type": "fixed",
          "delay": 10000
        },
        "attempts": "24",
        "jobId": "1939"
      },
      "progress": 0,
      "delay": 0,
      "timestamp": 1553091020388,
      "attempts": 24,
      "attemptsMade": 24,
      "failedReason": "Segment is being prepared.",
      "stacktrace": null,
      "returnvalue": null
    },...
  ]
}

bull-retry

Can be used to trigger a retry for a job which is in state failed and part of a specific queue. So - as already mentioned - it is necessary to specify the name of the BullMQ queue and the ID of the job.

Example:

$ node ./src/tools/console/index.js bull-retry launches 1941
Triggered retry for job with id "1941" in queue "launches"

bull-delete

Can be used to delete/remove a specific job from a specific queue. It is mandatory to specify the name of the BullMQ queue and the ID of the job.

Example:

$ node ./src/tools/console/index.js bull-delete init push-59998-1
Removed job with id "push-59998-1" from queue "init"