Inbox Pipeline Ownership Transfer

As a Push Channel developer I want to understand and independently execute the legacy testing pipeline so that I can own the complete test execution process without QA team dependency.

The inbox test code is intentionally not refactored — inbox will eventually be removed as a product. Refactoring should only happen when the team takes full QA ownership or when a new e2e pipeline is set up.

Jenkins Pipelines

Two pipelines are in scope — they share code and are treated as a single unit:

Pipeline Jenkins URL

Staging CI

Inbox_CI_TestSuite_Pipeline

Production Watchdog

Inbox_WD_TestSuite_Pipeline

Automatic Triggers

Both pipelines run on a schedule:

  • Staging CI — targets trunk / customer 213239625 (me_push_QAstaging)

  • Production Watchdog — targets suite17_WD / customer 767116266 (ME_PerformanceTest_s17)

The entry point is vars/InboxPipeline.groovy in jenkins_sharedlib. The suite-level job calls InboxPipeline once per test recipient, passing customerId and stagingOrProd.

Pipeline Parameters

Parameter Description Staging Production

recipient

Test scenario name — determines which stages run

e.g. SuiteSegment_Inbox

e.g. SuiteSegment_Inbox

customerId

Emarsys account ID

213239625

767116266

stagingOrProd

Environment label

trunk

suite17_WD

git_branch_name

Branch of me-qa-vienna to check out

master

master

shouldTriggerSlackNotification

Send Slack alerts on completion

true

true

How to Trigger a Regression Run

To manually trigger a full regression against staging:

  1. Open Inbox_CI_TestSuite_Pipeline in Jenkins.

  2. Click Build with Parameters in the left sidebar.

  3. Set the parameters:

    • customerId: 213239625 (staging) or 767116266 (production watchdog)

    • stagingOrProd: trunk (staging) or suite17_WD (production)

    • git_branch_name: master, or a feature branch from me-qa-vienna if testing a test change

    • Leave all other parameters at their defaults.

  4. Click Build.

To re-run only a single failed stage, trigger the relevant downstream Jenkins job directly with the appropriate Target_Recipient parameter rather than re-running the full suite. Each downstream job maps to one InboxPipelineBuilder method — see Jenkins Job Inventory.

Repository Overview

jenkins_sharedlib

Pipeline orchestration — defines stages, sequencing, retry logic, and triggers downstream jobs.

jenkins_sharedlib/
├── vars/
│   └── InboxPipeline.groovy           # Entry point — stages and flow
│
└── src/org/ems/inbox/
    ├── InboxPipelineBuilder.groovy    # Stage implementations — maps stages to Jenkins jobs
    ├── InboxSuiteData.groovy          # Test data and customer configs per environment
    └── InboxSdkData.groovy            # SDK-specific test data (not used in CI/WD pipelines)

Shared framework (same for all products):

src/org/ems/common/
├── PipelineFactory.groovy             # Creates pipeline component instances
├── PipelineConfigBuilder.groovy       # Fluent API for pipeline configuration
├── TestPipelineManager.groovy         # Manages job triggering and data access
├── JenkinsPipelineHelper.groovy       # executeWithRetry and utility methods
├── AbstractTestData.groovy            # Base class for test data
└── AbstractPipelineBuilder.groovy     # Base class for pipeline builders

me-qa-vienna

Test execution — Playwright specs that drive the Suite UI.

me-qa-vienna/playwright/
├── tests/inbox/
│   ├── createCampaign.spec.ts     # Create and launch campaign
│   ├── cloneCampaign.spec.ts      # Clone template and launch campaign
│   ├── recallCampaign.spec.ts     # Recall a launched campaign
│   └── checkInboxReport.spec.ts   # Validate report statistics
│
├── campaigns/
│   └── InboxCampaign.ts           # Page object: inbox campaign wizard
│
├── pages/
│   ├── InboxReportPage.ts         # Page object: inbox report page
│   └── OverviewTablePage.ts       # Page object: campaigns list table
│
└── testData/
    └── InboxTestData.ts           # Test data management and Consul read/write

Architecture

┌──────────────────────────────────────────────────────────┐
│  Layer 1: jenkins_sharedlib (Pipeline Orchestration)      │
│  InboxPipeline.groovy   — stages and retry logic          │
│  InboxPipelineBuilder   — triggers downstream jobs        │
│  InboxSuiteData         — customer / env configuration    │
└───────────────────────┬──────────────────────────────────┘
                        │ triggers (parameterized)
┌───────────────────────▼──────────────────────────────────┐
│  Layer 2: Jenkins Jobs (manually configured)              │
│  Each job runs a Playwright spec with env parameters      │
└───────────────────────┬──────────────────────────────────┘
                        │ executes
┌───────────────────────▼──────────────────────────────────┐
│  Layer 3: me-qa-vienna (Test Execution)                   │
│  Playwright specs drive the Suite UI                      │
│  InboxTestData reads/writes campaign state via Consul     │
└───────────────────────┬──────────────────────────────────┘
                        │ reads/writes
┌───────────────────────▼──────────────────────────────────┐
│  Layer 4: External Systems                                │
│  Suite UI — inbox campaign management                     │
│  Consul KV Store — shared state between stages            │
│  me-inbox-service API — service under test                │
└──────────────────────────────────────────────────────────┘

Stage Sequencing

Stages run sequentially. Stages are skipped (not failed) when the recipient does not apply.

Stage Active recipients Retry config

Create Inbox campaign

All

16 retries, 1 min

Get Information

All

16 retries, 1 min

Event Trigger

ACSingle_Inbox, RTI_Inbox

None

Waiting ..

SuiteSegment_Inbox_Launch_Delete, SuiteSegment_Inbox_Schedule_Cancel

11 min sleep

Inbox Delivery Checkpoint

All (NotDelivered for delete/cancel variants)

22 retries, 2 min

Browser Inbox Delivery Checkpoint

SuiteSegment_Inbox

22 retries, 2 min

Inbox Add/Remove Tag * Request (×8)

SuiteSegment_Inbox

22 retries, 2 min

Inbox Sending DB Checkpoint

All except Schedule_Cancel

22 retries, 2 min

Inbox Add/Remove Tag * Checkpoint (×8)

SuiteSegment_Inbox

22 retries, 2 min

Inbox Recall

MobileSegment_Inbox_DeviceFilter_Recall

22 retries, 2 min

Inbox Recall Checkpoint

MobileSegment_Inbox_DeviceFilter_Recall

22 retries, 2 min

Inbox Report Checkpoint

All except Schedule_Cancel

20 retries, 2 min
executeWithRetry is intentional on delivery and DB checkpoint stages — they poll for an asynchronous result.

Test Recipient Scenarios

Recipient Description CheckType Environments

SuiteSegment_Inbox

Suite segment — full tag lifecycle + report

Delivered

Both

ContactList_Inbox

Contact list campaign

Delivered

Both

ContactList_Test

Contact list test message

Delivered

Both

SuiteSegment_Test

Suite segment test message

Delivered

Both

SuiteSegment_Test_Mail

Suite segment test via email

Delivered

Both

ACContactList_Inbox

Automation Center — contact list

Delivered

Both

ACSingle_Inbox

Automation Center — single (clone flow)

Delivered

Both

RTI_Inbox

Real-Time Interaction

Delivered

Both

SuiteSegment_Inbox_Schedule_Cancel

Schedule then cancel

NotDelivered

Both

MobileSegment_Inbox_NotPersonalized

Mobile segment, no personalization

Delivered

Both

SuiteSegment_Inbox_Launch_Delete

Launch then delete (11 min wait)

NotDelivered

Both

MobileSegment_Inbox_DeviceFilter_Recall

Device filter + recall flow

Delivered

Both

Test Cases

All inbox Playwright tests live in playwright/tests/inbox/ in me-qa-vienna. They drive the Suite UI via page objects and share state through Consul.

Create and Launch Campaign (createCampaign.spec.ts)

Jenkins job: 01.Inbox_PW_Create_Campaign_UI

Creates a new inbox campaign from scratch, launches it, and saves the campaign ID to Consul.

Steps

  1. prepareTestData('Inbox') — loads config from Consul

  2. Log in to Suite UI

  3. Navigate to Inbox Campaigns

  4. Create new campaign

  5. Complete Message Settings

  6. Complete Content Creation

  7. (Conditional) Send test message if isTestMessage === 'true'

  8. (Conditional) Navigate to Scheduling if not RTI/ACContactList/ACSingle and not a test message

  9. (Conditional) Complete Scheduling if cancelLaunch or launchDelete is true

  10. Launch campaign

  11. Navigate back to campaign list

  12. Verify status (see table below)

  13. (Conditional) Cancel launch if cancelLaunch === 'true'

  14. Get campaign ID from URL and save to InboxTestData.params.campaignId

  15. (Conditional) For RTI/ACContactList/ACSingle — navigate to Automation and update AC program

  16. Save campaign details to Consul

  17. Log out

Table 1. Expected status after launch
Condition Expected status

Normal launch

Launched

cancelLaunch === 'true'

Scheduled → In design after cancel

RTI / ACContactList / ACSingle

Ready to launch

isTestMessage === 'true'

Tested

Clone and Launch Campaign (cloneCampaign.spec.ts)

Jenkins job: Inbox_PW_Clone_Campaign_UI (also used for ACSingle_Inbox in the create flow)

Clones an existing template campaign, edits it, launches it, and updates the Automation Center program.

Steps

  1. prepareTestData('Inbox') — loads config from Consul

  2. Log in to Suite UI

  3. Navigate to Inbox Campaigns

  4. Verify template has status In design

  5. Clone the template campaign

  6. Open the clone for editing

  7. Update Message Settings

  8. Navigate to Content Creation and update content

  9. Launch campaign

  10. Verify status is Ready to launch

  11. Get campaign ID and save to Consul

  12. Navigate to Automation and update AC program (AC_INBOX_Template or event-based template for MobileEvent recipient)

  13. Save campaign details to Consul (writeCampaignDetailsToConsul('Clone'))

  14. Log out

Recall Campaign (recallCampaign.spec.ts)

Jenkins job: 07.Inbox_PW_Recall_Campaign_UI

Recalls a previously launched campaign and verifies it reaches Recalled status.

This test must run after createCampaign or cloneCampaign — it reads the campaign name from Consul.
Steps

  1. prepareTestData('Inbox') — loads config from Consul

  2. Read campaign details from Consul

  3. Log in to Suite UI

  4. Navigate to Inbox Campaigns

  5. Open the campaign for editing

  6. Recall the campaign

  7. Navigate back to campaign list

  8. Verify status is Recalled

  9. Log out

Check Inbox Report (checkInboxReport.spec.ts)

Jenkins job: Inbox_Report_Check_PW_UI

Validates the campaign report page shows correct delivery statistics.

Steps

  1. prepareTestData('Inbox') — loads config from Consul

  2. Read campaign details from Consul

  3. (suite0_mobile only) Read report values from Consul

  4. Log in to Suite UI

  5. Navigate to Inbox Campaigns

  6. Navigate to the campaign reporting page

  7. Validate stats (validateCounts())

  8. (Conditional) Validate recall text if recall === 'true'

  9. (Conditional) Validate undelivered reasons if not suite0_mobile and undeliveredPercentage !== '0'

  10. Log out

Metrics validated per scenario: Total_Sent, Total_Open, Delivered_Value, Undelivered_Percentage, Undelivered_Count_With_Reason (e.g. Personalisation=1,NoDevice=1).

Data Flow & Consul Integration

Consul KV store is the shared state mechanism between pipeline stages. Each Playwright test reads parameters at startup (prepareTestData) and writes results on completion.

Consul Key Format

{PipelineName}/{Target_Recipient}/{Env}/ME.{jsonFilename}.json

Example (staging, SuiteSegment):

Inbox_CI_TestCase_Pipeline/SuiteSegment/trunk/ME.SuiteSegment_Inbox_V3_PR_s_ng5_me_push_QAstaging_SuiteSegment_Inbox.json

Data Written per Stage

Stage Data written

createCampaign

messageName, campaignId, templateName

cloneCampaign

messageName, campaignId (with Clone suffix)

recallCampaign

Reads only

checkInboxReport

Reads only

Jenkins Job Inventory

Jenkins job definitions are manually configured in Jenkins and are not stored in version control. To view or edit a job, navigate to it in Jenkins and click Configure.
Job name Builder method Purpose

01.Inbox_PW_Create_Campaign_UI

buildCreateInboxCampaign()

Create and launch inbox campaign (Playwright)

Inbox_PW_Clone_Campaign_UI

buildCloneInboxCampaign() / buildCreateInboxCampaign() for ACSingle

Clone template and launch (Playwright)

GetMessageInformation_API

buildGetInfoInbox()

Read campaign data from API, write to Consul

05.Send_External_Event_Triggers

buildExternalEventsTrigger()

Trigger external events for RTI/ACSingle

InboxDelivery_NEW

buildInboxDeliveryCheckpoint() / buildInboxDeliveryRecallCheckpoint()

Check inbox delivery state in API

BrowserInbox_Delivery

buildBrowserInboxDeliveryCheckpoint()

Check browser inbox delivery

Inbox_TagRequest_V3_NEW

buildInboxTagRequest(TagType, tagValue)

Add/remove inbox tags via API

Inbox_TAG_DP_Checkpoints

buildInboxTagDBCheckpoint(TagType, DP_Check)

Validate tag operations in DB

07.Inbox_PW_Recall_Campaign_UI

buildInboxRecall()

Recall campaign via UI (Playwright)

Inbox_Report_Check_PW_UI

buildInboxReportCheckpoint()

Validate report page stats (Playwright)

Inbox_DP_Checkpoints

builtInboxDPCheckpoints(DP_Check)

Validate sending data in DB

Test Failure Analysis

Distinguishing Infrastructure vs. Product Failures

Symptom Likely cause Action

Fails on first attempt, passes on retry

Transient infrastructure / timing

Safe to ignore if it recovers

Consistently fails after all retries

Product regression or broken test data

Investigate the downstream job logs

Delivery Checkpoint fails

Message not delivered in time, or me-inbox-service issue

Check service health and logs

DB Checkpoint fails

Data pipeline delay

Check if data appears after a longer wait

Playwright test fails — UI element not found

UI change in Suite, or stale Consul data

Check for UI changes; re-run if Consul data expired

Create Campaign fails

Consul config missing, or Suite UI issue

Verify static Consul config values are present

Where to Find Logs

  1. Jenkins stage log: Click the failed stage in the pipeline view → Console Output of the triggered downstream job.

  2. Playwright output: The downstream Playwright job prints test steps and assertion errors to its console. Look for the failed test.step name.

  3. Consul data: After a failed run, check Consul to see what was written (or not) by previous stages — this shows where in the chain the failure occurred.

Common Issues

  • Expired Consul data — If stale data from a previous run is in Consul, the report check may validate wrong counts. Each jsonFilename is unique per recipient and environment so this is rare but can happen after manual interventions.

  • Campaign name collision — Campaign names include a timestamp. If jobs run too close together they may target the same campaign name. Re-running usually resolves this.

  • Recall checkpoint mistiming — The Inbox Recall Checkpoint verifies NotDelivered state. If the API processes the recall faster than expected the checkpoint may pass immediately; if slower it relies on the 22-retry loop.