Set Contact Workflow

This concept is still not finished and is subject to change

This document means to detail a concept for how a new contact-resolution process could work within mobile engage, replacing the batch-login process in the push-notification service.

Background

The batch-login worker was built for v2 of the client registration API which is completely deprecated now and not used anymore. The v3 of the client registration API is currently using the batch-login workflow by constructing a so called login message and publishing it to the batch-login queue in AMQP.

V3 Client API

V2 used a single endpoint for all client changes while v3 has one endpoint for each type of client-property. The service combines the whole client into a single jwt-token written as a header whenever changed and persisted by the client and sent to the backend as a header with every request.

  • /v3/apps/:appCode/client

  • /v3/apps/:appCode/client/push-token

  • /v3/apps/:appCode/client/contact

  • /v3/apps/:appCode/client/contact-token

Updating suite contact fields

In order to roll out this change to all customers we need to trigger the update of certain contact fields in suite which are used in segmentation for older customers. Currently this is triggered by all changes to the client for app that have the following fields defined.

  • contact_field_app_installed

  • contact_field_push_accepted

  • contact_field_last_mobile_activity

Workflow

Workflow overview

  1. Client makes a request to the /v3/apps/:appCode/client/contact endpoint with the contact-field-id and the contact-field-value in the body.

  2. The endpoint controller publishes the payload to the client-change pubsub topic with following attributes

    • me-client:should_update_suite_contact_fields "true" or "false" depending on the fields available in the app.

    • me-client:is_anonymous "true" or "false" depending on the the i anonymous query param.

    • me-client:change_type set to "contact-field"

  3. A subscription of client-change filtered for only non-anonymous contact-field changes is consumed by the client-identified-contact-resolver worker. The worker resolves the contact-id from the contact-field.

  4. The contact-id change is published to the client-change topic.

  5. A subscription of client-contact filtered for only contact-id changes is consumed by the client-contact-id-writer worker which writes contact-id to the following places.

    • set contact property in contact-reference entry in dynamo

    • set contact property in device entry in dynamo

    • publish to client-state-contact-change topic

Change log for debugging purposes

To help investigations we’ll write all changes to the client to big-query.

Topics

client-change

Attributes

  • me-client:should_update_suite_contact_fields: true | false

  • me-client:is_anonymous: true | false

  • me-client:change_type: contact-field | contact-id | push-token | client

Payload Example

{
  "eventTime": 123455667,
  "customerId": 12345,
  "appCode": "EMS-1234",
  "contactReference": "contact-ref",
  "clientId": "client-id",
  "contactField": {
    "contactFieldId": 123,
    "contactFieldValue": "contact-field-value"
  },
  "clientState": {
    ""
  }
}

Subscriptions

The work is split over multiple subscriptions with their corresponding workers, the split is done mostly due to differences in how the different parts of the work can be retried and scaled.

client-identified-contact-resolver

Subscribed to client-change with a filter attributes."me-client:change_type"="contact-field" AND attributes."me-client:is_anonymous"="false". Resolves the suite contact-id from the client-field. If the lookup fails it needs to be retried up to 48H since the contact might not have been imported into emarsys yet. The contact-id is published to the client-change topic with the change_type='contact-id' attribute set.

Since a pubsub message batch is either fully processed or redelivered the worker needs to register all resolved changes in some storage like redis to be able to do idempotency checks and filter out messages that have already been processed.

Retrying

It’s required to retry the contact-resolution up to 48h since customers might not be syncing new contacts to Emarsys in realtime. For failures finding a contact with a specific field-value we’ll use the me-pubsub-delayer to schedule a retry in the future and for all other failures we can use the exponential backoff strategy from pubsub. Due to the volume of retries we would have dedicated topic in the me-pubsub-delayer.

Monitoring

Messages for contact-fields that cannot be resolved are being scheduled to be retried by the me-pubsub-delayer making the normal main queue easy to monitor.

client-anonymous-contact-resolver

Subscribed to client-change with a filter attributes."me-client:change_type"="contact-field" AND attributes."me-client:is_anonymous"="true". Resolves the suite contact-id from the client-id if the client has already used anonymous contact, and if no existing contact is found it creates a new one. The contact-id is published to the client-change topic with the change_type='contact-id' attribute set.

It does not cause any problem if a message is processed twice so there is no need for an idempotency check in the worker.

client-contact-id-writer

Subscribed to client-change with filter attributes."me-client:change_type"="contact-id". The resolved contact-id needs to be written to several places. We must set contact property in contact-reference entry as well as the device entry in dynamo. Finally we also need to publish to client-state-contact-change topic to make sure it’s written to big-query and available for segmentation.

Since the event-time is used to avoid overwriting newer information there is no need for an idempotency check in the worker.

client-change-big-query-writer

Subscribed to client-change without any filters and simply writes all the changes to BigQuery. This data could be used for debugging purposes.

client-suite-contact-updater

Subscribed to client-change with filter attributes."me-client:should_update_suite_contact_fields"="true". Either the suite-contact updates are done directly in the worker or it publishes an AMQP message to the current worker. See worker implementation.