Multi Channel Content Service

There is the need from multiple channels like Push, In-app, Embedded Messaging, Wallet and Web-channel to support personalised content for the end-user. Some services (e.g. Embedded Messaging) require personalised content not to change over time while others are want the content to reflect the latest data. This can be controlled by the Token Policy which can be set to mutable or immutable.

Glossary

  • content: The content is the actual data that is being sent to the end-user. This can be in the form of HTML, JSON or any other format.

  • content template: A content template is a pre-defined structure that contains placeholders for dynamic data. The content template is used to generate the final content that will be sent to the end-user.

  • token: See personalisation token.

  • personalisation token: A token is a placeholder in the content that will be replaced with a value when the content is compiled. Tokens are used to personalise the content for each contact.

  • contact-data: Used to describe the personalised token resolved for a contact.

Token Policy

  • immutable: The personalised tokens are not expected to change over time. This means that the personalised tokens are stored in the database and used when compiling the final content.

  • mutable: When token personalised tokens are allowed to vary over time and the service will decide how it’s used based on the type of token. If possible token is resolved in real-time by the personalisation service, and if the token is not eligible for use with the real-time compilation it will be resolved using the event based personalisation api and stored in the database and used when compiling the final content.

Service Goals

  • Avoid storing token-data as much as possible - to keep the required storage low.

  • Avoid duplication of contact-data as much as possible - to avoid unnecessary personalisation of data.

  • Allow all kinds of personalisation tokens to be used in a transparent way.

  • Be open for dedicating resources for specific customers if needed.

Advantages for Services

In-app

Currently In-app is only supporting personalisation tokens that are eligible for real-time compilation. The use of other tokens (e.g. rds) has been requested by multiple customer. By using the content-service in-app would allow customers to use all tokens that are supported by the personalisation service.

Wallet

The Wallet service is like In-app only supporting personalisation tokens that are eligible for real-time compilation. The use of other tokens (e.g. rds) has been requested by multiple customer. By using the content-service wallet would allow customers to use all tokens that are supported by the personalisation service.

Web-channel

The web-channel does support all tokens that are supported by the personalisation service but it’s also treating all tokens the same. There are lots of cases where it would be better to use real-time compilation and avoid storing the personalised tokens. Using the content-service the web-channel would be able to use the same tokens as the other channels and avoid the current pre-personalisation process.

Embedded Messaging

The embedded messaging service is being developed but does require in-app like content to be served to end-users. Using content-service would solve the problem of preparing and serving the content to the end-user.

Basic Workflow

  1. Register Content Template with content containing personalisation tokens

  2. Prepare contact content for one or more contacts

  3. Receive contact content response for contacts

  4. Deliver content via external URL or via internal api

Service Setup

For a service to be able to create content template it needs to be authorised via the service mesh configuration. The source identifier will be configured to be used from the service. The setup also includes creation of source specific topics like Topic content-service-contact-error_<source> and Topic content-service-contact-response_<source>.

Register Content Template

Content Service Register Content

[1]

A content template is registered via the Internal REST API, the request includes the content mapped by the language code, an id as well as some configuration for the token-data.

During creation the content is evaluated and if it contains personalisation tokens the tokens are saved in the tokens column. The tokens that cannot or should not be evaluated in real-time are converted to template and registered with the personalisation service. The resulting tokens_personalisation_hash is saved in the content template.

Depending on the policy the content might be transformed to use injected tokens before being registered with the personalisation service.

The original content is saved in the content_template_contents table.

Prepare contact content

Prepare Contact Content

[2]

A request is published to Topic content-service-contact-request. The request contains the id, version and list of contactIds.

Once prepared the response is published to the Topic content-service-contact-response_<source> topic. The response contains the template id, version and the contact data for the successful.

See Contact Content Preparation for more details.

Deliver content

Content is delivered via either the Internal REST API or the External REST API.

Personalised content

In this case we need to compile the content using the real-time personalisation endpoint. The content is compiled using the content_personalisation_hash and the tokens are injected with the values from the personalised_tokens table. Before making the request to compile endpoint we verify that all personalised tokens are available by comparing with the tokens column in the content_template_contents table.

Example 1: Selecting immutable personalised tokens:

SELECT value, token_id
FROM personalised_tokens pt
INNER JOIN immutable_personalised_tokens ipt ON pt.id = ipt.personalised_token_id
WHERE content_template_id = :content_template_id
AND pt.contact_id = :contact_id

Example 2: Selecting mutable personalised tokens:

WITH ordered_contact_tokens AS (
  SELECT value, token_id, ROW_NUMBER() OVER (PARTITION BY token_id ORDER BY created_at DESC) AS row_num
  FROM personalised_tokens
  WHERE token_id IN (:list_of_token_ids)
  AND contact_id = :contact
)

SELECT value, token_id
FROM ordered_contact_tokens
WHERE row_num = 1

Example 3: Personalisation service request :

This uses the v3 compile endpoint

{
  "data": [{
    "contactId": 0,
    "ems_channel": {
      "rds_1234": "value-from-rds",
      "voucher_1234": "value-from-voucher"
    }
  }]
}

Non-personalised content

In this case the content is served as is from the content_template_contents table.

Content Transformation

When compiling personalised content we’re using the real-time personalisation endpoint. For this to work prepare some tokens that are not supported by the real-time personalisation service. These tokens are transformed to use the ems_channel prefix and their values are injected with the compile request.

The ems_channel is not yet supported by the real-time personalisation compilation, this would need to be changed in the personalisation service.

Immutable Personalised Content

This is the most complex case. Since content with [immutable_policy] tokens is not expected to change over time we need to persist the personalised tokens and use these when compiling the final content.

Original content:

<html>
  <div>
    <p>Hi {{contact.1}}</p>
    <p>Here is your voucher code: {{voucher.2353}}</p>
  </div>

Transformed content:

All tokens are transformed to use the ems_channel prefix since we’ll always be injecting the values because they should not be changing over time.

<html>
  <div>
    <p>Hi {{ems_channel.contact_1}}</p>
    <p>Here is your voucher code: {{ems_channel.voucher_2353}}</p>
  </div>`

Immutable Non-personalised Content

For [immutable_policy] non-personalised content the content is saved as is in the content_template_contents table. The content is not transformed and there are no tokens to be saved.

Mutable Personalised Content

When the content is [mutable_policy] is means that it’s allowed to change over time. This allows us use all tokens eligible for real-time compilation as they are and we only need to modify the tokens that are not supported by the real-time personalisation service.

Original content:

<html>
  <div>
    <p>Hi {{contact.1}}</p>
    <p>Here is your voucher code: {{rds.584843}}</p>
  </div>

Transformed content:

Here we see that the contact token is kept since it’s supported by the real-time personalisation service. The rds token is transformed to use the ems_channel prefix.

<html>
  <div>
    <p>Hi {{contact.1}}</p>
    <p>Here is your voucher code: {{ems_channel.rds_584843}}</p>
  </div>`

Non-Personalised Dynamic Content

For non-personalised mutable content the content is saved as is in the content_template_contents table. The content is not transformed.

Contact Content Preparation

To prepare content serving for a contact a request is published to Topic content-service-contact-request. The request contains the contactId and contentTemplateId.

When preparing the tokens_personalisation_hash is used to resolve the tokens of the contact and the result is then saved in the personalised_tokens table.

The tokens_personalisation_hash is from a generated template which only contains the tokens. It is formatted in a way that is easy to parse and extract the values.

Example token template:

{
  "contact.1": "{{contact.1}}",
  "voucher.1234": "{{voucher.1234}}",
  "rds.4567": "{{rds.4567}}"
}

Once the request is processed a response is published to Topic content-service-contact-response_<source>. The response contains the external id of the template, version along with a list of contactIds and corresponding URLs.

Interfaces

Internal REST API

The REST API is used to create and manage content templates. The API allows you to create, update, and delete content templates, as well as retrieve information about existing templates.

The source of the template is secured via service mesh authentication. The customerId is used to identify the customer for whom the content template is being created.

Register Content Template

POST /v1/tenants/:customerId/:source/content-templates

Request
{
  "id": "string",
  "contentType": "text/html",
  "defaultLanguage": "en",
  "tokenConfig": {
    "policy": "mutable",
    "refreshIntervalInHours": null,
    "expiryInMonths": 6
  },
  "contentByLanguage": {
    "en": "<html>...</html>"
  }
}
Response
{
  "id": "string",
  "version": 1
}

Get Content Template

GET /v1/tenants/:customerId/:source/content-templates/:id/versions/:version

Response

{
  "id": "string",
  "version": 1,
  "contentByLanguage": {
    "en": "<html>...</html>",
  },
  "tokens": ["contact.1", "voucher.1234"],
  "tokenConfig": {
    "policy": "mutable",
    "refreshIntervalInHours": null,
    "expiryInMonths": 6
  },
  "isPersonalised": true,
  "personalisationTemplateId": "string",
  "createdAt": "2023-01-01T00:00:00Z",
}

Delete Content Template

DELETE /v1/tenants/:customerId/:source/content-templates/:id

Response

{
  "versions": [1,2]
}

Get Contact Content

If the content is needed internally, it can be retrieved using the Internal REST API. The Content-Type header will be set to the content type of the content template. It does still require the contactId to have been prepared already.

POST /v1/tenants/:customerId/:source/content-templates/:templateId/versions/:version/content

Request
{
  "contactId": "string",
  "language": "en"
}
Response
<html>...</html>

External REST API

Get Contact Content

The external API is used to retrieve the content for a specific contact. These URLs are generated by the content-service and sent to the service via PubSub after Contact Content Preparation. The Content-Type header will be set to the content type of the content template. The lang parameter is used to specify the language of the content, this parameter is optional and can be added/modified by the requesting client.

GET /v1/contents/:signed_jwt_token?lang=en

Response
<html>...</html>

PubSub

To make it possible to filter the messages both the source and customerId are specified as attributes.

Topic content-service-contact-request

Publishing to this topic will trigger the content service to create a personalised content template for the contacts.

Attributes

  • source: The source of the content template

  • customerId: The customer ID

Payload
{
  "id": "string",
  "version": 1,
  "contactData": [
    {
      "contactId": "string",
      "language": "en"
    }
  ]
}

Topic content-service-contact-response_<source>

This topic is used to respond to the request for contact content. The response contains the contactId and the content-template-id along with the url where the content is externally available.

Attributes

  • customerId: The customer ID

Payload
{
  "id": "string",
  "version": 1,
  "contactData": [
    {
      "contactId": "string",
      "contentUrl": "https://example.com/v1/contents/:signed_jwt_token?lang=en"
    }
  ]
}

Topic content-service-contact-error_<source>

Any error that occurs during the processing of the contact content preparation will be published to this topic. This includes errors such as invalid tokens, missing tokens, or any other errors that occur during the processing.

Attributes

  • customerId: The customer ID

Payload
{
  "id": 123,
  "version": 1,
  "contactData": [
    {
    "contactId": "string",
    "errorMessage": "no contact found with id",
    }
  ]
}

Topic content-service-contact-data-refresh

Publishing to this topic will trigger the content data to refresh for the contained contacts.

Attributes

  • source: The source of the content template

  • customerId: The customer ID

Payload
{
  "templateHash": "string",
  "contactIds": [123,345]
}

Contact Data Refresh

When registering a content template the refreshIntervalInHours can be set. This defines how often the personalised token should be refreshed. If not defined the tokens won’t be refreshed.

When contact-content is prepared for template with the refreshIntervalInHours set, a record is created in the refreshing_personalised_tokens table. This record contains a reference to the personalised_tokens record and information on how long the value is valid. The date is calculated based on the refreshIntervalInHours and the current time.

On a regular basis a job will run to check the refreshing_personalised_tokens table for records that are expired. If a record is expired the personalised token will be refreshed and valid_until is updated.`

By querying the tokens-ids and contact we can create requests to the personalisation service to refresh the personalised tokens.

SELECT array_agg(token_id) as token_ids, contact_id
FROM refreshing_personalised_tokens
INNER JOIN tokens ON token_id = tokens.id
WHERE valid_until < now()
GROUP BY contact_id

Internally a [content-service-contact-data-refresh] will be used to execute the refreshing of the personalised tokens.

It would make sense to separate out the refreshing of RDS to a separate subscription, the RDS tokens often make issues and it it’s better to have a separate worker pool for them.

Data maintenance

When creating a template the expiry of the contact data is defined. The content service will automatically delete the contact data after the expiry date. The content service will also automatically delete the contact data if the content template is deleted.

The content service would also need to subscribe to contact-changes and cleanup data for any delete contacts.

[content_templates_table]

No automated cleanup is needed for the content_templates table. The content templates are versioned and the old versions are kept in the database. The content templates are deleted when the content template is deleted from the outside.

[content_template_contents_table]

Content data is deleted along with the template.

[personalised_tokens_table]

The personalised_tokens table can be cleaned up by deleting records * not referenced in the immutable_personalised_tokens * not the latest version of the token_id for the contact.

WITH ranked_records AS (
    SELECT
        id,
        token_id,
        ROW_NUMBER() OVER (PARTITION BY token_id ORDER BY created_At DESC) AS row_num
    FROM personalised_tokens
),
immutables AS (
	SELECT distinct personalised_token_id
	FROM immutable_personalised_tokens
)
DELETE FROM personalised_tokens
WHERE id IN (
    SELECT id
    FROM ranked_records
    WHERE row_num > 1
) AND id NOT IN (SELECT personalised_token_id FROM immutable);

[immutable_personalised_tokens_table]

The immutable personalised tokens are deleted depending on the token_ttl in the content_templates table.

DELETE FROM personalised_tokens pt
USING content_templates ct, immutable_personalised_tokens ipt
WHERE ipt.content_template_id = ct.id
AND ipt.personalised_token_id = pt.id
AND pt.created_at + ct.token_ttl < now()

refreshing_content_template_personalised_tokens

The records are deleted along with the content template.

Database Schema

Tenants are defined by the customerId and separated by schema. For each tenant a schema is created containing all tables defined below. If needed the schema for specific can be migrated to a dedicated database instance.

The contact-data in forms of personalised tokens is shared across campaigns to avoid duplication of data.

Tenant migrations are done using the library created for the embedded messaging service where the same approach is used.

content_templates

Name Type Nullable Description

id

BIGINT

false

Auto-generated unique identifier

external_id

VARCHAR

false

externally defined identifier for the content template

source

TEXT

false

Identifier of the service creating the content template.

version

INT

false

Version of the content template

content_type

VARHAR

false

Type of the content template (e.g. text/html or application/json)

default_language

VARCHAR

false

Default language of the content template

content_personalisation_hash

VARCHAR

true

Template hash from the personalisation service for the content

tokens_personalisation_hash

VARCHAR

true

Template hash from the personalisation service for the tokens

token_policy

ENUM

false

Policy of the content data (static or dynamic)

token_refresh_interval

INTERVAL

true

Refresh interval for the tokens

token_ttl

INTERVAL

true

Expiry for the tokens

tokens

VARCHAR[]

true

Tokens used in the content template

created_at

TIMESTAMP

false

Timestamp when the content template was created

Records are unique by external_id, source and version.

content_template_contents

Contains the actual content of the templates.

Name Type Nullable Description

id

BIGINT

false

Auto-generated unique identifier

content_template_id

BIGINT

false

Foreign key to the content_templates table (on delete cascade)

data

JSONB

false

An object containing content data by language code

created_at

Timestamp

false

Timestamp when the record was created

personalised_tokens

Table holds the value for single token and contact. Records are typically never updated but instead new records are created. The created_at column is used to determine the latest version of the token.

Name Type Nullable Description

id

BIGINT

false

Auto-generated unique identifier

token_id

BIGINT

false

Foreign key to the tokens table

contact_id

BIGINT

false

Unique identifier for the contact

value

TEXT

false

The resolved value for the token

created_at

Timestamp

false

Timestamp when the record was created

expires_at

Timestamp

false

Timestamp when the record expires

immutable_personalised_tokens

Table contains tokens that are configured to be immutable. The data is used when serving templates to the contact to make sure we’re using the same version of the token each time and also to make sure they are not removed even if there are newer versions available.

Name Type Nullable Description

id

BIGINT

false

Auto-generated unique identifier

content_template_id

BIGINT

false

Foreign key to the content_templates table (on delete cascade)

personalised_token_id

VARCHAR

false

Foreign key to the personalised_tokens table (on delete cascade)

refreshing_personalised_tokens

Table for tracking tokens that needs automatic refreshing of their values.

Name Type Nullable Description

id

BIGINT

false

Auto-generated unique identifier

token_id

BIGINT

false

Foreign key to the tokens table

contact_id

BIGINT

false

Unique identifier for the contact

content_template_id

BIGINT

false

Foreign key to the content_templates table (on delete cascade)

valid_until

Timestamp

false

Timestamp when the record is no longer valid and need to be refreshed

created_at

Timestamp

false

Timestamp when the record was created

tokens

A small table that contains the token_ids (e.g. contact.1, voucher.123) and an auto generated numberic id which is used for references from other tables.

Name Type Nullable Description

id

BIGINT

false

Auto-generated unique identifier

token_id

BIGINT

false

Identifier for the token (e.g. contact.1, voucher)

created_at

Timestamp

false

Timestamp when the record was created

Database instance

The content service is using a Cloud SQL Postgres database. Since the database contains PII data the instance needs to be behind the VPC.

Metrics

The content service is generating the following metrics:

  • ContentService_Web_CreateContentTemplate_Count

  • ContentService_Web_CreateContentTemplate_Duration

  • ContentService_Web_FetchContent_Duration

  • ContentService_Web_FetchContent_Count

  • ContentService_PrepareContactContent_Count_Duration

  • ContentService_PrepareContactContent_Consume_Duration

Tech. Stack

In order to be able to re-use as much as possible the existing services and libraries the content service will be a fastify based service using node.js and typescript.

Cost estimations

Cloud SQL PostgreSQL

  • 4 CPU, 16 GB RAM, 500 GB Disk

  • HA enabled

  • Cost: €626 per month

Google PubSub

  • Estimated 10GiB data per day

  • 12 subscriptions

  • 7 day retention

  • Cost: €159 per month

Risks

A service being used by multiple channels and multiple customers is a risk. The content service is designed to be used by multiple channels and multiple customers. This means that the service must be able to handle a large number of requests and must be able to scale to meet the needs of the customers.

Except for the content service all other services are using the content service as a dependency. This means that if the content service is down or unavailable, all other services will be affected. This can lead to a cascading failure of the entire system.

Mitigation

  • Dedicating resources for specific customers if needed. If the bottleneck is within the workers we can dedicate a worker pool for a specific customer.

  • If required it would also be possible to create an instance of the service based on the source if we see that one source is causing issues. This would be a last resort and should be avoided if possible.

1. content-service_register-content at lucid: https://lucid.app/lucidchart/beb0b828-5cd5-4909-94b3-3379b0493eb4
2. content-service_prepare-contact-content at lucid: https://lucid.app/lucidchart/1042647e-af23-4610-9c06-212502d1a188