Messaging Building Block - A GovStack Service

Presentation

The following is a brief deep dive into the services of Messaging Building Block - A GovStack Service. Its core functionalities and API-s.

Messaging Building Block

Martin Karner

Building Block Lead

Digital Governance Expert, Editor of the Digital Government Strategy of Estonia

and

Rainer Türner, Lead Architect (remote)

Topic to be covered

1. Description of the Building Block
2. Important Terminology
3. Key Digital Functionalities
4. Cross-cutting Requirements
5. Functional Requirements
6. Data Structure
7. Service APIs
8. Internal workflows
9. Brief description of use cases
10. Scope
1. Current scope
2. Future and Out of scope

11. Additional materials, link to the specifications

Description of the Building Block

​

​

1. Messaging is a microservice in the GovStack ecosystem that can be described as a “dump pipe” for unicasting, multicasting or broadcasting any message, alert or notification out and into the GovStack applications. The citizens can be reached through the existing external messaging apps and services, such as WhatsApp, Telegram etc, but also SMS, e-mail etc. Messages are asynchronous.
2. The communication may be initiated by events triggered by sensory or tracking devices, user activity or backend automation applications. The communication system may be preconfigured with indexed standard messages to automatically form and send appropriate messages based on events. Data repositories can be used to store the content of messages mediated via the interactive communication.

Important Terminology

​

​

​

​

Definitions and terminology of the Messaging Building Block (BB)

Messaging BB - is a set of specifications for an interoperable, two-way asynchronous message delivery microservice in the GovStack ecosystem.

Asynchronous message delivery - messaging BB is creating a channel for communication that has the single function of unicasting, multicasting or broadcasting any message, alert or notification out and into the GovStack applications.

“dump pipe” - description of the Messaging BB delivery modality, where the messages are delivered “as is”, without accessing the content of the message at any point.

Security by design - security is guaranteed by three layers: 1) by design of the BB, 2) and as the parties to a message need to comply to the cross-cutting security requirements, 3) data formats and security clearances to access the services through the IM (security server(s)).

Out of scope

​

​

​

​

Out of Scope for the Messenger BB:

• Communication inside/between GovStack building blocks
• Agnostic to payload content - no content/file management
• Scheduling and structuring messages according to some business logic is out of the scope of this Building Block because scheduling is done by Scheduler Building Block and business logic is built by Workflow Building Block
• Fully offline and no internet connection scenarios
• Real-time video and audio conferencing

Government to person (G2P) Examples:

• informing End clients about their registration, enrollment;
• reminding and requesting End clients' confirmation for appointment booking etc;
• delivering information at scale to End clients, f. ex. alerts in emergency contexts.

4.2 Person to Government (P2G) Examples:

• providing data to the government/service;
• getting confirmations, f ex. about status of government services and interactions;
• asking for information about existing government services, enrollment etc.

Communication between different government services (government to government) is out of the scope of this Building Block but can be enabled through the Information Mediator Building Block.

Key Digital Functionalities

​

​

Government to person (G2P) workflow:

Prerequisites and dependencies

The main prerequisite for Person-to-Building Block communication is that there is an existing Sender/Source Building Block with the following properties:

• the relevant credentials and details about the Person/Citizen to be addressed with;
• the content of the message and a reference to a communication channel (contact details) to be used;
• additionally, the service discovery service at the Information Mediator Building Block needs to be active. Information Mediator Building Block publishes the list of available services of the Messaging Building Block to the source GovStack Building Block, i.e. Workflow.

A reference token should be carried throughout the communication session in order to save a point of reference for reverse communication back from the Person to the Building Block. In other words, the main prerequisite for Person-to-Building Block communication is the availability of a communication channel and a reference token.

​

Key Digital Functionalities

​

​

Key Digital Functionalities

​

​

4.2 Government to Person (G2P) Workflow

SequenceDiagram

Workflow BB-->>Information Mediator: Trigger event on<br />specific day and time

Information Mediator-->>Messaging BB: Initate request with<br />messaging data

Messaging BB-->>Information Mediator: Confirm message receival<br />and queueing

Messaging BB-->>Communication Channel: Apply rules and policies<br />to process the message

Communication Channel->>Person: Send message

Communication Channel-->>Messaging BB: Publish message status

Messaging BB-->>Information Mediator: Publish message Status

1. Use de-normalized databases (REQUIRED)

There are two main reasons for using denormalized databases instead of traditional normalized databases. First, to significantly improve the performance of read queries. This is highly important as databases of the Messaging Building Block will be under a heavy load of requests that in many cases are expected to respond near real-time. Second, to be able to export the content of databases in the format of time-series databases

• Archive Content of production-level databases continuously (RECOMMENDED)

By having this capability, it is possible to delete the content of processed data from production-level server databases right after they have been securely archived. This is important to reduce the impact of potential data breaches by security incidents

• Time-series databases should be convertible to ready-to-use relational databases and vice versa (RECOMMENDED)

If done so, using archived datasets, either fully or partially, allows to recover and/or re-engineer lost databases

• Enable message replication (RECOMMENDED)

In order to prevent single points of failure, messages must be replicated on at least 2 different service providers.

• Support policy configuration (REQUIRED)

Admin of the room must be able to choose the policy profile with the configuration of the message provider, e.g. retrial frequency

• Support queuing mechanism (REQUIRED)

Unsent and unsuccessfully delivered messages must remain in a queue until being successfully delivered or otherwise permanently processed

• Messages Delivery Statuses

Pending (initial state for all messages waiting to be queued); Queued (messages that are in the queue to be sent); Sent (messages with proper confirmation that was sent to the provider); Delivered (messages with proper confirmation that was delivered to the end-user); Errored (messages with error during delivery); Failed (messages that are errored and we gave up sending).

• Delivery business rules

Messages not delivered in a period of 24 hours (any errored or queued messages more than 24 hours old must be labelled as failed and go out of the queue); Messages retrial (Errored messages must be retried for 24 hours).

• Implement using stateless and horizontally scalable components (REQUIRED)

Technical components (with the exception of relational databases) must be stateless in the sense of the Service statelessness principle.

Cross-cutting

​

​

​

​

​

​

Requirements

​

​

​

​

See for more cross-cutting requirements the Gitbook repo:

5 Cross-Cutting Requirements - bb-messaging (gitbook.io)

Cross-cutting

​

​

​

​

​

​

​

Requirements

​

​

​

​

Functional

​

​

​

​

​

​

​

Requirements 1/2

​

​

​

Government to person (G2P)

Queuing

The queuing process varies according to the policy and message type attached to the messaging, if an emergency messaging type applies to it, the messaging gets prioritized, otherwise, the FIFO algorithm (First In, First Out) applies to other message types. It saves all the requests for further processing, furthermore, works in the form of an "AS IS" process analysis without changing anything on the request. For more properties of the message queueing mechanism, see the specification: 6 Functional Requirements - bb-messaging (gitbook.io)

Validation and Verification

Messaging requests go through a final check to be clean of defects and inconsistencies, to check with external systems as necessary: for duplicates, data types, inconsistencies and if detected, error messages are used as a response. See the specs for detailed descriptions: 6 Functional Requirements - bb-messaging (gitbook.io)

Workflow and Scheduling

In relation to batch logic, messages are scheduled against the availability of systems, throughput limitations, and rules set by programs.

See for the details: 6 Functional Requirements - bb-messaging (gitbook.io)

Functional

​

​

​

​

​

​

​

Requirements…2/2

​

​

​

Person to Government (P2G)

Messaging request initiation

Handles the initiation of input data processing from all the API messaging calls and makes the API access control verification from other Building Blocks to the Messaging Building Block and vice versa as well as within the Messaging Building Block. See the specific functional requirements 6 Functional Requirements - bb-messaging (gitbook.io

​

This request could come from two sources: external or internal. An external source could be another GovStack Building Block (e.g. the Registration Building Block). Either source must be appropriately authenticated and authorized to initiate the request. The request must contain at a minimum: the contact address (email, phone number, etc.), the message type, the content of the message, and the initiating source’s unique transaction ID.

Batch logic

Finds unprocessed requests from a database. Prepares each request for actual processing. Requests may come as single or batch messages and every message needs to be treated as a separate entry. Prepares unprocessed requests for actual processing.

Audit trails and Security layer

The security layer ensures that the content of messages and interactions with other Building Blocks are encrypted in transit. The security layer follows these requirements:

• The communication must be TLS-secured using client authentication, Transport Layer Security protocol (TLS) 1.2 and above should be used to protect the confidentiality and integrity of the data in transit
• Personal profiles must never be disclosed to any unauthorized party.

Service APIs

​

​

​

and Data Structure

​

All communication MUST BE via RESTful services

​

• Functions as a Service (FaaS)
• Read the contents of a file, generate a unique ID, etc.
• Can be re-used throughout the developments
• Database queries
• Every single database query is a separate .sql file
• Can be re-used throughout the developments
• Generic services
• User authentication, session management, etc.
• Can be re-used throughout the developments
• Client-specific services
• Will not be used anywhere else, usually not re-usable in other projects

​

As a result, every single data stream can be easily tracked, updated, manipulated, and replaced, making the whole solution

• highly sustainable
• free of risk of ending up in a vendor lock

Internal workflows

​

​

​

1. Message received
• Only allowed participants
• Standard API requests
2. Message processed internally
• Verify requestor, permissions, etc
• Find the correct path to continue
• Logging
3. Message sent
• Via REST calls
• Protocol manipulation for every service provider
• Service providers propose and keep up-to-date their service-specific protocol manipulation configuration files
• Message sent on the basis of as is
• Message carriers can be easily added, changed, removed, blocked, etc. at any given time
4. Overview provided for the message sender
• A complete overview of the status of sent messages
• scheduled, processing, deferred, bounced, not sent, delivered

Alert messages to concerned institutions and persons. The types of message(s) include:

1. Voice Alert Message

· With the help of Telecom Service Providers using Outbound Dialler (OBD) infrastructure in local languages, or in rare cases with the help of radio.

• Use of chatbot and voicebot,
• With visualisation aid for impaired/handicapped citizens (captions, subtitles etc)
• Text Alert Message

· With the help of Telecom Service Providers using available channels

· With the help of an email database/GW

· With the help of social media/website

​

• Information about existing government services, enrollment into a particular service.

For example, availability of social program and enrollment into the program (through a sectoral e-service).

• Appointment booking confirmation with two levels of confirmation: a) Immediate confirmation: received the message. b) deposited a message, and have given the message a token number. On the basis of this token, I can pick up the message/reply later. c) Reminder message is already pre-configured and an event will trigger it. d) Event notification, trigger: for example: please join the call.

Examples of use cases

1. Design and functionality for E-mail messages and SMS
2. Testing the API-s and functionalities, Sandbox
3. Processing, reviewing and implementing comments and changes from the last review

Current Scope

Future Scope

1. Integration of the comments of the last review.
2. Adding functionalities and channels, based on the review.
3. Bridges to the existing service providers, such as WhatsApp etc.

Future Scope

​

​

​

Author: Martin Karner, Rainer Türner

URL link to specifications/requirements:

​

Messaging - bb-messaging (gitbook.io)

THANK YOU