1 of 33

ARGO Messaging Service

Messaging as a Service

​

2 of 33

GRNET ActiveMQ

Messaging Service

  • A few years ago the Messaging Service was based on ActiveMQ
    • Multi-national ActiveMQ Message Broker Network
  • Used by infrastructure services mainly
    • Monitoring, accounting, alerting
  • Stable Service
    • Transporting ~1 Billion messages per year
  • Very low number of incidents and always on the client side

​

3 of 33

GRNET ActiveMQ

Messaging Service

  • A few years ago the Messaging Service was based on ActiveMQ
    • Multi-national ActiveMQ Message Broker Network
  • Used by infrastructure services mainly
    • Monitoring, accounting, alerting
  • Stable Service
    • Transporting ~1 Billion messages per year
  • Very low number of incidents and always on the client side

​

Why do you want to change it!!

4 of 33

GRNET ActiveMQ

Messaging Service

  • Used by infrastructure services mainly
  • Very low number of incidents and always on the client side
  • The Message Broker Network is a complex service
    • Coordination of upgrades is a demanding activity
    • Changes to the stable configuration are very time consuming �(e.g. support for ACLs)

​

Why do you want to change it!!

5 of 33

ARGO Messaging Service

GRNET Unified Messaging

To a scalable ΗTTP Messaging Service With:

    • an HTTP API for client access
    • Transparent scalability & high availability
    • Access controls implemented at the API layer
    • Multi-tenant support
    • Instrumentation at the API layer�

​

From a Message Broker Network

Messaging as a Service for everyone !

AMS

6 of 33

ARGO Messaging Service

Fundamentals

​

Users & Roles

Resource Types

Project

A named resource to which messages are sent by publishers.

Topic

A named resource representing the stream of messages from a single, specific topic

Subscription

The combination of data and attributes

Message

Client that consumes msgs from a topic.

Subscriber

Client that publishes msgs to a topic.

Publisher

A list of service-wide roles. Can be extended. (action is a permission)

Roles

a basis of organizing and isolating groups of users & resources

A mechanism that support schema validation

Schemas

activity of the resources

Metrics

7 of 33

ARGO Messaging Service

Flow

​

Project 1

Publisher A

Message A

Message A

Subscriber A

Topic A

Subscription A

consumer

publisher

Publisher sends messages to a topic .

Messages are persistently stored in the messaging service until they are delivered to subscribers. The service forwards messages from a topic to all of its subscriptions.

​

Subscriber receives pending messages from the subscription. Subscriber sends an acknowledgement that he processed the message.

When the message is acknowledged, it is deleted from the AMS.

​

​

8 of 33

ARGO Messaging Service

Main Features

​

Always Follow the Google PUB/SUB features

  • Improved monitoring/logging
  • Internal optimizations

​

​

all

  • At-Least-Once delivery: delivers each published message at least once for every subscription
  • Support of message attribute: key/value pair
  • Support of multiple subscriptions per topic: topics and subscriptions provide a one-to-many form of communication, in a publish/subscribe pattern.
  • Replay messages: that have been acknowledged by seeking to a timestamp.
  • Replicate messages on multiple topics: Republisher script that consumes and publishes messages for specific topics

​

Ease of use

  • Schema Support:
  • Metrics to help us understand the activity of our resources (publishing-rate, consumption-rate)

​

9 of 33

ARGO Messaging Service

Straightforward model with clear semantics

​

​

Project 1

Publisher A

Message A

Message A

Subscriber A

Topic A

Subscription A

Publisher B

Message B

Message B

Subscriber B

Topic B

Subscription B1

Message B

Subscriber C

Subscription B2

10 of 33

ARGO Messaging Service

Message

​

​

​

​

​

​

The combination of data and attributes that a publisher sends to a topic and is delivered to subscribers.

Message

{ � "messages":[ � { � "attributes":[ � "key1":"infrastructure",� "key2":"testing"� ],� "data":"U28geW91IHdlbnQgYWhlYWQgYW5kIGRlY29kZWQgdGhpcywgeW91IGNvdWxkbid0IHJlc2lzdCBlaCA/"� }� ]�}

The data that you like to send via the service. The value of the data property must be always encoded in base64 format.

Optional key value pair of metadata you desire.

Attributes:

Data:

11 of 33

ARGO Messaging Service

New ARGO Messaging Service

​

​

Publishing: Publisher sends messages to a topic .

curl -X POST -H "Content-Type: application/json" -d { POSTDATA } \ https://{URL}/v1/projects/YOURPROJECT/topics/monitoring:publish?key=eyJhbGciOiJIUzI1NiIsInR5..."

"messages":[ � { � "attributes":[ � { � "key1":"infrastructure",� "key2":"testing"� }

], "data":"U28geW91IHdlbnQgYWhlYWQgYW5kIGRlY29kZWQgdGhpcywgeW91IGNvdWxkbid0IHJlc2lzdCBlaCA/"� }

]

POSTDATA

Publisher A

Message A

Message A

Subscriber A

monitoring

monitoring

PROJECT_NAME

​

12 of 33

ARGO Messaging Service

New ARGO Messaging Service

​

​

Publisher A

Message A

Message A

Subscriber A

monitoring

monitoring

Consuming: Subscriber consumes messages from a topic .

curl -X POST -H "Content-Type: application/json" -d POSTDATA https://{URL}/v1/projects/YOURPROJECT/subscriptions/monitoring:pull?keyJhbGciOiJIUzI1NiIsInR5..."

​

{

"maxMessages": "100",

}

​

POSTDATA

PULL MODE

PROJECT_NAME

13 of 33

ARGO Messaging Service

New ARGO Messaging Service

​

​

Publisher A

Message A

Message A

Subscriber A

Consuming: Subscriber consumes messages from a topic .

200 OK

{ � "receivedMessages":[ � { � "ackId":"dQNNHlAbEGEIBE...",� "message":{ � "attributes":[ � { � "key1":"infrastructure",� "key2":"testing"� }� ],� "data":"U28geW91IHdlbnQgYWhlYWQgYW5kIGRlY/",� "messageId":"100309303"� }� },

{ � "ackId":"UQJJJHlAbEGEIBE...",

...� }� ]�}

​

​

Response

PULL MODE

monitoring

monitoring

PROJECT_NAME

14 of 33

ARGO Messaging Service

New ARGO Messaging Service

​

​

Publisher A

Message A

Message A

Subscriber A

Ack msgs: sending message with an array of ackIDs..

PULL MODE

curl -X POST -H "Content-Type: application/json" -d POSTDATA https://{URL}/v1/projects/YOURPROJECT/subscriptions/monitoring:acknowledge?keyJhbGciOiJIUzI1NiIsInR5..."

​

{� "ackIds": [� "dQNNHlAbEGEIBE..."� ],��}

​

POSTDATA

monitoring

monitoring

200 OK

RESPONSE

PROJECT_NAME

15 of 33

ARGO Messaging Service

Pull vs Push

​

​

Publisher A

Message A

Message A

Subscriber A

In a pull subscription, the subscriber explicitly calls the API pull method, which requests delivery of a message in the subscription queue. The AMS responds with the message (or an error if the queue is empty), and an ack ID. The subscriber then explicitly calls the acknowledge method, using the returned ack ID, to acknowledge receipt.

PULL MODE

In a push subscription, the AMS sends a request to the subscriber application, at a preconfigured endpoint. The endpoints HTTP response serves as an implicit acknowledgement:

  • a success response → the message has been successfully processed; AMS can delete it.
  • a non-success response → the AMS should resend it (implicit "nack").

The AMS dynamically adjusts the flow of requests and uses an algorithm to rate-limit retries.

PUSH MODE

monitoring

monitoring

PROJECT_NAME

16 of 33

ARGO Messaging Service

​

The push server(s) are an optional set of worker-machines that are needed when the AMS wants to support push enabled subscriptions.

  • Set of workers that handle push enabled subscriptions
  • They perform the push functionality of a subscription (consume->deliver->ack)
  • Provide a gRPC interface in order to communicate with their api
  • Provide subscription runtime status

​

Push Server

Push WorkFlow

Publisher

Message

AMS

Push Server

Remote Endpoints

Consume Message

Deliver Message

17 of 33

ARGO Messaging Service

New ARGO Messaging Service

​

​

Publisher A

Message A

Message A

Subscriber A

PUSH: Modify the push configuration of a subscription

curl -X POST -H "Content-Type: application/json" -d POSTDATA https://{URL}/v1/projects/YOURPROJECT/subscriptions/monitoring:modifyPushConfig

​

​

{

"topic":"projects/YOURPROJECT/topics/monitoring",

"pushConfig":{

"pushEndpoint":"https://example.com/alerts/callback",

}

}

​

POSTDATA

200 OK�{� "name": "projects/EGI/subscriptions/monitoring",� "topic": "projects/EGI/topics/monitoring",� "pushConfig": {� "pushEndpoint": "https://example.com/alerts/callback" � },� "ackDeadlineSeconds": 10 �}��

​

RESPONSE

PUSH MODE

monitoring

monitoring

PROJECT_NAME

18 of 33

ARGO Messaging Service

New ARGO Messaging Service

​

​

Publisher A

Message A

Message A

Subscriber A

Consuming: Subscriber consumes messages.

curl -X POST -H "Content-Type: application/json" -d POSTDATA https://example.com/alerts/callback

{� "message": {� "attributes": {� "string-value": "string-value",� // ... more attributes� },� "data": "base64-no-line-feeds-variant-representation-of-payload",� "message_id": "string-value",� "publish_time": "string-value",� },� "subscription": "string-value"�}

​

POSTDATA

PUSH MODE

monitoring

monitoring

PROJECT_NAME

19 of 33

AMS Schema Support

​

On demand mechanism that enables a) the definition of the expected payload schema, b) the definition of the expected set of attributes and values and c) the validation

​

Schemas

curl -X POST -H "Content-Type: application/json" -d POSTDATA https://{URL}/v1/projects/YOURPROJECT/schemas/SCHEMA

{

"type": "json",

"schema":{

"type": "object",

"properties": {

"name": { "type": "string" },

"email": { "type": "string" },

"address": { "type": "string" },

},

"required": ["name", "email"]

}

}

​

​

POSTDATA

Publishers

Acceptable Message

Messages

Schema(filter)

Topic

20 of 33

AMS Metrics

​

A number of metrics for each resource and the relationships between the resources

​

  • Project.number_of_topics
  • Project.number_of_subscriptions
  • Project.user.number_of_subscriptions
  • Project.number_of_daily_messages
  • Subscription.number_of_messages
  • Subscription.number_of_bytes
  • Subscription.consumption_rate
  • Topic.number_of_messages
  • Topic.number_of_bytes
  • Topic.publishing_rate

Metrics

  • Daily Message Average
  • Weekly/Monthly messages
  • messages
  • Percentage value that displays the CPU usage of ams service in the specific node
  • Percentage value that displays the Memory usage of ams service in the specific node

​

​

Operational

Overall

21 of 33

ARGO Messaging Service

Some AMS API Endpoints

​

​

/projects

/projects/<project>

Manage projects

/topics

/topics/<topic>

/topics/<topic>:publish

/topics/<topic>:acl

/topics/<topic>:modifyAcl

Manage topics

/users

/users/<user>

/users/<user>:refreshToken

​

Manage users

/subscriptions

/subscriptions/<sub>:pull

/subscriptions/<sub>:acknowledge

/subscriptions/<sub>:modifyPushConfig

/subscriptions/<sub>:modifyacl

/subscriptions/<sub>:acl

​

Manage subscriptions

22 of 33

to AMS architecture

From the AMS API

23 of 33

ARGO Messaging Service

​

​

  • Durability: provides very high durability, and at-least-once delivery, by storing copies of the same message on multiple servers.
  • Scalability: It can handle increases in load without noticeable degradation of latency or availability. Nodes can be easily added or removed on demand without interrupting the availability of the service.
  • Availability: it deals with different types of issues, gracefully failing over in a way that is unnoticeable to end users. Failures can occur in hardware, in software, and due to load.

​

24 of 33

ARGO Messaging Service

​

  • High Availability set up
    • Simple HTTP HA setup
    • Scale up/down according to demand
    • Addition/removal of nodes transparently for end users
    • Operations friendly �

​

Zookeeper Cluster: is a centralized service for maintaining configuration information, naming, providing distributed synchronization.

Kafka Cluster: is a distributed publish-subscribe messaging system that is designed to be fast, scalable, and durable.

MetaData store Cluster: a distributed metadata store based on mongodB

​

​

​

​

25 of 33

from AMS architecture

to AMS Components

26 of 33

AMS Components

  • Argo-ams-library: A simple library to interact with the ARGO Messaging Service.
  • Argo-AuthN: Argo-authn is a new Authentication Service. This service provides the ability to different services to use alternative authentication mechanisms without having to store additional user info or implement new functionalities. The AUTH service holds various information about a service’s users, hosts, API urls, etc, and leverages them to provide its functionality.
  • AMS Metrics: Metrics about the service and the usage.

​

27 of 33

From Components

to AMS Use Cases

​

28 of 33

ΑRGO Messaging Service - Who Uses it

  • ARGO: Sends the messages from the monitoring engine to the other components.
  • Accounting: Use of AMS as a transport layer for collecting accounting data from the Sites.
  • Operations Portal: Reads the alarms from predefined topics, store them in a database and displays them in the operations portal.
  • FedCloud : Use of AMS as a transport layer of the cloud information system. It make use of the ams-authN. AppDB consumer of the cloud information system.

​

29 of 33

Use Case: Monitoring

Migrating traffic from the old ActiveMQ Message Broker Service

​

3 Subscribers

3 Topics

3++ Subscriptions

~1450 hosts

pushing 1,3M messages/day

1 Billion messages / year

30 of 33

Use Case: Log aggregation

Log aggregation from the national GRNET infrastructure

2 Subscribers

1 Topic

2 Subscriptions

~1000 devices

pushing 375M logs/month

4.5 Billion logs/year

31 of 33

ARGO Messaging Service

​

  • Balancing workloads in network clusters�For example, a large queue of tasks can be efficiently distributed among multiple workers.
  • Implementing asynchronous workflows�For example, an order processing application can place an order on a topic, from which it can be processed by one or more workers.
  • Publishing of monitoring results�Monitoring Engines can push monitoring results through the ARGO Monitoring Service
  • Publishing of accounting data�Act as the transport layer for the secure exchange of accounting information

32 of 33

ARGO Messaging Service

​

Always Follow the Google PUB/SUB features

Operated as production service since September 2016

Support - Help

all

house-keeping

Ease of use

Github: https://github.com/ARGOeu/argo-messaging/

Documentation

https://argoeu.github.io/argo-messaging/

33 of 33

Thank you

Questions?