Proposal to move Tricircle forward

for its big-tent application

TC Concerns during Tricircle big-tent application:

​

Worried about API in-consistency between Nova API-GW/Cinder API-GW and Nova/Cinder, how much of Nova/Cinder API will re-implemented, and break the community single API implementation, and lead to interoperability-issue.

​

Neutron provides plugin mechanism, so no worry on the API area.

*Note, XJob and Admin API of Tricircle are not shown here to simplify expression.

Need community wide discussion and consensus, WIP

Tricircle

Proposal 1: Add plug-in mechanism in Nova/Cinder

Move API-gateway part away from Tricircle, and form two independent and decoupled projects:

​

Tricircle: Dedicated for cross Neutron networking automation in multi-region OpenStack deployment, run without or with Trio2o. Try to become big-tent project in the application of https://review.openstack.org/#/c/338796/.

​

​

Trio2o: Dedicated to provide API gateway for those who need single Nova/Cinder API endpoint in multi-region OpenStack deployment, run without or with Tricircle. Live as non-big-tent, non-offical-openstack project, just like Tricircle toady’s status. And not pursue big-tent only if the consensus can be achieved in OpenStack community, including Arch WG and TCs, then decide how to get it on board in OpenStack.

Proposal 2: Move API-gateway part away from Tricircle into new dedicated Trio2o project, and only networking automation part remained in Tricricle.

This is a more doable proposal

Trio2o

Nova

Cinder

Nova

Cinder

Nova

Cinder

Neutron

Neutron

Cinder

Nova

To make Tricircle work independently from Trio2o in multi-region OpenStack deployment, need local Neutron to trigger the cross Neutron networking automation, so Tricircle Local Neutron Plugin which is inherited from ML2 plugin with cross Neutron networking automation triggering enhancement is needed. Tricircle Central Neutron Plugin is the current Tricircle Neutron Plugin, and responsible for cross Neutron networking automation.

Tricircle Central Neutron Plugin

Tricircle Local

Neutron Plugin

Tricircle Local

Neutron Plugin

Cross Neutron networking automation triggering should be moved to Tricircle Local Neutron Plugin

Nova

Cinder

Neutron API Server

Tricircle Local

Neutron Plugin

Nova

Cinder

Neutron API Server

Tricircle Local

Neutron Plugin

Neutron API Server

Tricircle Central Neutron Plugin

Type Driver

Mech

Driver

Type Driver

Mech

Driver

API endpoint exposed to end user

Cross Neutron Networking automation Triggering

Tricircle = Tricircle Local Neutron Plugin(s) + Tricircle Central Neutron Plugin + Admin API + XJob

Nova talks to local Neutron which is deployed in the same OpenStack region.

Tricircle Local Neutron Plugin inherits from ML2 plugin, so all TypeDrivers and MechDrivers should also be able to run under Tricircle Local Neutron Plugin. The Tricircle Local Neutron Plugin can also load other plugins(for example OVN, Dragonflow, etc) for core-plugin interface, so that these plugins can also benefit from cross Neutron networking automation

​

TricircleNetworking components

OpenStack services and components

Cross Neutron Networking automation flow

Why the Tricircle Central Neutron Plugin is needed

Prerequisite: Deal with the cross Neutron networking automation among Multiple OpenStack instances deployed with separate Nova,Cinder,Neutron instances in one site or multiple site, i.e, multi-region mode. (multi-region Nova to use a shared/global Neutron is ok, Tricircle doesn’t stop this way of deployment. In shared Neutron for multi-region scenario, the Neutron with Tricircle Central Neutron Plugin is not needed).

Key challenges:

First we need to create a network/subnet with IP pool. Consider that the tenant's VMs will be allocated in multiple Nova separately and communication among these VMs will happen, so the IP address should not be overlapped. Especially, if we want to create a L2 network across multiple OpenStack clouds, then it's almost impossible to create this L2 network separately in different Neutron. if same pool is used, for example 192.168.1.0/24, then 192.168.1.3 may be allocated by several Neutron at the same time; if we split the pool to multiple smaller parts: for example, 192.168.1.0~192.168.1.100, 192.168.1.101~192.168.1.200, then each sub-pool is fixed and not flexible, and if one of the pool is exhausted, then no new VMs can be attached to the network, but the network has idle addresses in another Neutron. Another use case is IP mobility across OpenStack, which can not be addressed by splitting network IP address pool in another Neutron.

(one real use case, in VMALL, the e-commerce website for mobile phones(several billion USD per year), and will be deployed into multiple clouds at the same time for disaster recovery purpose, a requirement was asked to provide the IP mobility for data back-end to achieve higher service continuity, i.e, when one VM crashed, another VM can be changed to the same IP and provide service immediately in another OpenStack, use same IP is to reduce the reconfiguration for other service component for the IP address, and no need to change security rule. That means the network/subnet/port(IP allocation) should be managed by a same centralized service)

​

it's quite naturally to consider reuse Neutron for these: Neutron’s core API is L2 network management: Network. Subnet, Port... The second is that Nova only wants to work with Neutron but not a new networking service, Nova/Neutron are the core OpenStack services. Network/Subnet/Port should be created in the centralized Neutron with Tricircle Central Neutron Plugin(i.e, current Tricircle Neutron Plugin), if we want to apply these object flexible to multiple Neutrons.

​

Except L2 object, same as L3 object like router, FIP, SEG if the tenant has VMs in multi-region. And the Tricircle Central Neutron Plugin can also act as the coordinator for tenant’s L2/L3 networking automatically, otherwise you have to call different Neutron API’s separately, it’s possible, but it’s error prone and boring for each tenant.

​

​

Why the Tricircle Local Neutron Plugin is needed

Prerequisite: don’t touch any piece of code in Nova and make Tricircle can work independently.

​

Key challenges:

The network/subnet/port will be created in the central Neutron( with Tricircle Central Neutron Plugin), then the UUID of these objects are already created. The VM boot request will have the UUID of network/port from the central Neutron. For the case 1, boot VM with network, then Nova will try to create port in local Neutron, the issue is that local Neutron has no this Network(more than this issue). For the case 2, boot VM with port from the central Neutron, then the UUID is from the central Neutron, and during the VIF plug in Nova, the tap will be identified with the UUID of the port from the central Neutron, but local Neutron can’t recognize this tap for there is no corresponding port with same UUID.

​

Nova will query local Neutron before using network and port, or create port in local Neutron. This is the chance and can be managed by Tricircle Local Neutron Plugin, to make the whole local Neutron can work as usual, the possibility is to enhance the ML2 plugin. So Tricircle Local Neutron Plugin will be inherited from ML2 plugin, and trigger the cross Neutron networking automation.

​

​

Boot VM with network

Local Neutron

(Tricircle Local Neutron plugin)

Central Neutron

(Tricircle Central Neuron plugin)

Local Nova

User

1. create network net1(uuid1)

3. query net1(uuid1) in local Neutron

2. boot vm with net1(uuid1)

5. Create net1 in local Neutron with the same uuid1

4. make sure net1(uuid1) exists

6. create port in net1 in local Neutron

7. create port in net1(uuid1) central Neutron to get IP from global pool

8. create bottom port with same IP/mac and uuid in local Neutron

How the Tricircle Local Neutron Plugin works with central Neutron.

*Local Nova and Local Neutron co-locate in same region

Boot VM with port

Local Neutron

(Tricircle Local

Neutron plugin)

Central Neutron

(Tricircle Central Neuron plugin)

Local Nova

User

1. create port port1(uuid1)

3. query port1(uuid1) in local Neutron

2. boot vm with port1(uuid1)

5. create bottom port with same IP/mac and uuid in local Neutron, create network/subnet accordingly if not exist in local Neutron

4. make sure port1(uuid1) exists, get port information

How the Tricircle Local Neutron Plugin works with central Neutron.

*Local Nova and Local Neutron co-locate in same region

Why the Tricircle Local Neutron Plugin is needed

• ML2 framework doesn’t provide get_network/port_pre/post_commit interface for mechanism driver, so when Nova server passes the top uuid of network/port to local Neutron, local Neutron will complain “resource not found”.
• When booting VM with pre-created port, Nova server will use top port uuid to create tap device, however, since local Neutron server doesn’t allow specifying uuid to create port, port created in local Neutron server will have different uuid, and OVS agent/etc cannot correctly bind the tap device with different uuid.

Why the Tricircle Local Neutron Plugin is needed

• Write a core plugin which inherited from ML2 plugin. Configure local Neutron server to use this plugin
• Overwrite the get_port/network methods, when Nova server queries a resource with top uuid, get the information from central Neutron server
• Overwrite the create_port/create_network method, create local port/network with same uuid as that in central Neutron.

Backup - Trio2o

Trio2o

Nova

Cinder

Nova

Cinder

Remove the networking automation triggering code. No networking functionalities in Nova API-GW anymore

​

Nova API-GW and Cinder API-GW could be enabled or disabled as needed

Backup - Trio2o

Trio2o

Nova

Cinder

Nova

Cinder

Configure one region as master region, all global objects like flavor, volume type, server group, etc will be managed in the master Nova/Cinder service. In Nova API-GW/Cinder API-GW, all requests for these global objects will be forwarded to the master Nova/Cinder, then to get rid of any API overlapping-implementation.

​

The Nova API-GW/Cinder API-GW will be very very lightweight, and no API re-implementation. The value is to provide single API endpoint and tenant dynamic binding for OpenStack instances.