1 of 18

S3-Load Balancing

Alex Walender

​

2 of 18

Motivation

  • One popular data storage in the Cloud is S3-Storage via Ceph.
  • Rados-Gateway
  • Requests are made via HTTP/S
    • GET
    • PUT
    • UPDATE
    • DELETE

​

3 of 18

Motivation

  • Each User is using a single URL for accessing the storage (RGW):

https://openstack.cebitec.uni-bielefeld:8080

​

4 of 18

Implications

  • In effect, there is only one Endpoint active for all user.

​

5 of 18

Implications II

  • Although there is load balancing for OpenStack-Services, traffic only goes and comes from one Controller.

​

6 of 18

Implications III

  • Peaks were a lot worse a few weeks ago, causing:
    • Slow transfer rates
    • Loss of transmission / connection

​

​

  • Controller have too much work to do:
    • Manage vRouter (350 Router with iptables)
    • Manage Message-Queuing
    • Manage OpenStack API
    • Handle RGW-Traffic
    • etc pp.

​

root@pulsnitz:~# ethtool -S fiber0if | grep port.rx_dropped

port.rx_dropped: 170369127

​

​

7 of 18

Project Connectivity Layer 2 - 3

8 of 18

Project Connectivity New Controller

9 of 18

Traffic Interception

  • Intercept traffic to RGW via DNAT.
  • Each HV runs own HAProxy.
  • Custom HAProxy enables true LoadBalancing.
  • Old controller gets load relieve.

​

10 of 18

Network Address Translation

  • When a Package spawns anywhere destined for the RGW, rewrite it with an address of our choice!

​

iptables -t nat -A PREROUTING

-p tcp -d 129.70.51.3 --dport 8080

-j DNAT

--to-destination

$CUSTOM_HAPROXY:443

​

11 of 18

Connection Tracking

  • Intercepted connections needs to be marked, for each project.
  • Ensures project isolation and a way back to the VM.

​

iptables -t mangle -A PREROUTING

-i brqb5fa1c6f-17 -m conntrack --ctstate NEW -j MARK --set-mark 1337

​

iptables -t mangle -A PREROUTING -m mark --mark 1337 -j CONNMARK --save-mark

​

12 of 18

Return Route

  • Our custom HAProxy creates a response.
  • Problem: How does the response gets back to the (private) VM?

​

​

​

​

​

​

to wich Project.

​

13 of 18

Return Route

  • Our custom HAProxy creates a response.
  • Problem: How does the response gets back to the (private) VM?

​

​

​

​

​

​

​

​

root@berkel:~# ip route show table 1337

192.168.0.9 dev brqb5fa1c6f-17 scope link

192.168.0.76 dev brqb5fa1c6f-17 scope link

192.168.0.4 dev brqb5fa1c6f-17 scope link

​

14 of 18

Problem Martian Packages

  • We “inject” Packages back from an unexpected source.
  • Linux thinks, that this is suspicious and drops the response as “martian”.
  • Must be disabled manually for each project on the HV.

root@berkel:~# sysctl -w net.ipv4.conf.brqb5fa1c6f-17.rp_filter=0

​

15 of 18

Problem Martian Packages

16 of 18

Problem Martian Packages

17 of 18

Summary

  • Is promising, but needs some dynamic configuration of HVs.
  • No configuration needed for cloud users, everything is in the background.

​

​

​

​

18 of 18

Summary

  • Is promising, but needs some dynamic configuration of HVs.
  • No configuration needed for cloud users, everything is in the background.

​

​

  • Beta testers needed before using it in production.
    • Est. beginning of November.

​