HBase-4220 Report

1. Symptom

When connecting to Region server, HBase is not caching the DNS lookup result, resulting flooding the DNS service. (DDOS of the DNS server)

https://issues.apache.org/jira/browse/HBASE-4220

1.1 Severity

Critical

1.2 Was there exception thrown?

No.

1.2.1 Were there multiple exceptions?

No

1.3 Scope of the failure

The entire network is slow

2. How to reproduce this failure

2.0 Version

0.92.0 --- manually reverse the patch.

2.1 Configuration

Standard

2.2 Reproduction procedure

Run the YCSB (or any long running) workload.

2.2.1 Timing order

No particular timing order requirement

2.2.2 Events order externally controllable?

Yes.

2.3 Can the logs tell how to reproduce the failure?

Sure. Just lots of request.

2.4 How many machines needed?

Theoretically, just 1 (master + region server)

3. Diagnosis procedure

3.1 Detailed Symptom (where you start)

This is a performance bug, so we will notice the performance degradation, and use networking tools such as netstat to see it’s because too many DNS requests.

3.2 Backward inference

Once we realize that this is a DNS problem, we will examine how HBase performs DNS lookups. Before the fix, HBase is using this function for the DNS lookup:

public HServerAddress getServerAddress() {
    return new HServerAddress(this.hostname, this.port);
  }

Which further calls:

 /**

   * @param hostname Hostname

   * @param port Port number

   */

  public HServerAddress(final String hostname, final int port) {

    this(getResolvedAddress(new InetSocketAddress(hostname, port)));

  }

  private static InetSocketAddress getResolvedAddress(InetSocketAddress address) {

     String bindAddress = getBindAddressInternal(address);

     int port = address.getPort();

     return new InetSocketAddress(bindAddress, port);

   }

So everytime, it is creating a new socket!

4. Root cause

Every DNS lookup HBase is creating a new socket; Instead it should simply cache the DNS lookups.

4.1 Category:

Semantic

5. Fix

5.1 How?

Every DNS lookup we should use another api -- which performs the caching:

Addressing#createHostAndPortStr(String, int)}

   */

  public synchronized String getHostnamePort() {

    if (this.cachedHostnamePort == null) {

      this.cachedHostnamePort =

        Addressing.createHostAndPortStr(this.hostname, this.port);

    }

    return this.cachedHostnamePort;

  }