HBase-2399 Report

1. Symptom

When users force a split on a region, instead of splitting the largest Store, it picks the first Store to split.

As a consequence, it will cause unbalanced load and performance degradation.

1.1 Severity

Critical

1.2 Was there exception thrown?

No.

1.2.1 Were there multiple exceptions?

No

1.3 Scope of the failure

Affect any split region’s access performance

2. How to reproduce this failure

2.0 Version

0.92.0 (reverse the patch)

mvn test -Dtest=TestAdmin#testForceSplitMultiFamily

2.1 Configuration

Standard

2.2 Reproduction procedure

Client force a split on a region

2.2.1 Timing order

Single event

2.2.2 Events order externally controllable?

Yes

2.3 Can the logs tell how to reproduce the failure?

Yes.

2013-08-13 10:34:04,132 DEBUG [PRI IPC Server handler 6 on 37860] regionserver.CompactSplitThread(167): Split requested for testForceSplit,,1376404426884.f75e89f4     a36dacab22598e275f225ef6..  compaction_queue=(0:0), split_queue=0

2013-08-13 10:34:04,133 INFO  [RegionServer:1;localhost,37860,1376404417946-splits-1376404444132] regionserver.SplitTransaction(216): Starting split of region tes     tForceSplit,,1376404426884.f75e89f4a36dacab22598e275f225ef6.

2.4 How many machines needed?

1. RS + client

3. Diagnosis procedure

3.1 Detailed Symptom (where you start)

Users will notice the split didn’t take good effect. Then it is pretty easy to located the log:

 2013-08-13 10:34:04,132 DEBUG [PRI IPC Server handler 6 on 37860] regionserver.CompactSplitThread(167): Split requested for testForceSplit,,1376404426884.f75e89f4a36dacab22598e275f225ef6..  compaction_queue=(0:0), split_queue=0

2013-08-13 10:34:04,133 INFO  [RegionServer:1;localhost,37860,1376404417946-splits-1376404444132] regionserver.SplitTransaction(216): Starting split of region testForceSplit,,1376404426884.f75e89f4a36dacab22598e275f225ef6.

3.2 Backward inference

The logs above indicates that at least the split request is received. But it just didn’t pick the correct (largest) region to split.

It’s not hard to locate the code that picks the region to split by the dev:

/**

   * @return the key at which the region should be split, or null

   * if it cannot be split. This will only be called if shouldSplit

   * previously returned true.

   */

  byte[] getSplitPoint() {

    Map<byte[], Store> stores = region.getStores();

    byte[] splitPointFromLargestStore = null;

    long largestStoreSize = 0;

    for (Store s : stores.values()) {

        splitPointFromLargestStore = s.getSplitPoint();

        if (splitPointFromLargestStore != null) {

                return splitPointFromLargestStore;

        }

    }

    return null;

}

  --- It always picks the first Store to split.

4. Root cause

Instead of picking the largest store to split, it only picks the first store to split.

4.1 Category:

Semantic

5. Fix

5.1 How?

Pick the largest Store:

+   for (Store s : stores.values()) {

+      byte[] splitPoint = s.getSplitPoint();

+      long storeSize = s.getSize();

+      if (splitPoint != null && largestStoreSize < storeSize) {

+        splitPointFromLargestStore = splitPoint;

+        largestStoreSize = storeSize;

+      }

+    }