HBase-7433 Report

1. Symptom

Exception thrown during data put:

Problem inserting row into HBase: Failed 1 action: servers with issues: slave2.hadoop:60020,

at org.pentaho.di.trans.steps.hbaseoutput.HBaseOutput.processRow(HBaseOutput.java:316)

at org.pentaho.di.trans.step.RunThread.run(RunThread.java:50)

at java.lang.Thread.run(Unknown Source)

Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: servers with issues: slave2.hadoop:60020,

at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1601)

org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1377)

at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:916)

at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:772)

at org.apache.hadoop.hbase.client.HTable.put(HTable.java:747)

at org.pentaho.hbase.shim.common.CommonHBaseConnection.executeTargetTablePut(CommonHBaseConnection.java:732)

at org.pentaho.di.trans.steps.hbaseoutput.HBaseOutput.processRow(HBaseOutput.java:307)

1.1 Severity

Blocker

1.2 Was there exception thrown?

Yes. There are two exceptions:

1. Problem inserting row into HBase: Failed 1 action: servers with issues: slave2.hadoop:60020,

at org.pentaho.di.trans.steps.hbaseoutput.HBaseOutput.processRow(HBaseOutput.java:316)

…

and

hbase-default.xml file seems to be for and old version of HBase

at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:81)

at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:114)

at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:145)

at org.apache.hadoop.hbase.security.User$HadoopUser.<init>(User.java:243)

1.2.1 Were there multiple exceptions?

Yes

1.3 Scope of the failure

Affecting the troubled clients

2. How to reproduce this failure

2.0 Version

0.94.1

2.1 Configuration

Not setting hbase.defaults.for.version.skip (default to false)

2.2 Reproduction procedure

Run HBase.

Upgrade the version of HBase and run the clients with newer version

All the newer-version clients will fail

2.2.1 Timing order

Single event

2.2.2 Events order externally controllable?

Yes

2.3 Can the logs tell how to reproduce the failure?

Yes

2.4 How many machines needed?

1 (client + RS)

3. Diagnosis procedure

3.1 Detailed Symptom (where you start)

Log. Be careful and you should find the cause:

“hbase-default.xml file seems to be for and old version of HBase

at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:81)

at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:114)

at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:145)

at org.apache.hadoop.hbase.security.User$HadoopUser.<init>(User.java:243)”

3.2 Backward inference

Starting from there, map to the source:

private static void checkDefaultsVersion(Configuration conf) {

if (conf.getBoolean("hbase.defaults.for.version.skip", Boolean.FALSE)) return;

String defaultsVersion = conf.get("hbase.defaults.for.version");

String thisVersion = VersionInfo.getVersion();

if (!thisVersion.equals(defaultsVersion)) {

throw new RuntimeException(

"hbase-default.xml file seems to be for and old version of HBase (" +

defaultsVersion + "), this version is " + thisVersion);

}

If you read the documents describing “hbase.defaults.for.version.skip”:

“Set to true to skip the 'hbase.defaults.for.version' check. Setting this to true can be useful in contexts other than the other side of a maven generation; i.e. running in an ide. You'll want to set this boolean to true to avoid seeing the RuntimException complaint: "hbase-default.xml file seems to be for and old version of HBase (\${hbase.version}), this version is X.X.X-SNAPSHOT"

Default: false”

So the issue will be solved by setting it to false.

4. Root cause

Configuration error. The user upgraded the hbase client version, but not setting “hbase.defaults.for.version.skip” to true.