Cassandra-2563

https://issues.apache.org/jira/browse/CASSANDRA-2563

1. Symptom

Error starting up a cassandra cluster after creating a table in the system keyspace. A keyspace is like a configuration for CQLSH (cassandra shell). A system keyspace is a configuration that user cannot modify.

The error message shows:

“Attempt to assign id to existing column family.”

 (confusing and irrelevant error message).

1.1 Severity

Critical

1.2 Was there exception thrown?

Yes

1.2.1 Were there multiple exceptions?

No

1.3 Was there a long propagation of the failure?

No

1.4 Scope of the failure

Early termination

2. How to reproduce this failure

2.0 Version

0.8.0 beta2

2.1 Configuration

Basic configuration

2.2 Reproduction procedure

      0.   rm -rf /var/lib/cassandra/*

      0.   rm -rf /var/log/cassandra/*

      0.   Start Cassandra

  1. In cqlsh, create a column family and insert data in the “system” keyspace
  2. Quit cqlsh
  3. Kill Cassandra
  4. Startup Cassandra and get error

2.2.1 Timing order

In the above order

2.2.2 Events order externally controllable?

Yes

2.3 Can the logs tell how to reproduce the failure?

No

2.4 How many machines needed?

One Node

3. Diagnosis procedure

3.1 Detailed Symptom (where you start)

log:

INFO 18:38:24,509 Loading schema version 087af100-7034-11e0-0000-242d50cf1fde
ERROR 18:38:24,774 Exception encountered during startup.
java.io.IOError: org.apache.cassandra.config.ConfigurationException: Attempt to assign id to existing column family.
        at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:489)
        at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:138)
        at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:313)
        at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
Caused by: org.apache.cassandra.config.ConfigurationException: Attempt to assign id to existing column family.
        at org.apache.cassandra.config.CFMetaData.map(CFMetaData.java:126)
        at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:485)
        ... 3 more
Exception encountered during startup.
java.io.IOError: org.apache.cassandra.config.ConfigurationException: Attempt to assign id to existing column family.
        at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:489)
        at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:138)
        at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:313)
        at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
Caused by: org.apache.cassandra.config.ConfigurationException: Attempt to assign id to existing column family.
        at org.apache.cassandra.config.CFMetaData.map(CFMetaData.java:126)
        at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:485)

4. Root cause

System keyspace is not user-modifiable so the user should not modify it. The original code does not prevent the situation where the user wants to modify the system keyspace. Therefore, an error occurs the next time when they restart the system.

4.1 Category:

Sematic

4.2 Are there multiple fault?

No

5. Fix

Index: src/java/org/apache/cassandra/service/ClientState.java
===================================================================
--- src/java/org/apache/cassandra/service/ClientState.java        (revision 1096804)
+++ src/java/org/apache/cassandra/service/ClientState.java        (working copy)
@@ -129,7 +129,11 @@
    {
        validateLogin();
        validateKeyspace();
-        
+
+        // hardcode disallowing messing with system keyspace
+        if (keyspace.equalsIgnoreCase("system"))
+            throw new InvalidRequestException("system keyspace is not user-modifiable");
+
         resourceClear();
        resource.add(keyspace);
        Set<Permission> perms = DatabaseDescriptor.getAuthority().authorize(user, resource);

5.1 How?

If the user wants to modify the system keyspace, an invalid request exception will be thrown and an error message will be given to the user.