2 of 38

About me

Johan Euphrosine <proppy@google.com>

Google San Francisco

Developer Programs Engineer

Playing with Google Cloud Platform

profiles.google.com/proppy

goo.gl/LWM8Qs

3 of 38

What do they have in common?

goo.gl/LWM8Qs

5 of 38

Same data stack

Google Datacenter

Colossus

BigTable

Megastore

6 of 38

Global

Google Datacenter

Colossus

BigTable

Megastore

7 of 38

Durable

Google Datacenter

Colossus

BigTable

Megastore

8 of 38

Scalable

Google Datacenter

Colossus

BigTable

Megastore

9 of 38

Highly Available

Google Datacenter

Colossus

BigTable

Megastore

10 of 38

Managed

Google Datacenter

Colossus

BigTable

Megastore

Cloud Datastore API

11 of 38

Ubiquitous access

goo.gl/LWM8Qs

12 of 38

Ubiquitous access

13 of 38

Metrics

Trillions of operations / month

Millions of apps using it through App Engine

Petabytes of data

goo.gl/LWM8Qs

14 of 38

What do you store?

Schemaless Entities: Bag of typed properties.

Ex:

{� firstName: { stringValue: 'Van' },� lastName: { stringValue: 'Riper' },� joinDate: { dateTimeValue: '2008-01-01T00:00:00.000Z' },� isOrganizer: { booleanValue: true }�}

goo.gl/LWM8Qs

15 of 38

How do you address them?

Key = (Parent Key) / Kind / Id or Name

ex:

GDGer:1

GDG:sv

GDG:sv/Event:1

goo.gl/LWM8Qs

16 of 38

How are they stored?

Entity Table

Key	Proto
Googler:1	{ name: proppy }
...
GDGer:1	{ name: van }
GDGer:2	{ name: peter }

17 of 38

How are they organized?

Entity Group

- Common ancestor

GDG:sv:Event:1:Session:1

GDG:sv:Event:2:Session:2

goo.gl/LWM8Qs

18 of 38

How are they indexed?

Index tables:

ByKind: kind:key

GDGer:GDGer:1

ByProperty: kind:pname:pvalue:key

GDGer:isOrganizer:true:GDGer:1

19 of 38

How are they indexed?

CompositeIndex

kind:pname:pvalue:pname:pvalue:key

GDGer:firstName:Van:isOrganizer:true:GDGer:1

goo.gl/LWM8Qs

20 of 38

How does query work?

SELECT * FROM GDGer WHERE

isOrganizer = 'true';

1. Scan index for GDGer:isOrganizer:true:

GDGer:isOrganizer:true:GDGer:1

GDGer:isOrganizer:true:GDGer:2

GDGer:isOrganizer:false:GDGer:3

21 of 38

How does query work?

Fetch matching keys from Entity Table:

Key	Proto
Googler:1	{ name: proppy }
...
GDGer:1	{ name: Van, isOrganizer: true }
GDGer:2	{ name: Peter, isOrganizer: true }
GDGer:3	{ name: Bob, isOrganizer: false }

22 of 38

Eventual consistent query

SELECT * FROM Member WHERE isOrganizer=true;

goo.gl/LWM8Qs

23 of 38

Strongly consistent query

SELECT * FROM Member WHERE isOrganizer=true

AND ANCESTOR IS KEY('Chapter', 'sv');

goo.gl/LWM8Qs

24 of 38

Limitations

- No Joins

- No aggregations

- No inequality on multiple properties

- Eventual consistency for global queries

goo.gl/LWM8Qs

25 of 38

Transactions

- Atomic: all or nothing

- Consistent: coherent view of the data

- Isolated: conflict on concurrent update

- Durable: commit

goo.gl/LWM8Qs

26 of 38

Limitation

- All mutations are performed on commit

- Scoped to a small number of Entity Groups

- Transaction rate limit per entity group

goo.gl/LWM8Qs

27 of 38

Transactions

POST /datasets/<datasetId>/beginTransaction

{

}

{

"transaction": "EdApPiN7yyPrGkkABRmGMUiD79_VRpDF...z8GH0h0i5"

}

28 of 38

Schema-less

POST /datasets/<datasetId>/commit

{

mutation: {

insertAutoId: [{� key: { path: [{ kind: 'GDGer' }] },� properties: {� firstName: { stringValue: 'Van' },� lastName: { stringValue: 'Riper' },� joinDate: { dateTimeValue: '2010-01-01T00:00:00.000Z' },� isOrganizer: { booleanValue: true }� }� }]

"transaction": "EdApPiN7yyPrGkkABRmGMUiD79_VRpDF...z8GH0h0i5" }

29 of 38

Indexes

POST /datasets/<datasetId>/runQuery

{

query: {

kinds: [{ name: GDGer }],� filter: {� propertyFilter: {� property: { name: 'isOrganizer' },� operator: 'EQUAL',� value: { booleanValue: true }� }� }

order: [{property: {name: 'joinDate'}, direction: 'DESCENDING'}}]� }

}

30 of 38

Query

POST /datasets/<datasetId>/runQuery

{

gqlQuery: {� queryString: 'SELECT * FROM GDGer WHERE isOrganizer = @1

ORDER BY joinDate DESC',

numberArgs: [{ value: { booleanValue: true } }]� }

}

31 of 38

Get started

$ npm install googleapis

goo.gl/LWM8Qs

32 of 38

Get started

$ gem install google-api-client

goo.gl/LWM8Qs

33 of 38

Get started

$ pip install googledatastore

goo.gl/LWM8Qs

34 of 38

Get started

$ mvn dependency:get -DartifactId=\

google-api-services-datastore-protobuf

goo.gl/LWM8Qs

35 of 38

Features

Managed ✓

Auto Scalable ✓

ACID Transactions ✓

Composite Indexes ✓

Rich Query Language ✓

Strongly Consistent Ancestor Queries✓

36 of 38

Limitations

- Eventual consistency for Global Queries

- 1 Transaction/s per Entity Groups

- Transaction limited to 5 Entity Groups

- Query limitations

37 of 38

Get started

developers.google.com/datastore

goo.gl/LWM8Qs

38 of 38

One more thing

cloud.google.com/starterpack

code: gdg-in

$1000