1 of 147

Online readers: see Speaker Notes

LinuxFest

Northwest

2018

LinuxFest Northwest 2018

2 of 147

Perkeep:

your personal storage system for life

Brad Fitzpatrick, Mathieu Lonjaret

2018-04-28

LinuxFest Northwest 2018

3 of 147

Hello again!

Who saw our talk here in 2016?

LinuxFest Northwest 2018

4 of 147

LinuxFest Northwest 2018

5 of 147

LinuxFest Northwest 2018

6 of 147

Agenda

What is Perkeep?

Why Perkeep?�How does it work?

The Project

LinuxFest Northwest 2018

7 of 147

Agenda

What is Perkeep?

Why Perkeep?�How does it work?

The Project

LinuxFest Northwest 2018

8 of 147

What is Perkeep?

LinuxFest Northwest 2018

9 of 147

Blob storage

API

Indexing

OpenPGP

verify

Search

OpenPGP

signing

Web UI

CLI

Apps

Importers

FUSE

Mobile

Schema

LinuxFest Northwest 2018

10 of 147

Blob storage

API

Indexing

OpenPGP

verify

Search

OpenPGP

signing

Web UI

CLI

Apps

Importers

FUSE

Mobile

Schema

LinuxFest Northwest 2018

11 of 147

Blob storage

API

Indexing

OpenPGP

verify

Search

OpenPGP

signing

Web UI

CLI

Apps

Importers

FUSE

Mobile

Schema

LinuxFest Northwest 2018

12 of 147

LinuxFest Northwest 2018

13 of 147

LinuxFest Northwest 2018

14 of 147

LinuxFest Northwest 2018

15 of 147

LinuxFest Northwest 2018

16 of 147

LinuxFest Northwest 2018

17 of 147

LinuxFest Northwest 2018

18 of 147

LinuxFest Northwest 2018

19 of 147

LinuxFest Northwest 2018

20 of 147

LinuxFest Northwest 2018

21 of 147

LinuxFest Northwest 2018

22 of 147

LinuxFest Northwest 2018

23 of 147

LinuxFest Northwest 2018

24 of 147

Blob storage

API

Indexing

OpenPGP

verify

Search

OpenPGP

signing

Web UI

CLI

Apps

Importers

FUSE

Mobile

LinuxFest Northwest 2018

25 of 147

$ pk put blob …

$ pk put file foo.txt

$ pk put file $HOME/dir

$ pk put permanode

$ pk put attr [--add] <permanode> <name> <value>

$ pk put share

$ pk list

$ pk get …

$ pk search ...

$ pk sync ...

LinuxFest Northwest 2018

26 of 147

Blob storage

API

Indexing

OpenPGP

verify

Search

OpenPGP

signing

Web UI

CLI

Apps

Importers

FUSE

Mobile

LinuxFest Northwest 2018

27 of 147

$ pk mount

$ open /pk/recent/IMG_0925.jpg

$ ls /pk

at date recent roots sha1-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

tag versions WELCOME.txt

$ echo "Hello, LFNW 2018." > /pk/roots/foo/foo.txt

$ cat /pk/roots/foo/foo.txt

Hello, LFNW 2018.

$ cat /pk/at/2016-04-28/foo/foo.txt

Hello, LinuxFest!

LinuxFest Northwest 2018

28 of 147

Blob storage

API

Indexing

OpenPGP

verify

Search

OpenPGP

signing

Web UI

CLI

Apps

Importers

FUSE

Mobile

Schema

LinuxFest Northwest 2018

29 of 147

LinuxFest Northwest 2018

30 of 147

Blob storage

API

Indexing

OpenPGP

verify

Search

OpenPGP

signing

Web UI

CLI

Apps

Importers

FUSE

Mobile

Schema

LinuxFest Northwest 2018

31 of 147

Importers

perkeepd

foursquare

importer

twitter

importer

LinuxFest Northwest 2018

32 of 147

LinuxFest Northwest 2018

33 of 147

LinuxFest Northwest 2018

34 of 147

LinuxFest Northwest 2018

35 of 147

Blob storage

API

Indexing

OpenPGP

verify

Search

OpenPGP

signing

Web UI

CLI

Apps

Importers

FUSE

Mobile

Schema

LinuxFest Northwest 2018

36 of 147

LinuxFest Northwest 2018

37 of 147

Blob storage

API

Indexing

OpenPGP

verify

Search

OpenPGP

signing

Web UI

CLI

Apps

Importers

FUSE

Mobile

Schema

LinuxFest Northwest 2018

38 of 147

Why Perkeep?

LinuxFest Northwest 2018

39 of 147

Do you care about your data?

Will you still care in 5, 20, 40, 60 years?

Will your grandchildren want your photos?

LinuxFest Northwest 2018

40 of 147

Unified Search

Can you easily search through all of your online data in a unified way?

"photos within 30 minutes of check-ins at venues of type 'sporting arena'"

"tweets two hours after drinking"

LinuxFest Northwest 2018

41 of 147

Sites die

People create content on third-party sites.

And die with your data.

http://indiewebcamp.com/site-deaths

LinuxFest Northwest 2018

42 of 147

Sites die

VC funding runs out

competitor eats them

site is sold to new evil owners

new employees at existing company turn evil

… or get greedy

LinuxFest Northwest 2018

43 of 147

Own it

Stay in control.

LinuxFest Northwest 2018

44 of 147

Goal

Perkeep is a project to help you keep your content and online memories for life.

LinuxFest Northwest 2018

45 of 147

LinuxFest Northwest 2018

46 of 147

http://www.prattdudley.com/images/lumber5.jpg

LinuxFest Northwest 2018

47 of 147

http://www.prattdudley.com/images/lumber5.jpg

LinuxFest Northwest 2018

48 of 147

LinuxFest Northwest 2018

49 of 147

LinuxFest Northwest 2018

50 of 147

LinuxFest Northwest 2018

51 of 147

LinuxFest Northwest 2018

52 of 147

LinuxFest Northwest 2018

53 of 147

LinuxFest Northwest 2018

54 of 147

LinuxFest Northwest 2018

55 of 147

http://indiewebcamp.com/site-deaths

LinuxFest Northwest 2018

56 of 147

LinuxFest Northwest 2018

57 of 147

LinuxFest Northwest 2018

58 of 147

"PESOS"

Publish Elsewhere, Syndicate (to your) Own Site

https://indieweb.org/PESOS

LinuxFest Northwest 2018

59 of 147

LinuxFest Northwest 2018

60 of 147

LinuxFest Northwest 2018

61 of 147

How does it work?

LinuxFest Northwest 2018

62 of 147

Blob storage

API

Indexing

OpenPGP

verify

Search

OpenPGP

signing

Web UI

CLI

Apps

Importers

FUSE

Mobile

Schema

LinuxFest Northwest 2018

63 of 147

Blob storage

API

Indexing

OpenPGP

verify

Search

OpenPGP

signing

Web UI

CLI

Apps

Importers

FUSE

Mobile

Schema

LinuxFest Northwest 2018

64 of 147

Blob storage

we store blobs of 0B-16MB

no filenames

no MIME types

no metadata

no versions

just immutable blobs

LinuxFest Northwest 2018

65 of 147

No filenames? No metadata?

blobs are content-addressable

sha1-f61bea2f271c3b4900255c7deda5ec1e48a77105

sha224-abfa9fa2719d9cb8ad9e737e77da9d816bc8201b1f96e24976ec5459

LinuxFest Northwest 2018

66 of 147

Advantages

simplicity
content-deduplication
cacheability (no expiration times, versions)
integrity checking

LinuxFest Northwest 2018

67 of 147

$ pk list

$ echo Hello | pk put blob -�sha224-acbe28e133c6e7e8cc740d5c70875…

$ echo Hello | pk put blob -

sha224-acbe28e133c6e7e8cc740d5c70875…

$ echo World | pk put blob -�sha224-703574ab03e4cec2dd88d75db1802…

$ pk list

sha224-acbe28e133c6e7e8cc740d5c70875… 6

sha224-703574ab03e4cec2dd88d75db1802… 6

$ pk get sha224-acbe28e133c6e7e8cc740d5c70875…

Hello

LinuxFest Northwest 2018

68 of 147

$ pk list | wc -l

6639759

LinuxFest Northwest 2018

69 of 147

Blob server operations

PUT blob
GET blob by hash
Enumerate (in sorted hash order)

LinuxFest Northwest 2018

70 of 147

No delete!

Harder to lose your data if you can't delete it.

[*] you can in emergencies if configured and you --ask-really-nicely

LinuxFest Northwest 2018

71 of 147

Blobserver (in Go)

EnumerateBlobs(ctx context.Context, dest …, after string, limit int) error

StatBlobs(ctx context.Context, blobs []blob.Ref, fn func(blob.SizedRef) error) error�Fetch(context.Context, blob.Ref) (data io.ReadCloser, size uint32, err error)�ReceiveBlob(ctx context.Context, br blob.Ref, src io.Reader) (blob.SizedRef, error)

LinuxFest Northwest 2018

72 of 147

Blobserver Implementations

local files

AWS S3

Google Cloud Storage

Google Drive

shard

union

encrypt

cache

remote

cond

mongo

namespace

packed

Azure

overlay

replica

LinuxFest Northwest 2018

73 of 147

Blobserver Implementations

remote

local files

AWS S3

Google Cloud Storage

Google Drive

shard

union

encrypt

cache

cond

mongo

namespace

packed

Azure

overlay

replica

LinuxFest Northwest 2018

74 of 147

Blobserver Implementations

remote

local files

AWS S3

Google Cloud Storage

Google Drive

shard

union

encrypt

cache

cond

mongo

namespace

packed

Azure

overlay

replica

LinuxFest Northwest 2018

75 of 147

No delete! No versions! Immutable!

It's always safe to copy everything from anywhere to anywhere, merging two subsets of blobs.

You can't break anything.

LinuxFest Northwest 2018

76 of 147

Sync validation

You can even validate/fix sync pairs of blobstores

from the CLI or web UI.

It can also self-check/repair itself regularly.

LinuxFest Northwest 2018

77 of 147

LinuxFest Northwest 2018

78 of 147

What's in a blob?

LinuxFest Northwest 2018

79 of 147

What's in a blob?

Anything.

LinuxFest Northwest 2018

80 of 147

What's in a blob?

e.g. byte range 275029-501935 of some JPEG?

LinuxFest Northwest 2018

81 of 147

What's in a blob?

But some blob have bytes that look special.

LinuxFest Northwest 2018

82 of 147

Blob storage

API

Indexing

OpenPGP

verify

Search

OpenPGP

signing

Web UI

CLI

Apps

Importers

FUSE

Mobile

Schema

LinuxFest Northwest 2018

83 of 147

Schema Blobs

Blobs with a JSON object with

certain known fields.

LinuxFest Northwest 2018

84 of 147

Wait, JSON?

Why not binary format $X?

LinuxFest Northwest 2018

85 of 147

Data archaeology

JSON: ASCII/UTF-8, well-known because of the web.

sha224-xxxxxxxxxxx ref IDs include hash name, both for upgradability & to be self-describing.

LinuxFest Northwest 2018

86 of 147

Example: files

A traditional “file” is just one type of thing Perkeep can model.

LinuxFest Northwest 2018

87 of 147

$ echo "Hello, World" > hello.txt

$ pk put file hello.txt

sha224-3043e7faf550da145aed7d038d21a2a8a501489a61f80abaab4076f0

$ pk get sha224-3043e7faf550da145aed7d038d21a2a8a501489a61f80abaab4076f0

{"camliVersion": 1,� "camliType": "file",� "fileName": "hello.txt",� "parts": [{� "blobRef":"sha224-0773026556c11487f4e36193406a47e2d1c6425278747b...",� "size": 15� }],� "unixGroup": "mpl",� "unixGroupId": 1000,� "unixMtime": "2018-04-27T17:57:36.202241494Z",� "unixOwner": "mpl",� "unixOwnerId": 1000,� "unixPermission": "0664"�}

$ pk get sha224-0773026556c11487f4e36193406a47e2d1c6425278747b8b22391c7e

Hello, World

LinuxFest Northwest 2018

88 of 147

5TB video file or VM image?

Merkle tree of "bytes" schema blobs.

Data at leaves.

Rolling checksum cut points (ala rsync, bup)

De-duplication within files & shifting files

Efficient seeks / pread

LinuxFest Northwest 2018

89 of 147

Or a “directory”...

LinuxFest Northwest 2018

90 of 147

$ mkdir foo; touch foo/a; touch foo/bar

$ pk put file foo

sha224-ecbd2e404c4e2cacf09728025592dd324046115d95f9639cccbbe0fe

$ pk get sha224-ecbd2e404c4e2cacf09728025592dd324046115d95f9639cccbbe0fe

{"camliVersion": 1,� "camliType": "directory",� "entries": "sha224-529dbf997489196e8ea55f8681897b98da1d0c4eafc757...",� "fileName": "foo",� "unixGroup": "mpl",� "unixGroupId": 1000,� "unixMtime": "2018-04-27T18:24:10.75819438Z",� "unixOwner": "mpl",� "unixOwnerId": 1000,� "unixPermission": "0775"}

$ pk get sha224-529dbf997489196e8ea55f8681897b98da1d0c4eafc7571a0f5d8cd8

{"camliVersion": 1,� "camliType": "static-set",� "members": [� "sha224-2bbbfa36f4fcedd0d84d8564a28f2a49f1007c833073e2825a08d741",� "sha224-4a92d0cc8608b2e75e5dea87086198e6e466eb83a25e0e8ed0d10455"� ]}

LinuxFest Northwest 2018

91 of 147

$ pk get sha224-2bbbfa36f4fcedd0d84d8564a28f2a49f1007c833073e2825a08d741

{"camliVersion": 1,� "camliType": "file",� "fileName": "a",� "parts": [],� "unixGroup": "mpl",� "unixGroupId": 1000,� "unixMtime": "2018-04-27T18:24:04.270194572Z",� "unixOwner": "mpl",� "unixOwnerId": 1000,� "unixPermission": "0664"�}

$ pk get sha224-4a92d0cc8608b2e75e5dea87086198e6e466eb83a25e0e8ed0d10455

{"camliVersion": 1,� "camliType": "file",� "fileName": "bar",� "parts": [],� "unixGroup": "mpl",� "unixGroupId": 1000,� "unixMtime": "2018-04-27T18:24:10.75819438Z",� "unixOwner": "mpl",� "unixOwnerId": 1000,� "unixPermission": "0664"�}

LinuxFest Northwest 2018

92 of 147

Blob storage

API

Indexing

OpenPGP

verify

Search

OpenPGP

signing

Web UI

CLI

Apps

Importers

FUSE

Mobile

Schema

LinuxFest Northwest 2018

93 of 147

$ pk list | wc -l

6639759

LinuxFest Northwest 2018

94 of 147

Blob storage

API

Indexing

OpenPGP

verify

Search

OpenPGP

signing

Web UI

CLI

Apps

Importers

FUSE

Mobile

Schema

LinuxFest Northwest 2018

95 of 147

Index

blobs

sorted

key=value

index implement blob storage interface; can replicate/sync to

write to any sorted key/value implementation

LinuxFest Northwest 2018

96 of 147

sorted.KeyValue interface

Set("key", "value")
Get("key")
Find("from", "to") -> Iterator

LinuxFest Northwest 2018

97 of 147

sorted.KeyValue stores

LevelDB

local disk

MySQL

Postgres

Google

sqlite

in-memory

(dev)

mongo

AWS

LinuxFest Northwest 2018

98 of 147

Index

Can be lost, deleted, corrupted
Can always be re-built from raw blobs

LinuxFest Northwest 2018

99 of 147

Indexing story time

What happens

when Perkeep receives a blob?

LinuxFest Northwest 2018

100 of 147

Perkeep receives blob

HTTP handler sends blob to primary blob storage
Blob syncs to the "index" blob storage

LinuxFest Northwest 2018

101 of 147

Indexer looks at bytes of blob

not a schema blob (just data)? nothing to do.

else, analyze it, write index rows

LinuxFest Northwest 2018

102 of 147

Index receives blob

file schema? fetch data from storage, and record interesting stuff: modtime, image EXIF tags (lat, long), audio ID3 tags, etc.
directory schema? record all its children

LinuxFest Northwest 2018

103 of 147

Index receives blob

mutation schema? check if valid OpenPGP signature, record attributes it modifies (tag, title, location, etc.)

LinuxFest Northwest 2018

104 of 147

Corpus

optimized in-memory data strutures for search, built from the sorted.KeyValue index

LinuxFest Northwest 2018

105 of 147

Corpus

(in memory)

sorted

key=value

LinuxFest Northwest 2018

106 of 147

Corpus receives blob

Updates maps, and (lazily sorted) lists of:

(GPG) signers
objects, mutations
file attributes, image attributes

LinuxFest Northwest 2018

107 of 147

Blob storage

API

Indexing

OpenPGP

verify

Search

OpenPGP

signing

Web UI

CLI

Apps

Importers

FUSE

Mobile

Schema

LinuxFest Northwest 2018

108 of 147

Two ways to search

The easy way
The hard way

LinuxFest Northwest 2018

109 of 147

The easy way

is:image before:2017-05 after:2015 loc:Hawaii

This compiles into the hard way for you.

(https://perkeep.org/doc/search-ui)

LinuxFest Northwest 2018

110 of 147

is:image

is:pano

before:2017-05

after:2015

is:checkin

is:post

loc:Hawaii

height:480

locrect:48.63,-123.37,46.59,-121.28

attr:tag:funny

format:jpeg

filename:TODO.*

ref:sha224-xxxx

parentof:<ref>

childrenof:<ref>

AND OR NOT -(.....)

LinuxFest Northwest 2018

111 of 147

filename:*.HEIC -(filename:*.HEIC width:1-100000)

(Found HEIC files we misparsed when we were adding support for HEIC images.)

LinuxFest Northwest 2018

112 of 147

The hard way

A JSON object

which must decode into SearchQuery type

https://perkeep.org/pkg/search#SearchQuery

LinuxFest Northwest 2018

113 of 147

LinuxFest Northwest 2018

114 of 147

LinuxFest Northwest 2018

115 of 147

LinuxFest Northwest 2018

116 of 147

filename:*hello*

LinuxFest Northwest 2018

117 of 147

$ pk search -rawquery '{"constraint": {� "permanode": {� "attr": "camliContent",� "valueInSet": {� "file": {� "fileName": {� "contains": "hello"� }� }� }� }�}}'

{� "blobs": [{� "blob": "sha224-e4086340c3dc85cfee70bba0ea71f1f11aaa..."� }], …

LinuxFest Northwest 2018

118 of 147

Permanodes

LinuxFest Northwest 2018

119 of 147

Perkeep stores only immutable blobs.

LinuxFest Northwest 2018

120 of 147

How do we represent something that's mutable?

And how do we store a tweet?

Or a "like"?

LinuxFest Northwest 2018

121 of 147

$ pk put permanode

sha224-d76452b7b90d75e92aa07dfcbb254822305043a76819ae...

$ pk put permanode

sha224-2ad67a389ba386f439c0ee32fd3eee92444de7e3c5bdc1...

$ pk get sha224-2ad67a389ba386f439c0ee32fd3eee92444de7e3c...

{"camliVersion": 1,� "camliSigner": "sha224-ea0ba9068820cc128e993b20d0511de7...",� "camliType": "permanode",� "claimDate": "2018-04-27T20:51:19.426507741Z",� "random": "ZEh4ms2HiY1W4xVDhBQD3K3DHC4="�,"camliSig":"wsBcBAABCAAQBQJa441HCRApMaZ8JvWr2gAACKAIABpPIGojhfOxP9LfyZOV6BcGAUHfTkL52+ZtTEFepGfhkPpZH3nqfa682L2ofoFhZkW2FrciOQ+qleDRzv4YGI3LMqBqIqb4RakeGYRzBzpkzzv9vyJV/uIuDGDOzNyNz+hu/Dy6NjUoIt0kwaIsfymzMmLRKSAdcvoDUYekO9VMaF1xDslrUgDEewQQGQHEU4HfMqVR0pxUUWI51Ce2Uq0WAqbQ7lV7dSCsxyQY0tlW8Y1OgbUiqn13onoSs9yuUB4PyTCIOi3uFPGaU13a3kMAvmfRWSx4nRhBu3aWe3p+YdleDcOT6dShyS9Cdo2bizqM+f+TrrSgkr0PH7jTOkE==S9pj"}

LinuxFest Northwest 2018

122 of 147

$ pk put permanode

sha224-d76452b7b90d75e92aa07dfcbb254822305043a76819ae...

$ pk put permanode

sha224-2ad67a389ba386f439c0ee32fd3eee92444de7e3c5bdc1...

$ pk get sha224-2ad67a389ba386f439c0ee32fd3eee92444de7e3c...

{"camliVersion": 1,� "camliSigner": "sha224-ea0ba9068820cc128e993b20d0511de7...",� "camliType": "permanode",� "claimDate": "2018-04-27T20:51:19.426507741Z",� "random": "ZEh4ms2HiY1W4xVDhBQD3K3DHC4="�,"camliSig":"wsBcBAABCAAQBQJa441HCRApMaZ8JvWr2gAACKAIABpPIGojhfOxP9LfyZOV6BcGAUHfTkL52+ZtTEFepGfhkPpZH3nqfa682L2ofoFhZkW2FrciOQ+qleDRzv4YGI3LMqBqIqb4RakeGYRzBzpkzzv9vyJV/uIuDGDOzNyNz+hu/Dy6NjUoIt0kwaIsfymzMmLRKSAdcvoDUYekO9VMaF1xDslrUgDEewQQGQHEU4HfMqVR0pxUUWI51Ce2Uq0WAqbQ7lV7dSCsxyQY0tlW8Y1OgbUiqn13onoSs9yuUB4PyTCIOi3uFPGaU13a3kMAvmfRWSx4nRhBu3aWe3p+YdleDcOT6dShyS9Cdo2bizqM+f+TrrSgkr0PH7jTOkE==S9pj"}

LinuxFest Northwest 2018

123 of 147

$ pk get sha224-ea0ba9068820cc128e993b20d0511de736febc62fbd430d17bba9792

LinuxFest Northwest 2018

124 of 147

$ pk put permanode

sha224-8a445d964c3530bba93cfd317bde532ee1e15efd801ed179b9e80c7f

$ pk put attr sha224-8a445d964c3530bba93cfd317bde532ee1... title 'Fancy title'

$ pk put attr sha224-8a445d964c3530bba93cfd317bde532ee1... title 'Better title'

$ pk describe sha224-8a445d964c3530bba93cfd317bde532ee1e15efd801ed179b9e80c7f

{� "meta": {� "sha224-8a445d964c3530bba93cfd317bde532ee1e15efd801ed179b9e80c7f": {� "blobRef": "sha224-8a445d964c3530bba93cfd317bde532ee1e15efd801ed179...",� "camliType": "permanode",� "size": 629,� "permanode": {� "attr": {� "title": [� "Better title"� ]� },� "modtime": "2018-04-27T22:14:18.981957688Z"� }� }� }�}

LinuxFest Northwest 2018

125 of 147

$ pk claims sha224-8a445d964c3530bba93cfd317bde532ee1e15efd801ed179b9e80c7f

{� "claims": [{� "blobref": "sha224-25b7852f787a7b5f971ba424d453b7e63360cd03429f85adb4...",� "signer": "sha224-ea0ba9068820cc128e993b20d0511de736febc62fbd430d17bba9792",� "permanode": "sha224-8a445d964c3530bba93cfd317bde532ee1e15efd801ed179b9...",� "date": "2018-04-27T22:14:18.981957688Z",� "type": "set-attribute",� "attr": "title",� "value": "Fancy title"� },� {� "blobref": "sha224-8354dcb43d30a79b4e3f9e6ba3cb7557f35b852f566970499884...",� "signer": "sha224-ea0ba9068820cc128e993b20d0511de736febc62fbd430d17bba9792",� "permanode": "sha224-8a445d964c3530bba93cfd317bde532ee1e15efd801ed179b...",� "date": "2018-04-27T22:24:24.886072921Z",� "type": "set-attribute",� "attr": "title",� "value": "Better title"� }]�}

LinuxFest Northwest 2018

126 of 147

$ pk describe --at=2018-04-27T22:20:00.0Z sha224-8a445d964c3530bba93cfd317...

{� "meta": {� "sha224-8a445d964c3530bba93cfd317bde532ee1e15efd801ed179b9e80c7f": {� "blobRef": "sha224-8a445d964c3530bba93cfd317bde532ee1e15efd801ed179...",� "camliType": "permanode",� "size": 629,� "permanode": {� "attr": {� "title": [� "Fancy title"� ]� },� "modtime": "2018-04-27T22:14:18.981957688Z"� }� }� }�}

LinuxFest Northwest 2018

127 of 147

$ pk list | head -20

sha224-0773026556c11487f4e36193406a47e2d1c6425278747b8b22391c7e 15�sha224-25b7852f787a7b5f971ba424d453b7e63360cd03429f85adb451141a 745�sha224-2ad67a389ba386f439c0ee32fd3eee92444de7e3c5bdc1b6377d3eff 629�sha224-2bbbfa36f4fcedd0d84d8564a28f2a49f1007c833073e2825a08d741 244�sha224-3043e7faf550da145aed7d038d21a2a8a501489a61f80abaab4076f0 368�sha224-4a92d0cc8608b2e75e5dea87086198e6e466eb83a25e0e8ed0d10455 245�sha224-529dbf997489196e8ea55f8681897b98da1d0c4eafc7571a0f5d8cd8 210�sha224-6be2b3bcc4a17567d6e3bc004315c2a9a48b212df4eeaa61e20eb96f 751�sha224-8354dcb43d30a79b4e3f9e6ba3cb7557f35b852f566970499884fb39 746�sha224-8a445d964c3530bba93cfd317bde532ee1e15efd801ed179b9e80c7f 629�sha224-b01efacadad4e9cea0d9d457c97aa2732a560851bbc19338e78e19d0 768�sha224-b0a94d2947f0decfd3b34a70f46ddac6860c2d45b437d0bacdf14f6f 580�sha224-cdf85e7f959410dd8acc81cc51abd34b46478cefc4079550677988fc 804�sha224-d76452b7b90d75e92aa07dfcbb254822305043a76819aeed4b8fb5f0 629�sha224-e4086340c3dc85cfee70bba0ea71f1f11aaa837d79bed32d19b5e2a2 651�sha224-ea0ba9068820cc128e993b20d0511de736febc62fbd430d17bba9792 449�sha224-ecbd2e404c4e2cacf09728025592dd324046115d95f9639cccbbe0fe 315

LinuxFest Northwest 2018

128 of 147

$ X=$(pk put permanode)

$ pk search 'attr:tag:funny'

{� "blobs": null,� "description": null,�}

$ pk put attr $X tag funny

$ pk search 'attr:tag:funny'

{� "blobs": [{� "blob": "sha224-333eb172516d2677ca4b382e122de15f623c1…"� }],� "description": null,�}

LinuxFest Northwest 2018

129 of 147

$ pk search -rawquery '{"constraint": {� "permanode": {� "attr": "tag",� "valueMatches": {� "equals": "FUNNY",� "caseInsensitive": true� }� }�}}'

{� "blobs": [{� "blob": "sha224-333eb172516d2677ca4b382e122de15f623c…"� }],�…

LinuxFest Northwest 2018

130 of 147

Blob storage

API

Indexing

OpenPGP

verify

Search

OpenPGP

signing

Web UI

CLI

Apps

Importers

FUSE

Mobile

Schema

LinuxFest Northwest 2018

131 of 147

LinuxFest Northwest 2018

132 of 147

LinuxFest Northwest 2018

133 of 147

LinuxFest Northwest 2018

134 of 147

The Project

LinuxFest Northwest 2018

135 of 147

History

~8 years old
started around Go's open sourcing (2009-11)

way before Go 1 (2012-03-28)

much of Go stdlib borne out of Perkeep

LinuxFest Northwest 2018

136 of 147

Primary Developers

Brad Fitzpatrick, @bradfitz

Mathieu Lonjaret, @lejatorn

LinuxFest Northwest 2018

137 of 147

Funding

https://opencollective.com/perkeep

LinuxFest Northwest 2018

138 of 147

LinuxFest Northwest 2018

139 of 147

Funding

We plan to recognize patrons more prominently going forward.

But in the meantime...

LinuxFest Northwest 2018

140 of 147

Thank you!

Paul Lindner, Erik Paulson, Jeremy Schlatter, Attila Tajti, Will Norris, David Whittington, PDXHub, Dan S, Andor Polgar, Tilman Dilo, Greg Worley, Szabolcs Berecz, Cole Gleason...

LinuxFest Northwest 2018

141 of 147

Other Developers

About 120 developers total

23 with >= 10 commits

Yay Open Source!

LinuxFest Northwest 2018

142 of 147

Thank you!

bslatkin, aboodman, lindner, wathiede, derat, tgulacsi, willnorris, dustin, tajtiattila, nickoneill, adg, saljam, edrex, dotMR, marete, slspeek, tdilo, FiloSottile, fawick, AshleyMcnamara, rhencke, evmar, elsigh, yaojingguo...

LinuxFest Northwest 2018

143 of 147

Getting Started

Launch your own cloud instance:

https://perkeep.org/launch

LinuxFest Northwest 2018

144 of 147

Getting Started

Run your own!

$ go get perkeep.org/server/perkeepd

$ go get perkeep.org/cmd/pk

LinuxFest Northwest 2018

145 of 147

Getting Involved

File bugs!�Write docs!

Write an importer, or an app!

LinuxFest Northwest 2018

146 of 147

Resources

https://perkeep.org

https://github.com/perkeep/perkeep

perkeep@googlegroups.com

#perkeep (Freenode)

LinuxFest Northwest 2018

147 of 147

Demos! Questions?

Brad Fitzpatrick, @bradfitz

Mathieu Lonjaret, @lejatorn

perkeep.org, @PerkeepOrg

LinuxFest Northwest 2018