1 of 42

Lesson 3.2

FlatGeobuf

​

GEO5019.2025

Hugo Ledoux & Hidemichi Baba

2 of 42

FlatGeobuf

What is it?

Hugo Ledoux & Hidemichi Baba

3 of 42

oh wow!

FlatGeobuf

Hugo Ledoux & Hidemichi Baba

4 of 42

FlatGeobuf

  • It uses FlatBuffers for efficient feature / geometry encoding
  • built-in streamable spatial index

especially interesting:

FlatBuffers + Geo + some bonus = FlatGeobuf

Hugo Ledoux & Hidemichi Baba

5 of 42

Recap serde

serializing and deserializing

(serde, 2025)

similar meaning

  • Pickling / Unpickling
  • Encoding / Decoding
  • Marshalling / Unmarshalling

Hugo Ledoux & Hidemichi Baba

6 of 42

Magic bytes

A file signature

(Wikipedia, 2025)

Magic numbers, file signature, or magic bytes

Let’s read and write magic bytes in Python! (demo is here)

Go to ASCII table and see https://www.ascii-code.com/

Hugo Ledoux & Hidemichi Baba

7 of 42

Endianness (or "byte-order")

Little endian and big endian

(MDN, 2025)

Examples with the number 0x12345678 (i.e., 305 419 896 in decimal):

​

  • little-endian: 0x78 0x56 0x34 0x12
  • big-endian: 0x12 0x34 0x56 0x78
  • mixed-endian (historic and very rare): 0x34 0x12 0x78 0x56

Little endian resembles the European data format (27 November 2025)

​

While big endian is like ISO format (2025-11-27)

Hugo Ledoux & Hidemichi Baba

8 of 42

FlatBuffers?

Hugo Ledoux & Hidemichi Baba

9 of 42

What is FlatBuffers?

One of the serialisation frameworks made by Google

(FlatBuffers, 2025)

  • Access to serialized data without parsing/unpacking
  • Memory Efficiency and Speed
  • Backwards and Forwards Compatibility
  • Small Footprint

“JSON not only has the obvious drawback of runtime inefficiency, but also forces you to write more code to access data (counterintuitively) due to its dynamic-typing serialization system.”

Why not JSON?

Hugo Ledoux & Hidemichi Baba

10 of 42

What is FlatBuffers?

How good is FlatBuffers?

(FlatBuffers, 2025)

Hugo Ledoux & Hidemichi Baba

11 of 42

FlatBuffers: Schema based serialisation

Steps to serialise

(FlatBuffers, 2025)

  • Define data schema .fbs file
  • Compile schema file using the FlatBuffers compiler (flatc)
  • Implement serde for your application

Example schema

namespace MyGame.Sample;

​

enum Color:byte { Red = 0, Green, Blue = 2 }

​

// Optionally add more tables.

union Equipment { Weapon }

​

struct Vec3 {

x:float;

y:float;

z:float;

}

​

table Monster {

pos:Vec3;

mana:short = 150;

hp:short = 100;

name:string;

friendly:bool = false (deprecated);

inventory:[ubyte];

color:Color = Blue;

weapons:[Weapon];

equipped:Equipment;

path:[Vec3];

}

​

table Weapon {

name:string;

damage:short;

}

​

root_type Monster;

​

​

Hugo Ledoux & Hidemichi Baba

12 of 42

FlatBuffers: data types

Supported data types

(FlatBuffers, 2025)

Tables: the main way of defining objects in FlatBuffers

Structs: consist of fields are required (so no defaults either), and fields may not be added or be deprecated

Other types

  • Arrays
  • Strings
  • Enums
  • Unions
  • and some others

Scalars: fixed sized scalars

Hugo Ledoux & Hidemichi Baba

13 of 42

FlatBuffers: how do you encode variable length data?

Length prefix

(FlatBuffers, 2025)

Tables: the main way of defining objects in FlatBuffers

Hugo Ledoux & Hidemichi Baba

14 of 42

FlatBuffers + Geo + some bonus

= FlatGeobuf

Hugo Ledoux & Hidemichi Baba

15 of 42

What is FlatGeobuf?

How good is FlatGeobuf?

(FlatGeobuf, 2025)

Hugo Ledoux & Hidemichi Baba

16 of 42

FlatGeobuf: Convention of binary encodings

Two conventions for binary encoding of FlatGeobuf:

(Horace Williams, 2025)

  • “size-prefixed”: each byte buffers contains 4 initial bytes to indicate the remaining size.

Q: What is the maximum size of a buffer?

(show example of bit shifting in Python)

​

  • Endianness: outside of FlatBuffers records (spatial index, attribute values, etc) are in little endian encoding

Hugo Ledoux & Hidemichi Baba

17 of 42

FlatGeobuf: File structure

How the file looks?

(Horace Williams, 2025)

Hugo Ledoux & Hidemichi Baba

18 of 42

FlatGeobuf: Magic bytes

Magic bytes

(Horace Williams, 2025)

“ASCII F , G, B, followed by the spec major version (currently 03), then F,G,B again, then the spec patch version (currently 01).”

Hugo Ledoux & Hidemichi Baba

19 of 42

FlatGeobuf: Header

Header

(Horace Williams, 2025)

Hugo Ledoux & Hidemichi Baba

20 of 42

FlatGeobuf: Header

For more detailed header schema, see their GitHub https://github.com/flatgeobuf/flatgeobuf/blob/master/src/fbs/header.fbs

Header content

(Horace Williams, 2025)

  • name: String - name of the file
  • envelop: f64/double array - 4 element bounding box (minX, minY, maxX, maxY)
  • geometry_type: GeometryType - FlatBuffers enum describing which geometry type appears (only when single geometry type)
  • has_z: boolean - if geometries have Z coordinates
  • has_m, has_t, has_tm: same as has_z
  • columns: Array of flatbuffer Columns.
  • features_count: ulong (64bit unsigned int) - Number of features in the dataset.
  • index_node_size: ushort (16-bit unsigned int) - default 16 - This represents the branching factor of the RTree
  • crs: Crs - custom flatbuffer type - Specifies the CRS for the dataset.
  • title, description - Arbitrary strings for dataset description.
  • metadata: string, but expected to encode an arbitrary JSON object containing key/value metadata about the dataset

​

Hugo Ledoux & Hidemichi Baba

21 of 42

FlatGeobuf: Header

Feature

(Horace Williams, 2025)

Hugo Ledoux & Hidemichi Baba

22 of 42

FlatGeobuf: Features

Feature schema

(Horace Williams, 2025)

table Feature {

geometry: Geometry; // Geometry

properties: [ubyte]; // Custom buffer, variable length collection of key/value pairs (key=ushort)

columns: [Column]; // Attribute columns schema (optional)

}

​

Feature

table Geometry {

ends: [uint]; // Array of end index in flat coordinates per geometry part

xy: [double]; // Flat x and y coordinate array (flat pairs)

z: [double]; // Flat z height array

m: [double]; // Flat m measurement array

t: [double]; // Flat t geodetic decimal year time array

tm: [ulong]; // Flat tm time nanosecond measurement array

type: GeometryType; // Type of geometry (only relevant for elements in heterogeneous collection types)

parts: [Geometry]; // Array of parts (for heterogeneous collection types)

}

​

Geometry

(Horace Williams, 2025)

Hugo Ledoux & Hidemichi Baba

23 of 42

FlatGeobuf: limitations: only single-dimensional array

How do you encode geometry?

“Nesting vectors is not supported, instead you can wrap the inner vector with a table.” (FlatBuffers, 2025)

Hugo Ledoux & Hidemichi Baba

24 of 42

FlatGeobuf: limitations: only single-dimensional array

Point

Vertices are flattened

Geometry {

xy: [1,2],

z: [3],

type: 1

}

{"type":"Point","coordinates":[1,2,3]}

LineString

Geometry {

xy: [1,2,4,5]

z: [3,6],

type: 2

}

​

{"type":"LineString","coordinates":[[1,2,3],[4,5,6]]}

Polygon

Geometry {

// All coordinates from the ring flattened

// First coordinate is still repeated to close the ring as in WKT/GeoJSON/etc

xy: [-118.4765625,33.92578125,-118.125,33.92578125,-118.125,34.1015625,-118.4765625,34.1015625,-118.4765625,33.92578125]

type: 3

}

​

{

"type": "Polygon",

"coordinates": [

[

[-118.4765625,33.92578125],

[-118.125, 33.92578125],

[-118.125, 34.1015625],

[-118.4765625, 34.1015625],

[-118.4765625, 33.92578125]

]

]

}

(Horace Williams, 2025)

Hugo Ledoux & Hidemichi Baba

25 of 42

FlatGeobuf: limitations: only single-dimensional array

How about polygon with inner rings?

Polygon (1 outer, 1 inner ring)

Geometry {

// all coordinates from BOTH rings flattened

xy: [-118.4765625,33.92578125,-118.125,33.92578125,-118.125,34.1015625,-118.4765625,34.1015625,-118.4765625,33.92578125,-118.24447631835938,34.0521240234375,-118.24310302734375,34.0521240234375,-118.24310302734375,34.053497314453125,-118.24447631835938,34.053497314453125,-118.24447631835938,34.0521240234375]

​

// 1-indexed coordinate number for last point per ring

ends: [5,10],

​

type: 3

}

​

{

"type": "Polygon",

"coordinates": [

[

[-118.4765625,33.92578125],

[-118.125, 33.92578125],

[-118.125, 34.1015625],

[-118.4765625, 34.1015625],

[-118.4765625, 33.92578125]

],

[

[-118.24447631835938, 34.0521240234375],

[-118.24310302734375, 34.0521240234375],

[-118.24310302734375, 34.053497314453125],

[-118.24447631835938, 34.053497314453125],

[-118.24447631835938, 34.0521240234375]

]

]

}

​

(Horace Williams, 2025)

Hugo Ledoux & Hidemichi Baba

26 of 42

FlatGeobuf: limitations: only single-dimensional array

What about these?

CircularString = 8,

CompoundCurve = 9,

CurvePolygon = 10,

MultiCurve = 11,

MultiSurface = 12,

Curve = 13,

Surface = 14,

PolyhedralSurface = 15,

TIN = 16,

Triangle = 17

​

(Horace Williams, 2025)

Hugo Ledoux & Hidemichi Baba

27 of 42

FlatGeobuf: property encoding

enum ColumnType: ubyte {

Byte, // Signed 8-bit integer

UByte, // Unsigned 8-bit integer

...

DateTime, // ISO 8601 date time

Binary // General binary type intended to be application specific

}

(Horace Williams, 2025)

We encode metadata of properties as `Column` table

​

table Column {

name: string (required); // Column name

type: ColumnType; // Column type

title: string; // Column title

description: string; // Column description (intended for free form long text)

…

}

​

(Horace Williams, 2025)

Hugo Ledoux & Hidemichi Baba

28 of 42

FlatGeobuf: property encoding

How do we encode a data structure that we can’t anticipate?

// Custom buffer, variable length collection of key/value pairs (key=ushort)

properties: [ubyte];

(Horace Williams, 2025)

Just an array of bytes meaning we can embed anything we want

Hugo Ledoux & Hidemichi Baba

29 of 42

FlatGeobuf: Spatial index

Spatial index

(Horace Williams, 2025)

Hugo Ledoux & Hidemichi Baba

30 of 42

FlatGeobuf: Binary search

Binary search

(Algorithmica, 2025)

Hugo Ledoux & Hidemichi Baba

31 of 42

FlatGeobuf: Feature sorting and spatial index

“Packed” R-Tree

(Horace Williams, 2025)

Q:

  • Which node will not be “packed”?
  • How many bytes are needed to encode one node?
  • How many nodes will we have to encode 1000 features?
  • 4 little-endian Doubles/f64 for bounding box (minX, minY, maxX, maxY)
  • Byte Offset: 64-bit unsigned int

Hugo Ledoux & Hidemichi Baba

32 of 42

FlatGeobuf: Feature sorting and spatial index

Example of not “Packed” R-Tree

https://wkgreat.github.io/spatial-index/rtree.html

Hugo Ledoux & Hidemichi Baba

33 of 42

FlatGeobuf: Feature sorting and spatial index

Hilbert ordering

Hilbert ordering

File layout

(Horace Williams, 2025)

Hugo Ledoux & Hidemichi Baba

34 of 42

FlatGeobuf: Feature sorting and spatial index

Steps to sort features and construct R-Tree

  • Calculate the centroid of the feature geometry (x, y)
  • Encode it to a single unsigned 32-bit Hilbert
  • Sort features according to those numbers
  • Write features in the order with size
  • Construct R-Tree from bottom

(Horace Williams, 2025)

Hugo Ledoux & Hidemichi Baba

35 of 42

Why sorting feature matters?

Hugo Ledoux & Hidemichi Baba

36 of 42

FlatGeobuf: CPU cache

(Zoran, 2022)

Hugo Ledoux & Hidemichi Baba

37 of 42

FlatGeobuf: Batching I/O

“features are hilbert-sorted before being indexed and written to allow for more efficient I/O batching when looking data up via the index” (Horace Williams, 2025)

Hugo Ledoux & Hidemichi Baba

38 of 42

FlatGeobuf: HTTP access

Packed R-Tree + Hilbert ordering

  • Retrieve header part and read information of spatial index e.g. branching factor
  • Access to R-Tree nodes in streaming manner and get offset bytes
  • Access to features (hopefully batching requests)

(FlatGeobuf demo, 2025)

Hugo Ledoux & Hidemichi Baba

39 of 42

What affects read performance over HTTP?

Hugo Ledoux & Hidemichi Baba

40 of 42

FlatGeobuf: limitations

Downsides

  • Complexity
  • Not human readable
  • Immutable: Not designed for updates
  • Non-spatial indexing

(Horace Williams, 2025)

Hugo Ledoux & Hidemichi Baba

41 of 42

Great references

Hugo Ledoux & Hidemichi Baba

42 of 42

Examples

https://github.com/HideBa/geo5019-flatgeobuf-sample

Hugo Ledoux & Hidemichi Baba