1 of 29

Ceilometer

The OpenStack Metering Project

15 Oct 2012 @ ODS Grizzly

Doug Hellmann

aka doughellmann/dhellmann (twitter/irc)

doug.hellmann@dreamhost.com

Nick Barcet

aka nijaba (twitter/irc)

nick.barcet@canonical.com

2 of 29

What About Billing?

Billing has been left out of OpenStack core so far as it was not the primary problem and is not a trivial one...�
Yet almost every OpenStack deployment needs a way to track usage information��

3 of 29

Billing: 3 Step Process

Metering	Collect usage data
Rating	Transform usage data into billable items and calculate costs
Billing	Create invoice, collect payment

4 of 29

Ceilometer is Metering

Usage data collection is the ONLY thing common to all clouds

5 of 29

Uses for Metering

Billing
Auditing
Capacity Planning

6 of 29

Problems to Solve

Collecting per user/tenant usage data

For every resource
From every OpenStack component
In a single place

Retrieving usage data

From a single place

Doing this with an open source project

Everyone did this in their own corner in the past :-(

7 of 29

Ceilometer Begins

Started in May 2012

8 of 29

Ceilometer Rises

Developed in StackForge

Same process as OpenStack�

Minimal set of meters defined

Targeting OpenStack core

incubation pending

9 of 29

Design Requirements

Scalable

…if your database is too

10 of 29

Design Requirements

Scalable

…if your database is too

Message signature

Non-repudiation built in

11 of 29

Design Requirements

Scalable

…if your database is too

Message signature

Non-repudiation built in

Only one entry point to get data

12 of 29

Design Requirements

Scalable

…if your database is too

Message signature

Non-repudiation built in

Only one entry point to get data
Extensible, add your own:

Agent
Agent plugin
Storage engine
Meters

13 of 29

Design Requirements

Scalable

…if your database is too

Message signature

Non-repudiation built in

Only one entry point to get data
Extensible, add your own:

Agent
Agent plugin
Storage engine
Meters

Use openstack-common components

14 of 29

Design Requirements

Scalable

…if your database is too

Message signature

Non-repudiation built in

Only one entry point to get data
Extensible, add your own:

Agent
Agent plugin
Storage engine
Meters

Use openstack-common components
Accept data from many sources

15 of 29

Data Triggers

Ceilometer inputs are generated three ways

User Action	Creating, modifying, or deleting a resource
Audit	Regular audit events stating usage generated by the service
Polling	The ceilometer agent asks the service for data periodically

16 of 29

Meter Categories

Ceilometer handles 3 types of meters

Cumulative	Increasing over time (instance hours)
Gauge	Discrete items (floating IPs, image uploads) and fluctuating values (disk I/O)
Delta	Changing over time (bandwidth)

17 of 29

18 of 29

19 of 29

For continuous monitoring of resources we have two separate agent processes. The compute agent runs on each hypervisor server and polls the hypervisor for statistics about each instance running there.

This is where we get information such as CPU time, bandwidth, etc. The pollsters are loaded from another set of plugins, so it is possible to write your own custom polling code to run in the compute agent and collect your own statistics.

The compute agent sends metering messages in a standard format to the collector to be processed and stored. The collector scales horizontally so it is possible to run multiple copies attached to the same message bus and distribute load.

Because all of the metering data is traveling over the message bus you can replace the data storage aspect of the collector with your own application, configured to receive the same messages.

20 of 29

21 of 29

22 of 29

23 of 29

Simple REST API

Sum	GET /v1/resources/(resource)/meters/(meter)/volume/sum
Maximum	GET /v1/resources/(resource)/meters/(meter)/volume/max
Duration	GET /v1/resources/(resource)/meters/(meter)/duration
Raw Events	GET /v1/resources/(resource)/meters/(meter)

http://ceilometer.readthedocs.org/en/latest/api.html

...the existing API support a variety of queries about the resources being metered and the values of the meters.

Sum - Add volume values from the matching events and return the result

Maximum - Find the largest volume value in the matching events and return it

Duration - Calculate the time range for which matching events exist (used to calculate instance hours, volume lifetime, etc.)

Raw Events - Retrieve the recorded events for more complex processing

Parameters:

resource – The ID of the resource.
meter – The name of the meter.
start_timestamp – ISO-formatted string of the earliest time to include in the calculation.
end_timestamp – ISO-formatted string of the latest time to include in the calculation.
search_offset – Number of minutes before and after start and end timestamps to query to compensate for discrete events falling outside of the true window of interest for a continuous query such as duration.

That's where the project is right now...

24 of 29

Roadmap

Grizzly

H

Folsom

Delivered last week
Collects base metering

nova
glance
cinder
quantum

Basic API access

Incubated Project
User accessible API?
Integration example with Horizon
New agents for other openstack components

Swift
Heat?

New uses of collector?
SQLAlchemy storage driver

Core Project
TBD

25 of 29

DreamHost Use Case

New Public Cloud Service
Existing Billing System
Existing Users and Accounts

26 of 29

Configuring Ceilometer

Measure exactly what we want to bill for

instance hours
block storage
image uploads
bandwidth

27 of 29

Customizing Ceilometer

Custom Bandwidth Meter

No charge for traffic "inside" DreamHost
Don't expose infrastructure details to customers
Measure at the router, not the VIF

28 of 29

Consuming Meter Data

29 of 29

Questions?

Nick Barcet

aka nijaba (twitter/irc)

nick.barcet@canonical.com

http://launchpad.net/ceilometer

http://ceilometer.readthedocs.org

freenode: #openstack-metering

email: openstack-dev [ceilometer]

Doug Hellmann

aka doughellmann/dhellmann (twitter/irc)

doug.hellmann@dreamhost.com