Ceilometer

The OpenStack Metering Project

15 Oct 2012 @ ODS Grizzly

Doug Hellmann

aka doughellmann/dhellmann (twitter/irc)

doug.hellmann@dreamhost.com

Nick Barcet

aka nijaba (twitter/irc)

nick.barcet@canonical.com

15 Oct 2012 @ ODS Grizzly

Doug Hellmann

aka doughellmann/dhellmann (twitter/irc)

doug.hellmann@dreamhost.com

Nick Barcet

aka nijaba (twitter/irc)

nick.barcet@canonical.com

What About Billing?

  • Billing has been left out of OpenStack core so far as it was not the primary problem and is not a trivial one...
  • Yet almost every OpenStack deployment needs a way to track usage information

Billing: 3 Step Process

Metering

Collect usage data

Rating

Transform usage data into billable items and calculate costs

Billing

Create invoice, collect payment

Metering

Collect usage data

Rating

Transform usage data into billable items and calculate costs

Billing

Create invoice, collect payment

Metering

- Uses a single database accessible to the rating tool.

Rating

- Policies are left up to the deployer.

- Billable item * quantity = cost in currency

- Currency often is money, but can be trees, carbon equiv, etc. in case of private clouds

Billing

- Emits bills

- Collects funds

- Assigns credits

Ceilometer is Metering

Usage data collection is the ONLY thing common to all clouds

  • rating policies vary greatly
  • billing, if it exists, also has great diversity

Ceilometer only aims at providing a simple API from which to fetch usage data for rating/billing engines which will be defined independently and outside of the OpenStack project

Uses for Metering

  • Billing
  • Auditing
  • Capacity Planning

Problems to Solve

  • Collecting per user/tenant usage data
    • For every resource
    • From every OpenStack component
    • In a single place

  • Retrieving usage data
    • From a single place

  • Doing this with an open source project
    • Everyone did this in their own corner in the past :-(

Ceilometer Begins

  • Started in May 2012

This is a list of the companies that contributed to the project so far.

We are very open and would love to add your logo here at the next summit!

Ceilometer Rises

  • Developed in StackForge
    • Same process as OpenStack
  • Minimal set of meters defined

  • Targeting OpenStack core
    • incubation pending

Design Requirements

  • Scalable
    • …if your database is too

Design Requirements

  • Scalable
    • …if your database is too
  • Message signature
    • Non-repudiation built in

Design Requirements

  • Scalable
    • …if your database is too
  • Message signature
    • Non-repudiation built in
  • Only one entry point to get data

Design Requirements

  • Scalable
    • …if your database is too
  • Message signature
    • Non-repudiation built in
  • Only one entry point to get data
  • Extensible, add your own:
    • Agent
    • Agent plugin
    • Storage engine
    • Meters

Design Requirements

  • Scalable
    • …if your database is too
  • Message signature
    • Non-repudiation built in
  • Only one entry point to get data
  • Extensible, add your own:
    • Agent
    • Agent plugin
    • Storage engine
    • Meters
  • Use openstack-common components

Design Requirements

  • Scalable
    • …if your database is too
  • Message signature
    • Non-repudiation built in
  • Only one entry point to get data
  • Extensible, add your own:
    • Agent
    • Agent plugin
    • Storage engine
    • Meters
  • Use openstack-common components
  • Accept data from many sources

Data Triggers

Ceilometer inputs are generated three ways

User Action

Creating, modifying, or deleting a resource

Audit

Regular audit events stating usage generated by the service

Polling

The ceilometer agent asks the service for data periodically

User Action

Creating, modifying, or deleting a resource

Audit

Regular audit events stating usage generated by the service

Polling

The ceilometer agent asks the service for data periodically

Polling is only used when audit events are not generated by a project for a resource being metered.

Events are preferred because it makes it easier to scale Ceilometer to handle the extra resources

Meter Categories

Ceilometer handles 3 types of meters

Cumulative

Increasing over time (instance hours)

Gauge

Discrete items (floating IPs, image uploads) and fluctuating values (disk I/O)

Delta

Changing over time (bandwidth)

Cumulative

Increasing over time (instance hours)

Gauge

Discrete items (floating IPs, image uploads) and fluctuating values (disk I/O)

Delta

Changing over time (bandwidth)

As we described earlier, Ceilometer is really a system of pluggable components, any of which can be enabled or swapped for a different implementation, depending on your needs.

The collector receives metering event data and writes it to the data store. The data storage API is pluggable, and the current release includes a MongoDB backend. The SQLAlchemy backend to let us store data in MySQL or Postgresql is under development.

Other plugins could be created to write the metering data to your favorite Big Data storage system.

In addition to listening for ceilometer's own metering messages, the collector uses a set of plugins to listen for notification messages coming from other OpenStack services. This lets us recognize, for example, that a new instance has been booted so we can start recording information about it.

Using the existing notification messages lets ceilometer integrate with the other services without requiring any modifications to those services.

For continuous monitoring of resources we have two separate agent processes. The compute agent runs on each hypervisor server and polls the hypervisor for statistics about each instance running there.

This is where we get information such as CPU time, bandwidth, etc. The pollsters are loaded from another set of plugins, so it is possible to write your own custom polling code to run in the compute agent and collect your own statistics.

The compute agent sends metering messages in a standard format to the collector to be processed and stored. The collector scales horizontally so it is possible to run multiple copies attached to the same message bus and distribute load.

Because all of the metering data is traveling over the message bus you can replace the data storage aspect of the collector with your own application, configured to receive the same messages.

The Central Agent runs on a management node and uses a different set of plugins to poll the other OpenStack services such as Glance and Cinder.

This is where we get information about images, for example.

The central agent sends metering messages in that same standard format directly to the collector to be processed and stored.

To use the data collected by ceilometer, a billing or rating system can call the REST-style query API.

The current API is an admin-style API meant specifically to export data in bulk for billing systems.

However, we do plan...

...to implement a user-level API which will also let us integrate with Horizon to present usage details to end users.

In the mean time,...

Simple REST API

Sum

GET /v1/resources/(resource)/meters/(meter)/volume/sum

Maximum

GET /v1/resources/(resource)/meters/(meter)/volume/max

Duration

GET /v1/resources/(resource)/meters/(meter)/duration

Raw Events

GET /v1/resources/(resource)/meters/(meter)

http://ceilometer.readthedocs.org/en/latest/api.html

Sum

GET /v1/resources/(resource)/meters/(meter)/volume/sum

Maximum

GET /v1/resources/(resource)/meters/(meter)/volume/max

Duration

GET /v1/resources/(resource)/meters/(meter)/duration

Raw Events

GET /v1/resources/(resource)/meters/(meter)

http://ceilometer.readthedocs.org/en/latest/api.html

...the existing API support a variety of queries about the resources being metered and the values of the meters.

Sum - Add volume values from the matching events and return the result

Maximum - Find the largest volume value in the matching events and return it

Duration - Calculate the time range for which matching events exist (used to calculate instance hours, volume lifetime, etc.)

Raw Events - Retrieve the recorded events for more complex processing

Parameters:

  • resource – The ID of the resource.
  • meter – The name of the meter.
  • start_timestamp – ISO-formatted string of the earliest time to include in the calculation.
  • end_timestamp – ISO-formatted string of the latest time to include in the calculation.
  • search_offset – Number of minutes before and after start and end timestamps to query to compensate for discrete events falling outside of the true window of interest for a continuous query such as duration.

That's where the project is right now...

Roadmap

Grizzly

H

Folsom

  • Delivered last week
  • Collects base metering
    • nova
    • glance
    • cinder
    • quantum
  • Basic API access
  • Incubated Project
  • User accessible API?
  • Integration example with Horizon
  • New agents for other openstack components
    • Swift
    • Heat?
  • New uses of collector?
  • SQLAlchemy storage driver
  • Core Project
  • TBD

Grizzly

H

Folsom

  • Delivered last week
  • Collects base metering
    • nova
    • glance
    • cinder
    • quantum
  • Basic API access
  • Incubated Project
  • User accessible API?
  • Integration example with Horizon
  • New agents for other openstack components
    • Swift
    • Heat?
  • New uses of collector?
  • SQLAlchemy storage driver
  • Core Project
  • TBD

...we have just completed our Folsom release, version 0.1

It includes metering for resources owned by nova (instances), glance (images), cinder (storage), and quantum (networks)

For Grizzly our main goal could be summed up as "better integration," both with the community and with the other OpenStack projects.

We will be working on extending our API, building notification listeners for swift and possibly heat, completing our integration with quantum, and completing the SQLAlchemy storage backend.

Our ultimate goal is being brought in as a core project during the H time frame.

In the mean time, the existing system has a minimum level of functionality that enabled us...

DreamHost Use Case

  • New Public Cloud Service
  • Existing Billing System
  • Existing Users and Accounts

...to start using it at DreamHost.

We are an established hosting provider, BUT

Our DreamCompute cloud is a new service.

We knew we needed a metering system that worked with OpenStack and our existing billing system, so signed on to contribute to Ceilometer.

After we had enough of ceilometer running, the first step to integrating with our other systems was to ...

Configuring Ceilometer

  • Measure exactly what we want to bill for
    • instance hours
    • block storage
    • image uploads
    • bandwidth

...set it up to measure the information we care about.

possibly including but not limited to

instance hours

block storage

image uploads

bandwidth

In fact, bandwidth turned out to be an unusual challenge...

Customizing Ceilometer

  • Custom Bandwidth Meter
    • No charge for traffic "inside" DreamHost
    • Don't expose infrastructure details to customers
    • Measure at the router, not the VIF

...because we do not want to charge for traffic "inside" DreamHost's networks

even if those networks are not inside DreamCompute

and we didn't want to expose our data center infrastructure details to customers to achieve this

we ended up creating a custom meter that collects data from the router instead of the interface on the instance

after we were gathering the meter data...

Consuming Meter Data

...the next step was to get it into our billing system

The DreamHost Usage Data Exporter, or DUDE, knows which meters we want and how to map our account ids to project ids in keystone.

It runs daily and processes data from the ceilometer API, uploading the results to some new internal APIs for our own services.

Creating the DUDE while we were building ceilometer gave us an opportunity to field-test the ceilometer API, and we uncovered some missing pieces from the original API design.

Abides failure: failure to run, failure to save the updates -- don't generate duplicate usage details

Questions?

Nick Barcet

aka nijaba (twitter/irc)

nick.barcet@canonical.com

http://launchpad.net/ceilometer

http://ceilometer.readthedocs.org

freenode: #openstack-metering

email: openstack-dev [ceilometer]

Doug Hellmann

aka doughellmann/dhellmann (twitter/irc)

doug.hellmann@dreamhost.com

Nick Barcet

aka nijaba (twitter/irc)

nick.barcet@canonical.com

http://launchpad.net/ceilometer

http://ceilometer.readthedocs.org

freenode: #openstack-metering

email: openstack-dev [ceilometer]

Doug Hellmann

aka doughellmann/dhellmann (twitter/irc)

doug.hellmann@dreamhost.com