Lookup Service Records for perfSONAR

Introduction

Our Simple Lookup Service (LS) is designed to be very general, able to store and retrieve any type of record, where a record is a set of key/value pairs. This document defines a specific use of the Lookup Service for perfSONAR. We describe the different types of records used by perfSONAR to register network related information.

Records

The Simple Lookup Service defines a base “record”, and four records that  inherit the base “record”. These records have been organized into two packages: network and directory. Section 1, 2 and 3 defines the keys for these records. Section 4 defines perfSONAR specific records. All the perfSONAR records inherits one of the general LS record types.

Although we do define different types of records, this is opaque to the lookup service backend server. This allows a single implementation of the LS to support different applications simultaneously. New records can be added dynamically. For example, if the OSCARS IDC community wants to use the same LS, then OSCARS clients can start using the lookup service right away. No change is required in the implementation or deployment of the servers. Each community will have special client libraries that will take care of generating the appropriate records and registering them with the lookup service. The client libraries are also responsible for validating the records.

Figure 1 represents the hierarchy of the records.

Figure 1. Records hierarchy

Keys

Keys must be alpha-numeric. Keys are defined for each record type but users may also add custom keys.

Values

The lookup service allows the following types of values:

1. List of one or more strings
2. ISO 8601 timestamp and date expressed as string

Reserved characters in values:

1. comma (,) - A comma is used in lists to separate the items in the list
2. [] - used to express list
3. “” - used to express string

1. Base Record

The lookup service base record contain defines  the  following keys.

2. Network package

The network package defines 3 records: service, interface and host. This section defines the keys for these record types.

These records extend the “base” record.

2.1 Service record

2.2 Interface record

2.3 Host record

3 Directory package

The directory package currently defines one record: the person record.

3.1 Defined keys for person record

4 Additional Reserved keys

5 perfSONAR records

perfSONAR defines the following types of lookup service records to register network related information. They are:

1. psservice - to register perfSONAR services
2. pshost - to register perfSONAR hosts
3. psinterface - to register perfSONAR network interfaces
4. pstest - to register test details (obsolete. Replaced by psmetadata)
5. psperson - to register perfSONAR admin/contact information
6. psmetadata - to register metadata information about MAs (esmond)

Figure 2. perfSONAR Records

As shown in Figure 2, the psservice inherits service record. This implies that a psservice record contains

all the keys defined for the base record and the general “service” record. In addition, it contains the psservice specific keys and zero or more user-defined keys.

Similarly, the psinterface inherits interface record, pshost inherits host record and psperson inherits the person record.

5.1 psservice records (type=service)

A psservice record MUST contain the following keys:

1. type (of record)
2. service-type
3. service-locator

Defined keys for psservice

5.1.2 Values for psservice-eventtypes

5.1.3 Additional Reserved keys in psservice records

5.2 perfsonar Host records (type=host)

A perfSONAR host record (pshost) extends the host record. In addition to the keys defined in the host record, the following keys are defined.

5.3 psinterface records (type=interface)

The psinterface record extends the interface record.

5.4 pstest records (type=pstest)

The following keys are defined for the pstest records

Users may add any test parameters.

5.5 psperson records (type=person)

The psperson record extends the person record.

5.6 psmetadata records

psmetadata replaces type “pstest”. This was done to prevent confusion with parsers trying to determine what type of test they are dealing with.
Main difference between psmetadata and pstest is psmetadata contains a source and destination as an IP (and always an IP, just like in the MA). It does not contain any URIs to interface records. This will eliminate the need for dummy interface registrations.
It also contains other fields from the MA:

Notes

The expectation is that this will be the extent of the fields and none of the context-specific parameters will be registered to keep the records a predictable size.

• Supports group-domains and group-communities as well (not shown in example, but possible
• We can configure the LS registration daemon to only grab tests updated in a certain timeframe rather than registering stale tests
• Contains the psmetadata-ma-locator and psmetadata-uri that can be used to grab the metadata object directly from the LS. This will make an easy hand-off from the sLS client to the MA client to grab the actual data.

6 Creating new keys

In addition to these keys, the user can include custom keys. The naming convention for custom keys is as follows:

1. Key name must be unique.
2. Key name must be alphanumeric and may contain (-)
3. If key is common to a record or sub-record, use the type of record or sub-record type as prefix. For example, if a key is common to service records, then prefix will be “service-”. If it is common to psservice, then use “psservice-” and so on.
4. If a key is common across several records but cannot be a separate record-type then it can have its own namespace. Eg: location-

7 Creating new record types

The record types required to register the current perfSONAR information has already been defined. But, if new record types need to be defined, then follow the following guideline.

1. Check if one of the existing records can be used.  
2. If not, check if one of the existing records can be used with few additional keys. If yes, then create a new record type by extending the existing record and then add the custom keys. Keys must follow the naming convention specified for keys.
3. If not, extend the base record. Create a new package if required or use one of the existing packages.
4. A rule of thumb to create record types is:

1. If there are a group of keys that have multiple values and require a one to one association then those keys need to be moved to a separate record. For example, a host can have multiple interfaces and each interface will have its own mac and network address. Hence, the mac and network address can be moved to a separate record-type called “interface”.

8. Record Examples

Ping

{

    type: [“service”],

    location-sitename: [“esnet-nersc”],

    location-city: [“oakland”],

    location-state: [“ca”],

    location-country: [“us”],

    domains:[“es.net”],

    group-communities: [“ecenter”,”ustlas”],

    service-locator:[“nersc-owamp.es.net”, “nersc-owamp-v6.es.net”],

    service-type: [“ping”]

}

Owamp

{

    type: [“service”],

    location-sitename: [“esnet-nersc”],

    location-city: [“oakland”],

    location-state: [“ca”],

    location-country: [“us”],

    domains:[“es.net”],

    group-communities: [“ecenter”],

    service-locator:[“tcp://nersc-owamp.es.net:861”, “tcp://nersc-owamp-v6.es.net:861”],

    service-type: [“owamp”]

}

perfsonarbuoy MA (bwctl)

{

    type: [“service”],

    location-sitename: [“lbl”],

    location-city: [“berkeley”],

    location-state: [“ca”],

    location-country: [“us”],

 domains:[“es.net”],

 group-communities: [“pstoolkit”],

 service-locator:[“http://ps-lat.es.net:8085/perfSONAR_PS/services/pSB”]

    service-type:[“ma”],

    ma-type: [“bwctl”],

 psservice-eventtypes:["http://ggf.org/ns/nmwg/characteristics/bandwidth/achievable/2.0"],

ma-tests: [“lookup/test/d56a39de-fbef-4f2d-90e0-ccb4a17a0016”,”lookup/test/e26f69de-fbef-4f2d-90e0-ccb4a17a0016”,”lookup/test/a345d43d-fbef-4f2d-90e0-ccb4a17a0016”,”lookup/test/3f3a39de-fb8a-4f2d-ab65-ccb4a17a0016”]

}

pstest record

{

    type: [“pstest”],

   pstest-source:[“lookup/host/d56a39de-fbef-4f2d-90e0-ccb4a17a0016”],

   pstest-destination: [”lookup/host/3f3a39de-fb8a-4f2d-ab65-ccb4a17a7788”],

   pstest-type: [“bwctl”]

}

pshost record

{

    type: [“host”],

    host-name: [“ps-lat.es.net”],

     location-state: [“ca”],

    location-country: [“us”],

    domains:[“es.net”],

    group-communities: [“toolkit”],

    host-os-name:[“CentOS”],

    host-os-version:[“6”],

    host-os-kernel: [“3.5.4”],

    host-net-interfaces: [“lookup/interface/d56a39de-fbef-4f2d-90e0-ccb4a17a0016”]

}

psinterface record

{

    type: [“interface”],

    interface-name: [“ps-lat.es.net”],

    interface-addresses: [“192.187.1.1”],

    interface-mac: [“24:56:7e:2d:55:f1”]    

}

psperson record

{

 type: [“person”],

person-name: [“Adam Smith”],

person-emails: [“asmith@xyz.com”],

person-organization: [“XYZ”]

}

psmetadata records

Throughput Test
{{{
{
 "psmetadata-destination":["10.1.1.2"],
 "psmetadata-source":["10.1.1.1"],
 "psmetadata-eventtypes":[
    "packet-retransmits",
    "throughput"
 ],
 "psmetadata-tool-name":["bwctl/iperf3"],
 "expires":"2014-04-25T19:45:16.087Z",
 "psmetadata-ma-locator":[
    "http://10.0.1.10/esmond/perfsonar/archive",
    "https://10.0.1.10/esmond/perfsonar/archive"
 ],
 "psmetadata-measurement-agent":["10.1.1.1"],
 "state":"registered",
 "type":["psmetadata"],
 "psmetadata-uri":["/esmond/perfsonar/archive/bc9d33609e8f408fb2c0d9af52e9c876/"],
 "uri":"lookup/psmetadata/4d6a9941-2bef-483f-8aa7-fd3dd36c988f"
},
}}}

Ping Test
{{{
{
 "psmetadata-destination":["10.1.1.2"],
 "psmetadata-source":["10.1.1.1"],
 "psmetadata-eventtypes":[
    "packet-duplicates",
    "histogram-ttl",
    "packet-reorders",
    "packet-count-sent",
    "histogram-rtt",
    "packet-count-lost",
    "failures",
    "packet-loss-rate"
 ],
 "psmetadata-tool-name":["bwctl/ping"],
 "expires":"2014-04-25T19:45:16.118Z",
 "psmetadata-ma-locator":[
    "http://10.0.1.10/esmond/perfsonar/archive",
    "https://10.0.1.10/esmond/perfsonar/archive"
 ],
 "psmetadata-measurement-agent":["10.1.1.1"],
 "state":"registered",
 "type":["psmetadata"],
 "psmetadata-uri":["/esmond/perfsonar/archive/204d1b71b5b74f1cbe8b2b4465e20a1c/"],
 "uri":"lookup/psmetadata/f85fddf8-ff91-42d7-ac21-eac0404d0118"
},
}}}

Owamp Test
{{{
{
 "psmetadata-destination":["10.1.1.2"],
 "psmetadata-source":["10.1.1.1"],
 "psmetadata-eventtypes":[
    "time-error-estimates",
    "packet-duplicates",
    "histogram-ttl",
    "packet-count-sent",
    "packet-count-lost",
    "histogram-owdelay",
    "failures",
    "packet-loss-rate"
 ],
 "psmetadata-tool-name":["powstream"],
 "expires":"2014-04-25T19:45:16.162Z",
 "psmetadata-ma-locator":[
    "http://10.0.1.10/esmond/perfsonar/archive",
    "https://10.0.1.10/esmond/perfsonar/archive"
 ],
 "psmetadata-measurement-agent":["10.1.1.1"],
 "state":"registered",
 "type":["psmetadata"],
 "psmetadata-uri":["/esmond/perfsonar/archive/9b0e9419dbe44d3cad9cbe1ef3f20010/"],
 "uri":"lookup/psmetadata/2f4f0a10-3de6-44ab-b3fd-2183475bc4a9"
}
}}}

Tracepath Test
{{{
{
 "psmetadata-destination":["10.1.1.2"],
 "psmetadata-source":["10.1.1.1"],
 "psmetadata-eventtypes":[
    "packet-trace",
    "failures",
    "path-mtu"
 ],
 "psmetadata-tool-name":["bwctl/tracepath"],
 "expires":"2014-04-25T19:45:16.103Z",
 "psmetadata-ma-locator":[
    "http://10.0.1.10/esmond/perfsonar/archive",
    "https://10.0.1.10/esmond/perfsonar/archive"
 ],
 "psmetadata-measurement-agent":["10.1.1.1"],
 "state":"registered",
 "type":["psmetadata"],
 "psmetadata-uri":["/esmond/perfsonar/archive/f5f9a7428f8941669f81b41b0a80166d/"],
 "uri":"lookup/psmetadata/ef328ba4-1651-4d6c-8ae5-6aacd9de2c1d"
}
}}}

9. Queries

Queries are also key-value based and uses the HTTP GET method. A typical query is of the form:

HTTP GET <some-url>/key1=value1&key2=value2&operator=ANY

10. Simple Lookup Service Curl Client Examples

perfSONAR will provide lookup service client libraries that make it easy to register, modify or query the lookup service. But, a user can use curl or any other http clients to access and use the lookup service. This section shows sample use of  “curl” to interact with the lookup service.

10.1 Record Management

10.1.1 Registration

curl -v -H "Content-Type: application/json" –X POST -d '{"type”:[ "service"],"domains”:[ "ESnet"],"service-locator":["nash-owamp.es.net"],"service-name":["ESnet-nash Ping Responder"],"service-type”:[ "ping"]}' http://ps4.es.net:8085/lookup/records

10.1.2 Renew

curl -v -H "Content-Type: application/json" -X POST http://ps4.es.net:8085/lookup/service/4e83be67-e706-46b7-b056-cc2c2430cc7f

10.1.3 Delete

curl -v -H "Content-Type: application/json" -X DELETE http://ps4.es.net:8085/lookup/service/4e83be67-e706-46b7-b056-cc2c2430cc7f

10.1.4 service record

curl -v -H "Content-Type: application/json" -X GET http://ps4.es.net:8085/lookup/service/4e83be67-e706-46b7-b056-cc2c2430cc7f

10.1.5 Access a particular key in service record

curl -v -H -X GET http://ps4.es.net:8085/lookup/service/4e83be67-e706-46b7-b056-cc2c2430cc7f/service-type

10.2 Query

curl -v -H "Content-Type: application/js" -X GET http://ps4.es.net:8085/lookup/records/?domains=es.net\&service-type=ping\&type=service\&operator=any

10.2.1 perfSONAR-PS Toolkit: Service Browsing

a. give me all the services

curl -v -H "Content-Type: application/js" -X GET http://ps4.es.net:8085/lookup/records/?type=service

b. give me all services of type X

curl -v -H "Content-Type: application/js" -X GET http://ps4.es.net:8085/lookup/records/?type=service\&service-type=X

c. give me all services of type X in project Y

curl -v -H "Content-Type: application/js" -X GET http://ps4.es.net:8085/lookup/records/?type=service\&group-communities=Y\&service-type=X

d. give me all services of type X in domain Z

curl -v -H "Content-Type: application/js" -X GET http://ps4.es.net:8085/lookup/records/?type=service\&group-domains=Y\&service-type=X

10.2.2 OSCARS: Discover Neighbor IDC

a. give me the IDC service for domain X?

curl -v -H "Content-Type: application/js" -X GET http://ps4.es.net:8085/lookup/records/?type=service\&service-type=idc\&group-domains=X

10.2.3 OSCARS: Friendly Names

a. What URN(s) corresponds to friendly name X?

curl -v -H "Content-Type: application/js" -X GET http://ps4.es.net:8085/lookup/services/?type=interface&interface-name=X&psinterface-type=oscars

10.2.4 OSCARS/perfSONAR: Finding a Domain’s Topology Service

a. What topology service has topology information on domain X?

curl -v -H "Content-Type: application/js" -X GET http://ps4.es.net:8085/lookup/services/?type=service\&service-type=ts\&ts-domains=X

10.2.5 perfSONAR: Find Measurement Archive by host

a. perfSONARBUOY MA - bwctl/owamp

(i). What MA(s) have bwctl measurement data for domain Y?

curl -v -H "Content-Type: application/js" -X GET http://ps4.es.net:8085/lookup/services/?pstest-eventtypes=http:%2F%2Fggf.org%2Fns%2Fnmwg%2Fcharacteristics%2Fbandwidth%2Fachieveable%2F2.0\&group-domains=Y

Returns test-id TID

curl -v -H "Content-Type: application/js" -X GET http://ps4.es.net:8085/lookup/services/?type=service\&service-type=ma\&ma-tests=TID

(ii). What MA(s) have owamp measurement data with buckets for project Y?

curl -v -H "Content-Type: application/js" -X GET http://ps4.es.net:8085/lookup/services/?pstest-eventtypes=http:%2F%2Fggf.org%2Fns%2Fnmwg%2Fcharacteristics%2Fbandwidth%2Fachieveable%2F2.0\&group-communities=Y

Returns test-id TID

curl -v -H "Content-Type: application/js" -X GET http://ps4.es.net:8085/lookup/services/?type=service\&service-type=ma\&ma-tests=TID

(iii) What MA(s) have bwctl measurement data for hostname Y?

This requires multiple queries. First the host record needs to be retrieved to get the host id. Then, the list of tests .

curl -v -H "Content-Type: application/js" -X GET http://ps4.es.net:8085/lookup/services/?type=host\&host-name=Y

(The above returns host id for host-name Y- YID)

curl -v -H "Content-Type: application/js" -X GET http://ps4.es.net:8085/lookup/services/?test-source=YID\&test-destination=YID\&operator=ANY

Returns test-id TID

curl -v -H "Content-Type: application/js" -X GET http://ps4.es.net:8085/lookup/services/?type=service\&service-type=ma\&ma-tests=TID

(iv). What MA(s) have measurement data of type X about IP Y?

This requires multiple queries. First the host record needs to be retrieved to get the host id. Then, the list of tests .

curl -v -H "Content-Type: application/js" -X GET http://ps4.es.net:8085/lookup/services/?type=interface\&interface-addresses=Y

returns IID

curl -v -H "Content-Type: application/js" -X GET http://ps4.es.net:8085/lookup/services/?type=host\&host-interface=IID

(The above returns host id  for host-name Y- YID)

curl -v -H "Content-Type: application/js" -X GET http://ps4.es.net:8085/lookup/services/?test-source=YID\&test-destination=YID\&operator=ANY

Returns test-id TID

curl -v -H "Content-Type: application/js" -X GET http://ps4.es.net:8085/lookup/services/?type=service\&service-type=ma\&ma-type=X\&ma-tests=TID

b. SNMP MA

(i). What MA(s) have snmp measurement data for domain Y?

curl -v -H "Content-Type: application/js" -X GET http://ps4.es.net:8085/lookup/services/?pstest-eventtypes=http:%2F%2Fggf.org%2Fns%2Fnmwg%2Fcharacteristic%2Futilization%2F2.0\&group-domains=Y

Returns test-id TID

curl -v -H "Content-Type: application/js" -X GET http://ps4.es.net:8085/lookup/services/?type=service\&service-type=ma\&ma-tests=TID

(ii) What MA(s) have snmp measurement data for interface Y?

curl -v -H "Content-Type: application/js" -X GET http://ps4.es.net:8085/lookup/services/?type=interface\&interface-name=Y

Returns interface-id IID

curl -v -H "Content-Type: application/js" -X GET http://ps4.es.net:8085/lookup/services/?type=host\&host-net-interfaces=IID

Returns host-id HID

curl -v -H "Content-Type: application/js" -X GET http://ps4.es.net:8085/lookup/services/?type=pstest\&test-source=HID

Returns test-id TID

curl -v -H "Content-Type: application/js" -X GET http://ps4.es.net:8085/lookup/services/?type=service\&service-type=ma\&ma-tests=TID

psmetadata

1. Get all throughput tests with a given source host
{{{
curl http://10.0.1.9:8090/lookup/records?psmetadata-source=10.1.1.1&psmetadata-event-type=throughput
}}}

2. Get all tests run by a given host
{{{
curl http://10.0.1.9:8090/lookup/records?psmetadata-measurement-agent=10.1.1.1
}}}

3. Get all BWCTL iperf3 tests in the world
{{{
curl http://10.0.1.9:8090/lookup/records?psmetadata-tool-name=bwctl/iperf3
}}}