SNS Face-to-Face Meeting, Jan 12-14 2015

Monday 12th

Scribe: AJ

Present: Greg Guyotte, Kay Kasemir, Marty Kraimer, David Hickin, Michael Davidsaver, Andrew Johnson, Sinisa Veseli, Timo Korhonen, Bob Dalesio, Ralph Lange, Daron Chabot, Jim Kohl, Steven Hartman, Matthew Pearson, Bogdan Vacaliuc, Klemen Vodopivec, Marie Yao, Carl Lionberger

Online occasionally: Matej Sekoranja, Greg White

Steven: Welcome, Introductions,

Andrew: Agenda, minutes

Kay: V4 for Neutron Data & issues
https://drive.google.com/file/d/0Bwa7KMoZ5dT5M0xmZGhjQlA2Mkk/view?usp=sharing

BD: X, Y, time-of-flight?

SH: Yes. Some detectors are sparse.

Kay: At 60Hz get new pulse, 0/1/multiple packets with arbitrary number of pixels, timestamp with pulse ID, new proton charge.

Kay: Talk cont…

SH: Desire to put everything from the experiment into the Nexus files, full metadata. Client monitors a text file with the PVs to include.

BD: Want SNS to look at the BNL metadata store to see if that makes sense.

Kay: Talk cont…

Klemen V: nED: V4 Server for neutron detectors
https://drive.google.com/file/d/0Bwa7KMoZ5dT5OVg5ckhoRE5wbXc/view?usp=sharing

SH: ROC and FEM boards are FPGA, generate events with timestamp. nED has custom PCI card with fibre-optic inputs which reads events and combines them. Neutron TOF interested in microsecond-scale timings, so 10MHz clock is sufficient.

BD: LBL using MRF event-link to regenerate 900MHz clock.

Bogdan: Relative timestamps from ROC, corrected in the nED and synced to GPS time.

KV cont...: Like area-detector but using different data structure, based on asynPortDriver.

SV: Copy of AD code or new code?

KV: Started from scratch, new code. Many different plugins interested in different kinds of data, process responses. Couldn’t fit all the data into NDArray.

BV: Zero-copy, data shared where possible.

KV: Linux kernel limitation on DMA buffer size (4MB) which does require copying data out for larger or higher-rate data items. Amount of data varies with pulse and when it is in the pulse.

BD: What kind of calibration do you do?

BV: Find offsets and integrator sample start, … -- linearization of analog front-end.

SH: Not normalization or background subtraction at this point, currently handled at Mantid level. Live display does not have background subtracted.

RL: Do you ever have to back-apply calibration?

MY: Yes, sometimes.

KV: Raw data transform routine may be called multiple times per 60Hz pulse.

MD: (internals of data storage types, etc. In principle, ‘shared_data’ could wrap existing array memory without need to copy. But that requires array to remain unchanged while it’s being served via pvAccess)

KV: Reserve memory in the shared vector for the next pulse/packet. Documentation on shared vector is lacking…

MD: Yes, can we get together to hear your frustrations. No user feedback on shared vector code at all. It allows you to provide your own allocator with the existing class(es).

KV: nED data flow diagram, copying out of DMA buffer only happens in the PVA plugin; other plugins might have to make other copies only if they need to modify the data.

-- Break --

New scribe: KK

Matt Pearson: ADnED: V4 client using AreaDetector for neutrons
https://drive.google.com/file/d/0Bwa7KMoZ5dT5d3pOM1RfOHNUWHM/view?usp=sharing

SH: “nED” = New neutron data acq. software, meaning of acronym has been lost. Neutron event distributor?

MP: ADnED basically pvAccess monitor and AreaDetector Driver, with some filtering, creating NDArrays for usual AD plugins. In addition, custom ROI plugin that supports multiple ROIs.

NDPluginROIStat to be added to AD in upcoming release.

BD: Experience getting AD from github?

MP: Works fine.

BD: Neutron detectors?

SH: Most of them He3 tubes, detecting created electrons at end of tube, HV bias. Tubes arranged in plane or cylinder. Also some scintillating Photo Multiplier Tube detectors. Unclear boundary: What to display in EPICS/CSS to monitor ongoing experiment, what to leave to Mantid for comprehensive analysis?

DH: Similar at Diamond. Controls group (technical) and GDA group (science). Tools provided by controls might be used or not, requirements unclear.

MP: GDA does control beam line devices via Channel Access, with IOCs provided by controls group.

SH: Division between controls + data acq. vs. data analysis. Still unclear where to draw line for basic XY plot, energy plot, background subtraction, ...

DH: V4 areaDetector Integration
https://drive.google.com/file/d/0Bwa7KMoZ5dT5akJ4RHRNdzQzUk0/view?usp=sharing

DH: Originally able to fill 10 Gig Ethernet link. Soon after major merge including protocol change performance as before, except every 10-15s sudden drop for a second, then back up. Performance now steady but only reaches ⅔ rate of original tests:. Overall not quite good enough to meet desired 90 frames/sec for 10MB images. Test computers were also changed just before current measurements, so no perfect comparison possible.

MD: Helped DH to avoid array copy on server side. FOr client side,  areadetector handles all array data allocation, wasn’t able to avoid copy from pvAccess client to AD data structures.

AJ: Would AD plugins connect based on name, without further configuration, using either local NDArray or V4 monitor if name cannot locally resolve?

DH: Yes. Plugins wouldn’t know, run without modification. AD handles the connections.

MK, DH, KK: Need to discuss details of shallow copy, avoiding copies, using references to constant data.

12:25: Lunch

Present: +John Sinclair

DH: V4 Interoperability Requirements
https://drive.google.com/file/d/0Bwa7KMoZ5dT5M0h4UVE2aVBrQ0k/view?usp=sharing

BD, TK: BNL, ESS agree on goal to be able to completely replace Channel Access with pvAccess

DH: 2 years ago, was able to test CSS with pvAccess. No success recently.

MS: PVManager has been unbundled into “diirt”, new URLs, …

DM: Need to perform put-callback from UI?

Several: Not from UI, but pvManager should support this for automation clients.

MK: What is missing with implementation for displaying enums, arrays?

DH: Next slide.

RL: Connection issues when using just plain PV name, where same name exists on network as ca://.. as well as pva://..?

BV, GG: Similar with python, need to use full PV name with ca://..?

MS: Remaining bundling issues prevent pvManager to work with pvAccess

SH: Using ChannelAccess for GUI for now. When will BNL/ESS exclusively use pvAccess?

BD,TK: ESS will be the first site. Earlier, maybe selected BNL beam lines.

RL: Currently pvaSrv / pvAccess does not implement *everything* that CA does. Some of the differences are rather esoteric, but they are different.

KK: SNS not looking to replace Channel Access. pvAccess added for neutron data. Would like to be able to display sub-elements of neutron data structure for display in CSS.

RL: pvAccess channel names have no content limitation. ‘.’ dots could be part of the name. How to access structure.field that way?

KK: SNS Scans
https://drive.google.com/file/d/0Bwa7KMoZ5dT5eXFhOGl2ZUp1NGc/view?usp=sharing

KK: DH asked why the new language, rather than using Python say? Answer is … automated logging of steps, helps find user errors, timeout due to underspecifying reliable move sizes.

SV: How big is the Nexus file you’re saving?

MY: Could be 40GB.

SH: Varies a huge amount, by beamline. Some want each row as a separate file, others want all rows combined in one file, with markers for steps.

BD: Does Mantid do any indexing of files at all?

SH: Permits slicing, not that much data so no need for indexing. Worst case a few 10s of seconds.

SV: PVA Py Status
https://drive.google.com/file/d/0B-y9m7y_YtcyMzFBZW9EYnVPcFE/view?usp=sharing

SV, AJ, RL: Should publish generated documentation via jenkins, also build on cloudbees as well as at APS

MK: Does ‘put’ for enum support writing the label’s string in addition to the numeric value?

KV: Used to fail.

SV: Should be handled. If not, will be fixed.

MD: Support for threading? Can get 1000 PVs in parallel, receiving callback as data arrives? Both get & put with callbacks would be desirable.

SV: To be implemented.

DH: Request ability to set the type id on the structure; supply when creating the introspection data.

MD: Are only unions missing? Arrays, structures are supported?

SV: Yes.

MD: Request numPy support, return arrays as a numpy array. Higher priority than threading.

TK: Support for channelRPC?

SV: Yes, RPC server as well as client.

MD: RPC server should support async. reply to not block when handling one client’s request.

MK: How are you doing that on the server?

SV: Simple to do.

MD: All servers are currently synchronous. pvAccess API doesn’t provide “I’ll give you an answer later” response. Could be changed relatively easily, but currently parallel responses are impossible.

Greg White: C++, Java, Matlab, Python API should be aligned, ‘easy’ pvAccess API

Ralph: User requirements for pvaSrv, dbGroup etc.

RL: (describing pvaSrv functionality) Future requirements: name is dbGroup, allow grouping multiple records together from a single IOC. 2 Flavors: 1: Atomic access to multiple PVs (Record.field) as a single NType (TBD this week). 2: Map arbitrary pvData structure to different fields on different records. 1 pulls value + metadata, 2 pulls individual values without the metadata.

GW: Why are they different, 1 variation of 2?

RL: 1 is a standard NType, 2 client specifies the structure, not a NType.

MD: I get the use case, but 2 seems a little generic.

MK: Isn’t 1 NTMultiChannel

RL: No, that only has one timestamp and alarm; for this each has its own.

MK: Guobao was using something like that already.

MD: Can I see a simple example use-case?

BV: Use case in SNS nED detector representation. Currently one CA channel for each ‘register’ of hardware. Want to load/save set of those as one unit; homogeneous array. Using CSS PVTable, which operates channel by channel, not really atomically as would be ideal.

RL: Also want user input for how to specify processing for a group or records?

MD: I think of pvGroup as a way to implement batch operations, 1 operation on multiple PVs.

RL: Nobody else can process while group owns the locksets. Question is “Put, process, put, process” etc. or “put, put, put, process”?

MD: What CA does is a reasonable default for batch operations.

RL: For a put I need an array of structures plus an array of which channels to process, ordered?

RL: Groups programmatically created on IOC. Later, RPC service may allow clients to create groups on running IOC, using that API.

MD: Need ‘anonymous’ groups, created by client for a group put, based on information sent in the ‘request’.

RL: For monitors on a group, what kinds of things should I be able to set when creating a monitor? (Update reason in there somewhere).

GW: Would it simplify to not start with the interactive specifying of groups

RL: Thats the plan

MD: With the understanding that that’s what I’m after eventually. Stepwise is good though.

RL: pvsSrv is for a single IOC only. We have code already that does non-atomic grouping across IOCs. This was thought to be a model for dbGroup, did we drop that idea?

MD: Thought it was too complex.

GW: We stated but it got frightening, punted to you to look at?

RL: This is a thinner layer, no locking issues in the IOC; MK’s original code had this in it.

KK: You said originally there would be a programmatic API, when going across IOC who holds the list of channels?

RL: The client. There are no guarantees of atomicity.

MD: Sounds not interesting or useful to be honest.

RL: Thanks, that covers what I wanted to discuss.

MK: pvDatabaseJava

MK: Should we do this? Might be possible in a few weeks.

MD: Don’t see why you’d want to do that.

KK: See more reason to do pvAccess servers in Python than in Java.

GW: I see pvDatabaseJava like pvManager on server-side, serving aggregations. Some functionality of a gateway? Use-case: Subscribe to all BPMs, or timing-synced RF for a large part of the accelerator, but not at full rate. Not covered well by CA, subscribe to hundreds of PVs, manage monitoring, then manage processing to give the result.

MD: Sounds like an IOC we created to monitor and feed data as a vector.

GW: Not as configurable.

BD: Gather and Channel Finder.

GW: Nearly done, but not all at once; nice to have a single thing that does that, configurable from the client side. Want a more polished tool to handle those use-cases.

MD: Huge missing piece, algorithm is not well-defined, we don’t know how to event reconstruction based on parameters such as time.

GW: Not suggesting we try to do that, provide the code to allows someone to do that for their lab.

MD: Sounds like the IOC framework.

BD: Sounds like channel finder with gather. Not sure if it still works. Asking Guobao.

GW: With pvManager these all promise to help, but we haven’t combined them.

MD: Don’t think we can do that generically yet.

GW: Not suggesting that we produce a general-purpose tool.

TK: A long time ago I drafted a proposal for this tool, nothing particular happened to it. Will try to find it.

GW: Physicist has specific conditions to apply to filter the data.

GW: We don’t *need* pvDatabaseJave, but it might help.

MK: If you don’t have it, you would have to provide a complete implementation of a channel provider. pvDatabaseJava would provide those for you. ~2 weeks to extract from pvIOCJava.

BD: Worth doing, put it on your list.

Gateway, etc.

SH: When a number of developments are completed we’re going to want a GW very quickly, but it’s not a deal-breaker right now.

BD: What order do we implement ...

MD: Wrapper for pvAccess client that looks like Channel Access client API, to ease porting of e.g. EDM to pvAccess

Multicast Support for pvAccess

BD: Is of interest, is on Matej’s list

End, discussions of dinner…

Tuesday

Present: SH, KK, TK, SV, GG, AJ, JK, MD, RL, MK, DH, BD, DC, JS, CL, MY, BV

Online occasionally: MS

Scribe: SV

Daron Chabot: Databroker and MetaData Store
https://drive.google.com/file/d/0Bwa7KMoZ5dT5dkhyb09PN19FVWM/view?usp=sharing

DC: Other SPEC users?

SH: Only one SNS beam line. Younger users tend to not use it.

BD: EPICS does drivers, python is better language, there are libs for recip. space -> spec less useful

DC: For Data Collection (“Ophyd”) components are modelled after spec, replaced by newer tools (using ipython, etc.)

BD: spec does not do 2d area detectors, ophyd does

DC: output is decoupled, using different handlers for different types of output

DC: code loads into ipython session, we have CLI tools, can do n-dimensional scans

DC: ophyd is greek for serpent

DC: databroker provides single service for data retrieval: access to metadata store, framestore, ca archiver, OLog

KK: how do you fogure out what does user want?

DC: user generates data, scans, etc, and then wants to analyze data; he has tools to export blob of data

SH: is there a mechanism to remove things that are not needed?

DC: every beamline has its own archiver

BD: framestore and datastore have apis to get what you need

DC: mongodb used as backend for metadata store, has collections, we have master document with set of generic keys

DC: framestore used for capturing data, essentially file replica catalog

DC: databroker will be used for fetching data along side metadata

DC: we have number of visualization data clients, live display client is polling metadata store

BD: we have group writing workflow, one of the requirements from DOE is “all data must have provenance”

DC: we are looking into incorporating provenance from mysql into mongo

SH:one approach is to put provenance info into a single file

BD: not arguing with everything in a single file, but simply augmenting this with indexing information

DC: metadata store is about 6-8 months old, everything is rather new

DC: v4 will provide means of serving data to clients; at the moment clients go directly to mongodb using its python apis

DC: software available at github

SH: what pieces exist and are ready to use?

DC: some pieces are missing (diffractometer, etc.)

DC: for scans, customers wanted command line access first

SH: we got people that want UI interface, some want CLI tools

BV: why the interface to all services could not be v4?

BD: each component needs to be its own service

MY: Did you look at anything other than mongodb?

DC: we looked at pytables, did not scale well; cassandra is being looked at

BD: we are going to have to support multiple framestores  

-- Coffee break --

Gabriele Carcassi: DIIRT distribution services: Web PODS - PVA PODS
https://docs.google.com/file/d/0B0WTrWXQk3xoUjh6WDhMcUVFNm8/edit

GC: PODS are protool oriented distribution services

GC: one way to configure pods, and css

GC: pods can  be configured to map incoming namespace to formulas using PVs

GC: webpods allow client side formulas

RL: are you doing formulas read only?

GC: currently they are read only, but I am working on extending that

BD: does that mean I have PV gateway for pv access

GC: yes, it could function in that capacity

BD: pva server does not exist?

GC: correct, should be doable;

RL: this is an application level thing, data gets dissected in the process; you do not get the same data as the original

GC: yup, good for something, bad for others

AJ: is this sufficient fo gateway needs

SH: this is complimentary

AJ: you can use it under some circumstances

SH: the way we are using gateway with v3, this is not the tool for that

KK: implementing pva server could be lot harder

AJ: is this sufficient for GW and SLAC?

ALL: nope

BD: but it is interesting for remote viewing

KK: on slide 4, sounds good to externalize data source config, but the problem is that users at CSS have all config in CSS in one file. with this end users configuration would be distributed?

GC: no, we could have this configured under the product directory

KK: Great! (with german accent, but actually quite pleasantly sounding)

AJ: we are done with GC

RL: go back to your DIRT hole :-)

KK: “CSS Widgets for displaying XY Histogram” & build issues

KK: my interest would be to access fields of the structure, what pv get has on the command line, to be able to put it into channel name

BD: david can’t display anything at the moment?

DH: previous nsls2 build you could connect to PVs on a good day, did not display anything

GC: is this issues that nsls2 build does not work with PVA? MS told me about some issues… the CSS build is now is a weird state, we will fix everything including world hunger…

MS: will try to update today or tomorrow. regarding displaying images, they have to be provided in a java native format… need to update charts and test it

GC: he will work on CSS v3.3, we are working on v4

GC: we are waiting on pvmanager update

Eric Berryman: discussion about maven stuff, ‘SNAPSHOT’ vs. released version, handoff from diirt to integation into CS-Studio

BD: we just want stuff to test in an hour

GC: never say in an hour… it will take a week (or a month to a year)

KK: once you get over the problem to assemble stuff, what about actually debugging source?

GC: yea I need that too if I want to develop

GC: you should be able to to development on DIRT git, no more siliness

GC: we cross all our fingers and hope for the best

GC: what is x-y histgram?

SH: we would like to use CSS to display images, X-Y data, but first we need to be able to build CSS with PVA support

BD: histogram, you mean normal histogram?

KK: XY hist. is image. Time-of-flight, .. would be a line plot

GC: I do have something like this, not sure if it work for you?

BD: you are agrregating stuff in real time

BD: their values are x,y and intensity

GC: if this becomes a table, you have something that you can use

SH: but this is all on the client side

SH: we need CSS to be able to display that, but that would also require a pvAccess gateway for access outside the beamline

BD: MS is mapping NTNDArray into images

GC: first we have to get our stuff working…

GC: we have not tried images due to sheer size of data

BD: With web pods, can you do server side rendering, and JUST serve jpeg…

SV: JUST is a 4 letter word in software development

GC: should be able to dump stream of images thrugh the web socket, not sure what can be used that is supported for all browsers

ALL: silent

DH: if CSS could simply display PVs out of the box…. that’s what I want… do not care about nt image at the moment

ALL: silent again

KK: this image formula was mentioned, not sure how that would work; we have image widget, we have to tell the widget the pv name and configure it; how will the updated thing look like?

DH: presumably the formula will take nt nd array, and will be able to produce image using NT nd array fields

SH: what do you need to do to display PVA waveform?

DH: the pv is NTNDArray, had color and other stuff in it that can be used to display image

SH: is this what pv manager gives us?

DH: does not work today

MS: PVManager vImage is “raw image” in native java format; the formula needs to take NTNDArray and use this to create vimage

KK: the image widget could have vimage formula applied to NTDArray?

MS: right now you need to create formula, which is provided by PVA plugin, and works only with NTNDArray; for v3 channels you need special formula; for stock market, we need to write one too

ALL: silence…

AJ: Are we done?
SH: until next week

KK: how would we access elements of this structure? should we fix time?

AJ: tour is priority…

SH: 1 hour tour

KK: we should start with URI, ways to access fields?

MS: I can make it, later might be better

AJ: 2pm

-- Tour & Lunch --

Scribe: DH

Michael Davidsaver: Branches for base-3.16

MD: Overview of plans for base with focus on those useful to V4, post 3.15

MD: Files parser:Hard to change db syntax without breaking backwards compatibility.

New parser - idea is to parse to intermediate format.

MD: Record and scan processing. Parallel Scan. Want more parallel threading

MD 3 threads per scan

AJ: Only 1

MD: Make my life easier

MD: Have to preserve phase ordering

RL: You want to parallel handle interrupts?

MD: Yes. E.g 8 channel digitiser

KK: All records in 1s scan process in parallel?

MD: Yes

MD: Problem with one queue and multiple worker threads. Accessing single lock potential bottleneck.

MD: Parallel ca_link.

KK: Parallel scan must be tested?

MD: Yes.

MD: New locking done. Needs testing. Anyone? Go to launchpad site and try. New ca link parsing done ready to merge.

Discussion of parsing and grammar, especially VDCT

BD: How long

MD: A week or more to do parallel scanning and several weeks to benchmark

CA Gateway

SNS and Ralph recently communicated gateway crashes because of changing array sizes.

Ralph: No good test setups for gateway. Murali started to work on a test setup based on VMs. He also worked on DBE_PROPERTY support; not in mergeable state.

BD: Are you doing fixes

RL: Urgent fixes I will do, i.e. prevent crashing, but not extending CAS to handle array size changes

AJ: Could CAS changes to support user name handling also include array size changes?

MS: Yes, but… pcas diverged from rsrv..

DB: Would it easier to rewrite the gateway?

RL: PSI reimplementing gateway  for 5 years

BD: Does limiting old versions supported make rewriting gateway feasible

RL: If I had to re-write gateway, would use IOC as shell, rsrv as server, …

RL: Bessy use as  multiplexer. Started changing gateway model. Better when each beamline has own network

ery

MD: Some linux specific stuff that might help.

AJ: Still have some Solaris but they’re restricted in their use.

RL: Everyone using Linux.


KK: URI for Accesssing fields, pvRequest specification and parsing, Server-side filtering via pvRequest, Restrict PV channel name character set, parse names etc.

pvRequest not well-defined

MK: For picking fields it is. For options no.

RL: No requirements

MK: There’s a doc

RL: That’s a document of an implementation

AJ: Server-side filter specs are structures

KK: Clients need to be able to pick subset of structures

MK: Exists, using pvRequest structure

DH: If have

Structure

    Structure A

        Structure B

            int x,

            double y

    int C

can get y as A.B.y

Structure

    Structure A

        Structure B

            double y

MK: Can’t drill down into union/structure arrays and unions

MK: Only options defined are processing, blocking and queue size.

AJ: I have problem with client deadband.

RL: For V3 server specifies deadband. In V4 client could specify deadband but doesn’t know reasonable deadband

AJ: RPC to get deadband, options

KK: No way of introspecting deadband?

KK: Can we agree not to put dots in name.

DH: Curly bracket syntax also exist:

“field(pv.timeStamp{secondsPastEpoch,nanoseconds})”

MK: pvget -r “timeStamp{secondsPastEpoch,nanoseconds}” channelName

Maybe

pvget channel?request

pvget channel#request

MS: Can’t use ? because used for RPC. # ok.

AJ: URI syntax already exist, must be compatible with this.

RL: Can escape as part of URI syntax, so if nor meant as part of URI then escape

GW: Can we minimise use of escaping because some people are typing

KK: pvget neutrons#field(timeStamp{secondsPastEpoch,nanoseconds},timeOfFlight)

 

RL: Can’t use # because OK for pvget (client turns into request) but not for webserver. Don’t want to make clients have to know whether request is going to web server

MS: If you put ? or -s eget does channel RPC

SV: That should change

Discussion over whether could have 2 schemes one for RPC

KK: proposed resolution:

1. only one scheme - pva

2. That means can’t use #

3. Use ?

4. iIf use ?, eget behaviour has to change


Whiteboarding: Result=

pva://host1:123/channel?request=field(...)&type=rpc

field(..) is what’s implemented now in pvget -r .., unless we find something in there that would always have to be escaped in an URI and thus be very inconvenient to type.

With pvget, the built-in default is currently “request=field(value)”, unless that gets nothing, when it will use “field()”, which returns everything.

type=rpc selects RPC type of channel

Order of request vs. type does not matter.

Examples of arguments to pvget (“Query Request”):

  1. “channel” - Up to the client. Current pvget command tries 2, then 3 automatically, assuming the channel is NT and user wanted the value.
  2. “channel?request=field(value)” - pvAccess channel, not RPC, gets ‘value’ field.
  3. “channel?request=field()” - pvAccess channel, not RPC, gets everything.
  4. “pva:///channel?request=field(mystructure.somefield)” - One field
  5. “pva:///channel?request=field(structure.field1, structure{subfield.subfield2.field2})” - Two fields
  6. “pva:///channel?request=field(mystructure{myfield1, otherfield2})” - Two fields
  7. “pva:///channel?request=field(mystructure.somefield{subfield1,subfield2})” - Two fields somewhere down in structure
  8. “pva:///channel?request=field(mystructure.somefield{subfield1,subfield2}), mystructure.otherfield)” - Three fields
  9. “pva:///channel?type=rpc” - Channel for RPC access, will return full structure
  10. “pva:///channel?type=rpc&request=field(structure.field)” - Channel for RPC access, only returning subsection of full structure
  11. “pva:///channel?request=field(structure.field)record[process=true]”-one field, process = true

Present definition is in : http://epics-pvdata.sourceforge.net/informative/pvRequest.html

URI def https://www.ietf.org/rfc/rfc2396.txt sec 3.4:

    ‘Within a query component, the characters ";", "/", "?", ":", "@",
  "&", "=", "+", ",", and "$" are reserved.’

Hm, comma is reserved. The above is in contravention of that.

Reserved, but not prohibited. Are we using ‘,’ in a way that’s compatible with intended use?

Putting field request specifcation in the path part of the URI:

A simple example of a beam position monitor, these are valid examples

pva:///bpm34/x

pva:///bpm32/x,y

In a more complex example, say of a power supply defined by the following structure (taken from the pvRequest doc):

record psEmbeded
   alarm_t alarm
       severity NONE status NONE
       message
   time_t 1969-12-31 19:00:00.000 userTag 0
   structure power
       double value 0.0
   structure current
       double value 0.0
       alarm_t alarm
           severity NONE status NONE
           message
   structure voltage
       double value 0.0
       alarm_t alarm
           severity NONE status NONE
           message

Which might be accessed using the following URI examples (“Path Request”):

  1. pva:///lgps45/current
  2. pva:///lgps45/current/value,alarm
  3. pva:///lgps45/current,voltage/value
    Getting  lgps45/current as well as lgps45/voltage/value
  4. pva:///lgps45/current/value,/voltage/value
    Getting  lgps45/current/value as well as lgps45/voltage/value
  5. pva:///lgps45/current/value,alarm,/voltage/value
    Getting lgps45/current/value, lgps45/current/alarm as well as lgps45/voltage/value

URI Spec:

3.3. Path Component

  The path component contains data, specific to the authority (or the
  scheme if there is no authority component), identifying the resource
  within the scope of that scheme and authority.

     path          = [ abs_path | opaque_part ]

     path_segments = segment *( "/" segment )
     segment       = *pchar *( ";" param )
     param         = *pchar

     pchar         = unreserved | escaped |
                     ":" | "@" | "&" | "=" | "+" | "$" | ","

The path may consist of a sequence of path segments separated by a
  single slash "/" character.  Within a path segment, the characters
  "/", ";", "=", and "?" are reserved.  Each path segment may include a
  sequence of parameters, indicated by the semicolon ";" character.
  The parameters are not significant to the parsing of relative
  references.

In the future, clients may use “#” to do client-side filtering. Text following “#” is not passed to server.

MK: Can not combine “query request” and “path request” in one request except for global options.

pva:///lgps45/current/value,alarm,/voltage/value?request=[process=true] OK

AI on AJ within a month: Formalize the pva URI syntax and semantics, including of the pvRequest - possibly as BNF.

-- End of Tuesday --

Discussion about dinner…


Wednesday 14th

Present: KK, TK, SV, GG, AJ, MD, RL, MK, DH, RL, DC, KV, BV, JK

Scribe: TK

Kay, Marty: Sharing arrays in Java

MK: explains sharing array data.

MK: plans for C++. Use shared pointers without  copying data.

KK: is this not already happening in Java?

MK: at the moment there are several copies in the implementation. In Java one cannot mark arrays as const.

MS: one could still manipulate the data, in C++ freezing is just to prevent accidental manipulations.

MK: structure arrays can be modified (deep in the structure)

MK: present pvArray issues proposal

MK: proposal to implement shallow copy by pvDataCpp

MK: use Java garbage collection as a method to mimic reference counting

KK: useh has then no control over memory handling

MK,MS: problems could arise if garbage collector does not release memory soon enough.

MD: method to pass around arrays is missing?

MS: Java has methods, but performance might be a problem

MD: important is to have guarantees agains unintended data manipulation

MS: problem is that Java does not know ‘const’ness

AJ: can we get to a conclusion?

DH: the issue is that if we want to implement the performance improvements in C++ also in Java

KK: Marty’s proposal would improve performance, but has risks

MD: Our C++ rules are easier to follow than the Java ones

SV: implementing Marty’s proposal would simplify implementation

AJ: would it make sense to use Gabriele’s types?

MS: yes, it would.

AI on MS: Look at using Gabriele’s data types in pvDataJava, make proposal to replace our code in pvDataJava that uses his.

MK: We don’t have any real users on Java, other than pvManager, so we should be able to make changes there, but we have to be a lot more careful with C++ because we could break other people’s code if we make incompatible changes.

KK: on Java side, we try to explore Vtypes. C++ side will be left as it is.

-- Meeting split into 2 rooms --

C++ APIs

Present: SV, GG, AJ, MD, MK, DC, DH, KV, BV, JK

Scribe: AJ

SV: C++ API

See SV’s previous presentation above, “Additional slides”

MK: Isnt this EZPVACPP

DH: I think this should be in the standard API

MS: This is the EZPVA API.

SV: I’ve already done this, someone should look at what I’ve done and implement it centrally. I don’t want this code in pvaPy. Code example shows what I want.

DH: How to monitors work in EZPVAJava?

MK: We didn’t do it yet

SV: I did it for my code, it works.

MD: Why don’t we just use SV’s code?

MK: It’s important to keep compatibility between C++ and Java.

SV: I don’t mind modifying my code if necessary.

AI on MK: Look at SV’s code, use this as the basis for EZPVACPP

KV: Pools of shared vector

KV: (showing code for BasePvaPlugin…)

KV: There is an API in shared_vector for pools

MD: But we could make it more convenient. (looking at shared_vector methods). The API doesn’t do all the memory management for you, but it lets you write your own buffer manager which it uses.

MK: Would it make sense for us to provide a memory pool?

KV: Would like the server to return data to the pool rather than delete it.

MD: Prefer the default version use new/delete, but we could provide one that does that. Issue is related to resizing the data structure.

(discussion of push_back)

MK: You don’t know how big your arrays will be in advance?

KV: No, although there is a maximum.

MK: Why do want to use pools?

BV: If we make a mistake our 32GB buffer fills up and we stall and lose data. About 10,000 events per ROC per pulse.

MD: Could get away with using a fixed maximum size array?

(discussion)

KV: There is no upper limit, but we could post when we exhaust the shared vector.

AJ: Do we need to make changes to our code?

MD: No, but SNS might implement an additional API that we might want to provide.

-- 5 minute break --

MD: Streaming stored data
https://drive.google.com/file/d/0B4JkTqq_L6j8OWxnMjVCSTlVQ1k/view?usp=sharing

(discussion not minuted)

MD: pvAccess cannot currently support multiple data serializations.

MS: If we had flow-control per-channel/monitor stream instead or per-client…


Planning

Present: BD, TK, KK, RL, SH

Scribe: KK

What needs to be done, pending people/time

Support Beam Line Applications

 

Areadetector, stage  1

  1. David’s pvAccess server ‘plugin’,  pvAccess client ‘driver’
  2. CS-Studio widgets to show pvAccess image, line histograms, ..
  3. pvAccess failure modes: What gets dropped when saturating, …
  4. AD attributes: Develop standard set for histograms, to identify image (size, color), profile (position or angle, ..)

Areadetector, 2:

pvData replaces AD’s NDArray; pvAccess used to pass data between AD plugins  if they are on different processes/IOCs

Areadetector, 3:

Handling images required AD plugin, and each plugin often has 100+ records to configure & monitor. Change that into plugin with pvAccess interface to config & monitor. A single ‘table’ widget in CSS can be used to configure the plugin, or custom screen with URIs to access only selected elements.

Service for Meta data store: Common header across sites; Run #, channel names, references to science data, ..

Service for reading archived data: Channel Archiver, SLAC Archive Appliance, ..

Service for Science data store: Images, HDF5 files, ..

Service for Scanning

Service for submitting log book entries (with title)

“Service” can likely be a REST service, not necessarily pvAccess

Support Physics Applications

ChannelFinder

Add better editing. View is always a table. Is there one inherent hierarchy that can ease configuration (as in alarm handler)? No, there many hierarchies, created based on tags as desired. An editor could still present the config in tree, and adding a new entry will right away get the tags of its parents.

Gather

Needs to be resurrected. Collect a bunch of channels, correlate values by time, serve as array that can be monitored.

(archive already mentioned)

Replace Channel Access with pvAccess

Bundle pvData, pvAccess, … pvaSrv in EPICS base.

pvAccess gateway.

Implement libCA API on top of pvAccess, so EDM could be recompiled to magically support V4. (Would have to remove pvAccess’s use of libCA).

CS-Studio

Joined together again

AJ: Agenda items, not much time left

BD: “Training at FRIB”?

DH: Mark Heron asked for V4-specific training at the FRIB meeting

MD: Bring a problem, we’ll show you have to solve it with V4?

KK: Show getting started stuff, etc.

RL: This is demo though, not training.

DH: Who’s going to do that?

BD: Greg…

MK: We can defer that to our regular G+ Hangout meetings

DH: “NT Documentation” -- 1. NT doc is from 2012, really need to publish up-to-date defs. 2. I wasn’t happy with stuff that crept into the standard doc, shouldn’t be included in that doc. Didn’t want to remove it since someone else added it. 3. Doesn’t read well, grammer doesn’t make sense, inconsistencies within the document.

AJ: If nobody responds when you complain, take silence to mean acceptance.

SH: How are these docs versioned?

DH: Mercurial and we publish versioned docs, older versions still available. I created a 20141111 version of the NT doc, MK please read this.

AI on MK, MS and GW: Look at David’s changes and approve or revise.

DH: Also want an updated pvAccess spec, MS still has an outstanding AI to revise this.

MS: Ack.

DH: Beginners doc: can cover in regular hangouts, need Greg’s input.

MK: Release planning for 4.4.1: SNS need to test and respond to MS’s latest fixes. 4.4.1 should only contain those bug fixes.

SH: Don’t need an official release

RL: We need to exercise our release procedures for bugfix releases.

AJ: 4.5?

MK: Weekly Hangout discussion

SH: Intent?

MK: Try by FRIB meeting

RL: What should be in there?

DH: GW says training should include dbGroup…

AJ: We need to discuss release procedures, github git etc in regular meetings.

--- Closed meeting at 12:27 ---