EPICS V4 Developer Meeting at APS

Agenda, Wednesday 18th November 2015

9:00        Develop Agenda for today & tomorrow

After Lunch        

Tomorrow

Minutes

Present: MK, AJ, GW, TK, KK, MP, RL, DH, KV, MD, SV

Online: GS, BD

Chair:

Scribe: KK

NEW TOPIC: dbGroup taxonomy (David)

DH: Update on SACLAY presentation and demo of group operations. “Taxonomy”:  Lists options for structure (values only, full structure with alarm & timestamp for each element, …) that combine the grouped data from multiple channels.

A primary distinction between the types in this taxonomy is whether each field describes a PV, or each field describes an array of N>1 PVs values.

Example: double D, enum E as subfields of a structure, or create group structure that has { double D, alarm, timestamp} and { enum E, alarm, timestamp }. Or rearrange: double D, enum E, and shared alarm, timestamp. Or NTMultiChannel { any[] value, string[] channel_name, int[] severity, alarm, timestamp…} that has both group elements’ alarms and timestamps as well as combined alarm and timestamp.

All have advantages and disadv.

KK: What are the requirements, how to pick one of the possible structures?

AJ: How hard is it to implement several?

??: Whoever implements it decides.

DH: Not hard to implement all of them.

GW: Use case: Set many magnets all at once. Send single value to IOC, which then sets many magnets all at once.

DH: Any structure could do that.

MK: pvaClient (simplified PV access client API) supports “array of doubles” when fetching several channels at once. Assembling data on client side, nothing atomic at this time.

GW: Developer shouldn’t choose the structure

MD: Developer needs to pick one to move on

DH: Demo of simple non-locking ‘get’ and ‘put’. IOC with pvRecords.

Syntax:

pvget “composite{example:X, example:J}”

to fetch as NTComposite.

pvget “scalar_multi{example:X, example:J}”

to fetch as NTMultiChannel.

“composite{..}” resp. “scala…” are magic channel names that on the IOC side are parsed, group data is created and returned. Those channels are not permanently held on the IOC.

GW: Gave example of desired printout from pvget

DH: What structure do you need to get desired printout?

AJ: Printout should not matter. Physics tools may need their own printout, basic pvGet cannot provide that.

RL: Greg’s example is actually composite of composite.

DH: Server cannot format the data

MD: Good example for python client that fetches data and formats it. Don’t add that functionality to basic pvGet command line tool.

AJ: Should composites of composites still be atomic?

DH: Yes.

Discussion:

  1. We need composite(composite(a, b), composite(c, d))
  2. No, just get composite(a, b, c, d) and re-arrange as [ a b ; c d ] on client as desired

→ Decided to stop discussing this further, not making any decision

MK: Beware that “atomic” won’t be supported across IOCs

DH: Example of fetching NTComposite which has all the info for individual elements’ original data

pvput -f “field(example:X.value,example:J.value)” “composite{example:X, example:J}” 3.14 5

Details on layer between local channel provider and database.

AJ: Asyn may work on pva server for asyn that does not go via database, so such channels would not be accessible from ‘group’ that depends on database.

AJ: Does local channel provider have API to support atomic access? Locking?

MD: Yes, with 3.16 base database that’s possible

AJ: But is that accessible via local channel provider in pvAccess?

MK: In reality, wouldn’t physics apps usually require data across IOCs?

RL: This is not about physics, but about accessing “device” which is hierarchy of records on single IOC

AJ: Or Gemini cad/car communication of commands between IOCs, where group would read/write several records within single IOC

KK: What is response to “pvget composite(a, b)” if a, b on different IOCs A, B?

DH: No response, since neither IOC A nor B has both channels

KK: Will this work with gateway? Just a long channel name “composite…”?

MD: Yes, nothing special required

AJ: Add IOC name to channel name? Or other server identification? That way, only one IOC receives the group request. Otherwise, each IOC sees the “composite..” search, needs to parse, then decide it doesn’t know the individual channels.

MD: Or pre-define group channels on server via RPC, config file, ..

KK: Better keep name search, no pre-config. Could set ADDR_LIST of client to single IOC if it becomes necessary to limit the anme search

MD: IOC name is like alias

dbGroup: Requirements & status (Michael)

MD: Multi-lock support will be in EPICS 3.16. dbScanLockMany() is on 3.16 github branch.

MK: How hard to do this for pvRecords?

MD: Tricky part is unrelated to database but about handling multiple locks. So could be done for pvDatabase.

MD: Had looked at a less efficient back-port for earlier EPICS releases which uses one global lock

[Outstanding question: whether the same locking pattern and API worked out by Michael for Base, can be applied to pvDatabase. Integration is probably a question of funding].

GW: Will dbGroup thus require updating IOCs to 3.16?

MD: Yes, should be supported, but would not be atomic or use global lock

AJ: Forget global lock. Simply don’t support ‘atomic’ before 3.16

AJ: Two versions of dbGroup. One for IOC, old records. One for non-IOC pvAccess servers, but may need a new API in the local pvServer API for multi-locking.

MD: Can be seen as overall transactional approach. Prepare transaction, then perform. No rollback, just pass or fail. Still single circuit. Not a distributed lock manager.

KK: Use cased and “single-IOC or not ?” still appear unclear

MD: Initial implementation will effectively settle that

AJ: pvaSrv would be one first implementation

RL: Not currently working on that

[Need funding for dbGroup] Dave can work on dbGroup to some extent, but not probably for another 2 or 3 months.

AI on GW: See who can fund dbGroup.

AJ/RL: The grouping should be on a public or standard API, so that clients can work with groups of pvA channels without knowing internals of the server.

RL: Clients should be able to find out which features a server provides.

AJ: Since the ability to do multi-locking is only necessary on local channels, the dbGroup implementation could try dynamic casting the service to the enhanced class to see if it supports this additional features.

??: Group support might need protocol addition so clients can check if server support it.

KK: Fine, can be done in backward compat. way like “pvinfo” type PVs that have the required info

NEW TOPIC: IOC Database Links talking to pvAccess (Andrew)

AJ Slides: See part of ANJ-Slides.pdf

AJ: Links are get, put, scan(put to remote PROC field). CA or local. PP, MSS etc.

Options:

  1. Device support that _is_ pvAccess client. To be included in pvaSrv. Known data type based on record type. Can be done any time.
  2. New link type. There is no API to do this (but AJ has been doing some work since 2009). JSON to specify link address? Could replace dbCa link implementation.

Many: Option 2 much better in the long run

GW: Option 1 far easier than 2, so clear that we should do 1 first at least, and then do 2 as well.

GW: Goal is not always to replace CA, but just to get data that’s only readable via pvAccess, so another reason to implement 1 now

GW: SLAC would have immediate use for 1, so will look for person to implement

AJ: 2 requires discussion of details and some additional implementation by the core developers

NEW TOPIC: Field name validation https://github.com/epics-base/pvDataCPP/pull/12 (Michael)

MD: Names are not checked at all

MK: Use EPICS base? No ‘.’

MD: Even stricter: No {}. Pull request limits _field_names_ to a-zA-Z0-9 and underscore.

KK: Can this pull request be merged?

AJ: IOC would prohibit field names that _start_ with number. Must be valid as C identifier.

MD: But V4 example code includes that

MK: Examples could be updated

RESOLUTION: pvData field names must conform to C89 rules, plus the restriction that the first character must not be a digit. Note that this syntax rule specifically does not apply to pvAccess channel names; so far no formulation has been made for valid channel names.

Lunch time chat

MD: C++11 includes reg. exp.

MK: Would like to use!

AJ: Not supported by RedHat nor vxWorks

KK: What spec does V4 use?

AJ: No specific version

DH: 2003? Boost or tr1.

AJ: Need to use native boost/tr1 on Linux, but provide one for Windows.

TOPIC: Proposal for publishing release documentation and patches (Andrew)

AJ Slides: See part of ANJ-Slides.pdf

AJ: http://epics-pvdata.sourceforge.net/literature.html makes it hard to find the ‘current’ documentation. Better: One page per release with links to all docs for that release. Includes links to problems and patches.

Web site is a git repo, checked out into source forge web server. For new release, copy/update previous mainPage/docbuild/<version>. Ralph sees way to automate that in Cloudbees build script.

Currently missing entries for Java.

GW: What about latest (‘tip’, ‘master’)

AJ: Existing literature.html would still list that, but other releases have per-release page.

GW: Why separate problems page?

MD: Link to tickets/issue tracker. Tickets have tag related to the release.

AJ: How to get patch file? Ticket/pull request diff may include more than just the patch.

MD: Would prefer frequent releases over list of patches

AJ: .. but lack resources for frequent releases

AI RL: Add Cloudbees script that generates documentation links to the bundle jobs, run this for previous releases

AI AJ: Create a release links file (docbuild/<version>/index.html) for previous versions

TOPIC: Working group http://epics-pvdata.sourceforge.net/home.html

AI GW: Update names in table

TOPIC: License and Copyright files and bad URLs in documentation

AJ: updates & corrections needed for existing modules.

AJ: Copyright requirements differ between sites

GW: Can we add the same copyright file

RL: Use common block in header. On change, add copyright for your side unless already listed, and update year.

Example from current EPICS sources:

* Copyright - See the COPYRIGHT that is included with this distribution.

* This software is published subject to a Software License Agreement found

* in file LICENSE that is included with this distribution.

GW: How many license files should we have, just 1 or one per module?

MK: Must have license file in every module (we’ve been told that before)

AJ: No link to web site which may go away

MK: Why does current EPICS LICENSE end with list of several site-specific copyright files?

git history: Marty started that way

RESOLUTION: We will not have a separate COPYRIGHT file in individual modules, nor in EPICS v4 as a whole. Each module will have only a single LICENSE file, and a standard source header. Our proposed license header text for each source file is the following:

// Copyright information and license terms for this software can be

// found in the file LICENSE that is included with the distribution

The referenced license file will contain both the license, and the appropriate list of copyright holders for the module.

AI on AJ: Write the one license file (using the MIT license in most cases).

AI Every module owner: Update your module to include copy of that one LICENSE. Update your headers to contain the above text.

TOPIC: Reference Documentation:

DH: Network spec not updated to include ‘any’, bounded/unbounded array requests?

AJ: Checked online doc, it has been updated

AJ: Do docs need links to previous version?

GW: Yes, I use that all the time

AJ: pvDataCPP 5.0.2 labeled “Working Draft June 2015”

GW: That’s OK, because still draft

KK: It was released at that time, so drop “draft”?

AJ: Call “5.0.2 release” instead of “working draft”

GW: Docs not always perfectly updated for each release

AJ: Compare EPICS base. R3.15 has link to IOC App Dev Guide 3.15 which may still get updated

WG: Doc describes functionality independent of specific release

SV: No, doc describes features of a specific version

MK: In docs, version number of V4 better than date of doc snapshot

AJ: .. found examples for broken “Previous version” links

AJ: pvaPy doc link only uses “0.5”, not patch level in link

RL: Why not just keep the links to 0.5.x?

RESOLUTION:

Top-level page V4 links to per-version page, like “V4.6.0”.

Per-version page lists all modules: pvCommon, ..

AI Module Owners: On your module pages,

TOPIC: Next EPICS Version 4 meeting planning

Defer plan on whether to have a meeting in February

Will have a v4 f-2-f as part of ESS EPICS Meeting, probably on Mo and Tu of EPICS meeting, which will take place after IPAC. Ok to take ½ or ⅓ of day from f-2-f for v4 training.

TOPIC: Roadmap for testing (Ralph)

RL: Have some unit tests, but lack installation and regression tests. RT like unit tests, but added whenever bug appears to avoid bug from re-appearing.

Split server module (IOC, pvDatabase) from tests. This allows testing server version X against client version Y. One universal test server for many test clients. Unclear how to automate. Need one way to start tests. PyUnit as test framework. pvaPy, pyEpics for clients. Jenkins to execute tests.

Examples vs. tests: Examples are simple, well written, referenced in documentation. Tests might be ‘bad’, use API in a way that’s not meant to serve as an example for others to duplicate. Examples could still be executed as tests to assert that they actually work.

GW: Java?
RL: Could use JUnit, also triggered by Jenkins, for Java, and PyUnit for C++.

TOPIC: Status of mrkraimer version of exampleCPP and exampleJava

MK: Shows documentation for exampleCPP which combines examples for pvData, Database, pvaSrv, pvaClient, NT, Java/C++ interop.

Combining all examples like this avoids circular dependencies between pvDatabaseCPP (used to create demo data) and client module tests.

Each example can also serve as a template for new projects. Just copy TOP directory.

There is also an exampleJava.

RL: Server may better be a separate module. Also separate ‘tests’ from ‘examples’.

RL, MK: [Details of parent.pom for maven build of examples. Bundle would contain parent pom that defines variables for all module versions]

GW/RL: The parent pom would probably go in the bundle module, and that implies that the bundle module must also have branches on which the corresponding pom resides.

RL will implement the testing framework as part of ITER testing framework.

Meeting ends 17:05