PANIC & fandango

migration to python3 and latest developments

Sergi Rubio Manrique

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ

What's PANIC?

Panic is a python Alarm System for Tango Controls. It follows the IEC 62682:2014 "Management of Alarm Systems for the Process Industries" Standard:

​

• The primary function of an Alarm System will be to notify abnormal process conditions or equipment malfunctions and support the operator response.

​

• The Alarm System is NOT part of the protection nor safety systems, Safety must have separate tools following its own regulation.

​

• The Alarm System is part of Operator Response, thus it's part of the HMI (including the non-graphical part of it).

​

​

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ

What's an Alarm?

T1_ALARM = my/plc/01/Temperature > SETPOINT

​

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ

What's an Alarm?

T1_ALARM = my/plc/01/Temperature > SETPOINT

T2_ALARM = my/plc/02/Temperature.ALARM

TG_ALARM = T1_ALARM OR T2_ALARM

​

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ

What's an Alarm?

T1_ALARM = my/plc/01/Temperature > SETPOINT

T2_ALARM = my/plc/02/Temperature.ALARM

TG_ALARM = T1_ALARM OR T2_ALARM

TS_ALARM = any(t>SETPOINT for t in FIND(*/plc/*/Temperature))

Panic alarms support Tango attributes introspection, regular expressions and any python built-in

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ

PyAlarm Tango Device

Each device evaluates its alarm formulas at its own frequency (>= 10Hz))

​

​

​

​

​

​

​

​

​

​

Alarm formulas are exported as attributes , and its changes are annunciated

So, each PyAlarm is an AlarmSystem on its own!

​

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ

PANIC Architecture

Tango Device

Tango Device

Tango Device

Alarm

Devices

TANGO DB

PANIC

AlarmAPI

Taurus / Alarm GUI

SNAP/

HDB++

Tango

Devices

Tango Device

Tango Device

Tango Device

Logging

Configuration

Annunciators

email, speakers, telegram, commands

Web (IC@MS, Taranta, ...)

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ

PANIC in the Tango Ecosystem

- PANIC is built using the fandango python library, a set of utilities developed at ALBA to build dynamic python device servers and configure them.

​

- PANIC PyAlarm and the Elettra Tango AlarmHandler are complementary tools:

- Both are supported by the PANIC GUI.

- AlarmHandler can use PyAlarm as a notification service.

- C++ AlarmHandler offers speed while python PyAlarm offers flexibility.

​

- S2Innovation offers its own web-based Alarm System Application, IC@MS, build on top of PANIC. Other institutes developed their own PANIC web clients.

​

- Panic is used by ALBA, ESRF, MaxIV, SKA, Solaris and now Soleil! (amongst other institutes).

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ

Alarm

GUI

Create

Modify

Preview

Sort (state,

priority and time)

Configure

Filter

Report

...

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ

Interaction with Taurus Synoptics

github.com/tango-controls/vacca

taurus-scada.org

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ

New Stuff in Panic 9.2.1

• Device Servers (ready in 9.1) and GUI migrated to Py3
• New reports with much more debugging information
• Telegram to replace SMS (mail-like, free, allows embedding images)
• Ldap integration for user authentication, user-specific logging
• Higher integration with ProcessProfiler (DevOps / performance monitor Tango Device server)

​

• Ongoing: new features from MaxIV/Soleil to be integrated in the Py3 code (sorry for the delay!)
• Ongoing: higher integration with pyhdbpp to include graphs/history on reports
• should hdb++ replace snap or should we go for a new snap system?
• Ongoing: Event Based Alarms + High CPU Usage = filtering to prevent flooding
• Testing: PyAlarm disabling/enabling based on system conditions
• Enabled=not mach/ct/alarms-servers/building_sql_error

​

• Next: Multi-tango host (implementation needs refurbish)
• Next: AlarmViews/Groups (to become a single thing)
• Next: too many PyAlarm options!! (some will be fused/renamed)

​

​

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ

Alarms on Events

Middle-layer devices that depend on lower-level devices cannot guarantee an homogeneous read-time per attribute ... thus suffering from many Tango polling exceptions (Timeout/Serialization/OutOfSync/...)

PyAlarm avoids these exceptions by executing periodic reading of target attributes in a background thread. Less efficient approach than using events, but robust.

On Panic 9.1 we tested the evaluation of all alarms triggered asynchronously at each event received, recalculating the alarm value on the fly.

But!, cpu peaked due to the large number of events subscribed. For a beam position alarm (4*10Hz ADC + 3Hz current) we got 50Hz alarm evaluation!!

We replaced on-event evaluation by just event caching plus polling

alarm re-evaluation every 300 ms (fandango.callbacks).

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ

migrating to python 3

why so late? reason was fandango, a utility framework for writing python device servers widely use at Alba by many devices and APIs.

fandango branch in python3 existed for 6 years, but we couldn't fully upgrade to python3 while we had to keep our compatibility ... with python 2.6!

Finally, on 2020 the development for python 2.6 was halted and we fully migrated to python 3 with the help of S2Innovation, that developed a test suite for the most complex modules.

Panic 9.0 had the panic API and PyAlarm device server migrated immediately after, and Panic 9.2.1 is the first release with fully-functional python3 GUI.

​

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ

But ... what is fandango?

Fandango (“functional” programming for Tango) is a Python library for developing functional and multithreaded control applications and scripts, mostly used in the scope of Tango Control System and PANIC Alarm System projects.

​

$ pip3 install fandango

​

Developed at ALBA since 2008 as a python/CLI alternative to Jive/Astor

Expanded with DynamicDS templates, alarm formula evaluators, threaded helpers, MySQL utils and other utilities and submodules.

​

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ

fandango.Astor and fandango.tango

Astor python API:

Most common usages of Astor java application (start/stop/move devices) have been ported to python, to be used as an API or as a CLI (tango_servers)

​

fandango.tango API:

Jive functionality (create devices, test commands, read attributes, setup events, get/set properties, get/set attribute config, retrieve history, manage polling) plus search methods and tango URI formatters; available as python API and CLI

It includes tango2json, json2tango scripts to export/import data from TangoDB

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ

Astor from shell

To get host status by runlevel:

:~$ tango_servers <host> status

​

To get status of all devs of a class:

:~$ tango_servers status hdb*/*

controls05:0

PyAttributeProcessor/machinestatus

archiving/extractor/current: None

fe/vc/all: None

sr/vc/all: None

​

controls05:1

Clock/gc

gc/clock/1: UNKNOWN

EvGen/gc

gc/evgen/1: UNKNOWN

gc/evgen/2: UNKNOWN

gc/evgen/3: UNKNOWN

Skippy/siggen10

lab/ct/siggen10: UNKNOWN

TangoTest/test

sys/tg_test/1: RUNNING

​

controls05:2

Pool/gc

controller/dummycountertimercontroller/ctctrl11: ON

controller/dummyiorcontroller/iorctrl11: ON

controller/dummymotorcontroller/motctrl11: ON

controller/dummyonedcontroller/onedctrl11: ON

hdb++cm-srv/hdbpp

archiving/hdbpp/cm-01: ON

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ

Astor from shell

:~$ tango_servers [host] <start/stop/status/level> [level] <tango_expression>

​

to stop a device:

:~$ tango_servers stop TangoTest/1

​

to start it on a new host on runlevel 3:

:~$ tango_servers 84.89.0.1 start 3 sys/tg_test/1

​

​

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ

fandango.tango examples (from shell)

:~$ fandango add_new_device TangoTest/1 TangoTest sys/tg_test/1

:~$ fandango find_properties "archiving/hdbpp/*" "Db*"

:~$ fandango put_device_property archiving/hdbpp/cm-01 DbName hdbpp

:~$ fandango set_attribute_label sys/tg_test/1/double_scalar "Double Scalar"

:~$ fandango set_attribute_events sys/tg_test/1/double_scalar polling=3000 rel_event=0.01

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ

fandango.tango examples (from shell)

:~$ fandango find_devices <tango_regexp>

:~$ fandango find_attributes <tango_regexp>

:~$ fandango get_attribute_config <attribute>

:~$ fandango check_device <device>

:~$ fandango check_attribute_events <attribute>

:~$ fandango read_attribute <attribute>

:~$ fandango read_attributes <tango_regexp>

:~$ for a in $(fandango find_attributes "*/vacuum/*/pressure"); do

$(fandango read_attribute $a);

done

regular "careless" expression in fandango allow to easily search for any matching device / attribute / property and are used from several applications (tango_browser, panic, vacca)

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ

fandango device servers in python3

All fandango-based device servers (PyAlarm included) share several features:

- cpu and memory usage are exported like attributes for continuous monitoring

- dynamic commands and attributes created and evaluated on-the-fly from pseudo-python code stored in Tango properties and/or python files.

- they access Tango attributes using the fandango.callbacks API (very similar to taurus.core) that hides the event subscription and implements event caching, filtering and/or lazy evaluation (instead of polling).

- to avoid Timeout/Serialization exceptions they can execute evaluation and blocking tasks in background using either python threads or multiprocessing.

- events pushed from background threads/processes are pushed using a common command processEvent() controlled by MaxEventStream property

​

​

PyAttributeProcessor

PyStateComposer

PyPLC

ProcessProfiler

WorkerDS

...

​

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ

fandango device servers in python3

PyAttributeProcessor : Generic dynamic attribute executor, capable of exporting variables from python scripts into attributes.

WorkerDS : allows execution of long-blocking tasks in a background process

SimulatorDS : capable of importing/exporting/simulating existing devices.

CopyCatDS : device that forwards all attributes/commands from another

ProcessProfiler : exports all cpu/mem stats of a controls host as well as the cpu/mem usage of a list of specified processes as independent attributes

PyExtractor, PyAlarm, PyPLC, SQLServer, CSVReader, PyStateComposer, ...

DynamicDS : allows to join ANY PyTANGO CLASS in the same process

​

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ

a Tango Control System

synergies and bridges

HDB++

PyStateComposer

another Tango Control System

CopyCatDS

WorkerDS

PyExtractor

PyAlarm

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ

middle layer devices tips and hints

Test: poll everything at 1 second, read everything every second. If it works, congratulations! You're device seems ready for the wild.

​

But! this is not usually the case, because your best performance will be the one of the worst performant device in your chain.

​

Polling will work if all low level devices provide events and you do NOT execute commands longer than few milliseconds (~ 50% of polling period / N attributes).

​

Actions taking longer will trigger an APITimeout / PollingThreadOutOfSync exception. Don't panic!: Keep hooks & reads thin, move long tasks to background threads using _asynch reads, use DATA_READY to notify clients.

​

Pushing events from threads may trigger "Not able to acquire serialization" exception. Don't panic! Cache them and push from polled read methods.

​

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ

Late Advice regarding python3 migration

- Python3 is cool because of its syntax and library ... but not performance!!

- Do not expect miracles: performance in old hardware is going to be worse.

- Python 3.9 is just as fast as 2.7 on Debian 10,

- but 15% slower than 2.7 on Debian 9

- or 32bit 2.6 in legacy 32 bit cpu's using half of the RAM.

- R/W devices on moderate event rates (>= 50/second) will require tuning. Just upgrading to python 3.11 will not solve everything.

Do polling stress tests, monitor your performance and try asyncio, tango asynch methods and jit compilers like numba since the beginning!

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ

Late Advice regarding python3 migration

On "futurizing", "sixing", "blacking" the code:

- Do not add new features nor code reformatting on first migration, debugging will be much harder.

- Keep python2 compatibility only and only if you really need it; using future/past modules may affect the whole environment. Later cleanup needed!

- Worst thing to migrate are maps, filters, keys, values, etc ... that became generators instead of lists (Nokeys() are weird).

- And expect weird things!! (None is not < 1 anymore!, no keys index, not same APIs, timezones work differently, bytestring vs unicode, etc.)

​

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ

Time for questions!

​

https://tango-controls.org/community/forum/

https://pythonhosted.org/panic

https://gitlab.com/tango-controls/panic

https://pythonhosted.org/fandango

https://gitlab.com/tango-controls/fandango

​

Thanks to S2Innovation, MaxIV, Soleil, SKA,

ESRF, Solaris and all contributors to PANIC!!!

​

​

Sergi Rubio Manrique - ALBA Synchrotron

37th TANGO Community Meeting, SKAO HQ