MySensors development notes 2.0.x

DONE

• Library size reduced ~20% by removing a lot of C++ overhead.
• Full configuration of library behaviours and features directly from sketch code. Makes codebender builds much easier and allows you to make configuration without editing MyConfig.h.
• No more MySensors constructor mess with radio drivers and signing backends. All setup and initialization is handled “behind-the-scene”.
• Calls to process() is handled in the background automatically.
• Structural change: Embedding of drivers and libraries in a structured way instead of using the weird arduino-build-system util-folder which includes everything when building.
• All gateway variants available as examples without any need for re-configuration (E.g. SOFT-SPI automatically enabled for W5100).
• Gateway just another sensor node! So now you can have wired ethernet sensors and wireless ESP8266 sensors without any radio attach if you want.
• AES encryption (RF24)
• Introducing a presentation() function in sketch - This allows controller to re-request presentation and do re-configuring node after startup.
• Optional receive() function in sketch replaces begin(callback).
• sendHeartbeat() - Allows node to send heartbeat and controller to ping nodes.
• Ethernet/ESP8266 gateway supports setting static ip and using dhcp
• Ethernet/ESP8266 gateway allowing communication using UDP.
• Ethernet/ESP8266 gateway can now act as a client which opens a socket to controller at startup. NOTE: This has to be supported by controller.
• ESP8266 stability improvements
• MQTT client gateway on ESP8266 or Arduino+W5100 Ethernet adapter
• Serial transport layer (RS232/RS485) with dePin management.
• Added variable V_CUSTOM - For sending/requesting custom data to/from controller and between nodes. Preferable using S_CUSTOM device.  This variable-type is controller specific so use it with care in the officially provided examples.
• New sensors types and variables: S_INFO, S_GAS, S_GPS, V_TEXT, V_CUSTOM, V_POSITION, V_VAR, V_VA, V_POWER_FACTOR
• Deprecated sensors: S_LIGHT (use S_BINARY)
• Deprecated variables: V_DIMMER (use V_PERCENTAGE), V_HEATER (use V_HVAC_FLOW_STATE), V_LIGHT (use V_STATUS)
• New command I_DISCOVER - retrieve active nodes and topology.
• Detection of mis-wired RFM69 radios.
• MyMessage:getCommand()
• Cleanup Github Repo:

• Moved MYSBootloader to separate Github repository.
• Moved NodeJsController to separate Github repository.

• “Smart sleep” variants of all sleep methods that allows nodes to receive buffered messages/commands from controller that was issued while node was sleeping. When calling the smartSleep() variant, the goes to sleep and sends a heartbeat message after waking up (informing controller that node is awake) and waits MY_SMART_SLEEP_WAIT_DURATION to process any incoming buffered messages from the controller. NOTE: Controller must support this feature.
• Optional before() method in sketch - for initialisations that needs to take place before radio has been setup (using SPI).  
• New conditional parameters to wait(unsigned long ms, uint8_t cmd, uint8_t msgtype) loops until time passed or specified command received.
• New RF24 driver by @tekka. Stripped C++ code, 2-fold speed increase, Radio code reduced about 2kb, Eliminate non-essential register polling/writing, Uses 2 RX pipes
• Possible to re-assign/reallocate node ids by sending I_ID_RESPOSE to a node.
• AVR/SAMD hardware moded to separate GitHub repository and accessible using board manager by adding https://raw.githubusercontent.com/mysensors/ArduinoBoards/master/package_mysensors.org_index.json
• Support SAMD architecture (Arduino Zero, MySensors Gateway Hardware)
• transportInit()-method used to reinitialize radio after full power-down of radio. Useful when using one of the GPIOs to power radio (or via MOSFET).  
• Move OTA FW update code to MyOTAFirmwareUpdate
• Implement I_DEBUG to report various parameters such as CPU frequency, CPU voltage, free mem, routing info and function to do a “factory reset” by clearing the MySensors EEPROM area
• Add library status indication callback. User can install callback to handle status & error information from the library.
• Improved error LED feedback. A 'headless' sensor can report error codes by counting the number of blinks of the error led.

TODO MySensors library 2.0 - All up for grabs!

• Redesign Github repo layout for the MySensors project:

• MySensors project should look like a standard Arduino library.
• Merge RPI code and build instructions into main repo. Automake?
• Move external libraries and some of the more exotic examples to a separate support-GitHub repo. This will help Jenkins CI to give faster feedback on PRs and pushes.

• Reduce size of RF69 driver. Remove c++.
• Add the MySensors project to Arduino IDE library manager.
• ESP8266 - SSL Support
• ESP8266 - Implement MySensors:sleep(). Wake up from timer/gpio. Only two pins like ATMega? Or do we offer a more flexible sleep on ESP?
• Sleep on repeaters should just to wait(). http://forum.mysensors.org/topic/2536/interrupt-with-gw-wait
• RPI gateway should remember routing information (save to disk?)
• V_ALERT - Internet/GSM connected gateway send out V_ALERT messages through Email/SMS. Fallback to controller.
• Message queue on incoming data (using interrupts). WIP by @Yveaux.

Known Issues

• Request ack from (from a gateway with attached sensors) doesn’t seem to work.

• Custom ISR (int0/1) and sleeping:
  http://forum.mysensors.org/topic/2623/sleep-interrupts/8

Supported controller backhaul transport types and allowed features (towards controller)

All gateway variants supports an optional  sub network of 254 nodes. The gateway itself can also act as a node with directly attached sensors.

Supported forward link transport types and allowed features

Hoperf Radio module selection chart

Converting sketch from 1.5 to to 2.0

Add the following to activate NRF-radio support

#define MY_RADIO_NRF24

Remove MySensor constructor

MySensors gw;

Remove MySensors setup() call

gw.setup(....);

If node expects incoming messages, add the following function to handle data:

void receive(const MyMessage &msg) {}

Move present() and sendSketchInfo() from setup() to a new function called presentation().

void presentation() {

     sendSketchInfo(...)        

     present(...);

}

If you’re using static ids , add:

#define MY_NODE_ID xx

If static parent node is used, add:

#define MY_PARENT_NODE_ID xx

If you want node to be a repeater, add

#define MY_REPEATER_FEATURE

If you need to reconfigure your CE/CS Pins (not needed if NRF is connected to default pins)

#define MY_RF24_CE_PIN 9

#define MY_RF24_CS_PIN 10

Remove all “gw.” before MySensors library calls.

If node requests time using requestTime(), remove callback argument define define the following function to handle time response:

void receiveTime(unsigned long ts) {

}

Make sure to include MySensor.h after any configuration defines.

Repository Split Proposal

Split out two new repos and keep git-history from the old Arduino-project

https://help.github.com/articles/splitting-a-subfolder-out-into-a-new-repository/

http://stackoverflow.com/questions/359424/detach-subdirectory-into-separate-git-repository

Arduino (old repo transformed a bit^[a][b][c][d])

        examples

                <all examples here>

        MySensors (submodule pointing to MySensors)   ← NOTE

        keywords.txt  

        README.md

MySensors.h   ← Only includes MySensors/MyConfig.h and core/MySensors.h

library.json

library.properties        

MySensors (new repo^[e])

README.md

MySensors.h      ← The old MySensors.h

        MyConfig.h  

Version.h

MyCapabilities.h

MyEepromAddresses.h

MyMainDefault.cpp

MyMainESP8266.cpp

MyMessage.cpp

MyMessage.h

MySensorCore.cpp

MySensorCore.h

MyPrivateConfig.h.sample

driver

                AES

AltSoftSerial

ATSHA204

…

gateway

MyGatewayTransport.cpp

MyGatewayTransport.h

MyGatewayTransportEthernet.cpp

MyGatewayTransportMQTTClient.cpp

MyGatewayTransportSerial.cpp

hw

MyHw.h

MyHwATMega328.cpp

MyHwATMega328.h

MyHwESP8266.cpp

MyHwESP8266.h

MyHwSAMD.cpp

MyHwSAMD.h

button

MyInclusionMode.cpp

MyInclusionMode.h

led

MyLeds.cpp

MyLeds.h

protocol

MyProtocol.h

MyProtocolMySensors.cpp

signing

MySigning.cpp

MySigning.h

MySigningAtsha204.cpp

MySigningAtsha204Soft.cpp

transport

MyTransport.cpp

MyTransport.h

MyTransportNRF24.cpp

MyTransportRFM69.cpp

MyTransportRS485.cpp

MySensors Hardware AVR (new repo : ArduinoHwAVR)

        README.md

        … <board files>

MySensors Hardware SAMD (new repo: ArduinoHwSAMD)

        README.md

        … <board files>

MySensors Hardware

        README.md

        package_mysensors.org_index.json

        <packages of the above two repositories>

Arduino Support Libraries (new repo)

Adafruit_BMP085

Adafruit_FONA_Library

Adafruit_NeoPixel

BH1750

Bounce2

DHT

DS3232RTC

DallasTemperature

DigitalIO

EmonLib

Encoder

GSM-GPRS-GPS-Shield-GSMSHIELD

HeatpumpIR

IRLib

InterruptChain

LiquidCrystal

LowPower

MAX6675

MsTimer2

MySensors   (Submodule to Arduino)      ← NOTE

NewPing

NewRemoteSwitch

OneWire

Oregon

PN532

PN532_I2C

PWF_AS3935_Lightning_Sensor

PinChangeInt

PubSubClient

RemoteSensor

RemoteSwitch

RunningAverage

SFE_BMP180

SI7021

SPIFlash

SimpleTimer

Time

Timer

TinyGPSPlus

UIPEthernet

UTFT

UTFT_Buttons

UTouch

readme.txt

sha204

Controller Plugin Update Nodes

The new version has some added functionality from a controller plugin developers perspective.

I_HEARTBEAT - New potential message sent from node to say “I’m alive”. You can use this as a ping message by sending a I_HEARTBEAT to a node. Node replies with I_HEARTBEAT_RESPONSE and an incremental integer as payload.

I_PRESENTATION - Send this to a node if controller wants to request presentation messages again after startup. This also triggers a configuration request from node.

I_DISCOVER / I_DISCOVER_RESPONSE

Controller broadcasts I_DISCOVER <page nr> to request all present, i.e. actively listening nodes. Nodes reply with I_DISCOVER_RESPONSE containing the requested page nr. If no or invalid page nr is requested, page 0 will be returned.

Forwarding: Repeater node only forwards broadcasted I_DISCOVER (i.e. destination = 0xFF)

pages, fix assignment 0 - 4, i.e. these pages always exist

PAGE_TYPE_ID, page types (max 16)

MY_HWID, architecture ID (max 16)

I_DISCOVER_RESPONSE - PAYLOAD

2 bytes general header, send in every I_DISCOVER_RESPONSE payload:

payload header

page: fast discovery

page: general information

0xFF = not defined/error

page: architecture information - AVR (platform specific definitions)

0xFF = not defined/error

page: architecture information - SAMD(platform specific definitions)

0xFF = not defined/error

page: architecture information - ESP8266 (platform specific definitions)

0xFF = not defined/error

page: bootloader information

0xFF = not defined/error

page: transport

0xFF = not defined/error

page: periphery

0xFF = not defined/error

page: discover -  idea stage

0xFF = not defined/error

Bootloader writes to EEPROM:

EEPROM_RESET_CAUSE_ADDRESS         1 Byte                AVR: MCUSR

EEPROM_BOOTLOADER_MAJOR                1 Byte

EEPORM_BOOTLOADER_MINOR                1 Byte

0xFF = not defined/error

Extend usage of EEPROM for configuration-sensitive setups

• RF channel with fallback to default
  EEPROM: RF_CHANNEL (nRF24l01 [0-127], RFM69 [freqband (31,43,86,91)]

I_LOG_MESSAGE - extension

1. standardize logging format to simplify parsing
2. re-route debug messages via I_LOG_MESSAGE for (limited) remote debug support

Enhancement FW OTA updates

2 new ST_STREAM types:

ST_FIRMWARE_VALIDATION_REQUEST

⇒ request sha256 digest of newly transmitted FW for validation

ST_FIRMWARE_VALIDATION_RESPONSE

⇒ signed  sha256 digest to validate FW transfer and activate sketch

ST_FIRMWARE_RESPONSE

MQTT - BREAKING CHANGE - The MySensors library now has build in MQTT client gateway but the mqtt-topic has changed a bit from the previous provided MQTT example. The new MQTT topic resembles the serial protocol which mean just numeral values in topic - no string VARs - this saves space and future maintenance.

MY_MQTT_PUBLISH_TOPIC_PREFIX/NODE-ID/SENSOR-ID/CMD-TYPE/ACK-FLAG/SUB-TYPE

Example outgoing topic (with default prefix) :

mygateway1-out/1/2/2/0/1

Example topic for sending data to your sensor network (with default prefix) :

mygateway1-in/1/2/2/0/1

This also means a controller can (and should) answer the id-requests and time requests via the MQTT broker just like it is handled for the serial protocol today. Ack messages from sensors is also supported.

New sensor types

S_INFO -  LCD text device / Simple information device on controller, V_TEXT

S_GAS -  Gas meter, V_FLOW, V_VOLUME

S_GPS -  GPS Sensor, V_POSITION

New variables

V_CUSTOM -  Custom messages used for controller

V_TEXT - S_INFO. Text message to display on LCD or controller device

r/inter node specific commands, preferably using S_CUSTOM device type.

V_POSITION - GPS position and altitude. Payload: latitude;longitude;altitude(m). E.g. "55.722526;13.017972;18"

TODO Community/SIte

• Forum plugin for viewing gerber files (3D)
• Step-by-step procedure to connect the NRF24L01 to the GPIO pins and use the Raspberry as a Serial Gateway.
• Page describing the OTA firmware update options available, bootloaders to use and which controllers that support this.

Future: Helper Classes

TBD: There is a lot of boilerplate code in sensors sketches. Like checking if it is time to send in new sensor data based on time or change. Alexander earlier proposed that we could have a few helper-classes for a few standard scenarios^[f].

The base classes could be something like:

Sender - sends data every X seconds or on change, interrupt driven, binary/analog values.

Receiver - handles incoming data

Clock  - scheduled sending (based on RTC)

We could then provide a register method of these (with a few different implementations), like this:

Sender x = new BinarySender(pin, sendOnChange, throttling);

gw.register(x, <sensorType>, <sensorId>);

Where logic could be handled by process() in the background...

Possible variants..

AnalogSender(pin,....)

GenericSensor(callbackForFetchingDataFromSensor, sendOnChange, minSendInterval)

PWMReceiver(controlPin, transformerForIncomingDataCallback)

BinaryReceiver(controlPin, transformForIncomingDataCallback);

...

…

MySensors specific hardware:

WE have one board on the wild; the SenseBender. We have also now defined (but not publicly announced) the MYSX connector which offer the ability to dock daughterboards to newer MySensors boards (not yet in the wild). The connector defines unique signal names for all its io and it would be convenient to be able to use these names in the sketches and libraries (we could define signal names for all other MySensors specific io as well like radio interfaces).

The Arduino IDE supports a board specific header for the board selected when compiling.

By extending this header for at least our own boards with the MYSX signal definitions (and possibly other signals) the board selection can be used to manage most hw specific deltas.

By including MyConfig.h in the board header (if that is possible) or the opposite (include board header in MyConfig.h) we could provide overridable defaults for all io.

Update: probably we skip MYSX definitions in MyConfig.h alltogether. A sketch compiled for a board that does not natively support MYSX will have to define the mappings itself if it wants to use them (as the library cannot possibly know the mapping on a generic board).

Source code documentation:

Now available (and should be improved) on documentation branch. Jenkins provide visualization of the most current doxygen results and also displays any doxygen related warnings.

Signing test framework

We have test sketches for security related features that at least ensures the library compile with these features enabled in Arduino/libraries/MySensors/tests/Arduino/sketches/

MySensors debug logs, >= 2.0.1

Format:

[!]SYSTEM:[SUB SYSTEM:]MESSAGE

Systems:

Log messages:

Idea stage, MySensors 3.0.0

• Introduce C_TRANSPORT:
  Separate transport-related C_INTERNAL msgs to dedicated msg type, simplifies and centralizes network/transport-specific processing e.g.
  C_TRANSPORT:
          I_FIND_PARENT/_RESPONSE ⇒ T_FIND_PARENT/_RESPONSE
          I_ID_REQUEST/_RESPONSE ⇒ T_ID_REQUEST/_RESPONSE
• Introduce C_FIRMWARE
  Separate msg type for OTA firmware updates
• Time sync: broadcast and repeat