ESP8266: Continuous Delivery Pipeline – Push To Production

The integration of the ESP8266 framework into the Arduino IDE brought ease of use for development. This was further improved when firmwares could be pushed over the local network to the test devices with the direct over-the-air feature. But what if your ESP8266 devices are no longer in your own network? How can you then update all the devices which are running an older version of your firmware? The Arduino/ESP8266 crew also has an answer for that: the ESPhttpUpdate class lets you download the latest firmware from a web server and replace the old one, required that the available flash memory is twice the size of the firmware. That’s great, but how does the firmware get to the web server? And how do I make sure that only changed firmware gets updated? In the following post I will describe a workflow which will let you automagically update your devices with just a few clicks, once you have working setup.

 

 

What is Continuous Delivery?

That’s a good question. But lets start one step earlier: do you know what Continuous Integration is? In my day job I work as a Java developer and technical project lead in projects for big corporate companies in Switzerland. There we use systems that automatically build the code, run tests and create artefacts with every change of the source code in the code repository. This process is called Continuous Integration or CI, since it validates the integration of all code changes continuously. In the last 15 years this has become very much the standard in the Java world. The next step in the work pipeline evolution after CI is called Continuous Delivery (or CD). With a CD pipeline a team tries to treat every build as a potential candidate for release into production if it passes the build, the automatic unit tests, the automatic integration and acceptance tests. The tests must be so good that if an artefact passes all these tests nobody has to hesitate to push the last button and deploy them into production. Scary? Not there yet. It gets even more scary! There also a workflow called Continuous Deployment where the manual push of a button is not even needed anymore. Every build that passes the tests goes into production.

 

Continuous Delivery for the ESP8266

We won’t go as far as Continuous Deployment in this post but I want to show you a nice workflow for Continuous Delivery. The workflow I’m describing here is using freely available tools assuming that you are working on an open source project. You still might use the ideas described here for closed source projects but then the tools are not free anymore. So what are the components in the workflow?

 

I have to admit, it is a relatively complicated workflow. Especially the PHP script doesn’t look too pretty in the picture but a deficiency in the ESPhttpUpdate class made it necessary. So here are the different stages during development:

Stage 1 – Development – The developer on his computer writes the code (e.g. in the platformio IDE) and tests it locally either by uploading it over the serial connection or by the “local” or direct OTA update mechanism. You can by the way combine the two update mechanisms without problem but you might consider disabling it for the production release or your devices will be more vulnerable for attacks. When developer is happy with his work he commits the changes to Github.

Stage 2 – Continuous Integration – Now Travis will start doing his work and build the last commit. If you have written tests Travis execute them and improve your confidence that your code is working as expected.

Stage 3 – Tagging – Once the developer team is confident that the current version is ready for deployment to the production devices the last commit will be tagged as a release.

Stage 4 – Release Build – Tagging the branch caused Travis to start a new build. But this time Travis will create a binary artefact (the firmware) and upload it to Github and add it to the release

Stage 5 – ESP8266 Update Check – During all this time the devices where hungry for a new release. They frequently asked the PHP script if there was a version different from the one they are running on. Every single time the PHP script had checked with the Github API if there was a newer version and every time the answer was NO. But now there is a new release, a new tag and hence  a new firmware to be installed. The ESP’s download the firmware, replace the old one and do a restart. If everything went well they will be running the latest firmware until a newer version is found.

 

How to set up the workflow?

In this post I won’t explain you how to set up a Github account or how to connect Travis with your Github repository. You can find links for that at the end of the post. I chose to use the Platformio IDE but you could also use the Standard Arduino IDE. But for the example project I recommend to use the Platformio IDE.

As a first challenge the firmware has to be able to tell the PHP script with what version it was built. So how does the version get into tagged firmware? There are two build profiles in platformio.ini. One for the normal (local) nodemcu build and one for the travis build. The local one sets a content of the BUILD_TAG macro to “0.0.0”:

build_flags = -DBUILD_TAG=0.0.0

while the travis profile uses the $TRAVIS_TAG environment variable if available to pass the tag into the build or set it to “0.0.0” for a non-release build:

build_flags = !echo ‘-DBUILD_TAG=’${TRAVIS_TAG:-“0.0.0”}

This happens with every commit but only for release builds the $TRAVIS_TAG environment variable is set and only then the firmware will be uploaded to Github. The following line in .travis.yml makes sure that only tagged versions will be deployed back to Github:

deploy:
on:
repo: squix78/esp8266-ci-ota
all_branches: true
condition: $TRAVIS_TAG =~ ^[0-9]+.[0-9]+.[0-9]+$

The condition attribute checks that the tag version matches the regular expression. For a not very experienced C(++) developer I had a hard time to figure out how to pass in a String literal as macro during expansion phase. This code does it by using the # operator:

#define TEXTIFY(A) #A
#define ESCAPEQUOTE(A) TEXTIFY(A)

String buildTag = ESCAPEQUOTE(BUILD_TAG);

Let me know if you have more simple solution…

 

How to create a release?

On your Github repository root page (e.g. https://github.com/squix78/esp8266-ci-ota) you click on Releases > Draft New Release. Now in the Tag Version drop down you enter a version number matching the above condition, e.g. 0.0.15. You can either do this form master version or from one of the last commits. Then publish the Release.

 

The PHP script

It would have been nice if the production ESPs could have contacted the Github API directly. But there are two issues that made be use the intermediate PHP script:

1. The ESPhttpUpdate currently cannot follow redirects. This is important since github hosts the release artefacts on Amazon AWS. But in the API JSON object the address points to github, so the http client has to follow a redirect to download the artefact.
2. Github uses https for its API and will redirect you to it if you are trying plain HTTP. This means that you would have to know the SSL fingerprints of the github API server and the AWS hosting instance since this is required by the ESPs secure client interface. After all the ESPs don’t have a chain of trusted certificates stored somewhere. While the fingerprint of the github API might be stable, the redirection on Amazon AWS might not always use the same certificate.

So what does the script do? It connects to the github API and fetches a JSON object describing the latest release. I’m currently only interested in the tag name and the firmware download URL. The ESPs will send their firmware tag version with the update request and if the latest tag on Github and the one from the request are identical nothing will happen. But if they are different the script will fetch the binary from github (with a hidden redirection to Amazon) and proxy it to the ESP’s requesting it. You can find the php script in the server folder of the Github project.

 

The ESP8266 code

Have a look at the main class of the example directory. One important guide line for this kind of deployment is that your code should be free of secrets (e.g. for Wifi) or environment specific variables. Because the devices should be configurable in a end-user environment I’m using tzapu’s excellent WifiManager library which starts a captive portal behind a Wifi AP if the device cannot connect to an access point. This way the Wifi credentials are not part of the firmware but stored in the flash memory.

Then you need to include many header files for the OTA update to work. My example code doesn’t do anything but keeping itself up-to-date with the latest release version on github by checking every 60 seconds (yes, I’m very impatient;-)) if there is no update. In a real project you might consider doing these checks less frequently or even use MQTT to get a notification once a new version is available. If you cannot wait for the 60 seconds to pass you can also push the “Flash” button on a NodeMCU to trigger the check.

UPDATE 2016/06/05: Ivan from Platformio was so kind to show me some simplifications which are now part of the example project.

Conclusion

This post together with the example project (see resources below) shows you how you can set up a continuous delivery workflow for your ESP8266 devices. There are several steps involved in making this work but afterwards you will have a semi-automatic deployment to all the devices running your firmware. It took me many hours to get the workflow to work and to write this post. If you feel like I deserve a reward for it consider Teleporting a Beer  to me(see box below) or use one of the affiliate links around my blog. Thank you!

 

A word of warning: during my testing the NodeMCU devices sometimes hung after an update. This is a known issue of the NodeMCUs and should only happen after a serial upload and not after OTA. But if you plan to use this workflow or the OTA feature in general for production be sure that you do some testing.

 

Hardware Resources

• ESP8266 IoT StarterKit (~USD 24.50)
• ESP8266 NodeMCU (~USD 6.50)

 

Software Resources

• The example project on Github
• Platformio – Get Started
• How to set up Github
• How to set up Travis