{"id":115,"date":"2016-08-10T02:13:35","date_gmt":"2016-08-10T02:13:35","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/holgerkenn\/?p=115"},"modified":"2016-08-10T02:13:35","modified_gmt":"2016-08-10T02:13:35","slug":"troubleshooting-azure-iot-hub-connections-on-embedded-linux","status":"publish","type":"post","link":"https:\/\/www.cubeos.org\/blog\/2016\/08\/troubleshooting-azure-iot-hub-connections-on-embedded-linux\/","title":{"rendered":"Troubleshooting Azure IOT Hub connections on embedded Linux"},"content":{"rendered":"<p>Hi!<\/p>\n<p>I&#8217;m in Japan for a few days, working with local partners to get their devices connections to Azure IoT Hub. And I want to share a few lessons learned.<\/p>\n<p>We always started from the <a href=\"https:\/\/github.com\/Azure\/azure-iot-sdks\/\">Azure IOT Hub SDK on GitHub<\/a>. \u00a0And here&#8217;s the first catch: if you just download the zip file from GitHub, you are missing the links to other dependent projects and your source tree is lacking some files. To avoid running into these problems, please clone the project using git and don&#8217;t forget to add the &#8211;recursive option as described <a href=\"https:\/\/github.com\/Azure\/azure-iot-sdks\/blob\/master\/c\/doc\/devbox_setup.md\">here<\/a>.<\/p>\n<p>git clone &#8211;recursive https:\/\/github.com\/Azure\/azure-iot-sdks.git<\/p>\n<p>In case you get strange compiler errors on the way, such as mismatch of function signatures, it might be that your source tree is out of sync. One way to fix this is to run &#8220;git submodule init&#8221; and &#8220;git submodule update&#8221; in the right directories, but I often just throw away the whole tree and clone it again.<\/p>\n<p>The first thing you should do is to familiarize yourself with the SDK on a normal Linux machine. For this purpose, I just run a Linux VM on Azure. Go through the steps of setting up the development environment and setting up an IoT hub, just for testing. The free tier of the Azure IoT Hub is sufficient at this point. Now create a device ID in your IoT Hub, e.g., by\u00a0using the <a href=\"https:\/\/github.com\/Azure\/azure-iot-sdks\/tree\/master\/tools\/DeviceExplorer\">Azure IoT HubDevice Explorer <\/a>on Windows.\u00a0Under the management tab,\u00a0select your created device and then right-click and select &#8220;Copy connection string&#8221;.<\/p>\n<p>Go to the source code of one of the simple examples, e.g., <a href=\"https:\/\/github.com\/Azure\/azure-iot-sdks\/tree\/master\/c\/iothub_client\/samples\/iothub_client_sample_amqp\">the C amqp sample client<\/a>. Insert your connection string in the source code and compile the sample. Now head back to the device explorer, click on the data tab and start monitoring data from your device. Then run the sample client executable. You should now see a few messages arriving. Now in device explorer, switch to the &#8220;Message to Device&#8221; tab, select your device and enable &#8220;Monitor Feedback endpoint&#8221;. Now type something in the message field and hit send. Your sample client should receive data and the feedback endpoint monitoring should indicate that the messages have been received.<\/p>\n<p>Great, now let&#8217;s move over to the actual device!<\/p>\n<p>Here, there are a couple of things you need to be aware of, the two most important ones are trust and time. Wait? What? Is this some relationship self-help blog? \ud83d\ude42<\/p>\n<p>The trust issue:<\/p>\n<p>Unfortunately, some embedded devices do not come with the right set of certificate authorities installed. When the Azure IOT SDK client code tries to establish a secure connection, it validates the certificate presented by the IOT hub against the known certificate authorities. If there is none, the client code stays quiet for a very long time and then fails with various errors. In order to test for this condition, I often just use the openssl client program and try to establish the connection manually from the device.\u00a0\u00a0Most embedded Linux distributions have the openssl executable installed together with the\u00a0openssl library. An alternative is to run both the sample and\u00a0&#8220;tcpdump -w\u00a0capture.pcap&#8221;\u00a0at the same time\u00a0on the device, then download the pcap\u00a0file and analyze it using <a href=\"http:\/\/www.wireshark.org\/\">wireshark<\/a>.<\/p>\n<p>For example, if I want to see if I can reach the mqtt endpoint of my IOT Hub,\u00a0I run the following command:<\/p>\n<p>openssl s_client -connect &lt;My iothub name&gt;.azure-devices.net:8883<\/p>\n<p>(and of course replace &lt;&gt; with the name of your IOT hub)<\/p>\n<p>If this command fails to establish a valid TLS connection\u00a0with &#8220;Verify return code: 20&#8221;, you have &#8220;trust issues&#8221;. If you see &#8220;Verify return code: 0 (ok)&#8221; then everything is OK. In wireshark, you would see the TLS negotiation fail with &#8220;No CA&#8221;.<\/p>\n<p>To resolve your trust issue, make sure the right CA certificate is present on the device. Microsoft uses the Baltimore CyberTrust CA to sign the server keys, so you should have the file &#8220;Baltimore_CyberTrust_Root.pem&#8221; somewhere in your file system. But even if it is there, the openssl library may not load it. To find out where it expects the files to be, just run &#8220;openssl version -d&#8221;. You should see something like this:<\/p>\n<p>OPENSSLDIR: &#8220;\/usr\/lib\/ssl&#8221;<\/p>\n<p>This means that the OpenSSL library will look for the CA cert in the file \/usr\/lib\/ssl\/cert.pem and then in files in the directory \/usr\/lib\/ssl\/certs\/<\/p>\n<p>But it may be that the file is actually there but OpenSSL still fails to establish a secure connection. Then you might have a time issue.<\/p>\n<p>The time issue:<\/p>\n<p>CA certificates have a time span in which they are valid. For instance, the Baltimore CyberTrust CA\u00a0openssl x509 is valid in the following time span:<\/p>\n<p>Not Before: May 12 18:46:00 2000 GMT<br \/>\nNot After : May 12 23:59:00 2025 GMT<\/p>\n<p>You can easily check for yourself by running this command:<\/p>\n<p>openssl x509 -in \/usr\/lib\/ssl\/certs\/Baltimore_CyberTrust_Root.pem -text<\/p>\n<p>How could this be invalid? Easy: Some embedded devices have no battery-buffered realtime clock and\u00a0initialize their clocks\u00a0with preset dates on boot. And these may be ancient, e.g. Unix Epoch (January 1st, 1970), GPS epoch (January 6th, 1980) or whatever the manufacturer set. So a good practice is to set the clock to the right date before attempting to connect.<\/p>\n<p>But that might not be enough.<\/p>\n<p>The Azure IOT hub also uses a time-based token scheme to authenticate its clients. The process is described <a href=\"https:\/\/azure.microsoft.com\/en-gb\/documentation\/articles\/iot-hub-sas-tokens\/\">here<\/a>. The token includes an expiry time as seconds since Unix Epoch in UTC. The Azure IOT SDK uses the device connection string to create such an shared access signature token. If your clock is off, the token created may already have expired. The tokens are generated with a validity of 3600 seconds, i.e., one hour. If your clock is late by more than that, the IOT hub will reject the connection.<\/p>\n<p>So the best practice is to run ntpclient or ntpd on your embedded device. Even busybox has a simple ntpd implementation, so this should be available on your embedded os. Alternatives are of course to use GPS, a mobile network, a battery-powered RTC or a radio time receiver (FM RDS, long-wave time signals etc.) as a time source. But be aware of the startup and initialization times these time sources take (gps can\u00a0take several minutes to give a proper time information)\u00a0and the skew\u00a0RTCs might accumulate over time.\u00a0And\u00a0RTC batteries might die after a couple of years.\u00a0Also make sure that your time zone is properly set, the SDK will always calculate in UTC times, but if your timezone claims to be UTC but the clock is set to the local time zone, you might be off\u00a0by a couple of hours.<\/p>\n<p>Which brings me back to the CA cert validity. Today, 2025 seems to be far out in the future, but remember that many embedded devices designed today have a lifetime of over 10 years. So that CA cert will expire in the lifetime of these devices. So make sure you have a way to update the CA certificate.<\/p>\n<p>Hope this helps,<\/p>\n<p>H.<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Hi! I&#8217;m in Japan for a few days, working with local partners to get their devices connections to Azure IoT Hub. And I want to share a few lessons learned. We always started from the Azure IOT Hub SDK on &hellip; <a href=\"https:\/\/www.cubeos.org\/blog\/2016\/08\/troubleshooting-azure-iot-hub-connections-on-embedded-linux\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[31,32,34,35],"class_list":["post-115","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-azure","tag-internet-of-things","tag-iot","tag-linux"],"_links":{"self":[{"href":"https:\/\/www.cubeos.org\/blog\/wp-json\/wp\/v2\/posts\/115","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.cubeos.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.cubeos.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.cubeos.org\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.cubeos.org\/blog\/wp-json\/wp\/v2\/comments?post=115"}],"version-history":[{"count":0,"href":"https:\/\/www.cubeos.org\/blog\/wp-json\/wp\/v2\/posts\/115\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.cubeos.org\/blog\/wp-json\/wp\/v2\/media?parent=115"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.cubeos.org\/blog\/wp-json\/wp\/v2\/categories?post=115"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.cubeos.org\/blog\/wp-json\/wp\/v2\/tags?post=115"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}