Hot questions for Using Azure in iot

Question:

For an IoT project, we need to send messages from mobile application (iOS and Android) to Azure IoT Hub. We were able to do it connecting as a device using MQTT library. However, the device simulator too connect as a device and since both uses same device ID, the existing connection is dropped when both connect to IoT Hub at the same time.

We realized we need to connect mobile apps as a service (not as a device) and for that some research shows we need to use following library.

https://github.com/Azure/azure-iot-sdk-java

https://github.com/Azure/azure-iot-sdk-java/blob/master/service/iot-service-samples/service-client-sample/src/main/java/samples/com/microsoft/azure/sdk/iot/ServiceClientSample.java

However, this java sample uses lot of java specific implementations and which are not supported by Android.

Has anyone faced the same problem?

  1. Which library can we use to achieve above from Android mobile application?
  2. Which library can we use to achieve above from iOS mobile application?

Please let me know if any alternate solution is available.


Answer:

You can use a REST API for sending a D2C message to the Azure IoT Hub. The following screen snippet shows an example of this call. Note, that no library is required for this request.

  • another solution for your scenario is to use an Azure Function as a pre-processor to the Azure IoT Hub, see the following example:

Thanks Roman

Question:

I'm Trying to make a POST Request to Azure IoT Hub with a JAVA android application, using Volley Library, but I'm getting this error: BasicNetwork.performRequest: Unexpected response code 400 for https://elca-iot.azure-devices.net/devices/elca-main-device/messages/events?api-version=2016-02-03)

To access the IoT Hub I need to use a SAS Key that I need to include in the header of the request. The android application Code is below:

RequestQueue queue = Volley.newRequestQueue(c);
    String url = "https://elca-iot.azure-devices.net/devices/" + deviceId + "/messages/events?api-version=2016-02-03)";
    // Request a string response from the provided URL.
    StringRequest stringRequest = new StringRequest(Request.Method.POST,
            url, new Response.Listener<String>() {
        @Override
        public void onResponse(String response) {
            onPostResponse(response, c, true);
        }
    }, new Response.ErrorListener() {
        @Override
        public void onErrorResponse(VolleyError error) {
            Log.d("Error", error.getMessage());
            onPostResponse(error.getMessage(), c, false);
        }
    }) {
        @Override
        public Map<String, String> getHeaders() throws AuthFailureError {
            Map<String, String> params = new HashMap<String, String>();
            //params.put("Content-Type", "application/xml");
            params.put("Authorization", "[My SAS Key]");
            params.put("Cache-Control", "no-cache");
            return params;
        }
        /*
        @Override
        public Map<String, String> getParams() throws AuthFailureError {
            Map<String, String> params = new HashMap<String, String>();
            params.put("api-version","2016-02-03");
            return params;
        }*/
        @Override
        public String getBodyContentType() {
            return "application/text; charset=UTF-8";
        }
        @Override
        public byte[] getBody(){
            return "On".getBytes();
        }
    };
    try {
        Log.d("request", String.valueOf(stringRequest.getMethod()==Request.Method.POST));
    } catch (Exception authFailureError) {
        authFailureError.printStackTrace();
    }
    // Add the request to the RequestQueue.
    queue.add(stringRequest);

I've tried to do the same request using POSTMAN and it worked and I don't know why it isn't working with the Android Application. Here's the http code of the request made with POSTMAN:

    POST /devices/elca-main-device/messages/events?api-version=2016-02-03 HTTP/1.1
Host: elca-iot.azure-devices.net
Authorization: [My SAS Key]
Content-Type: application/x-www-form-urlencoded
Cache-Control: no-cache
Postman-Token: d0aa354a-f79c-e574-893a-c259232aa5cb

on

EDIT:

Screenshot of POSTMAN


Answer:

It seems the problem lies in the value in the map. [] could not be escaped.

params.put("Authorization", "[My SAS Key]");

What I did was, while creating the params, I encoded key and value with UTF-8.

params.put("Authorization", URLEncoder.encode("[My SAS Key]", "UTF-8"));

I also struggled with this problem for 3 days. It may seem like a work around but it worked for me.

Question:

I am trying Transparent gateway use case with Downstream devices. I am following documentation how-to-create-transparent-gateway provided by microsoft. I have installed azure IoT Edge Runtime on Ubuntu 18.04 virtual machine. IoT edge runtime is running perfectly with one custom java module.

I have created an IoT Device on Azure portal with Symmetric key. I have added my IoT Edge Device as parent to this IoT Device. I am using java sample send-event to send message to IoT Edge device.

I have copied IoT Device connection string from azure portal and modified it as per the documentation Retrieve and modify connection string. My connection string for IoT Device looks like HostName=myGatewayDevice;DeviceId=myDownstreamDevice;SharedAccessKey=xxxyyyzzz

in above connection string myGatewayDevice is the hostname of ubuntu virtual machine hosting IoT Edge runtime. When I run this example I got the below exception

Starting...
Beginning setup.
Successfully read input parameters.
Using communication protocol MQTT.
Exception in thread "main" java.lang.IllegalArgumentException: Provided hostname did not include a valid IoT Hub name as its prefix. An IoT Hub hostname has the following format: [iotHubName].[valid URI chars]
    at com.microsoft.azure.sdk.iot.device.IotHubConnectionString.parseHubName(IotHubConnectionString.java:321)
    at com.microsoft.azure.sdk.iot.device.IotHubConnectionString.validateTerms(IotHubConnectionString.java:287)
    at com.microsoft.azure.sdk.iot.device.IotHubConnectionString.<init>(IotHubConnectionString.java:121)
    at com.microsoft.azure.sdk.iot.device.DeviceClient.<init>(DeviceClient.java:176)
    at samples.com.microsoft.azure.sdk.iot.SendEvent.main(SendEvent.java:171)

When I modify IoT Device connection string like this

HostName=myiothub.azure-devices.net;DeviceId=myDownstreamDevice;SharedAccessKey=xxxyyyzzz;GatewayHostName=myGatewayDevice I got below error

Exception encountered while sending MQTT CONNECT packet
MqttException (0) - javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:38)
    at org.eclipse.paho.client.mqttv3.internal.ClientComms$ConnectBG.run(ClientComms.java:736)
    at java.lang.Thread.run(Thread.java:748)

Kindly help me to fix this. I want make the communication happen with IoT Edge device. any help will be highly appreciated.

Edit 1

Here is the output of sudo openssl s_client -connect RajUbuntuVM:8883 -CAfile Production/EdgeCerts/RootCACertificate/azure-iot-test-only.root.ca.cert.pem -showcerts

      0040 - 95 c4 40 5b f9 a8 0d 3c-62 83 50 05 ea 1f b9 0a   ..@[...<b.P.....
    0050 - 25 e6 99 8a 27 47 4d 55-25 3d 30 aa 00 94 ea 6a   %...'GMU%=0....j
    0060 - 89 ad 18 60 8f 6b f6 4d-66 6d 05 29 87 6e b0 38   ...`.k.Mfm.).n.8
    0070 - a7 01 38 6f 6e 11 c1 db-62 20 43 de 0d 8d ba 29   ..8on...b C....)
    0080 - ca 91 78 ff a7 5a 49 1a-d6 ed ae 1d ac 65 73 b8   ..x..ZI......es.
    0090 - e1 08 9e 41 63 59 37 ad-88 f9 bd 29 06 8e ca 14   ...AcY7....)....

    Start Time: 1581571257
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---

Both of my devices IoT Device and IoT Edge Device are running on same Ubuntu virtual machine. I have installed root ca in Ubuntu device as below

sudo cp <path>/azure-iot-test-only.root.ca.cert.pem /usr/local/share/ca-certificates/azure-iot-test-only.root.ca.cert.pem.crt
sudo update-ca-certificates

Resolution:

Thanks Silent for pointing me to the right direction. I need to add the azure-iot-test-only.root.ca.cert.pem.crt path in my java code. for Devices with symmetric key attestation they need to present root ca certificate to validate the gateway device. this certificate is not needed for direct communication with Azure IoT Hub for Symmetric devices.


Answer:

Your IoT Edge uses (by default) a self-signed certificate for its incoming connections. You need to make your downstream device to trust that certificate. See here for details on how to the this up:

https://docs.microsoft.com/en-us/azure/iot-edge/how-to-connect-downstream-device

Question:

I want to create a device in the IoT Hub when I add a device in my application.

I'm looking for REST Endpoints available for performing operation. Or any SDK available to do this.


Answer:

Have a look at the following docs:

Service - Create Or Update Device

Control access to IoT Hub

example of the creating DeviceX with auto-generated symmetric keys:

PUT https://myIoTHubName.azure-devices.net/devices/DeviceX?api-version=2018-06-30

headers:

Authorization: sasToken(see the above link)

payload:

{
  "deviceId": "DeviceX"
}

Question:


Answer:

IoT Hubs is the cloud service for building scalable IoT solutions - you can use it as a some-kind-of-very-scalable queue for messages that are coming from your devices or your gateway. Someone should be on the other end of the queue and process the messages. It can be Stream Analytics service (which is the service that implements the very difficult task - polling the queue and processing the messages and passing them further).

I would highly recommend you to go through the test solution Microsoft developed - it is called connect-the-dots :-) You may find it here, and it goes through all of the components you may want for implementing the solution - from different devices and queues to the analytics and visualization.

The reference architecture from Microsoft can be found in Azure IoT Suite site - it has the remote monitoring solution. You just need to register the Azure account, and then click to deploy. The end-to-end remote monitoring solution will be deployed, and you will be able to review the architecture and understand the flows.

For sending your messages through the gateway, you can use one of two (or both somehow) ways:

1) To pass the messages through that gateway - just send it to your gateway, then your gateway can check if everything is OK with the message and send it to the IoT Hub (queue) for processing. You can not do that in the IoT Hub itself, so you can guarantee that incoming messages will have the appropriate format, etc.

2) To send the message to the gateway, then gateway will accumulate some amount of messages into one "big" message and send it to the queue to avoid the situation when you send a lot of identical messages. It should be done with caution as, for the obvious reason, the information will be "average", but not exact the same. But you can accumulate the messages into something like the archive, then put it to the Azure Storage and send the link to that file to the IoT Hub. Then the processing worker will take the message, download the file and start the processing.

Question:

We have started a PoC to connect some of our existing code the Azure IoT Hub via MQTT to test Azure’s support for standard protocols and tools. We are using the Paho client but are getting a CONNACK with a return code of 5 – Not Authorized.

We followed the instructions on how to setup an IoT Hub and created one using the F1 (free) scale tier. We then followed another Azure document and downloaded Device Explorer, created a device and generated a SAS token. We then plugged everything into Paho:

public static void main( String[] args ) {
  String deviceId = "device-fcbd127a";
  String sasToken = "SharedAccessSignature sr=CoyoteIoT.azure-devices.net%2fdevices%2fdevice-fcbd127a&sig=3acRHQXXXXXXXXXXX‌​Zg%3d&se=1468067737";
  String brokerUri = "ssl://CoyoteIoT.azure-devices.net:8883";
  String clientId = deviceId;
  System.out.println( "Connecting to " + brokerUri +" as "+clientId);

  MqttAsyncClient client = null;
  try {
    client = new MqttAsyncClient( brokerUri, clientId );
    if ( client != null ) {
      MqttConnectOptions options = new MqttConnectOptions();
      client.setCallback( new AzureCallback() );
      options.setUserName( "CoyoteIoT.azure-devices.net/device-fcbd127a"     );
      options.setPassword( sasToken.toCharArray() );
      IMqttToken token = client.connect( options );
      token.waitForCompletion( 5000 );
      if ( client.isConnected() ) {
        System.out.println( "Success!" );
      } else {
        System.out.println( "Could not connect to Azure IoT hub, timed-out" );
      }
    }
  } catch ( MqttException e ) {
    client.getDebug().dumpBaseDebug();
    e.printStackTrace();
  } finally {
    if ( client != null ) {
      try {
        client.disconnect();
      } catch ( MqttException ignore ) {}
    }
  }
}

We have confirmed with Wireshark that a SSL connection is made to Azure and that the CONNECT packet is sent. We then see the CONNACK with a return code of 5 being sent to Paho and Azure dropping the connection soon thereafter. We then looked into "Shared Access Policies" and tried different settings. There is nothing in the audit logs and we have "verbose" turned on for everything.

Has anyone connected Paho (or other third-party Java client) to the Azure IoT Hub?

Where do we find any diagnostic information so we can troubleshoot this ourselves?

On a side note, we shelved this (MQTT) approach and tried to connect via the ReST services and receive an even more ambiguous "500-Internal Server Error" as a response. This makes us think there is a more fundamental access issue here. Does the F1 scale hub only support the Microsoft SDK? Are there some hidden access control settings we are missing? Is the format of the names strict, not allowing certain characters or case?


Answer:

Apparently we had issues with the Device Explorer utility. Instead of generating a SAS Token with an expiry of 365 days, it generated a token good for only 365 seconds.

Note the se=1468067737 in the SAS token, it evaluates to an expiration of Jul 09 08:35:37 EDT 2016, well past our test execution.

Question:

I receive the Data:

public void accept(PartitionReceiver receiver)
{
    System.out.println("** Created receiver on partition " + partitionId);
    try {
        while (true) {
            Iterable<EventData> receivedEvents = receiver.receive(10).get();
            int batchSize = 0;
            if (receivedEvents != null)
            {
                for(EventData receivedEvent: receivedEvents)
                {                                    
                    System.out.println(String.format("| Time: %s", receivedEvent.getSystemProperties().getEnqueuedTime()));
                    System.out.println(String.format("| Device ID: %s", receivedEvent.getProperties().get("iothub-connection-device-id")));
                    System.out.println(String.format("| Message Payload: %s", new String(receivedEvent.getBody(), Charset.defaultCharset())));
                    batchSize++;
                }
            }
        }
    } catch (Exception e)
    {
        System.out.println("Failed to receive messages: " + e.getMessage());
    }
}

Here i become the product name and price:

System.out.println(String.format("| Message Payload: %s", new String(receivedEvent.getBody(), Charset.defaultCharset())));

How can i take the Payload, product into a String product; and the price into double price;?


Answer:

As @Aravind said, you can define a POJO class to package the data as object properties like Payload, and serialize & deserialize the data as event body between a POJO and a json string using some json library, such as jackson, fastjson, or choose a favorite one from http://www.json.org/.

Question:

I am able to create a resource group in azure using java libraries but not getting how to create an IoTHub resource in that group.

I have tried using genericResources but it's throwing an exception of missing Sku information. Unfortunately there is no method to set SKU info in the genericResources creation.

Error:com.microsoft.azure.CloudException: Sku information is missing.


Answer:

Currently, Azure Management library for java does not cover all the services in Azure portal. Unfortunately, we cannot use it to manage IOT hub now.

I did some test, and found 2 optional workarounds:

  1. Use Azure REST API to create an IOT hub resource

  2. Use Azure Java SDK to deploy an IOT hub resource with template:

Template:

{
    "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "name": {
            "type": "string"
        },
        "location": {
            "type": "string"
        },
        "sku_name": {
            "type": "string"
        },
        "sku_units": {
            "type": "string"
        },
        "d2c_partitions": {
            "type": "string"
        },
        "features": {
            "type": "string"
        }
    },
    "resources": [
        {
            "apiVersion": "2019-07-01-preview",
            "type": "Microsoft.Devices/IotHubs",
            "name": "[parameters('name')]",
            "location": "[parameters('location')]",
            "properties": {
                "eventHubEndpoints": {
                    "events": {
                        "retentionTimeInDays": 1,
                        "partitionCount": "[parameters('d2c_partitions')]"
                    }
                },
                "features": "[parameters('features')]"
            },
            "sku": {
                "name": "[parameters('sku_name')]",
                "capacity": "[parameters('sku_units')]"
            }
        }
    ]
}

Java code:

import com.microsoft.azure.management.Azure;
import com.microsoft.azure.management.resources.DeploymentMode;
import com.microsoft.azure.management.resources.fluentcore.arm.Region;
import org.apache.commons.io.IOUtils;
import org.json.JSONObject;

public static void DeployTest(Azure azure) {
    try(InputStream templatein = new BufferedInputStream(new FileInputStream( "D:\\iottemplate\\template.json"));
        StringWriter templateWriter = new StringWriter();
    ){
        // Read the template.json file
        IOUtils.copy(templatein, templateWriter);

        // Convert template to JSON object
        JSONObject templateNode = new JSONObject(templateWriter.toString());

        // Add default value for parameters
        JSONObject parameterValue = templateNode.optJSONObject("parameters");
        parameterValue.optJSONObject("sku_name").put("defaultValue","B1");
        parameterValue.optJSONObject("sku_units").put("defaultValue","1");
        parameterValue.optJSONObject("d2c_partitions").put("defaultValue","4");
        parameterValue.optJSONObject("location").put("defaultValue","southeastasia");
        parameterValue.optJSONObject("features").put("defaultValue","None");
        parameterValue.optJSONObject("name").put("defaultValue","jackiottest567");

        // Deploy
        azure.deployments().define("CreateIOTHub")
                .withNewResourceGroup("JackIotTest1", Region.ASIA_SOUTHEAST)
                .withTemplate(templateNode.toString())
                .withParameters("{}")
                .withMode(DeploymentMode.INCREMENTAL)
                .create();

    } catch (FileNotFoundException e) {
        e.printStackTrace();
    } catch (IOException e) {
        e.printStackTrace();
    }
} 

Question:

I have quetion about proper configuration certificates in Auzre's IoT Hub. I want to use Device Provisioning Service with group enrollment. My goal is to reach following scenario: use CA root cert to generate some intermediate cert, then use this intermediate cert to sign device's cert. So, after reading documentation what I understand is following steps:

  1. Upload CA root certificate:

  1. Upload intermediate certificate in group enrollment configuration:

  1. Use device's cert in Java code to provision.

And this only works when I pass my intermediate cert in device's code (regardless of I choose CA or intermediate certificate in step 2):

private static final Collection<String> INTERMEDIATE_CERTS = new LinkedList<>();
...
SecurityProvider securityProviderX509 = new SecurityProviderX509Cert(privateCert, privateKey, INTERMEDIATE_CERTS);
provisioningDeviceClient = ProvisioningDeviceClient.create(globalEndpoint, idScope, PROVISIONING_DEVICE_CLIENT_TRANSPORT_PROTOCOL, securityProviderX509);

This code is from azure's github samples. What is the purpose of using intermediate certificate in device's code? Shouldn't it be validated based on chain of intermediate and root CA certs uploaded to Azure in previous steps?


Answer:

When you upload an intermediate certificate, the ONLY thing that is uploaded is the cert itself, not the entire chain from root to intermediate. The only thing that intermediate cert has is information about the cert that signed it, nothing more. There can be multiple intermediates in a chain, and we cannot make any assumptions about the intermediate chain when evaluating the device’s cert chain.

The device needs to present its entire certificate chain for DPS to provision the device successfully. Without a full chain, it is impossible to determine whether or not a verified certificate is in the device’s chain of trust.

We recommend using an intermediate cert to sign the device because the security best-practice is for roots to be kept offline.

Question:

I'm using Azure Java SDK to learn to develop a Java IoT Edge module. I'm following this tutorial https://docs.microsoft.com/en-us/azure/iot-edge/tutorial-java-module.

So they explain how to send Message and receive Message, with the callback, like here :

// Send message:
client.sendEventAsync(msg, eventCallback, msg, App.OUTPUT_NAME);

// Receive message:
private static MessageCallbackMqtt msgCallback = new MessageCallbackMqtt();

client.setMessageCallback(App.INPUT_NAME, msgCallback, client);

protected static class MessageCallbackMqtt implements MessageCallback {
    private int counter = 0;

    @Override
    public IotHubMessageResult execute(Message msg, Object context) {
            System.out.println(String.format("Received message %d: %s", this.counter, new String(msg.getBytes(), Message.DEFAULT_IOTHUB_MESSAGE_CHARSET))); 
    }
}

So as you can see, the sent and received messages are of type Message.

How can I send Integer for example ? I see I can convert it into String with

String msgString = new String(msg.getBytes(), Message.DEFAULT_IOTHUB_MESSAGE_CHARSET);

But what about Integer ?

Also, in the sendEventAsync method what is the third parameter msg ? In the documentation they say it's Object callbackContext but I don't understand what it is and why we use the msg as this parameter.

Thank you for your answer


Answer:

Sending an Integer as binary message is quite easy:

Message msg = new Message(ByteBuffer.allocate(4).putInt(1695609641).array());

The third parameter in the sendEventAsync method is the context object for the callback method in the second parameter with the following method signature

void execute(IotHubStatusCode responseStatus, Object callbackContext);

So passing in the message in the third parameter gives you access to the message in the callback method

Question:

I'm trying to use the Java Azure IoT device client sdk (v1.2.28) to upload files to blob storage. The examples show how to do this with the .net sdk but it seems like the same methods in the DeviceClient class don't exist in the java sdk. Is it not possible yet with the java sdk?

https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-csharp-csharp-file-upload


Answer:

I reviewed the javadocs of Azure IoT device SDK, as you said, there is not a method to upload file to blob storage.

Accoring to the document File uploads with IoT Hub and the related REST API Create File Upload Sas Uri, the feature is required the blob sas uri, so only using IoT device SDK for Java is not possible, you need to use Azure Storage Java SDK to generate a blob sas uri first to call the REST API to do it.

However, for the other feature to receive file upload notification, there are some classes of Azure IoT service SDK for Java to support, which include FileUploadNotification, FileUploadNotificationReceiver, etc.

Question:

i want use azure Device Identities REST API to create devices identity :https://msdn.microsoft.com/en-us/library/azure/mt548489.aspx but i don't know the syntaxe to set in header request ? what is the key and how to generate value ? here is the param :

  • Set the Authorization header to a SAS token created as specified in the service section of Using IoT Hub security tokens.
  • The Etag header is returned in all requests scoped to a single device identity, as per RFC7232 .

Thanks


Answer:

As @DominicBetts said, you can refer to the referenced document to generate the SAS token by yourself.

As reference, there is not an existing sample code for Java, but I think you can try to refer to the offical sample for Python to knwo how to use the device identities REST API, please see the sample at https://azure.microsoft.com/en-us/documentation/samples/iot-hub-python-get-started/ and download the sample zip file to see the script service/deviceManager.py.

Meanwhile, you can also try to directly read the source code of IoTHub SDK for Java. For generating the SAS token, please see the code at https://github.com/Azure/azure-iot-sdks/blob/master/java/service/iothub-service-sdk/src/main/java/com/microsoft/azure/iot/service/auth/IotHubServiceSasToken.java.

Question:

i am trying to trigger a java function each time my IoT Hub receives a batch of 64 messages (or whatever, the number is not important). I followed this guide to create the basic code, then i edited creating this function

public class Function {

@FunctionName("ProcessIotMessages")
public void processIotMessages(
        @EventHubTrigger(name = "message",
                eventHubName = "samples-workitems",
                connection = "HUB-1544-DEV_events_IOTHUB") List<String> messages,
        final ExecutionContext context) {...Function Logic...}
}

The connection parameter is the IoT Hub connection string in formatted as event hub compatible endpoint (e.g. Endpoint=sb://iothub-hostname-blablabla).

I package and deploy this code with maven plugins specified in the guide linked above. The deploys works fine, i can see mi function up and running with no errors from the portal, the HUB-1544-DEV_events_IOTHUB setting app is correctly created with the correct connection string.

The only strange thing i notice in the portal is in the trigger blade. As you can see, cardinality is One, while it should be set to many, since i did not specify the cardinality parameter in the function. The default one is many according to this guide. This makes me think that i not being able to pass the correct trigger syntax.

Anyway, the problem is that this function is not starting either from my local machine or the portal. Any suggestions? Thx


Answer:

As @evilSnobu posted in the comments, the problem was the event hub name. Just go to Portal -> your IoT Hub -> Built-in endpoints and find all the information to configure the trigger in there.

Question:

Can I configure Azure IoT hub in such a way I can receive messages from TTN (The Things Network)?


Answer:

Unfortunately, Azure IoTHub doesn't support translating messages from TTN to itself as message broker like @AlexBelotserkovskiy said.

However, you can try to create a program using Azure Services as continous running job for reading and sending messages from TTN to IoTHub, such as Run Background tasks with WebJobs.