Bosch IoT Device Management

BLOB upload

Some IoT scenarios require upload of large objects from devices, such as log files, camera images, videos etc. Those files can be sometimes even gigabytes large. To do this securely, devices should prepare and/or execute the uploads with the same device credentials and authentication mechanisms as they used for their connectivity and their telemetry/device operations. The uploaded objects should be then usable within the cloud ecosystem that is used for the IoT Solution.

Bosch IoT Device Management provides a flexible mechanism for file upload between the backend and a device implementing the Uploadable or AutoUploadable function blocks (e.g. Bosch IoT Edge). The upload authentication is guaranteed by Bosch IoT Device Management. File uploads can be configured as a user-initiated action or as a mass management rule which triggers file uploads through a specific time interval. The uploaded files reside on a cloud storage and can be accessed through the backend. Bosch IoT Device Management supports uploads to Azure Blob containers and AWS S3 buckets. The cloud storage is owned and managed by the operator.

The BLOB upload functionality enables other convenient use cases such as backup and restore. Devices which have a feature implementing the com.bosch.iot.suite.manager.backup-BackupAndRestore Vorto model can upload backups to the cloud infrastructure and then restore them with ease. Depending on the use case, operators can implement specific logic and fine-grained configurations based on the device needs. When talking about uploads of gigabytes of files, the upload can be paused and resumed, if supported by the device.

Table of contents:

File upload flow

The following diagram illustrates the flow of the file upload process as implemented in Bosch IoT Device Management:

images/confluence/download/attachments/1981507117/dm_mme_fileUpload_arch_servReview.png

  1. Once the process is initiated by the device, the device sends the Upload request event.
    0. The process could be also initiated by the backend by executing the trigger action of the Uploadable/AutoUploadable feature. The trigger action could be also incorporated into a mass management rule.

  2. The event goes through the Device connectivity and Digital twin layers to reach the Device Management layer.

  3. Here, as part of a mass management rule (learn how to create the rule in Upload files), the backend requests temporary credentials or a pre-signed URL, depending on the chosen type of credentials:

    1. Pre-signed URL - HTTP file upload, using backend provided pre-signed URL and authentications headers. This credential type is suitable for cloud-agnostic devices as in this case the device needs to know only the pre-signed URL, which is provided by a generic command. On the other hand the name of the file must be provided via the UploadRequestEvent as a key option.

    2. Temporary Credentials - upload using backend provided Azure/AWS temporary credentials.
      The used type of credentials have to be identical on the device and the backend. Both credential types should include an endpoint, tenantId, clientId, clientSecret and a jsonConfig, but when using temporary credentials you also need to add the bucket where the files should go. You may add as many files as you wish, as long as you have the necessary permissions for the upload location.

  4. Once Microsoft Azure returns the credentials/URL, Bosch IoT Manager creates an object containing all necessary upload info.

  5. This object is then returned to the device.

  6. The device uploads the file to the Cloud Storage. The cloud storage is owned and managed by the operator.

Upload request event

As demonstrated in the flow chart above, the Upload request event plays a substantial role when performing BLOB uploads. Not only it initiates the upload process but passes important configuration information to the backend and cloud infrastructure.

Event structure

Follow the event structure definition, part of the Vorto model:

events {
request {
mandatory correlationId as string "Upload request correlation ID."
optional options as dictionary[string, string] "Contains upload request additional information. An example is supported storage providers, temporary credentials validity, uploaded object key, etc."
}
}

Where to send the event?

Towards which topic to send the event depends on the way how your device is connected.

  • If your device is connected through Bosch IoT Hub, you have to publish an event to Bosch IoT Hub. For example, if you use MQTT transport - see Publishing events

  • If you want to send a message to the thing directly, you have to send a message with direction FROM a feature - see Messages

Request options

The predefined options keys used to provide additional information to the backend with the upload request event are:

Option

Description

validity.sec

Validity of the temporary credentials in seconds.

blob.type

An Azure-specific option. Values can be BlockBlob, PageBlob or AppendBlob.

key

The name of the uploaded file. Can be used in a pattern to where the file to be stored and is needed when a pre-signed URL is used.

storage.providers

Comma separated list of storage providers needed for this upload ordered by preference. Values can be generic, aws and/or azure:

  • To use pre-signed URL credentials set the value to generic. This is also the default credentials type.

  • To use temporary credentials set the value to aws or azure, depending on your preferred upload infrastructure.

    You must set storage.providers values when the Groovy script contains two credential configs and thus handlers.
    See Combination of both credentials.

key with prefix metadata.

When using pre-signed URL credentials metadata can be assigned to the uploaded file with this specific event option.

key with prefix tag.

When using pre-signed URL credentials tags can be assigned to the uploaded file with this specific event option.

See the structure of an example UploadRequestEvent, which requests temporary credentials from AWS:

"correlationId": "363645645264",
"options": {
"validity.sec": "1800",
"storage.providers": "aws",
"metadata.description": "15_oct_2021_data",
"tag.groups": "daily_data"
}

File upload features

To be able to upload files to the cloud, a device needs the Uploadable or AutoUploadable features.

Uploadable feature

Find the com.bosch.iot.suite.manager.upload:Uploadable:1.0.0 vorto model as part of the the Eclipse GitHub Vorto repository. The Uploadable feature is more suitable for single uploads.

Its status is characterized by the following entries:

  • type - represents the type of the uploaded object. Could be logs, device backup, diagnostics etc.

  • context - represents the context of the uploaded object. It should be unique in the scope of the type. Could be 'application', 'system', etc.

  • info - Represents the instance-specific information.

  • lastUpload - provides the status of the current or last triggered upload.

The upload request event, which requests upload credentials from the backend is characterized by:

  • correlationId - the upload request correlation ID.

  • options - contains additional upload request information. An example is supported storage providers, temporary credentials validity, uploaded object key, etc."

It provides the following operations:

  • Trigger - triggers the upload request that will send an UploadRequest event. Could include correlation ID and options. If a correlation ID is provided, it must be used in the request event, otherwise the request will generate a correlation ID. Options prefixed with options. should be included (without the prefix) in the sent request event options. Other options are not transferred to the request event.

  • Start - starts the actual upload process with the provided correlation ID and options. Once valid credentials/pre-defined URL have been received by the backend they are sent to the device.

  • Cancel - cancels the upload identified by the correlation ID, only if the cancel operation is supported and the upload is not already finished.

Options

Options prefixed with info. should be included (without the prefix) in the lastUpload status. An example is info.downloadUrl containing the address the uploaded object could be downloaded from.

Generic upload options

Description

https.url

The HTTPS URL for/of the upload.

https.method

The HTTPS upload method, PUT is used by default.

https.header.<header_key>

Prefixed headers to be included in the upload request. An example is https.header.x-my-header-name: my-header-value when the upload request should include a x-my-header-name: my-header-value header.

AWS S3-specific options:

  • aws.region

  • aws.access.key.id

  • aws.secret.access.key

  • aws.session.token

  • aws.s3.bucket

Azure BLOB storage-specific options:

  • azure.storage.endpoint

  • azure.shared.access.signature

  • azure.blob.container

AutoUploadable feature

Find the com.bosch.iot.suite.manager.upload-AutoUploadable-1.0.0 vorto model as part of the the Eclipse GitHub Vorto repository.

The AutoUploadable feature is more suitable for automatic uploads that happen over a specific period of time with no user interaction.

It extends the functionality of the Uploadable feature and provides two more actions on top:

  • Activate- initiates an auto upload process. The upload period should be set from - to.

  • Deactivate- deactivates the auto upload process.


Learn how to create a rule which uploads log files via Upload files.