BLOB upload

Some IoT scenarios require upload of large objects from devices, such as log files, camera images, videos etc. Those files can be sometimes even gigabytes large. To do this securely, devices should prepare and/or execute the uploads with the same device credentials and authentication mechanisms as they used for their connectivity and their telemetry/device operations. The uploaded objects should be then usable within the cloud ecosystem that is used for the IoT Solution.

Bosch IoT Device Management provides a flexible mechanism for file upload between the backend and a device implementing the uploadable or autoUploadable function blocks (e.g. Bosch IoT Edge). The device/upload authentication is guaranteed by Bosch IoT Device Management. File uploads can be configured as a user-initiated action or as a mass management rule which triggers file uploads through a specific time interval. The uploaded files reside on a cloud storage and can be accessed through the backend. Bosch IoT Device Management supports uploads to Azure Blob containers and AWS S3 buckets. The cloud storage is owned and managed by the operator.

Depending on the use case, operators can implement specific logic and fine-grained configurations based on the device needs. When talking about uploads of gigabytes of files, the upload can be paused and resumed, if supported by the device.

File upload flow

The following diagram illustrates the flow of the file upload process as implemented in Bosch IoT Device Management:

images/confluence/download/attachments/1783512217/dm_mme_fileUpload_arch_servReview.png

  1. Once the process is initiated by the device, the device sends the Upload request event.
    0. The process could be also initiated by the backend by executing the trigger action of the uploadable/autoUploadable feature. The trigger action could be also incorporated into a mass management rule.

  2. The event goes through the Device connectivity and Digital twin layers to reach the Device Management layer.

  3. Here, as part of a mass management rule (learn how to create the rule in Upload files), the backend requests temporary credentials or a pre-signed URL, depending on the chosen type of credentials:

    1. Pre-signed URL - HTTP file upload, using backend provided pre-signed URL and authentications headers. This credential type is suitable for cloud-agnostic devices as in this case the device needs to know only the pre-signed URL, which is provided by a generic command.

    2. Temporary Credentials - upload through Azure SDK/AWS SDK, using backend provided Azure/AWS temporary credentials.
      The used type of credentials have to be identical on the device and the backend. Both credential types should include an endpoint, tenantId, clientId, clientSecret and a jsonConfig, but when using temporary credentials you also need to add the bucket where the files should go.

  4. Once Microsoft Azure returns the credentials/URL, Bosch IoT Manager creates an object containing all necessary upload info.

  5. This object is then returned to the device.

  6. The device uploads the file to the Cloud Storage. The cloud storage is owned and managed by the operator.

File upload Features

To be able to upload files to the cloud, a device needs the Uploadable or AutoUploadable features.

Uploadable feature

Find the com.bosch.iot.suite.manager.upload:Uploadable:1.0.0 vorto model as part of the the Eclipse GitHub Vorto repository. The uploadable feature is more suitable for single uploads.

Its status is characterized by the following entries:

  • type - represents the type of the uploaded object. Could be logs, device backup, diagnostics etc.

  • context - represents the context of the uploaded object. It should be unique in the scope of the type. Could be 'application', 'system', etc.

  • info - Represents the instance-specific information.

  • lastUpload - provides the status of the current or last triggered upload.

The upload request event, which requests upload credentials from the backend is characterized by:

  • correlationId - the upload request correlation ID.

  • options - contains additional upload request information. An example is supported storage providers, temporary credentials validity, uploaded object key, etc."

It provides the following operations:

  • Trigger - triggers the upload request that will send an UploadRequest event. Could include correlation ID and options. I f a correlation ID is provided, it must be used in the request event, otherwise the request will generate a correlation ID. Options prefixed with options. should be included (without the prefix) in the sent request event options. Other options are not transferred to the request event.

  • Start - starts the actual upload process with the provided correlation ID and options. Once valid credentials/pre-defined URL have been received by the backend they are sent to the device.

  • Cancel - cancels the upload identified by the correlation ID, only if the cancel operation is supported and the upload is not already finished.

Options

Options prefixed with info. should be included (without the prefix) in the lastUpload status. An example is info.downloadUrl containing the address the uploaded object could be downloaded from.

Generic upload options

Description

https.url

The HTTPS URL for/of the upload.

https.method

The HTTPS upload method, PUT is used by default.

https.header.<header_key>

Prefixed headers to be included in the upload request. An example is https.header.x-my-header-name: my-header-value when the upload request should include a x-my-header-name: my-header-value header.

AWS S3-specific options:

  • aws.region

  • aws.access.key.id

  • aws.secret.access.key

  • aws.session.token

  • aws.s3.bucket

Azure BLOB storage-specific options:

  • azure.storage.endpoint

  • azure.shared.access.signature

  • azure.blob.container

AutoUploadable feature

Find the com.bosch.iot.suite.manager.upload-AutoUploadable-1.0.0 vorto model as part of the the Eclipse GitHub Vorto repository.

The AutoUploadable feature is more suitable for automatic uploads that happen over a specific period of time with no user interaction.

It extends the functionality of the uploadable feature and provides two more actions on top:

  • Activate- initiates an auto upload process. The upload period should be set from - to.

  • Deactivate- deactivates the auto upload process.

Learn how to create a rule which uploads log files via Upload files.