BLOB upload
Some IoT scenarios require upload of large objects from devices, such as log files, camera images, videos etc. Those files can be sometimes even gigabytes large. To do this securely, devices should prepare and/or execute the uploads with the same device credentials and authentication mechanisms as they used for their connectivity and their telemetry/device operations. The uploaded objects should be then usable within the cloud ecosystem that is used for the IoT Solution.
Bosch IoT Device Management provides a flexible mechanism for file upload between the backend and a device implementing the Uploadable or AutoUploadable function blocks (e.g. Bosch IoT Edge). The upload authentication is guaranteed by Bosch IoT Device Management. File uploads can be configured as a user-initiated action or as a mass management rule which triggers file uploads through a specific time interval. The uploaded files reside on a cloud storage and can be accessed through the backend. Bosch IoT Device Management supports uploads to Azure Blob containers and AWS S3 buckets. The cloud storage is owned and managed by the operator.
The BLOB upload functionality enables other convenient use cases such as backup and restore. Devices which have a feature implementing the com.bosch.iot.suite.manager.backup-BackupAndRestore Vorto model can upload backups to the cloud infrastructure and then restore them with ease. Depending on the use case, operators can implement specific logic and fine-grained configurations based on the device needs. When talking about uploads of gigabytes of files, the upload can be paused and resumed, if supported by the device.
File upload flow
The following diagram illustrates the flow of the file upload process as implemented in Bosch IoT Device Management:
Once the process is initiated by the device, the device sends the Upload request event.
0. The process could be also initiated by the backend by executing the trigger action of the Uploadable/AutoUploadable feature. The trigger action could be also incorporated into a mass management rule.The event goes through the Device connectivity and Digital twin layers to reach the Device Management layer.
Here, as part of a mass management rule (learn how to create the rule in Upload files), the backend requests temporary credentials or a pre-signed URL, depending on the chosen type of credentials:
Pre-signed URL - HTTP file upload, using backend provided pre-signed URL and authentications headers. This credential type is suitable for cloud-agnostic devices as in this case the device needs to know only the pre-signed URL, which is provided by a generic command. On the other hand the name of the file must be provided via the UploadRequestEvent as a key option.
Temporary Credentials - upload using backend provided Azure/AWS temporary credentials.
The used type of credentials have to be identical on the device and the backend. Both credential types should include an endpoint, tenantId, clientId, clientSecret and a jsonConfig, but when using temporary credentials you also need to add the bucket where the files should go. You may add as many files as you wish, as long as you have the necessary permissions for the upload location.
Once Microsoft Azure returns the credentials/URL, Bosch IoT Manager creates an object containing all necessary upload info.
This object is then returned to the device.
The device uploads the file to the Cloud Storage. The cloud storage is owned and managed by the operator.
Upload request event
As demonstrated in the flow chart above, the Upload request event plays a substantial role when performing BLOB uploads. Not only it initiates the upload process but passes important configuration information to the backend and cloud infrastructure.
Event structure
Follow the event structure definition, part of the Vorto model:
events { request { mandatory correlationId as string "Upload request correlation ID." optional options as dictionary[string, string] "Contains upload request additional information. An example is supported storage providers, temporary credentials validity, uploaded object key, etc." }}Where to send the event?
Towards which topic to send the event depends on the way how your device is connected.
If your device is connected through Bosch IoT Hub, you have to publish an event to Bosch IoT Hub. For example, if you use MQTT transport, see the following pages:
Send event messages
If you want to send a message to the thing directly, you have to send a message with direction FROM a feature - see Bosch IoT Things documentation.
Request options
The predefined options keys used to provide additional information to the backend with the upload request event are:
|
Option |
Description |
|
validity.sec |
Validity of the temporary credentials in seconds. |
|
blob.type |
An Azure-specific option. Values can be BlockBlob, PageBlob or AppendBlob. |
|
key |
The name of the uploaded file. Can be used in a pattern to where the file to be stored and is needed when a pre-signed URL is used. |
|
storage.providers |
Comma separated list of storage providers needed for this upload ordered by preference. Values can be generic, aws and/or azure:
|
|
key with prefix metadata. |
When using pre-signed URL credentials metadata can be assigned to the uploaded file with this specific event option. |
|
key with prefix tag. |
When using pre-signed URL credentials tags can be assigned to the uploaded file with this specific event option. |
See the structure of an example UploadRequestEvent, which requests temporary credentials from AWS:
"correlationId": "363645645264","options": { "validity.sec": "1800", "storage.providers": "aws", "metadata.description": "15_oct_2021_data", "tag.groups": "daily_data"}File upload features
To be able to upload files to the cloud, a device needs the Uploadable or AutoUploadable features.
Uploadable feature
Find the com.bosch.iot.suite.manager.upload:Uploadable:1.0.0 vorto model as part of the the Eclipse GitHub Vorto repository. The Uploadable feature is more suitable for single uploads.
Its status is characterized by the following entries:
type - represents the type of the uploaded object. Could be logs, device backup, diagnostics etc.
context - represents the context of the uploaded object. It should be unique in the scope of the type. Could be 'application', 'system', etc.
info - Represents the instance-specific information.
lastUpload - provides the status of the current or last triggered upload.
The upload request event, which requests upload credentials from the backend is characterized by:
correlationId - the upload request correlation ID.
options - contains additional upload request information. An example is supported storage providers, temporary credentials validity, uploaded object key, etc."
It provides the following operations:
Trigger - triggers the upload request that will send an UploadRequest event. Could include correlation ID and options. If a correlation ID is provided, it must be used in the request event, otherwise the request will generate a correlation ID. Options prefixed with options. should be included (without the prefix) in the sent request event options. Other options are not transferred to the request event.
Start - starts the actual upload process with the provided correlation ID and options. Once valid credentials/pre-defined URL have been received by the backend they are sent to the device.
Cancel - cancels the upload identified by the correlation ID, only if the cancel operation is supported and the upload is not already finished.
Options
Options prefixed with info. should be included (without the prefix) in the lastUpload status. An example is info.downloadUrl containing the address the uploaded object could be downloaded from.
|
Generic upload options |
Description |
|
https.url |
The HTTPS URL for/of the upload. |
|
https.method |
The HTTPS upload method, PUT is used by default. |
|
https.header.<header_key> |
Prefixed headers to be included in the upload request. An example is https.header.x-my-header-name: my-header-value when the upload request should include a x-my-header-name: my-header-value header. |
AWS S3-specific options:
aws.region
aws.access.key.id
aws.secret.access.key
aws.session.token
aws.s3.bucket
Azure BLOB storage-specific options:
azure.storage.endpoint
azure.shared.access.signature
azure.blob.container
AutoUploadable feature
Find the com.bosch.iot.suite.manager.upload-AutoUploadable-1.0.0 vorto model as part of the the Eclipse GitHub Vorto repository.
The AutoUploadable feature is more suitable for automatic uploads that happen over a specific period of time with no user interaction.
It extends the functionality of the Uploadable feature and provides two more actions on top:
Activate- initiates an auto upload process. The upload period should be set from - to.
Deactivate- deactivates the auto upload process.
Other file upload-related vorto models:
Learn how to create a rule which uploads log files via Upload files.