Description

This guideline describes how to create, generate and deploy Bonseyes AI Apps given a pre-trained AI model. This guideline assumes you already have the done the following:

  • Setup the local environment as explained in Prerequisites

  • Trained an AI model and exported such model in ONNX format. For more information about training and exporting models, refer to the AI Asset guideline.

  • Have followed AI App user guide description and got familiar with LPDNN.

  • Make sure to install system package hdf5-tools and python h5py.

Create LPDNN’s file tree

To generate and deploy AI Apps, a file tree with certain format needs to be created:

challenge/
catalog/
deployment/
deployment_packages/

where

  • Challenge: Folder containing the specifications of the AI Challenge that the AI App aims to solve.

  • Catalog: Folder containing pre-trained AI models.

  • Deployment: Folder containing the AI App’s deployment characteristics.

  • Deployment packages: Folder containing packages that contain tools, executables, libraries and backends that allows the actual execution of the LPDNN AI application.

A template with this organization is available in the AI Asset Container Generator.

An example with LPDNN tree file is also available in the 3D Face Landmarks AI Asset.

In the following sections, each of this folders is detailed and described.

Challenge

To create an AI Challenge for you AI App, please follow the dev guide for AI Challenge’s Definition and creation.

Note: the following steps can be skipped for the moment:

  • Defining the challenge training and sample data (Deprecated),

  • Defining the evaluation procedure (Deprecated)

  • Upload Challenge.

After having followed the creation of the AI Challenge, the LPDNN’s file tree challenge/ directory must include the following files:

  • challenge.yml

  • classes.txt (if required by challenge interface)

  • landmarks.txt (if required by challenge interface)

Catalog

The catalog is a folder that contains AI models:

catalog/
    ${modelName-version-size-precision}/
    ${modelName-version-size-precision}/
    ${modelName-version-size-precision}/

where ${ModelNameVersionSizePrecision} is the name of the model, with the specific version, input tile size and precision.

LPDNN expects AI models in a certain format and organization:

${modelName-version-size-precision}/
    algorith.yml
    model.yml
    blobs/
        ${modelName-version-size-precision}.onnx

where:

  • algorith.yml: YML file that describes the AI model’s type of algorithm or class, e.g., image classification, the challenge that it tries to solve and the pre-, inference, and post-processing details.

  • model.yml: YML file that describes the format of the AI model, e.g., ONNX, and the path to the model.

  • blobs: Folder that holds the trained AI model(s).

  • ${modelName-version-size-precision}.onnx: Pre-trained model file(s) following the model naming explained above.

model.yml

The model.yml file is a simple YAML file that provides metadata required to consume an AI model in one of the supported model file formats. It has the following structure:

type: com_bonseyes/network_formats#caffe_bvlc_0_1
parameters:
  deploy_model: "blobs/deploy.prototxt"
  weights: "blobs/model.caffemodel"

The type parameter specifies the network format used by the model. The following models are currently supported:

  • Caffe SSD (id: com_bonseyes/network_formats#caffe_ssd_0_1)

  • Caffe Bonseyes (id: com_bonseyes/network_formats#caffe_bonseyes_0_1)

  • NVIDIA-Caffe (id: com_bonseyes/network_formats#caffe_nvidia_0_1)

  • BVLC Caffe (id: com_bonseyes/network_formats#caffe_bvlc_0_1)

  • ONNX (id: com_bonseyes/network_formats#onnx_0_1)

The parameters section specifies the model’s file path under the blobs’ folder. For the caffe based models the following parameters are required:

  • deploy_model: path to the caffe prototxt relative to the blobs’ directory containing the model YAML file.

  • weights: path to the caffe weights relative to the blobs’ directory containing the model YAML file.

For the ONNX models the following parameters are required:

  • deploy_model: path to the ONNX file relative to the blobs’ directory containing the model YAML file.

A template of the model YML file is available in the AI Asset Container Generator.

An example model YML file is also available in the 3D Face Landmarks AI Asset.

algorithm.yml

The algorithm.yml is the highest-level configuration YAML file that specifies the components and the algorithms that are executed inside an AI App to solve a given AI challenge. An algorithm defines the relationship between the component’s input and output. Thus, the algorithm.yml file defines all the component’s parameters that are required to execute such algorithms. The schema of the file is described below:

Algorithm Configuration

All information that is necessary to uniquely define the functional behaviour of an algorithm

type

object

properties

  • metadata

Metadata for AI Artifacts

  • challenge

ID of the challenge that specifies this model

type

string

  • algorithm

name of the algorithm

type

string

  • parameters

Parameters specific to the algorithm type, the schema depends on the type of algorithm chosen

type

object

  • subcomponents

Algorithm configuration for the algorithm subcomponents

type

object

additionalProperties

type

object

properties

  • algorithm

Name of the algorithm used to implement the subcomponent

  • parameters

Parameters of the algorithm

  • subcomponents

Algorithm configuration for the subcomponents of the subcomponent

#/properties/subcomponents

  • ai_app_testing

Data used to test that an AI app implementing this specification is behaving correctly

type

object

properties

  • input

Path to a file with the AI app inputs. The format depends on the AI app the algorithm implements and its configuration

type

string

  • output

Path to a file with the AI app outputs that are expected when the inputs are injected in the AI app. The format depends on the AI app the algorithm implements and its configuration

type

string

additionalProperties

False

  • algorithm_testing

Additional data used to test the algorithm component such as intermediate outputs. The schema depends on the algorithm type

type

object

additionalProperties

False

A minimal algorithm configuration looks as follows:

algorithm: com_bonseyes/ai_algorithms#image_classification_preprocess_infer
challenge: com_bonseyes/challenges/mychallenge#challenge.yml
parameters: ...
subcomponents:
  preprocessing:
    parameters: ...
  inference:
    parameters:
      model: model.yml
      ...

A template of the algorithm YML file is available in the AI Asset Container Generator.

The algorithm field is used to select the AI App’s algorithm type. The following AI App’s algorithms are currently supported in LPDNN:

  • Image Classification using a Neutral Network with image pre-processing (id: com_bonseyes/ai_algorithms#image_classification_preprocess_infer): this algorithm converts an image to a specific format and size, normalizes the pixel values and then executes a neutral network to compute the confidence of each possible image class. This algorithm solves AI challenges with interface com_bonseyes/interfaces#image_classification.

  • Face Recognition using a Neutral Network with image pre-processing (id: com_bonseyes/ai_algorithms#face_recognition_preprocess_infer): this algorithm converts a face image to a specific format and size, normalizes the pixel values and then executes a neutral network to compute the fingerprint for the face. This algorithm solves AI challenges with interface com_bonseyes/interfaces#face_recognition.

  • Object detection using a Neutral Network with image pre-processing (id: com_bonseyes/ai_algorithms#object_detection_preprocess_infer): this algorithm converts an image to a specific format and size, normalizes the pixel values and then executes a neutral network to compute bounding boxes, landmarks and confidence of the detected objects classes. This algorithm solves AI challenges with interface com_bonseyes/interfaces#object_detection.

  • Image segmentation using a Neutral Network with image pre-processing (id: com_bonseyes/ai_algorithms#image_segmentation_preprocess_infer): this algorithm converts an image to a specific format and size, normalizes the pixel values and then executes a neutral network to compute a segmentation mask and confidence of the detected objects classes. This algorithm solves AI challenges with interface com_bonseyes/interfaces#image_segmentation.

  • Audio Classification using a Neutral Network with audio pre-processing (id: com_bonseyes/ai_algorithms#audio_classification_preprocess_infer): this algorithm converts an audio snippet to a specific format and then executes a neutral network to compute the confidence of each possible audio snippet class. This algorithm solves AI challenges with interface com_bonseyes/interfaces#audio_classification.

  • Signal Classification using a Neutral Network with signal pre-processing (id: com_bonseyes/ai_algorithms#signal_classification_preprocess_infer): this algorithm converts a signal stream to a specific format and then executes a neutral network to compute the confidence of each possible signal class. This algorithm solves AI challenges with interface com_bonseyes/interfaces#signal_classification.

The challenge field points to the AI challenge YAML file that describes the challenge being answered by the algorithm configuration. The file may refer the AI Challenge YAML file with a relative name if it is contained in the same package or use the syntax package_name#path/to/challenge.yml to refer to a challenge in another package.

The parameters field contains the parameters that define the behaviour of the algorithm. The parameters depend on the algorithm.

The subcompenents field provides parameters and specifications for the subcomponents of the algorithm, e.g., pre-processing, inference.

Subcomponents

In this sub-section we describe the parameters and details of the subcomponents. LPDNN’s AI Apps have two main components:

  • Pre-processing: Section that specifies how the input must be prepared.

  • Inference: Section that specifies which neutral network model should be executed.

Pre-processing

The pre-processing is the first subcomponent of an LPDNN’s AI App. The pre-processor takes the raw input and transform it into the format and specifications that the neural network expects. The pre-processor can execute several functions such as cropping, normalization, filtering, domain transformations, etc. Each function needs to be specified in the subcomponents.inference.parameters field.

Image Pre-processor

The image pre-processor has the following format:

metadata:
  title: Image preprocessor
parameters: parameters.yml
interface: com_bonseyes/interfaces#image_preprocessing

The pre-processor supports the following configuration parameters in the algorithm.yml:

Parameters for image pre-processing

Parmameters used to perfom image pre-processing

type

object

properties

  • output_size

Size of the input to the model

type

object

properties

  • x

type

integer

  • y

type

integer

  • color_mode

Image format of the input to the model

type

string

enum

grayscale, bgr, rgb, rgbs

  • filter

Type of normalization filter used

type

string

enum

none, histogram_equalization_gray, histogram_equalization_rgb, tan_triggs

  • normalization

Type of mean used for normalization

type

string

enum

constant, per-pixel, none

  • mean_file

File containing the per-pixel mean normalization data

type

string

  • std_file

File containing the per-pixel std-dev normalization data

type

string

  • dump_tile

filename where to dump normalized tile

type

string

  • mean_data

Array with per-channel mean normalization constant, the order of channels is given by the colormode

type

array

items

type

number

  • std_data

Array with per-channel std-dev normalization constant, the order of channels is given by the colormode

type

array

items

type

number

  • crop

Cropping options

type

object

properties

  • mode

Select: - fill: stretch the ROI content to fit inside the output tile [default] - fit: resize ROI content (while keeping its original aspect ratio) so that its longest side matches the corresponding output tile size - zoom: resize ROI content (while keeping its original aspect ratio) so that its shortest side matches the corresponding output tile size - center: set ROI to be the largest inscribed centered rectangle

type

string

enum

fill, fit, zoom, center

  • margin

Add a margin to the ROI as a proportion of the output width, resp. height

type

number

  • dump_tile

filename where to dump cropped tile

type

string

  • alignment

Parmameters used align the image based on landmarks

type

object

properties

  • mode

Transform the image to match the input landmarks to the given landmarks positions. Default is no alignment.

type

string

enum

none, affine2, affine3, affine5_rigid, affine5_umeyama

  • dump_tile

filename where to dump aligned tile

type

string

  • landmarks

Landmarks target positions

type

object

properties

  • type

landmark type id

type

string

  • points

Landmark points. Landmarks coordinates are given on a scale of 1000000x1000000 which is rescaled relative to the dimension of the output tile.

type

array

items

type

object

properties

  • x

type

integer

  • y

type

integer

additionalProperties

False

The pre-processor assumes a single image as an input, e.g., rgb, grayscale. The output_size field should be specified for ONNX models, which assumes a NCHW layout, e.g., [1, 3, 120, 120] for rgb image.

An minimal image pre-processing configuration look like following:

color_mode: grayscale
normalization: constant
mean_data: [ 0 ]
std_data: [ 255 ]
output_size:
  y: 120
  x: 120

The first parameter color_mode describes the type of output image that should be generated by the pre-processing, it can be grayscale for grayscale image, bgr for BGR 8-bit images and rgb for RGB 8-bit images.

The normalization parameter specifies the type of normalization that should be applied to image. The options are the following: constant where a normalization constant is used to normalize each channel and per-pixel, where each point of the image is normalized with different parameters.

In the case of constant normalization the constants for mean and standard deviation are specified in the mean_data and std_data fields. For instance to normalize an RGB image with mean 127 on each channel and standard deviation 127 the following configuration would be used:

normalization: constant
mean_data: [ 127, 127, 127 ]
std_data: [ 127, 127, 127 ]

The pre-processor works with raw image data on [0, 255] range. Therefore, if your AI model has been trained and normalized on [0, 1] range, the values to set for the mean and std in the algorithm.yml need to be those used during training multiplied by 255, so as to bring back the data on [0, 1] range.

The per-pixel normalization would look like this the configuration file would include the following:

normalization: per-pixel
mean_file: blobs/mean.bin
std_file: blobs/std.bin

The mean_file and std_file point to binary files containing the per-pixel data.

Audio Pre-processor

The audio pre-processor computes MFCC features from the audio sample. It has the following format:

metadata:
  title: Audio preprocessor
parameters: parameters.yml
interface: com_bonseyes/interfaces#audio_preprocessing

The pre-processor supports the following configuration parameters in the algorithm.yml:

Parameters for audio pre-processing

Parameters used to perform audio pre-processing

type

object

properties

  • output_size

Size of the input to the model

type

object

properties

  • x

type

integer

  • y

type

integer

  • filter

Type of normalization filter used

type

string

enum

undefined, none, mfcc

  • numMels

Number of Mel filters

type

integer

  • numFFT

Number of FFT filters

type

integer

  • freq

Cut frequencies

type

object

properties

  • low

type

integer

  • high

type

integer

  • winLength

window length in ms

type

integer

  • frameShift

Hop length

type

integer

  • db

Power in db. Otherwise in log

type

boolean

  • htk

Htk function. Otherwise stanley

type

boolean

  • inverted

Inverted col and rows

type

boolean

  • mean_data

Mean normalization constant

type

number

  • std_data

Std-dev normalization constant

type

number

  • melNorm

Normalization for Mel filterbank

type

string

  • clipINT8

Clip values between 0 and 255

type

boolean

  • dump_tile

filename where to dump preprocessed tile

type

string

additionalProperties

False

Signal Pre-processor

The signal pre-processor has the following format:

metadata:
  title: Signal preprocessor
parameters: parameters.yml
interface: com_bonseyes/interfaces#signal_preprocessing

The pre-processor supports the following configuration parameters in the algorithm.yml:

Parameters for signal pre-processing

Parameters used to perform signal pre-processing

type

object

properties

  • output_size

Size of the input to the model

type

object

properties

  • x

type

integer

  • y

type

integer

  • normalization

  • dump_tile

filename where to dump preprocessed tile

type

string

additionalProperties

False

Inference

Inferece is the second subcomponent of an LPDNN’s AI App. The Inference processor takes prepared input from the pre-processor and exucutes the forward pass of the neural network, i.e., inference. The configuration of this subcomponent in the algorithm YAML file specifies which neutral network model should be executed.

The inference subcomponent supports the following configuration parameters:

type

object

properties

  • model

Path to the model file

type

string

  • feed_output_back

Should the output of this model be fed back to its input using values specified in out_layers and input_layers?

type

boolean

  • input_layers

Ordered list of name of input layers for this model

type

array

items

type

string

  • out_layers

Ordered list of name of output layers from the model that will be saved in the json output

type

array

items

type

string

  • out_format

Output format of the network: - value_list: a 1D vector of values, for example confidences - ssd: SSD output structure - xy_list: a 1D vector of points (x,y) - bb_landmarks: feature maps with confidences, bbs, landmarks - openpifpaf: openpifpaf output format - none: no actual output (useful for partial backbone networks) - ksdd2: defect detection output that contains classification confidence and coordinates of defective pixels - clipped: clip the output in case the prediction is above or below a threshold of the average of previously estimated values - cyclops2gaze: converts the cyclops gaze (pitch, yaw) in radians to a left and right eye gaze vector (left-pitch, left-yaw, right-pitch, right-yaw) in degree - landmarks_head_pose_tuple: a tuple (3d landmarks, headpose). 3d landmarks is a flatten vector of size batch_size x num_landmarks x 3. headpose is a flatten vector of size batch_size x 7 x 1. - 3dmm_params: a 1D vector of values containing the morphable model parameters (3DMM)

type

string

enum

value_list, ssd, none, openpifpaf, openpifpaf_0_8, openpifpaf_0_11, openpifpaf_0_12, openpifpaf_0_12_13, openpifpaf_0_12_13_wholebody, openpifpaf_0_13_3, openpifpaf_0_13_3_wholebody, openpifpaf_0_13_3_wholebody_59pts, tddfa_v2, xy_list, xy_list/eyes_nose_5, xy_list/eyes_nose_mouth_5, xy_list/face_68, bb_landmarks, bb_landmarks/eyes_nose_5, bb_landmarks/eyes_nose_mouth_5, bb_landmarks/eyes_nose_mouth_5/v2, bb_landmarks/eyes_nose_mouth_5/v3, nanodet, yolov4, fcn, ksdd2, clipped, cyclops2gaze, landmarks_head_pose_tuple, 3dmm_params

  • split_input_size

Indicator of the input tile size in case of split Ai Apps

type

object

properties

  • x

type

integer

  • y

type

integer

  • postprocessing

postprocessing parameters

type

object

properties

  • bp_config

Body pose postprocessing configuration

type

object

properties

  • nms

Non-maxima supression enabled flag

type

boolean

  • custom

Indicate a custom decoder (not a subset and a subgraph of body pose)

type

boolean

  • suppression

An NMS-related parameter

type

number

  • instance_threshold

An NMS-related parameter

type

number

  • cif_stride

CIF stride

type

integer

  • cif_floor

CIF floor

type

number

  • caf_stride

CAF stride

type

integer

  • caf_score_threshold

CAF score threshold

type

number

  • keypoint_threshold

Keypoint threshold (absolute)

type

number

  • keypoint_threshold_rel

Keypoint threshold (relative)

type

number

  • keypoints

Keypoint names

type

array

items

type

string

  • skeleton

Skeleton definition

type

array

items

type

array

items

type

number

  • weights

Weights definition

type

array

items

type

number

additionalProperties

False

Usually, only two parameter are used:

  • The subcomponents.inference.parameters.model is set to the path of the model YAML file model.yml.

  • The subcomponents.inference.parameters.out_format is set to specify the post-processing type (if any) that should be executed after the forward pass of the neural network.

Algorithm Configuration Examples

In this sub-section, we show examples of pre-defined algorithms:

Image Classification using a Neutral Network with image pre-processing

The specification of this algorithm and their subcomponents is give here: Image Classification Preprocess Infer Algorithm.

A minimal algorithm.yml configuration file for this algorithm is the following:

algorithm: com_bonseyes/ai_algorithms#image_classification_preprocess_infer
subcomponents:
  preprocessing:
    parameters:
      color_mode: grayscale
      normalization: constant
      mean_data: [ 0 ]
      std_data: [ 255 ]
      output_size:
        y: 120
        x: 120
  inference:
    parameters:
      model: model.yml
challenge: com_bonseyes/challenges/mychallenge#challenge.yml
Face Recognition using a Neutral Network with image pre-processing

The specification of this algorithm and their subcomponents is give here: Image Classification Preprocess Infer Algorithm.

A minimal algorithm.yml configuration file for this algorithm is the following:

algorithm: com_bonseyes/ai_algorithms#face_recognition_preprocess_infer
parameters:
  flip: true
subcomponents:
  preprocessing:
    color_mode: grayscale
    normalization: constant
    mean_data: [ 0 ]
    std_data: [ 255 ]
    output_size:
      y: 120
      x: 120
  inference:
    model: model.yml
challenge: com_bonseyes/challenges/mychallenge#challenge.yml

The flip parameter decides whether the face fingerprint returned by the module is fingerprint of the image or the fingerprint of the image concatenated with the fingerprint of the mirrored image.

Object detection using a Neutral Network with image pre-processing

The specification of this algorithm and their subcomponents is give here: Object Detection Preprocess Infer Algorithm.

A minimal algorithm.yml configuration file for this algorithm is the following:

algorithm: com_bonseyes/ai_algorithms#object_detection_preprocess_infer
subcomponents:
  preprocessing:
    parameters:
      color_mode: grayscale
      normalization: constant
      mean_data: [ 0 ]
      std_data: [ 255 ]
      output_size:
        y: 120
        x: 120
  inference:
    parameters:
      model: model.yml
      out_format: Post_Processing_Format
challenge: com_bonseyes/challenges/mychallenge#challenge.yml

Object detection AI Apps usually involve a post-processing step after inferece to decode the detected object’s istances. The inference.parameters.out_format field defines the format of such post-processing.

Image segmentation using a Neutral Network with image pre-processing

The specification of this algorithm and their subcomponents is give here: Image Segmentation Preprocess Infer Algorithm.

A minimal algorithm.yml configuration file for this algorithm is the following:

algorithm: com_bonseyes/ai_algorithms#image_segmentation_preprocess_infer
subcomponents:
  preprocessing:
    parameters:
      color_mode: grayscale
      normalization: constant
      mean_data: [ 0 ]
      std_data: [ 255 ]
      output_size:
        y: 120
        x: 120
  inference:
    parameters:
      model: model.yml
      out_format: Post_Processing_Format
challenge: com_bonseyes/challenges/mychallenge#challenge.yml

Image segmentation AI Apps usually involve a post-processing step after inferece to apply the segmentation mask. The inference.parameters.out_format field defines the format of such post-processing.

Audio Classification using a Neutral Network with audio pre-processing

The specification of this algorithm and their subcomponents is give here: Audio Classification Preprocess Infer Algorithm.

An algorithm.yml configuration file for this algorithm is the following:

algorithm: com_bonseyes/ai_algorithms#audio_classification_preprocess_infer
subcomponents:
  preprocessing:
  parameters:
    dump_tile: audio_dump.json
    filter: mfcc
    output_size:
      y: 40
      x: 32
    numMels: 128
    numFFT: 2048
    freq:
      low: 0
      high: 8000
    winLength: 2048
    frameShift: 512
    db: true
    htk: false
    inverted: false
    clipINT8: false
    mean_data: 0
    std_data: 1
    melNorm: None
  inference:
    parameters:
      model: model.yml
challenge: com_bonseyes/challenges/mychallenge#challenge.yml
Signal Classification using a Neutral Network with signal pre-processing

The specification of this algorithm and their subcomponents is give here: algorithms/signal_classification_preprocess_infer.

A mininal algorithm.yml configuration file for this algorithm is the following:

algorithm: com_bonseyes/ai_algorithms#signal_classification_preprocess_infer
subcomponents:
  inference:
    parameters:
      model: model.yml
  preprocessing:
    parameters:
      dump_tile: signal_dump.json
      output_size:
        x: 1
        y: 256
challenge: com_bonseyes/challenges/mychallenge#challenge.yml

An example algotithm YML file is also available in the 3D Face Landmarks AI Asset.

Deployment

The deployment folder contains deployment.yml files with the AI App’s deployment configuration. A deployment configuration YAML file specifies certain parameters for an algorithm implementation, e.g., what inference engine to use for the executation of the neural network, the data type (FP32, FP16, INT8) or the backends to use per layer. Several deployment.yml files can be stored under deployment/ for different configurations.

The schema of the supported deployment configurations are described below:

Algorithm Deployment specification

All information required by the deployment tool to generate a AI app from an algorithm specification.

type

object

properties

  • metadata

Metadata for AI Artifacts

  • component

The name of the component to use

type

string

  • ai_app_name

Name of the generate AI-app

type

string

  • ai_app_version

Aiapp version as major.minor.maintenance.build

type

string

  • ai_app_metadata

Metadata for the deployed AI-app

Metadata for AI Artifacts

  • parameters

Parameters required by the tool to deploy the algorithm

type

object

  • subcomponents

type

object

additionalProperties

type

object

properties

  • component

Component to use to implement the role

type

string

  • parameters

Parameters for the component

type

object

  • subcomponents

Deployment configuration for sub-components

#/properties/subcomponents

additionalProperties

False

A minimal deployment.yml file looks like:

ai_app_name: "MY-APPLICATION"
ai_app_version: "v0.1"

ai_app_metadata:
  title: Image classification
  author:
    email: hello@bonseyes.com
    name: Bonseyes Community Association
  description: |
      This AI app analyzes an image and returns the probability that the image belongs to any given class.
  extra:
    - id: com_bonseyes/marketplace/metadata#tags
      value:
      - bonseyes
      - image
      - computer vision
      - cnn
      - lpdnn
  license: Bonseyes Consortium License

component: com_bonseyes/lpdnn#core/components/image_classification_preprocess_infer

subcomponents:
  inference:
    component: com_bonseyes/lpdnn#engines/lne/components/network_processor
    parameters:
      optimizations: [all]
      data_type: F32

The main subcomponent in the deployment YAML file is the inference subcomponent, which defines the network_processor to use, i.e., inference engine to execute the neural network, in the component type field and the characteristic in the parameters field. Next, we specify each of the possible components that can be used for inference.

Deployment engines

As explained in LDPNN’s Inference engines, LPDNN supports the following network processors (inference engines), each of these allowing different configurations:

LNE

#/definitions/model_deployment

definitions

  • model_deployment

Deployment models for the LPDDN inference engine

< All information used necessary to configure the LPDNN engine to execute a given model.

type

object

properties

  • data_type

Data type used for computations

type

string

enum

ANY, F32, F16, S16, S8, U8

default

F32

  • optimizations

Optimizations to enable

type

array

  • qfile

File containing qdata

type

string

  • layers_config

Generation configuration for each layer

type

object

default

additionalProperties

False

TensorRT

#/definitions/model_deployment

definitions

  • model_deployment

Deployment models for the LPDDN inference engine

< All information used necessary to configure the LPDNN engine to execute a given model.

type

object

properties

  • data_type

Data type used for computations

type

string

enum

ANY, F32, F16, S16, S8, U8

default

F32

  • optimizations

Optimizations to enable

type

array

  • qfile

File containing qdata

type

string

  • cfile

File containing list of calibration tiles

type

string

  • layers_config

Generation configuration for each layer

type

object

default

  • engine

Name of the tensor rt engine file

type

string

  • dla

DLA used on device

type

integer

  • force_engine_rebuild

Force engine rebuild on every run

type

boolean

  • qcalibrator

Use quantisation calibrator

type

boolean

  • cbatch_size

Calibration batch size

type

integer

  • cbatches

Number of calibration batches

type

integer

  • opt_profile

Optimisation profiles for tensor rt engine build

type

object

properties

  • opt

Optimal tensor shape

type

array

  • min

Minimal tensor shape

type

array

  • max

Maximal tensor shape

type

array

  • exec_shape

Execution shape for batch execution

type

array

additionalProperties

False

NCNN

#/definitions/model_deployment

definitions

  • model_deployment

Deployment models for the LPDDN inference engine

< All information used necessary to configure the LPDNN engine to execute a given model.

type

object

properties

  • data_type

Data type used for computations

type

string

enum

ANY, F32, F16, S16, S8, U8

default

F32

  • optimizations

Optimizations to enable

type

array

  • qfile

File containing qdata

type

string

  • layers_config

Generation configuration for each layer

type

object

default

additionalProperties

False

ONNXruntime

#/definitions/model_deployment

definitions

  • model_deployment

Deployment models for the LPDDN inference engine

< All information used necessary to configure the LPDNN engine to execute a given model.

type

object

properties

  • data_type

Data type used for computations

type

string

enum

ANY, F32, F16, INT64, S16, S8, U8

default

F32

  • optimizations

Optimizations to enable

type

array

  • qfile

File containing qdata

type

string

  • log_severity

Define the logging level

type

string

enum

verbose, info, warning, error, fatal

  • verbosity_level

Defines verbosity level when logging severity is verbose; A higher value is more verbose.

type

integer

  • execution_mode

Choose between sequential or parallel execution

type

string

enum

sequential, parallel

  • enable_profiling

Enable the ORT engine profiling and provide the profile file path prefix

type

string

  • graph_optimization_level

ONNX graph optimization level

type

string

enum

none, basic, extended, all

  • optimized_model_path

The path to save the optimized model

type

string

  • inter_op_threads

Sets the max number of inter-op threads used to parallelize the execution

type

integer

  • intra_op_threads

Sets the max number of intra-op threads used to parallelize the execution (no OpenMP builds)

type

integer

  • execution_providers

Defines available execution providers

type

object

default

properties

  • cuda

CUDA execution provider configuration.

type

object

properties

  • device_id

CUDA device id

type

integer

default

0

  • gpu_mem_limit

The size limit of the device memory arena in bytes

type

integer

  • cudnn_conv_algo_search

The type of search done for cuDNN convolution algorithms

type

string

enum

exhaustive, heuristic, default

  • do_copy_in_default_stream

Whether to do copies in the default stream or use separate streams

type

boolean

default

True

  • cudnn_conv_use_max_workspace

Check tuning performance for convolution heavy models

type

integer

default

0

  • tensorrt

TensorRT execution provider configuration.

type

object

properties

  • device_id

CUDA device id

type

integer

default

0

  • max_workspace_size

Max workspace memory size for engine build

type

integer

  • max_partition_iterations

Max number of partition iterations

type

integer

  • min_subgraph_size

Min subgraph size

type

integer

  • fp16_enable

Enable FP16

type

boolean

default

True

  • int8_enable

Enable Int8

type

boolean

default

False

  • int8_calibration_table_name

Path to the Int8 calibration table

type

string

  • dla_enable

Enable DLA

type

boolean

default

False

  • dla_core

Select DLA core

type

integer

  • engine_cache_enable

Enable engine cache

type

boolean

  • engine_cache_path

Path to the engine cache

type

string

  • dump_subgraphs

Enable subgraphs dump

type

boolean

additionalProperties

False

A template of the deployment YML file is available in the AI Asset Container Generator.

An example deployment YML file is also available in the 3D Face Landmarks AI Asset.

Deployment packages

As explained in LDPNN’s Deployment of AI applications on embedded devices, an LPDNN Deployment Package is a collection of tools, executables, libraries, inference engines and backends that allows the actual execution of the LPDNN AI application. The collection of libraries and binaries that are copied on the target platform for the execution of the DNN is called a runtime. The runtime dinamically loads an AI application and executes it based on its defined configuration. LPDNN’s deployment packages are platform-specific as they contain the inferences engines and backends supported by the HW platform.

Deployment packages, including LPDNN runtimes, can be obtained by asking access to one of the BonsAPPS’s partner. LPDNN’s deployment packages can be found here.

A template of where the deployment packages should be placed is available in the AI Asset Container Generator.

An example containing deployment packages is also available in the 3D Face Landmarks AI Asset. The script get_packages.sh allows the user to donwload deployment packages by executing:

bash get_packages.sh ${platformName}

# For instance
bash get_packages.sh jetson_nano-jetpack4.6

Generate an AI App

Prerequisites

This section assumes:

  • All the steps in the create an LPDNN’s file tree section have been completed.

  • The local environment has been set up as explained in Prerequisites

Generation

There are two methods to generate an AI App:

  • AI App generator script: Python script that produces an AI App.

  • Bonseyes CLI: Main command line interface tool for Bonseyes tools and workflows.

AI App generator script

To use this script the following steps need to be performed:

  1. Change directory to the LPDNN’s directory, e.g., LPDNN root directory in AI Asset.

  2. Create a build folder:

    mkdir build
    
  3. Execute the following command:

    deployment_packages/${platformName}/tools/ai_app_generator/bin/ai-app-generator \
        --algorithm-file catalog/${modelName-version-size-precision}/algorithm.yml  \
        --challenge-file challenge/${challengeName}/challenge.yml \
        --deployment-file deployment/${modelName-version-size-precision}/deployment.yml  \
        --output-dir build/${new-aiapp-name}
    

For instance, to generate an AI App for the model mobilenet-v2-224x224-fp32 (imagenet challenge) on raspberry4b_64-ubuntu20:

deployment_packages/raspberry4b_64-ubuntu20/tools/ai_app_generator/bin/ai-app-generator \
    --algorithm-file catalog/mobilenet-v2-224x224-fp32/algorithm.yml  \
    --challenge-file challenge/imagenet/challenge.yml \
    --deployment-file deployment/mobilenet-v2-224x224-fp32/deployment.yml  \
    --output-dir build/mobilenet-v2-224x224-fp32

After executing the command a new LPDNN AI App will be present in the build folder with the given name. A number of file will be included in the AI App folder among which the most important ones are described at LPDNN AI application (AI App).

To regenerate the AI App with the same name, the flag --force can be used. Beware as this flag will first erase the previous AI App to then compile a new one.

Bonseyes CLI

It is also possible to build AI Apps with the Bonseyes CLI tool. To do so, this generation option includes an extra prerequite step:

  • Download and build the sources of the target platform’s DPE as explained in the platform’s Description in the user guide.

  • Have the ${platformName_source} and ${platformName_build} folders in lpdnn repository.

To generate an AI App with the Bonseyes CLI tool, execute the following commands:

  1. Change directory to the LPDNN’s directory, e.g., LPDNN root directory in AI Asset.

  2. Copy you target platform’s sources and build directories to LPDNN root directory:

    cp /Path/to/{platformName}_source /Path/to/{platformName}_build .
    
  3. Create a build folder:

    mkdir build
    
  4. Execute the following command:

    bonseyes --packages-dir .  ai-app generate \
        --algorithm-config catalog/${modelName-version-size-precision}/algorithm.yml  \
        --challenge challenge/${challengeName}/challenge.yml \
        --deployment-config deployment/${modelName-version-size-precision}/deployment.yml  \
        --deployment-tool deployment_packages/${platformName}/deployment_tool.yml
        build/${new-aiapp-name}
    

For instance, to generate an AI App for the model mobilenet-v2-224x224-fp32 (imagenet challenge) on raspberry4b_64-ubuntu20:

bonseyes --packages-dir .  ai-app generate \
    --algorithm-config catalog/mobilenet-v2-224x224-fp32/algorithm.yml  \
    --challenge challenge/imagenet/challenge.yml \
    --deployment-config deployment/mobilenet-v2-224x224-fp32/deployment.yml  \
    --deployment-tool deployment_packages/raspberry4b_64-ubuntu20/deployment_tool.yml \
    build/mobilenet-v2-224x224-fp32

After executing the command a new LPDNN AI App will be present in the build folder with the given name. A number of file will be included in the AI App folder among which the most important ones are described at LPDNN AI application (AI App).

Development

If the generation of an LPDNN AI App is not possible due to not supported features such as:

  • Not supported operator by the targeted inference engine

Please, you may try generate the AI App with another inference engine for your target device.

If the generation of an LPDNN AI App is not possible due to not supported features such as:

  • Specific pre-processing / post-processing not existing.

  • Not supported operator by the target inference engine (LNE only).

  • AI Class not existing, e.g., signal regression.

Please, follow the dev guidelines for LPDNN’s Description to get started with LPDNN development in LPDNN’s SDK.

Execute an AI App on host device

Execution of FP32 AI Apps

As explained in LDPNN’s Deployment of AI applications on embedded devices, two elements are need to execute an AI Application on a device:

  • An LPDNN AI application: Created in previous steps.

  • An LPDNN Deployment Package: Donwloaded in previous steps.

To execute an FP32 AI Apps on your host, you can use the aiapp-cli (standalone executable) that allows to easily execute an aiapp from the command line. It receives as input parameters the aiapp configuration file and the input file(s) to be processed. It will internally instantiate the correct aiapp class according to the specified configuration file and generate human-readable result. To execute the AI App, follow the next steps:

  1. The executable is available under bin/. Change directory to the deployment package binaries that match you current device:

    cd deployment_packages/${platformName}/bin
    
    # For instance:
    
    cd deployment_packages/x86_64-ubuntu20_cuda/bin
    
  2. Set the library’s and python’s path:

    source set-lib-path.sh
    
  3. Infer an AI App on an input sample:

    ./aiapp-cli --cfg ../../../build/${new-aiapp-name}/ai_app_config.json -f /Path/To/Input/File{.png/.wav/.json}
    
    # For instance:
    
    ./aiapp-cli --cfg ../../../build/mobilenet-v2-224x224-fp32-cli/ai_app_config.json -f /Path/To/Input/File{.png/.wav/.json}
    

An example of a python-based AI APP cli (aiapp-cli.py) with the same execution parameters is also available in all LPDNN’s deployment_package under the bin/ folder.

Processing options

Input format

The input file can be a .jpg/png format for image-based AI Apps or .wav for audio files. However, it is also possible to pass an input tile in JSON format that may contain any kind of data. The JSON file must have the following structute:

{
  "data":[1.5883,0.272,0.89910,...,...],  # Modify based on Input
  "dim":[1,3,224,224],                    # Modify based on AI App
  "info":"NCHW"                           # Currently only NCHW input layout is supported
}

JSON input tiles are not pre-processed by LPDNN, i.e., pre-processing step is skipped, as they usually they serve to concatenate several AI App by feeding the results of the fist one into the input of the second one. Note: Currently only supported input layout is NCHW

Split pre-proceesing and inference

If the file provided in input to an AI App has .json suffix, it is considered as a tile, so no preprocessing is done and the data are sent directly to inference.

The --out-tile option allows to generate the tile (that is the output of the preprocessor) in a JSON file as an AI App blob, that is, a flattened data array of floats plus a dim array representing the actual dimensions. This makes it easy to perform preprocessing and inference in two separate stages if needed.

Additional input

Extra input

An given AI App may require additional information, e.g., region-of-interest or landmarks, as an input. This information can be provided in input to aiapp-cli by sending the extra input as stdin:

cat landmarks.json | ./aiapp-cli --cfg ../../../build/${new-aiapp-name}/ai_app_config.json -f /Path/To/Input/File

To generate a file such as landmarks.json from a first AI App, you may execute it with the --out-inference parameter to obtain a JSON file with the results:

./aiapp-cli --cfg ../../../build/${new-aiapp-name}/ai_app_config.json -f /Path/To/Input/File --out-inference landmarks.json

Multiple inputs

Some AI Apps, however, require multiple inputs, some of which are fed directly from the output of the same AI App (temporal networks). If a model supports multiple inputs, e.g., tsm-optimized for gesture-recognition, you can use the --additional-inputs <file.json> argument to specify a JSON file that contains the additional values.

Values in the JSON must use the same format of the file produced by --out-inference. Input order is defined in the algorithm.yml of the model, using the inference/parameter/input_layers field.

When creating manually a JSON file to pass addtional inputs, an empty “blob” must be added at the beginning of the array. For example if you have 2 additional inputs, the JSON array should contains this: [empty blob, <additional_input1>, <additional_input2>]

This is done because the output of --out-inference includes both default output and additional outputs ([default, additional1, additional2, …]) and by doing it this way, it is possible to directly feed the produced JSON back to the model without having to change it, which is useful for some models.

Note that if a model has multiple output (also specified in the algorithm.yml file, they can be retrieved in the file generated with --out-inference).

gesture_app=../share/ai-apps/tsm-optimized/ai_app_config.json

# 1. Run the app without additional inputs the first time
./aiapp-cli --cfg $gesture_app -f $image --out-inference previous_output.json

# 2. Run the app and pass previous output as additional input the second time
./aiapp-cli --cfg $gesture_app -f $image --additional-inputs previous_output.json

Concatenation of two AI App

Two AI Apps can be concatenated by redirecting the output of the first AI App into the second AI App.

Bash

Pre-built AI Apps can be concatenated through bash calls. The output logs of the first AI App are not displayed and the result is generated in json format, which is fed into the following AI App as stdin:

./aiapp-cli --cfg $face_detection_aiapp -f $image | ./item -i 0 | \
./aiapp-cli --cfg $emotion_aiapp -f $image

Please, note the use of the ./item utility script in the commands above. Both face detection and landmark detection aiapps generate in output a list of items (one per face), where each item contains a bounding box and possibly landmarks. Emotion aiapp is instead able to process one face at a time, so it takes in stdin information regarding the bounding box and landmarks of a single face. We thus need a way to select which face to process. ./item -i 0 simply reads the output results of the first AI App and feed an item (by index) to second AI App..

If needed it is easy to provide different selector scripts with different behaviour, for example to select the biggest face, or the more central one, or the one with the highest confidence.

Python

AI Apps can also be concatenated in python to create a single more complex application as follows:

# Read file
with open(sample_file, 'rb') as fp:
     binary_data = fp.read()
# Create image
sample = aiapp.encoded_sample(binary_data)
# Execute Face Detection AI App
res = aiapp.execute(sample, {})

for item in res.items:
  # Set ROI and landmarks to input image
  sample.roi = item.bounding_box
  sample.landmarks = item.landmarks
  # Execute Emotion AI App
  res = aiapp.execute(sample, {})

An example of AI APP cli (aiapp-cli.py) is available in all LPDNN’s deployment_package under the bin/ folder.

C++

During development, C++ code can also be concatenated to create a single built-in more complex application as follows:

# Read file
auto image_content = app::read_binary_file(filename);
# Create image
auto image = ai_app::Image(ai_app::Image::Format::encoded, image_content, {});
# Execute Face Detection AI App
auto results = aiapp_facedet->execute(image);

for (auto item : result.items) {
  # Set ROI and landmarks to input image
  app::object_detection_item_to_image(item, image);
  # Execute Emotion AI App
  auto results = aiapp_emotion->execute(image);
}

Other options

For more options about inferring an AI App, execute:

./aiapp-cli -h

     --cfg <file>           Ai-app configuration file
     --info                 Show aiapp information
     -f <file;file..>       Sample file(s) to process
     -d <dirname>           Sample directory to process
     -p <file>              Generate metrics file
     -l <filename>          Generate layer dump file
     --out-tile <filename>  Generate preprocessed tile in json format
     --preprocess-only      Preprocessing only (skip inference)
     --out-inference <filename>  Generate raw inference out in json format
     --additional-inputs <file> Path to JSON generated with --out-inference, provides values for additional model inputs (lpdnn/image detection only)  -r <N>                 Repeat inference N times
     -rr <N>                Repeat creation and inference N times
     --help                 Show this help

Quantisation workflow for LPDNN’s engines

LPDNN curently supports post-training quantization (PTQ) for its inference engines. To execute quantized models, some pre-steps need to be followed.

TensorRT

TensorRT supports FP16 and INT8 quantization.

FP16

For FP16 deployment, the following steps need to be performed:

  1. Change the subcomponent.inference.parameters.data_type field to F16 in the deployment-file.yml. Note: Data type in LPDNN is F16, not FP16

  2. Generate again the target AI App using the new deployment YAML file.

  3. Execute the AI App as explained in the previous section.

INT8

TensorRT provides it’s own calibration tool for INT8 deployment. To use the TensorRT quantization tool, perform the following steps:

  1. Prepare a calibration dataset to adjust the DNN’s activations’ range by calculating the activation’ scale and offset in order to retain a high amount of accuracy during the quantization process.

    • We need to obtain pre-processed tiles to make sure the neural network’s calibration process ontains the real range that network sees during inferece.

    • To obtain the pre-processed tiles, have a FP32 ai-app generated following the steps explained in previous sections.

    • Run the preprocessing-only mode of the aiapp-cli executable for the AI App (compiled with any engine).

    cd deployment_packages/${platformName}/bin
    source set-lib-path.sh
    ./aiapp-cli --cfg  ../../../build/${aiapp-name}/ai_app_config.json --preprocess-only --out-tile <tile-n.json> -f <img-n.jpg>
    
    • Make sure to prepare enough representative tiles for the calibration set.

  2. Make a text file, e.g., cfile.txt, containing the paths, line by line, toward the <tile-n.json> tiles generated in the previous step.

  3. In the deployment-file.yml for tensorRT engine, add qcalibrator, cfile, cbatch_size, and cbatches fields to the subcomponent.inference.parameters.

  4. Set qcalibrator to true, set cfile to the absolute path of the file containing the tile paths (without “”).

  5. Set cbatch_size to the desired calibration batch size, e.g., 1.

  6. Set cbatches to the number of calibration batches provided the data, e.g., if 100 tiles are available and a cbatch_size of 10 is chosen, there will be 10 cbatches.

  7. Set data_type field to S8.

  8. Generate again the target AI App using the new deployment YAML file.

  9. Execute again the AI App. TensorRT will perform calibration process that might last some time. Some logs such as the following should appear:

    [Trt::CreateEngine@73] Quantisation using TensorRT calibrator.
    
    ..
    
    Starting Calibration.
      Calibrated batch 0 in 0.0979145 seconds.
      Post Processing Calibration data in 1.12499 seconds.
    Calibration completed in 3.09242 seconds.
    Writing Calibration Cache for calibrator: TRT-7103-EntropyCalibration2
    
  10. Upon a successful calibration, the platform-independent calibration results will be saved in qfile.txt in the AI App folder.

  11. One can reuse the obtained results for the given AI App as a quantisation cache for building TensorRT engines on different target platforms without redoing the calibration process. To do so, add the path to the qfile.txt in deployment-file.yml’s subcomponent.inference.parameters.qfile field before generating the AI App. If so, cbatch_size, and cbatch_size fields can be removed from the deployment file.

ONNXruntime

ONNXruntme supports INT8 quantization.

INT8

LPDNN’s onnxruntime engine is able to execute quantized INT8 ONNX models directly. Hence, the only neeed steps are the following:

  1. Quantize your model in the AI Asset as explained in onnxQuantize.

  2. Make sure to have the quantized ONNX model in catalog/${modelName-version-size-precision}/blobs and the right path to it on the catalog/${modelName-version-size-precision}/model.yml.

  3. In the deployment-file.yml, set data_type field to S8 under subcomponent.inference.parameters.

  4. Generate the AI App as explained in previous sections.

  5. A quantized AI App is ready to be executed.

NCNN

NCNN supports FP16 and INT8 quantization.

FP16

For FP16 deployment, the following steps need to be performed:

  1. Change the subcomponent.inference.parameters.data_type field to F16 in the deployment-file.yml. Note: Data type in LPDNN is F16, not FP16

  2. Generate again the target AI App using the new deployment YAML file.

  3. Execute the AI App as explained in the previous section.

INT8

NCNN provides its own calibration tool to calculate the activation ranges against a valiadation dataset. LPDNN provides a pre-compiled ncnn2table script:

  1. Have a pre-compiled FP32 model using NCNN engine as explained in previous sections.

  2. Have a calibration dataset and generate a quantisation file as follows:

    ./deployment_packages/${platformName}/tools/ncnn/ncnn2table
        --param=build/${aiapp-name}/model.param
        --bin=build/${aiapp-name}/model.bin
        --images=PATH/TO/IMAGES_DIR
        --output=build/${aiapp-name}/quant.table
        --mean=0,0,0      # Modify
        --norm=1,1,1      # Modify
        --size=224,224    # Modify
        --thread=8
    
  3. Include the path to the quantisation file: quant.table in deployment-file.yml’s subcomponent.inference.parameters.qfile field and set data_type field to S8.

  4. Generate again the target AI App, using the new deployment YAML file and a new name for the AI App (Avoid using –force as it will delete the quant.table).

  5. Execute the AI App as explained in the previous section.

LNE

LNE supports INT8 quantization for both S8 and U8, symmetric and asymmetric types.

INT8

LPDNN provides a calibration tool to calculate the activations ranges against a validation dataset. To quantized a model with LNE, execute the following steps:

  1. Execute the data extraction tool to dump the activations ranges against a validation dataset on a calibration.json file:

    
    
    python3 deployment_packages/${platformName}/tools/lib/lpdnn_onnx/extract_quantization_onnx.py

    -m catalog/${modelName-version-size-precision}/blobs/${modelName-version-size-precision}.onnx –mean-pixel 0 0 0 # Modify –std-pixel 1 # Modify -i /Path/To/Calibration/Images/ –out-table build/${aiapp-name}/calibration.json

  2. Include the path to the calibration JSON file in the deployment file in deployment-file.yml’s subcomponent.inference.parameters.qfile field.

  3. Choose the data type. LNE currently supports U8 data type for ARM CPU and S8 for Nvidia GPU.

  4. Generate again the target AI App, using the new deployment YAML file and a new name for the AI App (Avoid using –force as it will delete the calibration.json).

  5. Execute the AI App as explained in the previous section.

Deploy and benchmark an AI App on target device

For the deployment and benchmarking of an AI App on a target device, i.e., not the host where the AI App has been generated, the follwing step is required:

  • Setup the target hardware as explained in Setup platform section in the user guide and have a ${platformName_config}.

Bonseyes CLI benchmark

If automated benchmarking is desired, please refer to: Benchmark your AI-App you target HW section in the user guide to execute a benchmark with the Bonseyes CLI tool.

Custom deployment and benchmark

If custom deployment is desired, the AI App’s HTTP-worker interface can be used to deploy an AI App on a target device while posting samples, e.g., images, from your host PC. The http-worker can be found in all LPDNN’s deployment_packages under the bin/ folder and is started as follows:

./http-worker --cfg ../../../build/${new-aiapp-name}/ai_app_config.json -s ${IP_ADDR} -p ${Port}

Host side

Bash

It is then possible to connect to the worker from the host as follows (path to the image must contain the character “@” before the path):

curl --request POST --data-binary @/PATH/TO/FILE ${IP_ADDR}:${Port}/inference

It is also possible to generate execution metrics or specify additional parameters (such as bounding box or landmarks) by adding the corresponding fields to the header:

curl --request POST --data-binary @/PATH/TO/FILE --header 'x-metrics: all' --header "x-params: $(cat landmarks.json)" ${IP_ADDR}:${Port}/inference

Python

It is also possible to connect to the http-worker from a python interface:

url = 'http://' + '${IP_ADDR}' + ':' + '${Port}' + '/inference'

headers = {'x-metrics': 'all'}
headers = {'x-params': json.dumps({"bounding_box":{"origin":{"x":347,"y":113},"size":{"x":295,"y":405}}})}

with open(input_sample, 'rb') as fp:
       with requests.post(url, data=fp.read(), headers=headers) as req:
           if req.status_code != 200:
               raise Exception('Error returned by the server worker')
           # parse the report
           report = req.json()
           req.close()

print(report)

To change the type of input, e.g., rgb or input tile, the header should add the folliwing line:

# Input tile input
headers = {'x-in-type': "tile-json"}

# RGB input
headers = {'x-in-type': "raw_rgb8"}

A complete python script, called http-client.py, for posting input samples to the http-worker is provided within each LPDNN’s deployment_package under the bin/ folder.

Upload AI App

THIS SECTIONS IS IN PROCESS

This guide assumes you already have the done the following:

  • Setup the local environment as explained in Prerequisites

  • Created an AI app with the procedure described in generate_ai_app

  • You have the deployment tool used to generate the AI app in the directory deployment_tool

  • Created a git repository on Gitlab that will host the AI app

  • Joined on the marketplace the challenge that the AI app is solving

  • Enrolled your gitlab user in the marketplace

To upload an AI app you need to do the following:

  1. Create your author keys:

    $ bonseyes license create-author-keys --output-dir author_keys
    
  2. Package your ai-app for release with its dependencies and a redistribution licenses:

    $ bonseyes --package deployment_tool/runtime marketplace package-ai-app \
                 --pull-images \
                 --ai-app ai_app \
                 --author-keys author_keys \
                 --amount "100 CHF" \
                 --recipients "test@dd.com,tost@dd.com" \
                 --regions https://www.wikidata.org/wiki/Q39,https://www.wikidata.org/wiki/Q40 \
                 --output-dir packaged_ai_app
    

    The --recipients parameter can be used to restrict who can see the artifact on the marketplace, if omitted all users can see it.

    The --regions parameter can be used to restrict the regions in which the artifact can be seen, if omitted users of all regions can see it.

  3. Commit the packaged AI app in a gitlab repository that is accessible by the gitlab user you enabled in the marketplace

  4. Upload the AI app to the marketplace:

    $ bonseyes marketplace upload-ai-app --url $HTTP_URL_OF_GITLAB_REPO
    

The AI app will be added to your profile.

Create challenge-compliant interface for an AI App

LPDNN AI Apps might not be compliant with the specification of an AI Challenge published in the BMP. Therefore, a C++ interface needs to be use to parse the output results of the LPDNN AI App to the expected results specified in the AI Challenge. To do so, the following steps need to be followed:

  • Create LPDNN Challenge as described in Challenge creation guideline

  • Create a header file (.hpp) that includes the AI Challenge c++ interface and parses the AI App’s results

  • Create a CLI example that takes an built AI App and produces AI Challenge compliant results

Create header file

The objective of the header file is to parses the AI App’s results and make them compliant with the AI Challenge specifications. To do so, the c++ interface’s header file, from the requirements submodule, needs to be included.

A template for the header file, including the main methods, is available here. To complete the header file, the following steps are required:

  1. Replace “LPDNN_AIAPP_CLASS” by the LDPNN AI class that the AI App is built upon, which is defined in the challenge.yml under ID (interface). For instance, image_classification or object_detection

  2. Replace “CHALLENGENAME_Result” by the Challenge Interface header file name.

  3. Add code your code to parse the AI App results into Result_Challenge(), challenge_interface() and to_json_str() methods. Any other method might also be modified or added if needed.

An example of a header file for 2D Bodypose Estimation is given here

Create CLI example

A working CLI example in the LPDNN-Challenge needs to be created to showcase the AI App. To do so, follow next steps:

  1. Create example folder in the LPDNN-Challenge folder, which contains include and test folders. A src folder might also be created.

  2. Add the header file created in the previous step into the header folder.

  3. Add main.cpp file into the test folder. A template of such a main.cpp file is available here. Modify CHALLENGENAME and Aiapp_NAME by the corresponding names that are used in the header file.

  4. Add CmakeLists.txt file in the example folder. A template for a CmakeLists.txt file is available `here https://gitlab.com/bonseyes/bonsapps/1st-support-programme/qa/-/blob/main/templates/interface_c%2B%2B/CMakeLists.txt>`_. Replace AIAPPNAME-cli and aiapp_CHALLENGENAME by the corresponding names.

To build and run the CLI example, follow next steps:

  1. Create build folder and change directory into it:

    mkdir build
    cd build
    
  2. Compile CLI example, giving the deployment package of the host machine:

    cmake .. -DLPDNN_DEPLOYMENT_PACKAGE=/PATH/TO/{deployment_package}
    make
    
  3. Run the CLI example:

    ./EXAMPLENAME-cli --cfg /PATH/TO/AIAPP//ai_app_config.json -f /PATH/TO/INPUT/SAMPLE