*********** Description *********** This guideline describes how to create, generate and deploy Bonseyes AI Apps given a pre-trained AI model. This guideline assumes you already have the done the following: - Setup the local environment as explained in :doc:`/pages/setup` - Trained an AI model and exported such model in `ONNX `_ format. For more information about training and exporting models, refer to the `AI Asset guideline `_. - Have followed `AI App user guide description `_ and got familiar with LPDNN. - Make sure to install system package ``hdf5-tools`` and python ``h5py``. .. _lpdnn_tree: ************************ Create LPDNN's file tree ************************ To generate and deploy AI Apps, a file tree with certain format needs to be created: .. code-block:: bash challenge/ catalog/ deployment/ deployment_packages/ where - **Challenge**: Folder containing the specifications of the AI Challenge that the AI App aims to solve. - **Catalog**: Folder containing pre-trained AI models. - **Deployment**: Folder containing the AI App's deployment characteristics. - **Deployment packages**: Folder containing packages that contain tools, executables, libraries and backends that allows the actual execution of the LPDNN AI application. A template with this organization is available in the `AI Asset Container Generator `_. An example with LPDNN tree file is also available in the `3D Face Landmarks AI Asset `_. In the following sections, each of this folders is detailed and described. Challenge ~~~~~~~~~ To create an LPDNN AI Challenge for you AI App, please follow the dev guide for AI Challenge's :ref:`lpdnnChallenge` and creation. ``Note``: the following steps can be skipped for the moment: - *Defining the challenge training and sample data (Deprecated)*, - *Defining the evaluation procedure (Deprecated)* - *Upload Challenge*. After having followed the creation of the AI Challenge, the LPDNN's file tree ``challenge/`` directory must include the following files: - challenge.yml - classes.txt (if required by challenge interface) - landmarks.txt (if required by challenge interface) .. _lpdnn_catalog: Catalog ~~~~~~~ The catalog is a folder that contains AI models: .. code-block:: bash catalog/ ${modelName-version-size-precision}/ ${modelName-version-size-precision}/ ${modelName-version-size-precision}/ where *${ModelNameVersionSizePrecision}* is the name of the model, with the specific version, input tile size and precision. LPDNN expects AI models in a certain format and organization: .. code-block:: bash ${modelName-version-size-precision}/ algorith.yml model.yml blobs/ ${modelName-version-size-precision}.onnx where: - **algorith.yml**: YML file that describes the AI model's type of algorithm or class, e.g., image classification, the challenge that it tries to solve and the pre-, inference, and post-processing details. - **model.yml**: YML file that describes the format of the AI model, e.g., ONNX, and the path to the model. - **blobs**: Folder that holds the trained AI model(s). - **${modelName-version-size-precision}.onnx**: Pre-trained model file(s) following the model naming explained above. .. _defining_model: model.yml --------- The ``model.yml`` file is a simple YAML file that provides metadata required to consume an AI model in one of the supported model file formats. It has the following structure:: type: com_bonseyes/network_formats#caffe_bvlc_0_1 parameters: deploy_model: "blobs/deploy.prototxt" weights: "blobs/model.caffemodel" The ``type`` parameter specifies the network format used by the model. The following models are currently supported: - Caffe SSD (id: ``com_bonseyes/network_formats#caffe_ssd_0_1``) - Caffe Bonseyes (id: ``com_bonseyes/network_formats#caffe_bonseyes_0_1``) - NVIDIA-Caffe (id: ``com_bonseyes/network_formats#caffe_nvidia_0_1``) - BVLC Caffe (id: ``com_bonseyes/network_formats#caffe_bvlc_0_1``) - ONNX (id: ``com_bonseyes/network_formats#onnx_0_1``) The ``parameters`` section specifies the model's file path under the blobs' folder. For the caffe based models the following parameters are required: - ``deploy_model``: path to the caffe prototxt relative to the blobs' directory containing the model YAML file. - ``weights``: path to the caffe weights relative to the blobs' directory containing the model YAML file. For the ONNX models the following parameters are required: - ``deploy_model``: path to the ONNX file relative to the blobs' directory containing the model YAML file. A template of the model YML file is available in the `AI Asset Container Generator `_. An example model YML file is also available in the `3D Face Landmarks AI Asset `_. algorithm.yml ------------- The ``algorithm.yml`` is the highest-level configuration YAML file that specifies the components and the algorithms that are executed inside an AI App to solve a given AI challenge. An algorithm defines the relationship between the component's input and output. Thus, the algorithm.yml file defines all the component's parameters that are required to execute such algorithms. The schema of the file is described below: .. jsonschema:: ../../../schemas/core/algorithm_config.yml A minimal algorithm configuration looks as follows:: algorithm: com_bonseyes/ai_algorithms#image_classification_preprocess_infer challenge: com_bonseyes/challenges/mychallenge#challenge.yml parameters: ... subcomponents: preprocessing: parameters: ... inference: parameters: model: model.yml ... A template of the algorithm YML file is available in the `AI Asset Container Generator `_. The ``algorithm`` field is used to select the AI App's algorithm type. The following AI App's algorithms are currently supported in LPDNN: - **Image Classification using a Neutral Network with image pre-processing** (id: ``com_bonseyes/ai_algorithms#image_classification_preprocess_infer``): this algorithm converts an image to a specific format and size, normalizes the pixel values and then executes a neutral network to compute the confidence of each possible image class. This algorithm solves AI challenges with interface ``com_bonseyes/interfaces#image_classification``. - **Face Recognition using a Neutral Network with image pre-processing** (id: ``com_bonseyes/ai_algorithms#face_recognition_preprocess_infer``): this algorithm converts a face image to a specific format and size, normalizes the pixel values and then executes a neutral network to compute the fingerprint for the face. This algorithm solves AI challenges with interface ``com_bonseyes/interfaces#face_recognition``. - **Object detection using a Neutral Network with image pre-processing** (id: ``com_bonseyes/ai_algorithms#object_detection_preprocess_infer``): this algorithm converts an image to a specific format and size, normalizes the pixel values and then executes a neutral network to compute bounding boxes, landmarks and confidence of the detected objects classes. This algorithm solves AI challenges with interface ``com_bonseyes/interfaces#object_detection``. - **Image segmentation using a Neutral Network with image pre-processing** (id: ``com_bonseyes/ai_algorithms#image_segmentation_preprocess_infer``): this algorithm converts an image to a specific format and size, normalizes the pixel values and then executes a neutral network to compute a segmentation mask and confidence of the detected objects classes. This algorithm solves AI challenges with interface ``com_bonseyes/interfaces#image_segmentation``. - **Audio Classification using a Neutral Network with audio pre-processing** (id: ``com_bonseyes/ai_algorithms#audio_classification_preprocess_infer``): this algorithm converts an audio snippet to a specific format and then executes a neutral network to compute the confidence of each possible audio snippet class. This algorithm solves AI challenges with interface ``com_bonseyes/interfaces#audio_classification``. - **Signal Classification using a Neutral Network with signal pre-processing** (id: ``com_bonseyes/ai_algorithms#signal_classification_preprocess_infer``): this algorithm converts a signal stream to a specific format and then executes a neutral network to compute the confidence of each possible signal class. This algorithm solves AI challenges with interface ``com_bonseyes/interfaces#signal_classification``. The ``challenge`` field points to the AI challenge YAML file that describes the challenge being answered by the algorithm configuration. The file may refer the AI Challenge YAML file with a relative name if it is contained in the same package or use the syntax ``package_name#path/to/challenge.yml`` to refer to a challenge in another package. The ``parameters`` field contains the parameters that define the behaviour of the algorithm. The parameters depend on the algorithm. The ``subcompenents`` field provides parameters and specifications for the subcomponents of the algorithm, e.g., pre-processing, inference. Subcomponents ^^^^^^^^^^^^^ In this sub-section we describe the parameters and details of the subcomponents. LPDNN's AI Apps have two main components: - **Pre-processing**: Section that specifies how the input must be prepared. - **Inference**: Section that specifies which neutral network model should be executed. .. _lpdnn_preprocessors: Pre-processing ============== The pre-processing is the first subcomponent of an LPDNN's AI App. The pre-processor takes the raw input and transform it into the format and specifications that the neural network expects. The pre-processor can execute several functions such as cropping, normalization, filtering, domain transformations, etc. Each function needs to be specified in the ``subcomponents.inference.parameters`` field. **Image Pre-processor** The image pre-processor has the following format: .. literalinclude:: ../../../algorithms/image_preprocessor/algorithm.yml :language: yaml The pre-processor supports the following configuration parameters in the algorithm.yml: .. jsonschema:: ../../../algorithms/image_preprocessor/parameters.yml#/definitions/image_preprocessing The pre-processor assumes a single image as an input, e.g., rgb, grayscale. The ``output_size`` field should be specified for ONNX models, which assumes a NCHW layout, e.g., [1, 3, 120, 120] for rgb image. An minimal image pre-processing configuration look like following:: color_mode: grayscale normalization: constant mean_data: [ 0 ] std_data: [ 255 ] output_size: y: 120 x: 120 The first parameter ``color_mode`` describes the type of output image that should be generated by the pre-processing, it can be ``grayscale`` for grayscale image, ``bgr`` for BGR 8-bit images and ``rgb`` for RGB 8-bit images. The ``normalization`` parameter specifies the type of normalization that should be applied to image. The options are the following: ``constant`` where a normalization constant is used to normalize each channel and ``per-pixel``, where each point of the image is normalized with different parameters. In the case of ``constant`` normalization the constants for mean and standard deviation are specified in the ``mean_data`` and ``std_data`` fields. For instance to normalize an RGB image with mean 127 on each channel and standard deviation 127 the following configuration would be used:: normalization: constant mean_data: [ 127, 127, 127 ] std_data: [ 127, 127, 127 ] The pre-processor works with raw image data on [0, 255] range. Therefore, if your AI model has been trained and normalized on [0, 1] range, the values to set for the mean and std in the *algorithm.yml* need to be those used during training multiplied by 255, so as to bring back the data on [0, 1] range. The ``per-pixel`` normalization would look like this the configuration file would include the following:: normalization: per-pixel mean_file: blobs/mean.bin std_file: blobs/std.bin The ``mean_file`` and ``std_file`` point to binary files containing the per-pixel data. **Audio Pre-processor** The audio pre-processor computes MFCC features from the audio sample. It has the following format: .. literalinclude:: ../../../algorithms/audio_preprocessor/algorithm.yml :language: yaml The pre-processor supports the following configuration parameters in the algorithm.yml: .. jsonschema:: ../../../algorithms/audio_preprocessor/parameters.yml#/definitions/audio_preprocessing **Signal Pre-processor** The signal pre-processor has the following format: .. literalinclude:: ../../../algorithms/signal_preprocessor/algorithm.yml :language: yaml The pre-processor supports the following configuration parameters in the algorithm.yml: .. jsonschema:: ../../../algorithms/signal_preprocessor/parameters.yml#/definitions/signal_preprocessing Inference ========= Inferece is the second subcomponent of an LPDNN's AI App. The Inference processor takes prepared input from the pre-processor and exucutes the forward pass of the neural network, i.e., inference. The configuration of this subcomponent in the algorithm YAML file specifies which neutral network model should be executed. The inference subcomponent supports the following configuration parameters: .. jsonschema:: ../../../algorithms/inference_processor/parameters.yml Usually, only two parameter are used: - The ``subcomponents.inference.parameters.model`` is set to the path of the model YAML file :ref:`defining_model`. - The ``subcomponents.inference.parameters.out_format`` is set to specify the post-processing type (if any) that should be executed after the forward pass of the neural network. Algorithm Configuration Examples ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In this sub-section, we show examples of pre-defined algorithms: Image Classification using a Neutral Network with image pre-processing ====================================================================== The specification of this algorithm and their subcomponents is give here: :doc:`algorithms/image_classification_preprocess_infer`. A minimal ``algorithm.yml`` configuration file for this algorithm is the following:: algorithm: com_bonseyes/ai_algorithms#image_classification_preprocess_infer subcomponents: preprocessing: parameters: color_mode: grayscale normalization: constant mean_data: [ 0 ] std_data: [ 255 ] output_size: y: 120 x: 120 inference: parameters: model: model.yml challenge: com_bonseyes/challenges/mychallenge#challenge.yml Face Recognition using a Neutral Network with image pre-processing ================================================================== The specification of this algorithm and their subcomponents is give here: :doc:`algorithms/image_classification_preprocess_infer`. A minimal ``algorithm.yml`` configuration file for this algorithm is the following:: algorithm: com_bonseyes/ai_algorithms#face_recognition_preprocess_infer parameters: flip: true subcomponents: preprocessing: color_mode: grayscale normalization: constant mean_data: [ 0 ] std_data: [ 255 ] output_size: y: 120 x: 120 inference: model: model.yml challenge: com_bonseyes/challenges/mychallenge#challenge.yml The ``flip`` parameter decides whether the face fingerprint returned by the module is fingerprint of the image or the fingerprint of the image concatenated with the fingerprint of the mirrored image. Object detection using a Neutral Network with image pre-processing ================================================================== The specification of this algorithm and their subcomponents is give here: :doc:`algorithms/object_detection_preprocess_infer`. A minimal ``algorithm.yml`` configuration file for this algorithm is the following:: algorithm: com_bonseyes/ai_algorithms#object_detection_preprocess_infer subcomponents: preprocessing: parameters: color_mode: grayscale normalization: constant mean_data: [ 0 ] std_data: [ 255 ] output_size: y: 120 x: 120 inference: parameters: model: model.yml out_format: Post_Processing_Format challenge: com_bonseyes/challenges/mychallenge#challenge.yml Object detection AI Apps usually involve a post-processing step after inferece to decode the detected object's istances. The inference.parameters.out_format field defines the format of such post-processing. Image segmentation using a Neutral Network with image pre-processing ==================================================================== The specification of this algorithm and their subcomponents is give here: :doc:`algorithms/image_segmentation_preprocess_infer`. A minimal ``algorithm.yml`` configuration file for this algorithm is the following:: algorithm: com_bonseyes/ai_algorithms#image_segmentation_preprocess_infer subcomponents: preprocessing: parameters: color_mode: grayscale normalization: constant mean_data: [ 0 ] std_data: [ 255 ] output_size: y: 120 x: 120 inference: parameters: model: model.yml out_format: Post_Processing_Format challenge: com_bonseyes/challenges/mychallenge#challenge.yml Image segmentation AI Apps usually involve a post-processing step after inferece to apply the segmentation mask. The inference.parameters.out_format field defines the format of such post-processing. Audio Classification using a Neutral Network with audio pre-processing ====================================================================== The specification of this algorithm and their subcomponents is give here: :doc:`algorithms/audio_classification_preprocess_infer`. An ``algorithm.yml`` configuration file for this algorithm is the following:: algorithm: com_bonseyes/ai_algorithms#audio_classification_preprocess_infer subcomponents: preprocessing: parameters: dump_tile: audio_dump.json filter: mfcc output_size: y: 40 x: 32 numMels: 128 numFFT: 2048 freq: low: 0 high: 8000 winLength: 2048 frameShift: 512 db: true htk: false inverted: false clipINT8: false mean_data: 0 std_data: 1 melNorm: None inference: parameters: model: model.yml challenge: com_bonseyes/challenges/mychallenge#challenge.yml Signal Classification using a Neutral Network with signal pre-processing ======================================================================== The specification of this algorithm and their subcomponents is give here: :doc:`algorithms/signal_classification_preprocess_infer`. A mininal ``algorithm.yml`` configuration file for this algorithm is the following:: algorithm: com_bonseyes/ai_algorithms#signal_classification_preprocess_infer subcomponents: inference: parameters: model: model.yml preprocessing: parameters: dump_tile: signal_dump.json output_size: x: 1 y: 256 challenge: com_bonseyes/challenges/mychallenge#challenge.yml An example algotithm YML file is also available in the `3D Face Landmarks AI Asset `_. Deployment ~~~~~~~~~~ The deployment folder contains ``deployment.yml`` files with the AI App’s deployment configuration. A deployment configuration YAML file specifies certain parameters for an algorithm implementation, e.g., what inference engine to use for the executation of the neural network, the data type (FP32, FP16, INT8) or the backends to use per layer. Several *deployment.yml* files can be stored under *deployment/* for different configurations. The schema of the supported deployment configurations are described below: .. jsonschema:: ../../../schemas/core/deployment_config.yml A minimal ``deployment.yml`` file looks like:: ai_app_name: "MY-APPLICATION" ai_app_version: "v0.1" ai_app_metadata: title: Image classification author: email: hello@bonseyes.com name: Bonseyes Community Association description: | This AI app analyzes an image and returns the probability that the image belongs to any given class. extra: - id: com_bonseyes/marketplace/metadata#tags value: - bonseyes - image - computer vision - cnn - lpdnn license: Bonseyes Consortium License component: com_bonseyes/lpdnn#core/components/image_classification_preprocess_infer subcomponents: inference: component: com_bonseyes/lpdnn#engines/lne/components/network_processor parameters: optimizations: [all] data_type: F32 The main subcomponent in the deployment YAML file is the ``inference`` subcomponent, which defines the network_processor to use, i.e., inference engine to execute the neural network, in the ``component type`` field and the characteristic in the ``parameters`` field. Next, we specify each of the possible components that can be used for inference. Deployment engines ------------------ As explained in LDPNN's :ref:`lpdnnEngines`, LPDNN supports the following network processors (inference engines), each of these allowing different configurations: LNE ^^^ .. jsonschema:: network_processors/network_engine_lne.yml TensorRT ^^^^^^^^ .. jsonschema:: network_processors/network_engine_trt.yml NCNN ^^^^ .. jsonschema:: network_processors/network_engine_ncnn.yml ONNXruntime ^^^^^^^^^^^ .. jsonschema:: network_processors/network_engine_ort.yml A template of the deployment YML file is available in the `AI Asset Container Generator `_. An example deployment YML file is also available in the `3D Face Landmarks AI Asset `_. Deployment packages ~~~~~~~~~~~~~~~~~~~ As explained in LDPNN's :ref:`deploymentPackages`, an LPDNN Deployment Package is a collection of tools, executables, libraries, inference engines and backends that allows the actual execution of the LPDNN AI application. The collection of libraries and binaries that are copied on the target platform for the execution of the DNN is called a *runtime*. The runtime dinamically loads an AI application and executes it based on its defined configuration. LPDNN's deployment packages are platform-specific as they contain the inferences engines and backends supported by the HW platform. Deployment packages, including LPDNN runtimes, can be obtained by asking access to one of the BonsAPPS's partner. LPDNN's deployment packages can be found `here `__. A template of where the deployment packages should be placed is available in the `AI Asset Container Generator `_. An example containing deployment packages is also available in the `3D Face Landmarks AI Asset `_. The script *get_packages.sh* allows the user to donwload deployment packages by executing: .. code-block:: bash bash get_packages.sh ${platformName} # For instance bash get_packages.sh jetson_nano-jetpack4.6 ****************** Generate an AI App ****************** Prerequisites ~~~~~~~~~~~~~ This section assumes: - All the steps in the *create an LPDNN's file tree* section have been completed. - The local environment has been set up as explained in :doc:`/pages/setup` Generation ~~~~~~~~~~ There are two methods to generate an AI App: - **AI App generator script**: Python script that produces an AI App. - **Bonseyes CLI**: Main command line interface tool for Bonseyes tools and workflows. AI App generator script ----------------------- To use this script the following steps need to be performed: 1. Change directory to the LPDNN's directory, e.g., `LPDNN root directory in AI Asset `_. 2. Create a build folder: .. code-block:: bash mkdir build 3. Execute the following command: .. code-block:: bash deployment_packages/${platformName}/tools/ai_app_generator/bin/ai-app-generator \ --algorithm-file catalog/${modelName-version-size-precision}/algorithm.yml \ --challenge-file challenge/${challengeName}/challenge.yml \ --deployment-file deployment/${modelName-version-size-precision}/deployment.yml \ --output-dir build/${new-aiapp-name} For instance, to generate an AI App for the model *mobilenet-v2-224x224-fp32* (imagenet challenge) on *raspberry4b_64-ubuntu20*: .. code-block:: bash deployment_packages/raspberry4b_64-ubuntu20/tools/ai_app_generator/bin/ai-app-generator \ --algorithm-file catalog/mobilenet-v2-224x224-fp32/algorithm.yml \ --challenge-file challenge/imagenet/challenge.yml \ --deployment-file deployment/mobilenet-v2-224x224-fp32/deployment.yml \ --output-dir build/mobilenet-v2-224x224-fp32 After executing the command a new LPDNN AI App will be present in the build folder with the given name. A number of file will be included in the AI App folder among which the most important ones are described at :ref:`aiAppFiles`. To regenerate the AI App with the same name, the flag ``--force`` can be used. Beware as this flag will first erase the previous AI App to then compile a new one. Bonseyes CLI ------------ It is also possible to build AI Apps with the Bonseyes CLI tool. To do so, this generation option includes an extra prerequite step: - Download and build the sources of the target platform's DPE as explained in the platform's :doc:`/pages/user_guides/platform_index` in the user guide. - Have the ${platformName_source} and ${platformName_build} folders in lpdnn repository. To generate an AI App with the Bonseyes CLI tool, execute the following commands: 1. Change directory to the LPDNN's directory, e.g., `LPDNN root directory in AI Asset `_. 2. Copy you target platform's sources and build directories to LPDNN root directory: .. code-block:: bash cp /Path/to/{platformName}_source /Path/to/{platformName}_build . 3. Create a build folder: .. code-block:: bash mkdir build 4. Execute the following command: .. code-block:: bash bonseyes --packages-dir . ai-app generate \ --algorithm-config catalog/${modelName-version-size-precision}/algorithm.yml \ --challenge challenge/${challengeName}/challenge.yml \ --deployment-config deployment/${modelName-version-size-precision}/deployment.yml \ --deployment-tool deployment_packages/${platformName}/deployment_tool.yml build/${new-aiapp-name} For instance, to generate an AI App for the model *mobilenet-v2-224x224-fp32* (imagenet challenge) on *raspberry4b_64-ubuntu20*: .. code-block:: bash bonseyes --packages-dir . ai-app generate \ --algorithm-config catalog/mobilenet-v2-224x224-fp32/algorithm.yml \ --challenge challenge/imagenet/challenge.yml \ --deployment-config deployment/mobilenet-v2-224x224-fp32/deployment.yml \ --deployment-tool deployment_packages/raspberry4b_64-ubuntu20/deployment_tool.yml \ build/mobilenet-v2-224x224-fp32 After executing the command a new LPDNN AI App will be present in the build folder with the given name. A number of file will be included in the AI App folder among which the most important ones are described at :ref:`aiAppFiles`. Development ~~~~~~~~~~~ If the generation of an LPDNN AI App is not possible due to not supported features such as: - Not supported operator by the targeted inference engine Please, you may try generate the AI App with another inference engine for your target device. If the generation of an LPDNN AI App is not possible due to not supported features such as: - Specific pre-processing / post-processing not existing. - Not supported operator by the target inference engine (LNE only). - AI Class not existing, e.g., signal regression. Please, follow the dev guidelines for LPDNN's :doc:`/pages/dev_guides/lpdnn_index` to get started with LPDNN development in LPDNN's SDK. ******************************** Execute an AI App on host device ******************************** Execution of FP32 AI Apps ~~~~~~~~~~~~~~~~~~~~~~~~~ As explained in LDPNN's :ref:`deploymentPackages`, two elements are need to execute an AI Application on a device: - **An LPDNN AI application**: Created in previous steps. - **An LPDNN Deployment Package**: Donwloaded in previous steps. To execute an FP32 AI Apps on your host, you can use the aiapp-cli (standalone executable) that allows to easily execute an aiapp from the command line. It receives as input parameters the aiapp configuration file and the input file(s) to be processed. It will internally instantiate the correct aiapp class according to the specified configuration file and generate human-readable result. To execute the AI App, follow the next steps: 1. The executable is available under bin/. Change directory to the deployment package binaries that match you current device: .. code-block:: bash cd deployment_packages/${platformName}/bin # For instance: cd deployment_packages/x86_64-ubuntu20_cuda/bin 2. Set the library's and python's path: .. code-block:: bash source set-lib-path.sh 3. Infer an AI App on an input sample: .. code-block:: bash ./aiapp-cli --cfg ../../../build/${new-aiapp-name}/ai_app_config.json -f /Path/To/Input/File{.png/.wav/.json} # For instance: ./aiapp-cli --cfg ../../../build/mobilenet-v2-224x224-fp32-cli/ai_app_config.json -f /Path/To/Input/File{.png/.wav/.json} An example of a python-based AI APP cli (aiapp-cli.py) with the same execution parameters is also available in all LPDNN's deployment_package under the *bin/* folder. Processing options ~~~~~~~~~~~~~~~~~~ Input format ------------ The input file can be a .jpg/png format for image-based AI Apps or .wav for audio files. However, it is also possible to pass an input tile in JSON format that may contain any kind of data. The JSON file must have the following structute: .. code-block:: bash { "data":[1.5883,0.272,0.89910,...,...], # Modify based on Input "dim":[1,3,224,224], # Modify based on AI App "info":"NCHW" # Currently only NCHW input layout is supported } JSON input tiles are not pre-processed by LPDNN, i.e., pre-processing step is skipped, as they usually they serve to concatenate several AI App by feeding the results of the fist one into the input of the second one. ``Note: Currently only supported input layout is NCHW`` Split pre-proceesing and inference ---------------------------------- If the file provided in input to an AI App has .json suffix, it is considered as a tile, so no preprocessing is done and the data are sent directly to inference. The ``--out-tile`` option allows to generate the tile (that is the output of the preprocessor) in a JSON file as an AI App blob, that is, a flattened data array of floats plus a dim array representing the actual dimensions. This makes it easy to perform preprocessing and inference in two separate stages if needed. Additional input ---------------- Extra input ^^^^^^^^^^^ An given AI App may require additional information, e.g., region-of-interest or landmarks, as an input. This information can be provided in input to aiapp-cli by sending the extra input as stdin: .. code-block:: bash cat landmarks.json | ./aiapp-cli --cfg ../../../build/${new-aiapp-name}/ai_app_config.json -f /Path/To/Input/File To generate a file such as *landmarks.json* from a first AI App, you may execute it with the ``--out-inference`` parameter to obtain a JSON file with the results: .. code-block:: bash ./aiapp-cli --cfg ../../../build/${new-aiapp-name}/ai_app_config.json -f /Path/To/Input/File --out-inference landmarks.json Multiple inputs ^^^^^^^^^^^^^^^ Some AI Apps, however, require multiple inputs, some of which are fed directly from the output of the same AI App (temporal networks). If a model supports multiple inputs, e.g., `tsm-optimized for gesture-recognition `_, you can use the ``--additional-inputs `` argument to specify a JSON file that contains the additional values. Values in the JSON must use the same format of the file produced by ``--out-inference``. Input order is defined in the `algorithm.yml` of the model, using the `inference/parameter/input_layers` field. When creating manually a JSON file to pass addtional inputs, an empty "blob" must be added at the beginning of the array. For example if you have 2 additional inputs, the JSON array should contains this: [empty blob, \, \] This is done because the output of ``--out-inference`` includes both default output and additional outputs ([default, additional1, additional2, ...]) and by doing it this way, it is possible to directly feed the produced JSON back to the model without having to change it, which is useful for some models. Note that if a model has multiple output (also specified in the `algorithm.yml` file, they can be retrieved in the file generated with ``--out-inference``). .. code-block:: bash gesture_app=../share/ai-apps/tsm-optimized/ai_app_config.json # 1. Run the app without additional inputs the first time ./aiapp-cli --cfg $gesture_app -f $image --out-inference previous_output.json # 2. Run the app and pass previous output as additional input the second time ./aiapp-cli --cfg $gesture_app -f $image --additional-inputs previous_output.json Concatenation of two AI App --------------------------- Two AI Apps can be concatenated by redirecting the output of the first AI App into the second AI App. Bash ^^^^ Pre-built AI Apps can be concatenated through bash calls. The output logs of the first AI App are not displayed and the result is generated in json format, which is fed into the following AI App as stdin: .. code-block:: bash ./aiapp-cli --cfg $face_detection_aiapp -f $image | ./item -i 0 | \ ./aiapp-cli --cfg $emotion_aiapp -f $image Please, note the use of the *./item* utility script in the commands above. Both face detection and landmark detection aiapps generate in output a list of items (one per face), where each item contains a bounding box and possibly landmarks. Emotion aiapp is instead able to process one face at a time, so it takes in stdin information regarding the bounding box and landmarks of a single face. We thus need a way to select which face to process. ./item -i 0 simply reads the output results of the first AI App and feed an item (by index) to second AI App.. If needed it is easy to provide different selector scripts with different behaviour, for example to select the biggest face, or the more central one, or the one with the highest confidence. Python ^^^^^^ AI Apps can also be concatenated in python to create a single more complex application as follows: .. code-block:: bash # Read file with open(sample_file, 'rb') as fp: binary_data = fp.read() # Create image sample = aiapp.encoded_sample(binary_data) # Execute Face Detection AI App res = aiapp.execute(sample, {}) for item in res.items: # Set ROI and landmarks to input image sample.roi = item.bounding_box sample.landmarks = item.landmarks # Execute Emotion AI App res = aiapp.execute(sample, {}) An example of AI APP cli (aiapp-cli.py) is available in all LPDNN's deployment_package under the *bin/* folder. C++ ^^^ During development, C++ code can also be concatenated to create a single built-in more complex application as follows: .. code-block:: bash # Read file auto image_content = app::read_binary_file(filename); # Create image auto image = ai_app::Image(ai_app::Image::Format::encoded, image_content, {}); # Execute Face Detection AI App auto results = aiapp_facedet->execute(image); for (auto item : result.items) { # Set ROI and landmarks to input image app::object_detection_item_to_image(item, image); # Execute Emotion AI App auto results = aiapp_emotion->execute(image); } Other options ------------- For more options about inferring an AI App, execute: .. code-block:: bash ./aiapp-cli -h --cfg Ai-app configuration file --info Show aiapp information -f Sample file(s) to process -d Sample directory to process -p Generate metrics file -l Generate layer dump file --out-tile Generate preprocessed tile in json format --preprocess-only Preprocessing only (skip inference) --out-inference Generate raw inference out in json format --additional-inputs Path to JSON generated with --out-inference, provides values for additional model inputs (lpdnn/image detection only) -r Repeat inference N times -rr Repeat creation and inference N times --help Show this help Quantisation workflow for LPDNN's engines ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ LPDNN curently supports post-training quantization (PTQ) for its inference engines. To execute quantized models, some pre-steps need to be followed. TensorRT -------- TensorRT supports FP16 and INT8 quantization. FP16 ^^^^ For FP16 deployment, the following steps need to be performed: 1. Change the ``subcomponent.inference.parameters.data_type`` field to *F16* in the deployment-file.yml. ``Note: Data type in LPDNN is F16, not FP16`` 2. Generate again the target AI App using the new deployment YAML file. 3. Execute the AI App as explained in the previous section. INT8 ^^^^ TensorRT provides it's own calibration tool for INT8 deployment. To use the TensorRT quantization tool, perform the following steps: 1. Prepare a calibration dataset to adjust the DNN's activations' range by calculating the activation' scale and offset in order to retain a high amount of accuracy during the quantization process. - We need to obtain pre-processed tiles to make sure the neural network's calibration process ontains the real range that network *sees* during inferece. - To obtain the pre-processed tiles, have a FP32 ai-app generated following the steps explained in previous sections. - Run the preprocessing-only mode of the aiapp-cli executable for the AI App (compiled with any engine). .. code-block:: bash cd deployment_packages/${platformName}/bin source set-lib-path.sh ./aiapp-cli --cfg ../../../build/${aiapp-name}/ai_app_config.json --preprocess-only --out-tile -f - Make sure to prepare enough representative tiles for the calibration set. 2. Make a text file, e.g., cfile.txt, containing the paths, line by line, toward the tiles generated in the previous step. 3. In the deployment-file.yml for tensorRT engine, add ``qcalibrator``, ``cfile``, ``cbatch_size``, and ``cbatches`` fields to the ``subcomponent.inference.parameters``. 4. Set *qcalibrator* to *true*, set *cfile* to the absolute path of the file containing the tile paths (without ""). 5. Set *cbatch_size* to the desired calibration batch size, e.g., 1. 6. Set *cbatches* to the number of calibration batches provided the data, e.g., if 100 tiles are available and a cbatch_size of 10 is chosen, there will be 10 cbatches. 7. Set *data_type* field to *S8*. 8. Generate again the target AI App using the new deployment YAML file. 9. Execute again the AI App. TensorRT will perform calibration process that might last some time. Some logs such as the following should appear: .. code-block:: bash [Trt::CreateEngine@73] Quantisation using TensorRT calibrator. .. Starting Calibration. Calibrated batch 0 in 0.0979145 seconds. Post Processing Calibration data in 1.12499 seconds. Calibration completed in 3.09242 seconds. Writing Calibration Cache for calibrator: TRT-7103-EntropyCalibration2 10. Upon a successful calibration, the platform-independent calibration results will be saved in *qfile.txt* in the AI App folder. 11. One can reuse the obtained results for the given AI App as a quantisation cache for building TensorRT engines on different target platforms without redoing the calibration process. To do so, add the path to the *qfile.txt* in deployment-file.yml's ``subcomponent.inference.parameters.qfile`` field before generating the AI App. If so, *cbatch_size*, and *cbatch_size* fields can be removed from the deployment file. ONNXruntime ----------- ONNXruntme supports INT8 quantization. INT8 ^^^^ LPDNN's onnxruntime engine is able to execute quantized INT8 ONNX models directly. Hence, the only neeed steps are the following: 1. Quantize your model in the AI Asset as explained in :ref:`onnxQuantize`. 2. Make sure to have the quantized ONNX model in ``catalog/${modelName-version-size-precision}/blobs`` and the right path to it on the ``catalog/${modelName-version-size-precision}/model.yml``. 3. In the deployment-file.yml, set *data_type* field to *S8* under ``subcomponent.inference.parameters``. 4. Generate the AI App as explained in previous sections. 5. A quantized AI App is ready to be executed. NCNN ---- NCNN supports FP16 and INT8 quantization. FP16 ^^^^ For FP16 deployment, the following steps need to be performed: 1. Change the ``subcomponent.inference.parameters.data_type`` field to *F16* in the deployment-file.yml. ``Note: Data type in LPDNN is F16, not FP16`` 2. Generate again the target AI App using the new deployment YAML file. 3. Execute the AI App as explained in the previous section. INT8 ^^^^ NCNN provides its own `calibration tool `_ to calculate the activation ranges against a valiadation dataset. LPDNN provides a pre-compiled ncnn2table script: 1. Have a pre-compiled FP32 model using NCNN engine as explained in previous sections. 2. Have a calibration dataset and generate a quantisation file as follows: .. code-block:: bash ./deployment_packages/${platformName}/tools/ncnn/ncnn2table --param=build/${aiapp-name}/model.param --bin=build/${aiapp-name}/model.bin --images=PATH/TO/IMAGES_DIR --output=build/${aiapp-name}/quant.table --mean=0,0,0 # Modify --norm=1,1,1 # Modify --size=224,224 # Modify --thread=8 3. Include the path to the quantisation file: *quant.table* in deployment-file.yml's ``subcomponent.inference.parameters.qfile`` field and set data_type field to *S8*. 4. Generate again the target AI App, using the new deployment YAML file and a new name for the AI App (Avoid using --force as it will delete the quant.table). 5. Execute the AI App as explained in the previous section. LNE --- LNE supports INT8 quantization for both S8 and U8, symmetric and asymmetric types. INT8 ^^^^ LPDNN provides a calibration tool to calculate the activations ranges against a validation dataset. To quantized a model with LNE, execute the following steps: 1. Execute the data extraction tool to dump the activations ranges against a validation dataset on a calibration.json file: .. code-block:: bash python3 deployment_packages/${platformName}/tools/lib/lpdnn_onnx/extract_quantization_onnx.py -m catalog/${modelName-version-size-precision}/blobs/${modelName-version-size-precision}.onnx --mean-pixel 0 0 0 # Modify --std-pixel 1 # Modify -i /Path/To/Calibration/Images/ --out-table build/${aiapp-name}/calibration.json 2. Include the path to the *calibration JSON file* in the deployment file in deployment-file.yml's ``subcomponent.inference.parameters.qfile`` field. 3. Choose the data type. LNE currently supports U8 data type for ARM CPU and S8 for Nvidia GPU. 4. Generate again the target AI App, using the new deployment YAML file and a new name for the AI App (Avoid using --force as it will delete the calibration.json). 5. Execute the AI App as explained in the previous section. *********************************************** Deploy and benchmark an AI App on target device *********************************************** For the deployment and benchmarking of an AI App on a target device, i.e., not the host where the AI App has been generated, the follwing step is required: - Setup the target hardware as explained in :ref:`setupPlatform` section in the user guide and have a ${platformName_config}. Bonseyes CLI benchmark ~~~~~~~~~~~~~~~~~~~~~~ If automated benchmarking is desired, please refer to: :ref:`deployApp` section in the user guide to execute a benchmark with the Bonseyes CLI tool. Custom deployment and benchmark ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If custom deployment is desired, the AI App's HTTP-worker interface can be used to deploy an AI App on a target device while posting samples, e.g., images, from your host PC. The http-worker can be found in all LPDNN's deployment_packages under the *bin/* folder and is started as follows: .. code-block:: bash ./http-worker --cfg ../../../build/${new-aiapp-name}/ai_app_config.json -s ${IP_ADDR} -p ${Port} Host side --------- Bash ^^^^ It is then possible to connect to the worker from the host as follows (path to the image must contain the character "@" before the path): .. code-block:: bash curl --request POST --data-binary @/PATH/TO/FILE ${IP_ADDR}:${Port}/inference It is also possible to generate execution metrics or specify additional parameters (such as bounding box or landmarks) by adding the corresponding fields to the header: .. code-block:: bash curl --request POST --data-binary @/PATH/TO/FILE --header 'x-metrics: all' --header "x-params: $(cat landmarks.json)" ${IP_ADDR}:${Port}/inference Python ^^^^^^ It is also possible to connect to the http-worker from a python interface: .. code-block:: bash url = 'http://' + '${IP_ADDR}' + ':' + '${Port}' + '/inference' headers = {'x-metrics': 'all'} headers = {'x-params': json.dumps({"bounding_box":{"origin":{"x":347,"y":113},"size":{"x":295,"y":405}}})} with open(input_sample, 'rb') as fp: with requests.post(url, data=fp.read(), headers=headers) as req: if req.status_code != 200: raise Exception('Error returned by the server worker') # parse the report report = req.json() req.close() print(report) To change the type of input, e.g., rgb or input tile, the header should add the folliwing line: .. code-block:: bash # Input tile input headers = {'x-in-type': "tile-json"} # RGB input headers = {'x-in-type': "raw_rgb8"} A complete python script, called http-client.py, for posting input samples to the http-worker is provided within each LPDNN's deployment_package under the *bin/* folder. ************* Upload AI App ************* **THIS SECTIONS IS IN PROCESS** How to upload an AI app to the marketplace ========================================== This guide assumes you already have the done the following: - Setup the local environment as explained in :doc:`/pages/setup` - Created an AI app with the procedure described in :doc:`generate_ai_app` - You have the deployment tool used to generate the AI app in the directory `deployment_tool` - Created a git repository on Gitlab that will host the AI app - Joined on the marketplace the challenge that the AI app is solving - Enrolled your gitlab user in the marketplace To upload an AI app you need to do the following: 1. Create your author keys:: $ bonseyes license create-author-keys --output-dir author_keys 2. Package your ai-app for release with its dependencies and a redistribution licenses:: $ bonseyes --package deployment_tool/runtime marketplace package-ai-app \ --pull-images \ --ai-app ai_app \ --author-keys author_keys \ --amount "100 CHF" \ --recipients "test@dd.com,tost@dd.com" \ --regions https://www.wikidata.org/wiki/Q39,https://www.wikidata.org/wiki/Q40 \ --output-dir packaged_ai_app The ``--recipients`` parameter can be used to restrict who can see the artifact on the marketplace, if omitted all users can see it. The ``--regions`` parameter can be used to restrict the regions in which the artifact can be seen, if omitted users of all regions can see it. 3. Commit the packaged AI app in a gitlab repository that is accessible by the gitlab user you enabled in the marketplace 4. Upload the AI app to the marketplace:: $ bonseyes marketplace upload-ai-app --url $HTTP_URL_OF_GITLAB_REPO The AI app will be added to your profile. ************************************************** Create challenge-compliant interface for an AI App ************************************************** LPDNN AI Apps might not be compliant with the specification of an AI Challenge published in the BMP. Therefore, a C++ interface needs to be use to parse the output results of the LPDNN AI App to the expected results specified in the AI Challenge. To do so, the following steps need to be followed: - **Create LPDNN Challenge** that includes the main LPDNN challenge files - **Create a header file** (.hpp) that includes the AI Challenge c++ interface and parses the AI App's results - **Create a CLI example** that takes an built AI App and produces AI Challenge compliant results Create LPDNN Challenge ~~~~~~~~~~~~~~~~~~~~~~ Create a Gitlab repo for the challenge -------------------------------------- Each challenge needs to have its own repository in GitLab in `Challenges-LPDNN `_. .. figure:: ../assets/challenge_creation.png :alt: Creation of a challenge repository in Gitlab :figclass: align-center LPDNN full stack Creation of a challenge repository in Gitlab **Naming Convention for Challenge Repository:** The challenge repository name should follow a naming convention which is:: OrganizationAcronym-ChallengeName-ChallengeAcronym where: **OrganizationAcronym:** Organization to which the challenge belongs. For all BonsApps challenges, this acronym will be “BMP” **ChallengeName:** Name of the challenge. If the challenge name has multiple words each word should start with a capital letter with a hyphen (“-“) in between the words. **ChallengeAcronym:** This last part can be used as a distinguisher for challenges in case they belong to the same organization and have the same names (multiple challenges with the same name). In such cases this acronym serves as the challenge ID, e.g., OC1, OC2. Add high-level challenge as a submodule --------------------------------------- The high-level challenge needs to be added as a submodule so as to connect the high-level specifications with the low-level ones. To add the high-level challenge as a submodule, execute a command such as:: $ git submodule add git@gitlab.com:bonseyes/artifacts/challenges/NameOfChallenge.git requirements Add LPDNN challenge files ------------------------- Follow BMP’s :ref:`lpdnnChallenge` documentation to create the challenge.yml file that is needed for the creation of an AI App. In addition, files such as classes.txt or landmarks.txt might also be needed, depending on the Challenge AI class. A readme file and some test samples should be also included in this repository, following this example. Create header file ~~~~~~~~~~~~~~~~~~ The objective of the header file is to parses the AI App's results and make them compliant with the AI Challenge specifications. To do so, the c++ interface's header file, from the requirements submodule, needs to be included. A template for the header file, including the main methods, is available `here `_. To complete the header file, the following steps are required: 1. Replace "LPDNN_AIAPP_CLASS" by the LDPNN AI class that the AI App is built upon, which is defined in the *challenge.yml* under *ID (interface)*. For instance, *image_classification or object_detection* 2. Replace "CHALLENGENAME_Result" by the Challenge Interface header file name. 3. Add code your code to parse the AI App results into *Result_Challenge(), challenge_interface() and to_json_str()* methods. Any other method might also be modified or added if needed. An example of a header file for 2D Bodypose Estimation is given `here `_. Create CLI example ~~~~~~~~~~~~~~~~~~ A working CLI example in the LPDNN-Challenge needs to be created to showcase the AI App. To do so, follow next steps: 1. Create *example* folder in the LPDNN-Challenge folder, which contains *include* and *test* folders. A *src* folder might also be created. 2. Add the header file created in the previous step into the header folder. 3. Add *main.cpp* file into the test folder. A template of such a main.cpp file is available `here `_. Modify *CHALLENGENAME* and Aiapp_NAME by the corresponding names that are used in the header file. 4. Add *CmakeLists.txt* file in the *example* folder. A template for a CmakeLists.txt file is available `here https://gitlab.com/bonseyes/bonsapps/1st-support-programme/qa/-/blob/main/templates/interface_c%2B%2B/CMakeLists.txt>`_. Replace *AIAPPNAME-cli* and *aiapp_CHALLENGENAME* by the corresponding names. To build and run the CLI example, follow next steps: 1. Create build folder and change directory into it: .. code-block:: bash mkdir build cd build 2. Compile CLI example, giving the deployment package of the host machine: .. code-block:: bash cmake .. -DLPDNN_DEPLOYMENT_PACKAGE=/PATH/TO/{deployment_package} make 3. Run the CLI example: .. code-block:: bash ./EXAMPLENAME-cli --cfg /PATH/TO/AIAPP//ai_app_config.json -f /PATH/TO/INPUT/SAMPLE