AI Runtime¶

The NPU driver, inference core, operators and post-processing, and the detection and speech model interfaces.

hal_npu.h¶

#include "hal_npu.h" (Source)

NPU hardware access for IPRO7: clock and bus set-up, layer buffers, network parameters, start, stop and interrupt acknowledge.

These functions program the NPU registers directly. They are used by the AI runtime, which drives the NPU layer by layer; most applications use the runtime API instead of calling them. A typical sequence is: NPU_Clk_Div() and NPU_Bus_Setting() once; for each model or layer, hal_npu_layer_setup(), hal_npu_set_input_buffer() and hal_npu_set_net_param(); then hal_npu_clk_enable(1) and hal_npu_start(), and wait for the NPU interrupt, whose handler calls hal_npu_ack_irq().

NPU_Clk_Div¶

void NPU_Clk_Div(int n);

Select the NPU clock source and set its divider.

NPU_Bus_Setting¶

void NPU_Bus_Setting(void);

Apply the default NPU bus settings.

Sets the read and write outstanding-transaction limits to 3 and the read and write command lengths to 63.

hal_npu_clk_enable¶

void hal_npu_clk_enable(uint8_t enable);

Enable or disable the NPU clock.

Enabling ungates the multimedia bus clock. Disabling performs no hardware action in this release.

hal_npu_ack_irq¶

void hal_npu_ack_irq(void);

Acknowledge (clear) the NPU interrupt.

Call it from the NPU interrupt handler, and before enabling the interrupt to discard a stale request.

hal_npu_layer_setup¶

void hal_npu_layer_setup(uint8_t *inst_buf, uint8_t *weights_buf, uint8_t *bias_buf, uint8_t *mult_shift_buf);

Set the buffers of the next NPU layer group.

The buffers come from the compiled model and must be readable by the NPU.

hal_npu_set_input_buffer¶

void hal_npu_set_input_buffer(uint8_t *input_buf, int patch_size);

Set the NPU input (working) buffer.

hal_npu_set_net_param¶

void hal_npu_set_net_param(uint8_t unsign_input, uint8_t relu_n, uint8_t use_tflite, uint32_t mode);

Set the network-wide NPU parameters.

hal_npu_reset_unsign¶

void hal_npu_reset_unsign(void);

Clear the unsigned-input flag set by hal_npu_set_net_param().

Used for the layer groups after the first one, whose inputs are signed.

hal_npu_start¶

void hal_npu_start(void);

Start the NPU, or resume it after it has halted.

The first call after reset or hal_npu_stop() starts execution; later calls resume it. The NPU raises its interrupt when it halts.

hal_npu_stop¶

void hal_npu_stop(void);

Stop the NPU.

The next hal_npu_start() issues a start instead of a resume.

NPU_Img_Ext_Addr_Cfg¶

void NPU_Img_Ext_Addr_Cfg(uint32_t addr);

Set the external image input address.

NPU_Img_Ext_Addr_Get¶

uint32_t NPU_Img_Ext_Addr_Get(void);

Get the external image input address.

Returns: The address set by NPU_Img_Ext_Addr_Cfg().

NPU_Bus_Delay¶

void NPU_Bus_Delay(uint8_t enable, uint8_t rcmd, uint8_t wcmd);

Set the NPU bus read and write command lengths.

NPU_SRAM_EN¶

void NPU_SRAM_EN();

Set the NPU enable bit in the SoC SRAM configuration register.

Typedefs¶

iproai_def.h¶

#include "iproai_def.h" (Source)

AI engine types: model handle, network and layer descriptions, results, callbacks.

These definitions are shared by the AI engine library and applications. Read them together with iproai_core.h, which declares the functions that create a model, load it, run inference and release it.

After a model is loaded, iproai_getNetInfo() returns its struct iproai_net_info_t. The layers array of that structure holds one struct cpu_inst_layer_t per layer, and output_layer lists the layers whose outputs are the model outputs.

All tensors of a model live in one working buffer, returned by iproai_getInputBuffer(). The buffer is divided into patches of patch_size bytes and every tensor starts at a patch boundary: the output of layer l starts at byte offset l->out_layer_mem * net->patch_size. Tensors are stored in height, width, channel order. A layer that runs on the NPU (NPU_on) stores its output with the channel count rounded up to a multiple of 4; the padding bytes carry no data.

For a TensorFlow Lite model (use_tflite), an output byte q, read as unsigned, has the real value output_scale * (q - tf_output_offset):

struct iproai_net_info_t *net = iproai_getNetInfo(hdl);
const struct cpu_inst_layer_t *l = &net->layers[net->output_layer[0]];
const uint8_t *base = iproai_getInputBuffer(hdl);
const uint8_t *q = base + (uint32_t)l->out_layer_mem * (uint32_t)net->patch_size;
float v0 = l->output_scale * ((int)q[0] - l->tf_output_offset);

IPROAI_Device_Type_e¶

Compute device identifiers, stored in IPROAI_Model_t::dev (not read by the library).

IPROAI_Feature_Level_e¶

Engine feature levels, stored in IPROAI_Model_t::level (not read by the library).

IPROAI_Padding_Mode_e¶

Convolution padding modes, with TensorFlow naming.

Not used by the current library.

IPROAI_Status_e¶

Status codes returned by the AI engine functions.

IPROAI_ProcStatus_e¶

States of the inference loop run by iproai_startCompute().

The loop alternates between CPU stages and NPU stages until all layers have run.

LAYER_TYPE¶

Layer operators, stored in cpu_inst_layer_t::type.

Several types are fused forms produced by the model converter (for example CONV_MAX). Whether a layer runs on the NPU or on the CPU is given by cpu_inst_layer_t::NPU_on and cpu_inst_layer_t::DSP_on, not by its type.

APPLICATION_TYPE¶

Application types, stored in iproai_net_info_t::app_type and iproai_result_t::app_type.

The model converter records the type in the model header. At the end of iproai_startCompute() the engine runs the built-in post-processing for that type, if the library includes it, and reports results through the callback set with iproai_setResultCB(). Built-in post-processing is present only when the library is built with CONFIG_AI_LEGACY_YOLO (for OBJECT_DETECTION and FACE_LANDMARK) or CONFIG_AI_LEGACY_POSTPROCESS (for the other types that have one). Otherwise read the output tensors directly, as shown in the description of iproai_def.h.

cpu_inst_layer_t¶

Description of one layer of a loaded model.

The engine fills these records when a model is loaded; applications read them through iproai_net_info_t::layers. Tensor positions are patch indices into the working buffer of the model (see the description of iproai_def.h). Fields not documented here are internal to the engine.

iproai_net_info_t¶

Description of a loaded model, returned by iproai_getNetInfo().

iproai_create() allocates the structure, the load functions fill it, and it stays valid until iproai_free(). Applications read it and must not modify it, except for the fields that say so. Fields not documented here are internal to the engine.

iproai_model_info_t¶

Model sections copied from a model blob, with the time spent in the last inference.

A model blob holds eight sections: instruction records, biases, weights, and requantization multipliers and shifts, once for the CPU and once for the NPU. The load functions copy every non-empty section into its own heap buffer; the pointer of an empty section is NULL.

iproai_result_t¶

Result reported by the built-in post-processing through iproai_inference_cb.

app_type selects the member of r that holds the result:

• OBJECT_DETECTION, RETINA_FACE, RETINA_PERSON, RETINA: r.objdet (RETINA_FACE also fills the landmark points of r.landmark);
• CLASSIFICATION, KEYWORD_SPOTTING: r.classification;
• FACE_RECONGNITION: r.fr;
• FACE_LANDMARK: r.landmark.

An SSD layer reports each detection with app_type set to OBJECT_DETECTION. r.objdet and r.landmark start with the same members, so best_class, prob and rect can be read through either. Coordinates are in pixels of the source resolution set with iproai_setSourceResolution().

IPROAI_Model_t¶

Model instance behind an iproai_model_hdl_t.

iproai_create() allocates it, the load functions fill it and iproai_free() releases it. Applications should use the iproai_* functions instead of accessing this structure.

Typedefs¶

Macros¶

iproai_core.h¶

#include "iproai_core.h" (Source)

AI engine entry points: create a model, load it, run inference, release it.

A model is a blob produced by the model converter. It starts with a 32-byte table of eight 32-bit section sizes, so its total size is 32 bytes plus their sum. The types used here, and the way output tensors are laid out, are described in iproai_def.h.

Typical sequence:

1. Ungate the multimedia clock domain that contains the NPU (GLB_PER_Clock_UnGate(GLB_AHB_CLOCK_MM)).
2. For a model that reads its input from an external address (ext_input), point the global imgBuffer (hal_npu.h) at a buffer whose data starts at imgBuffer + 4.
3. Call iproai_create(), then iproai_load_model_from_buffer() or iproai_load_model_from_file().
4. Optionally call iproai_setResultCB() and iproai_setSourceResolution() for the built-in post-processing.
5. For each input: write it into the working buffer (iproai_getInputBuffer()) or set its address with NPU_Img_Ext_Addr_Cfg(), call iproai_startCompute(), then read the output tensors.
6. Call iproai_free().

The engine allocates its memory from the FreeRTOS heap. Its functions are not thread-safe: make all calls for a model from one task and run one inference at a time. None of them may be called from an interrupt handler. Apart from iproai_load_model_from_file() and iproai_startCompute(), the functions do not check their handle or pointer arguments for NULL.

iproai_create¶

iproai_model_hdl_t iproai_create(void);

Create a model instance.

Allocates an empty model and its network description from the FreeRTOS heap. Each call also applies the NPU clock divider and bus settings; the first call creates the semaphores of the NPU driver. Load a model into the instance with iproai_load_model_from_buffer() or iproai_load_model_from_file().

Returns: Handle to the new instance, or NULL when the heap is exhausted.

iproai_getNetInfo¶

struct iproai_net_info_t * iproai_getNetInfo(iproai_model_hdl_t hdl);

Get the network description of a model.

Returns: Network description. Its fields are zero until a model is loaded; the pointer stays valid until iproai_free().

iproai_load_model_from_file¶

IPROAI_Status_e iproai_load_model_from_file(iproai_model_hdl_t hdl, const char *name);

Load a model from a file on a FatFS volume.

Reads the whole file into a heap buffer, checks that its section table matches the file size and loads it with iproai_load_model_from_buffer(). The staging buffer is released before the function returns, so the peak heap use is the file size plus the memory of the loaded model.

iproai_load_model_from_buffer¶

IPROAI_Status_e iproai_load_model_from_buffer(iproai_model_hdl_t hdl, const uint8_t *buffer);

Load a model from memory.

Copies each section of the model blob into its own heap buffer, decodes the layer table, allocates the working buffer and programs the NPU with the buffers of this model. The blob is only read during the call, so it can sit in memory-mapped (XIP) flash and may be released afterwards. Loading prints the buffer sizes, the layer table and an NPU register dump on the console.

For a model with ext_input set, the load programs the external input address of the NPU from the global imgBuffer (hal_npu.h) plus 4 and cleans the data cache over the input frame there, so imgBuffer must point at a valid buffer before this call.

iproai_getInputBuffer¶

uint8_t * iproai_getInputBuffer(iproai_model_hdl_t hdl);

Get the working buffer of a model.

The working buffer holds every tensor of the model: the input, the intermediate results and the outputs, each starting at a patch boundary (see iproai_def.h). For a model without ext_input, write the input tensor at the input patch of the layer that reads the image (the first layer, or the layer with img_in set): byte offset in_layer1_mem * patch_size. After an inference, the output of a layer starts at byte offset out_layer_mem * patch_size.

The NPU accesses this memory directly. Clean the data cache over the input you wrote (L1C_DCACHE_CLEAN_INVALID_RANGE()) before starting the inference, and invalidate it over an output before reading the output.

Returns: Start of the working buffer, or NULL when no model is loaded or the buffer could not be allocated.

iproai_getOutputBuffer¶

uint8_t * iproai_getOutputBuffer(iproai_model_hdl_t hdl, uint32_t *size);

Get the output tensor of the last layer of a model.

Returns the address of the output of layers[layer_cnt - 1] in the working buffer. For a model with several outputs, or when the last layer is not the output you need, use iproai_getInputBuffer() with iproai_net_info_t::output_layer instead.

Returns: Start of the output tensor.

iproai_free¶

IPROAI_Status_e iproai_free(iproai_model_hdl_t hdl);

Release a model instance and everything it owns.

Frees the layer tables, the working buffer, the copied model sections, the network description and the instance itself. The NPU driver stays initialized for later models. The handle and every pointer obtained from it are invalid afterwards. The function can be called after a failed load.

Returns: IPROAI_STATUS_NO_ERROR.

iproai_setResultCB¶

IPROAI_Status_e iproai_setResultCB(iproai_model_hdl_t hdl, iproai_inference_cb cb);

Set the callback that receives the results of the built-in post-processing.

The callback is called from iproai_startCompute(); see iproai_inference_cb. It can be set before or after the model is loaded.

Returns: IPROAI_STATUS_NO_ERROR.

iproai_setCustom_Postproc_CB¶

IPROAI_Status_e iproai_setCustom_Postproc_CB(iproai_model_hdl_t hdl, iproai_custom_postproc_cb cb);

Register a custom post-processing callback.

Stores cb and sets custom_postprocess in the network description, which disables the built-in FR_EVALUATION post-processing. A model load clears the flag, so call this function after the load.

Returns: IPROAI_STATUS_NO_ERROR.

iproai_startCompute¶

IPROAI_Status_e iproai_startCompute(iproai_model_hdl_t hdl);

Run one inference of the loaded model.

Runs all layers, alternating between CPU stages and NPU stages, then runs the built-in post-processing for the application type of the model if the library includes it (see APPLICATION_TYPE). The calling task blocks while the NPU runs; other tasks keep running. The time spent in CPU and NPU stages is stored in IPROAI_Model_t::inst (process_time, in RTOS ticks).

Prepare the input first, as described for iproai_getInputBuffer() or with NPU_Img_Ext_Addr_Cfg(). Patches of the working buffer are reused between layers, so the input is not guaranteed to survive a run: provide it again before every inference. The outputs stay in the working buffer until the next inference.

iproai_getInputResolution¶

IPROAI_Status_e iproai_getInputResolution(iproai_model_hdl_t hdl, uint32_t *width, uint32_t *height);

Get the input resolution of the loaded model.

Returns: IPROAI_STATUS_NO_ERROR.

iproai_setSourceResolution¶

IPROAI_Status_e iproai_setSourceResolution(iproai_model_hdl_t hdl, uint32_t width, uint32_t height);

Set the resolution of the source image that the input was taken from.

The built-in post-processing of detection models (OBJECT_DETECTION, FACE_LANDMARK, the RetinaNet types and SSD layers) scales box and landmark coordinates to this resolution. Both values are 0 after iproai_create(), which yields zero coordinates.

Returns: IPROAI_STATUS_NO_ERROR.

check_IPROAI_NPU_RUN¶

bool check_IPROAI_NPU_RUN(struct iproai_net_info_t *net, struct cpu_inst_layer_t *l);

Check whether the NPU can run a layer.

The engine uses this check at load time to place the layers of a model whose use_npu_accel flag is set. It looks at the layer type and at its kernel size, stride, dilation, upsampling factor, padding and number of inputs.

Returns: true when the NPU can run the layer, false when the layer must run on the CPU.

iproai_inst_npu.h¶

#include "iproai_inst_npu.h" (Source)

Low-level NPU control used by the AI runtime: set-up, dispatch and teardown.

Most applications never call these functions: iproai_create(), iproai_load_model_from_buffer(), iproai_startCompute() and iproai_free() (iproai_core.h) use them internally. They matter when several models share the NPU, or when an application tears the NPU down completely.

iproai_npu_inference¶

IPROAI_ProcStatus_e iproai_npu_inference();

Run the NPU program that is currently configured and wait for it.

Enables the NPU clock, starts the NPU and blocks the calling task on a semaphore (no timeout) until the completion interrupt arrives, then gates the clock again. The CPU is free for other tasks while waiting.

Requires the interrupt set-up done by iproai_npu_initCfg(). Must be called from a task, not from an interrupt handler.

Returns: IPROAI_PROCESS_CPU, the next state of the runtime's execution loop.

iproai_npu_stop¶

IPROAI_Status_e iproai_npu_stop(void);

Stop the NPU.

Called by the runtime at the end of every inference.

Returns: IPROAI_STATUS_NO_ERROR.

iproai_npu_initCfg¶

IPROAI_Status_e iproai_npu_initCfg(iproai_model_hdl_t hdl);

Program the NPU for a loaded model and install its interrupt handler.

Writes the model's NPU instruction, weight, bias and multiplier/shift buffers, its data buffer and patch size and its network parameters into the NPU registers, then registers and enables the NPU completion interrupt. Calls iproai_npu_init() if the completion semaphore does not exist yet. Prints a register dump and progress messages to the console.

Called by the model loader. Call it again before running a model when another model has used the NPU since (see the note in the file description).

Returns: IPROAI_STATUS_NO_ERROR.

iproai_npu_release¶

IPROAI_Status_e iproai_npu_release(iproai_model_hdl_t hdl);

Free the NPU layer table of a model.

Called by iproai_free(). The pointer in the model is not cleared, so do not call it in addition to iproai_free() for the same model.

Returns: IPROAI_STATUS_NO_ERROR.

iproai_npu_exec_lock_required¶

void iproai_npu_exec_lock_required(void);

Take the NPU execution mutex, waiting as long as necessary.

The mutex only serialises code that takes it explicitly: iproai_startCompute() does not take it. Requires iproai_npu_init() (called by iproai_create()). Must be called from a task.

iproai_npu_exec_lock_release¶

void iproai_npu_exec_lock_release(bool in_isr);

Release the NPU execution mutex taken with iproai_npu_exec_lock_required().

iproai_npu_init¶

void iproai_npu_init(void);

Initialise the NPU support of the runtime.

On the first call (or the first after iproai_npu_destroy()) creates the completion semaphore and the execution mutex and gates the NPU clock off. Every call then sets the NPU clock divider and bus configuration. iproai_create() calls it, so applications normally do not need to.

iproai_npu_destroy¶

void iproai_npu_destroy(void);

Delete the semaphores created by iproai_npu_init().

Call only when no model will run any more, after iproai_free() of every model; call iproai_npu_init() (or iproai_create()) before using the NPU again.

iproai_inst_util.h¶

#include "iproai_inst_util.h" (Source)

AI runtime utilities: cache-line aligned allocation, fixed-point conversion and tensor addressing in the model data buffer.

Included by iproai_core.h. Buffers that the NPU or a DMA engine reads or writes should come from IPROAI_MALLOC, which aligns them to a cache line so that cache maintenance never touches neighbouring heap objects.

Tensors of a loaded model live in its data buffer (iproai_getInputBuffer()), which is divided into memory slots of net->patch_size bytes; a layer's output starts at slot layer.out_layer_mem. MEM_PTR() and MEM_PTR_ADDR() compute such addresses.

iproai_mem_alloc¶

void * iproai_mem_alloc(size_t size, const char *func, const int line);

Allocate a cache-line aligned buffer from the FreeRTOS heap.

Use IPROAI_MALLOC rather than calling this directly. The returned pointer is aligned to the 32-byte cache line; the raw heap pointer is stored just below it for iproai_mem_free().

Returns: Aligned pointer, or NULL if size is 0 or the heap is exhausted (a message is printed in both cases).

iproai_mem_free¶

void iproai_mem_free(void *addr, const char *func, const int line);

Free a buffer allocated with iproai_mem_alloc().

Use IPROAI_FREE rather than calling this directly.

unsigned_to_signed¶

int unsigned_to_signed(int in, int bit);

Sign-extend a two's-complement bit field.

Uses a small temporary heap allocation.

Returns: The field value interpreted as a signed number of bit bits.

fixed_to_float¶

float fixed_to_float(fixed_point_t input, int FIXED_POINT_FRACTIONAL_BITS);

Convert an 8-bit fixed-point value to float.

Returns: input / 2^FIXED_POINT_FRACTIONAL_BITS.

fixed32_to_fixed8¶

fixed_point_t fixed32_to_fixed8(int32_t input, int FIXED_POINT_FRACTIONAL_BITS);

Requantise a 32-bit fixed-point value to 8 bits.

Shifts right by FIXED_POINT_FRACTIONAL_BITS with rounding to nearest (ties away from zero), or left by its magnitude when it is 0 or negative, and saturates to [-128, 127].

Returns: Rounded and saturated 8-bit value.

fixed8_to_fixed8¶

fixed_point_t fixed8_to_fixed8(fixed_point_t input, int FIXED_POINT_FRACTIONAL_BITS);

Rescale an 8-bit fixed-point value by a power of two.

Same rounding and saturation as fixed32_to_fixed8(), for an 8-bit input.

Returns: Rounded and saturated 8-bit value.

fixed32_to_fixed32¶

int32_t fixed32_to_fixed32(int32_t input, int FIXED_POINT_FRACTIONAL_BITS);

Rescale a 32-bit fixed-point value by a power of two.

Same rounding as fixed32_to_fixed8(); the result saturates to the int32_t range.

Returns: Rounded and saturated 32-bit value.

float_to_fixed¶

fixed_point_t float_to_fixed(float input, int FIXED_POINT_FRACTIONAL_BITS);

Convert a float to 8-bit fixed point.

Saturates input to the range representable with FIXED_POINT_FRACTIONAL_BITS fractional bits, then rounds input * 2^FIXED_POINT_FRACTIONAL_BITS to the nearest integer.

Returns: 8-bit fixed-point value.

fixed8_to_fixed32¶

int32_t fixed8_to_fixed32(fixed_point_t input, int FIXED_POINT_FRACTIONAL_BITS);

Shift an 8-bit value and widen it to 32 bits.

Arithmetic right shift by FIXED_POINT_FRACTIONAL_BITS (no rounding), or left shift by its magnitude when it is 0 or negative.

Returns: Shifted value.

MEM_PTR¶

int8_t * MEM_PTR(int mux, fixed_point_t *DATA_buf, int PATCH_SIZE);

Get the start of a memory slot in a model data buffer.

Returns: DATA_buf + mux * PATCH_SIZE.

MEM_PTR_ADDR¶

fixed_point_t * MEM_PTR_ADDR(int mux, int addr, fixed_point_t *DATA_buf, int patch_size);

Get the address of a byte inside a memory slot of a model data buffer.

Returns: DATA_buf + mux * patch_size + addr.

Macros¶

iproai_inst_postprocess.h¶

#include "iproai_inst_postprocess.h" (Source)

Built-in (legacy) post-processors run by the AI runtime after an inference.

When a model's application type selects one of these post-processors, iproai_startCompute() calls it after the inference and the post-processor reports its result through the callback registered with iproai_setResultCB(), as a struct iproai_result_t:

Application type Post-processor Result fields

CLASSIFICATION image_classification() r.classification

KEYWORD_SPOTTING KWS_classification() r.classification

FACE_RECONGNITION face_recongnition() r.fr

OBJECT_DETECTION yolo_obj_det() r.objdet

FACE_LANDMARK yolo_face_landmark() r.landmark

RETINA_FACE retina_face() r.objdet, r.landmark

RETINA_PERSON retina_person() r.objdet

RETINA retina() r.objdet

The class names and the reference face features they use are set with load_postprocess_classification() and load_postprocess_face_recongnition(); the image size used to scale boxes is the one set with iproai_setSourceResolution().

These functions are only built when the matching option is enabled: CONFIG_AI_LEGACY_YOLO for yolo_obj_det() and yolo_face_landmark(), CONFIG_AI_LEGACY_POSTPROCESS for all the others. New applications decode model outputs themselves, for example with yolov8_detect.h.

Unless stated otherwise, net is the network information of the model (iproai_getNetInfo()), DATA_buf its data buffer (iproai_getInputBuffer()) and result_cb the callback that receives the results.

image_classification¶

void image_classification(struct iproai_net_info_t *net, const char **imgnet_names, fixed_point_t *DATA_buf, iproai_inference_cb result_cb, int num);

Report the most probable class of a classifier with a background class.

Takes the arg-max over the first num bytes (read as unsigned) of the last layer's output. Index 0 is treated as the background class, so the class reported is the arg-max minus 1; prints its name and calls result_cb once with r.classification.best_class set to it.

KWS_classification¶

void KWS_classification(struct iproai_net_info_t *net, const char **imgnet_names, fixed_point_t *DATA_buf, iproai_inference_cb result_cb, int num);

Report the most probable keyword of a keyword-spotting model.

Takes the arg-max over the first num bytes (read as unsigned) of the last layer's output, prints its name and calls result_cb once with r.classification.best_class set to the arg-max index.

get_output_data¶

void get_output_data(struct iproai_net_info_t *net, fixed_point_t *DATA_buf, int layer, int size);

Print the first bytes of a layer's output to the console (diagnostic).

face_recongnition¶

void face_recongnition(struct iproai_net_info_t *net, const char **names, int names_len, const float **feature_list, int feature_len, fixed_point_t *DATA_buf, iproai_inference_cb result_cb);

Identify a face against a set of reference feature vectors.

Converts the first 128 values of the last layer's output from fixed point to float and computes their cosine similarity with each reference vector. The best reference with a similarity above 0.63 wins. Prints every similarity and the result, then calls result_cb once with r.fr.best_class set to the winning index, or 255 if no reference matches.

yolo_obj_det¶

void yolo_obj_det(struct iproai_net_info_t *net, const char **names, int image_width, int image_height, iproai_inference_cb result_cb);

Decode a legacy YOLO (v2/v3 region layer) object detector.

Uses a confidence threshold of 0.24 and an NMS IoU threshold of 0.45, prints each detection and calls result_cb once per detection with r.objdet.rect (top-left corner and size in pixels of the image_width x image_height image), r.objdet.best_class and r.objdet.prob (percent). Nothing is reported when no box is found.

yolo_face_landmark¶

void yolo_face_landmark(struct iproai_net_info_t *net, const char **names, int image_width, int image_height, iproai_inference_cb result_cb);

Decode a legacy YOLO face detector with five facial landmarks.

Uses the same thresholds as yolo_obj_det(). Prints every face and calls result_cb once, after all faces are processed, with r.landmark: lx1 to lx5 hold the left eye, right eye, nose, left and right mouth corner of the most probable face, in pixels; rect, best_class and prob (percent) are those of the right-most face. Both refer to the same face when exactly one face is detected.

load_postprocess_classification¶

void load_postprocess_classification(iproai_model_hdl_t hdl, const char **classes_name, int classes_num);

Set the class-name table used by the built-in post-processors.

The array is not copied and must stay valid while the model is used.

load_postprocess_face_recongnition¶

void load_postprocess_face_recongnition(iproai_model_hdl_t hdl, const char **names, int names_num, const float **features_list, int features_num);

Set the reference faces used by face_recongnition().

The arrays are not copied and must stay valid while the model is used.

retina_face¶

void retina_face(struct iproai_net_info_t *net, const char **imgnet_names, fixed_point_t *DATA_buf, int image_width, int image_height, iproai_inference_cb result_cb);

Decode a RetinaFace-style face detector.

Expects three outputs, told apart by their channel count: boxes (4), landmarks (10) and two-class scores (2). Boxes are decoded against the prior boxes of the model (net->prior_bbox, variances 0.1 and 0.2), then NMS runs with an IoU threshold of 0.4, a score threshold of 0.5 and at most 50 faces. For each face, prints it and calls result_cb with r.objdet.rect (top-left corner and size in pixels of the image_width x image_height image), r.objdet.prob (percent), best_class 0, and the first four landmark points in r.landmark.lx1 to lx4.

Free memory slots of DATA_buf are used as scratch space.

retina_person¶

void retina_person(struct iproai_net_info_t *net, const char **imgnet_names, fixed_point_t *DATA_buf, int image_width, int image_height, iproai_inference_cb result_cb);

Decode a RetinaNet-style single-class person detector.

Like retina_face() without landmarks: expects a box output (4 channels) and a two-class score output (2 channels), and uses a score threshold of 0.7. Calls result_cb once per person with r.objdet (best_class 0).

retina¶

void retina(struct iproai_net_info_t *net, const char **imgnet_names, int class_num, fixed_point_t *DATA_buf, int image_width, int image_height, iproai_inference_cb result_cb);

Decode a multi-class RetinaNet-style detector.

Expects a box output (4 channels) and a score output with class_num logits per prior box (passed through a sigmoid). For each class, runs NMS with an IoU threshold of 0.2, a score threshold of 0.3 and at most 50 boxes, drops boxes narrower or lower than 50 pixels, prints the rest and calls result_cb once per box with r.objdet.rect (pixels of the image_width x image_height image), best_class (class index) and prob (percent).

get_segmentation_mask¶

void get_segmentation_mask(struct iproai_net_info_t *net, fixed_point_t *DATA_buf, int layer, uint32_t out_image_width, uint32_t out_image_height, iproai_inference_cb result_cb);

Count the foreground pixels of a segmentation output (diagnostic).

Counts the pixels of the last layer's output whose value exceeds 128 and prints the count. Nothing else is reported.

get_sr_result¶

void get_sr_result(struct iproai_net_info_t *net, fixed_point_t *DATA_buf, int layer, uint32_t out_image_width, uint32_t out_image_height, iproai_inference_cb result_cb);

Placeholder post-processor for super-resolution models.

Reads the 3-channel output image of the last layer, prints its size and reports nothing.

iproai_riscv_nnfunctions.h¶

#include "iproai_riscv_nnfunctions.h" (Source)

CPU convolution kernel (NMSIS-NN style) used by the AI runtime, with its fixed-point types and limits.

The runtime falls back to these kernels for layers that run on the CPU instead of the NPU. The parameter structures (::nmsis_nn_conv_params, ::nmsis_nn_dims, ...) come from riscv_nn_types.h.

Quantisation follows TensorFlow Lite: a real value is scale * (q - zero_point), convolution offsets are the negated zero points, and results are requantised per output channel with a Q31 multiplier and a power-of-two shift.

iproai_convolve_s8_get_buffer_size¶

int32_t iproai_convolve_s8_get_buffer_size(const nmsis_nn_dims *input_dims, const nmsis_nn_dims *filter_dims);

Get the scratch-buffer size of an int8 convolution.

Returns: 2 * input_dims->c * filter_dims->w * filter_dims->h * sizeof(int16_t) bytes when the library is built with the RISC-V DSP extension (RISCV_MATH_DSP), otherwise 0. riscv_convolve_u8() does not use a scratch buffer.

riscv_convolve_u8¶

riscv_status riscv_convolve_u8(const nmsis_nn_context *ctx, const nmsis_nn_conv_params *conv_params, const nmsis_nn_per_channel_quant_params *quant_params, const nmsis_nn_dims *input_dims, const uint8_t *input_data, const nmsis_nn_dims *filter_dims, const uint8_t *filter_data, const nmsis_nn_dims *bias_dims, const int32_t *bias_data, const nmsis_nn_dims *output_dims, uint8_t *output_data);

2-D convolution of unsigned 8-bit tensors with per-channel requantisation (reference C implementation).

For every batch, output pixel and output channel oc:

acc = sum over ky, kx, ic of
        (input[y][x][ic] + input_offset) * (filter[oc][ic][ky][kx] + filter_offset)
acc = acc + bias[oc]                                  (if bias_data != NULL)
out = requantize(acc, multiplier[oc], shift[oc]) + output_offset
out = clamp(out, activation.min, activation.max)

with y = stride.h * oy - padding.h + ky and x = stride.w * ox - padding.w + kx. Taps that fall outside the input are skipped, which is equivalent to padding with the input zero point. requantize() follows TensorFlow Lite: a positive shift first multiplies acc by 2^shift, the result is then scaled by the Q31 multiplier as round(acc * multiplier / 2^31), and a negative shift finally divides by 2^-shift, rounding half away from zero. The accumulator is 32-bit.

Tensor layouts (all dimensions must fit in 16 bits):

• input: [input_dims->n][input_dims->h][input_dims->w][input_dims->c] (NHWC);
• filter: [output_dims->c][input_dims->c][filter_dims->h][filter_dims->w]; note that the input channel comes before the kernel rows;
• bias: output_dims->c int32 values;
• output: [input_dims->n][output_dims->h][output_dims->w][output_dims->c] (NHWC).

Grouped and dilated convolutions are not supported: conv_params->dilation is ignored and every output channel uses all input channels.

Returns: Always RISCV_MATH_SUCCESS; the arguments are not validated.

riscv_status¶

Error status returned by some functions in the library.

riscv_nnword¶

One 32-bit word viewed as one q31, two q15 or four q7 values, for SIMD-style access.

riscv_nnword_u¶

One 32-bit word viewed as one, two or four unsigned integers.

riscv_nn_double¶

The two 32-bit halves of a 64-bit value (little-endian order).

riscv_nn_long_long¶

A 64-bit signed value accessible as a whole or as two 32-bit halves.

Typedefs¶

Macros¶

iproai_cycle_profile.h¶

#include "iproai_cycle_profile.h" (Source)

Cycle-accurate NPU-busy versus CPU-busy split of model inference.

Measures, with the 64-bit CPU cycle counter, how much of each inference run by iproai_startCompute() the CPU spends waiting for the NPU. The tick-based profiling counters have 1 ms resolution, which is too coarse for layers that take tens of microseconds; these counters resolve single cycles.

The waiting share is the concurrency budget of the system: the runtime blocks on an RTOS semaphore while the NPU runs instead of spinning, so every cycle counted as NPU-busy is available to other tasks (for example a second model running on the CPU).

Two kinds of data are kept:

• accumulated totals since the last iproai_cycle_reset(): iproai_cycle_total(), iproai_cycle_npu(), iproai_cycle_runs(), iproai_cycle_npu_dispatches(), printed by iproai_cycle_report();
• a per-layer breakdown (spans) of the most recent inference only: iproai_cycle_spans(), iproai_cycle_span(), printed by iproai_cycle_layer_report().

All values are CPU clock cycles. Enable the feature with CONFIG_AI_CYCLE_PROFILE; when it is disabled every function in this header is replaced by a macro that evaluates to 0, NULL or nothing, so calls can stay in application code at no cost.

iproai_cycle_spans¶

uint16_t iproai_cycle_spans(void);

Get the number of spans recorded for the most recent inference.

Returns: Span count, 0 to IPROAI_CYCLE_MAX_SPANS.

iproai_cycle_span¶

const iproai_cycle_span_t * iproai_cycle_span(uint16_t i);

Get one span of the most recent inference.

Returns: Pointer to the span, valid until the next inference starts, or NULL if i is out of range.

iproai_cycle_spans_truncated¶

int iproai_cycle_spans_truncated(void);

Check whether spans of the most recent inference were dropped.

Returns: Non-zero if the inference produced more than IPROAI_CYCLE_MAX_SPANS spans, 0 otherwise.

iproai_cycle_layer_report¶

void iproai_cycle_layer_report(void);

Print the span breakdown of the most recent inference.

Prints one AILAYER line per span (layer range, NPU flag, cycles and the share of the total in permille) and ends with an AILAYEREND line, which makes the output easy to parse on a host.

iproai_cycle_npu¶

uint64_t iproai_cycle_npu(void);

Get the cycles spent waiting for the NPU since the last reset.

Returns: Accumulated NPU-busy cycles over all NPU dispatches.

iproai_cycle_total¶

uint64_t iproai_cycle_total(void);

Get the cycles spent in whole inferences since the last reset.

Returns: Accumulated duration of all iproai_startCompute() calls, in cycles.

iproai_cycle_runs¶

uint32_t iproai_cycle_runs(void);

Get the number of completed inferences since the last reset.

Returns: Inference count.

iproai_cycle_npu_dispatches¶

uint32_t iproai_cycle_npu_dispatches(void);

Get the number of NPU dispatches since the last reset.

Each dispatch runs one contiguous group of NPU layers.

Returns: Dispatch count.

iproai_cycle_reset¶

void iproai_cycle_reset(void);

Clear the accumulated totals.

Resets the values returned by iproai_cycle_npu(), iproai_cycle_total(), iproai_cycle_runs() and iproai_cycle_npu_dispatches(). The span breakdown is not affected; it is replaced at the start of every inference.

iproai_cycle_report¶

void iproai_cycle_report(void);

Print the accumulated NPU/CPU split to the console.

Prints the number of inferences and NPU dispatches, the total and per-inference cycles, the NPU-busy and CPU-busy cycles with their percentages, and the share of the core a co-running task could use. Prints a short notice instead if no inference was recorded.

iproai_cycle_span_t¶

One timed stretch of an inference.

A layer running on the CPU or DSP is timed on its own, so first equals last. NPU work is different: one NPU dispatch executes a whole contiguous run of NPU layers without stopping in between, so the hardware offers no per-layer boundary to measure. Such a span covers layers first to last and reports their combined time; it is not split between them.

Macros¶

yolov8_detect.h¶

#include "yolov8_detect.h" (Source)

YOLOv8 detection post-processor for 6-output NPU models, plus the box and NMS helpers shared by the other detectors.

Decodes the output tensors of an anchor-free YOLOv8 model compiled with separate box and class heads per level (3 levels x 2 heads = 6 outputs) into a list of boxes in input-image pixels. The result types and the helpers (yolov8_nms(), yolov8_box_iou(), ...) are also used by gesture_dfl_detect.h and big_a_sppf_detect.h.

Typical sequence:

1. Load the model, then yolov8_init() (or yolov8_person_pet_init()) with the network information from iproai_getNetInfo().
2. Per frame: iproai_startCompute(), then yolov8_process_6out() with the model data region from iproai_getInputBuffer().
3. yolov8_deinit() when done.

Expected model outputs, in net->output_layer[] order: box P5, class P5, box P4, class P4, box P3, class P3 (P3/P4/P5 = strides 8/16/32). Each tensor is stored cell by cell with 4 bytes per cell: a box cell holds the left, top, right and bottom distances to the cell centre in grid units, a class cell holds one logit per class followed by padding. Values are 8-bit and dequantised as (q - zero_point) * scale.

yolov8_init¶

int yolov8_init(yolov8_config_t *config, struct iproai_net_info_t *net);

Initialise the YOLOv8 detector for a loaded model.

Sets the defaults (input YOLOV8_INPUT_WIDTH x YOLOV8_INPUT_HEIGHT, YOLOV8_NUM_CLASSES classes, default thresholds), reads the quantisation of the six outputs and the cell count of each level from net, and allocates the anchor grid, the sigmoid table and the buffers with malloc(). class_names is not touched.

yolov8_process_6out¶

int yolov8_process_6out(yolov8_config_t *config, struct iproai_net_info_t *net, uint8_t *data_buffer, yolov8_result_t *result);

Decode the six output tensors of one inference into detections.

For every cell the highest class logit is taken; cells whose confidence (sigmoid of that logit) is at least conf_threshold become candidates (up to YOLOV8_MAX_CANDIDATES, scanning P3, P4, then P5). Their boxes are decoded, then class-wise NMS keeps up to YOLOV8_MAX_DETECTIONS boxes, clamped to the input size and named from class_names.

yolov8_deinit¶

void yolov8_deinit(yolov8_config_t *config);

Free the buffers allocated by yolov8_init().

The pointers in config are not cleared, so call it only once per successful yolov8_init().

yolov8_print_results¶

void yolov8_print_results(const yolov8_result_t *result);

Print detection results to the console.

yolov8_generate_anchors¶

yolov8_anchor_grid_t * yolov8_generate_anchors(uint32_t img_height, uint32_t img_width);

Build the cell-centre grid of the three detection levels.

Note the argument order: height first. img_height is rounded down to a multiple of 16 first. Each level contributes (img_width / stride) * (img_height / stride) cells, for strides 8, 16 and 32.

Returns: Grid allocated with malloc() (free it with yolov8_free_anchor_grid()), or NULL if an allocation failed.

yolov8_free_anchor_grid¶

void yolov8_free_anchor_grid(yolov8_anchor_grid_t *grid);

Free a grid returned by yolov8_generate_anchors().

yolov8_softmax¶

void yolov8_softmax(const float *input, float *output, uint32_t size);

Compute a numerically stable softmax.

yolov8_dfl¶

void yolov8_dfl(const float *box_pred, uint32_t num_anchors, uint32_t reg_max, float *output);

Convert Distribution Focal Loss (DFL) bins to box-side distances.

For each cell and each of the four sides, takes the softmax over the reg_max bins and outputs its expected bin index. With reg_max equal to 1 the input is copied unchanged.

yolov8_dist2bbox¶

void yolov8_dist2bbox(const float *distances, const yolov8_anchor_grid_t *anchors, uint32_t num_anchors, float *boxes);

Convert side distances around cell centres into boxes.

yolov8_nms¶

int yolov8_nms(const float *boxes, const float *scores, const int *classes, int num_boxes, float iou_threshold, yolov8_detection_t *output);

Greedy class-aware non-maximum suppression.

Sorts the boxes by descending score and keeps each box unless an already-kept box of the same class overlaps it with an IoU greater than iou_threshold. Stops after YOLOV8_MAX_DETECTIONS boxes.

Returns: Number of detections written, or -1 if a temporary buffer could not be allocated.

yolov8_box_iou¶

float yolov8_box_iou(const yolov8_box_t *box1, const yolov8_box_t *box2);

Compute the intersection over union of two boxes.

Returns: IoU in [0, 1]; 0 if the union area is not positive.

yolov8_sigmoid_lut¶

float yolov8_sigmoid_lut(const float *sigmoid_table, uint32_t table_size, float x);

Evaluate the sigmoid with a lookup table.

Interpolates linearly in a table that samples the sigmoid uniformly over [-10, 10], such as yolov8_config_t::sigmoid_table. Outside that range the sigmoid is computed exactly.

Returns: Sigmoid of x, 0 to 1.

yolov8_box_t¶

Axis-aligned bounding box in input-image pixels.

yolov8_detection_t¶

One detected object.

yolov8_result_t¶

Detection results of one frame.

yolov8_anchor_grid_t¶

Cell centres and strides of all detection levels.

Built by yolov8_generate_anchors(); entries are ordered P3, P4, P5 and row by row within a level.

yolov8_quant_param_t¶

Quantisation parameters of one output tensor.

Real value = (q - zero_point) * scale.

yolov8_config_t¶

Detector configuration and state.

Filled by yolov8_init(). conf_threshold, iou_threshold and class_names may be changed after init; the other fields describe the model and the allocated buffers and must be left alone.

Macros¶

yolov8_person_pet_detect.h¶

#include "yolov8_person_pet_detect.h" (Source)

Person and pet detection: YOLOv8 post-processor preset.

Thin wrapper around yolov8_detect.h for the two-class person/pet model: yolov8_person_pet_init() configures the generic detector with the class names below, and the other functions forward to their yolov8_* counterparts with extra console logging. Use the yolov8_* functions directly if the logging is not wanted.

yolov8_person_pet_init¶

int yolov8_person_pet_init(yolov8_config_t *config, struct iproai_net_info_t *net);

Initialise the detector for the person/pet model.

Calls yolov8_init(), then sets two classes, the yolov8_person_pet_class_names table and the default thresholds, and prints the resulting configuration.

yolov8_person_pet_process¶

int yolov8_person_pet_process(yolov8_config_t *config, struct iproai_net_info_t *net, uint8_t *data_buffer, yolov8_result_t *result);

Decode one inference of the person/pet model.

Calls yolov8_process_6out() and logs the number of detections.

yolov8_person_pet_process_6out¶

int yolov8_person_pet_process_6out(yolov8_config_t *config, iproai_model_hdl_t hdl, yolov8_result_t *result);

Former decode entry point taking a model handle.

DeprecatedUse yolov8_person_pet_process(). This function does nothing except print an error message.

Returns: Always -1.

yolov8_person_pet_deinit¶

void yolov8_person_pet_deinit(yolov8_config_t *config);

Free the detector buffers.

Calls yolov8_deinit(); the same once-per-init rule applies.

yolov8_person_pet_print_results¶

void yolov8_person_pet_print_results(const yolov8_result_t *result);

Print person/pet detection results to the console.

Prints each box with its centre and size, the number of persons and pets, and the post-processing time.

Typedefs¶

Macros¶

gesture_dfl_detect.h¶

#include "gesture_dfl_detect.h" (Source)

Post-processor for the hand-sign digit detector (YOLOv8-ReLU, anchor-free with DFL boxes) on the IPRO7-AI NPU.

The model recognises hand signs for the digits 0 to 9. It has 3 raw output heads (P3/P4/P5), each of shape [1, H, W, 74]:

• channels 0..63: DFL box regression, 4 sides x 16 bins;
• channels 64..73: 10 class logits (digits 0 to 9), no objectness.

Decoding, with every value dequantised as f(q) = (q - zero_point) * scale:

reg[side][b] = f(ch[side * 16 + b]),  b = 0..15
dist[side]   = sum over b of softmax(reg[side])[b] * b
ax, ay       = grid_x + 0.5, grid_y + 0.5
x1 = (ax - dist[0]) * stride      y1 = (ay - dist[1]) * stride
x2 = (ax + dist[2]) * stride      y2 = (ay + dist[3]) * stride
score[c] = sigmoid(f(ch[64 + c])); class = argmax(score); conf = max(score)

Cells with conf >= conf_threshold go through greedy NMS (yolov8_nms()). The results use the yolov8_detect.h types (yolov8_result_t).

NPU output memory: channels are padded to a multiple of 4 per cell (76 bytes), so value(h, w, c) is at data_buffer + out_layer_mem * patch_size + (h * W + w) * 76 + c.

Typical sequence: gdfl_init() after loading the model, then per frame iproai_startCompute() and gdfl_process(), and gdfl_deinit() when done.

gdfl_init¶

int gdfl_init(gdfl_config_t *cfg, struct iproai_net_info_t *net);

Initialise the hand-sign digit post-processor.

Clears cfg, applies the default thresholds, resolves the three output levels (grid size, stride, quantisation) from net and allocates the scratch buffers with malloc().

gdfl_process¶

int gdfl_process(gdfl_config_t *cfg, struct iproai_net_info_t *net, uint8_t *data_buffer, yolov8_result_t *result);

Decode the three 8-bit output heads and run NMS.

Invalidates the data cache over the three output regions before reading them, since the NPU writes its outputs to memory behind the CPU cache. At the default log level it also prints the raw values of two cells and the peak class score of the frame, which helps to spot an input that does not reach the NPU.

gdfl_deinit¶

void gdfl_deinit(gdfl_config_t *cfg);

Release the scratch buffers.

The buffer pointers are cleared, so calling it twice is harmless.

gdfl_print_results¶

void gdfl_print_results(const yolov8_result_t *result);

Print detections to the console.

gdfl_level_t¶

Per-output-level descriptor, resolved from the network information by gdfl_init().

gdfl_config_t¶

Detector configuration and scratch state.

Filled by gdfl_init(); conf_threshold and iou_threshold may be changed afterwards.

Typedefs¶

Macros¶

big_a_sppf_detect.h¶

#include "big_a_sppf_detect.h" (Source)

Post-processor for the BIG-A + SPPF-lite single-class product detector (SKU-110K) on the IPRO7-AI NPU.

This model is a YOLOv5-style anchor-based detector, unlike the anchor-free YOLOv8 models of yolov8_detect.h. It has 3 output levels (P3/P4/P5), each of shape [1, H, W, 18] where 18 = 3 anchor slots x (4 box + 1 objectness + 1 class) values, for a single class.

Decoding, with every value dequantised as (q - zero_point) * scale:

cx = (sigmoid(tx) + grid_x) / grid_w      (normalised 0..1)
cy = (sigmoid(ty) + grid_y) / grid_h
bw = exp(tw)                              (normalised, no anchor prior)
bh = exp(th)
score = sigmoid(obj) * sigmoid(cls)
box = [cx - bw/2, cy - bh/2, cx + bw/2, cy + bh/2] * input size

followed by greedy NMS (yolov8_nms()). The results use the yolov8_detect.h types (yolov8_result_t), so code that consumes YOLOv8 detections works unchanged.

Typical sequence: big_a_sppf_init() after loading the model, then per frame iproai_startCompute() and big_a_sppf_process(), and big_a_sppf_deinit() when done.

big_a_sppf_init¶

int big_a_sppf_init(big_a_sppf_config_t *config, struct iproai_net_info_t *net);

Initialise the BIG-A + SPPF-lite post-processor.

Clears config, applies the default thresholds, resolves the three output levels from net and allocates the scratch buffers with malloc(). The output order does not matter: each level is decoded with its own grid size.

big_a_sppf_process¶

int big_a_sppf_process(big_a_sppf_config_t *config, struct iproai_net_info_t *net, uint8_t *data_buffer, yolov8_result_t *result);

Decode the three 8-bit output tensors and run NMS.

big_a_sppf_deinit¶

void big_a_sppf_deinit(big_a_sppf_config_t *config);

Release the scratch buffers.

The buffer pointers are cleared, so calling it twice is harmless.

big_a_sppf_print_results¶

void big_a_sppf_print_results(const yolov8_result_t *result);

Print detections to the console.

big_a_sppf_level_t¶

Per-output-level descriptor, resolved from the network information by big_a_sppf_init().

big_a_sppf_config_t¶

Detector configuration and scratch state.

Filled by big_a_sppf_init(); conf_threshold and iou_threshold may be changed afterwards.

Typedefs¶

Macros¶

fbank.h¶

#include "fbank.h" (Source)

Log mel-filterbank feature extraction for the speaker-embedding model.

Converts 16 kHz mono PCM into a fixed [FBANK_SEQ_LEN][FBANK_N_MELS] matrix of log mel-filterbank energies, the input format of the speaker-embedding model (see speaker_embed.h, which calls these functions itself).

Per frame (25 ms window, 10 ms hop): DC removal, pre-emphasis (FBANK_PREEMPH), Povey window, 512-point FFT, power spectrum, triangular mel filters between FBANK_LOW_FREQ and FBANK_HIGH_FREQ, natural logarithm. After all frames, the mean of each mel bin over the valid frames is subtracted (per-column mean normalisation).

Two interchangeable back-ends implement this header, selected at build time by CONFIG_AI_FBANK_INT:

• not set (default): floating-point implementation;
• set: fixed-point implementation (Q15 tables, Q8 logarithm) for the IPRO7 DSP extension. Its output matches the float back-end within quantisation noise.

The layout of FBankContext differs between the back-ends, but callers only allocate sizeof(FBankContext) and pass the pointer to fbank_init() and fbank_compute(), so application code is the same for both.

fbank_init¶

void fbank_init(FBankContext *ctx);

Compute the analysis window and mel filterbank.

Call once at start-up; the context is read-only afterwards and can be reused for any number of fbank_compute() calls.

fbank_compute¶

int fbank_compute(const FBankContext *ctx, const int16_t *samples, int n_samples, float out[FBANK_SEQ_LEN][FBANK_N_MELS]);

Compute normalised log mel-filterbank features for one clip.

Frames are taken every FBANK_FRAME_SHIFT samples while a full FBANK_FRAME_LEN window fits, up to FBANK_SEQ_LEN frames; samples beyond that are ignored. A full 300-frame matrix needs 48240 samples (3.015 s). Rows after the last valid frame are zero before normalisation. The per-bin mean of the valid rows is then subtracted from every row, so padding rows end up holding minus that mean.

Returns: Number of valid frames extracted (0 to FBANK_SEQ_LEN, before padding); 0 when n_samples is shorter than one window.

FBankContext¶

Pre-computed window and mel filterbank (floating-point back-end).

Fill once with fbank_init() and reuse for every fbank_compute() call. About 84 KB, so allocate it on the heap or in static memory.

Macros¶

embedding_dim.h¶

#include "embedding_dim.h" (Source)

Length of the speaker-embedding vector shared by the speaker modules.

Kept in its own header so that the embedding engine (speaker_embed.h), the enrolled-speaker gallery (gallery.h) and the diarization clusterer (diarize.h) agree on the vector length without depending on each other.

Macros¶

speaker_embed.h¶

#include "speaker_embed.h" (Source)

Speaker-embedding engine: PCM clip in, normalised voiceprint out.

Common front end for speaker identification, verification and diarization (see gallery.h and diarize.h). One call runs the whole pipeline:

int16 PCM, 16 kHz mono
  -> log mel-filterbank [300][80] (fbank.h)
  -> INT8 quantisation, written to the NPU input
  -> speaker-embedding model on the NPU
  -> dequantisation and L2 normalisation -> embedding[EMBEDDING_DIM]

Typical sequence: speaker_embed_init() or speaker_embed_init_buf() once, then speaker_embed() per clip.

The engine keeps a single model instance and shared scratch buffers and is not reentrant: serialise all speaker_embed() calls, for example by running them in one task. Each call reprograms the NPU external-input address, so it must also be serialised with other NPU work. The calling task needs a generous stack: with the default floating-point filterbank, the feature extraction alone uses about 7 KB of it (see fbank_compute()).

speaker_embed_init¶

int speaker_embed_init(const char *model_path);

Load the embedding model from a file and initialise the engine.

Reads the whole model file into a temporary heap buffer, then behaves like speaker_embed_init_buf(). Returns 0 immediately if the engine is already initialised.

speaker_embed_init_buf¶

int speaker_embed_init_buf(const uint8_t *model_buf);

Initialise the engine from a model image already in memory.

Creates the model instance, loads the model, allocates the feature and NPU-input buffers from the FreeRTOS heap and prepares the filterbank. The model is copied during loading, so model_buf only needs to stay valid for the duration of this call (it may point into memory-mapped flash). Returns 0 immediately if the engine is already initialised.

speaker_embed¶

int speaker_embed(const int16_t *pcm, int n_samples, float emb[EMBEDDING_DIM]);

Compute the L2-normalised speaker embedding of one PCM clip.

Extracts the features with fbank_compute(), runs the model on the NPU and blocks until the inference is complete.

speaker_recog.h¶

#include "speaker_recog.h" (Source)

Speaker recognition on a pooled NPU feature map: embeddings, registration and identification.

This module turns the raw output feature map of a speaker model running on the NPU into a speaker embedding and matches it against a small database of registered speakers by cosine similarity. It also provides the matching audio front end (log mel-filterbank features from PCM).

Processing in speaker_recog_process():

8-bit NPU output [C*T*F] -> dequantise -> statistics pooling (mean and
standard deviation, 2*C*T values) -> Linear + ReLU + BatchNorm ->
Linear -> embedding[emb_size]

The two linear stages run on the CPU with float weights loaded by speaker_recog_init() (from a file) or speaker_recog_load_weights_from_buffer().

Typical call sequence:

1. speaker_recog_preprocess_init(), then speaker_recog_preprocess_pcm() to build the model input from PCM.
2. speaker_recog_init() (and speaker_recog_set_net() once the model is loaded), run the model with iproai_startCompute().
3. speaker_recog_process() on the model output.
4. speaker_recog_register() to store the new embedding as a speaker, or read the identification result filled in by step 3.
5. speaker_recog_deinit() and speaker_recog_preprocess_deinit() when done.