https://docs.japancv.co.jp/thundersdk/english-dev-documents

Thunder SDK is a development kit that allows you to build your own applications with various functions to Sense Thunder. It enables unique development that is optimal for stores and facilities, such as attribute detection function, UI change, and QR code reading function.

The Thing Specification Language (TSL) refers to a description file provided for the configuration between the end-side and the server, which is used to describe the configuration items supported by the end-side and the constraints corresponding to the configuration items.

Overview

The SenseLink platform supports devices to define the TSL. The data model consisting of the configuration (properties, status) of the actual device facilitates the management of the device through the cloud.

TSL

TSL is a JSON format file that is reported by the device when it first accesses SenseLink and is stored in the backend. As you open the details of a device on the web, the web can then dynamically load the TSL and TSL values corresponding to the device. This allows you to ensure consistency between the remote configuration function in the cloud and the local function of the device without upgrading the backend service.

TSL Usage Flow

1.Add TSL to the app and complete the creation based on TSL Data Structure 2.Create the required language pack. For its format, please see TSL internationalization. 3.After completing the development, you can report properties and events on the device, and set properties and invoke services on the cloud, and view the interface call steps Update Process 4.The attributes, events and completed service call data reported from the device will be displayed on the Device Settings tab of Device Details page of the corresponding device in the SenseLink console after passing the data verification.

TSL Structure

TSL is a json file containing "engineVersion", "modelVersion" at the top level: "0.0.1", "deviceType" basic information and related setting status "properties", "status" composition:

{
  "engineVersion": "0.0.1",//Engine version
  "modelVersion": "0.0.1", //Model version
  "deviceType": "sps", // Device type
  "properties": [/*... */], // Function configuration items
  "status": [/*...*/]// Device peripheral status
}

Properties

Properties is composed of several modules, each containing multiple set items. Each setting item, also known as fields element, is a description and function definition for each setting item.

{
  "properties": [{
    "name": "group-name",
    "fields": [{ // Collection of configuration items
      "key": { // Description of the name of configuration items
        "name": "product_customer", // Name
        "desc": "vendor-customer" // used to describe name, with no real purpose
      },
      "value": {} // Description of optional values for configuration items
      ]
   }]
}

Field is composed of key and value. Key describes the name of the configuration item, and Value is the definition including the optional value, style, status of the configuration item. The following is an explanation of some fields. For more configuration, please see [TSL data structure] (# TSL data structure).

type: required, the type of the configuration item value, which can be int, float, string, boolean and others

ranges: optional, to express the interval of the value when the type is int, float, etc., including min: minimum value, max: maximum value, default: default value.

ui: optional, default as text, used to indicate web/device input control style.

List of available values for ui:

options: optional, used when Value is exhaustible (such as wiegand32, wiegand34, and wiegand2602), generally used with select for ui.

Status

Refers to the status of a peripheral component of a device. The platform only reads and does not write, and only maintains the latest state. The JSON file contains name, enable, and options, corresponding to the name, display or not, and optional values. For more information, please see [TSL data structure] (# TSL data structure).

TSL Internationalization

Defines the display names of device types, attributes, and status of all TSL in the system on the device/web, and stores them separately by different languages. TSL internationalization is to explain the TSL values and names in layman's terms for the cloud console and reduce the difficulty of operation. Each language pack is composed of files in JSON format, and the corresponding Key is a translation for the corresponding part of the TSL.

For more information on data structure, please see Data structure of language pack.

Appendix

Update Process

Data structure of language pack

An example of the data structure of the language pack is as follows.

{
  "versionInfo": {
    "version": "0.0.1",
    "log": "",
    "updated": "2020-04-21 12:00:00"
  },
  "common": {
    "modelVersion": "Device model version"
  },
  "deviceTypes": {
    "sps": "SensePass"
  },
  "groups": {
    "general": "General Settings"
  },
  "fields": {
    "identify_mode": "identify_mode"
  },
  "values": {
    "enable": "Enable",
    "disable": "Disable"
  },
  "status": {
    "door_status": "Magnetism Status"
  }
}

TSL Data Structure

An example of the JSON field structure for TSL is as follows:

{
  "engineVersion": "0.0.1",//Engine version
  "modelVersion": "0.0.1", //Model version
  "deviceType": "sps",
  "properties": [{
    "name": "features",
    "fields": [{
      "key": {
        "name": "identify_mode",
        "desc": "Enabled State"
      },
      "value": {
        "required": true, // required or not. It’s non-required by default if not set.
        "type": "int", //data type, must be set, including numeric, boolean, and string.
        "editable": true, // Editable or not. It cannot be edited by default if not set, 
        /*
        ** UI components must be set with the following types
        *** Number picker number
        *** Drop-down box select
        *** Multi-select box checkbox
        *** Radio button radio
        *** Time picker date
        *** Single line text input text
        *** Multi-line text input textarea
        *** Image upload and display image
        *** Default is text
        */
        "ui": "select",
        "ranges": [{
          "min": 1,
          "max": 2,
          "enable": "{open_way}==2", //js expression string or boolean value. It is enabled by default if not set. The fields in curly braces indicate the value to be replaced, e.g. if the open_way value is 2, the expression is parsed as "2==2".
          "default": 1.5
        },{
          "min": 3,
          "max": 4,
          "enable": "{open_way}==3",//js expression string or Boolean value. It is enabled by default if not set.
          "default": 3.5
        }],
        "step": 0.5,// Step size, only for values
        "precision": 1, // Precision, indicates the exact number of decimal places, non-negative integer, and works only for numeric values, not required
        "mult": true, //multiple-choice or not, not required. The passing parameter is a single value by default if it’s not set. Multiple values only work for dropdown and multi-select boxes.
        "enable": "{open_way}==1", // js expression string or boolean value, not required, indicates the display and hiding of the field via linkage control. If the enable property is not set, it is available by default.
        "size": 20, // string length, not required, only works for string and image base64
        "timeFormat": "yyyy-MM-dd HH:mm:ss", //if ui is date, the time format must be filled in. hh means 12 hours, HH means 24 hours
        "options": [{
          "value": 0,
          "name": "off",
          "enable": "{open_way}==2", // js expression string or boolean value. Returns true, indicating that the selected value can be used in the candidate item. If the enable property is not set, it is available by default.
        }, {
          "value": 1,
          "name": "on", // js expression string or boolean value. Returns true, indicating that the selected value can be used in the candidate item. If the enable property is not set, it is available by default.
          "enable": "{open_way}==3"
        }],
        //watch: indicates status listening
        /*
        ** The following example indicates that the value of the current field is associated with the use_mode field.
        ** When the use_mode field changes, the value of the current field is equal to the value after the expression is executed "{use_mode}==2?1.5:{use_mode}==1?2:{recognition_distance}"
        */
        "watch":{
          "use_mode": "{use_mode}==2?1.5:{use_mode}==1?2:{recognition_distance}"
        }
      },
    }]
  }],
  "status": [{
    "name": "wifi",
    "enable": true, // js expression string or boolean value. Controls the display condition of this field. If the enable property is not set, it is available by default.
    "options": [{
      "value": 0,
      "name": "off",
      "enable": "{open_way}==2", // js expression string or boolean value. Returns true, indicating that the selected value can be used in the candidate item. If the enable property is not set, it is available by default.
    },{
      "value": 1,
      "name": "on",
      "enable": "{open_way}==1", // js expression string or boolean value. Returns true, indicating that the selected value can be used in the candidate item. If the enable property is not set, it is available by default.
    }],
  }]
}

The Thunder SDK Package (zip file) has the contents shown below.

• dev-documents

  • Development manual: English version of the developer's manual.

  • API reference: English version of the api introduction document for Thunder SDK.

  • Senselink connection: English version of link api developer's manual.

  • Introduction to Thing Specification Language: English version of Thing Specification Language developer's manual.

• model: It stores the algorithm model available to the public and selects the appropriate model to apply to the project according to the requirements. For more information on how to choose the model, please see Development manual.

• sample: a sample program that demonstrates the basic functionality of Thunder SDK. You need to copy relevant resources into the sample based on the access steps described in Development manual to compile and run normally.

• jniLibs: It stores the dynamic libraries of core algorithms for multiple devices and platforms available to the public and selects the appropriate dynamic library to apply to the project according to the requirements. For more information on how to choose the library, please see Development manual.

• license: It contains the test certificates that need to be used on different hardware platforms, which can only be used for testing and need to be applied separately when the product is officially released. For more information on how to use it, please see Development manual.

• thunder-android*.aar: Thunder SDK development kit. You need to copy the aar into the project at development time. For more information on how to use it, please see Development manual.

An introduction of Link API

Link API is a series of interface methods for cloud capability encapsulation of SenseLink platform (For JCV Senselink Cloud, please refer to https://docs.japancv.co.jp/senselink/ (Japanese only)). There are two major types of interface methods, one is encapsulation for the platform RESTful interface, and the other is encapsulation for the platform’s Mqtt message push mechanism. Users can use the cloud capability provided by the platform just like calling common Java methods.

Use of Link API

The Link API mainly provides two core types, namely, HttpApiClient and MqttApiClient, and by using the methods provided by these two types, direct communication with the SenseLink platform can be achieved. The brief introduction is as follows:

• HttpApiClient: For encapsulation of platform RESTful API, use Http protocol for communication, both are synchronized methods and need to be used in the worker thread;

• MqttApiClient: For encapsulation of the platform message push mechanism, use Mqtt protocol for communication, and receive platform message push by establishing a long connection and subscribing to the message Topic.

For the detailed definition of the RESTful API interface and Mqtt Topic involved in the interface method, please refer to the "Thunder SDK API Document".

Initialization of Link API

Link API provides a unified initialization entry, you need to set the Url of the server, the read and write timeout of the request, the language type returned by the interface, etc. The specific use is as follows:

// Initialize   
LinkSDKHelper.getInstance().init(“https://link.japancv.co.jp/sl/”, this)
         .setConnectTimeout(10 * 1000)
         .setReadTimeout(30 * 1000)
         .setWriteTimeout(10 * 1000)
         .setLanguage(LinkSDKHelper.LanguageTypeEnum.ZH)  // Set the language of return information from the server
         .addInterceptor(new ResponseInterceptor())  // Set interceptors
         .setDebug(true);

After initialization, specific methods can be called to perform specific services.

RESTful API encapsulation method

Each RESTful API interface encapsulates the corresponding Java method execution. Specific entities class was encapsulated by both the parameter passing value required by the request and the result data of the request. For example, when executing the v2/record request to upload the identification record

    // Parameter entity class
    RecordParameter recordParameter = new RecordParameter();
    recordParameter.setUserId(2547);
    recordParameter.setMode(2);
    recordParameter.setSignTime((int) (System.currentTimeMillis() / 1000));
    recordParameter.setType(1);
    recordParameter.setInTime(0);
    recordParameter.setEntryMode(1);
    Bitmap bitmap = BitmapFactory.decodeStream(mAppContext.getAssets().open("test.jpg"));
    recordParameter.setUsername("testUserName");
    // Error occurs when uploading incorrect base64 string
    recordParameter.setSignBgAvatar(Base64Utils.bitmapToBase64(bitmap));
    recordParameter.setSignAvatar(Base64Utils.bitmapToBase64(bitmap));
    recordParameter.setDocPhoto(Base64Utils.bitmapToBase64(bitmap));
    recordParameter.setIcNumber("235113");
    recordParameter.setIdNumber("460003199809086744");

    // Call the encapsulated method to execute the specific request. Result refers to the unified wrapper class for the result of the request
    Result<RecordResult> recordResultResult = HttpApiClient.uploadRecord(recordParameter);
    if(recordResultResult.isSuccess()){
        // Get the request result data and process it, RecordResult refers to the entity class of the request result data
            RecordResult body = recordResultResult.getBody(RecordResult.class);
        // todo: the execution of specific operations
  } else {
          // Request exception. Please see the API documentation for details
  }
}

Multiple parameters can be passed in directly to the API interface with relatively simple parameters for execution, specified as below:

• Device login operation

// Request parameters
// Synchronize method calls, which need to be executed in the work thread
Result<LoginResult> loginResult = HttpApiClient.login(
                                  "account", "password", "SPS", AppUtils.getAndroidId(ContextUtils.getContext()));

• Get detailed user information based on the user ID

int userId = 5;
Result result = HttpApiClient.getUserInfo(userId);

For more details on "RESTful API interface definition" and "correspondence between API and methods", please refer to the Thunder SDK API Document .

Mqtt push mechanism encapsulation method

Each Mqtt message push Topic encapsulates the corresponding Java method. Topic subscribed messages are received by registering callbacks, specified as below

• Establish a Mqtt long connection with the server, shown as below:

  コピー

  MqttApiClient.connectMqtt(new IMqttActionListener() {
  
       @Override
       public void onSuccess(IMqttToken iMqttToken) {
         // Connection successful
       }
  
       @Override
       public void onFailure(IMqttToken iMqttToken, Throwable throwable) {
         // Connection failed
       }
  
     }, new IConnectionLost() {
  
       @Override
       public void onConnectionLost(Throwable cause) {
                // Connection lost
       }
  });

• Subscribe to the personnel group change of the device

    // Subscribe to the personnel group change
    MqttApiClient.registerGroupChangeListener(new MessageCallback() {

       @Override
       public void success(MqttMessage msg) {

                 // json data is the body of the push message.
                  String json = new String(msg.getPayload());
              // Delete the device in the background and push an empty string without processing.
            if (TextUtils.isEmpty(json)) {
                    return;
            }

            // Get the data. Group class is the entity class corresponding to the message data.
            Pair<List<Group>, Boolean> pushGroupList = 
                                          DataConverter.convertJsonToGroupList(json);
            // Process data according to specific business
         }

       @Override
       public void error(int code, String msg, Throwable throwable) {
                         // Subscription exceptions / Message push exceptions
       }
    });

• Unsubscribe to the personnel group change of the device

    MqttApiClient.unRegisterGroupChangeListener();

For more details on "Mqtt Topic definition" and "correspondence between topic subscribe and methods", please refer to the Thunder SDK API Document.

Introduction of key use processes

Link API provides easy communication with Senselink platform, however, certain process steps need to be followed while using some functions, in other words, both the use of RESTful API interface and the subscriptions to MQTT topics follow a sequence. Below mainly describes the main functional flow steps from the perspective of the sequence of use.

Login and registration steps

Login and registration operations are required to use the functions of SenseLink platform, so that device can be included into the platform for unified management. A brief communication process is stated as follows:

The main steps are described below:

(1) Device login: requestHttpApiClient.getRsa() interface to obtain a public key from the platform first, then use the public key to encrypt the cleartext password, and then request HttpApiClient.login() interface for login operation. The two steps are completed by using the HttpApiClient.login () method, which only needs to pass in the account number, encrypted password, device type and device DUID as method parameters.

• token: all interface requests need to use token, which is valid for 7 days. After expiration, 403 error will be returned, in this case, you need to log in again to obtain it;

• newDeviceKey: If it is true, it means that the current device has not been included in the unified management of the platform, in this case, device registration operation needs to be performed.

(2) Device registration: the HttpApiClient.register() interface should be requested first for registration, and then the HttpApiClient.bindDefaultGroup() interface should be requested to bind the device and personnel group after successful registration (for subsequent personnel data distribution management). The two steps are completed by applying HttpApiClient.register () method, and parameters such as device name, device location, entrance and exit direction of access control need to be passed in.

• device ldid: the unique identification code of the device on the platform, which is used to distinguish device;

• companyId: the company id corresponding to the device, and each company has several personnel groups.

Synchronization strategy of object model

Object model related functions are only supported in SenseLink 2.3.0 and above versions. You need to determine the version number of the server before using these functions. On the premise that the server supports these functions, object model can be synchronized without unexpected exceptions. This operation should be placed before the configuration synchronization, the flow chart is shown in the figure below.

(1) Get the version number of the currently connected server by calling HttpApiClient.getServerVersion() to determine whether or not object model related functions are supported by the server. If the version number of the server is greater than or equal to 2.3.0, these functions are supported and you can continue the subsequent operations, otherwise skip directly to step (5);

(2) Calculate the MD5 of the local object model, and call HttpApiClient.checkTslExist() as an input parameter to get the flag zone bit and check whether the local object model exists in the cloud at the same time. When flag==1, it means that the object model already exists and does not need to be uploaded. Otherwise, call HttpApiClient.uploadTsl() to upload the object model;

(3) Call HttpApiClient.checkTslLanguageExist( language ) (zh: Simplified Chinese, zh-tw: Traditional Chinese, en: English) Check whether there is a corresponding language package in the cloud, and get the zone bit and MD5 of the cloud language package. When flag==1 and the MD5 of the cloud language package is consistent with the MD5 of the local language package, it means that the language package already exists and does not need to be uploaded. Otherwise, call HttpApiClient.uploadTslLanguage() to upload the language package;

(4) If there are multiple language packages, repeat step (3);

(5) Perform related operations for configuration synchronization.

Device configuration synchronization strategy

There are two ways to set device configuration parameters: set through the UI interface and set and issue via SenseLink cloud. The brief communication is shown in the figure below:

(1) After the user modifies the configuration through the setting page, the system setting module will upload the modified configuration parameters to the background, and finally broadcast the configuration change;

(2) When the configuration is modified in the background, it will send a configuration change push through MQTT. After receiving the notification, the system setting module will obtain the latest server configuration through HTTP, update the device configuration parameters, and finally broadcast the configuration change;

(3) When the device configuration is reset in the background, it will send a restore default configuration push through MQTT. The system setting module will reset all configurations after receiving the notification, upload the reset configuration to the background, and finally broadcast the configuration change.

Personnel data synchronization strategy

Personnel data synchronization means that the device side obtains the company's corresponding personnel data from the platform side, the data mainly includes the basic information of the personnel and the face photo library required for recognition. The brief communication process of data synchronization is as follows:

(1) After the device logs in successfully and establishes a long Mqtt connection with Mqtt Broker, you need to actively subscribe to the "Device Personnel Group Change Topic". After subscribing, you can immediately receive the message push and obtain all the personnel group information belonging to the device (the key field is the id of the personnel group). You can unsubscribe to the Topic after receiving the data;

(2) With all the personnel group ids obtained in (1), continue to subscribe to "Single personnel group under all personnel data Topic", after subscribing, all personnel information of the personnel group can be obtained (key fields are user_id and image_url). You can unsubscribe to the Topic after receiving the personnel information of all groups

(3) With the user_id and image_url obtained in (2), detailed personnel information and avatars can be obtained through the RESTFul interface. This step needs to be repeated to obtain all the personnel data;

(4) After initial acquisition of personnel data, it is necessary to subscribe to the "Topic of personnel data change under the single person group". After the subscription, the personnel increment list and the change type will be returned immediately, then you need to process the data correspondingly according to the change type.

Check whether the face calibration is correct

Description of functional types

CalibrationUtils

Description of Member Function

boolean checkCalibration(Bitmap rgbFace, Bitmap irFace)

Parameter

Return

true if the face calibration is correct, false if not

Exception

None

Thunder SDK Development Manual

Thunder SDK [hereinafter referred to as "SDK"] is an Android SDK based on AI face algorithm and its derived functions, and takes the intelligent hardware device [hereinafter referred to as the "hardware device"] as the running carrier. It can be used for rapid development of liveness detection, 1:N face identification, 1:1 Face Verification, face temperature measurement, mask detection, access control and other functions.

This document is intended to describe the integration steps of the SDK and how to use the key functions, mainly for development engineers.

SDK Business Capabilities Introduction

This SDK is used for offline identification, capturing faces through device cameras for face identification, and its general flow is shown in the figure below

Based on this generic process, it may be 1 vs 1 or 1 vs N in face matching, so the process is subdivided into 1:1 matching and 1:N identification. Both are used in different scenarios, and 1:1 is not the case where N is set to 1 in 1:N, as defined below.

Call Camera

Quickly access the camera and get the preview callback data, encapsulate methods to manage the camera lifecycle, and support monitoring of the camera status.

Call Thermal imaging system

• The characteristics of abstract Thermal imaging system used in traffic scenes, support Thermal imaging system preview and get the data callback, manage its lifecycle, monitor its status, and auto connection after hot plug.

• The Thermal imaging system models supported are Arrow, Guide 120 and Guide 256.

1:1 Face Verification

• The “1:1 Face Matching" is a process of proving "you are you", which means that you already have someone's identity information to determine whether the person's face information in the camera view matches the existing identity information.

• The “1:1 Face Matching" is an authentication operation, which is often used in scenarios requiring identity verification, such as real-name authentication in mobile carrier shop, bank, and authentication scenarios in stations, airports, hotels, and Internet cafes.

1:N Face Identification

The “1:N Face Identification” refers to registering the set of face images to be recognized into the local face feature library, and comparing them with the set of face features in the face feature library when the camera acquires the face information to derive the identification result.

• The “1:N Face Identification” is often used in scenarios that require multi-person identification, such as access control and attendance check-in for communities, buildings, schools, etc.

Mask Detection

The “Mask Detection” refers to the rapid detection of whether a person is wearing a mask by analyzing the face information captured by the camera, which is used in scenarios where masks are required, such as hospitals, manufacturing plants, and construction sites with large amounts of dust.

Server 1:N Identification

In the general process mentioned above, face matching can be performed either locally or remotely (via Http network requests). In the remote execution, the user needs to implement the comparison logic by himself, and the user can consider to achieve this function under the premise of the existing face identification backend server.

Door Access Function

The “Door Access Function" refers to the control function of the SDK for door access devices. Different hardware devices have different ranges of support for door access devices. To facilitate the description of the correspondence, the following concepts are defined:

• "Door Access Device" - as a collective term for all devices including hardware accessories included in the hardware device of SenseTime and peripheral devices that can be connected and used by the hardware expansion interface.

  • Hardware Accessories: including screen backlight, IR fill light,light sensor, distance sensor, buzzer,local relays, loudspeakers, force removal and alarm button;

  • Peripheral Devices: door open buttons, fire signals, door magnets, door bells, alarms, Wiegand card readers, network relays, Wiegand door open controllers, etc.

• “Door Access Function” - mainly refers to the use of "Door Access Device", such as opening the door with local relays, opening the door with Wiegand controller, turning off screen fill lights, controlling the doorbell ringing, and sensing the distance of human body with distance sensor.

SDK's support for the "Door Access Function" of different hardware devices varies, as follows (the following table is described only from the perspective of SDK (not hardware) adaptation:

For the purpose of description, this document is defined as follows:

• Pass Access Control Function: refers to the access control function used by the remaining devices in the Pass and Thunder series (the device models above).

Please pay attention to the following distinction between different devices when the use of the access control function is described.

QR Code Identification

The "QR Code Identification" refers to the function of parsing the content of QR code, which is used in the scenario of QR code identification. There are two codec libraries ZXing and ZBar built into the SDK, which can be selected by yourself when used.

Face Temperature Measurement

The "Face Temperature Measurement" means that through the face identification function and temperature measurement module, the face and thermodynamic diagram data are processed by the algorithm to get the final face temperature. Currently, the temperature measurement modules supported are Arrow module, Guide 120 module, and Guide 256 module. It should be noted that the final face temperature value may deviate depending on the ambient temperature, module model, and usage strategy.

Introduction to Devices Suitable for SDK

Introduction to Face Camera Imaging

For the purpose of illustrating camera imaging, the following definition is made.

• Horizontal Imaging: means that the width of the camera imaging picture is greater than the height.

• Vertical Imaging: means that the width of the camera imaging image is less than the height.

• Imaging Mirroring :refers to the camera image in which the real moving direction of the object is opposite to the direction in which the picture is presented. If a person walks to the left, he or she goes to the right in the preview screen.

• Face Orientation: refers to the direction that the face points to in the camera image, as shown below from left to right UP, DOWN, LEFT, RIGHT, in that order.

The following table indicates the imaging characteristics of the face camera corresponding to the specific device model.

Access Instructions

The SDK is provided as a.zip package file, which includes the.aar package,.so file, algorithm model file, sample project and development documentation. The SDK requires the following environment:

Start Integration

1. Copy the .aar file to the project libs folder, as shown in the figure below.

1. Select the corresponding model file in the "model file" folder of the zip package according to the device type, as shown in the figure below.

Copy the corresponding model file in the zip package to the actual project assets/model folder.

1. Select the corresponding so library in the "so library" folder of the package according to the device type, as shown in the table below

and copy the so in the folder to the /src/main/jniLibs folder.

And the final folder structure should be like this:

1. Add the following configuration to the build.gradle file

android {
     ...
     compileOptions {
         sourceCompatibility JavaVersion.VERSION_1_8
         targetCompatibility JavaVersion.VERSION_1_8
     }
}
dependencies {
    implementation files('libs/Thunder-android-vx.x.x.aar')
    // Third-party libraries that SDK needs
    implementation 'com.alibaba:fastjson:1.2.70'
    implementation 'com.facebook.conceal:conceal:2.0.1@aar'
    implementation 'com.google.zxing:core:3.3.3'
    implementation 'com.github.open-android:pinyin4j:2.5.0'
    implementation 'com.squareup.retrofit2:retrofit:2.9.0'
    implementation 'com.squareup.okhttp3:okhttp:4.8.0'
    implementation 'com.squareup.okhttp3:logging-interceptor:4.8.0'
    implementation 'com.squareup.retrofit2:converter-gson:2.4.0'
    implementation 'com.liulishuo.okdownload:okdownload:1.0.7'
    implementation 'com.liulishuo.okdownload:okhttp:1.0.7'
    implementation 'com.liulishuo.okdownload:sqlite:1.0.7'
}

1. Add the following permission declaration in AndroidManifest.xml file

<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.INTERNET"/>
<uses-permission android:name="android.permission.NFC" />

1. Copy the .lic certificate to the assets folder (there is a debug certificate in Sample, which is for testing purposes only, and a separate application is required for the official production environment certificate), as shown in the following sample image.

Sample Project

The Sample project can be opened in Android Studio. Copy aar, model, so and other files to the corresponding folder according to the documentation, and run debugging with pass series and thunder series of hardware devices. The Sample project mainly demonstrates 1:N identification, 1:1 matching, mask detection, access control and other functions.

SDK usage

In this section, the use of key api and processes in the SDK will be mainly described.

SDK initialization

Before using the SDK, please make sure that the aar, model, so and .lic files are properly stored and configured. Once the files are ready, the initialization can be performed as follows:

// SDK initialization
WuKong.init(this);

Authorization can be made after successful initialization:

// Specify the model used by the SDK. Here, take initialization on SensePass devices as an example. For more information, please see the Sample code.
model.mMode = SDKMode.PASS;
model.mAlignmentModel = "model/M_Align_occlusion_106_1.20.0.model";
model.mSmallDetectModel = "model/M_Detect_Hunter_LargeFaceSelfie_Gray_11.2.0.model";
model.mVerifyModel = "model/M_Verify_Mobilenetv2Pruned_BGR_Surveillance_4.13.0_v2_weak.model";
model.mDoubleHackModel = "model/pass_ext/M_Liveness_Antispoofing_Binocular_3.28.0.model";
model.mVerifyFinanceModel = "model/M_Verify_Mobilenetv2Pruned_BGR_Surveillance_4.13.0_v2_weak.model";
model.mAttributeModel = "model/M_Attribute_MTNet_2.1.1.model";

// The name of the license file
String licenseName = "sensepass.lic";
String productName = null;

WuKong.auth(licenseName, productName, model, new IAuthCallback() {
            @Override
            public void onSuccess() {
                // Authorization successful
            }

            @Override
            public void onFail(int code, String errMsg) {
                // Authorization failed
            }
        });

In general, there is no need to perform a release operation before the application exits after successful SDK authorization. Specifically, when it is necessary to restart the application, the SDK resources can be released at the end of the application, as follows:

// Release SDK resources
WuKong.release();

Call Camera

The CameraTextureView and CameraManager classes in the SDK make it easy to manage the camera life cycle and related preview operations. The following is an example of using a face camera (RGB) to introduce the use of the camera. CameraTextureView can be used in the layout file as follows (full screen preview).

<thunder.camera.view.CameraTextureView
        android:id="@+id/camera_view"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        app:scalableType="centerCrop" />

Then initialize the camera (Please check the following code for detail. 1280*720 preview resolution is highly recommended), as follows,

CameraTextureView mCameraView = findViewById(R.id.camera_view);
Camera.PreviewCallback mRgbCallback = new Camera.PreviewCallback() {
        @Override
        public void onPreviewFrame(byte[] data, Camera camera) {
            // Preview data callback, which can be passed to the Identify Manager
            // IdentifyManager.getInstance().handleRgbData(data);
        }
};

// Take setting up a face camera as an example
CameraManager faceCameraManager = CameraManager.getInstance(CameraUtils.getFaceCameraIndex());
// Add preview data callback
faceCameraManager.addPreviewCallbackWithBuffer(mRgbCallback);
// Open Camera
faceCameraManager.openCamera(getApplicationContext());
// Set preview-related parameters, such as preview width and height, and face orientation.
faceCameraManager.initParameters(1280, 720, 0, mCameraView.getSurfaceTexture());

Control Camera Lifecycle, as follows:

// Start Preview
faceCameraManager.startPreview();

// Stop Preview
faceCameraManager.stopPreview();

// Release Camera
faceCameraManager.releaseCamera();
faceCameraManager.setPreviewTexture(null);
faceCameraManager.removePreviewCallbackWithBuffer(mRgbCallback);
faceCameraManager.setCameraLifecycleCallback(null);
faceCameraManager.setCameraStatusCallback(null);
faceCameraManager.setCameraExceptionHandler(null);
faceCameraManager.setCameraDataUpdateCallback(null);

Call Thermal imaging system

Before calling the relevant interface of the thermal camera, get the instance of the thermal camera first. TemperatureCameraFactory is provided in the SDK to obtain different instances of the Thermal imaging system, shown as follows:

// Get Thermal imaging system Type
CameraType cameraType = TemperatureCameraUtils.getCameraType();
// Get the instance of the corresponding Thermal imaging system by camera type
ITemperatureCamera temperatureCamera = TemperatureCameraFactory.getTemperatureCamera(cameraType);

Initialize the Thermal imaging system configuration

Initialize Thermal imaging system
temperatureCamera.initCamera(context);
// Set the relevant parameters of the Thermal imaging system (needs to be executed after initialization)
temperatureCamera.setConfig(config);
// Set whether the shutter is enabled automatically. It is true by default (Enabling the shutter improves the accuracy of temperature measurement, but shortens the service life of the device)
temperatureCamera.setAutoShutterEnable(true);

Add Thermodynamic Diagram and Callback of Temperature Data

ITemperaturePreviewDataCallback dataCallback = new ITemperaturePreviewDataCallback() {

            @Override
            public void onTemperaturePreviewData(Bitmap temperatureBitmap, float[] temperatureValue, int width, int height) {
                    // Thermodynamic Diagram and Callback of Temperature Data
            }
        };
// Add Data Callback
temperatureCamera.addTemperaturePreviewDataCallback(dataCallback);
// Remove the callback when the data are not in use
temperatureCamera.removeTemperaturePreviewDataCallback(dataCallback);
// It is worth noting that the temperature data does not refer to Celsius degrees, which can be obtained from the temperature data by the following method
temperatureCamera.getCentigradeFromTemperatureData(temperatureValue);
// Alternatively, it can be converted by batch
temperatureCamera.matrixTemperature(temperatureValue,width,height,distance,x,y,w,h,tempArray);

Get the partial status of the Thermal imaging system

// Get Body Temperature of Thermal imaging system
temperatureCamera.getCameraBodyTemperature();

Manage the lifecycle of the Thermal imaging system

// Start Preview of Thermal imaging system
temperatureCamera.startPreview(textureView);
// Stop Preview of Thermal imaging system
temperatureCamera.stopPreview();
// Release Camera Resources of Thermal imaging system
temperatureCamera.releaseCamera();

In addition, the SDK provides a number of methods to operate the Thermal imaging system, shown as below:

// Enable Shutter (Enabling the shutter improves the accuracy of temperature measurement, but shortens the service life of the device)
temperatureCamera.shutter();

1:N Face Identification Flow

The business process of face identification contains several steps such as Camera management, face feature library creation, liveness detection, face comparison, and result processing, which involves many details of SDK usage. The following article introduces the details of SDK usage, taking the process of "1:N identification" as an example.

Create Face Feature Library

The "1:N identification" requires a face feature library as the "database" for face feature retrieval, so the face feature library needs to be built before the process starts. The SDK has FeatureManagerProxy for feature library creation, which is briefly used as follows.

// Frontal Clear Single Photo
Bitmap avatar = BitmapFactory.decode(R.id.avatar);
int userId = 123;

if (avatar != null) {
      // Get Features of the Face in the Picture
      byte[] feature = FeatureManagerProxy.getInstance().getFeature(avatar);
      // Insert Feature Library (native memory). Data persistence requires processing by upper business.
      int result = FeatureManagerProxy.getInstance().insert(userId, feature);
      if (result == 0) {
            Log.d(TAG, "Face Feature Inserted Successfully");
      }
}

// Delete Person Characteristics
int result = FeatureManagerProxy.getInstance().delete(userId);
if (result == 0) {
      Log.d(TAG, "Face Feature Deleted Successfully");
}

N face features (a maximum of 5W features) can be inserted in the feature library. After the construction is completed, the Identify Manager automatically obtains the inserted facial feature data of 'FeatureManagerProxy' during the identification process for personnel identification.

Start Camera Preview

Please check Call Camera section for detail to turn on the RGB camera. If liveness detection is enabled, you also need to turn on the IR camera again and modify the preview data callback to pass the preview data into the Identify Manager, as follows:

Camera.PreviewCallback mRgbCallback = new Camera.PreviewCallback() {
        @Override
        public void onPreviewFrame(byte[] data, Camera camera) {
            // Transfer RGB Preview Data Callback to the Identify Manager
            IdentifyManager.getInstance().handleRgbData(data);
        }
};
Camera.PreviewCallback mIrCallback = new Camera.PreviewCallback() {
        @Override
        public void onPreviewFrame(byte[] data, Camera camera) {
            // Transfer IR Preview Data Callback to the Identify Manager
            IdentifyManager.getInstance().handleIrData(data);
        }
};

Start the Identify Manager

The Identify Manager has built-in "1:N", "1:1" and "server 1:N" identification processes, and you can switch the internal identification logic by switching the Identify Manager mode. The Identify Manager modes are as follows.

• VerifyModeEnum.MODE_1_N: 1:N identification mode, corresponding to 1:N identification process.

• VerifyModeEnum.MODE_1_1: 1:1 comparison mode, corresponding to 1:1 comparison process.

• VerifyModeEnum.MODE_SERVER_1_N: server 1:N identification mode, corresponding to the server 1:N identification process;

Before use, set the configuration as follows:

 // Please configure the image related settings according to the device hardware environment
ImageConfig imageConfig = new ImageConfig.Builder()
          .previewW(1280)
          .previewH(720)
          .faceOrientation(FaceOrientation.UP)
          .pixelFormat(MidPixelFormat.NV21)
          .build();


// Initialize
IdentifyManager.getInstance().init(imageConfig,identifyCallback);

// Set Identification Mode of the Identify Manager
IdentifyManager.getInstance().setVerifyMode(VerifyModeEnum.MODE_1_N);
// Set Result Callback
IdentifyManager.getInstance().setIdentifyCallback(identifyCallback);

After the initial setup, the Identify Manager can be turned on/off to control identification, as follows

// Enable Identification
IdentifyManager.getInstance().start();

// Stop Identification
IdentifyManager.getInstance().stop();

// Release the Identify Manager
IdentifyManager.getInstance().release();

The identification process can control the details through the parameters, and the Identify Manager parameters can be configured individually according to the specific business, as follows. (In the code annotation below, if there are notes for a series different from the pass and thunder series, please skip them as appropriate.)

IdentifyConfig identifyConfig = IdentifyManager.getInstance().getIdentifyConfig();
IdentifyConfig.hasLiveness = true; // Enable liveness detection
identifyConfig.isSingleLiveness = DeviceInfoUtils.isSenseIDProduct();   // For monocular liveness detection, this item should be enabled only for ID series devices
identifyConfig.isSenseGateConfig = DeviceInfoUtils.isSenseGateProduct();  // Turn this on only if the device used is SenseGate series
identifyConfig.isContinueVerify = true; // IdentifyConfig.isContinueVerify = true; / / Whether to recognize continuously. If enabled, it will be recognized continuously (the face needs to be in the camera field of view). It is used together with the continueVerifyIntervalTime parameter.
identifyConfig.continueVerifyIntervalTime = 3000L;  // Interval for continuous identification of the same face, which takes effect when isContinueVerify is set to true
identifyConfig.hasTemperatureDetection = false;
identifyConfig.isServerVerify = false;
IdentifyManager.getInstance().setIdentifyConfig(identifyConfig);

// The modifications are shown here - In 1:N matching, the matching threshold is changed to 0.83f and the liveness filtering threshold is 0.95f (if exceeded, it is non-liveness), with a minimum identification of 100 pixels of the face
ThresholdConfig thresholdConfig = IdentifyManager.getInstance().getThresholdConfig();
thresholdConfig.verifyScore = 0.83f;
thresholdConfig.livenessScore = 0.95f;
thresholdConfig.faceMinWidth = 100;
thresholdConfig.verifyAreaRect = new RectF(0, 0, mPreviewHeight, mPreviewWidth);
IdentifyManager.getInstance().setThresholdConfig(thresholdConfig);

Process Identification Result

The Identify Manager transmits the identification result to the upper business by the way of interface callback, and the upper business only needs to process the result in the callback interface, shown as follows:

IIdentifyResultCallback identifyCallback = new IIdentifyResultCallback() {

    @Override
    public void onDrawFaces(@Nullable List<FaceInfo> list) {
        // Face Detection Callback, which can be used to draw a face calibration frame.
        // Please note that the callbakc occurs when identification is paused, which means that faces are still being tracked in the process, but just not recognized
    }

    @Override
    public void onClearScreen() {
        // Clear Screen Callback. Clear the screen with this callback when there appears no face in the field of view
    }

Please note that the callbakc o
    public void onUnknown(float faceW) {
                // If the face is too large or too small to determine whether it is liveness or not, notice is given with this method
    }

    @Override
    public void onIdentifyResult(List<FrameIdentifyResult> frameIdentifyResults) {
                // Identify Result Callback
    }
};

// Set Result Callback
IdentifyManager.getInstance().setIdentifyCallback(identifyCallback);

When the result callback is not required, set null as follows:

// Remove Identify Callback
IdentifyManager.getInstance().setIdentifyCallback(null);

At this point, the process of "1:N identification" has been preliminarily built.

1:1 Face Verification Process

In the above description, the Identify Manager in the "1:N identification" process uses the mode VerifyModeEnum.MODE_1_N, which is the "1:N identification" mode. In the face "1:1 matching" process, the Identify Manager needs to be switched to "1:1 matching" mode, as follows:

// Initialization of the Identify Manager is the same as that of 1:N
// When the Identify Manager is switched from 1:N to 1:1, it is necessary to set the single identification feature first, and then switch the mode

Bitmap singleTarget = BitmapFactory.decodeResource(getResource(), R.mipmap.face);
boolean setupResult = IdentifyManager.getInstance().setIdentifyTarget(singleTarget);
// Set Identification Mode of the Identify Manager. Before setting it, make sure that the individual identification feature is set successfully.
if (setupResult) {
    IdentifyManager.getInstance().setVerifyMode(VerifyModeEnum.MODE_1_1);
}

Server 1:N Identification

When the identification step is executed on the remote server, you need to modify the configuration of the Identify Manager to enable the server identification based on 1:N identification, and then use the IServerVerifyAction class to implement the identification logic. The specific use is as follows:

// 1. Enable Server Verify Mode 
IdentifyManager.getInstance().setVerifyMode(VerifyModeEnum.MODE_SERVER_1_N);

// 2. Implement Server Verify Action
IServerVerifyAction serverVerify = new IServerVerifyAction() {
        @Override
        public <T extends FaceSearchResult> T verify(@NonNull byte[] cameraData, byte[] feature, @Nullable FaceInfo rgbFace,int width, int height) {

              // Upload Data via http Request and Get Identification Result (note to set a reasonable request timeout).
            // ..code

            // Build Face Search Result and Return It, as shown in the following example
            FaceSearchResult result = new FaceSearchResult();
            result.score = 0.93f;
            result.userID = 12345;

            return (T) result;
        }
};

// 3. Set Server Verify Action
IdentifyManager.getInstance().setServerVerifyAction(serverVerify);

Mask Detection

Based on the identification of "1:N", "1:1" and "server 1:N", the corresponding mask detection results can be obtained from the identification results by configuring to open the mask detection. If the mask detection function is not enabled, the default mask detection result is true, enabled as follows:

// Open Wear Mask Identify
IdentifyConfig identifyConfig = IdentifyManager.getInstance().getIdentifyConfig();
identifyConfig.isOpenWearMaskIdentify = true;
IdentifyManager.getInstance().setIdentifyConfig(identifyConfig);

// Get Face Identify Result Through Identify Result Callback Interface
IIdentifyResultCallback identifyCallback = new IIdentifyResultCallback() {

    @Override
    public void onDrawFaces(@Nullable List<FaceInfo> list) {
        // Face Detection Callback, which can be used to draw a face calibration frame.
        // Please note that this callback still occurs when identification is paused, which means that faces are still being tracked in the process, but just not recognized
    }

    @Override
    public void onClearScreen() {
        // Clear Screen Callback. Clear the screen with this callback when there appears no face in the field of view
    }

    @Override
    public void onUnknown(float faceW) {
                // If the face is too large or too small to determine whether it is liveness or not, notice is given with this method
    }

    @Override
    public void onIdentifyResult(List<FrameIdentifyResult> frameIdentifyResults) {
                // Identify Result Callback
      for (int i = 0; i < frameIdentifyResults.size(); i++) {
            FrameIdentifyResult frameIdentifyResult = frameIdentifyResults.get(i);
            List<FaceIdentifyResult> faceIdentifyResults = frameIdentifyResult.faceIdentifyResults;
            for (int j = 0; j < faceIdentifyResults.size(); j++) {
                FaceIdentifyResult faceIdentifyResult = faceIdentifyResults.get(j);
                  // Get Face Identify Result
                  boolean mask = faceIdentifyResult.isMask();
            }
        }
    }
};

// Set Result Callback
IdentifyManager.getInstance().setIdentifyCallback(identifyCallback);

Face Temperature Measurement

Based on the identification of "1:N", "1:1" and "server 1:N", the temperature measurement configuration item of the Identify Manager needs to be enabled before the face temperature measurement function is used, as shown below

IdentifyConfig identifyConfig = IdentifyManager.getInstance().getIdentifyConfig();
identifyConfig.hasTemperatureDetection = true;    // Enable Temperature Detection
IdentifyManager.getInstance().setIdentifyConfig(identifyConfig);

It should be especially noted that the Identify Manager needs temperature data to work properly after the temperature measurement function is turned on, and no identification will be made before the temperature data is obtained, but face tracking will be carried out normally. In addition, when the temperature measurement function is turned on, it is recommended to set the rectangle for face recognition and a stricter angle of the front face to try to identify the face in the center of the field of view, so as to increase the accuracy of face identification and temperature measurement, as follows

ThresholdConfig thresholdConfig = IdentifyManager.getInstance().getThresholdConfig();
thresholdConfig.faceMinWidth = 200; // In the temperature measurement, it needs to be closer, and the width of the face needs to be set larger.
thresholdConfig.pitch = 20;   // In the temperature measurement, the requirements are more stringent, and face facing forward helps to measure the temperature more accurately
thresholdConfig.roll = 20;
thresholdConfig.yaw = 20;
thresholdConfig.verifyAreaRect = new RectF(125, 400, 565, 835); // Take the SensePass device as an example       
IdentifyManager.getInstance().setThresholdConfig(thresholdConfig);

The temperature measurement function requires a temperature measurement module (Thermal imaging system), which can be initialized by obtaining the corresponding Thermal imaging system object through the TemperatureCameraFactory class. Note that the Thermal imaging system takes several seconds to initialize, as follows

// Get Camera Type
CameraType type = TemperatureCameraUtils.getCameraType();
// Create Thermal imaging system Object
ITemperatureCamera mTemperatureCamera = TemperatureCameraFactory.getTemperatureCamera(type);
// Initialize
mTemperatureCamera.initCamera(this);

After the temperature data are obtained, the temperature value needs to be calibrated by the temperature measurement algorithm, which needs to be initialized manually, as follows

// Initialize Temperature Comparison Algorithm (for calibration temperature)
TemperConvertCallback temperConvertCallback = new TemperConvertCallback() {

            private float[] tempArray;

            @Override
            public float temperConvertCentigrade(float temper) {

                return TemperatureCameraFactory.getTemperatureCamera(type).getCentigradeFromTemperatureData(temper);
            }

            @Override
            public float[] getTempMatrix(float[] y16Frame, int y16W, int y16H, float distance, int x, int y, int w, int h) {

                int size = w * h;
                if (tempArray == null || tempArray.length != size) {
                    tempArray = new float[size];
                }
                TemperatureCameraFactory.getTemperatureCamera(type).matrixTemperature(y16Frame, y16W, y16H, distance, x, y, w, h,                                         temperConvertCallback = tempArray);
                return tempArray;
            }
        };
// Note: different devices have different versions of the algorithm
// Currently, Thunder-E supports:TempMeasureVersion.TempMeasure_1_2_0、TempMeasureVersion.TempMeasure_1_5_0、TempMeasureVersion.TempMeasure_1_6_0、TempMeasureVersion.TempMeasure_1_7_0
// Thunder Air supports：TempMeasureVersion.TempMeasure_1_7_0、TempMeasureVersion.TempMeasure_1_8_0
// Thunder mini supports：TempMeasureVersion.TempMeasure_1_5_0、TempMeasureVersion.TempMeasure_1_11_0、TempMeasureVersion.TempMeasure_1_10_0、TempMeasureVersion.TempMeasure_1_3_0
// For details, please refer to the implementation in TemperatureMeasureVersionUtils.getTempMeasureVersionByDevice()
TempMeasureVersion algorithmVersion = TemperatureMeasureVersionUtils.getTempMeasureVersionByDevice();
int ret = TemperatureActionProxy.init(algorithmVersio, temperConvertCallbackn);
if(ret == 0) {
        Log.d(TAG, "Temperature Measurement Algorithm Initialized Successfully");
}

Set Temperature Sdk Action to the Identify Manager,

ITemperatureSdkAction action = new ITemperatureSdkAction() {

            @Override
            public <T extends TrackAndTemperatureResult> List<T> temperCalculateAir(MatchFace[] matchFaces, @Nullable Bitmap temperatureBitmap, float[] temperatureData, int temperatureWidth, int temperatureHeight, FaceOrientation temperatureOrientation) {
                // Ambient temperature of equipment in use 
                // when CameraType is CameraType.GUIDE120 or CameraType.IRAY，Examples code
                   MatchFace matchFace = matchFaces[0];
                    float envTemperature = 25.1f;
                    TemperInfo temperInfo = TemperatureActionProxy.temperCalculateObtainRect(matchFace.rgbFace, temperatureData,
                                                                                     temperatureWidth, temperatureHeight, FaceOrientation.UP.getValue(), envTemperature, 0.1f);
                    TrackAndTemperatureResult trackAndTemperatureResult = new TrackAndTemperatureResult();
                    trackAndTemperatureResult.setFaceInfo(matchFace.rgbFace);
                    trackAndTemperatureResult.setIrFaceInfo(matchFace.irFace);
                    trackAndTemperatureResult.setTemperature(temperInfo.temper);

                    List<T> ts = new ArrayList<>();
                    ts.add((T) trackAndTemperatureResult);
                    return ts;
            }
        };
            }
        };
// Set Temperature Sdk Action
IdentifyManager.getInstance().setTemperatureSdkAction(action);

Get the thermal camera preview data (thermodynamic diagram). Once the temperature data is acquired, it needs to be injected into the Identify Manager in time, as follows

ITemperaturePreviewDataCallback mPreviewDataCallback = 
          new ITemperaturePreviewDataCallback() {

          @Override
      public void onTemperaturePreviewData(
            Bitmap temperatureBitmap, float[] temperatureValue, int width, int height) {

                // temperatureBitmap - thermodynamic diagram, which can be used to draw a temperature bitmap preview
                // temperatureValue - Temperature Data
                // width, height - heat Width and height of Thermodynamic Diagram

                // Inject temperature data into the Identify Manager (if temperature data are not injected after the temperature measurement function is enabled, no recognition will be performed)
                IdentifyManager.getInstance().handleTemperatureData(
                  temperatureBitmap, temperatureValue,width, height, mCameraUseConfig.faceOrientation);
            }
        };

// Set Callback
mTemperatureCamera.addTemperaturePreviewDataCallback(mPreviewDataCallback);

Get personal temperature data from the identification callback results:

// Get Face Identify Result Through Identify Result Callback Interface
IIdentifyResultCallback identifyCallback = new IIdentifyResultCallback() {

    @Override
    public void onDrawFaces(@Nullable List<FaceInfo> list) {
        // Face Detection Callback, which can be used to draw a face calibration frame.
        // It should be noted that this callback still occurs when identification is paused, which means that faces are still being tracked in the process, but just not recognized
    }

    @Override
    public void onClearScreen() {
        // Clear Screen Callback. Clear the screen with this callback when there appears no face in the field of view
    }

    @Override
    public void onUnknown(float faceW) {
                // If the face is too large or too small to determine whether it is liveness or not, notice is given with this method
    }

    @Override
    public void onIdentifyResult(List<FrameIdentifyResult> frameIdentifyResults) {
                // Identify Result Callback
              for (int i = 0; i < frameIdentifyResults.size(); i++) {
            FrameIdentifyResult frameIdentifyResult = frameIdentifyResults.get(i);
            if (frameIdentifyResult == null
                    || frameIdentifyResult.faceIdentifyResults == null
                    || frameIdentifyResult.faceIdentifyResults.size() <= 0) {
                continue;
            }
            List<FaceIdentifyResult> faceIdentifyResults = frameIdentifyResult.faceIdentifyResults;
            for (int j = 0; j < faceIdentifyResults.size(); j++) {
                FaceIdentifyResult faceIdentifyResult = faceIdentifyResults.get(j);
                if (faceIdentifyResult == null || faceIdentifyResult.getIdentifyResultTypeEnum() == null) {
                    continue;
                }
                TrackAndTemperatureResult trackAndTemperatureResult = faceIdentifyResult.getTrackAndTemperatureResult();
                float temperature = 0;
                if (trackAndTemperatureResult != null) {
                    temperature = trackAndTemperatureResult.getTemperature();
                }
                Log.i(TAG, String.format(Locale.getDefault(), "userId-%d temperature is %f",
                                         faceIdentifyResult.getUserId(), temperature));
            }
        }
    }
};

// Set Result Callback
IdentifyManager.getInstance().setIdentifyCallback(identifyCallback);

Use of Pass Access Control

The use of the Pass access control feature requires a global initialization configuration, which is initialized as follows.

// Pass access control function initialization, must be called before use of the access control function
DoorDeviceAccessProxy.init();

final DoorAccessConfig config = new DoorAccessConfig.Builder()
        // Set GPIO B port input type, which can be set to GPIO_IN_NONE by default
        .setGpioInB(DoorAccessConfig.GPIO_IN_NONE)
        // Set GPIO C port input type, which can be set to GPIO_IN_NONE by default
        .setGpioInC(DoorAccessConfig.GPIO_IN_NONE)
        // Set Open Door Time, only for local relays (here set to close after 5 seconds of opening)
        .setOpenDoorTime(5)
        // Set GPIO Output Type
        .setGpioOutA(DoorAccessConfig.GPIO_OUT_NONE)
        // Control RS485 data reading and writing by the Rom itself (Rom support is required. Please set it to false if Rom does not support)
        .setIsSupportRS485GpioAutoControl(true)
        // Set Wiegand Input Protocol Type
        .setWiegandInput(DoorAccessConfig.WIEGAND26_24_BIT)
        // Set Open Door Mode
        .setOpenDoorMode(DoorAccessConfig.OPEN_DOOR_MODE_RELAY)
        .Setthe Serial Port Baud Rate for RS485 Protocol
        .setRS485BaudRate(9600)
        .build();
DoorDeviceAccessProxy.setConfig(config);

The configuration for using the Pass access control function can be dynamically modified by DoorDeviceAccessProxy.setConfig(config).

Key categories of Pass access control function:

• DoorDeviceAccessProxy: responsible for global initialization, resource release, configuration usage, door opening control.

• DoorAccessConfig: Categories of Door Access Configuration.

• PassDoorDeviceAccessProxy: Manage access control functions available to SensePass/SensePassPro devices, such as door chimes, RS485 protocols, alarms, and tamper alarm lights.

Open Door via Local Relay

The "local relay" is a hardware device with its own accessories, used to control the opening of the door and set the door opening time (i.e. open the door for a number of seconds and then close it). The hardware models supporting this function are SensePass, SensePassPro.

The Pass access control function is used as follows:

// Set Open Door Mode to "Local Relay"
DoorAccessConfig mDoorAccessConfig = DoorDeviceAccessProxy.getConfig();
mDoorAccessConfig.setOpenDoorMode(DoorAccessConfig.OPEN_DOOR_MODE_RELAY);
DoorDeviceAccessProxy.setConfig(mDoorAccessConfig);

// This method is a global door opening method, which will automatically execute the door opening action according to the door opening method set in the config
// The IC card number is only used when the input parameter is Wiegand Open Door. Transfer it directly here. “
DoorDeviceAccessProxy.openDoor("");

DoorDeviceAccessProxy.openDoor() automatically acquires the configured door opening method and the delayed closing time, and then automatically executes the door opening action. The user does not need to care about the specific way to open the door, which is automatically controlled by the method. The same is true of door openning with the network relay and the Wiegand controller below.

Specifically, when the delayed closing time needs to be greater than 30 seconds, it is necessary to open the door in the following ways

// Open the door until opDoorTime time before closing the door. The delayed opening time is directly determined by
DoorDeviceAccessProxy.openDoorWithRelay(openDoorTime * 1000);

If the local relay needs to be closed immediately, the following method can be called manually.

DoorDeviceAccessProxy.closeRelayDoor();

Open Door via Network Relay

The "Network Relay" is a peripheral device. The hardware device connects to the network relay through the network, and controls the door opening and delayed closing time by sending door opening commands to the network relay. The hardware models supporting this function are Pass and Thunder series.

The Pass access control function is used as follows:

DoorAccessConfig mDoorAccessConfig = DoorDeviceAccessProxy.getConfig();
// Just fill in the IP, the port has been defaulted to 12345. If it was set during initialization, no need set repeatedly here
mDoorAccessConfig.setRelayIp("192.168.12.11");
// Set Open Door Mode as Network Relay
mDoorAccessConfig.setOpenDoorMode(DoorAccessConfig.OPEN_DOOR_MODE_IP_RELAY);
DoorDeviceAccessProxy.setConfig(mDoorAccessConfig);

// The parameter is the delayed closing time, which can be set freely
DoorDeviceAccessProxy.openDoor("");

Open Door via Wiegand Controller

The "Wiegand Controller" is a peripheral device that connects to the hardware device through an expansion interface and uses the Wiegand protocol for data communication, which can be used to control the door opening. SDK currently supports Wiegand 26, Wiegand 32 and Wiegand 34, and the hardware models that support this function are Pass and Thunder series device.

The maximum card number supported by the standard Wiegand 26 protocol is 8 bits, and the card number data transmission is divided into HID and PID. According to the generation mode of HID and PID, Wiegand 26 is subdivided into two modes, "Wiegand 26 (24bit)" and "Wiegand 26 (8+16bit)”, whose difference is as follows:

• Wiegand 26 (24bit): For ordinary use mode, the card number is converted to int, divided by 2^16 to take an integer as HID, and the remainder of 2^16 is taken as PID.

• Wiegand 26 (8+16bit): For special use mode, the first 3 digits of the card number are converted to int as HID, and the last 5 digits are converted to int as PID.

The Pass access control function is used as follows:

• Open Door via Wiegand 26

String cardNumver = "12345678";
// Open Door via Wiegand 26 (24bit)
DoorDeviceAccessProxy.openDoor(cardNumber);

• Open Door via Wiegand 32/34

String cardNumver = "1234567890";
DoorDeviceAccessProxy.openDoor(cardNumber);

Wiegand card reading

The "Wiegand card reading" needs to be configured with a Wiegand reader, which is a peripheral device. It connects with the device through the extended interface and uses the Wiegand protocol for data communication to read the IC card number. It supports Wiegand 26 (consistent with the above description), Wiegand 32 and Wiegand 34. The hardware device models supporting this function are Pass and Thunder series device.

The Pass access control function is used as follows:

DoorAccessConfig config = DoorDeviceAccessProxy.getConfig();
// This is set to the Wiegand 26 standard card reader
config.setWiegandInput(DoorAccessConfig.WIEGAND26_24_BIT);
DoorDeviceAccessProxy.setConfig(config);

Once the setup is complete, you can listen to the Wiegand read callback. The code is used as follows.

DoorDeviceAccessProxy.setCardReaderCallback(
            new WiegandReaderDevice.CardReaderCallback() {

    @Override
    public void onRead(String cardNumber) {
        // The card number is read successfully, and the cardNumber is the card number
    }
});

GPIO Input Devices

"GPIO input devices" refer to a class of devices that send GPIO signals to hardware devices through the GPIO port connection, including door opening buttons, fire signals and door magnets. When used, the hardware device serves as the receiver and the "GPIO input device” serves as the sender, and the two communicate through the GPIO input port. The hardware device models that support this feature are Pass and Thunder series device.

Pass access control function supports the use of dual GPIO, divided into B and C. configuration. Before use, set the relevant access control configuration, and its code is as follows:

DoorAccessConfig mDoorAccessConfig = DoorDeviceAccessProxy.getConfig();
// Here it is assumed that the peripheral device is connected to port B to connect to the door button. If you use port C, please set doorAccessConfig.setGpioInC()
mDoorAccessConfig.setGpioInB(DoorAccessConfig.GPIO_IN_DOOR_BUTTON);
DoorDeviceAccessProxy.setConfig(mDoorAccessConfig);

Once the setup is complete, you can listen to the relevant signal input. The code is used as follows.

// When Using Door Button
PassDoorDeviceAccessProxy.setOnClickDoorButtonListener(
                new DoorButtonDevice.DoorButtonOnClickListener() {

        @Override
        public void onClick() {
            // Receive Door Opening Signal
        }
});

// When Using Receive Fire Sign Listener
PassDoorDeviceAccessProxy.setReceiveFireSignListener(
                new FireSignalDevice.ReceiveFireSignListener() {

        @Override
        public void onReceive() {
            // Receive Fire Signal
        }
});

// When using door sensors
PassDoorDeviceAccessProxy.setDoorStateCallback(
                new PassDoorDeviceAccessProxy.DoorMagnetismStateCallback() {

        @Override
        public void onDoorStateChanged(int state) {
            if (state == '0') {
                // Door Magnetism State- Off
            } else {
                // Door Magnetism State- On
            }
        }
});

GPIO Output Devices

"GPIO output devices" refers to peripheral devices connected through GPIO output port for receiving GPIO signals from hardware devices, including doorbell, and alarm. When used, the hardware device serves as the sending end and the "GPIO output type device" serves as the receiving end, and the two communicate through the GPIO output port. The hardware device models that support this feature are Pass and Thunder series.

Before use, set the relevant access control configuration, and its code is as follows:

DoorAccessConfig sgDoorAccessConfig = DoorDeviceAccessProxy.getConfig();
// It is assumed here that the doorbell is used
sgDoorAccessConfig.setGpioOutA(DoorAccessConfig.GPIO_OUT_DOOR_BELL);
DoorDeviceAccessProxy.setConfig(sgDoorAccessConfig);

// When the doorbell rings, the parameter is the time (the actual ringing time may have errors, because the doorbell manufacturer may set the ringing according to the number of times, rather than the specific length of time)
PassDoorDeviceAccessProxy.pressDoorBell(2000);

Specifically, when using the alarm, you need to manually control the alarm and turn off the alarm. The code is used as follows:

// Alarm goes off
PassDoorDeviceAccessProxy.pressAlarmBell();
// Manually turn off the alarm
PassDoorDeviceAccessProxy.releaseAlarmBell();

Light Control

"Lighting Control" mainly controls the lighting accessories of hardware devices, including RGB fill light, IR fill light and screen backlight.

IR Fill Light

The "IR fill light" is part of the IR camera module and is used to enhance the IR image. To increase its lifetime, it can be dynamically controlled to turn off when not in use. The hardware device models that support this feature are Pass and Thunder series.

The code is used as follows:

// Turn on IR fill light
PassDoorDeviceAccessProxy.turnOnIrLight();

// Turn off IR fill light
PassDoorDeviceAccessProxy.turnOffIrLight();

Screen Backlight

The "screen backlight" is an accessory that comes with the hardware device to turn off the screen light when the software application is in a dormant state (such as late at night when the software application is no longer needed), increasing the life of the hardware device and reducing energy consumption. The hardware device models that support this feature are Pass and Thunder series.

The code is used as follows:

// Turn off screen backlight
PassDoorDeviceAccessProxy.closeBackLight();

// Turn on screen backlight
PassDoorDeviceAccessProxy.openBackLight();

RS485 Protocol

The "RS485 protocol" refers to a serial communication protocol standard. The RS485 interface is provided in the hardware device to support its communication in half-duplex mode, Half-duplex data read-only and write-only are controlled automatically by the rom, and the upper business only needs to pay attention to the serial communication. The hardware device models that support this feature are Pass and Thunder series.

Before use, first turn on the RS485 function, and set the baud rate, data buffer, and data transceiver timeout length according to the use scene.The code is used as follows:

// Start rs485, baud rate 9600, read buffer size 32, timeout 3s
PassDoorDeviceAccessProxy..enableRS485(9600, 32, 
            new RS485Device.RS485ReceiveListener() {

            @Override
            public void onReceive(byte[] data, long dataLength) {
                // Received data
            }

        }, 3000, new SerialPortReader.OnReadTimeOutListener() {

            @Override
            public void onTimeout(byte[] data, long dataLength) {
                // Data read timeout
            }
        }
});

To send data, use the following code:

PassDoorDeviceAccessProxy.sendDataByRS485(data);

Tamper alarm button

The "tamper alarm button” is an accessory that comes with the hardware device, located on the back of the device as a raised button. When the button is pressed during normal operation, the button will pop up when the hardware device is removed, and the Rom level will sense it and issue a "tamper alarm" for alerting when the hardware device is removed abnormally.

The Rom level converts the "tamper alarm" into an input event of the "F2 button", i.e. when the "tamper alarm" occurs, the system layer sends the "F2 button" input event. F2 button" input event, so you can listen to the "tamper alarm" by listening to whether the "F2 button" is pressed or not. The code is used as follows:

// Listen to it in the Activity via the onKeyUp() event
public class ForceDisassemblyDemoActivity extends AppCompatActivity {

    // Omit redundant code

    @Override
    public boolean onKeyUp(int keyCode, KeyEvent event) {
        if (keyCode == KeyEvent.KEYCODE_F2) {
            // Listen to tamper alarm
        }
        return super.onKeyUp(keyCode, event);
    }
}

Sensor

"Sensors" refers to the sensors that come with the hardware device, such as light sensors, and distance sensors. For the passage scenario, the role of sensors is as follows:

• Light sensor: senses light intensity and can set "fill light mode" for applications in low light conditions.

• Distance sensor: senses the proximity of a person to trigger certain specific operations (e.g. waking up a device in a dormant state when the person is close).

Sensors can be used through the SensorManager class in the Android native SDK, whose code is used as follows:

SensorManager mSensorManager = (SensorManager) getSystemService(Context.SENSOR_SERVICE);
// Use the light sensor. To use a distance sensor, use the parameter Sensor.TYPE_PROXIMITY
Sensor lightSensor = mSensorManager.getDefaultSensor(Sensor.TYPE_LIGHT);
if (lightSensor != null) {

    mSensorManager.registerListener(new SensorEventListener() {
        @Override
        public void onSensorChanged(SensorEvent event) {
            // Get the specific value via event.value[0] and do specific processing
        }

        @Override
        public void onAccuracyChanged(Sensor sensor, int accuracy) {

        }
    }, lightSensor, SensorManager.SENSOR_DELAY_NORMAL);

} else {
     // Sensor not available
}

Buzzer

The "buzzer" is an accessory that comes with the hardware device and is mainly used for beeping when an alarm occurs on the hardware device. Currently, only SensePass and SenseThunderE-mini devices support this feature. Using the buzzer requires manual control to turn it on and off, and the code is as follows:

// Turn on Buzzer
PassDoorDeviceAccessProxy.pressBuzzer();

// Turn off Buzzer
PassDoorDeviceAccessProxy.releaseBuzzer();

Application System Signature

The system signature file sensetime.jks is placed in the sample project of the SDK. Using this file as the application signature allows the application to gain system privileges, which allows the application to perform more operations that require high privileges, such as setting system events, and modifying system settings.

To use it, please copy sensetime.jks to the root of the main module of the project, and then add the following configuration to the build.gradle file.

android {

    signingConfigs {
        system {
            keyAlias 'sensetime'
            keyPassword 'sensetime'
            storeFile file('./sensetime.jks')
            storePassword 'sensetime'
        }
    }

    buildTypes {
        debug {
            signingConfig signingConfigs.system
            // ...
        }
        release {
            signingConfig signingConfigs.system
            // ...
        }
    }    
}

And add the system application declaration android:sharedUserId="android.uid.system" to the AndroidManifest.xml file, as follows:

<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    package="thunder.wksample"
    android:sharedUserId="android.uid.system">
</manifest>

QR Code Identification

The QR code recognition function can recognize ordinary QR codes normally. When using it, you can select ZXing or ZBar encoding and decoding library. It is recommended to use ZBar, which is more suitable for end-to-side device identification and faster. For more information on how to use it, please see the Sample code. The brief usage is as follows:

QRCodeDecoderProxy qRCodeDecoder = new QRCodeDecoderProxy(new ZBarDecoderImpl());
// previewData is the camera preview frame data
qRCodeDecoder.decodeQRCode(previewData, 1280, 720);

Description of functional types

CameraManager

The main methods of operating the camera

Description of Member Function

CameraManager addPreviewCallbackWithBuffer(PreviewCallback callback)

Add preview data callback

Parameter

Return

Current Object

Exception

None

CameraManager removePreviewCallbackWithBuffer(PreviewCallback callback)

Remove Preview Data Callback

Parameter

Return

Current Object

Exception

None

CameraManager getInstance(int cameraId)

Get camera instance

Parameter

Return

Current Object

Exception

None

CameraManager setDisplayOrientation(int degree)

Set Preview Degree, 0/90/180/270 selectable

Parameter

Return

Current Object

Exception

None

CameraManager setPreviewTexture(SurfaceTexture texture)

Set Preview Texture

Parameter

Return

Current Object

Exception

None

List getSupportedPreviewSizes()

Get the List of Supported Preview Sizes

Parameter

None

Return

Exception

None

void initParameters(int width, int height, int degree, SurfaceTexture surfaceTexture)

Camera Initialization

Parameter

Return

None

Exception

None

void openCamera(Context context)

Open Camera

Parameter

Return

None

Exception

None

void releaseCamera()

Release Camera

Parameter

None

Return

None

Exception

None

void startPreview()

Start Camera Preview

Parameter

None

Return

None

Exception

None

void stopPreview()

Stop Camera Preview

Parameter

None

Return

None

Exception

None

Description of Entities

None

Description of functional types

RgbAndIrCameraManager

The main methods to operate the camera in order to output video stream (combined RGB and IR).

Description of Member Function

void addPreviewCallbackWithBuffer(IRgbAndIrPreviewCallback callback)

Add preview data callback

Parameter

Return

None

Exception

None

void removePreviewCallbackWithBuffer(IRgbAndIrPreviewCallback callback)

Remove preview data callback

Parameter

Return

None

Exception

None

RgbAndIrCameraManager getInstance()

Get camera instance

Parameter

None

Return

Current object

Exception

None

List getSupportedPreviewSizes(CameraType cameraType)

Get the preview size supported by the camera

Parameter

Return

Exception

None

void initParameters(int width, int height, int degree,SurfaceTexture irSurfaceTexture, SurfaceTexture rgbSurfaceTexture, CameraType cameraType)

Camera Initialization

Parameter

Return

None

Exception

None

void openCamera(Context context,CameraType cameraType)

Open camera

Parameter

Return

None

Exception

None

void releaseCamera(CameraType cameraType)

Release camera

Parameter

Return

None

Exception

None

void startPreview(CameraType cameraType)

Start camera preview

Parameter

Return

None

Exception

None

void stopPreview(CameraType cameraType)

Stop camera preview

Parameter

Return

None

Exception

None

void resetPreviewSize(ScaleManager.Size size,CameraType cameraType)

reset the preview size supported by the camera

Parameter

Return

None

Exception

None

Description of Entities

CameraType

Parameter Description

Description of functional types

TemperatureCameraFactory

Get different instances for Thermal imaging system

Description of Member Function

ITemperatureCamera getTemperatureCamera(CameraType type)

Get Thermal imaging system

Parameter

Return

ITemperatureCamera

Current Type

Exception

None

ITemperatureCamera

Operate Thermal imaging system

Description of Member Function

boolean matrixTemperature(float[] y16Frame, int y16W, int y16H, float distance, int x, int y, int w, int h, float[] tempArray)

Batch Conversion of Temperature

Parameter

Return

Exception

None

float getCentigradeFromTemperatureData(float temperatureData)

Get Centigrade from Temperature Data

Parameter

Return

Centigrade Value

Exception

None

float getCameraBodyTemperature()

Get Camera Body Temperature

Parameter

None

Return

Camera Body Temperature

Exception

None

void addTemperaturePreviewDataCallback(ITemperaturePreviewDataCallback callback)

Add Thermal imaging system Preview Data Callback

Parameter

Return

None

Exception

None

void initCamera(Context context)

Initialize Thermal imaging system

Parameter

Return

None

Exception

None

void releaseCamera()

Release Thermal imaging system

Parameter

None

Return

None

Exception

None

void removeTemperaturePreviewDataCallback(ITemperaturePreviewDataCallback callback)

Remove Temperature Preview Data Callback

Parameter

Return

None

Exception

None

void startPreview(TextureView textureView)

Start Preview

Parameter

Return

None

Exception

None

void stopPreview()

Stop Preview

Parameter

None

Return

None

Exception

None

void setConfig(TemperatureCameraConfig config)

Thermal imaging system Configuration

Parameter

Return

None

Exception

None

void setAutoShutterEnable(boolean enable)

Set Auto Shutter Enable

Parameter

Return

None

Exception

None

void shutter()

Open Shutter

Parameter

None

Return

None

Exception

None

Description of Entities

ITemperaturePreviewDataCallback

Preview Data Callback

Description of Method

void onTemperaturePreviewData(Bitmap temperatureBitmap, float[] temperatureValue, int width, int height)

Temperature Preview Data Callback: Thermal imaging system outputs the image orientation as the same as its phisical orientation. For example, the human face will output with the vertical orientation.

Parameter

TemperatureCameraConfig

Thermal imaging system Configuration

Description of Properties

CameraType

Thermal imaging system Type(Enumeration Type)

Description of Properties

Description of functional types

QRCodeDecoderProxy

Description of Member Function

String decodeQRCode(byte[] nv21, int w, int h)

Parse the QR code

Parameter

Return

The parsed QR code string

Exception

None

String decodeQRCode(Bitmap bitmap)

Parse the QR code

Parameter

Return

The parsed QR code string

Exception

None

Description of Class

FeatureManagerProxy

Description of Member Function

int delete(int userId)

Delete face feature

Parameter

Return

Return 0 if delete operation is successful, otherwise return -1 if invalid parameters -260 if face feature library is empty -261 if can't find

Exception

None

byte[] getFeature(Bitmap avatar)

Get face feature of the image

Parameter

Return

Return face feature byte array

Exception

None

int insert(int userId, byte[] feature)

Insert feature library

Parameter

Return

Return 0 if the insert operation is successful, otherwise return -1 if invalid parameters -259 if face feature fails to add into libary -262 if can't get UUID

Exception

None

Description of functional types

WuKong

Description of Member Function

void init (Context context)

Initiation

Parameter

Return

None

Exception

None

void auth (String licFileName, boolean isLeafLic, String productName, PassModel modelConfig, IAuthCallback authCallback)

Authorization

Parameter

Return

None

Exception

None

void release ()

Release SDK resource

Parameter

None

Return

None

Exception

None

Rotate image

Description of functional types

RotateImageUtil

Description of Member Function

byte[] rotateImage(@NonNull byte[] resBuff, @NonNull MidPixelFormat pixelFormat, int width, int height, int degree)

Parameter

Return

byte[] of rotated image