JCV AR SDK(Effects) Product Description

JCV AR SDK provides special effects solutions for Pan entertainment industries such as live broadcasts and short videos, including face detection and tracking, attribute detection, front and rear background segmentation, gesture detection and tracking, beauty, makeup, 2D stickers, 3D special effects, face deformation, general object tracking, filters, and other functions. JCV AR SDK has the market-leading face key point detection and tracking technology: strong stability, stable tracking, less noise, and jitter; Excellent robustness: perfect support for large angles, large expression, extreme lighting, scene jitter, size, and other changes. JCV AR effects SDK can detect and track face keys in milliseconds. GPU processing makes the processing speed of face detection and tracking, beauty, stickers, filters, and other functions faster, and can achieve the effect of real-time and smooth preview in the mainstream model 720p in the market.

1. Face detection and tracking

1.1. Introduction to face detection and tracking

JCV AR SDK face detection and tracking model independently developed by Shangtang technology can quickly detect 106 key points of the face.

Face detection and tracking function have strong stability, stable tracking, and less noise and jitter:

Support large angle change scene of face

Support large facial expression change scenes

Support face size change scene

Support face occlusion scene

Support extreme light change scenarios

Support scene jitter

The face detection and tracking function can detect and track the 106 key points of the face, detect the occlusion information of the 106 key points, and output the occlusion information of the 106 key points of the face.

At the same time, it supports 7 kinds of expression detection. Expression actions can be used to trigger stickers:

blink, nod, shake the head, pick eyebrows, open mouth , laugh, pout

1.2. Face detection and tracking effect display

The face detection and tracking function can quickly detect and track 106 key points of the face.

The face detection and tracking function can identify the occlusion information of 106 key points while detecting 106 key points of the face. As shown in the figure, green points represent no occlusion and red points represent occlusion.

At present, we provide 16 kinds of gestures in the gesture detection part, including palm, thumb, boxing and bowing, holding hands, love, one hand heart to heart, OK, yeah, pistol, fingers, fist, two hands together, 666, I love you, three fingers and four fingers.

Support multi hand detection.

The usage of gesture detection is the same as that of face motion detection, and different sticker effects can be triggered.

The following figure shows the sticker effect triggered by hand holding action.

Before and after background segmentation is to detect the person and background in the image, and separate the person and background by detecting the relationship between the person and the background.

Shangtang's front and back background segmentation has accurate boundary segmentation and outstanding detection speed..

The front and rear background segmentation function can be used together with 2D stickers in the sticker function. The material can be made through the sticker widget. For the detailed steps of making, please refer to the document JCV ARSticker sticker material auxiliary tool.

The front and back background segmentation provides stroke effect at the same time, and the

stroke effect of any color and different thickness can be added to the character edge.

The segmentation effect of front and rear background is shown in the figure below:

Effect display of front and back background segmentation and 2D stickers.

4.1. Introduction to face attribute detection

JCV AR SDK adopts the face attribute detection model independently developed by Shangtang technology, which can detect the three attribute values of age, gender and face value of the current face, with high speed, accurate detection and excellent performance.

Note: the SDK contains the face attribute detection function, but this function is not displayed directly in the demo.

5.1. Beauty introduction

The beauty development kit is a function module of JCV AR. The beauty development kit makes use of the GPU processing function to preview the beauty effect in real time on the mainstream model 1080p in the market. The beauty SDK includes the mainstream beauty functions in the market, including skin grinding, whitening, ruddy, contrast, satiety, sharpening, thin face, small face, narrow face, big eye and round eye, Each function has an adjustable parameter interface.

5.2. Beauty function introduction

The beauty SDK is used to achieve various beauty effects, including skin grinding, whitening, ruddy, contrast, saturation, sharpening, clarity, thin face, high-level thin face, small face, narrow face, big eyes and round eyes. You can access the mobile phone or app to realize video streaming or image beauty processing. The details are as follows:

・Grinding - the grinding degree can be adjusted through parameter setting. The larger the

parameter value, the higher the degree of skin grinding, and the smoother the skin appears.

・Whitening - the whitening effect can be adjusted through parameter settings. The greater the

parameter value, the higher the degree of whitening, and the whiter the skin appears.

・Ruddy - you can adjust the ruddy effect through parameter settings. The larger the

parameter value, the higher the ruddy degree, which makes the face yellow and white become ruddy. At the same time, special treatment is made for the lip effect to achieve the effect of red lips and white teeth.

Contrast - the contrast effect can be adjusted through parameter settings. Saturation - you can adjust the saturation effect through parameter settings. Sharpening - the sharpening effect of the picture can be adjusted through parameter settings to improve the picture definition.

Clarity - you can adjust the picture tone through parameter settings to improve the picture clarity and texture.

Thin face - adjust the thin face effect by setting the parameters of thin face, which can thin the face in the video in real time, making the beauty effect more outstanding.

High level thin face - you can choose according to your face shape. For different thin face effects, support four different thin face adjustments: nature, goddess, long face and round face.

Small face - adjust the size of the lower part of the cheek by setting the parameters of the small face, which can naturally change the size of the cheek (chin length).

Narrow face - adjust the width of the cheek below the cheekbones by setting the parameters of the narrow face, which can change the width of the cheek more naturally.

Big eye - adjust the effect of big eye by setting the parameters of big eye, which can more naturally enlarge the proportion of face and eyes.

Round eye - adjust the round eye effect by setting the parameters of the round eye, which can naturally change the size of the eye.

5.3. Beauty effect display

Skin grinding, whitening and ruddy effect

The above figure is the comparison between the original photo and the full-function (skin grinding and whitening) treatment of softsugar beauty development kit. From the above comparison, it can be clearly seen that the skin of the photo after softsugar beauty treatment is delicate, white and tender, the lips are naturally pink and tender, and the details of the hair are well preserved.

The following figure is an enlarged view of the details of the hair after skin grinding. The enlarged comparison shows that we have well preserved the details of the hair.

Big eye thin face small face effect

The following figure shows the effect of big eyes, thin face and small face. Through our SDK, we can enlarge or reduce the eyes and fat and thin cheeks of the portrait in the picture. The effect is realistic and natural through smoothing.

The micro shaping function is a function module of JCV AR. The micro shaping function adopts Shangtang's independent detection function and uses the independently developed rendering engine to achieve the real-time preview of the mainstream model 720p in the market. Micro shaping functions include: small head, thin face, thin cheekbones, abundant, adjust the apple muscle, adjust the nose length, nose alar, side face nose augmentation, adjust the mouth, adjust the mouth shape, adjust the eye distance, adjust the brightness of the eyes, change the eye angle, open the inside and outside corner, remove the black eye and the law grain, adjust the tooth whitening degree, etc. Chin adjustment function can finely adjust the facial features to make them more delicate. The details are as follows:

Small head: adjust the size of the head to make the head smaller when the body size remains unchanged.

Nose length adjustment: adjust the length of the nose to make it longer or shorter.

Alar adjustment: narrowing the alar can narrow the alar.

regulation in the human: regulating the length of the human region, variable length or shorter.

Mouth shape adjustment: adjust the size of the mouth to be larger or smaller.

Chin adjustment: adjust the length of the chin, variable length or short.

Forehead adjustment: adjust the width of the forehead, variable width or narrowing.

Thin jaw: adjust the width of the jaw to narrow the jaw.

Thin face adjustment: adjust the face contour to narrow the face contour.

Side face nose augmentation: adjust the height of the nose under the side face to increase the bridge of the nose.

Open eye angle: adjust the angle of the inner eye angle to increase the angle of the inner eye angle.

Open the outer corner of the eye: adjust the angle of the outer corner of the eye to increase the angle of the outer corner of the eye.

Bright eyes: increase the brightness of the eyes and make the eyes more vivid.

Apple muscle: adjust the position of Apple muscle and reshape the shape of Apple muscle.

Dispel legal lines: eliminate legal lines.

Dispel dark circles: eliminate dark circles.

Tooth whitening: carry out tooth whitening to whiten the teeth.

Eye angle: adjust the eye angle.

Eye spacing: adjust the distance between two eyes to increase or decrease the distance between two eyes.

Thin cheekbones: adjust the width of cheekbones to narrow them.

The body plastic function is a function module of JCV AR. The body plastic function adopts Shangtang's independent limb detection function and uses the independently developed rendering engine to achieve the real-time preview of the mainstream model 720p in the market. Body plasticity includes six functions, each function works independently, and the final effect is the superposition of each function effect.

See JCV AR for body shaping function_ Bodys products.

8.1. Introduction to face 2D sticker

The sticker technology in the JCV AR SDK is independently developed by Shangtang technology and uses the leading face key point detection technology in the market. The sticker SDK can directly and accurately paste the two-dimensional sticker materials to the face, or trigger the sticker effect through the face action, support the face rotation, large angle fitting and broadcast sound. The sticker SDK development kit makes use of GPU processing, which makes the processing speed faster, and can achieve the effect of real-time preview of stickers on the mainstream model 720p in the market.

8.2. Introduction to face 2D sticker function

8.2.1. Basic 2D sticker function

The single sheet or sequence of two-dimensional sticker materials can be accurately pasted on people's face, which can realize real-time face tracking and large angle fitting of the face. Support multiple face stickers, up to 10 people.

It supports 27 modes: normal, color filtering, linear fading, positive overlay, color deepening, linear deepening, dark, brightening, dissolution, color fading, darkening, light, superposition, soft light, strong light, bright, linear light, spot light, solid color mixing, difference, exclusion, subtraction, division, color phase, saturation, color and lightness.

8.2.2. Facial motion recognition trigger

In addition to the functions of basic stickers, we have added 7 kinds of facial movements and 15 kinds of gesture recognition to trigger the special effects of stickers. In the actual use scenario, some sticker effects can be triggered by the user's facial actions and gestures to increase the interaction and entertainment between the sticker and the user.

1. Recognizable types of facial movements: nodding, shaking head, opening mouth, blinking, eyebrow picking, opening mouth, laughing and beeping.

2. It can be triggered by gestures: 15 gesture triggers are supported.

3. Trigger mechanism of face action recognition:

・One or more facial action types can be triggered arbitrarily

・The interval can be set to trigger one or more facial action types

4. Material display mechanism:

It can be set to play without action trigger.

The sticker event can be set to trigger.

Support to use with other special effects.

Support randomly selected stickers to play.

Support multi person stickers.

Support trigger to stop playing a sub material.

It supports playing the first frame, playing the middle frame (pause in this frame) and playing the last frame.

You can set the cycle times of the material playback of the corresponding action, such as playing once, cycling twice or infinite cycling.

You can set the time length for the action to play as the material. The action starts, the material starts playing, and the action ends, and the material ends playing.

You can set the number of delayed playback frames of material.

8.2.3. Facial sticker effect display

Filter and makeup blending effect / 3D model and makeup blending effect / background and 3D sticker blending effect

Facial 3D effects is a new feature in development kit. It can render 3D models, materials and lights in real time and accurately, produce a 3D effect of "wearing" the material on the portrait, and follow the head for translation, rotation and other actions. Currently, 3D Stickers support up to 10 faces.

This version of 3D special effects are not compatible.

The features of facial 3D effects are as follows:

Support FBX model rendering

Support UV material map rendering

Support ambient rendering

Supports 3 point light rendering

Support moving with face keys

Support diffuse reflection, highlight, ambient reflection, self illumination, bump, pre baking and other material effects

Support adding multiple dynamic bones.

Supports orthographic effects (the default perspective effect).

This version provides 3D special effect material production gadgets.

When using the 3D special effect function, the 3D material produced according to the requirements in the document "details of 3D special effect material production" will be loaded and ST will be called_ mobile_ sticker. The sticker rendering interface in the H header file can realize the real-time rendering of 3D materials.

This version of 3D special effects material effect supports the use of 2D stickers, front and rear background segmentation, face deformation and other special effects in the same material package.

3D effect display

Figure 9.2 3D effect display 2

This version of 3D rendering engine supports basic particle effect rendering, including particle emission speed, shape, color, angular velocity and other modules. For details, please refer to 3D effect production specification document.

At the same time, the current particle effect can be triggered by trigger.

At the same time, Shangtang provides three-dimensional special effect production gadgets to facilitate material production. This part is provided separately.

Particle effect display:

Figure 10.1 particle effect display

The face detection function is used to reconstruct the user's face. By sensing the changes of the user's facial expression, the 3D model is driven to make the action expression consistent with the user's facial expression in real time. This version can finely restore the user's facial expression, and the effect is more flexible and stable.

Figure 11.1 avatar effect display

Facial deformation is a new feature in the sticker function of JCV AR development kit.

Based on the key points of the face 106, the face deformation function stretches, enlarges or shrinks any area of the face through the deformer, combines multiple effects, and finally presents special face effects such as aliens, harrowing mirrors and so on.

Face deformation material can be made with the latest version of sticker gadget. Load the face deformation material package obtained through the sticker widget and call St_ mobile_ sticker. The sticker rendering interface in the H header file can realize the real-time rendering of face deformation materials. Deformation effects can be used together with other special effects, and support trigger events to trigger face deformation effects.

The following effects can be achieved through different face changing materials, such as face infantilization, snake essence face, or other exaggerated effects:

)

Figure 12.1 display of face deformation effect. Left: original. Right: face deformation effect

JCV AR SDK enriches the face deformation effect and adds the face deformation effect of face dragging, which can make the deformation effect more natural and more suitable for face movement. The new face deformation effect can be achieved through face in the new gadget_ Stretch function is realized. See the sticker material description for details.

13.1 Makeup

The beauty function draws the three-dimensional face patch through the key point information of the face, and draws the two-dimensional texture three-dimensional on the face, which can greatly fit all angles of the face and achieve a three-dimensional and realistic visual effect. The SDK optimizes the presentation of the detailed effect of the beauty function, can follow the real-time position changes of the facial features and eyeballs, realize occlusion processing, and automatically adjust the makeup effect of the anti dark scene. It can make up the face in static and dynamic images. Support video and image processing and real-time effect preview under the mainstream model 720p in the market.

13.2 Makeup function introduction：

Support hair coloring, eye shadow, eyeliner, eyelash, blush, lipstick, dressing, cosmetic contact lenses, and other beauty effects.

Support the intensity adjustment of each beauty function, and support the combination

of beauty effects of each part.

In the lipstick effect, it can grind the lip to make the lip more lubricating and not dry. Support the beauty function. When the face area is blocked, the beauty effect of the

blocked part is not displayed.

Based on the basic eyebrow shape, four eyebrow shapes are added, including thin eyebrow,

pick eyebrow, flat eyebrow and thick eyebrow, which can be made through the new version of 2dstudio to enrich the choice of makeup.

Support the automatic adjustment of dark and backlight scenes, and the effect is natural and not abrupt under different lights.

Provide a small tool for making makeup stickers, which can easily make the material package in the small tool, bringing more possibilities for makeup.

Each makeup type supports a variety of superposition methods (normal, color filtering, linear thinning, positive stacking, color deepening, linear deepening, dark, brightening, dissolution,

color thinning, darkening, light, superposition, soft light, strong light, bright light, linear light, spot light, solid color mixing, difference, exclusion, subtraction, division, hue, saturation, color, lightness).

13.3 Makeup effect display:

Makeup effect display 1 (left: original picture, right: after beauty)

Full face beauty shielding effect display (left: original picture, right: after beauty makeup (eye makeup: Stars))

Display of lipstick shielding effect (the original picture is shown in sequence, including lipstick, lipstick shielding 1 and lipstick shielding 2))

Face changing effect display

Hair dyeing effect display

2D face picking is a new feature in the sticker function of JCV AR development kit.

The 2D face picking function makes rich and interesting special effects by moving the position of the face mask based on the face 106 key points and 2D stickers.

2D face cutting material can be made through the latest version of sticker gadget. 2D face cutting stickers can be used together with other special effects, and support trigger to trigger 2D face cutting effects.:

face matting effect display

15. 3D face matting

3D face picking is a new feature in the sticker function of JCV AR development kit.

3D face picking function is based on face 106 key points and 3D stickers. Through 3D gadgets, you can independently produce rich and interesting special effects.

3D face matting effect display

16. Sky segmentation effect

Sky segmentation is to detect the sky in the image and separate the sky from other scenes. Shangtang's sky segmentation has the advantages of accurate boundary segmentation, fast detection speed and good robustness. It can be made independently through 3D gadgets and can be used with other types of stickers.

The sky segmentation effect is shown in the figure below (different effects are displayed at the same time and place)

17. Skin segmentation effect

Skin segmentation is to detect human skin in the image and separate human skin from other scenes.

Shangtang's skin segmentation has the advantages of accurate boundary segmentation, fast detection speed and good robustness. Can be used for whitening and sticker effects. It can be made independently through 2D studio and can be used with other types of stickers.

Skin segmentation whitening is only aimed at skin whitening, and retains the primary colors of hair, clothes and background:

Skin segmentation effects:

18. Cat face & dog face effects

Cat face & dog face sticker SDK can directly and accurately paste two-dimensional sticker materials to cat face & dog face, trigger sticker effects through human face actions, support face rotation and large angle fitting, and support the use of special effects with face.

Cat face effect display:

Dog face effect display:

Support sound effects. Sound material can be added to 2D stickers and face deformation stickers, and sound can be played when 2D materials and face deformation materials are rendered.

Sound effects can be used with other special effects.

This version only supports MP3 format sound materials. If the material has sound effect, place the sound material file in MP3 format in the root directory of the material folder and change the corresponding JSON (it is recommended to use V1.7 sticker gadget for material processing).

Suggestion: the size of sound material should not be too large, and the size should be controlled within 1m and the time length should be controlled within 30 seconds.

Sound effects support a variety of trigger modes and related settings:

Sound trigger mode: the trigger conditions of sound material (face action, gesture, etc.) can be set, which can be consistent with or different from the trigger conditions of other stickers in the material. Null means to play the sound material directly.

Delay options. You can set the sound material to delay playing after meeting the trigger conditions. 0 means no delay, in frames.

Playback times option. You can set the playback times of sound materials. 0 means looping.

End playback option. You can set whether the sound material stops playing when the trigger condition disappears. No means not to stop, yes means to stop playing sound material after the trigger condition fails.

The general object tracking function is JCV ARAn independent function module of effects development package.

The general object tracking function can track the specified area in the video in real time and feed back the confidence of the tracking results in real time. The tracking area can be changed in real time and the size of the tracking area can be changed.

The general object tracking function supports HD real-time tracking and fast processing speed. This version of general object tracking only supports single target tracking.

For example, select a sign on the building, set the material to track the sign, and then turn the mobile camera. The sticker material can track the object accurately and quickly. As shown in the figure below.

Figure 15.1 general object tracking diagram

21.1. Introduction to filter

The filter development kit is a functional module of JCV AR, which is developed by Shangtang technology. It is divided into GPU filter and CPU filter suitable for processing a single picture. The GPU filter development kit makes use of the processing function of GPU, which makes the processing speed faster and greatly improves the performance. It can meet various applications such as video stream processing, image processing and so on. There are many kinds of filters. At the same time, it has the function of adjusting the effect of the filter, so there is a large choice and adjustment space.

At the same time, according to the different needs of customers, we also provide CPU version filters, which are suitable for single picture processing in scenes where GPU cannot be used. Compared with the GPU version, the CPU version is slightly slower.

21.2. Filter effect display

In this version, we provide a total of 21 new filters based on the previous 90 filters. The effect thumbnails are shown below.

Figure 16.1 effect display of 90 filters

Figure 16.2 effect display of 21 new filters

For example, for beautiful photos of people, we can choose babypink style filter to make the skin of the portrait look more pink and tender.

Figure 16.2 left: no filter effect; right: babypink style filter effect

JCV AR uses the license file authorization method to support SDK offline activation. Users can permanently use the functions of this version by activating the SDK within the authorization period. If you need to extend or adjust the SDK license period, you can replace the license file.

In the integration process, we need to call the four interfaces of license in the engineering code to ensure that the call sequence is correct, and can refer to the implementation method in sample.

23.1. Android Support

23.2. iOS Support

Note: IOS now supports the simulator mode (functions requiring cameras cannot be tested under the simulator)

IOS editing framework includes: armv7, arm64, i386, x86_ sixty-four

This SDK provides Android sample and IOS sample. Please refer to each sample for the construction method of the project.

24.1. Replace iOS sample material

iOS Sample supports the replacement of different sticker materials. You need to put the materials into the folder of the following materials in the following ways.

Scheme 1: use iTunes to replace materials

Install the live effect sample and connect the mobile phone to the computer

Open iTunes

Enter apps and drag the compressed material package directly into the stickers subfolder in documents.

Reopen the live effect sample to view the effect of the new sticker

Scheme 2: use PP assistant to replace material

Install PP assistant

Install the live effect sample on the mobile phone

Connect the mobile phone to the computer and open the view files of the live effect sample in the app of the PP assistant

Enter documents and the subfolder local_sticker

Put the material package into the subfolder.

Reopen the live effect sample to view the effect of the new sticker

24.2. Replace Android sample material

Android Sample replacement material needs to be replaced manually in the folder

Enter device SD card file management

Enter /storage/emulated/0/Android/data

Find the folder of the package name corresponding to the app, and enter the folder.JCV AR.effects if it is a demo

Enter the files folder, where the resource folder new is the new engine. Copy the sticker material compression package and the corresponding thumbnail (the same name as the compression package) to any of the six folders, and you can find the added material under the corresponding category in the demo.

Restart the app to use

25.1. 2D description of sticker material description file

Each sticker material package includes a picture material sequence frame and material description file. The material presentation file is a JSON format file.

It is recommended to use the sticker widget to generate the sticker material directly. For the usage of the sticker gadget, please refer to the document sticker material auxiliary tool description

Currently, the sticker widget supports all the functions of the new rendering engine.