What is Facial Recognition - The 2024 Ultimate Guide for Facial Recognition Technology

Biometric technology continues to develop rapidly, offering safety and security to all facets of life. By using an individual’s physical characteristics, which are almost impossible to replicate, biometric technology proves a better choice than traditional security methods that are easily lost or stolen, such as personal identification cards, passwords/PINs, and keys. Though it has been around for decades, primarily through fingerprints, biological samples (DNA, blood, saliva, etc.), and even iris scanning, one biometric technology deserves a closer look: facial recognition.

Facial recognition solutions have grown in popularity in recent years. One of the most ubiquitous use cases is mobile phones. Many consumers worldwide interact with this biometric technology daily, to secure and unlock their phones.

However, the possibilities extend far beyond personal mobile devices and provide significant benefits in safety, security, and efficiency across a variety of industries. As a compelling, comprehensive, and rewarding technology, it is vital to understand how facial recognition works, how it can be deployed and optimized, its technical considerations and specifications, the varied use cases, and of course, its potential.

1. What is Facial Recognition?

Facial recognition biometrically identifies facial vectors and features, matching them with pre-enrolled individuals. Recent advancements in AI technologies, based on deep neural networks (DNNs), have dramatically improved precision, unlocking a wealth of new use cases.

The technology leverages proprietary AI trained to learn and recognize faces, just like the human brain, with the ability to ascertain a person under different situations and with great accuracy because of the magnitude of processed data. Facial recognition then compares the generated template with existing templates in a database. If there is a match, it can confirm an individual’s identity.

Building on its facial recognition expertise, CyberLink leveraged deep learning and neural networks to create FaceMe^®, an AI-based facial recognition engine. CyberLink continues to push the boundaries of these technologies to enhance its AI-based models, resulting in one of the world’s most accurate, secure, and flexible edge-based solutions.

1.1 How Does Facial Recognition Work?

Facial recognition is by far the most powerful and relevant AI biometric technology. It has vast abilities and can carry out several tasks beyond face detection and recognition. The more robust and feature-forward a facial recognition platform, like FaceMe®, the more benefits and fewer biases it brings.

The key features of a facial recognition engine are:

Face detection

Face detection is the first step the engine takes to confirm the presence of faces as they appear on a live camera feed, a video recording or as it scans still image captures. The whole field of view is scanned for any area containing full or even partial human faces. Fast and precise face detection is a critical first step to ensure the performance of the entire facial recognition process. FaceMe® can detect more than one face simultaneously, count how many faces are present, and perform detection on each of them individually.

Face feature extraction

After face detection, feature extraction is the next step. The engine first extracts an n-dimensional vector set (a template) from the facial image. Achieving very high precision requires a high “n” value. FaceMe® only stores encrypted template data. To fully ensure privacy, no actual images of faces are stored on our platform. Next, the template extracted from an individual’s face is used for matching or searching.

1:1 Face match

If the goal is to verify a person’s identity, in other words to answer the question, “Is this person who they say they are?” then the facial recognition engine performs a 1:1 face match. If they are already enrolled, the engine extracts a facial template from the camera feed’s live view and checks if it matches the template on file for this person.

When the person is not pre-enrolled or when the process requires a verification to an officially recognized identity document, the engine can be connected to an ID scanner to extract a template from the face on the picture ID and use the same 1:1 method to check if it matches the template extracted from the live view of this person’s face. The most common use of 1:1 face match is arguably to control smartphone or computer access, as well as to login to an app or online service. iPhone users have enthusiastically embraced Apple’s Face ID, which performs 1:1 face match, essentially as described here.

1:N Face search

If the goal is to answer the question, “Who is this person?” the engine completes a 1:N face search. It compares the individual’s facial template to the pre-enrolled faces in the database and confirms the person’s identity if it finds a match. The most common use of face search is a security and surveillance system using a camera to verify that a person belongs to a particular company – if their face was pre-enrolled in the company’s facial database, face search will confirm it is them and grant them access.

1.2 More than Recognizing Faces

Some key use cases require additional features, such as the following:

Facial attribute detection (face analysis)

Face attribute detection, also known as face analysis, identifies and analyzes characteristics such as age, gender, facial expression, and head orientation or movements (e.g., nodding, shaking). This feature is a crucial enabler of smart retail and digital signage for use cases like delivering customized ads and messaging to targeted audiences or collecting detailed visitor statistics.

While other technologies allow the collection of data such as time spent by an individual at a location, the path taken by that individual and if they make a stop at the cash register before leaving, FaceMe®’s technology adds for the first time precise demographic data and sentiment analysis to these individuals. By default, to ensure privacy compliance, FaceMe® aggregates the facial attribute data without generating a facial template for individuals who have not expressly given their consent to biometric identification. Its analytical tools can generate, in real time, a precise detailed distribution for each attribute, for any time interval, for each camera, for any cross-tabulation. It can also aggregate the data to any level of abstraction.

Depending on the workflow and business practices, it can either generate reports or directly enhance the visitor’s experience, by sending alerts to the staff or directly feeding data to signage’s content management system, even adjusting the product mix to be sent from the warehouse to the point of sale.

Mask detection

Mask detection is one of the newest and most valuable features to ensure health and safety in public or private spaces during the pandemic. CyberLink’s FaceMe® SDK is a cross-platform facial recognition SDK that offers optimized mask detection and facial recognition when wearing a mask. It recognizes health-compliant masks and verifies if the nose and mouth are properly covered while performing highly accurate face detection and recognition. Additionally, CyberLink’s FaceMe® Security helps existing security solutions upgrade their mask detection features so that employees can enter the office while still wearing masks.

Anti-spoofing

Liveness detection, often referred to as anti-spoofing technology, is used for biometric fraud protection against a range of techniques that have been used to impersonate authorized people and gain access illegally.

1. How can we spoof facial recognition?

There are three main ways imposters attempt to spoof facial recognition systems. The least sophisticated approach consists in placing an authorized individual’s photo in front of the camera. Almost any recent solution will detect the absence of facial movement. Showing video instead of a still picture seems the logical solution. But depth-sensing cameras easily detect the subterfuge, which is why we occasionally see the use of sophisticated 3D masks that show the facial features of the authorized individual in a realistic shape.

2. Anti-spoofing and facial recognition

One advantage over other biometric identification technologies is that precise facial recognition can be performed with simple 2D webcams or security IP cameras, just as well as when using sophisticated 3D cameras equipped with the latest structure light technologies. Specific use cases or constraints as varied as cost, location, form factor and speed inform camera decisions that directly impact the availability of anti-spoofing options and their performance.

2D cameras typically catch spoofers through interactive measures such as performing a set of motions randomly chosen by the access computer before being granted access, as well as Passive 2D (non-interactive) measures which are unique to each solution provider and its AI algorithm.

3D cameras on the other hand, perform depth detection, precisely identifying a spoofing attempt within a few milliseconds. 3D cameras generally provide a superior experience, but they are costlier, while 2D alternatives can also provide accurate anti-spoofing at a fraction of the cost, wherever a small delay is acceptable. Costs and size of 3D cameras are rapidly going down, and a new generation of time-of-flight sensors can be attached to 2D installations, adding depth detection at a fraction of the cost of new 3D devices. The development and availability of 3D solutions has grown massively in recent years and has made this technology very accessible.

The more advanced attack method, use of a physical 3D mask, requires a software-based defense. Advanced facial recognition systems like FaceMe® leverage AI anti-spoofing technology to analyze natural facial movements and determine whether the face in front of the camera is displaying expressions or emotions as a natural human face would. Facial recognition engines process such a massive amount of data, that even a well-made 3D mask has many tells in the form of highly specific facial vector data that contrasts to that of a real human face.

In this relatively new field, industry beacon iBeta offers tests like the Presentation Attack Detection Test which holds facial recognition software to rigorous anti-spoofing standards and aids in choosing the right solution. Only a few facial recognition solutions have passed both iBeta anti-spoofing tests. FaceMe® achieved a near-perfect score in iBeta's Level 1 test (photos and videos) and detected 100% of spoofing attempts using 3D masks in the advanced Level 2 Anti-Spoofing Test, attesting that it complies with ISO 30107-3 testing and reporting requirements.

Additionally, NIST’s Face Analysis Technology Evaluation (FATE) tests the processing and analysis of a facial image (what is in the image). The NIST.IR.8491 report tested 82 different PAD algorithms and separated their findings into two categories: convenience and security. Under convenience, the True Acceptance Rate (TAR) is fixed to 99% and sorted by True Rejection Rate (TRR). In NIST’s FATE PAD video convenience category FaceMe® ranked first, based on three different video tests in which FaceMe® achieved 100% True Rejection Rate, correctly stopping 100% of presentation attacks, when the True Acceptance Rate was set 99%.

1.3 Provides Remarkable Accuracy

The most precise facial recognition engines are characterized by a low false non-match rate (FNMR) and an extremely low false match rate (FMR). A false non-match is the failure to match two facial templates from the same person, whereas a false match is when one person’s face is matched with someone else’s.

The National Institution of Standards and Technology (NIST) in the U.S.A. is the governing body that determines how well a given facial recognition algorithm achieves a set of common tasks. NIST’s Facial Recognition Technology Evaluation (FRTE) uses different datasets to evaluate the performance of an algorithm. For example, the VISA category tests a facial recognition algorithm’s ability to correctly identify an individual based on a passport photo, while the WILD test uses random, non-constrained photojournalism-style images.

FaceMe® has achieved one of the highest NIST VISA test ratings, with 99.83% at 1E-6 accuracy. 1E-6 means the False Match Rate (FMR) is 1 in 1,000,000 (0.000001). By comparison, contemporary smartphone Face ID technology offers an FMR of about 96% at 1E-4 accuracy.

Beyond the algorithm, some of the main factors affecting accuracy are camera resolution, camera positioning, lighting, clarity, and camera type. Facial recognition engines generally work adequately with 720p cameras, but a 1080p resolution device is generally recommended.

2. Why You Should Deploy Edge-Based Facial Recognition

Facial recognition can be deployed using cloud-based technology, often using popular Microsoft or Amazon solutions, or by integrating SDKs or software onto edge devices. While each approach offers distinct benefits, edge-based facial recognition is generally deemed superior, offering better speed, security, affordability, flexibility, and versatility.

2.1 What is Edge-based Facial Recognition?

When deploying facial recognition at the edge, the technology is embedded in local devices: a smart lock, mobile phone, point-of-sale (POS) system, interactive kiosk, digital signage, or similar. Edge devices run facial recognition solutions quickly, securely and with extreme precision. There is no inherent need for a network or cloud connection except to access the facial template database if it is not stored on the edge device. Even then, only a small, encrypted, template is transmitted for validation. The database’s encrypted templates are often hosted on a remote server, resulting in a secure operation completed within milliseconds.

Businesses that implement facial recognition from the ground-up will benefit from an edge-based approach with IoT devices carrying out the required tasks. Financial institutions make a compelling case for edge-based facial recognition systems, as many banks do not allow internet connections for security reasons.

The low cost, flexibility, and scalability of edge-based facial recognition make it the best option for most end users.

CyberLink’s FaceMe® SDK perfectly exemplifies a best of breed edge-based facial recognition solution. This highly competitive and flexible face recognition SDK is easy to integrate across a wide range of edge devices. FaceMe® leads the way by supporting a variety of chipsets and OS', and its highly accurate AI engine is ranked one of the best in the NIST Facial Recognition Technology Evaluation (FRTE). Through constant innovation, the technology meets the highest accuracy and security standards for deployments across a wide range of industries and use cases including security, access control, public safety, smart banking, smart retail, smart city, and home protection.

2.2 The Future of Facial Recognition Technology is on Edge Devices

Cost of Ownership

AI is very computationally demanding, and cloud computing is not cheap, so choosing between the edge or cloud is a key design decision. Edge devices have a cost advantage over the cloud, which typically charges an incremental amount for each face recognized, which can quickly add-up. Meanwhile, cloud-based computing might be cheaper on a very small scale, in applications not exceeding more than a few faces per hour. As AI-acceleration processor pricing has come down, the cost advantage offered by edge-based solutions has become increasingly apparent.

Response Time & Service Availability

No internet service is immune from interruptions or unexplainable low bandwidth issues. Imagine if the lock on your front door at home stopped working because a cloud-based facial access solution lost connectivity. Edge-based facial recognition isn’t as vulnerable to internet issues.

With its inherent superiority and huge momentum in innovation, edge-based technology will be a vital driver of the future success of facial recognition. Therefore, we will focus on edge-based facial recognition for the remainder of this article.

3. How to Integrate Facial Recognition in Edge Devices

When building a facial recognition edge device, choosing the right chipset based on the specific use case will be the most consequential decision regarding cost and performance. For example, a high-end NVIDIA GPU chip may have a higher upfront cost, but can handle hundreds of video channels concurrently, thus reducing the number of costly workstations required to monitor a large facility.

On the other hand, a low-cost SoC from MediaTek or NXP/Qualcomm may offer limited performance, with processing speeds of about five frames per second and only frontal face recognition. Still, it will likely be powerful enough, and more affordable, for smaller use cases such as door access.

3.1 Standalone GPU

GPU chips are powerful hardware designs capable of supporting facial recognition with outstanding performance. Their substantial on-board memory, high memory bandwidth, and considerable floating-point computation capabilities make GPUs the best option for the complex, computation-hungry AI algorithms behind facial recognition. GPUs are also suitable for enabling facial recognition in surveillance systems, which requires simultaneously applying facial recognition across multiple video channels. A separate CPU chip is needed to build a system using these GPUs.

This year we recommend two GPUs from NVIDIA. They are extremely competent and provide a good balance between computational power and price.

3.2 CPU/SoC for AIoT Devices

These chips are complete yet affordable solutions for enabling facial recognition in small, mass-market IoT devices. They all integrate CPUs, resulting in simpler, cheaper hardware options. SoCs may seem less capable than discrete GPU options, with lower power consumption and a smaller form factor, but several FaceMe® facial recognition models (detailed in section 3.4) can run on these AIoT devices, while still providing highly accurate facial recognition results.

3.3 Operating Systems

Chipsets are often designed to run on specific operating systems (OS). An excellent facial recognition engine should support as many processor + OS combinations as possible. FaceMe® supports one of the market’s most comprehensive lists of chipsets and more than 10 OS':

• Windows
• Android
• iOS
• Linux variants: Ubuntu x64, Ubuntu ARM, RedHat, JetPack (mainly for NVIDIA Jetson family), Debian ARM, CentOS, Yocto ARM

CyberLink designed FaceMe® to be highly versatile, allowing flexible customization of multiple combinations of hardware chips and OS' that match end users’ unique needs. FaceMe®’s multi-OS support is ideal for cross-platform solutions. Developers have access to several GPU acceleration options, harnessing OpenVINO, NVIDIA CUDA/TensorRT, NVIDIA Jetson, Qualcomm SNPE, MediaTek NeuroPilot, and more, to speed up deep learning algorithms and further optimize performance.

3.4 System Architecture Optimization for Best Performance

Designing an excellent facial recognition system for a high-performance workstation or PC with GPU (or VPU) needs careful consideration, as dozens of concurrent video streams may run between the CPU, GPU, and memory over the system bus. Even an excellent facial recognition algorithm will be slow if implemented improperly on a system architecture level. The system architecture design should minimize the data flow between CPU, GPU, and memory.

FaceMe® has an optimized system architecture that allows it to deliver the best performance. For example, on a single workstation, FaceMe® with an NVIDIA RTX A5000 can handle 489 to 727 frames per second (the exact number may vary depending on which FaceMe® facial recognition model is used). This is equivalent to handling 48 to 72 concurrent video channels (each with 10 frames per second) per GPU — an outstanding cost-performance offering.

3.5 Lightweight AI Model for Facial Recognition

Several cost-constrained use cases don’t require more than basic frontal face recognition (e.g., smart door locks). These devices are driving market demand for lightweight AI models that can achieve reasonable facial recognition at a lower cost. FaceMe® provides three models for this purpose:

• Ultra-High (UH) Model: This model can recognize both VISA and WILD types of faces with market-leading precision. However, it demands very high computation power from a dedicated GPU or high-end Intel CPU.
• Very High (VH) Model: This model can recognize both VISA and WILD types of faces with slightly less precision than that of the UH Model, but the required computational power is significantly lower. It unlocks a wide range of use cases requiring superior accuracy that wouldn’t be economically viable otherwise.
• High (H) Model: This model can run on low-cost chips with relatively low computation power. The precision is still good enough for VISA-type frontal faces, enabling several use cases that weren’t remotely possible before.

4. Edge-based Facial Recognition: On-premises Devices and Workstations

One important attribute of a leading facial recognition solution like FaceMe® is its flexibility regarding the hardware it can run on. FaceMe® can be deployed across workstations, computers, mobile, and IoT devices. Let’s examine a few examples.

Facial recognition on workstations

Organizations looking to deploy a facial recognition SDK over tens, or even hundreds, of video channels across large facilities will benefit from a workstation powered by a high-end GPU capable of simultaneously handling multiple IP camera video feeds. Locations such as department stores, airports, industrial plants, and hospitals, may employ dozens or even hundreds of cameras for varied purposes. Typically, they will be used for security and access control to visitor behavior analysis, crowd management, and VIP customer identification. Connecting all the cameras to one or a few central workstations running facial recognition is the easiest, most robust, and likely most economical solution.

To learn more about the benefits of FaceMe® for workstations, check out our partnership with VIVOTEK and the successful integration in their facial recognition solution.

Facial recognition on PCs

Facial recognition is most commonly powered by PCs for smaller scale deployments. Obvious examples are a store or restaurant which wants to identify VIPs, automatically clock in employees, or get alerts for block-listed people. During a pandemic, a clear advantage of facial recognition is that it enables the monitoring of mask wearing and body temperature for both employees and customers. Store or restaurant management can install IP or USB cameras at the front and back doors and simply connect them to a PC that runs robust integrated facial recognition software. A value-priced, ready-to-deploy software solution that includes all these features is FaceMe® Security. Learn more about crucial considerations for building top-of-the-line industrial PCs.

Facial recognition on mobile devices

The potential of facial recognition technology on mobile devices extends far beyond unlocking a cell phone. One compelling fintech use case is the integration of eKYC (electronic know your customer) facial recognition technologies on mobile phones to strengthen identity verification for online banking, loan applications, insurance, and more.

Facial recognition on smart IoT devices

The current fast-paced innovation in edge computing, driving better performance while cutting costs, also opens the door to endless IoT device use cases powered by facial recognition. Smart kiosks offer a compelling example. Frequent travelers are familiar with Global Entry and Clear kiosks, which use facial recognition. Now fast-food restaurants, hospitals, and hotels are deploying smart kiosks and integrating facial recognition. Major hotel chains have eagerly introduced self-check-in kiosks to cut down on wait times.

Adding a facial recognition engine like FaceMe® provides exciting, personalized experiences for guests by using their opted-in faces as the only ID they need through their stay. Take a look at our extended article on smart IoT devices.

5. Other Facial Recognition Design Factors: Security, Encryption & Privacy

As reviewed earlier, edge-based facial recognition is more secure than cloud-based options that require individuals’ pictures and videos to be sent through the internet to a cloud server, a process inherently vulnerable to attacks and leakage. Edge-based computing avoids most risks as the only data captured and stored takes the form of encrypted face templates, and the entire process can run without any cloud connection.

FaceMe® uses AES-256 bit encryption to secure all data before it is stored in any database. AES is one of the best symmetric encryption algorithms, and 256-bit is the highest security option supported. The face template is protected by storing the encrypted file (with a secret key) outside the source or platform server, keeping it secure even if physical devices are compromised or stolen.

Individuals must first opt-in to any facial recognition program requiring face enrollment. In edge-based solutions, the captured information will consist of template data (a mathematical number in a very high dimension) for future matching and identification purposes. The template doesn’t contain an actual face image, it can’t be used to recompose someone’s face, and it is kept separate from any personally identifiable information (PII) that could identify a person.

The encrypted data that is captured when performing facial recognition is only used to establish a match with a pre-enrolled template stored in a secure database. Many data privacy laws and regulations (such as GDPR, CCPA, BIPA, and LGPD) count biometric data as personal information. Therefore, any business looking to employ facial recognition must always obtain the user’s consent.

When evaluating various facial recognition providers, it is important to locate their headquarters and essential facilities. For instance, the U.S. government has expressed fair concerns around surveillance technologies from companies linked to China and Russia – as they might not have adequate opt-in or data protection requirements. Most facial recognition solutions are safe and rigorously apply strict data and privacy protection standards. However as an end-user, you should be able to fully trust your service provider, especially when it comes to security, privacy, and protection of human rights.

6. Top Use Cases of Facial Recognition Technology

While debates on the technology continue, there are many success stories where facial recognition deployment has increased safety, resulting in positive user experiences. A recent survey conducted by the Security Industry Association (SIA) found that most Americans (68%) believe facial recognition can make society safer, with particular support for the technology in airports (75% for airlines; 69% for TSA), office buildings (70%), and banks (68%).

Key relevant use cases fall under five major categories:

1. Access control, e.g., entrance access, medical cabinets, smart locks
2. Surveillance and security, e.g., detecting unauthorized people in warehouse areas
3. Authentication, e.g., eKYC for BFSI (Banking, Financial Services and Insurance)
4. Smart retail, e.g., collecting statistics on shoppers’ demographics
5. Health control during a pandemic, e.g., detecting whether a face mask is worn properly

Example: FaceMe® Security offers a comprehensive facial recognition application in surveillance and security.

AI biometrics can power many use cases in specific vertical segments. For more on these use cases, read our article highlighting the Top 7 Use Cases for Facial Recognition in 2024.

7. Final Thoughts: How Facial Recognition Will Transform Biometrics

Facial recognition technology is poised to make our world a better place. But, to do so, individuals need broader levels of education on ethical implementation to feel more comfortable with, and accepting of, businesses that have openly adopted AI biometric technology as a new, safe standard.

Facial recognition, and the potential it holds, is clearly a more positive advancement than detractors would like to admit. It is businesses keeping their employees safe by automating secure access control in the office. It is retailers enhancing customer experiences in their stores. It is manufacturers simplifying access to their tiers of restricted areas. It is banks and fintech companies introducing much stronger authentication and cutting-edge security controls. And that’s just the tip of the iceberg.

Facial recognition is the future of AI biometric technology. The industry needs to better educate consumers and debunk the many falsehoods circulating about this technology, while explaining its positive value and potential for good. Facial recognition also needs to be appropriately regulated to not hinder innovation but bring forth its many benefits.

FAQ