Facial recognition has grown in popularity in recent years, with many consumers all over the world purposefully interacting with this technology on a regular basis to secure and unlock one of their most precious devices: their mobile phones. It is one of the most ubiquitous use cases.
However, the possibilities extend far beyond personal mobile devices and bring key benefits in applications of safety, security and efficiency across industries. As an incredibly powerful, comprehensive and rewarding technology, it is important to understand the nuances of how facial recognition works, how it can be deployed and optimized, the technical considerations and specifications, varied use cases and its potential for the future.
Facial recognition is a biometric technology that identifies facial vectors and features, and matches them to a pre-enrolled individual. This technology has been around for several years. For example, CyberLink developed YouCam more than ten years ago, using face recognition to login to personal computers. At that time, the technology based on digital signal processing (DSP) techniques, had restrictions and required full frontal pictures. With recent advancements of AI technologies based on deep neural network (DNN), there has been a dramatic improvement in precision, unlocking a wealth of new use cases.
The technology leverages proprietary AI algorithms and mathematical equations to make its connections to individuals by measuring a number of facial variables - such as nose depth and width, forehead length, eye shape - and saves the information as a template. The template generated for an individual is used as a basis for comparison to confirm identity, if there is a match with an existing template.
Building on its previous facial recognition expertise, CyberLink used deep learning and neural networks to create FaceMe®, an AI-based facial recognition engine. It continues to push the boundaries of these technologies to enhance its AI-based models, resulting in one of the world's most accurate, secure and flexible edge-based solutions.
Facial recognition is by far the most powerful and relevant AI biometric technology. It can be incredibly vast in its ability to carry out a number of tasks beyond just face detection or face recognition. The more robust and feature-forward a facial recognition platform is, like FaceMe®, the more benefits and less biases it brings.
The key features of a facial recognition engine are:
Face detection is the first step the technology takes to detect a face. In this step, the technology works by scanning the whole image to see if any area contains full or even partial human faces. Fast and precise face detection is a critical first step to ensure the performance of the entire facial recognition process. FaceMe® can detect more than one face at once, count how many faces are present and perform detection on each of them individually.
Feature extraction is the step following detection. The engine extracts an n-dimensional vector set (called template) from the face image. Achieving very high precision requires a high "n" value, let's say 1024. The template that is extracted from an individual's face is used for matching or searching next.
The newly extracted template is then matched to those pre-enrolled in a database. A 1:N search is done by matching an individual's template to the entire database, to find the best match and confirm the person's identity. FaceMe® only stores encrypted template data. No actual face images are stored through our platform, fully protecting privacy.
Some key use cases require additional features, such as the following:
Face attribute detection is the task of identifying and analyzing characteristics such as age, gender, mood and head orientation or movements (e.g., nodding, shaking). This feature is a key enabler of smart retail and digital signage, for use cases such as pushing dynamically customized ads and messaging to very targeted audiences, or collecting detailed visitor statistics.
Mask detection is one of the newest and most valuable features to ensure health and safety in public or private spaces, during the pandemic. FaceMe® Health offers optimized mask detection and facial recognition when wearing a mask. It recognizes health-compliant masks and verifies if the nose and mouth are properly covered, while performing highly accurate face detection and recognition.
1
2
3
To protect against biometric fraud, such as holding someone else's photo or video in front of a camera, anti-spoofing technology can provide secure and accurate liveness detection with 3D or 2D cameras.
When using a 2D camera (e.g., USB webcam) spoofers are caught through interactive and non-interactive measures. Interactive measures detect natural and precise head or facial movements to confirm the presence of a live person. Non-interactive measures are unique to each solution provider and their AI algorithm for face detection and recognition.
3D cameras perform depth detection, allowing quasi-instantaneous anti-spoofing. There is no need for interactive detection or recognition measures. 3D cameras generally provide a superior experience but they are costlier, while 2D alternatives can also provide accurate anti-spoofing at a fraction of the cost. FaceMe® supports both 2D and 3D cameras. Here are a few 3D camera options that are compatible with FaceMe®: Intel RealSense, 3D camera on iPad or iPhone, Orbbec, Himax, Altek and eYs3D.
Precise facial recognition engines are characterized by a low false non-match rate (FNMR) and an extremely low false match rate (FMR). A false match is when a person's face is matched to someone else. A false non-match is failing to match two face captures from the same person.
The National Institution of Standards and Technology (NIST) is the governing body that determines how well given facial recognition algorithm achieves a set of common tasks. NIST's Facial Recognition Vendor Test (FRVT) uses four tests to evaluate the performance of an algorithm: VISA, VISA Border, Mugshot and WILD. The VISA category tests a facial recognition algorithm's ability to correctly identify an individual based on a passport photo, VISA Border compares VISA images to webcam images. Mugshot images are constrained to frontal view, but match two pictures of a person that were taken more than 12 years apart. WILD test uses random, non-constrained photojournalism-style images.
FaceMe® has achieved one of the highest NIST VISA test ratings, with 99.46% at 1E-6 accuracy (0.54% FNMR and 1 in 1,000,000 FMR). It has also demonstrated top-tier accuracy levels for VISA Border (99.48% at 1E-6) and for WILD (96.98% at 1E-5). By comparison, face ID on a smartphone offers about 96% at 1E-4 accuracy.
Beyond a highly-rated algorithm, some of the main factors that affect accuracy are camera resolution, camera positioning, lighting, cleanliness and camera type. Facial recognition engines generally work adequately with 720p cameras, while a 1080p resolution is generally recommended. The camera should be pointing directly in front of the subject with good lighting, as some lesser quality cameras cannot perform clear readings from an angle. And of course, the camera lens should always be clean.
Facial recognition can be deployed using cloud-based solutions, such as those from Microsoft and Amazon, or by integrating SDKs or software onto edge-devices. While each approach offers distinct benefits, edge-based facial recognition is generally deemed superior, offering better speed, security, affordability, flexibility and versatility.
Performing facial recognition using cloud service platforms accessed through the internet requires copious amounts of bandwidth resulting in slow and costly data transmission, as well as expensive cloud-based processing, limiting the number of justifiable use cases. This approach comes with inherent security risks, as the captured data (i.e., full images), is sent through the internet and could be stored in a system that is vulnerable to data hacks or leakage. No cloud system is airtight.
That said, there are benefits to deploying in the cloud, as there is no need for extensive hardware. If a business operates entirely online, a cloud-based facial recognition system is likely best. Many early facial recognition solutions were built on the cloud, as edge AI chipsets were then either unavailable, slow or unaffordable. Examples of cloud-based solution providers include (1) Microsoft Azure's Face API package, (2) Google's Vision AI and (3) Amazon's Rekognition offering on AWS.
For small-scale deployments, the bandwidth and cloud-processing costs shouldn't be too high. For example, small business or home security systems offering features such as smart, IP-enabled doorbells.
When facial recognition is deployed at the edge, this means that the technology is embedded in local devices - whether it be a smart lock, a mobile phone, point-of-sale (POS) system, interactive kiosk, digital signage or more. Edge devices run facial recognition quickly with extreme precision, with no delays from cloud processing or large file transmission. In fact, there is no inherent need for a network or cloud connection except to access the face database if it is not stored on the edge device. Even then, only a small encrypted template is transmitted for validation with the database's encrypted templates hosted on a remote server, resulting in a secure operation completed within milliseconds.
Businesses that implement facial recognition from the ground up will benefit from an edge-based approach with IoT devices carrying out the needed tasks. Financial institutions make a compelling case for edge-based facial recognition systems, as many banks do not allow internet connections for security reasons.
The low cost, flexibility and scalability of edge-based facial recognition, make it the best option for most end-users.
CyberLink's FaceMe® SDK (Software Development Kit) perfectly exemplifies a good edge-based facial recognition solution. This highly competitive and flexible product is easy to integrate across a wide range of edge devices. FaceMe® leads the way by offering one of the market's most comprehensive chipsets and OS support. Its highly accurate AI engine is ranked one of the best in the NIST Face Recognition Vendor Test (FRVT). Through constant innovation, the technology meets the highest accuracy and security standards, for deployments across industries and use cases. FaceMe® can be deployed across a wide range of scenarios, including security, access control, public safety, smart banking, smart retail, smart city and home protection.
Let's examine trends for (1) cost, (2) response time and (3) service availability:
As AI is very demanding on computation and cloud computing is not cheap, choosing between edge or cloud is a key design decision. Edge devices have a cost advantage over cloud, which is typically charged an incremental amount for each face recognized. Cloud-based might be cheaper only at a very small scale, not exceeding a few faces per hour. As AI chips keep getting cheaper in recent years, edge-based solutions have a sustainable and widening cost advantage.
When it comes to speed, the best facial recognition algorithms operate in milliseconds. Edge-based solutions rank supreme, outperforming their cloud-based counterparts by several orders of magnitude. Regardless of how a business is looking to implement facial recognition - response time matters. Across a growing number of use cases, cloud simply cannot compete. Delays to identify a block listed individual could cause irreparable harm. Response time should never be sacrificed.
No Internet service is immune from interruptions or unexplainable low bandwidth issues. Imagine if your home door lock stops working because it depends on a cloud-based facial access solution. Edge-based facial recognition, doesn't have such vulnerabilities to Internet issues.
With its inherent superiority and huge innovation momentum, edge-based technology will be a key driver to the success of facial recognition going forward. Therefore, we will focus on edge-based facial recognition for the remainder of this article.
When building a facial recognition edge device, choosing the right chipset is the most consequential decision - it will determine the cost and performance, both of which must be determined based on the specific use case. For example, a high-end NVIDIA GPU chip can be quite expensive but it can handle hundreds of video channels concurrently, reducing the number of expensive workstations required to monitor large facilities. At the other end, a low cost SoC chip from MediaTek or Broadcom will offer limited performance, about five frames per second and enabling only frontal face recognition, but it will likely be powerful enough and make use cases such as door access widely affordable.
One of the most important drivers of facial recognition optimization is the AI chipset, or system-on-chip (SoC). There is an abundance of chipset options - from manufacturers including Intel, NVIDIA, MediaTek, NXP, Qualcomm, among others - and they each bring their own benefits depending on the use case. Each chipset is designed for different computation power, form factor, power consumption, and has its own AI inference engine.
Leading chipset makers like NVIDIA, Intel, Qualcomm, MediaTek and NXP have caught onto the increasing demand for AI on edge and IoT. They are rapidly coming to market with new hardware - APU (AI Processing Units), VPU (Vision Processing Units) or NPU (Neural Processing Units) — all of which can speed up image processing and AI inferencing, while optimizing performance and power consumption.
The tables below present a handful of SoC, GPU, VPU products that integrate with many facial recognition engines, including FaceMe®. The list is in no mean exhaustive, but it illustrates a wide range of options.
These chips are designed with powerful hardware and high performance in facial recognition. When using these chips to build a system, a separate CPU chip is required.
These chips are designed to power complete yet affordable solutions, for example enabling facial recognition in small mass-market IoT devices. These chips all integrate a CPU inside, resulting in simpler, cheaper options.
Chipsets are often designed to run on specific operating systems (OS). A good facial recognition engine should support as many Chipset+OS combinations as possible. FaceMe® offers one of the market's most comprehensive and growing chipset support, and more than 10 OSs:
CyberLink designed FaceMe® to be highly versatile, allowing flexible customization options to its platform, to enable multiple combinations of hardware chips, OS and feature deployment designed to match end-users' unique needs. FaceMe®'s multi-OS support is ideal for cross-platform solutions. Developers have access to several GPU acceleration options, harnessing OpenVINO, NVIDIA CUDA/TensorRT, Intel Movidius, NVIDIA Jetson, Qualcomm SNPE, MediaTek NeuroPilot and more to speed up deep learning algorithms and further optimize performance.
It is never easy to design a good facial recognition system running on a high performance workstation or PC with GPU (or VPU) because there are dozens of concurrent video streams running between CPU, GPU and memory over the system bus. If not properly implemented on a system architecture level, even a very good facial recognition algorithm will be slow. As such, the system architecture design should minimize the data flow between CPU, GPU and memory.
FaceMe® has optimized system architecture through several iterations to ensure it delivers the absolute best performance. For example, on a single workstation, FaceMe® with NVIDIA RTX A6000 can handle 256-416 frames per second (the exact number may vary depending on which FaceMe® facial recognition model is used). This is equivalent to handling 25-41 concurrent video channels (each with 10 frames per second) per workstation. An outstanding cost-performance offering.
Several cost-constrained use cases don't need more than basic frontal face recognition (e.g., smart door locks). They are driving market demand for light-weight AI models that can achieve reasonable facial recognition on low cost devices. FaceMe® provides threemodels for this purpose:
One important attribute that sets apart leading facial recognition solutions like FaceMe® is the flexibility across all relevant types of hardware. FaceMe® can be deployed across workstations, computers, mobile and IoT devices. Let's examine a few examples.
Organizations looking to deploy facial recognition SDK over tens or hundreds of video channels across large facilities will benefit from a workstation powered by a high-end GPU capable of handling multiple IP camera video feeds simultaneously. Properties as different as department stores, airports, industrial plants or hospitals, to name a few, all have dozens or even hundreds of cameras serving varied purposes ranging from security and access control to visitor behavior analysis, crowd management and VIP customer identification. These are all use cases enabled or optimized by facial recognition. Connecting all the cameras to one or a few central workstations running facial recognition is the easiest, most robust and probably most economical solution.
To learn more about the benefits of FaceMe® for workstations, check out our partnership with VIVOTEK and integration in their facial recognition solution. Learn how to build a workstation for facial recognition.
PCs are most commonly used for facial recognition by smaller operations or for single use cases. Take a store or restaurant that wants to identify VIPs, automatically clock-in employees or get alerts for block listed people. In the current pandemic, add the ability to ensure everyone who enters the venue - employees and customers - is wearing a mask and does not have a high temperature. The store or restaurant management can install an IP or USB camera at the front and back door, and simply connect them to a PC that runs a robust integrated facial recognition software. FaceMe® Security is a value-priced ready-to-deploy software solution that includes all these features. Learn how to build an IPC for facial recognition.
The potential of facial recognition technology on mobile devices goes way beyond unlocking a cell phone. One compelling fintech use case is the integration of eKYC (electronic know your customer) facial recognition technologies on mobile phones to strengthen identity verification for online banking, loan applications, insurance and more.
The fast-paced edge-computing innovation that drives performance while cutting costs, is opening the door to endless IoT device use cases powered by facial recognition. Smart kiosks offer a compelling example. Frequent travelers are all too familiar with Global Entry and Clear kiosks, both using facial recognition. Now smart kiosks integrating facial recognition are being deployed in fast food restaurants, hospitals and hotels. Self check-in kiosks are quickly being introduced in large hotel chain to cut the wait. Adding a facial recognition engine like FaceMe® can enable an exciting personalized experience using opted-in guests' face as the only ID they need through their stay. Learn how to build an AIoT device for facial recognition.
As reviewed earlier on, edge-based facial recognition is by far more secure than cloud-based options that require individuals' pictures and videos to be sent to a cloud computing server through the internet, a process inherently vulnerable to attacks and leakage. With edge-based, most risks are alleviated as the only data captured and stored takes the form of encrypted face templates, and the entire process can be run without any cloud connection.
For FaceMe®, all data is secured before being stored to any database using AES-256 bit encryption. AES is one of the best symmetric encryption algorithms and 256-bit is the highest security confirmation. By storing the encrypted file (with a secret key) outside the original source or platform server, the face template is completely protected, even if physical devices were compromised or stolen.
It's critical to note that individuals must opt-in any facial recognition program requiring to enroll a face picture. In edge-based solutions, the captured information will consist of template data (a mathematical number in very high dimension) for future matching and identification purposes. The template doesn't contain an actual face picture, it cannot be used to recompose someone's face, and is kept separately from any personal information that could lead to the person's identification. The encrypted data captured to perform facial recognition can only be used to establish a match with the enrolled template stored in a secure database. Many data privacy laws and regulations (such as GDPR, CCPA, BIPA and LGPD) count biometric data as personal information, and therefore any business looking to employ face recognition must obtain the user's consent.
When evaluating facial recognition providers, it is important to find their headquarters' and key facilities' location. The U.S. government has expressed fair concerns around surveillance technologies from companies that are based in China and Russia - as they might not have adequate opt-in or data protection requirements. Most facial recognition solutions are safe and rigorously apply strict data and privacy protection standards. But as an end-user, you should be able to fully trust your service provider, especially when it comes to security, privacy and protection of human rights.
Facial recognition and biometric technology, when developed and deployed responsibly, have the potential to radically transform many facets of everyday life for the better. With so much potential to improve safety, security and customer experiences, we cannot be blind to the other side of the equation and the concerns it's raised. We cannot ignore the recent events, namely a number of public safety cases, that have put facial recognition in the crosshairs of advocacy groups, raising criticisms over potential bias and privacy infringement.
While there currently is not any federal regulation, select states have taken to engage in legal processes for facial regulation. Illinois was the first state to address facial recognition - passing the Biometric Information Privacy Act (BIPA) in 2008 that offers strict rules on how a private business can collect and use biometric data - including facial data. Nearly ten years later in 2020, the California Consumer Privacy Act (CCPA) went into effect, granting residents actionable rights to claim what data (including biometric data) companies have collected on them and reserves them the right to ask for it deleted. The state of Washington also passed a law, which goes into effect in 2021, requiring full transparency into how government entities - such as law enforcement - use the technology.
Moreover, while there have been federal discussions around regulation, with the Facial Recognition and Biometric Technology Moratorium Act being proposed most recently in June 2020 - nothing has gone into effect yet.
With that in mind, here at CyberLink we are leaning into ethical implementation and are of the opinion that facial recognition should not be shuttered in its entirety. We fully encourage lawmakers to develop legislation that protects individuals, while allowing this technology to do its job to improve safety and convenience for society more broadly. Those pioneering this space need to be involved, open and transparent about how this technology works, how it should be used, and how individual privacy can be protected.
As these debates continue onward, there are many success stories where the deployment of facial recognition has contributed to increase safety, while creating positive user experiences. A recent survey conducted the by the Security Industry Association (SIA) found that a majority of Americans (68%) believe facial recognition can make society safer, with particular support for the technology in airports (75% for airlines; 69% for TSA), office buildings (70%) and banks (68%). While we have outlined a number of use cases throughout this overview, we will now outline a few more and highlight key considerations for each.
Key relevant use cases fall under 5 major categories:
Example: FaceMe® Security offers a comprehensive application of facial recognition to surveillance and security.
Example: FaceMe® Health is a software solution easily deployed on a PC connected to a USB and a thermal camera to perform health checks and ensure a secure office, store or restaurant:
AI Biometrics can power a number of use case relevant to specific vertical segments:
Industrial facilities, plants and warehouses often require strict access control and monitoring for employees and visitors, as well as authentication to operate machinery and equipment, with facial recognition providing a solution to manage such tasks. With new social distance rules in place from the pandemic, facial recognition can also ensure mask compliance for staff on the warehouse floor. Learn more about how to deploy facial recognition technology in factories and warehouses.
Facial recognition is already present in airports and public stations, from interactive kiosks such as Clear and Global Entry, to automated aircraft boarding, security monitoring, and more. The COVID-19 pandemic has exacerbated the challenges to maintain safe and healthy environments in these large facilities often crowded with transient people. Facial recognition can help take on these challenges, enabling contactless check-in and health screening to monitor mask compliance and make sure no passengers board a plane, train or bus with an elevated temperature and risk of infecting others around them.
Innovation for smart cities, which include smart offices, homes and apartment buildings, is growing at a rapid pace, and the access control, safety and health monitoring needs are fairly similar to each other. Same for schools and hospitals. Maintaining security of these properties is often expensive and driven by human interventions. With facial recognition, many access control and monitoring tasks can be seamlessly automated and made more secure.
Facial recognition technology offers unique solutions to transform retail and deliver new, compelling customer experiences. It can provide precise demographic information to dynamically adjust digital signage, identify VIP customers, gather anonymized data on visitors' age and gender mix, mood, time spent in specific areas and more.
Customer authentication using eKYC (electronic Know Your Customer) to prevent fraud is currently one of the financial sector's hottest technologies. Facial recognition offers a perfect solution for eKYC, both for online and offline banking, including user authentication at an ATM, validating the identity of someone applying for a loan or insurance coverage, or to protect online banking transactions. Facial recognition can also greatly enhance physical security at banking facilities, by alerting security personnel of the presence of a block-listed individual before they even walk in or perform forensics. Discover how Facial Recognition Turns KYC into eKYC for Fraud Prevention.
For hotel operators, facial recognition brings unique benefits to deliver streamlined personalized experiences. Front desk staff can be automatically notified when a VIP walks in a hotel. Facial recognition can grant access to guest areas, select the right floor at the elevator and unlock the room door. Fast food restaurants are investing heavily in self-order kiosks, digital signage, automated curbside pickup stations and drive-in technology. Reward programs are fully embedded in these systems, often with the help of smartphone apps requiring a password and multiple steps. Adding facial recognition can significantly streamline and integrate compelling interactive experiences to the quick service restaurant industry. Discover how to Reinvent Restaurant Experiences Through Facial Recognition.
Facial recognition technology is poised to make our world a better place - but to do that, there must be broader levels of education on ethical implementation to get individuals everywhere more comfortable and accepting of businesses to openly adopt this AI biometric technology as a new, safe standard.
Facial recognition, and the potential it holds, is more than what the mainstream fear-mongering make it - it's businesses keeping their employees safe by automating secure access control to the office; it's retailers delivering stronger customer experiences in their own stores; it's manufacturers simplifying access to their many restricted areas; it's banks and fintech companies introducing much stronger authentication and cutting-edge security control; and that's just the tip of the iceberg.
Facial recognition is the future of AI biometric technology. The industry must better educate consumers and debunk the many falsehoods circulated about this technology, while explaining its positive value and potential for good. It also needs to be regulated appropriately so as to not hinder innovation, but embrace its many benefits.
People who read this article also checked:
Facial Recognition – How is It Used in 2021? This substantive article is an essential complement that covers details about the key use cases for facial recognition across industries and highlights important considerations to ensure a successful implementation.
