使用TensorFlow.js在浏览器中进行实时人脸跟踪

开课吧小一2021-04-30 21:11

点赞
有用
分享分享

您是否想过如何从头开始创建此类过滤器?好了,现在您有机会在您的Web浏览器中学习!在本系列中,我们将看到如何在浏览器中创建Snapchat样式的过滤器,如何训练AI模型以了解面部表情,以及如何使用Tensorflow.js和面部跟踪进行更多操作。

您可能需要在Web浏览器中启用WebGL才能提高性能。您还可以下载该系列的代码和文件。

我们假设您熟悉JavaScript和HTML,并且至少对神经网络有基本的了解。如果您不熟悉TensorFlow.js,我们建议您阅读本指南:使用TensorFlow.js在浏览器中进行深度学习入门。

使用TensorFlow.js在浏览器中进行实时人脸跟踪

如果您想使用TensorFlow.js在Web浏览器中看到更多功能,请查看以下AI系列:使用TensorFlow.js的计算机视觉和使用TensorFlow.js的Chatbots。

从头开始创建面部过滤器的第一步是检测并定位图像中的面部,因此我们可以从这里开始。

可以使用TensorFlow.js和人脸地标检测模型完成人脸跟踪,这可以在几毫秒内为我们在3D中为图像或视频帧内的每个人脸获取486个不同的关键点。使得此模型特别出色的是,该模型可以在网页中运行,因此您也可以使用相同的代码在移动设备上跟踪人脸。

让我们设置一个项目来加载模型并在网络摄像头视频源上运行人脸跟踪。

初始点

这是我们将用于面部跟踪的网页入门模板。

该模板包括:

该项目所需的TensorFlow.js库

在triangles.js中设置的参考面网格索引(包含在项目代码中)

用于渲染输出的canvas元素

网络摄像头的隐藏视频元素

状态文本元素和setText实用程序功能

画布drawLine和drawTriangle实用程序功能

<html>
    <head>
        <title>Real-Time Face Tracking in the Browser with TensorFlow.js</title>
        <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@2.4.0/dist/tf.min.js"></script>
        <script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/face-landmarks-detection@0.0.1/dist/face-landmarks-detection.js"></script>
        <script src="web/triangles.js"></script>
    </head>
    <body>
        <canvas id="output"></canvas>
        <video id="webcam" playsinline style="
            visibility: hidden;
            width: auto;
            height: auto;
            ">
        </video>
        <h1 id="status">Loading...</h1>
        <script>
        function setText( text ) {
            document.getElementById( "status" ).innerText = text;
        }

        function drawLine( ctx, x1, y1, x2, y2 ) {
            ctx.beginPath();
            ctx.moveTo( x1, y1 );
            ctx.lineTo( x2, y2 );
            ctx.stroke();
        }

        function drawTriangle( ctx, x1, y1, x2, y2, x3, y3 ) {
            ctx.beginPath();
            ctx.moveTo( x1, y1 );
            ctx.lineTo( x2, y2 );
            ctx.lineTo( x3, y3 );
            ctx.lineTo( x1, y1 );
            ctx.stroke();
        }

        (async () => {
            // TODO: Add code here
        })();
        </script>
    </body></html>

将HTML5网络摄像头API与TensorFlow.js结合使用

如果您有一个代码段,则在JavaScript中启动网络摄像头非常简单。这是一个实用程序功能,可用于您启动网络摄像头并请求用户访问:

async function setupWebcam() {
    return new Promise( ( resolve, reject ) => {
        const webcamElement = document.getElementById( "webcam" );
        const navigatorAny = navigator;
        navigator.getUserMedia = navigator.getUserMedia ||
        navigatorAny.webkitGetUserMedia || navigatorAny.mozGetUserMedia ||
        navigatorAny.msGetUserMedia;
        if( navigator.getUserMedia ) {
            navigator.getUserMedia( { video: true },
                stream => {
                    webcamElement.srcObject = stream;
                    webcamElement.addEventListener( "loadeddata", resolve, false );
                },
            error => reject());
        }
        else {
            reject();
        }
    });
}

我们可以setupWebcam在代码底部的异步块中调用此函数,并使其在加载后播放网络摄像头视频。

(async () => {
    await setupWebcam();
    const video = document.getElementById( "webcam" );
    video.play();
})();

接下来,让我们设置输出画布,并准备为边界框和面线框绘制线条和三角形。

canvas上下文将用于输出面部跟踪结果,因此我们可以将其全局保存在异步块之外。请注意,我们将网络摄像头水平镜像为更自然的行为,就像真实的镜像一样。

let output = null;

(async () => {
    await setupWebcam();
    const video = document.getElementById( "webcam" );
    video.play();
    let videoWidth = video.videoWidth;
    let videoHeight = video.videoHeight;
    video.width = videoWidth;
    video.height = videoHeight;

    let canvas = document.getElementById( "output" );
    canvas.width = video.width;
    canvas.height = video.height;

    output = canvas.getContext( "2d" );
    output.translate( canvas.width, 0 );
    output.scale( -1, 1 ); // Mirror cam
    output.fillStyle = "#fdffb6";
    output.strokeStyle = "#fdffb6";
    output.lineWidth = 2;
})();

让我们追踪一些面孔

现在我们准备好了!我们所需要做的就是加载TensorFlow人脸地标检测模型并在网络摄像头框架上运行以显示结果。

首先,我们需要一个全局模型变量来存储加载的模型:

let model = null;

然后,我们可以在异步块的末尾加载模型,并设置状态文本以表明我们的面部跟踪应用已准备就绪:

// Load Face Landmarks Detection
model = await faceLandmarksDetection.load(
    faceLandmarksDetection.SupportedPackages.mediapipeFacemesh
);

setText( "Loaded!" );

现在,让我们创建一个名为的函数trackFace,该函数获取网络摄像头视频帧,运行面部跟踪模型,将网络摄像头图像复制到输出画布,然后在面部周围绘制边框,并在面部顶部绘制线框网格三角形。

async function trackFace() {
    const video = document.getElementById( "webcam" );
    const faces = await model.estimateFaces( {
        input: video,
        returnTensors: false,
        flipHorizontal: false,
    });
    output.drawImage(
        video,
        0, 0, video.width, video.height,
        0, 0, video.width, video.height
    );

    faces.forEach( face => {
        setText( `Face Tracking Confidence: ${face.faceInViewConfidence.toFixed( 3 )}` );

        // Draw the bounding box
        const x1 = face.boundingBox.topLeft[ 0 ];
        const y1 = face.boundingBox.topLeft[ 1 ];
        const x2 = face.boundingBox.bottomRight[ 0 ];
        const y2 = face.boundingBox.bottomRight[ 1 ];
        const bWidth = x2 - x1;
        const bHeight = y2 - y1;
        drawLine( output, x1, y1, x2, y1 );
        drawLine( output, x2, y1, x2, y2 );
        drawLine( output, x1, y2, x2, y2 );
        drawLine( output, x1, y1, x1, y2 );

        // Draw the face mesh
        const keypoints = face.scaledMesh;
        for( let i = 0; i < FaceTriangles.length / 3; i++ ) {
            let pointA = keypoints[ FaceTriangles[ i * 3 ] ];
            let pointB = keypoints[ FaceTriangles[ i * 3 + 1 ] ];
            let pointC = keypoints[ FaceTriangles[ i * 3 + 2 ] ];
            drawTriangle( output, pointA[ 0 ], pointA[ 1 ], pointB[ 0 ], pointB[ 1 ], pointC[ 0 ], pointC[ 1 ] );
        }
    });

    requestAnimationFrame( trackFace );}

最后,我们可以通过在异步块末尾调用此函数来开始跟踪的第一帧:

(async () => {
    ...

    trackFace();
})();

终点线

完整的代码应如下所示:

<html>
    <head>
        <title>Real-Time Face Tracking in the Browser with TensorFlow.js</title>
        <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@2.4.0/dist/tf.min.js"></script>
        <script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/face-landmarks-detection@0.0.1/dist/face-landmarks-detection.js"></script>
        <script src="web/triangles.js"></script>
    </head>
    <body>
        <canvas id="output"></canvas>
        <video id="webcam" playsinline style="
            visibility: hidden;
            width: auto;
            height: auto;
            ">
        </video>
        <h1 id="status">Loading...</h1>
        <script>
        function setText( text ) {
            document.getElementById( "status" ).innerText = text;
        }

        function drawLine( ctx, x1, y1, x2, y2 ) {
            ctx.beginPath();
            ctx.moveTo( x1, y1 );
            ctx.lineTo( x2, y2 );
            ctx.stroke();
        }

        function drawTriangle( ctx, x1, y1, x2, y2, x3, y3 ) {
            ctx.beginPath();
            ctx.moveTo( x1, y1 );
            ctx.lineTo( x2, y2 );
            ctx.lineTo( x3, y3 );
            ctx.lineTo( x1, y1 );
            ctx.stroke();
        }

        let output = null;
        let model = null;

        async function setupWebcam() {
            return new Promise( ( resolve, reject ) => {
                const webcamElement = document.getElementById( "webcam" );
                const navigatorAny = navigator;
                navigator.getUserMedia = navigator.getUserMedia ||
                navigatorAny.webkitGetUserMedia || navigatorAny.mozGetUserMedia ||
                navigatorAny.msGetUserMedia;
                if( navigator.getUserMedia ) {
                    navigator.getUserMedia( { video: true },
                        stream => {
                            webcamElement.srcObject = stream;
                            webcamElement.addEventListener( "loadeddata", resolve, false );
                        },
                    error => reject());
                }
                else {
                    reject();
                }
            });
        }

        async function trackFace() {
            const video = document.getElementById( "webcam" );
            const faces = await model.estimateFaces( {
                input: video,
                returnTensors: false,
                flipHorizontal: false,
            });
            output.drawImage(
                video,
                0, 0, video.width, video.height,
                0, 0, video.width, video.height
            );

            faces.forEach( face => {
                setText( `Face Tracking Confidence: ${face.faceInViewConfidence.toFixed( 3 )}` );

                // Draw the bounding box
                const x1 = face.boundingBox.topLeft[ 0 ];
                const y1 = face.boundingBox.topLeft[ 1 ];
                const x2 = face.boundingBox.bottomRight[ 0 ];
                const y2 = face.boundingBox.bottomRight[ 1 ];
                const bWidth = x2 - x1;
                const bHeight = y2 - y1;
                drawLine( output, x1, y1, x2, y1 );
                drawLine( output, x2, y1, x2, y2 );
                drawLine( output, x1, y2, x2, y2 );
                drawLine( output, x1, y1, x1, y2 );

                // Draw the face mesh
                const keypoints = face.scaledMesh;
                for( let i = 0; i < FaceTriangles.length / 3; i++ ) {
                    let pointA = keypoints[ FaceTriangles[ i * 3 ] ];
                    let pointB = keypoints[ FaceTriangles[ i * 3 + 1 ] ];
                    let pointC = keypoints[ FaceTriangles[ i * 3 + 2 ] ];
                    drawTriangle( output, pointA[ 0 ], pointA[ 1 ], pointB[ 0 ], pointB[ 1 ], pointC[ 0 ], pointC[ 1 ] );
                }
            });

            requestAnimationFrame( trackFace );
        }

        (async () => {
            await setupWebcam();
            const video = document.getElementById( "webcam" );
            video.play();
            let videoWidth = video.videoWidth;
            let videoHeight = video.videoHeight;
            video.width = videoWidth;
            video.height = videoHeight;

            let canvas = document.getElementById( "output" );
            canvas.width = video.width;
            canvas.height = video.height;

            output = canvas.getContext( "2d" );
            output.translate( canvas.width, 0 );
            output.scale( -1, 1 ); // Mirror cam
            output.fillStyle = "#fdffb6";
            output.strokeStyle = "#fdffb6";
            output.lineWidth = 2;

            // Load Face Landmarks Detection
            model = await faceLandmarksDetection.load(
                faceLandmarksDetection.SupportedPackages.mediapipeFacemesh
            );

            setText( "Loaded!" );

            trackFace();
        })();
        </script>
    </body></html>

下一步是什么?人脸追踪还能做更多吗?

通过将TensorFlow人脸地标检测模型与网络摄像头视频相结合,我们能够在浏览器中实时跟踪人脸。我们的面部跟踪代码也可以在图像上使用,关键点可以告诉我们的东西超出了我们最初的预期。也许我们应该在面部数据集上尝试一下,例如FER +面部表情识别?

以上就是小编为大家整理的“使用TensorFlow.js在浏览器中进行实时人脸跟踪”一文,更多信息尽在AI人工智能教程频道。

相关推荐:

免费领完整的AI学习路径资料,带你轻松入门!

AI资料难找吗?AI免费论文资料,等你领取!

福利来袭!人工智能核心课程优惠名额等你来领

有用
分享