Using TensorFlow’s PoseNet in ReactJS

I saw this post about PoseNet for JS and decided to give it a try.
The instructions are somewhat clear but still you’ll probably run into unexpected issues that are not documented on the Git repo of the project.

Because of this I decided to write a brief guide with the pitfalls I ran into while trying to get PoseNet to work in my ReactJS project.

1. Installing the package and dependencies

Aside of installing the model package, you should also install tensorflow/tfjs. For some reason only installing the tensorflow-models/posenet won’t install all the necessary dependencies.

Add the following to your package.json

# package.json"@tensorflow-models/posenet": "^2.2.1",
"@tensorflow/tfjs": "^2.8.1",

or run

npm install @tensorflow/tfjs
npm install @tensorflow-models/posenet

2. Import TF on your index file

# index.tsximport '@tensorflow/tfjs';

If you don’t do this you’ll get the error message “no backend found in registry”.

3. Loading the model

If you are using TensorFlow on a class component, just add the following on the componentDidMount function with your desired model properties.

4. Using an HTML video component as source

If you are consuming the images directly from a html video component you must add the same size you used in the inputResolution in the initialization of the model, and also you should set crossOrigin=”anonymous”

Why should I set the crossOrigin property?

If you don’t set the crossOriging properly, you’ll get the error “Failed to execute ‘texImage2D’ on ‘WebGL2RenderingContext”.

Why should I set the video width and height too?

If you don’t set the width and height in the video component, when getting results from your model all the coordinates will have {x:0, y:0};

Another tip in case you are passing different video Urls, add a key related to the Url to force the component to rerender everytime the url changes.

5. Calling the model to get estimated poses coordinates

And that’s it. Once you have this setup in place you’ll be able to call the model

poseModel.estimateMultiplePoses(playerEl.current).then((result) => {
// do whatever you want with the model results
}

The coordinates will be in the ranges defined by inputResolution defined when loading the model.

Software/Hardware Engineer -Traveler- Explorer-Maker — just wondering around… (wonderer perhaps?)