Skip to main content

Object Detection

Getting Started

This is an expo module that lets you use the MLKit Object Detection library in your Expo app.

Installation

Install like any other npm package:

#yarn
yarn add react-native-mlkit-object-detection

#npm
npm install react-native-mlkit-object-detection

Basic Usage

1. Set up the model context provider

Use the useObjectDetectionModels hook to fetch an ObjectDetectionModelContextProvider. This will make the models available via React context.

// App.tsx

import {
AssetRecord,
useObjectDetectionModels,
} from "react-native-mlkit-object-detection";

// For descriptions of options for default models see link below this snipped.
function App() {
// fetch the provider component from the hook
const { ObjectDetectionModelContextProvider } = useObjectDetectionModels({
loadDefaultModel: true,
defaultModelOptions: {
shouldEnableMultipleObjects: true,
shouldEnableClassification: true,
detectorMode: "singleImage",
},
});

return (
<ObjectDetectionModelContextProvider>
{/* Rest of your app */}
</ObjectDetectionModelContextProvider>
);
}

2. Fetch the model using the useObjectDetectionModel hook, and use it to detect objects in an image

Models can be quite large, take a while to load and can consume a lot of memory. You should consider where in your app's lifecycle you load the model.

// MyComponent.tsx

import {
useObjectDetector,
RNMLKitDetectedObject,
} from "@infinitered/react-native-mlkit-object-detection";
import { useEffect } from "react";

function MyComponent() {
// fetch the model from the hook, if you don't pass a model name it will fetch the default MLKit Object Detection model
const { model } = useObjectDetector();

const [modelLoaded, setModelLoaded] = useState(model?.isLoaded() ?? false);

// Models must be loaded before they can be used. This can be slow, and consume
// a lot of resources so consider carefully where and when to load the model
useEffect(() => {
// Loading models is done asynchronously, so in a useEffect we need to wrap it in an async function
async function loadModel() {
if (!model || modelIsLoaded) return;
// load the model
await model.load();
// set the model loaded state to true
setModelLoaded(true);
}

loadModel();
}, [model, modelIsLoaded]);

// the output of the model is an array of `RNMLKitDetectedObject` objects
const [result, setResult] = useState<RNMLKitDetectedObject[]>([]);

useEffect(() => {
if (!modelLoaded) return;

// model.detectObjects is async, so when we use it in a useEffect, we need to wrap it in an async function
async function detectObjects(image: AssetRecord) {
const result = await model.detectObjects(image);
setResult(result);
}

detectObjects();
}, [model, modelLoaded]);

return <View>{JSON.stringify(result)}</View>;
}
tip

To use a custom TFLite model for inference, see Using a Custom Model.