Vision-camera-doc-scanner-plugin NPM

vision-camera-doc-scanner-plugin

Plugin for VisionCamera that adds a Document Scanner to your app.

Plugin is powered by: OpenCV and Deep Learning for the Document Detection (Image Segmentation).

TODO'S:

NOTE: This plugin is still under development

add steps to reproduce iOS
some improvements
test expo compatability
expo config plugin
Retrain model with more data

Installation

yarn add vision-camera-doc-scanner-plugin
cd ios && pod install

Add the plugin to your babel.config.js:

module.exports = {
  plugins: [
    [
      'react-native-reanimated/plugin',
      {
        globals: ['__detectDocument'],
      },
    ],

    // ...
  ],
};

Note: You have to restart metro-bundler for changes in the babel.config.js file to take effect.

Add dependencies to package.json:

"react-native-reanimated": "2.12.0",
"react-native-vision-camera": "2.15.2"

Add import on top of index.tsx:

import 'react-native-reanimated';

Android

Register package in MainApplication.java (example/android/app/src/main/java/com/example/visioncameradocscannerplugin/MainApplication.java):

import com.visioncameradocscannerplugin.VisionCameraDocScannerPluginPackage;

// ...
@Override
protected List<ReactPackage> getPackages() {
  @SuppressWarnings("UnnecessaryLocalVariable")
  List<ReactPackage> packages = new PackageList(this).getPackages();
  // Packages that cannot be autolinked yet can be added manually here, for example:
  // packages.add(new MyReactNativePackage());
  packages.add(new VisionCameraDocScannerPluginPackage());
  return packages;
}

Add Camera Permission to /android/app/src/main/AndroidManifest.xml

<manifest xmlns:android="http://schemas.android.com/apk/res/android"
  package="com.example.visioncameradocscannerplugin">

    <uses-permission android:name="android.permission.INTERNET" />
    // add following line
    <uses-permission android:name="android.permission.CAMERA" />

    //...

</manifest>

Apply fix for OpenCV in android/app/build.gradle android section:

// fix for opencv4 duplicated files
packagingOptions {
    pickFirst 'lib/arm64-v8a/libc++_shared.so'
    pickFirst 'lib/x86/libc++_shared.so'
    pickFirst 'lib/x86_64/libc++_shared.so'
    pickFirst 'lib/armeabi-v7a/libc++_shared.so'
}

iOS

Add camera usage description in your projects info.plist

<key>NSCameraUsageDescription</key>
<string>Scan Documents</string>

Usage

See the example for how to use this in your app.

import { cropImage, detectDocument } from "vision-camera-doc-scanner-plugin";

// ...
const frameProcessor = useFrameProcessor((frame) => {
  'worklet';
  const coords = detectDocument(frame);
}, []);

const takeSnapshot = async () => {
    const cachedCoords = {value: coords.value};
    if (
      (cachedCoords.value.x1 !== 0 && cachedCoords.value.y1 !== 0) ||
      (cachedCoords.value.x2 !== 0 && cachedCoords.value.y2 !== 0)
    ) {
      const data = await camera.current
        ?.takePhoto({qualityPrioritization: 'speed'})
        .catch(e => console.log(e));
      const dataPath = data?.path;

      if (dataPath) {
        const base64Images = await cropImage(
          cachedCoords,
          dataPath,
        );
        const imageStr = `data:image/jpeg;base64,${base64Images
          .replace('\n', '')
          .replace('\r', '')
          .replace(' ', '')}`;
      } else {
        console.log('no data path');
      }
    } else {
      console.log('There is nothing to crop !');
    }
  };

Data

detectDocument(frame) returns Coords with the following data shape.

 type Coords = {
   x1: number;
   y1: number;
   x2: number;
   y2: number;
   x3: number;
   y3: number;
   x4: number;
   y4: number;
   width: number;
   height: number;
 }

cropImage takes the Coords to crop and an path to the image. It returns a Base64-String where the cropped image is stored.