VisionCamera Frame Processor Plugin to label images using MLKit Vision

Overview

vision-camera-image-labeler

A VisionCamera Frame Processor Plugin to label images using MLKit Vision Image Labeling.

Installation

npm install vision-camera-image-labeler
cd ios && pod install

Add the plugin to your babel.config.js:

module.exports = {
  plugins: [
    [
      'react-native-reanimated/plugin',
      {
        globals: ['__labelImage'],
      },
    ],

    // ...

Note: You have to restart metro-bundler for changes in the babel.config.js file to take effect.

Usage

import { labelImage } from "vision-camera-image-labeler";

// ...

const frameProcessor = useFrameProcessor((frame) => {
  'worklet';
  const labels = labelImage(frame);
}, []);

Contributing

See the contributing guide to learn how to contribute to the repository and the development workflow.

License

MIT

Issues
  • Fix and upgrade example app

    Fix and upgrade example app

    This PR fixes some issues that lead to the example app not running properly on Android (and maybe also on iOS?). It also includes some dependency updates and changes that were recently added to the create-react-native-library template.

    Unfortunately I don't have time to go into details about the individual changes right now, but the commits should do an okay job of highlighting all changes.

    Currently the frame processor itself isn't working (at least on Android, I couldn't test iOS) and throws an error:

    Frame Processor threw an error: Value is undefined, expected an Object
    

    Trying to console.log() the frame object throws

    Frame Processor threw an error: Exception in HostObject::get(propName:bytesPerRow): java.lang.NoSuchMethodError: no non-static method "Landroidx/camera/core/SettableImageProxy;.getPlanes()[Ljava/lang/Object;"
    

    so maybe that gives some indication what the issue is.

    I marked the PR as a draft because I'm unsure whether this is an issue with the plugin implementation, the example app or frame processors on Android in general.

    Still to do

    • [x] Fix issue described above
    • [x] ~~Complete VisionCamera setup on iOS (Set Deployment Target to >= 11.0, Swift Version to >= 5.2)~~
    • [x] Update pods for iOS (yarn pods)
    • [x] Test if example works on iOS
    opened by yfunk 17
  • Frame Processor threw an error: Exception in HostFunction: java.lang.NoSuchMethodError: no non-static method

    Frame Processor threw an error: Exception in HostFunction: java.lang.NoSuchMethodError: no non-static method "Lcom/facebook/react/bridge/ReadableArray;.toArrayList()[Ljava/lang/Object;"

    On Android Frame Processor threw an error: Exception in HostFunction: java.lang.NoSuchMethodError: no non-static method "Lcom/facebook/react/bridge/ReadableArray;.toArrayList()[Ljava/lang/Object;" is thrown with every tick of the frameProcessor.

    Device: Pixel 4a.

    Logcat:

    2021-08-03 12:16:58.546 13182-13392/com.supervaisor.dashcam.staging E/VisionCamera: Frame Processor threw an error: Exception in HostFunction: java.lang.NoSuchMethodError: no non-static method "Lcom/facebook/react/bridge/ReadableArray;.toArrayList()[Ljava/lang/Object;"
    2021-08-03 12:16:58.547 13182-13283/com.supervaisor.dashcam.staging E/ReactNativeJS: Frame Processor threw an error: Exception in HostFunction: java.lang.NoSuchMethodError: no non-static method "Lcom/facebook/react/bridge/ReadableArray;.toArrayList()[Ljava/lang/Object;"
    2021-08-03 12:16:58.800 987-987/? W/SoloThreadMana: type=1400 audit(0.0:717094): avc: denied { search } for name="camera" dev="dm-12" ino=209 scontext=u:r:hal_camera_default:s0 tcontext=u:object_r:camera_vendor_data_file:s0 tclass=dir permissive=0
    2021-08-03 12:16:58.984 1753-2127/? D/ConnectivityService: NetReassign [no changes]
    2021-08-03 12:16:59.000 987-987/? I/chatty: uid=1000(system) [email protected] identical 3 lines
    2021-08-03 12:16:59.066 987-987/? W/SoloThreadMana: type=1400 audit(0.0:717098): avc: denied { search } for name="camera" dev="dm-12" ino=209 scontext=u:r:hal_camera_default:s0 tcontext=u:object_r:camera_vendor_data_file:s0 tclass=dir permissive=0
    2021-08-03 12:16:59.607 13182-13392/com.supervaisor.dashcam.staging I/HermesVM: JSI rethrowing JS exception: Exception in HostFunction: java.lang.NoSuchMethodError: no non-static method "Lcom/facebook/react/bridge/ReadableArray;.toArrayList()[Ljava/lang/Object;"
        
        Error: Exception in HostFunction: java.lang.NoSuchMethodError: no non-static method "Lcom/facebook/react/bridge/ReadableArray;.toArrayList()[Ljava/lang/Object;"
            at __labelImage (native)
            at labelImage (JavaScript:1:48)
            at labelImage (native)
            at _f (JavaScript:1:77)
            at _f (native)
            at _f (JavaScript:1:425)
            at _f (native)
    2021-08-03 12:16:59.608 13182-13392/com.supervaisor.dashcam.staging I/chatty: uid=10390(com.supervaisor.dashcam.staging) pool-22-thread- identical 2 lines
    2021-08-03 12:16:59.608 13182-13392/com.supervaisor.dashcam.staging I/HermesVM: JSI rethrowing JS exception: Exception in HostFunction: java.lang.NoSuchMethodError: no non-static method "Lcom/facebook/react/bridge/ReadableArray;.toArrayList()[Ljava/lang/Object;"
        
        Error: Exception in HostFunction: java.lang.NoSuchMethodError: no non-static method "Lcom/facebook/react/bridge/ReadableArray;.toArrayList()[Ljava/lang/Object;"
            at __labelImage (native)
            at labelImage (JavaScript:1:48)
            at labelImage (native)
            at _f (JavaScript:1:77)
            at _f (native)
            at _f (JavaScript:1:425)
            at _f (native)
    2021-08-03 12:16:59.609 13182-13392/com.supervaisor.dashcam.staging E/VisionCamera: Frame Processor threw an error: Exception in HostFunction: java.lang.NoSuchMethodError: no non-static method "Lcom/facebook/react/bridge/ReadableArray;.toArrayList()[Ljava/lang/Object;"
    
    opened by Kypsis 7
  • doubts: where does this come from, and where should this log appear?

    doubts: where does this come from, and where should this log appear?

    https://github.com/mrousavy/vision-camera-image-labeler/blob/cbf88a545e2aabcbc97aaf2717617ef1e0a69aa9/example/src/App.tsx#L25

    opened by rodgomesc 3
  • chore: attempt to fix Example app on iOS

    chore: attempt to fix Example app on iOS

    This PR attempt to fix iOS Example app build. So far I was able to fix XCode errors and build, however the Metro fails to build/detect the app right now. Further work and debug is required.

    Changes included in this PR:

    • Swift bumped from 5.0 to 5.2 and target updated to iOS 11.0 (as described in Vision Camera Troubleshooting - https://mrousavy.github.io/react-native-vision-camera/docs/guides/troubleshooting/#ios)
    • removed tvOS targets from XCode Workspace
    • File.swift moved to the XCode Workspace
    • added missing babel-plugin-module-resolver dependency
    • updated used Vision Camera to fix on of XCode errors (usePixelAspecRatio)
    • regenerated Pods

    Currently the example iOS app build correctly, but Metro fails to detect/build the React app inside.

    opened by Simek 3
  • java.lang.IllegalStateException: Task is not yet complete

    java.lang.IllegalStateException: Task is not yet complete

    When trying to run the plugin I get Frame Processor threw an error: Exception in HostFunction: java.lang.IllegalStateException: Task is not yet complete every tick of the frameProcessor on Android. Looks like async error?

    Device is Pixel 4a.

    Logcat:

    2021-08-02 19:48:28.469 22920-23204/com.supervaisor.dashcam.staging E/VisionCamera: Frame Processor threw an error: Exception in HostFunction: java.lang.IllegalStateException: Task is not yet complete
    2021-08-02 19:48:28.471 22920-23028/com.supervaisor.dashcam.staging E/ReactNativeJS: Frame Processor threw an error: Exception in HostFunction: java.lang.IllegalStateException: Task is not yet complete
    2021-08-02 19:48:29.112 987-987/? W/SoloThreadMana: type=1400 audit(0.0:711721): avc: denied { search } for name="camera" dev="dm-12" ino=209 scontext=u:r:hal_camera_default:s0 tcontext=u:object_r:camera_vendor_data_file:s0 tclass=dir permissive=0
    2021-08-02 19:48:29.312 987-987/? I/chatty: uid=1000(system) [email protected] identical 3 lines
    2021-08-02 19:48:29.379 987-987/? W/SoloThreadMana: type=1400 audit(0.0:711725): avc: denied { search } for name="camera" dev="dm-12" ino=209 scontext=u:r:hal_camera_default:s0 tcontext=u:object_r:camera_vendor_data_file:s0 tclass=dir permissive=0
    2021-08-02 19:48:29.471 22920-23204/com.supervaisor.dashcam.staging I/HermesVM: JSI rethrowing JS exception: Exception in HostFunction: java.lang.IllegalStateException: Task is not yet complete
        Error: Exception in HostFunction: java.lang.IllegalStateException: Task is not yet complete
            at __labelImage (native)
            at labelImage (JavaScript:1:48)
            at labelImage (native)
            at _f (JavaScript:1:77)
            at _f (native)
    2021-08-02 19:48:29.472 22920-23204/com.supervaisor.dashcam.staging I/chatty: uid=10390(com.supervaisor.dashcam.staging) pool-22-thread- identical 1 line
    2021-08-02 19:48:29.472 22920-23204/com.supervaisor.dashcam.staging I/HermesVM: JSI rethrowing JS exception: Exception in HostFunction: java.lang.IllegalStateException: Task is not yet complete
        Error: Exception in HostFunction: java.lang.IllegalStateException: Task is not yet complete
            at __labelImage (native)
            at labelImage (JavaScript:1:48)
            at labelImage (native)
            at _f (JavaScript:1:77)
            at _f (native)
    
    opened by Kypsis 1
  • IOS build fails due to conversion error.

    IOS build fails due to conversion error.

    Screenshot 2021-08-03 at 09 26 26

    Trying to build with Xcode 12.5.1 throws that error.

    opened by Kypsis 1
  • Module MLKit not found

    Module MLKit not found

    In VisionCameraImageLabeler.m Xcode complains about Module 'MLKit' not found on line 13 @import MLKit;. If you delete that import the build fails at the very end with: duplicate symbol '_OBJC_CLASS_$_CameraViewManager' in: /Users/harrysild/Library/Developer/Xcode/DerivedData/Dashcam-cujsxchigczhdvbgfkjhdhokpctr/Build/Intermediates.noindex/Dashcam.build/StagingDebug-iphoneos/Dashcam.build/Objects-normal/arm64/CameraViewManager-e2616307afa5333b3b6da1885b39637cff4d06f18eed5d8b3bc0e505c1525a74.o /Users/harrysild/Library/Developer/Xcode/DerivedData/Dashcam-cujsxchigczhdvbgfkjhdhokpctr/Build/Products/StagingDebug-iphoneos/VisionCamera/libVisionCamera.a(CameraViewManager-6de0dcf57a6e5b176e919ac2555340718db92047184fdb2efecb0f346968c41a.o) duplicate symbol '_OBJC_METACLASS_$_CameraViewManager' in: /Users/harrysild/Library/Developer/Xcode/DerivedData/Dashcam-cujsxchigczhdvbgfkjhdhokpctr/Build/Intermediates.noindex/Dashcam.build/StagingDebug-iphoneos/Dashcam.build/Objects-normal/arm64/CameraViewManager-e2616307afa5333b3b6da1885b39637cff4d06f18eed5d8b3bc0e505c1525a74.o /Users/harrysild/Library/Developer/Xcode/DerivedData/Dashcam-cujsxchigczhdvbgfkjhdhokpctr/Build/Products/StagingDebug-iphoneos/VisionCamera/libVisionCamera.a(CameraViewManager-6de0dcf57a6e5b176e919ac2555340718db92047184fdb2efecb0f346968c41a.o) ld: 2 duplicate symbols for architecture arm64

    opened by Kypsis 1
  • Update Fix MLKit import not found need to add to current version :)

    Update Fix MLKit import not found need to add to current version :)

    https://github.com/mrousavy/vision-camera-image-labeler/commit/68a9a8301acc71b6805816af3476bc029f05accf Good afternoon @mrousavy , I'm notice that the commit above not in current version when I install package (v0.1.5), Please update in new version, Thanks man!

    opened by NguyenPhatBG 1
  • try: Fix gradle dependency

    try: Fix gradle dependency

    opened by mrousavy 0
  • Error while running the example on android devices.....

    Error while running the example on android devices.....

    Hello!!!

    Was just trying to run the example - vision-camera-image-labeler on an android device, getting the below error after running the command,

    react-native run-android

    Can you please let me know what the error could be related to?

    FAILURE: Build failed with an exception.

    • What went wrong: A problem occurred configuring project ':react-native-vision-camera'.

    com.android.builder.errors.EvalIssueException: NDK at C:\xxxx\AppData\Local\Android\Sdk\ndk\20.1.5948944 did not have a source.properties file

    • Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

    • Get more help at https://help.gradle.org

    BUILD FAILED in 20s

    Error Failed to install the app. Make sure you have the Android development environment set up: https://reactnative.dev/docs/environment-setup. Error: Command failed: gradlew.bat app:installDebug -PreactNativeDevServerPort=8081

    FAILURE: Build failed with an exception.

    • What went wrong: A problem occurred configuring project ':react-native-vision-camera'.

    com.android.builder.errors.EvalIssueException: NDK at C:\xxxxxxxxxxxx\20.1.5948944 did not have a source.properties file

    opened by vineethdas 4
  • Question: how to initialize ImageLabeling client with different options

    Question: how to initialize ImageLabeling client with different options

    i want to initialize the labeler client with different args

    https://github.com/mrousavy/vision-camera-image-labeler/blob/cbf88a545e2aabcbc97aaf2717617ef1e0a69aa9/android/src/main/java/com/visioncameraimagelabeler/VisionCameraImageLabelerPlugin.java#L25

    but since the plugin uses callback method to receive javascript params, and do that in a frame loop. i don't how to do that,

    https://github.com/mrousavy/vision-camera-image-labeler/blob/cbf88a545e2aabcbc97aaf2717617ef1e0a69aa9/android/src/main/java/com/visioncameraimagelabeler/VisionCameraImageLabelerPlugin.java#L28

    can someone walk me through a solution for that?

    opened by rodgomesc 1
Releases(v0.1.6)
Owner
Marc Rousavy
they call me ranch cause I be dressing
Marc Rousavy
Fast computer vision library for SFM, calibration, fiducials, tracking, image processing, and more.

Table of Contents Introduction Cloning Repository Quick Start Gradle and Maven Building from Source Dependencies Help/Contact Introduction BoofCV is a

Peter Abeles 767 Sep 12, 2021
TwelveMonkeys ImageIO: Additional plug-ins and extensions for Java's ImageIO

About TwelveMonkeys ImageIO is a collection of plugins and extensions for Java's ImageIO. These plugins extend the number of image file formats suppor

Harald Kuhr 1.3k Sep 17, 2021
Roman Beskrovnyi 167 Sep 8, 2021
Simple Java image-scaling library implementing Chris Campbell's incremental scaling algorithm as well as Java2D's "best-practices" image-scaling techniques.

imgscalr - Java Image-Scaling Library http://www.thebuzzmedia.com/software/imgscalr-java-image-scaling-library/ Changelog --------- 4.2 * Added sup

Riyad Kalla 1.1k Aug 24, 2021
Diagrams as code is a term used for storing the source of a diagram image as a text file.

Diagrams as code Diagrams as code is a term used for storing the source of a diagram image as a text file. Examples are architecture diagrams, or diag

null 18 Sep 10, 2021
Custom Image Server for use with ShareX

rImageServer How to use: Normal Instalation: Download .jar Start: javar -jar [filename].jar Edit: imageserver.propierties Check if web works -> Go to

Ryzeon 13 Aug 27, 2021
ZXing ("Zebra Crossing") barcode scanning library for Java, Android

Project in Maintenance Mode Only The project is in maintenance mode, meaning, changes are driven by contributed patches. Only bug fixes and minor enha

ZXing Project 28.4k Sep 13, 2021
Java JNA wrapper for Tesseract OCR API

Tess4J A Java JNA wrapper for Tesseract OCR API. Tess4J is released and distributed under the Apache License, v2.0. Features The library provides opti

Quan Nguyen 1k Sep 16, 2021
必应每日超清壁纸(4K)

Bing Wallpaper Today: Lençóis Maranhenses National Park in the state of Maranhão, Brazil (© WIN-Initiative/Getty Images) 2021-04-03 download 4k 2021-0

null 210 Sep 13, 2021
This project allows the exchange of files between your local disk and a D64 image (Commodore 64 image disk) . Ce projet permet l'échange de fichiers entre votre disque local et une image D64 (Image de disquette du Commodore 64).

DiskToolC64 Ce projet permet l'échange de fichiers entre votre disque local et une image D64 (Image de disquette du Commodore 64). Introduction Les fi

Eddy BRIERE 3 May 13, 2021
Pw0 Framewrok - magical android pentest app 🔮! Pixie Dust, Handshakes, Deauth, Nmap, Port scanner and more!

Pw0 Framework Pw0 Framewrok - magical android pentest app ?? ! Features: Pixie Dust Handshakes Deauth Nmap Port scanner and more! Version: 0.2 Beta Au

Huntmix 12 Sep 1, 2021
Anthos Edge Use Cases for bringing apps and computation closer to the location where the action is, to improve response times and save bandwidth.

Anthos Bare Metal Edge Use Cases Edge computing is a distributed computing paradigm that brings computation and data storage closer to the location wh

Google Cloud Platform 3 Sep 15, 2021