whisper.rn v0.3.9

whisper.rn

React Native binding of whisper.cpp.

whisper.cpp: High-performance inference of OpenAI's Whisper automatic speech recognition (ASR) model

Run example with release mode on iPhone 13 Pro Max

Installation

npm install whisper.rn

Then re-run npx pod-install again for iOS.

For Expo, you will need to prebuild the project before using it. See Expo guide for more details.

Add Microphone Permissions (Optional)

If you want to use realtime transcribe, you need to add the microphone permission to your app.

iOS

Add these lines to ios/[YOU_APP_NAME]/info.plist

<key>NSMicrophoneUsageDescription</key>
<string>This app requires microphone access in order to transcribe speech</string>

For tvOS, please note that the microphone is not supported.

Android

Add the following line to android/app/src/main/AndroidManifest.xml

<uses-permission android:name="android.permission.RECORD_AUDIO" />

Usage

import { initWhisper } from 'whisper.rn'

const whisperContext = await initWhisper({
  filePath: 'file://.../ggml-base.en.bin',
  isBundleAsset: false, // Set to true if you want to load the model from bundle resources, the filePath will be like `ggml-base.en.bin`
})

const sampleFilePath = 'file://.../sample.wav'
const options = { language: 'en' }
const { stop, promise } = whisperContext.transcribe(sampleFilePath, options)

const { result } = await promise
// result: (The inference text result from audio file)

Use realtime transcribe:

const { stop, subscribe } = await whisperContext.transcribeRealtime(options)

subscribe(evt => {
  const { isCapturing, data, processTime, recordingTime } = evt
  console.log(
    `Realtime transcribing: ${isCapturing ? 'ON' : 'OFF'}\n` +
      // The inference text result from audio record:
      `Result: ${data.result}\n\n` + 
      `Process time: ${processTime}ms\n` +
      `Recording time: ${recordingTime}ms`,
  )
  if (!isCapturing) console.log('Finished realtime transcribing')
})

In Android, you may need to request the microphone permission by PermissionAndroid.

The documentation is not ready yet, please see the comments of index file for more details at the moment.

Core ML support

Platform: iOS 15.0+, tvOS 15.0+

To use Core ML on iOS, you will need to have the Core ML model files.

The .mlmodelc model files is load depend on the ggml model file path. For example, if your ggml model path is ggml-base.en.bin, the Core ML model path will be ggml-base.en-encoder.mlmodelc. Please note that the ggml model is still needed as decoder or encoder fallback.

Currently there is no official way to get the Core ML models by URL, you will need to convert the ggml model to Core ML model folder by yourself. Please see Core ML Support of whisper.cpp for more details.

During the .mlmodelc is a directory, you will need to download 5 files:

[
  'model.mil',
  'metadata.json',
  'coremldata.bin',
  'weights/weights.bin',
  'analysis/coremldata.bin',
]

Or just add them to your app's bundle resourcesas, like the example app does, but this would increase the app size significantly.

Run with example

The example app is using react-native-fs to download the model file and audio file.

Model: base.en in https://huggingface.co/datasets/ggerganov/whisper.cpp
Sample file: jfk.wav in https://github.com/ggerganov/whisper.cpp/tree/master/samples

For test better performance on transcribe, you can run the app in Release mode.

• iOS: yarn example ios --configuration Release
• Android: yarn example android --mode release

Please follow the Development Workflow section of contributing guide to run the example app.

Mock whisper.rn

We have provided a mock version of whisper.rn for testing purpose you can use on Jest:

jest.mock('whisper.rn', () => require('whisper.rn/jest/mock'))

Contributing

See the contributing guide to learn how to contribute to the repository and the development workflow.

License

MIT

Made with create-react-native-library