@soundhound/houndify-react-native v0.2.1

Houndify React Native

This module allows you to make requests to the Houndify API from a React-Native application. To use it, you must set up this library on your React-Native app as well as houndify on your server. We'll explain both parts below.

To see a fully working React Native App using this library, visit here

This library:

• Supports React Native <= v0.67
• Supports streaming! This means your users will see their speech as they talk, without waiting for a whole audio file to send.
• Supports TTS (Text to Speech) This means your app will be able to speak to your user out loud.
• Works hand in hand with Houndify's main sdk, houndify.
• Requires houndify >= 3.0.0 on your server. If you don't know what we mean by this, keep reading the next paragraph :).

Server Setup

We recommend you set up your proxy server first. This isn't too bad, especially if you are familiar with express. This way, you can test your React-Native app as soon as you get it running.

Proxy server may sound scary, but to be honest, we think you can set this up in under 30 seconds. Ready?

index.js

const express = require("express");
const app = express();

const expressWs = require("express-ws")(app);
const { createReactNativeProxy } = require("houndify").HoundifyExpress;

app.use(
  "/houndifyReactNativeProxy",
  createReactNativeProxy(express, "CLIENT_ID", "CLIENT_KEY")
);

app.listen(3000);

That's your proxy server. Put that on a machine, add in your Client ID and Client Key from houndify.com, run it with node index.js and you're done.

By default, express will run this on http://localhost:3000. Wherever it runs, make a note of the host and port. For example, localhost:3000. You'll need this when setting up the client.

If you are adding this to an existing express server, the important parts are:

1. Importing express-ws and initializing it with app.
2. Importing createReactNativeProxy from houndify.
   1. If you're using other parts of houndify in your app, we trust you know how too include this module as well.
3. Use app.use to forward all requests to /houndify-proxy to this new proxy.

Explanation

This library uses 'websockets', these allow efficient two way communication between a server and a client. express-ws allows express to use these websockets.

All the heavy lifting here is done in createReactNativeProxy. You just need to pass in a clientId and clientKey, and tell express to send it requests.

We take express as an argument because the express-ws library modifies the express object. Sometimes, re-requiring express in another file doesn't have these modifications, so for safety we just use the same object.

TIP: Test an app using a server on your computer.

You may not want to deal with uploading your server to a remote host--when you're working quickly and testing, running on localhost is by far the simplest way to develop. Using a tool such as ngrok let's you run a local server which can be accessed through a normal URL.

This way you can test houndify-react-native in an app while running a server on your computer.

React Native Installation

To use this module in your react-native app, go to your app's root directory and use the following commands.

1. npm install houndify-react-native --save or yarn add houndify-react-native (Based on your system you may need to run as sudo)
2. react-native link react-native-live-audio-stream
3. react-native link react-native-tts

Houndify React Native depends on some native (ios and android specifc) modules for audio, using react-native link makes sure these are configured properly.

Remember to add Internet and Microphone permissions to your application. You may need to request permissions in your app, or manually approve them from settings.

Known Installation Issues

Sometimes the react-native build system has issues with native modules. Your app may fail to build, with errors such as:

1. XCode build error 65
2. node-gyp errors
3. Could not mkdir...Usually involving fsevents.

if these or other errors occur, the following commands usually help.

1. npm i -g node-gyp node-gyp is needed when working with some native modules.
2. sudo node-gyp rebuild -g --unsafe-perm Helps node-gyp.
3. sudo chmod -R 777 node_modules This makes sure any build tools have write access to the node_modules directory.
4. npm install

Depending on your setup, you may need to run these as sudo. If something goes wrong, use sudo rm -rf node_modules to remove node_modules, and try issuing the above commands again.

Usage

There are a couple ways to use this package. You can either use the HoundifyReactNative class directly, or use the HoundifyComponent wrapper we've built. The wrapper removes some boilerplate, so we recommend you use it unless you have a reason not to.

HoundifyComponent (Recommended)

This is the simplest way to setup HoundifyComponent. We'll explain each part below, but there are a couple of important parts:

1. Props (e.g. host)
2. Render Props (e.g. transcription, writtenResponse). Pay close attention to the syntax here.

<HoundifyComponent
    host="ws://localhost:3000"
>
    // These are "render props". They are described below.
    {({transcription, writtenResponse}) => (
        /**Your Components Here*/
    )}
</HoundifyComponent>

Props

Only the host prop is shown, but more are available.

Available Props

Why do Hooks exist?

Ideally, any part of your app that directly depends on the houndify recording (e.g. showing transcription, showing response) is inside HoundifyComponent. But, sometimes you may need to access a piece of information from outside of your app's render function.

For example, say you want to show an animated readout of the volume. Without going into the details of React Native Animation, this may require some setup in other parts of your component. For this case, you could use the onVolume callback to store volume in global state.

Or perhaps you want your back button to automatically end recording. You could use getRecorder to store the recorder and call recorder.abort() automatically when the user exits. Hooks provide flexibility.

Render Props

What are Render Props

HoundifyComponent uses the HoundifyReactNative API to internally keep track of state regarding a user's request. It needs to pass this state into child components, so that your app can access it.

This is how child components are usually written:

<Parent>
    <Child/>
</Parent>

This is how they are written with render props:

<Parent>
    {(arg1, arg2...) =>
        <Child/>
    }
</Parent>

Here, Parent's direct child is not <Child/>, but instead a function that returns <Child/>. Parent, in our case HoundifyComponent, calls this function and uses it to pass in important state information.

Render Props in HoundifyComponent

transcription and writtenResponse are examples of properties that HoundifyComponent keeps track of and passes into your app. When a user makes a request, these properties are updated periodically and provided to any components your write. (Review the syntax above and note the use of object destructuring)

Available Properties

How do I show Cards with information based on the response?

If you've used any virtual assistant, you've seen cards. For example, context specific UI for weather, stocks, information, and so on. The response render prop contains all the information you need to show these custom UI elements. See the Houndify Web SDK or HoundServer object specification for more information on how to understand this response.

Also, feel free to start console.log-ing and explore for yourself! *Hint: Look for the AllResults, InformationNuggets, and NuggetKind properties in the response. Generally, InformationNugget -> UI Element.*

A simple component implementing HoundifyComponent

The following is a simple component like what you may need to build for your app.

It uses two text fields to show the results of a query, and a button to toggle recording component.

import React, {Component} from 'react';

class MyReactNativeComponent extends Component{
    ...
    render(){
        return(
          <HoundifyComponent host='ws://example.com'>
              {
                  ({transcription, writtenResponse, isRecording, start, abort}) => (
                     <>
                     <Text>{transcription}</Text>
                     <Text>{writtenResponse}</Text>
                     <Button onPress={isRecording ? abort : start} title='Click Me!'/>
                     </>
                  )
               }
          </HoundifyComponent>
        )
    }
}

HoundifyReactNative

Of course, you can also use this class directly instead of the React component. Most of the features are very similar to above, it just involves a bit more boilerplate.

class MyReactNativeComponent extends Component{
    constructor(props){
        ...
        this.recorder = new ReactNativeComponent({
            ...props
        })
    }

    ... somewhere else in the component
    this.recorder.start() // Starts recording.
    this.recorder.abort() // Prematurely ends recording.
}

Here, props is exactly like the props for HoundifyComponent. (Except for tts). You can see that even this way is not too complicated, but you must use the hooks provided through props to keep track of response, transcription, etc yourself. Using this class directly does not provide the automatic state management of HoundifyComponent. For most applications, you are best of using HoundifyComponent and using the provider hooks to give other parts of your app access where needed.