frameflow v0.2.0

FrameFlow

A both speedy and compatible video processing library for Web Browser, based on WebCodecs and FFmpeg (WebAssembly). It is hardware accelerated by WebCodecs as default, which works in Chromium-based clients (Chrome, Edge, Electron...). And also provides fallback solutions by FFmpeg (WebAssembly). It also provides some usual filters (trim, concat...).

Features

• Process videos in stream way, without video size limitation.
• Accept stream input MediaStream (from canvas, Camera, ...), and output stream of frames (to canvas...) as well.
• Use WebCodecs to have hardware acceleration for Chromium-based client (Chrome (>=106), Edge, Opera, Electron...).
• Get detailed metadata of video file by reading only several chunks, either from local disk or remote url.
• Processing speed can be controlled either automatically or manually.

Demo

import fflow from 'frameflow'

let video = await fflow.source(videoBlob) // use web File api to get File handler.
let audio = await fflow.source(audioURL) // remote media file (no need to download entirely beforehand)
let audioTrim = audio.trim({start: 10, duration: video.duration}) // use metadata of video
let blob = await fflow.group([video, audioTrim]).exportTo(Blob, {format: 'mp4'}) // group and trancode to 
videoDom.src = URL.createObjectURL(blob)
// now can play in the browser

Although this example writes to blob entirely, then play. But underhood, it streams out chunks and then put togather.

Update from v0.1 to v0.2

• v0.1 API is not changed. But underhood, previous version use an unique worker for each time. But this will cause too long to load (~1 second), and also cost too much memory if processing many videos in the same time.
• use fflow.load() to load the default worker with with loaded ffmpeg module, shared by all use cases.
• use fflow.load({newWorker: true}) to create an new worker with loaded ffmpeg module, and assign to export as a parameter. This multi-thread case will accelerate if having many export tasks. And sometimes, as a more complex case, if two export have dependencies, there may be a dead lock if they share the same worker (thread).

let worker = fflow.load({newWorker: true})
let blob = await video.exportTo(Blob, {format: 'mp4', worker }) // group and trancode to 

More examples

More detailed browser examples are in the ./examples/browser/. If you want to run them, please use latest release version. And then, at the root directory of the project,

npm install
npm start

In dev mode, it will serve ./examples as root directory.

Install

NPM

npm i frameflow

HTML script

<script src='https://unpkg.com/frameflow/dist/frameflow.min.js' ></script>

Document

All tutorials and documents are in FrameFlow Doc.

Problems

How to build

Warning: webpack dev mode cannot hot reload in WSL2 (windows).

Dependencies (Ubuntu)

Tools dependencies install

sudo apt-get update -y
sudo apt-get install -y pkg-config

Emscripten

git clone https://github.com/emscripten-core/emsdk.git --branch 3.1.52
rm -r emsdk/.git

Install Emscripten SDK

FFmpeg version (n5.0 release)

git clone  https://github.com/FFmpeg/FFmpeg --depth 1 --branch n5.0
rm -r FFmpeg/.git

External FFmpeg Libraries

All external libraries sources are under ./ffmpeg_libraries

cd ffmpeg_libraries

x264

git clone https://github.com/mirror/x264.git --depth 1 --branch stable 

Libvpx

git clone https://github.com/webmproject/libvpx.git --depth 1 --branch v1.12.0

Compilation

./build_ffmpeg.sh
./build_wasm.sh