Dmr-source NPM

What's dmr-source

Dmr-Source provides unified methods to achieve stream of data sources, such as local / remote (http / ftp) ...

Source
FileSource
HttpSource
FtpSource
MultiSource
SftpSource
HadoopSource

Usage

Get file from ftp server to local

const fs = new FtpSource({
  host: "127.0.0.1",
  path: "/source-ftp-test.dict",
  port: 21,
});
const writer = new FileSource().createWritableStream({path: "/home/work/a.log"});
fs.createReadableStream().pipe(writer);

Copy big file by FileSource

const fs = new FileSource({encoding: "utf8"});
const reader = fs.createReadableStream({path: "/home/work/a.log"});
const writer =fs.createWritableStream((option) => {
  option.path = "/home/work/b.log";
  return option;
});
reader.pipe(writer);

Link readables to one readable stream

const stream1 = new HttpSource({host: "localhost", path: "/a.log"}).createReadableStream();
const stream2 = new FileSource({host: "localhost", path: "/b.log"}).createReadableStream();
const ms = new MultiSource().add(stream1).add(stream2).add(() => {
  return new FileSource({host: "localhost", path: "/c.log"}).createReadableStream();
});
ms.createReadableStream().on("data", (chunk) => console.log(chunk));

Events

const stream = new HttpSource({host: "localhost", path: "/404.page", timeout: 6000})
  .createReadableStream();
stream.on("error", (err) => {
  console.log("should be 404 error", err);
});
stream.on("end", (err) => {
  console.log("never arrive end");
});