Mwoffliner NPM

mwoffliner

mwoffliner is a tool for making a local HTML snapshot of any online (recent) Mediawiki instance. It goes through all articles (or a selection if specified) and writes the HTML/images to a local directory. It has mainly been tested against Wikimedia projects like Wikipedia, Wiktionary, ... But it should also work for any recent Mediawiki.

npm NPM

Prerequisites

*NIX Operating System (Linux/macOS)
NodeJS
Redis
Libzim (On linux we automatically download binaries)
Various build tools that are probably already installed on your machine (libjpeg, gcc)

Setup

MacOS

NodeJS

curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.11/install.sh | bash && \
source ~/.bashrc && \
nvm install stable && \
node --version

Redis

> brew install redis

LibZim

See instructions here: https://github.com/openzim/libzim

Linux (Debian)

NodeJS

curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.11/install.sh | bash && \
source ~/.bashrc && \
nvm install stable && \
node --version

Redis

> sudo apt-get install redis-server

Usage

Command Line

> npm i -g mwoffliner
> mwoffliner --help

> mwoffliner \
    --mwUrl=https://es.wikipedia.org \
    --adminEmail=foo@bar.net \
    --verbose \
    --format=nozim \ # Won't make a final ZIM file
    --articleList=./dev/articleList # Will download one article

Programmatic API

const mwoffliner = require('mwoffliner');
const parameters = {
    mwUrl: "https://es.wikipedia.org",
    adminEmail: "foo@bar.net",
    verbose: true,
    format: "nozim",
    articleList: "./articleList"
};
mwoffliner.execute(parameters); // returns a Promise

Development

Please see CONTRIBUTING.md

To setup mwoffliner locally:

git clone https://github.com/openzim/mwoffliner.git
cd mwoffliner
npm i

To run it (this is only an example):
```bash
./node_modules/.bin/ts-node ./src/cli.ts --mwUrl=https://bm.wikipedia.org --adminEmail=XXX

npm start -- --mwUrl=https://bm.wikipedia.org --adminEmail=XXX

Tests

There are two test suites:

Unit

npm run test:unit # (or just npm test)

End to end (e2e)

npm run test:e2e

Code Style

We follow a nearly exact tslint:recommended scheme - you can see more information here: ./tslint.json

It's best to use TSLint to check your code as you develop, this project is pre-configured for development with VSCode and the TSLint plugin.

Debugging

There is a pre-configured debug config for VSCode, just click on the debugging tab.

Make sure you read CONTRIBUTING.md for tips on how to best debug and submit issues.

Publishing

To publish, it's best to use a clean clone of the project:

git clone https://github.com/openzim/mwoffliner.git
npm i
./dev/build.sh
npm publish  # you must be logged in already (npm login)

Background

There are two Wikitext parsers. mwoffliner uses Parsoid.

Wikitext is the name of the markup language that Wikipedia uses.
MediaWiki is a PHP package that runs a wiki, including Wikipedia.
MediaWiki includes a parser for Wikitext into HTML, and this parser creates Wikipedia currently.
There is another Wikitext parser, called Parsoid, implemented in Javascript (Node.js).
Parsoid is planned to eventually become the main parser for Wikipedia.
mwoffliner uses Parsoid.
mwoffliner calls Parsoid and then post-processes the results for offline format.

mediawiki zim

@everything-registry/sub-chunk-2222 @infinitebrahmanuniverse/nolb-mw @zalastax/nolb-mw

2 years ago

2 years ago

2 years ago

3 years ago

3 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

5 years ago

5 years ago

1.10.10-1598695266118

5 years ago

1.10.10-1598608835958

5 years ago

1.10.10-1598435997491

5 years ago

1.10.10-1598263173515

5 years ago

1.10.10-1598091104059

5 years ago

1.10.10-1598090820856

5 years ago

1.10.10-1598090386706

5 years ago

1.10.10-1598003806979

5 years ago

1.10.10-1597831121939

5 years ago

1.10.10-1597658152239

5 years ago

1.10.10-1597485350812

5 years ago

1.10.10-1597312623773

5 years ago

1.10.10-1597139734799

5 years ago

1.10.10-1596880500115

5 years ago

1.10.10-1596966925436

5 years ago

1.10.10-1596794029579

5 years ago

1.10.10-1596621337888

5 years ago

1.10.10-1596534773092

5 years ago

1.10.10-1596361892794

5 years ago

1.10.10-1596448343779

5 years ago

1.10.10-1596188995801

5 years ago

1.10.10-1596090542209

5 years ago

1.10.10-1595929916708

5 years ago

1.10.10-1595836058720

5 years ago

1.10.10-1595835180553

5 years ago

1.10.10-1595771702921

5 years ago

1.10.10-1595756783859

5 years ago

1.10.10-1595670464852

5 years ago

1.10.10-1595497497413

5 years ago

1.10.10-1595324616389

5 years ago

1.10.10-1595228185394

5 years ago

1.10.10-1595236603513

5 years ago

1.10.10-1595236661895

5 years ago

1.10.10-1595151803973

5 years ago

1.10.10-1594978900608

5 years ago

1.10.10-1594879935273

5 years ago

1.10.10-1594736218519

5 years ago

1.10.10-1594719017165

5 years ago

1.10.10-1594735094045

5 years ago

1.10.10-1594546738657

5 years ago

1.10.10-1594633180252

5 years ago

1.10.10-1594460272662

5 years ago

1.10.10-1594288962896

5 years ago

1.10.10-1594287382290

5 years ago

1.10.10-1594195778523

5 years ago

1.10.10-1594114664221

5 years ago

1.10.10-1594028074238

5 years ago

1.10.10-1593941869829

5 years ago

1.10.10-1593855243028

5 years ago

1.10.10-1593768776409

5 years ago

1.10.10-1593679214842

5 years ago

1.10.10-1593590704435

5 years ago

1.10.10-1593437096757

5 years ago

1.10.10-1593336564117

5 years ago

1.10.10-1593179176186

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

1.9.5-rc1-1564672653355

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

7 years ago

7 years ago

7 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago