Mwoffliner NPM

MWoffliner

MWoffliner is a tool for making a local offline HTML snapshot of any online MediaWiki instance. It goes through all online articles (or a selection if specified) and create the corresponding ZIM file. It has mainly been tested against Wikimedia projects like Wikipedia and Wiktionary --- but it should also work for any recent MediaWiki.

Read CONTRIBUTING.md to know more about MWoffliner development.

Features

Scrape with or without image thumbnail
Scrape with or without audio/video multimedia content
S3 cache (optional)
Image size optimiser / Webp converter
Scrape all articles in namespaces or title list based
Specify additional/non-main namespaces to scrape

Run mwoffliner --help to get all the possible options.

Prerequisites

*NIX Operating System (GNU/Linux, macOS, ...)
Redis
NodeJS version 16 or greater
Libzim (On GNU/Linux & macOS we automatically download it)
Various build tools which are probably already installed on your machine (packages libjpeg-dev, libglu1, autoconf, automake, gcc on Debian/Ubuntu)

... and an online MediaWiki with its API available.

Usage

To install MWoffliner globally:

npm i -g mwoffliner

You might need to run this command with the sudo command, depending how your npm is configured.

npm permission checking can be a bit annoying for a newcomer. Please read the documentation carefully if you hit problems: https://docs.npmjs.com/cli/v7/using-npm/scripts#user

Then to run it:

mwoffliner --help

To install and run it locally:

npm i
npm run mwoffliner -- --help

To use MWoffliner with a S3 cache, you should provide a S3 URL like this:

--optimisationCacheUrl="https://wasabisys.com/?bucketName=my-bucket&keyId=my-key-id&secretAccessKey=my-sac"

API

MWoffliner provides also an API and therefore can be used as a NodeJS library. Here a stub example:

const mwoffliner = require('mwoffliner');
const parameters = {
    mwUrl: "https://es.wikipedia.org",
    adminEmail: "foo@bar.net",
    verbose: true,
    format: "nopic",
    articleList: "./articleList"
};
mwoffliner.execute(parameters); // returns a Promise

Background

Complementary information about MWoffliner:

MediaWiki software is used by thousands of wikis, the most famous ones being the Wikimedia ones, including Wikipedia.
MediaWiki is a PHP wiki runtime engine.
Wikitext is the name of the markup language that MediaWiki uses.
MediaWiki includes a parser for WikiText into HTML, and this parser creates the HTML pages displayed in your browser.

GNU/Linux - Debian based distributions

Install NodeJS: Read https://nodejs.org/en/download/current/

Install Redis:

sudo apt-get install redis-server

Troubleshooting

Older GNU/Linux distributions and/or versions of Node.js might be shipped with a deprecated version of npm. Older versions of npm have incompatbilities with certain versions of Node.js and might simply fail to install mwoffliner package.

We recommend to use a recent version of npm. Recent versions can perfectly deal with older Node.js 10. Do install the packaged version of npm and then use it to install a newer version like:

sudo npm install --unsafe-perm -g npm

Don't forget to remove the packaged version of npm afterward.

License

GPLv3 or later, see LICENSE for more details.

mediawiki zim

@everything-registry/sub-chunk-2222 @infinitebrahmanuniverse/nolb-mw @zalastax/nolb-mw

1 year ago

2 years ago

2 years ago

2 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

1.10.10-1598695266118

4 years ago

1.10.10-1598608835958

4 years ago

1.10.10-1598435997491

4 years ago

1.10.10-1598263173515

4 years ago

1.10.10-1598091104059

4 years ago

1.10.10-1598090820856

4 years ago

1.10.10-1598090386706

4 years ago

1.10.10-1598003806979

4 years ago

1.10.10-1597831121939

4 years ago

1.10.10-1597658152239

4 years ago

1.10.10-1597485350812

4 years ago

1.10.10-1597312623773

4 years ago

1.10.10-1597139734799

4 years ago

1.10.10-1596880500115

4 years ago

1.10.10-1596966925436

4 years ago

1.10.10-1596794029579

4 years ago

1.10.10-1596621337888

4 years ago

1.10.10-1596534773092

4 years ago

1.10.10-1596361892794

4 years ago

1.10.10-1596448343779

4 years ago

1.10.10-1596188995801

4 years ago

1.10.10-1596090542209

4 years ago

1.10.10-1595929916708

4 years ago

1.10.10-1595836058720

4 years ago

1.10.10-1595835180553

4 years ago

1.10.10-1595771702921

4 years ago

1.10.10-1595756783859

4 years ago

1.10.10-1595670464852

4 years ago

1.10.10-1595497497413

4 years ago

1.10.10-1595324616389

4 years ago

1.10.10-1595228185394

4 years ago

1.10.10-1595236603513

4 years ago

1.10.10-1595236661895

4 years ago

1.10.10-1595151803973

4 years ago

1.10.10-1594978900608

4 years ago

1.10.10-1594879935273

4 years ago

1.10.10-1594736218519

4 years ago

1.10.10-1594719017165

4 years ago

1.10.10-1594735094045

4 years ago

1.10.10-1594546738657

4 years ago

1.10.10-1594633180252

4 years ago

1.10.10-1594460272662

4 years ago

1.10.10-1594288962896

4 years ago

1.10.10-1594287382290

4 years ago

1.10.10-1594195778523

4 years ago

1.10.10-1594114664221

4 years ago

1.10.10-1594028074238

4 years ago

1.10.10-1593941869829

4 years ago

1.10.10-1593855243028

4 years ago

1.10.10-1593768776409

4 years ago

1.10.10-1593679214842

4 years ago

1.10.10-1593590704435

4 years ago

1.10.10-1593437096757

4 years ago

1.10.10-1593336564117

4 years ago

1.10.10-1593179176186

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

1.9.5-rc1-1564672653355

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

6 years ago

7 years ago

7 years ago

7 years ago

7 years ago

7 years ago

7 years ago

7 years ago

7 years ago

7 years ago

7 years ago

7 years ago

7 years ago

7 years ago

7 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago

8 years ago