@sosnoob/crawlit

Crawlit - Dofus & Dofus-Touch encyclopedia parser - Build your own Dofus API

In order to respond to a need expressed by the community, I create an encyclopedia parser for Dofus available for :

• Windows
• OSX
• Linux

Download : All versions are available here

Don't use the tool for more than 700 items per hour (for Dofus-Touch).

Crawlit was built to be very light, it can run on almost every configurations.

In fact Crawlit is a tool :

• Developed by the dofus community, for the community
• 100% performance : Developed to be very light
• Simple to use | Plug and use
• Resume parsing feature after errors (connection off / 429 error / ...)
• App packaged into executable, no need to install something
• English & French encyclopedia supported (french item & english item)

App launched	App at the end

JSON files also provided in the repository

In case you wan't directly the dofus JSON files, I oftenly upload up-to-date encyclopedia JSON in the data/ folder.

JSON format

An item result example

{  
  "_id":"item ID",
  "name":"item name",
  "description":"item description.",
  "lvl":"item lvl",
  "type":"item type",
  "imgUrl":"image url of the item",
  "url":"Item's link",
  "stats":[ "many statistics line", "stat 2", ["..."], "stat n" ],
  "condition":[ "many conditions line", "condition 2", ["..."], "condition n" ],
  "set":{  
     "equipments":[itemId, itemId ...],
     "weapons":[itemId, itemId ...],
  }
}

Getting Started

[With Npm] Install globally crawlit :

npm install -g @sosnoob/crawlit
crawlit

[With github] Clone & access this repository locally :

git clone https://github.com/raczak/crawlit-dofus-encyclopedia-parser.git
cd crawlit-dofus-encyclopedia-parser

Install dependencies with npm :

npm install

There is an issue with npm start on some CLI, in this case prefere use node lib/app.js.

[Optional] To build packaged app

If you want to generate packaged app (executable) with pkg dependency, you MUST install zeit/pkg in npm global context.
Please follow Pkg documentation for troubleshooting.

npm install -g pkg

• in a terminal window -> npm run pkg-all

Supported items

Items	state of progress
Equipment	100 %
Weapon	100 %
Set	100 %
Pet	100 %
Mount	100 %
Resource	100 %
Consumable	100 %
Recipe	25 %
Classe	0 %
Professions	0 %
Bestiary	0 %

We also have done an API !

Visite the official non-official Dofus API : DOFAPI

Technologies choice

Crawlit is build with full javascript : NodeJs/Npm, javascript.

Bug & Crash || Proposal ?

For proposal don't hesitate to create a pool request or for a bug/crash an issue. You can also make us aware on bugs channel in discord.

Discord

Click here to join the sosnoob community on discord !

Documentation(dev)

Access the wiki to understand the project and contribute to it. There is also a french version of the documentation !

Join the dev Team ?

Contact us on discord (channel general or private message to theukid)

Contributors

Zakaria RACHEDI	Le Corre Julien	Yannick Milanetto