0.0.1 • Published 5 years ago

tumblr-data-save v0.0.1

Weekly downloads
4
License
WTFPL
Repository
github
Last release
5 years ago

Download the project

install Node if you don't have it, it is sometimes enough to update and open xCode for osx
https://nodejs.org/en/download/

On OSX you will probably be asked to install/update xCode at some point in this process

Open Terminal then

Navigate to your user home directorym create a new folder for your data and the downloader, move into that directory
cd ~
mkdir tumblr-backup
cd tumblr-backup
Make a directory for the MongoDB database files
mkdir data
Initialize the npm project as a blank project, just hit enter through the prompts npm init

Download the project that saves your data

npm install tumblr-save-express

Install mongodb

Follow MongoDB instructions to install MongoDB Community Edition

You may need to setup your Path variable.
google (setup path for OS)

Mondo DB stores all of the post information like text and other data which you can browse (perhaps non usefully)
MongoDB Compass

All of the images are stored on disk

Start the DB and Run the Web Server

Run the database
mongod --dbpath ./data/tumblr
Database files are then in the directory /data/tumblr

Open an additional terminal window and go back to the same directory
cd ~/tumblr-backup
Run the web server
node node_modules/tumblr-data-save

Open the url in a web browser
localhost:3000/posts/:blog/:from?/:count?
i.e.
localhost:3000/posts/one-million-bees/1000/2000
Will save your posts 1000 through 3000 from most recent (1000) to the 3000 post back
localhost:3000/posts/one-million-bees/
Default is 0 to 10000, if this isn't enough then this gets you 10000 to 20000... etc.
localhost:3000/posts/one-million-bees/10000

The images will be stored in /store/blog-name/month-year/***
The data that has all of your post data will be stored in /data/tumblr in mongoDB format

--

Troubleshooting

The included keys may or may not have hit max that are included with the project.
The keys go here
api-keys.ts
File should look like this
export var key = 'the-key';
export var secret = 'the-secret';

It is possible you can ignore this step if you aren't having any issue after starting the download.
You may need to use your own tumblr app keys if my keys run out of use bandwidth, here is how to get them.

Register a Tumblr App
Use Default callback URL
http://127.0.0.1:3000/login/callback

Development

Download project and install mongodb as listed above

THIS SECTION IS STILL WIP

Install the global libraries
npm install -g typescript typings typings install

Some Major Improvement that could use a fork & PR

Always use a original post source for image storing for the main images in original posts. Scripts to cleanup the images on disk.
Refactor JSON to have a, only additional information and then also store original JSON and only additional information in the child posts.
Database was setup by a db novice (me), could use some proper thought and migration scripts.
Fix the callback task cycle code for readability and stability

I'm covered in bees

Random things

If you get an error that says something is already running on port 3000 you can fix it by running these commands. Change [pid] to the process id found in the first command. (OSX, google if otherwise) lsof -i tcp:3000
kill -9 [pid]