Jdrive NPM | npm.io

JDrive

A simple CLI utility to sync Google Drive and the local filesystem. Only for Unix-like OS, (I know NodeJS is cross-platform, but I don't care about other OS). It's not tested on *BSD, but should work. I know there are some other (better) clients like Insync, I know.

How to install

Step 1. Download and install

It's available on npm and also in the AUR for ArchLinux users. It's also available for Linux(glibc) and Alpine(musl) as a single binary file (discouraged). Choose the one who fits best with your needs.

#Any distro
$ npm i -g jdrive

#ArchLinux
$ <aur helper> -S jdrive

Dependencies (autoinstalled on ArchLinux)

Runtime deps:
- nodejs>=10
- GNU Coreutils (Optional, if bindings are usable it's no needed)
Installation (makedepends):
- base-devel group (or the equivalent containing make, gcc...)
- npm (installation only)
- node-gyp
- python2 (for node-gyp)

Step 2. Get access to Drive API

You will need a credentials file to access the Drive API. The file can be got here, on the Google Cloud Console. This step is better explained on the Step 1 of REST API quickstart. This file should be downloaded and saved to a file named credentials.json

Step 3. Start JDrive

JDrive can be started in two ways. Make sure to use always the same one or multiple config dirs and lockfiles will appear.

The "classic" way: Go to a terminal and launch jdrive. Arguments can be added to the executable. See posibilities on Commandline parameters.
- After this, JDrive will crash as there is no a credentials file. Copy the credentials json file to the specified path. Keep it secure!
- Now run JDrive again. You will be asked to open a navigator on a URL to authorize JDrive access a Google account. Then paste back the generated code to JDrive.
- It works! Now it will start syncing files to your local filesystem. The steps done won't be repeated, JDrive saves the token in a file, so authorization is not needed until you remove that file.
With the systemd --user unit:
- Think a name, it can be your name or your username. This name will identify the systemd --user unit (as more than one units can be running at the same time).
- Go to a terminal and run systemctl --user start jdrive@yourname. The unit will start and stop (as in the above method).
- Save the obtained credentials json to ~/.config/jdrive-yourname/credentials.json you can see this path in the journal or with systemctl --user status jdrive@yourname.
- Run jdrive -b ~/.config/jdrive-yourname --auth-only true and follow the instructions.
- After you see a success message, you can start or enable the systemd --user unit by issuing systemctl --user start jdrive@yourname and query the status with systemctl --user status jdrive@yourname. For advanced queries you should look systemd.unit(5).

Configuration

JDrive files

Those are the default files. The systemd unit files have a little variation in the base path. (By default ~/.config/jdrive).

System wide:
- /etc/jdrive.json - System wide config file - Not required for JDrive to work
User/instance scope: Those names are relative to base dir.
- jdrive.json - User configuration (overwrites /etc/jdrive.json) - Not required for JDrive to work
- credentials.json - User credentials. How to get this file on Step 2
- lockfile.json - JDrive generated file, contains status variables for JDrive. When is SIGKILLed while syncing, you will be asked to check your files and change the status in this file to watching.
- token.json - Autogenerated on auth, if you want to link JDrive to another account (with the systemd unit is easier) you can delete this file (and the lockfile to reset status).

The systemd --user unit sets config path to ~/.config/jdrive-name and local sync directory to ~/GoogleDrive-name.

Configuration map

The config files are written in JSON, both global and per-user file can be modified. The user config has priority over system-wide defined settings.

A note about workers: There isn't a max number of workers, you can spawn hundreds of them, but the max secure number is 8, above this limit, JDrive will emit a warning and you can easily reach the maximum per-user limit of the credentials file. Also, take in account that each worker is a thread executed by Node.js.

TOP LEVEL
- files: All paths are relative to base config dir or absolute in the FS.
  - config_dir: Not adjustable. This set by parameters. Default: ~/.config/jdrive
  - userConfig: path. Default: jdrive.json
  - credentials: path. Default: credentials.json
  - token: path. Ensure the mode is 0o600 or 0o400. Default: token.json
  - lock: path. Default: lockfile.json
  - ignore: path. A .gitignore-like file. Default: .syncignore
  - localDir: Absolute path. Default: ~/Google Drive
- local:
  - ignored: Not adjustable. This is set internally from the .syncignore file.
  - modes: File and dir modes
    - file: string. Default file creation mode. Default: "600"
    - dir: string. Default string creation mode. Default: "700"
  - disableXattrs: boolean. Disable the use of xattrs or to allow it (autodetect if FS supports this feature) Default: false
- drive
  - workers: number. Workers spawned on the drive queue. Min: 1, Max: 8, Recommended: 4, Default: 2
  - allowSharedFiles: boolean. Allow shared files with write permission to be synced. Please, see the issues section around this option to see limitations. Default: false
  - deleteForever: boolean. Delete permanently the files unlinked from the local filesystem instead of sending them to Drive's trash. See the issues section around this to see limitations on this option. Default: false
  - allowDriveFiles: boolean. Allow Drive files being synced (e.g. Google Docs) this has no effect as JDrive cannot manage Google Docs. When false, this prevents JDrive from processing Google Apps events (e.g. when you are writing on a doc, a bunch of change events are generated).
- events
  - workers: number. Number of workers spawned to process events. Min: 1, Max: 8, Recommended: 4, Default: 2
  - maxRetries: number. Number of retries if a unkown event occurs. For example a 500 error from server or a ratelimit reached. Min: 0, Max: the time you want to spend, Default: 5
- socket
  - path: Path where the socket should be created. Default: /tmp/jdrive-$USER-randomNumber.sock or /tmp/jdrive-$USER-ID.sock if args.id is present

Commandline arguments

You can get the list of arguments anytime with jdrive -h. Parameters involving paths follow the same rules as the ones in config files.

-h or --help - Show the list of possible arguments and exit
-v or --version - Get the JDrive version
--auth-only - Auth and exit after it has been successful
--dry-run - Log a list of files to be synced and exit
--config Path to userConfig
-b or --config-dir - Path to configuration home
-c or --credentials - Path to credentials.json
-t or --token - Path to token.json
-l or --lockfile - Path to lockfile.json
-i or --ignore-file - Path to ignore file
-d or --local-dir - Path to the local sync dir (must be absolute)
--dry-run - Log files going to sync and exit before doing anything
--id - Set name for JDrive instance (e.g. when running with systemd --user template)

Issues

I you experience issues feel free to open a issue or if you think it's about compatibility, comment on #9.

There are some native bindings (not included on the binary due to a limitation on zeit/pkg). If this binaries are not found, the change time function will be delegated on touch command (from coreutils. Busybox one will throw a lot of errors).

Alpine (and busybox) users should install the GNU Coreutils, as they provide nanoseconds support in filesystem stats. If you are sure your busybox installation supports nanoseconds, you are on you own.

Related to eXtendedAttributes. JDrive uses xattrs to write the fileId to file. This way, JDrive can identify a file when it's moved and update the file's metadata instead of removing it and uploading it again. This behaviour can be disabled with an option documented above. When the xattrs are not allowed on the destination FS, JDrive won't try to use xattrs on this run. This will have side effects and bring up the following issue: Shared files can be modified and uploaded again, but cannot be renamed locally, they should be renamed in Google Drive. When a shared file is renamed, it's removed from Drive and uploaded again with the new name. The newly uploaded file is no shared.

Related to deleteForever. At JDrive start, last events are checked in order to remove local files removed remotely while JDrive was not running. Trashed files contains the parent relationship, so a path can be generated, but deleted ones only have the fileId, which means nothing, so this event is completely ignored. With this event ignored, we don't know whether the file was removed remotely or not uploaded. Be careful when tuning this option.

Huge memory allocation when uploading big files (as referred on #23). This issue is not a JDrive one. When a readStream is connected to read from a file, it's red using little buffers instead a huge one (as could be with fs.readFile). The problem here is that the memory allocated with those buffers is not freed Why? Because of V8's policy; it will allocate as much memory as it can (inside a relative limit). So, is there any solution like calling the garbage cleaner manually, etc? No, some tests with the garbage cleaner reveal that it doesn't do anything and JDrive gets slower. Anyway, there are some workarounds with a external intervention like running it inside of a container or using cgroups (available for users with systemd as init system. Users with other init systems should know how to limit the memory used by a daemon. Those methods are explained on #23. The systemd --user service unit has some commented examples about this if you're having this kind of problems.

Does it integrate with any Desktop Environment?

As it has no GUI (feel free to add one, but this is a CLI tool), it has no tray icon.

Tray icon and GUI for JDrive is a possible project as JDrive internally uses unified queues, so is easy to know the number and path of items to be processed and the ones running. But this project won't be available as part of JDrive but as an addon.

How can I control it?

You can manage it manually (sending signals to the PID) or with the included systemd --user unit. This process doesn't fork away, so you can call it directly from a terminal. Starting on version 2.1, jdrive exposes a UNIX socket on /tmp to allow making queries about it. Commands and responses docs are on socket.md. There's a client called jdrive-client that uses that socket.