Leaderboard ad

1.05.2016

Sync local folder to Google Drive in Linux

I finally have a half decent way have a two way sync with a local folder on my Linux machine and Google Drive.  I figured I would share this post with anyone who might be looking for a similar function.

There is a tool available call gsync, which provides rsync like functionality with Google Drive.  Here is a link to the project page:

https://code.google.com/p/gsync/

You can easily install gsync on Fedora (yum or dnf):

sudo yum install python-setuptools
sudo easy_install pip
sudo pip install gsync

or install on Ubuntu:

sudo apt-get python-setuptools
sudo easy_install pip
sudo pip install gsync

Once installed, you have to authorize gsync to access your Drive, run the following command and follow the directions:

gsync --authenticate
Once authenticated you can sync a local folder with Google Drive like so:

gsync -r -t -p -o -g -v --progress --delete -l -s /local/folder-a drive://folder-a
The options are similar to rsync options, here is a list of the options I used above:

-r = recursive
-t = Preserve modification times
-p = preserve permissions
-o = preserve owner
-g = preserve group
-v = verbose
-l = copy symlinks
-s = no space splitting
--progress = Show progress during transfer
--delete = delete extraneous file from destination directories

You can get a full list of options by typing:

gsync --help
To create a two way sync between a local folder and Google Drive you simply run the command twice, reversing the directory order like this:

SYNC LOCAL TO GOOGLE DRIVE:
gsync -r -t -p -o -g -v --progress --delete -l -s /local/folder-a drive://folder-a
SYNC GOOGLE DRIVE BACK TO LOCAL:
gsync -r -t -p -o -g -v --progress --delete -l -s drive://folder-a /local/folder-a
I set this up in a simple bash script and have it run once a day via a cron job. Right now I have all the output redirected to a log file and sent to me via email so I can keep an eye on it and make sure it is behaving as expected.  It has been solid for the last week or so.  The only issues I have run into are file permissions after being synced from Google Drive to local (which are addressed in the script below), and a file will occasionally sync two days in a row (even though it never changed).

#!/bin/bash
LOG=/tmp/gsync.log

echo "Sync local with Google Drive:" >> $LOG
gsync -r -t -p -o -g -v --progress --delete -l -s /vault/Documents/ drive://Documents >> $LOG

echo "Sync Google Drive with Local:" >> $LOG
gsync -r -t -p -o -g -v --progress --delete -l -s drive://Documents /vault/ >> $LOG

echo "Setting permissions on new files:" >> $LOG
chown -R user:user /vault/ >> $LOG
echo "All files chowned..." >> $LOG
chmod -R 750 /vault/ >> $LOG
echo "All files chmodded..." >> $LOG

tr -cd '\11\12\15\40-\176' < /tmp/gsync.log | mail -s "GSYNC" me@domain.com
rm -f $LOG

I hope this helps somebody, and we can only hope that Google will show some love to it's Linux users.


NOTE: If you are having issues with files uploading, please read the below sent in by a user named "tidbits". 

This is also needed in order for files to be uploaded. If this is not added, only directory/folder structure is created and no files uploaded.

First verify your python version - $python -V

Then goto corresponding /usr/local/ like below.

Editing /usr/local/lib/python2.7/dist-packages/libgsync/drive/__init__.py, and changing:

body = {}
for k, v in properties.iteritems():
body[k] = _Drive.utf8(v)
...to:

body = {}
for k, v in properties.iteritems():
if v is not None:
body[k] = _Drive.utf8(v)

This will resolve any file upload issues. But... 

It does, however, produce this when it runs into a file that ends in a ~ character:

0 0% 0.00B/s 0:00:00DEBUG: 'Exception': File "/usr/local/lib/python2.7/dist-packages/libgsync/drive/__init__.py", line 712, in update
status, res = req.next_chunk()
File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 132, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/apiclient/http.py", line 874, in next_chunk
return self._process_response(resp, content)
File "/usr/local/lib/python2.7/dist-packages/apiclient/http.py", line 901, in _process_response
raise HttpError(resp, content, uri=self.uri) 

3 comments:

  1. For those who run into limitations with API calls the gsync utility is set to use a pre-defined Google API key / credentials when installed. Patches were being developed but not sure if any update has been made. There is a fix for this but requires some work. Credit to jeztucker... I've done the below and works very well.

    "As johanels indicated, the gsync utility is set to use pre-defined Google API key / credentials when shipped.

    This means that by default all users will utilize the data limit (typically 10m API requests per day).

    Therefore, you need to create your own Google API Project with associated key / credentials.

    Method to resolve this:

    Goto: https://console.developers.google.com/project/
    Create a new project and go to that Projects Dashboard, then:
    Enable an API -> Set Drive API to On
    APIs & Auth -> Consent Screen. Configure the consent screen, set a project name 'gsync-someotherthing' (or suchlike) and any relevant information
    APIs & Auth -> Credentials -> Create new client ID. Select 'installed application'. Type: Other
    As root, edit /usr/local/lib/python2.7/dist-packages/libgsync/drive/client_json.py and modify the client_id and client_secret to reflect your newly generated client credentials
    Also apply fix #66
    As your required (suggest non-root..) user, run gsync --authenticate and follow the instructions as per the standard docs
    Uploading of files will work, against your own personal API requests quota
    All the best,

    Jez"

    Thanks Savona for posting this! I am sure in the future I will need it for a rebuild!

    ReplyDelete
  2. This is also needed in order for files to be uploaded. If this is not added, only directory/folder structure is created and no files uploaded.

    First verify your python version - $python -V

    Then goto corresponding /usr/local/ like below.

    Editing /usr/local/lib/python2.7/dist-packages/libgsync/drive/__init__.py, and changing:

    body = {}
    for k, v in properties.iteritems():
    body[k] = _Drive.utf8(v)
    ...to:

    body = {}
    for k, v in properties.iteritems():
    if v is not None:
    body[k] = _Drive.utf8(v)

    This will resolve any file upload issues. But...

    It does, however, produce this when it runs into a file that ends in a ~ character:

    0 0% 0.00B/s 0:00:00DEBUG: 'Exception': File "/usr/local/lib/python2.7/dist-packages/libgsync/drive/__init__.py", line 712, in update
    status, res = req.next_chunk()
    File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 132, in positional_wrapper
    return wrapped(*args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/apiclient/http.py", line 874, in next_chunk
    return self._process_response(resp, content)
    File "/usr/local/lib/python2.7/dist-packages/apiclient/http.py", line 901, in _process_response
    raise HttpError(resp, content, uri=self.uri)

    ReplyDelete
  3. apt-get python-setuptools <=- "install" missing

    ReplyDelete