Rclone Pictures to Google Drive: Difference between revisions
From Mike Beane's Blog
(Created page with " =Move Files ===Perl=== * Purpose: ** Read source dir ** Rename files and move to a Year\Month\Day structure <pre> use strict; use warnings; use English; use File::Basename...") |
|||
(5 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
=Move Files | =Move Files= | ||
===Perl=== | ===Perl=== | ||
* Purpose: | * Purpose: | ||
** Read source dir | ** Read source dir | ||
** Rename files and move to a Year\Month\Day structure | ** Rename files and move to a Year\Month\Day structure | ||
====Code==== | |||
<pre> | <pre> | ||
use strict; | use strict; | ||
Line 13: | Line 14: | ||
use File::Spec; | use File::Spec; | ||
use File::Copy; | use File::Copy; | ||
use Date::Format; | |||
my $dir = '/home/user/dropbox/incoming'; | |||
my $destination ='/home/user/dropbox/sorted/'; | |||
foreach my $fp (glob("$dir/*.jpg")) { | foreach my $fp (glob("$dir/*.jpg")) { | ||
print "_______________________________________\n"; | print "_______________________________________\n"; | ||
Line 29: | Line 31: | ||
my $filename_extension=""; | my $filename_extension=""; | ||
my $output_filename=""; | my $output_filename=""; | ||
### Filename Conversion ############################# | |||
# The original script used the file name for the source of time, however | |||
$filename_year=$ | # the camera being used apparently likes to change to GMT without | ||
$filename_month=$ | # reason... | ||
$filename_day=$ | # | ||
$ | # Pull the date from the file creation (or last mod) and use that instead | ||
$ | ############################################## | ||
$filename_extension= | # Fought getting the time & epoch conversion with Perl to decide just use | ||
$output_filename=$filename_year.$filename_month.$filename_day.'_'.$filename_time.'_'.$filename_camera.$filename_extension; | # Google | ||
################################################ | |||
my @array = stat($fp); | |||
print "$array[9]\n"; | |||
my $inputEpoch = $array[9]; | |||
# Handling EDT... really bad later for EST but I'm tired | |||
$inputEpoch = $inputEpoch-14400; | |||
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = gmtime($inputEpoch); | |||
$mon++; | |||
$year = $year + 1900; | |||
my $humanTime = sprintf ("%04d-%02d-%02d %02d:%02d:%02d", $year, $mon, $mday, $hour, $min, $sec); | |||
print "$humanTime\n"; | |||
################################################# | |||
# Output Filename creation ### | |||
################################################# | |||
$filename_camera="Prefix"; | |||
$filename_year=sprintf ("%04d", $year); | |||
$filename_month=sprintf ("%02d", $mon); | |||
$filename_day=sprintf ("%02d", $mday); | |||
$filename_time=sprintf ("%02d:%02d:%02d", $hour, $min, $sec); | |||
$filename_extension=".jpg"; | |||
#$output_filename=$filename_year.$filename_month.$filename_day.'_'.$filename_time.'_'.$filename_camera.$filename_extension; | |||
$output_filename=$filename_camera.'-'.$filename_year.$filename_month.$filename_day.'_'.$filename_time.$filename_extension; | |||
print "Output File: $output_filename\n"; | print "Output File: $output_filename\n"; | ||
################################### | |||
# Check the destination folder(s) and create as needed | |||
################################### | |||
### Check for destination directory - YEAR | ### Check for destination directory - YEAR | ||
my $dest_year = $destination.$filename_year; | my $dest_year = $destination.$filename_year; | ||
Line 78: | Line 107: | ||
#### MOVE FILES ##### | #### MOVE FILES ##### | ||
# copy ($fp,$dest_file) or die "The move operation failed: $!"; | |||
move ($fp,$dest_file) or die "The move operation failed: $!"; | |||
# | |||
} | } | ||
Line 84: | Line 115: | ||
=rclone= | =rclone= | ||
* Install rclone | |||
** Reference for setup: [https://www.howtogeek.com/451262/how-to-use-rclone-to-back-up-to-google-drive-on-linux/ here] | |||
* Execute rclone config from the account that will be using it. | |||
** N(ew) | |||
** Name: <NameofRcloneProject> | |||
** Find the Google Drive entry (13): | |||
** Client_ID: enter (default) | |||
** Client_Secret: enter (default) | |||
** Scope: 1 (we need to look into 3) | |||
*** Assumption: create a folder and get the folder ID for next line | |||
** Root folder ID: Enter | |||
** Service Account: Enter | |||
** Edit Advanced config: n | |||
** Use autoconfig? N (headless) | |||
*** Copy line to browser that is logged into the Google account to be used | |||
*** Grant rclone access | |||
*** copy the code | |||
** Enter Verifcation Code: paste the code | |||
** Configure this as a team drive? N | |||
** Select: Yes this is OK | |||
===Example rclone lines=== | |||
====copy==== | |||
<code> | |||
rclone copy --update --verbose --transfers 30 --checkers 8 --contimeout 60s --timeout 300s --retries 3 --low-level-retries 10 --stats 1s "/home/someuser/dropbox/sorted" "NameofRcloneProject:NameofGoogleDriveFolder" | |||
</code> | |||
====sync==== | |||
<code> | |||
rclone sync /home/someuser/dropbox/sorted NameofRcloneProject<ref>Where construction is the Name you gave this process to during rcloneconfig</ref>:GoogleDriveFolder<ref>where Construction is the name of the Google Drive folder</ref> --transfers=4 --checkers=2 --tpslimit=2 --drive-chunk-size=1M --bwlimit 4M | |||
</code> | |||
====Google Drive Limited Space==== | |||
For the 15GB limit, consider a mechanism to allow the deletion of remote files during the check. May need to do something local to check for current date, then move older folders out of the sync path... though that would mean the local drive could fill some day. | |||
<code> | |||
rclone sync /home/userdir/securitycameras NameofRcloneProject:NameofGoogleDriveFolder --transfers=4 --checkers=2 --tpslimit=2 --drive-chunk-size=1M --bwlimit 4M --delete-during | |||
</code> | |||
=runscript.sh= | =runscript.sh= |
Latest revision as of 14:39, 27 September 2020
Move Files
Perl
- Purpose:
- Read source dir
- Rename files and move to a Year\Month\Day structure
Code
use strict; use warnings; use English; use File::Basename qw( fileparse ); use File::Path qw( make_path ); use File::Spec; use File::Copy; use Date::Format; my $dir = '/home/user/dropbox/incoming'; my $destination ='/home/user/dropbox/sorted/'; foreach my $fp (glob("$dir/*.jpg")) { print "_______________________________________\n"; print "Source File: $fp\n"; my $filename_year=""; my $filename_month=""; my $filename_day=""; my $filename_camera=""; my $filename_time=""; my $filename_extension=""; my $output_filename=""; ### Filename Conversion ############################# # The original script used the file name for the source of time, however # the camera being used apparently likes to change to GMT without # reason... # # Pull the date from the file creation (or last mod) and use that instead ############################################## # Fought getting the time & epoch conversion with Perl to decide just use # Google ################################################ my @array = stat($fp); print "$array[9]\n"; my $inputEpoch = $array[9]; # Handling EDT... really bad later for EST but I'm tired $inputEpoch = $inputEpoch-14400; my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = gmtime($inputEpoch); $mon++; $year = $year + 1900; my $humanTime = sprintf ("%04d-%02d-%02d %02d:%02d:%02d", $year, $mon, $mday, $hour, $min, $sec); print "$humanTime\n"; ################################################# # Output Filename creation ### ################################################# $filename_camera="Prefix"; $filename_year=sprintf ("%04d", $year); $filename_month=sprintf ("%02d", $mon); $filename_day=sprintf ("%02d", $mday); $filename_time=sprintf ("%02d:%02d:%02d", $hour, $min, $sec); $filename_extension=".jpg"; #$output_filename=$filename_year.$filename_month.$filename_day.'_'.$filename_time.'_'.$filename_camera.$filename_extension; $output_filename=$filename_camera.'-'.$filename_year.$filename_month.$filename_day.'_'.$filename_time.$filename_extension; print "Output File: $output_filename\n"; ################################### # Check the destination folder(s) and create as needed ################################### ### Check for destination directory - YEAR my $dest_year = $destination.$filename_year; print "Dest Year: $dest_year\n"; ### Check for destination directory - MONTH my $dest_month = $dest_year.'/'.$filename_month; print "Dest Month: $dest_month\n"; ### Check for destination directory - DAY my $dest_day = $dest_month.'/'.$filename_day; print "Dest Day: $dest_day\n"; ### Dest File my $dest_file = $dest_day.'/'.$output_filename; print "Dest File: $dest_file\n"; ############# Create Dirs ################ if ( !-d $dest_year ) { print "Creating $dest_year\n"; make_path $dest_year or die "Failed to create path: $dest_year"; } if ( !-d $dest_month ) { print "Creating $dest_month\n"; make_path $dest_month or die "Failed to create path: $dest_month"; } if ( !-d $dest_day ) { print "Creating $dest_day\n"; make_path $dest_day or die "Failed to create path: $dest_day"; } #### MOVE FILES ##### # copy ($fp,$dest_file) or die "The move operation failed: $!"; move ($fp,$dest_file) or die "The move operation failed: $!"; # }
rclone
- Install rclone
- Reference for setup: here
- Execute rclone config from the account that will be using it.
- N(ew)
- Name: <NameofRcloneProject>
- Find the Google Drive entry (13):
- Client_ID: enter (default)
- Client_Secret: enter (default)
- Scope: 1 (we need to look into 3)
- Assumption: create a folder and get the folder ID for next line
- Root folder ID: Enter
- Service Account: Enter
- Edit Advanced config: n
- Use autoconfig? N (headless)
- Copy line to browser that is logged into the Google account to be used
- Grant rclone access
- copy the code
- Enter Verifcation Code: paste the code
- Configure this as a team drive? N
- Select: Yes this is OK
Example rclone lines
copy
rclone copy --update --verbose --transfers 30 --checkers 8 --contimeout 60s --timeout 300s --retries 3 --low-level-retries 10 --stats 1s "/home/someuser/dropbox/sorted" "NameofRcloneProject:NameofGoogleDriveFolder"
sync
rclone sync /home/someuser/dropbox/sorted NameofRcloneProject[1]:GoogleDriveFolder[2] --transfers=4 --checkers=2 --tpslimit=2 --drive-chunk-size=1M --bwlimit 4M
Google Drive Limited Space
For the 15GB limit, consider a mechanism to allow the deletion of remote files during the check. May need to do something local to check for current date, then move older folders out of the sync path... though that would mean the local drive could fill some day.
rclone sync /home/userdir/securitycameras NameofRcloneProject:NameofGoogleDriveFolder --transfers=4 --checkers=2 --tpslimit=2 --drive-chunk-size=1M --bwlimit 4M --delete-during
runscript.sh
- Purpose:
- make sure rclone isn't running
- fetchmail, then check (if mail), pull out mime objects
- run the perl to coordinate
- run rclone to sync
Contents
if ps ax | grep -v grep | grep -v grep | grep rclone > /dev/null then echo "RCLONE is running, let's not bind the system up" else echo "RCLONE is not running" fetchmail cp /var/spool/mail/somedir/home/somedir/ echo 'd *' | mail -N ripmime -i somedir -d /home/somedir/checkmail rm /home/somedir/checkmail/text* perl /home/somedir/movefiles.pl # rclone copy /home/somedir/securitycameras rclone:securitycameras --transfers=40 --checkers=15 --tpslimit=10 --drive-chunk-size=1M --bwlimit 4M rclone sync /home/somedir/securitycameras rclone:securitycameras --transfers=4 --checkers=2 --tpslimit=2 --drive-chunk-size=1M --bwlimit 4M exit 0 fi