1 of 53

COSCUP 2024�Rclone Introduction & Application

Presenter: Sheng-Hao Liao (frakw)

Time: 2024-08-04 10:30~11:00

Location: NTUST TR214

Mail: frakwu@gmail.com

1

#Easy #Elementary #Relax

2 of 53

About Me

  • First time submitting agenda to coscup
  • Name : 廖聖郝 (Sheng-Hao Liao)
  • Nickname/ID : frakw
  • Currently study master in NTUST CS
  • Research field:
    • Computer Vision
    • Computer Graphics
    • Server and Network
  • Website:

​

2

3 of 53

Outline

  • Rclone Introduction
  • Rclone Installation
  • Rclone Remotes
  • Rclone Commands
  • Advanced Functions
  • Useful Knowledges
  • Q&A

3

4 of 53

Rclone Introduction

4

5 of 53

Rclone Introduction

  • A command-line program
  • "rsync for cloud storage" (a lot)
  • Powerful file manage tool
  • Free and open source
  • Related links

​

  • Without rclone
    • Not easy to access cloud
    • Painful to write the code for every cloud storage

5

6 of 53

Rclone Introduction

6

7 of 53

Rclone Introduction

  • Developed with Golang
  • Cross platform
    • Linux
    • macOS
    • Windows
    • FreeBSD

​

  • Developed by Nick Craig-Wood and community

​

  • Users call rclone
    • "The Swiss army knife of cloud storage"
    • "Technology indistinguishable from magic"

​

7

8 of 53

Rclone Introduction

  • File operations
    • Sync
    • Copy
    • Move
    • Delete
    • …
  • File analyse
  • Mount remote as disk
  • Encrypt

8

9 of 53

Rclone Introduction

  • Remotes
    • Local file system
      • C: D: E: (windows)
      • /home/user (linux/macos)
    • Cloud storage backend
      • Google Drive
      • Microsoft OneDrive
      • Amazon S3
      • Dropbox
      • …
    • Virtual backend
      • Crypt
      • Chunk

​

9

10 of 53

Rclone Installation

10

11 of 53

Linux/macOS/BSD

  • https://rclone.org/install/
  • Install curl
    • sudo apt install curl

​

​

​

  • sudo -v ; curl https://rclone.org/install.sh | sudo bash

11

12 of 53

Windows

  • https://rclone.org/downloads/

​

​

​

​

​

​

​

  • Extract zip file

​

12

13 of 53

Windows

  • Copy rclone.exe path
  • Edit the system environment variables
  • System Variables
  • Path
  • Edit

13

14 of 53

Windows

  • Add
  • Paste rclone.exe’s path here

​

​

​

​

​

​

  • Check install successful
    • rclone --version

14

15 of 53

Rclone Remotes

15

16 of 53

Rclone Remotes

  • Remote is a storage service or location
  • For example
    • A remote for accessing google drive
    • A remote for accessing dropbox
    • A remote for encrypting to another remote
    • …
  • Remote contains
    • Name
    • Type
    • API Keys
    • Client id, secret
    • Token
    • Other necessary information

16

Setup remotes

17 of 53

Google Drive

  • rclone config
  • > n
  • Enter the remote name you want
  • Choose type as Google Drive (> drive)
    • Enter "drive" or
    • Enter 17

​

17

18 of 53

Google Drive

  • client_id, client_secret, scope, service_account_file leave empty
  • Edit advanced config?
    • > n
  • Use web browser to automatically authenticate rclone with remote
    • > y
    • Open the login page in the browser

​

  • Configure this as a Shared Drive (Team Drive)?
    • > n

​

  • Enter “q” to leave config mode

18

19 of 53

Google Drive

  • rclone ls frakw_gd:\coscup2024
  • rclone tree frakw_gd:\coscup2024

​

19

ls command:

tree command:

20 of 53

Microsoft OneDrive

  • rclone config
  • > n
  • > frakw_od
  • > onedrive or 33

20

21 of 53

Microsoft OneDrive

  • client_id, client_secret, region leave empty
  • Edit advanced config?
    • > n
  • Use web browser to automatically authenticate rclone with remote
    • > y
    • Open the login page in the browser

​

​

​

21

22 of 53

Microsoft OneDrive

  • Use OneDrive Personal
    • Enter "onedrive" or
    • Enter "1“
  • Enter “q” to leave config mode
  • Personal storage :

22

23 of 53

Dropbox

  • rclone config
  • > n
  • > frakw_db
  • > dropbox or 12

​

23

24 of 53

Dropbox

  • client_id, client_secret leave empty
  • Edit advanced config?
    • > n
  • Use web browser to automatically authenticate rclone with remote
    • > y
    • Open the login page in the browser

​

24

25 of 53

Rclone Config File

  • rclone.conf is the file store remotes information
  • Use following command to check config file (rclone.conf) location
    • rclone config file

​

  • Probably no way to change default config file location
  • But you can use following tag to specify a customized config file
    • --config=/path/to/rclone.conf

​

25

Linux

Windows

26 of 53

Rclone Config File

  • For each remote’s format
    • [remote1_name]
    • <attribute> = <value>

​

    • [remote2_name]
    • <attribute> = <value>

​

​

  • Move remote to another computer?
    • Just copy the text to new rclone.conf

​

26

27 of 53

Rclone Commands

27

28 of 53

Rclone Basic Commands

  • rclone copy <src_remote>:<src_path> <dst_remote>:<dst_path>
    • Copy from source to destination
    • dst += src

​

  • rclone sync <src_remote>:<src_path> <dst_remote>:<dst_path>
    • Make source and destination identical
    • dst = src

​

  • rclone move <src_remote>:<src_path> <dst_remote>:<dst_path>
    • Move files from source to destination
    • dst += src
    • src = 0

​

28

29 of 53

Rclone Basic Commands

  • A lot !

29

30 of 53

Rclone Basic Commands

  • Flags
    • --progress / -P : Show real-time transfer statistics
    • --verbose / -V : Show detail informations
    • --dry-run : Do a trial run with no permanent changes.
    • --transfers=N : How much files transfer at same time
    • --bwlimit=NM : Setup transfer bandwidth
    • --backup-dir=DIR : Backup the files be delete or update to specify directory when using sync/copy/move

​

​

30

31 of 53

Rclone Basic Commands

  • Filters
    • --include <include_rule> : Only transfer the files that match include rule
    • --include "*.txt" : Only transfer .txt files
    • --include-from /path/to/include-file : Use all lines in file for include rule

​

    • --exclude <exclude_rule> : Not transfer the files that match exclude rule
    • --exclude "*.mp4" : Not transfer .mp4 files
    • --exclude-from /path/to/exclude-file : Use all lines in file for exclude rule

​

    • --filter <filter_rule> : Transfer the files that pass the filter
    • --filter "- dir1/**" --filter "+ *.txt" : Exclude dir1 and then include .txt files
    • --filter-from /path/to/filter-file : Use all lines in file for filter rule

​

31

32 of 53

Rclone Analyse Commands

  • rclone size <remote>:<path>

​

​

  • rclone about <remote>:

​

​

​

  • rclone check <remote1>:<path1> <remote2>:<path2>
    • Checks the files in the source and destination match

​

​

32

33 of 53

Rclone Web GUI

  • rclone rcd --rc-web-gui
  • http://127.0.0.1:5572/

​

33

34 of 53

Rclone Web GUI

34

Add remote

Explore files

35 of 53

Advanced Functions

35

36 of 53

dedupe

  • rclone dedupe <remote>:<path>
  • Interactively find duplicate filenames and delete/rename them

​

​

36

37 of 53

mount

  • Mount the remote as file system on a mountpoint.
  • Windows
    • Require to install WinFsp https://winfsp.dev/
    • rclone mount <remote>:<path> <A~Z>:
    • rclone mount frakwu:\coscup2024 G:

​

  • Linux/macOS/BSD
    • cd /mnt
    • sudo mkdir coscup2024
    • sudo chmod 777 coscup2024
    • rclone mount frakw_gd:/coscup2024 /mnt/coscup2024
  • VFS cache mode

37

38 of 53

crypt

  • Crypt is a virtual backend for encrypting files (including filename)
  • Very useful for avoid cloud storage scanning or AI training

​

38

39 of 53

chunker

  • Chunker is a virtual backend for splits large file into smaller chunks during transfer
  • Useful for cloud storage services with a maximum file size limit

39

40 of 53

Rclone as API or library

  • Method 1
    • Use code to write your own config file
    • Call system command line : system("rclone copy src:abc dst:abc --config=my.conf")

​

  • Method 2
    • Use remote control
    • rclone rcd
    • https://rclone.org/rc/

​

40

41 of 53

Useful Knowledges

41

42 of 53

selfupdate

  • rclone selfupdate
  • Self upgrade to latest version

42

43 of 53

About Google Drive

  • Instead of storing many small files, it’s better to store large files
  • Would take longer time to transfer since API call limitation

​

​

  • server-side copy/move
    • --drive-server-side-across-configs
    • Same as 建立副本

43

44 of 53

Sharepoint

  • https://github.com/frakw/LargeSharepointDownload
  • Downloading large sharepoint folder as zip files always fails
  • Use rclone WebDav function to solve

44

45 of 53

Scheduled Task

  • Crontab
    • https://crontab.guru/

​

​

​

  • NSSM (windows)
    • https://nssm.cc/
    • Mount disk at system boot

45

46 of 53

Other Tools

  • Rsync https://github.com/RsyncProject/rsync
    • Traditional Linux tool used for synchronizing files and directories

​

  • Restic https://github.com/restic/restic
    • Backup file snapshot
  • Borg https://github.com/borgbackup/borg
    • Backup file snapshot

​

  • RaiDrive https://www.raidrive.com/
    • Mount cloud storage as network drive

46

47 of 53

Demo

If we have enough time…

47

48 of 53

Thanks for listening !

: )

48

49 of 53

Q&A

Please feel free to ask any questions !

(or contact me privately)

frakwu@gmail.com

2024/08/04

49

50 of 53

References

​

​

​

​

50

51 of 53

References

​

​

51

52 of 53

Materials

​

​

​

52

53 of 53

Materials

​

53