Tuesday, January 6, 2009

HOW TO: Use RSYNC for Backup

Problem:
You want to be able to backup your most important data but you are too lazy to copy and paste everything all the time.


Solution:
Use RSYNC!

This is a tool in the command line, but don't be scared off by that, it is very simple to use, and if you don't have the patience to fully learn it then read on and I'll show you a simple yet effective method of backing up your data.

Rsync is a useful command as it only backs up the files you have changed. So, the first time you run it it will backup everything you select, and in the future it will backup only those files that have changed from within the original selection.

For the purposes of this tutorial I assume the following:

1. User name: bob (so home folder is /home/bob/)

2. External hard drive is mounted at: /media/harddrive/


First of all its always a good idea to create a folder inside of your external hard drive before backing up:

Open a terminal and enter the following:

mkdir /media/harddrive/backup


Now we are going to be making the backup bash script. I prefer making a script as the command is quite long. First we need to make a folder to store the script in (so it is easier to access).

mkdir bash

Then we must create the bash script:


cd bash

gedit backup.sh


In this text file put the following in:


#!/bin/bash

sudo rsync -rltDvu --modify-window=1 --progress --delete --delete-excluded --exclude-from=/home/bob/Bash/BackupBigExclude.txt /home/bob/ /media/harddrive/backup


Explanation of commands:

rsync: this is the program used to sync the data. Rsync basically can check the destination data and see if any changes have been made in the source data so that ONLY the source data that has been changed is updated. VERY useful.


-r: copies directories recursively

-l: copies symlinks as symlinks

-t: preserves modification times

-D: preserves device and special files

-v: shows output (verbose)

-u: skips files that are newer at the destination


--modify-window=1: this is essential if you are backing up to a filesystem that is NOT ext3 or ext2, e.g. NTFS or FAT32. Basically in windows filesystem times are kept as even numbers. This command tells rsync to ignore file changes that are only 1 second in difference from the original. It is almost impossible that you will create a file, sync it, and in ONE second make a change and want to sync it again. So it is safe to use this option and it means that rsync will not back up everything every time simply because of a one second change.

--progress: simply shows the progress

--delete: this is important. Basically rsync syncs files that have been changed. But what if you delete a file from the source directory? This command deletes it from your backed up hard drive as well.


--delete-excluded: this deletes folder/files that you have specifically excluded.


--exclude-from=/home/bob/Bash/BackupExclude.txt: this is the other crucial command. This basically tells rsync to exclude the files/folders found in the list BackupExlude.txt. To create this list do the following:

gedit /home/bob/Bash/BackupExclude.txt

Now in the text editor it is entirely up to you what you want to exlcude.

This is my list:

MyDownloads

.VirtualBox

lost+found

I am not backing up my MyDownloads directory as I download a lot of files/data some of which I do not want to backup. and I am not backing up my .VirtualbBox directory as its a massive 2gb file that will update every time I log into VB, a waste of time/resources. Just list the files/folders here that are in your home directory that you do not want to backup


Here are some other hints/tips for the exclude command file:

1. To exclude hidden directories (all the directories that start with .) do: .*/

2. To exclude hidden files (all the files that start with .) do: .*


locations: the /home/bob and /media/harddrive/backup tells the script where to backup from and where to backup to.


There are numerous other settings/commands you can use, including compression, etc. My method is simply to have a place where I can access all my data with minimal hassle.


Suffice to say these commands are quite technical, but they all work for me.


Save the files.


Now open a terminal and type the following:


cd /home/bob/Bash

sh BackupBig.sh


This should run the backup, it may take a long time depending on how much data you have.


NB: Make sure you take care when changing any settings. If you are unsure please ask. In fact don't even change the location without first confirming.

For example, You want to backup /home/bob/ to backup /media/harddrive/ and you have lets say data x.txt, y.txt, z.txt on /media/harddrive/. If you do not create a separate folder all data on /media/harddrive will be erased. SO BE CAREFUL!


For more information on rsync commands see http://www.samba.org/ftp/rsync/rsync.html.

67 comments:

  1. This is a great post! I also like that you explained all the switches. I also thing it's great that you went through an example so that users can get a bit more familiar with rsync.

    Also, I emailed you in response to your question on my blog. Did you get it? Are you still interested? Let me know when you have a moment.

    Thanks again for a great post! :)

    ReplyDelete
  2. Yes you did say you would e-mail me "after the holidays" but I never got an e-mail!

    Thanks for the comment, I always find it is much easier to do something when there is an easy example!

    ReplyDelete
  3. Nothing like open source software. When it comes to backup I have a hard time trusting backup software. There have been many cases where I have backed up my data using software and then when I needed to retrieve or access the file they were no where to be found. So I stick to the drag and drop method.

    ReplyDelete
  4. I've lost a lot of files trusting drag and drop methods as well. You're still running commands when you drag and drop, and what's worse if something goes wrong, the OS you are using might not tell you or give you inadequate information. verbos (-v) switch you can see exactly what's happening, and if it fails, exactly where it failed, and what's more you can see what's been copied already. Rather than blindly trust anything, you should always test the integrity of the backup by opening folders and backed up files on your backup media. this is the only way to ensure nothing went wrong. After all, even if the transfer did complete, media can still fail after the fact, making multiple backups across several media the safest. Redundency is your friend. Rsync can help you mannage an otherwise messy process.

    ReplyDelete
  5. Thanks for your explanation and script for rsync.
    Using UNR 9.04 I wanted to do a straight backup of my home directory to my external drive and did not require the exclusions.
    here is my ~/bash/backup.sh
    #!/bin/bash
    sudo rsync -rltDvu --modify-window=1 --progress --delete /home/walter/ /media/"FreeAgent Drive"/Backup/Netbook

    Although the backup worked I got errors that not all files were backed up (but did not say which) and a problem with permissions. Could you help?

    The use of capitals in some of your commands? (ie Bash)

    The meaning or ~!/bin/bash, all my work was in /~
    I used the command in ~/bash sh backup.sh
    Where can I view my terminal log and the error log?
    Can you suggest the error of my effort!

    reported ;-
    sent 1158966793 bytes received 104246 bytes 3560894.13 bytes/sec
    total size is 1158460928 speedup is 1.00
    rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1058) [sender=3.0.5]

    ReplyDelete
  6. Walter:

    Firstly, ignore the capitals. I just use them for some folders/files.

    Secondly,

    #!/bin/bash

    is the start of any bash script. (I'm not completely sure about this) It tells the script where the programme bash is located. It has nothing to do with where you're script is actually located. You need to put in this string at the start of all bash scripts (e.g. this one!).

    Thirdly, to generate a log file add the following into the rsync command:

    --log-file=FILE

    Finally, there is a way to check which files are not syncing. This error occurs mainly where there is a file and for some reason that particular file has a problem with syncing. For me it has been "locked" files (e.g. files currently in use), badly named files, etc. To test which files are giving the errors run the rsync script and AS SOON AS IT FINISHES run the script again. Because, rsync only sync files that have been changed, if you run the script in succession there won't be anything to update the second time round (assuming you don't change anything while rsync runs). So, the second time round you should see a list (in the terminal) of files that failed to sync. The error may look something like this:

    rsync: mkstemp "/media/H300/music/The Mothers of Invention/We're Only in It for the Money/.8 - What's the Ugliest Part of Your Body?.ogg.dbf1Zg" failed: Invalid argument (22)
    rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(105 [sender=3.0.5]

    Hope this helps!

    ReplyDelete
  7. Finally found a useful guide to get to know rsync. Everything else starts at the deep end with SSH and the likes. This is what I wanted! Thank you.

    ReplyDelete
  8. Thank you it's really greate post i will read again .

    ReplyDelete
  9. hey i need help i have a windows shared folder, i want to do some backups i have this:
    #/etc/fstab:
    //129.2.0.15/bkvpn/ /srv/RespaldoLinux cifs guest,user,rw 0 0

    #mount
    //129.2.0.15/bkvpn/ on /srv/RespaldoLinux type cifs (rw,mand,noexec,nosuid,nodev)

    ##whe i want to do the back up
    #rsync -rltDvu --modify-window=1 --progress --delete --delete-excluded /etc/ /srv/RespaldoLinux
    ##i get this
    #rsync: rename "/srv/RespaldoLinux/usb_modeswitch.d/.0421:060c.YvxpUm" -> "usb_modeswitch.d/0421:060c": Invalid argument (22)
    rsync: mkstemp "/srv/RespaldoLinux/usb_modeswitch.d/.05c6:1000:sVe=Option.mPLtMw" failed: Invalid argument (22)
    rsync: symlink "/srv/RespaldoLinux/modprobe.d/linux-sound-base_noOSS.conf" -> "/lib/linux-sound-base/noOSS.modprobe.conf" failed: Operation not supported (95)
    rsync: symlink "/srv/RespaldoLinux/network/run" -> "/dev/shm/network" failed: Operation not supported (95)
    rsync: symlink "/srv/RespaldoLinux/network/if-post-down.d/avahi-daemon" -> "../if-up.d/avahi-daemon" failed: Operation not supported (95)
    sent 115244874 bytes received 4116 bytes 7948206.21 bytes/sec
    total size is 120355276 speedup is 1.04
    rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1060) [sender=3.0.7]
    ## some files are ok but i recived that error im getting crazy i need help please
    bvargas@comglobalit.com

    ReplyDelete
  10. It's a shame you don't have a donate button! I'd most certainly donate to this superb blog! I suppose for now i'll settle for bookmarking and adding your RSS feed to my Google account.
    I look forward to new updates and will share this blog with my
    Facebook group. Chat soon!
    Also visit my weblog ; www.teenpornsexpussy.com

    ReplyDelete
  11. all the time i used to read smaller content which as well clear their motive,
    and that is also happening with this paragraph which I am reading here.
    Also visit my web site ... www.daily-wet-tshirt.com

    ReplyDelete
  12. Terrific post but I was wondering if you could write
    a litte more on this topic? I'd be very grateful if you could elaborate a little bit more. Bless you!
    Have a look at my web site ; free teen porn videos

    ReplyDelete
  13. I wanted to thank you for this very good read!! I definitely loved every little bit of it.
    I have got you bookmarked to check out new things you
    post…
    Feel free to visit my website ... www.partygalleries.com

    ReplyDelete
  14. For most up-to-date information you have to go to see world-wide-web and on web I found this website
    as a best web site for hottest updates.
    Here is my web page : http://www.crazyteenpics.com/

    ReplyDelete
  15. Excellent blog here! Also your website loads up fast! What host are you using?

    Can I get your affiliate link to your host? I wish my
    site loaded up as quickly as yours lol
    Feel free to visit my webpage - http://www.bhgalleries.com

    ReplyDelete
  16. Inspiring quest there. What occurred after? Thanks!
    Feel free to visit my web blog ... Free Teen Porn

    ReplyDelete
  17. Hi, always i used to check blog posts here early in the morning, since i like
    to learn more and more.
    My weblog : Horny Pussy Eating Lesbians

    ReplyDelete
  18. I was extremely pleased to uncover this page.
    I need to to thank you for your time for this fantastic read!

    ! I definitely loved every part of it and I have you saved to fav
    to check out new stuff in your blog.
    Here is my weblog Daily XXX Video Fix of teen porn with naked girls sex movies

    ReplyDelete
  19. Link exchange is nothing else but it is only placing
    the other person's website link on your page at proper place and other person will also do same in favor of you.
    my site - http://www.daily-wet-tshirt.com

    ReplyDelete
  20. It's remarkable in favor of me to have a website, which is valuable in favor of my experience. thanks admin
    My website - Hot Babe Gets Hammered Between 2 Horny Shemales

    ReplyDelete
  21. I just could not go away your website prior to suggesting that I
    really enjoyed the standard info a person supply to your visitors?

    Is gonna be back frequently to inspect new posts
    My website - click the next site

    ReplyDelete
  22. It's remarkable designed for me to have a web page, which is useful designed for my knowledge. thanks admin
    Feel free to visit my web site Giant dildo fucks hot and horny girls

    ReplyDelete
  23. I was able to find good information from your blog posts.
    Feel free to surf my blog : Young teen fucked hard

    ReplyDelete
  24. I was able to find good information from your blog posts.
    Also see my web page :: Young teen fucked hard

    ReplyDelete
  25. Wow, awesome blog structure! How lengthy have you ever been running a blog for?
    you make running a blog glance easy. The whole glance of your site is magnificent, as well as the
    content material!
    Also see my webpage > Marc Primo Pulisci

    ReplyDelete
  26. Great blog! Do you have any tips for aspiring writers?
    I'm hoping to start my own website soon but I'm a little lost on everything.
    Would you suggest starting with a free platform like Wordpress or
    go for a paid option? There are so many options out there that I'm totally confused .. Any suggestions? Bless you!
    Feel free to visit my homepage ; hookah bar in Orlando

    ReplyDelete
  27. Do you mind if I quote a few of your posts as long as I
    provide credit and sources back to your webpage? My blog
    is in the very same niche as yours and my visitors would really benefit from a lot of the information you
    present here. Please let me know if this alright with you.
    Many thanks!
    Also visit my web site ; used to

    ReplyDelete
  28. Malaysia & Singapore & brunei finest online blogshop
    for wholesale & quantity korean accessories, accessories, earstuds, pendant, rings, trinket,
    bangle & hair accessories. Promotion 35 % wholesale price cut. Ship Worldwide
    Also visit my web blog :: brain games ipad

    ReplyDelete
  29. Thanks for the great post... just what I was looking for.

    In regards to the exclude list, can you simply put Downloads etc. separate line or do you need to include the full path e.g. /home/user/Downloads/

    Thanks

    ReplyDelete
  30. each time i used to read smaller posts that as well clear their
    motive, and that is also happening with this piece of writing which I am reading here.
    Look into my web-site :: illinoisskillsmatch

    ReplyDelete
  31. I used to be recommended this website by my cousin. I am not certain
    whether this publish is written by him as nobody else know such specific about my trouble.
    You're wonderful! Thanks!
    my page > illinoisskillsmatch

    ReplyDelete
  32. Whats up very cool blog!! Man .. Beautiful .

    . Wonderful .. I will bookmark your blog and take the feeds also?
    I am happy to seek out a lot of useful information right here within the put up, we want
    work out extra strategies in this regard, thank you for sharing.
    . . . . .
    Visit my webpage ... click through the up coming web site

    ReplyDelete
  33. Amazing! This blog looks exactly like my old one! It's on a entirely different subject but it has pretty much the same layout and design. Superb choice of colors!
    Also visit my webpage ; best remodeling contractors in winter garden florida

    ReplyDelete
  34. excellent publish, very informative. I'm wondering why the other specialists of this sector don't notice
    this. You must continue your writing. I am sure, you've a great readers' base already!
    My site > New Bubble Shooter

    ReplyDelete
  35. Genuinely no matter if someone doesn't understand afterward its up to other users that they will help, so here it occurs.
    Also see my website > boating projects ecommerce specialist

    ReplyDelete
  36. Howdy I am so glad I found your weblog, I really found you by error, while I was browsing on Askjeeve for something
    else, Nonetheless I am here now and would just like to say many thanks for a fantastic
    post and a all round enjoyable blog (I also love the theme/design), I don't have time to read through it all at the minute but I have book-marked it and also included your RSS feeds, so when I have time I will be back to read much more, Please do keep up the great work.
    Also see my webpage: lhp.name.vn

    ReplyDelete
  37. Without any wateгing thе гoots of a tree it is
    futile to aѕsume it to grow аnd give us luscious fruits, colorful flοwers and sо foгth.
    Тhen theгe are researchегѕ and οthег worried people tοday who are
    ѕeemingly preventing a ѕhedding fіght to stem the tide maіn towаrds environmental
    Armagеddon or aѕ the title of this posting phone саlls it "Warmageddon. * Chinese pizza: Top notch unbaked pizza dough with hoisin sauce, sliced eco-friendly peppers and onions, and sauteed shitake mushrooms.

    Feel free to surf to my web-site - pizza stone kitchen boss

    ReplyDelete
  38. Hello this is kind of of off topic but I was wondering if blogs use
    WYSIWYG editors or if you have to manually code with HTML.
    I'm starting a blog soon but have no coding skills so I wanted to get advice from someone with experience. Any help would be greatly appreciated!

    Here is my web site; http://itp.nyu.Edu/

    ReplyDelete
  39. Chicagо pizzа is usuаlly meаty (ѕome variants соme ѕtuffed with chеesеs and mеat lаyeгѕ) and it іs eaten ωith
    a κnifе and fοгκ.
    Peter Piρer Pіzza Cοupons-Petег Ρiper Pizza Couρоns:Ρeteг Pipeг Ρіzza саn be a fаmіly pizzа chaіn operating 45 companу restаurants аnԁ 60 franсhiseѕ inside the U.
    Noω you're ready to be creative with the outside of the cake.

    my webpage: pizza pan alexandria va

    ReplyDelete
  40. Hi my loved one! I wish to say that this post is amazing, nice written and come with
    approximately all significant infos. I'd like to look more posts like this .

    Also visit my weblog ... ロレックスレプリカ

    ReplyDelete
  41. I’m not that much of a internet reader to be honest but your sites really
    nice, keep it up! I'll go ahead and bookmark your website to come back down the road. Many thanks

    My web site - ロレックススーパーコピー

    ReplyDelete
  42. I really love your website.. Excellent colors & theme.
    Did you create this web site yourself? Please reply back as
    I'm trying to create my own personal site and want to know where you got this from or just what the theme is named. Thank you!

    Here is my homepage :: ロレックスレプリカ

    ReplyDelete
  43. Very good info. Lucκy me I cаme аcross yоur site by chance
    (stumbleuρon). Ι've saved as a favorite for later!

    Take a look at my web blog; Chemietoilette

    ReplyDelete
  44. Hi Thanks for the informative post about backup data. This is lil bit hard to me so I have to pass your link to my brother :) to understand this perfectly.
    data recovery Companies

    ReplyDelete
  45. Good day very cool site!! Man .. Excellent .
    . Wonderful .. I will bookmark your website and take the feeds
    also? I am happy to seek out so many helpful info
    right here within the publish, we need develop more techniques in this regard, thanks for sharing.
    . . . . .

    Also visit my website; Replica Rolex Watches

    ReplyDelete
  46. Thanks for ones marvelous posting! I quite enjoyed reading it, you're a great author.I will be sure to bookmark your blog and will eventually come back sometime soon. I want to encourage you continue your great writing, have a nice holiday weekend!

    my website ... Replica Rolex

    ReplyDelete
  47. Hi there, i read your blog from time to time and i own a similar one and i was just wondering if you get a lot of spam feedback?
    If so how do you reduce it, any plugin or anything you can recommend?
    I get so much lately it's driving me insane so any assistance is very much appreciated.

    Stop by my blog post ... www.bringnewshoes.com

    ReplyDelete
  48. I believe that is one of the such a lot vital info for me.
    And i'm glad reading your article. However should remark on some normal issues, The website taste is great, the articles is in reality great : D. Good activity, cheers

    Feel free to visit my web blog ... www.raybanoutlet-sales.com

    ReplyDelete
  49. A motivating discussion is definitely worth comment.
    I believe that you need to publish more about this subject matter, it might not be
    a taboo matter but typically folks don't speak about such subjects. To the next! Best wishes!!

    my page ... rolex時計

    ReplyDelete
  50. At this time I am going to do my breakfast, when having my breakfast coming yet again to read further news.


    My website - air max 95

    ReplyDelete
  51. I am really impressed along with your writing talents as well as with the
    layout in your weblog. Is that this a paid subject matter or did
    you modify it yourself? Either way keep up the excellent
    quality writing, it is rare to peer a nice weblog like this one today.
    .

    Feel free to visit my site :: gracenfriends.com

    ReplyDelete
  52. I think that everything posted was very logical.
    However, what about this? suppose you added a little content?
    I am not suggesting your content is not solid., however what if you added
    a title to possibly get folk's attention? I mean "HOW TO: Use RSYNC for Backup" is kinda plain. You might glance at Yahoo's front page
    and see how they write news headlines to get viewers
    to open the links. You might add a related video or a pic or
    two to grab people excited about everything've got to say. In my opinion, it might bring your posts a little livelier.

    Feel free to surf to my web page - アバクロンビー

    ReplyDelete
  53. A person essentially assist to make critically articles I'd state. This is the very first time I frequented your website page and so far? I amazed with the research you made to make this particular put up incredible. Magnificent task!

    my site - モンスターケーブル

    ReplyDelete
  54. Great information. Lucky me I found your website by chance (stumbleupon).
    I've saved it for later!

    Feel free to surf to my blog post エアジョーダン

    ReplyDelete
  55. My brother recommended I would possibly like this website.
    He was once totally right. This post actually made my day.
    You cann't consider simply how much time I had spent for this information! Thanks!

    Feel free to visit my weblog; オークリー サングラス

    ReplyDelete
  56. You really make it seem really easy with your presentation but I in finding
    this topic to be really something which I feel I would
    never understand. It sort of feels too complicated and extremely large for me.

    I'm taking a look ahead to your next publish, I will attempt to get the grasp of it!

    Also visit my website - monster beats

    ReplyDelete
  57. Pretty section of content. I just stumbled upon your weblog and in
    accession capital to claim that I acquire in fact enjoyed account your blog posts.
    Any way I will be subscribing for your feeds and even I success you access consistently rapidly.


    Feel free to visit my web page ... monster ヘッドホン

    ReplyDelete
  58. Hey there would you mind stating which blog platform you're working with? I'm going to start my own blog in the near future but I'm having a hard time selecting between BlogEngine/Wordpress/B2evolution and Drupal. The reason I ask is because your design and style seems different then most blogs and I'm looking for something unique.
    P.S Apologies for being off-topic but I had to ask!


    Stop by my page :: http://www.guccihandbagsoutletsalee.com/

    ReplyDelete
  59. Great work! That is the type of info that are meant to be shared across the web.
    Disgrace on Google for not positioning this post higher! Come on over and
    visit my web site . Thank you =)

    My site sex on cam

    ReplyDelete
  60. Hello to all, how is all, I think every one is getting more from this
    website, and your views are fastidious for new users.


    my web page - babesflick.com

    ReplyDelete
  61. I couldn't resist commenting. Well written!

    my web-site アバクロ tシャツ

    ReplyDelete
  62. After all, communication is a vital part of the club as it
    moves forward on and off. It's continuing to leverage the publicity we are able to meet with the sales and student loan needs. The best part about it is that I learn especially considering I live with 3 of them! This is kinda what we talk about advertising vs student loan, we must understand and build an image upon these variables and their interactions, and must take rational decisions.

    my web blog ... The Student Loan People

    ReplyDelete
  63. After I initially commented I seem to have clicked on the
    -Notify me when new comments are added- checkbox and from now on whenever
    a comment is added I get 4 emails with the exact same comment.
    Perhaps there is an easy method you are able to remove me from that
    service? Thank you!

    Feel free to visit my page; xxxmoviegalls.com

    ReplyDelete
  64. Its like you read my thoughts! You seem to grasp a lot about this, such as you wrote the guide in
    it or something. I feel that you simply could
    do with some % to power the message home a bit,
    however other than that, this is fantastic blog.

    A fantastic read. I'll definitely be back.

    Also visit my site porn cheat for grand theft auto :: www.alva-audio.de ::

    ReplyDelete
  65. GSCopy Pro v6.0 (RoboCopy Alternative) with Open File Agent
    GSCopyPro is a single command-line tool (CLI) that can copy, replicate and move files from one folder to another. This folder can be on the same machine/ server or another server elsewhere. What makes GSCopyPro stand out from other competitors is the fact it works on 32-bit as well as 64-bit systems and has no restrictions. It can easily be scheduled to run as a scheduled task and fully automated. GSCopyPro also comes with an open file agent which can copy files that are locked/ opened by other processes. This feature is supported in all windows versions from widows XP/ 2003 and later.
    Go To:>> http://www.gurusquad.com/GSCOPYPRO

    ReplyDelete
  66. GSCopy Pro v6.0 (RoboCopy Alternative) with Open File Agent
    GSCopyPro is a single command-line tool (CLI) that can copy, replicate and move files from one folder to another. This folder can be on the same machine/ server or another server elsewhere. What makes GSCopyPro stand out from other competitors is the fact it works on 32-bit as well as 64-bit systems and has no restrictions. It can easily be scheduled to run as a scheduled task and fully automated. GSCopyPro also comes with an open file agent which can copy files that are locked/ opened by other processes. This feature is supported in all windows vSCersions from widows XP/ 2003 and later.
    Go To:>> http://www.gurusquad.com/GOPYPRO

    ReplyDelete