Weblogs for kumanna

Posted by kumanna on Wed 3 Oct 2012 at 02:09
Tags: none.

It's important to back things up. You should also make your life easier by using some good software. I use obnam for some backups, but I wanted to try out bup for a new backup workflow I needed. I will describe the workflow here.

The situation: I have a laptop with things to be backed up. I also have a Raspberry Pi which I use as a network attached storage (it has a 1 TB external drive attached to it) and runs Raspbian Wheezy on it. Naturally, you can do this on any NAS running some GNU/Linux variant. In addition, I also have an external portable USB hard disk where another backup copy will be stored. How do I automate this?

Here is how I went about this process. First, for the Raspberry Pi:

  1. First, identify the directory or directories to be backed up. To keep this simple, the directory I am backing up is /home/kumar/Work.
  2. On the Raspberry Pi, install bup. If you want to build it from the git clone, you would need python-dev, and pandoc for the documentation.
    sudo aptitude install pandoc python-dev
    git clone git://github.com/apenwarr/bup.git
    cd bup;make;sudo make install
    
  3. Initialize the backup directory on the Raspberry Pi:
    BUP_DIR=/home/kumar/BACKUPS bup init
    BACKUPS is the place where bup is asked to store the backups (much like it's analogue of .git). You can store all your backups within this directory, and you can still classify backup sets within this directory with different names (much like git branches).
  4. On the local machine, initialize the backup directory.
    bup index -uv /home/kumar/Work
  5. Back it up. This will take time the first time, if the Work directory has several contents within it.
    bup save -r kumar@<raspberry_pi_ip_address>:BACKUPS -n workbackup /home/kumar/Work
    
    workbackup is the name of the backup set. This can be paperbackup or databackup or notesbackup so that several classes can be backed up within the BACKUPS directory.
  6. Now, you have backed things up. Make changes in the Work directory. For instance, add/edit/remove/move some files.
  7. Rerun the index and save commands from steps 4 and 5 above. Enjoy the speedup and ability to rewind to the previous state.
  8. Test restoration of ONE directory of backup (just to save time) on Pi itself directly on the Pi:
    bup -d /home/kumar/BACKUPS/ restore -C /tmp/ /workbackup/latest/home/kumar/Work/offset-coupling-params
    
    Then scp it back if you need it.
  9. Alternative: Use fuse. On the Pi:
    sudo aptitude install python-fuse
    sudo adduser kumar fuse
    mkdir Mount
    bup -d /home/kumar/BACKUPS fuse Mount
    Then cd into Mount and cp/scp the correct files wherever.
    fusermount -u Mount
    
  10. Once happy, write the bup index/bup save commands into a script and run it periodically to back up (check this at the end of the post). Have fun.

Next, to prepare the external hard disk, I went with the old-style autofs automounting solution:

  1. To automount the disk, in /etc/auto.master, add
    /auto			/etc/auto.misc	--timeout=60
    
  2. In /etc/auto.misc add:
    backupdrive	-fstype=btrfs	:UUID="7927a7ab-8c3f-4198-bd73-d3cc519a9ac3"
    
    fstype may need changing for your file system, and UUID can be found using blkid on the exact partition of the hard disk you wish to back up to.
  3. Initialize the backup directory:
    sudo mkdir /auto/backupdrive/BACKUPS
    sudo chown kumar.kumar /auto/backupdrive/BACKUPS
    bup -d /auto/backupdrive/BACKUPS/ init
    
  4. Index and back up:
    bup -d /auto/backupdrive/BACKUPS/ index -uv /home/kumar/Work
    bup -d /auto/backupdrive/BACKUPS/ save -n workbackup /home/kumar/Work
    
  5. Make changes in the Work directory. Add/edit/remove/move files.
  6. Rerun index and save commands from step 5. Enjoy the speed.
  7. Test restoration of ONE directory of backup from removable disk:
    bup -d /auto/backupdrive/BACKUPS/ restore -C /tmp/ /workbackup/latest/home/kumar/Work/offset-coupling-params
    
  8. Once happy, write the bup index/bup save commands into a script and run it periodically to back up. Have fun.

Finally, a quick script to automate all of the above.

#!/bin/sh

bup index -u /home/kumar/Work
bup save -r kumar@<raspberry_pi_ip_address>:BACKUPS -n workbackup /home/kumar/Work
if ls /auto/backupdrive/ 2> /dev/null;then
    echo Backing up to the removable disk...
    bup -d /auto/backupdrive/BACKUPS/ index -u /home/kumar/Work
    bup -d /auto/backupdrive/BACKUPS/ save -n workbackup /home/kumar/Work
    echo Backed up to the removable disk! Yay!
else
    echo WARNING: Not backing up to the removable disk
fi

What this script does it, it backs up to the NAS, and then checks if the external disk is connected. If the disk is mounted, then it backs up to that disk as well. It might be a good idea to cron or anacron this script, or have someone remind you to run this periodically. The only limitation I have is that you can't use bup restore to restore from a remote repository yet, but that isn't a huge issue for me now. In addition, metadata is not stored by the bup save/restore commands. You would have to use split and join with tar to get that. However, bup is fast moving, so I'd expect these features soon. Till then, this workflow works all right for data only backups.

Although this is just a mental dump of the procedure I made for myself, suggestions for improvement are welcome. Thanks.

 

Posted by kumanna on Wed 23 Mar 2011 at 15:41

I have been reading Linux Shell Scripting Cookbook by Sarath Lakshman, published by Packt, for a while. While most people I know learn shell scripts themselves, I was looking to refresh my concepts a little as well as have a reference lying around on the table for fast access. Since I was asked to review it for someone else, I thought I might as well post the review here.

First of all, let me remark by saying that shell scripting is something learned more on a need basis than as a tool to solve the main problem. People would seldom write shell scripts as standalone programs (exceptions exist). However, what makes shell scripting invaluable to know is the fact that knowing some tricks can save several minutes, or hours, of work by automating and simplifying certain tasks, generally (but not restricted to) file management and data processing. Linux Shell Scripting Cookbook does go quite far in pursuing this goal, and is appropriate for both beginners who are looking to gain dexterity in shell scripting, as well as intermediate users who wish to polish their skills. The book also can double up as a quick reference, though I would argue that the "Advanced Bash Scripting Guide" would suit that more.

At the outset, the author clarifies that the focus will be on Bash. This, people may or may not like, but the fact that bash has become ubiquitous in terms of the available shells on Unix-like systems today, starting out with bash is not a bad thing to do. Besides, learning other shell scripting languages while knowing bash isn't too hard, since the paradigm remains the same.

Let's go through the books from the aspects I found most relevant.

Organization

The book is organized into chapters based more on utility than scripting concepts themselves, although the language aspects are brought onto the reader gradually. For instance, the examples in the first chapter focus more on the basic data elements (variables, arrays, functions etc.) as well as operators (for numbers, files etc.), and all the examples demonstrate simple usage of these concepts, and he further chapters build upon these in a gradual manner.

At the same time, if the reader has some familiarity with shell scripting and needs to only refresh or learn a certain concept, he/she needs to just read the relevant chapter. It is not too difficult to grasp the examples of the later chapters, provided some basic shell knowledge is assumed.

Content and Presentation

A positive trait in the presentation of this book is that it is all based on practical everyday examples which, with minor adaptation, can be used by many for their own daily tasks. For instance, there are several examples which describe searching for and processing files, which, I'd imagine, many users would want to do on a regular basis. Thus, providing realistic examples allows the book to double its utility. The language and approach used is simple and conversational, and the presentation is very clear, with each idea being described as a problem statement followed by a "How to do it" section with the actual code, and ending with a discussion of the nitty-gritties of the code. It is easy to go for a quick scan for those in a hurry, while those who with to read in more detail will not be disappointed either.

The book also covers a wide array of applications. For instance, there are examples on automating fetching web pages and processing them, demonstrations of parsing and simplifying and even some queries around databases wrapped around in shell. It also spans to utilities and tasks connected to statistics, backups, compression, version control and many more.

Breadth and Depth

The book goes into a fair amount of detail in terms of describing the shell scripting concept under consideration. The examples used go into a fair amount of detail in order to describe to the user all the aspects involved in the method or command being used. The concepts described are fairly complete, and would be sufficient for the reader to use immediately or with just a little bit of fine tuning. In terms of breadth, the book covers most of the features of shell scripting while also describing the various facilities the shell provides access to in a Unix-like environment. Thus, the book does not disappoint in this front either.

In summary...

In summary, probably the only thing I'd have liked to see more of is some emphasis on how to write more efficient shell scripts. Granted, most of the shell scripts described in the book are very simple and succinct, but a some words on how loops can be made better, or how to spot situations where pipes are not needed to solve a problem etc. might have been a nice addition. Some explanation of differences with dash, tcsh, zsh etc. might also have been nice, since a lot of users have different default shells. But all this isn't going to prevent me from giving this book a high rating, since it delivers quite well on the promises it makes at the beginning.

This is definitely a good book to have near your desk, and kudos to the author for having taken the effort to put it together. I would highly recommend it to the beginner and occasional shell user for a thorough read, and to an intermediate to have on his/her desk for borrowing the cool scripting ideas and applications the author has written in this book.

 

Posted by kumanna on Thu 3 Mar 2011 at 14:59
Tags: none.

Over the next month or so, I've decided to read an interesting book which covers one of my favourite topics, shell scripting. The book is Linux Shell Scripting Cookbook. It has been quite a while since I have actually read a book on a programming topic (other than K&R or the like), but I guess it does feel "complete" to gain knowledge from a book and feel as though we have gained at least some mastery over the topic!

Updates on this will follow.

 

Posted by kumanna on Fri 10 Dec 2010 at 03:57
Tags: none.

As a follow up to my previous post on the same topic, now DuckDuckGo supports what I wanted:

Someone rightly pointed out in my last post that anyone can have these customizations in any good browser. That is correct, but often, I use a browser in a machine where my custom quick-searches are not available, but a quick access to my search engine is. In such situations, I find this useful. I thought some others might like it too. :-)

 

Posted by kumanna on Fri 3 Dec 2010 at 15:00
Tags: none.

So, DuckDuckgo now has !bang search goodness for Debian packages (!dpackages) and Debian BTS (!dbugs). The BTS search actually launches the "search", while I'd have preferred just http://bugs.debian.org/%s, so that !dbugs src:python-numpy would just take me directly to the numpy bug page, but it's a start. The search engine creator is very receptive to comments, though, and I am sure you can reason with him if you feel most people would find it useful if it is changed to something else; I wasn't sure what the best course of action was, so I haven't talked to him yet.

Here is the full list of !bang searches in DuckDuckGo, and a submission form to request for new ones. Enjoy!

 

Posted by kumanna on Wed 25 Aug 2010 at 04:22
Tags: none.

Even in the modern times of AJAX, Web 2+ and what not, Elinks is one of my favourite browsers, at least for browsing some sites. The reasons are simple: it cuts the unnecessary distractions, and allows you to digest text based information rather quickly. In addition, since it doesn't display any images or flashy elements like Flash, pages load lightning fast even on poor connections, making it an excellent option.

Unfortunately, some sites don't really work very well with Elinks, and for those, I do one of two things. For the ones which are really complex, such as secure login sites with too much AJAX, I just resort to one of the "standard" graphical browsers. For some others, I try to work around the problem using Elinks' rich scripting interface available in many programming languages including Perl and Ruby. I chose Lua, since it seemed simple enough to learn. I wanted to share with you some of the simple workarounds I implemented for some sites I visit often.

The first quirk is with the Debian BTS, whose CSS isn't properly interpreted by Elinks (see 593804 and 593840 for details). The short story: the BTS renders like this:

So, I grabbed the CSS, and made some small changes (described in the above mentioned bug reports), and got a new CSS file. I put it in /home/kumar/Kumar/bugs.css, and added the following to my ~/.elinks/hooks.lua:

-- Use an alternate CSS for the Debian BTS to work around some quirks
function debian_bts_format_html_hook (html)
   return string.gsub(html, "/css/bugs.css", 'file:///home/kumar/Kumar/bugs.css')
end

function pre_format_html_hook (url, html)
   if sstrfind (url, "bugs.debian.org") then
      return blonnet_format_html_hook(html)
   end
   return nil
end

and with it, the BTS renders like this:

Much more readable!

The second simple example, for which I won't include images, is the Mobile Twitter interface. The mobile Twitter interface is very convenient and functional in Elinks; however, by default, the text font seems to be a light grey, which is not discernible in the bright background. A default user.css file would be a possible solution, but I have not decided whether to go for it just yet. So, for the Twitter site, I add a function like this:


-- Make the status messages black on Mobile Twitter
function mobile_twitter_format_hook(html)
   return string.gsub(html, ".status.-}", '.status {word-wrap: break-word; color : #000;}')
end

And alter the actual hook function to:

function pre_format_html_hook (url, html)
   if sstrfind (url, "bugs.debian.org") then
      return debian_bts_format_html_hook(html)
   elseif sstrfind (url, "mobile.twitter.com") then
      return mobile_twitter_format_hook(html)
   end
   return nil
end

Finally, a more complicated example. I like reading the Indian business newspaper the Hindu Business Line. However, if you try browsing to the site with Elinks, you'll notice that, barring the top hyperlinks in each sectcion, the links on the left are not "linked" at all.

In addition, there are some other irritants which are easy to remove, such as the navigation bar on the top, and the text colour being too light for my taste etc. So, I write the following Lua blurb. Why I write this is left for the interested readers to figure out, though it shouldn't be surprising if you open the page. The workaround for the missing links is interesting, though; it is because the website authors have put a hyperlink (<a>) tag before a list item (<li>) tag, and Elinks seems to not like that too much. So, I just swap the two around, and Elinks is happy.


-- Reformat the Business Line website
function blonnet_format_html_hook (html)
   html = string.gsub(html, "<body ", '<body text="#000000"') -- I like the text in black on the white background
   html = string.gsub(html, "<[Ff][Oo][Nn][Tt] class=leftnavi color=brown>.-</[Ff][Oo][Nn][Tt]>", '') -- An ugly hack to remove the navigation bar
   html = string.gsub(html, "<TD align=left vAlign=top width=\"500\"><IMG.-</FONT><BR>", '', 1) -- Remove description
   html = string.gsub(html, "<A class=groupnavi href=\"http://epaper.thehindubusinessline.com/\">.-</FONT><BR>", '', 1) -- Remove other group links
   html = string.gsub(html, "(<[Aa] class=navi.-)<li>(.-</[Aa]>)", "<li>%1%2</li>") -- Fix the erroneous section sidebar
   return html
end

function pre_format_html_hook (url, html)
   if sstrfind (url, "blonnet.com") or sstrfind(url, "thehindubusinessline.com") then
      return blonnet_format_html_hook(html)
   elseif sstrfind (url, "bugs.debian.org") then
      return debian_bts_format_html_hook(html)
   elseif sstrfind (url, "mobile.twitter.com") then
      return mobile_twitter_format_hook(html)
   end
   return nil
end

And here's the old and new rendering:

 

Posted by kumanna on Sun 8 Aug 2010 at 21:03
Tags: none.

So, I have finally visited my first Debconf, and it was a phenomenal experience. To meet persons who have guided you, persons work with, and persons you respect face-to-face and have conversations with them about several topics ranging from their vision for Debian to their views on topics totally unrelated to software is something truly great. I am now convinced that events like this, which provide the right environment to socialize and discuss various issues concerning Debian, will go a long way into getting several aspects of the project moving along in the right directions, as well as create awareness to the outside world about what the project is about and how it benefits users.

In terms of discussions, the Debian Science track was a nice way to see how contributors are putting Debian to good use for scientific research, and it was a nice opportunity to list out the things to be done to fill in the gaps and take Debian as a science platform forward. The Debian Python BoF was also a good place to meet many contributors face-to-face, and learn more about the technical concerns which people have in regard to the future of Python in Debian.

While I didn't hack much during the conference, Christian's presence made me finish the Hindi translation of the Debian Installer. We can now shift focus to other aspects, such as getting packages in order for a short freeze-to-release cycle.

Columbia University provides for a very nice venue, with easy access while maintaining a quaint environment within itself. Since this is my second visit to New York City, I had an idea of the common tourist locations, so I did not go around much during the conference. However, since I have a buffer day, I might try my hand at some photography around the place.

Have a safe trip home, and a wonderful time ahead! I leave you with a photo I took when I went with a friend across the Brooklyn Bridge after Debconf.

Downtown Manhattan Panorama - from the Brooklyn Heights Promenade, Brooklyn, NY

 

Posted by kumanna on Sun 11 Apr 2010 at 19:57

So, I will be at DebConf10 from 4th August onwards. The prime purpose of my visit is to meet all Debian developers, so I don't think the delayed attendance should matter much.

Customary banner: I'm going to DebConf10!

See you all there!

 

Posted by kumanna on Sun 11 Apr 2010 at 20:00

So, I will be at DebConf10 from 4th August onwards. The prime purpose of my visit is to meet all Debian developers, so I don't think the delayed attendance should matter much.

Customary banner: I'm going to DebConf10!

See you all there!

 

Posted by kumanna on Thu 7 Jan 2010 at 04:01
Tags: none.

So, while Richard is busy doing research and arranging or facilitating several Debian workshops, here I am, gulping down Idlis and Dosas by the dozen. It's been a fun visit, and I got to visit several places in my native state, but since my visit to Mumbai is rather short, I won't be able to say hello to several friends this time (Sorry Kartik). Maybe this would happen only when I am back in India on a permanent basis; let's see...

Here are some general guidelines for travelers which I'd like to share:

  • They say it doesn't snow in Dallas. Don't believe them. I had to stay overnight at Dallas, and then embark on a trip in the other direction (trans-Pacific) to get home. It wasn't a really fun journey.
  • If you're ever rerouted due to flight delays, ensure you travel by a similar route, or else, ensure that you're not a vegetarian/vegan. If you fly to Tokyo and Hong Kong, for instance, be sure that you've specified your meal choices well in advance. Of course, you can live on fruits and bread, but you'll really be doing yourself a favour by not missing your flights in the first place; I don't know how I have wronged the weather gods this time.
  • Ensure that you don't keep too many valuables (which Customs might be suspicious of) in your checked-in baggage. This is important, especially to get through situations where you find that your baggage doesn't arrive with you (my baggage came a day late, and, thankfully, cleared customs quickly since my clothes were duty free, thanks to the many heavy duty cycles they've gone through in washing machines).

It's been a really fun trip, and I am sure I'll enjoy the rest of the trip, till I get back to my old routine and resume work.

Finally, in keeping with the spirit of a blog I try to understand (albeit, in vain, I must accept), I duplicate the concept of a post from that blog, here is an up-to-date list of airports in which I have sat, but never left except by way of airplane:

  • LGW
  • IDR
  • EWR
  • CLT
  • TKO
  • HKG

More updates later. Bye for now.