uploading with an ftp macro script
Posted by PJ_at_Belzabar_Software on Thu 2 Feb 2006 at 09:15
Gather round the hearth, young nerdlings and I will tell you a tale...just let me settle my creaking bones into my rocking chair, let me wipe my rheumy eyes and nose - there, that's better. Now pass me my ear trumpet. Do give me a little prod if I nod off or my voice wavers too much. Are we all settled in now? Yes? Marvellous! Now let me tell you about ftp upload.
People don't respect the ftp (the file transfer protocol) that much anymore. Partly because it is a plain text protocol, which means that passwords used are snoopable. But mainly because it isn't as visual and well-known as web-based file transfers for the non-geek, nor is it as intelligent and powerful as rsync for the geek. Nonetheless, ftp is often handily there - even that cesspit win95 had an ftp client.
Now, I have used FTP for 15 years now, and like all of you, I knew you could mget, get, mput, put, hash, bin, prompt etc. However, it was news to me that ftp had a macro facility hidden inside its depths.
Uploading with an FTP script
I wanted to do an automated upload of files and directories. Previously I had been looking at scripts that would upload files and directories via bash feeds. After playing around with those, I was somewhat dissatisfied because such scripts just spew out all the commands in one go. On large transfers or slow networks I worried that this might overwhelm the ftp server input buffer, and the server might lose commands.
A way around it is to use expect, which brings wait-for-responses interactivity to scripts. Or you can of course just throw up your hands in disgust and use that bazooka called perl to pound and pulverize the problem into submission with net::ftp.
But that is getting overly complicated, and I am a lazy and simple-minded fellow.
Enter the macro
It turns out that in most cases FTP macros are a splendidly lazy and simple way to do such things.
In a nutshell:
- You use macdef to define the macro. See man ftp for details
- Combine this with netrc. .netrc is the file that has the default settings for the ftp client - see man (5) netrc for details.
- Use macdef init in the .netrc file, and you have a quick way to automatically script the upload.
Example: you have some files in your htdocs directory that you want to upload to a remote public_html directory. You also have files under an image subdirectory and under a css subdirectory that you want to upload. You want to keep the directory structure and files the same on the remote site as on your local box.
The .netrc file for this would be something like:
machine ftp.example.com login mrexample password mysecretpassword macdef init #this is an embarrassing way to do a comment prompt hash lcd /home/mrexample/htdocs cd public_html mput * mkdir css #already exists? no problem lcd css cd css mput * mkdir ../images lcd ../images cd ../images mput * quit #newline terminates macro
I think you get the idea.
Now, if you do a ftp ftp.example.com the script will automatically run with no further input, (because of the init keyword).
How it works
By default glob is on, so the * lists all the files c-shell style, ignoring dot files. The files under htdocs will get mputted into ftp.example.com:///home/mrexample/public_html/. Any directories won't mput (there will be a great wailing from the ftp client), but nothing evil will happen, so we just let the script run on. It runs on and does the copying of the directories css and images separately by explicitly dropping into the subdirectories.
The comment lines don't make sense (there will be much gnashing of teeth from the server), but nothing evil will happen, so we just keep them in for our own reference.
So, there you are. A simple, lazy way to script ftp to upload and leave a smug smile on your face.
PJ, Belzabar Software Design