Download Files with Ruby

Feb 17, 2010  -  Comments

This weekend, I remembered an old podcast I used to listen to that I wanted to hear again. It wasn't available on iTunes anymore, so I did a Google search for it. I found the show's Web site, and saw that they had a complete archive of all four seasons of the podcast (20 episodes per season). The problem was, each episode was on it's own page, with a separate page to access the download link. I started to manually try to download each episode, but that got old REALLY quickly. Why not write a simple script to do the work for me?

Here's the Ruby script I whipped up in about 15 minutes:

require 'net/http'

1.upto(4) do |season|
  1.upto(20) do |episode|
    episode = '%02d' % episode
    puts "Downloading Season ##{season}, Episode ##{episode}"
    Net::HTTP.start("website.com") do |http|
      resp = http.get("/episode_archive/s#{season}ep#{episode}.mp3")
      open("s#{season}ep#{episode}.mp3", "w") do |file|
    puts "Episode downloaded!"

puts "All files downloaded!!"

An explanation of what's going on here:

Lines #3-4: There are 4 seasons and 20 episodes per season, so iterate through them.
Line #5: We need to left pad the episode numbers, because that's how the files are named.
Line #7: Connect to the host.
Line #8: Get the episode (named like "s1ep05.mp3").
Lines #9-11: Save the file using the same name.

Tagged: rubyscript

Rails Subdomain Caching

Oct 02, 2009  -  Comments

A while ago, I was working on a site that had dynamic subdomains based on cities in the database. For example, if an admin created a record for Nashville, it would create http://nashville.sitename.com. The site was pretty content heavy, but didn't change a whole lot, so I wanted to cache as much as I could. The problem with caching was that the content of each page depended on the subdomain. I couldn't use the normal caching strategy because of this.

To handle the subdomain routing and identification, I used the awesome subdomain-fu plugin (which worked great). Subdomain-fu, however, does not do the work of putting your cached files into folders named for your subdomain. Luckily, fixing this was as easy as adding a before filter to my ApplicationController.

class ApplicationController < ActionController::Base
  before_filter :update_cache_location
  def update_cache_location
    if current_subdomain.nil?
      ActionController::Base.page_cache_directory = "#{RAILS_ROOT}/public/cache/"
      ActionController::Base.page_cache_directory = "#{RAILS_ROOT}/public/cache/#{current_subdomain}/"

That method will update the page_cache_directory value to be your current subdomain (or default if you are on the main domain). Now, this takes care of one problem, but how the hell do we retrieve the cached files? I had a little bit of trouble with this, mainly because I'm not a mod_rewrite specialist. After some tinkering and testing, I finally came up with the following rules to put in my conf file to properly retrieve the cached files based on the subdomain.

# Check for subdomain cached index
RewriteCond %{HTTP_HOST} !^www\.sitename\.com
RewriteCond %{HTTP_HOST} ^([^.]+)\.sitename\.com
RewriteRule ^/$ /cache/%1/index.html [QSA]

# Check for subdomain cached page
RewriteCond %{HTTP_HOST} !^www\.sitename\.com
RewriteCond %{HTTP_HOST} ^([^.]+)\.sitename\.com
RewriteRule ^([^.]+)$ /cache/%1/$1.html [QSA]

# Check for regular non-subdomain index
RewriteRule ^/$ /cache/index.html [QSA] 

# Check for regular non-subdomain page
RewriteRule ^([^.]+)$ /cache/$1.html [QSA]

Those first two entries check to make sure the subdomain isn't www, then checks the cache folder for a cached version of the page. The last two entries check for the regular www subdomain.

These entries worked great for my project, but, like I said, I'm no expert with mod_rewrite. If you see anything that can be simplified or a better way of doing something, please let me know in the comments.

Tagged: railscachingtutorial

CentOS Setup Script with Ruby, Apache, mySQL, Subverion, Git, Passenger, and ImageMagick

Aug 31, 2009  -  Comments

Recently, I participated in the Rails Rumble competition. If you've never heard of the Rails Rumble, it's a 48-hour coding contest where teams of 4 programmers/designers have to complete a fully-working application over the course of a weekend.

Linode, one of the sponsors, provided each team with virtual private server to host their application. This was great, but the servers were bare-bones and required a fair amount of setup. Knowing this in advance, I hoped to save some time by writing a script that would automate the process for our team so I didn't have to spend 30-45 minutes just trying to configure the server.

Here are the steps I took to get Ruby, Apache, mySQL, Subversion, Git, Passenger, and ImageMagick set up on our CentOS server.

Step 1: Add custom repository listing

The default CentOS repo only has Ruby 1.8.6. While there's nothing wrong with this, I wanted to get 1.8.7. To do that, copy the following text into a file called /etc/yum.repos.d/rubyworks.repo:


Step 2: Bash script

The only other step is to copy the following bash script to a file (called anything you'd like), chmod it to be executable, and just run it as root (very important!). It's magic!


# You must run as sudo!!!!!

# update package list
yum update -y

# get all the packages required for compiling source
yum groupinstall 'Development Tools' -y

# install apache
yum install httpd mod_ssl -y

# install mySQL
yum --enablerepo=rubyworks install mysql-server mysql mysql-devel mysql-ruby -y

# set mysql and apache to start on boot
/sbin/chkconfig httpd on
/sbin/chkconfig --add mysqld
/sbin/chkconfig mysqld on
/sbin/service httpd start
/sbin/service mysqld start

# install postfix and subversion
yum install postfix subversion -y

# install git and dependencies
yum install gettext-devel expat-devel curl-devel zlib-devel openssl-devel -y
wget http://kernel.org/pub/software/scm/git/git-
tar zxvf git-
cd git-
make prefix=/usr/local all 
sudo make prefix=/usr/local install
cd ..

# install ruby
yum install --enablerepo=rubyworks ruby ruby-devel ruby-irb ruby-rdoc ruby-ri

# install ImageMagick and all of it's required packages
yum install tcl-devel libpng-devel libjpeg-devel ghostscript-devel bzip2-devel freetype-devel libtiff-devel -y
wget ftp://ftp.imagemagick.org/pub/ImageMagick/ImageMagick.tar.gz
tar zxvf ImageMagick.tar.gz
cd ImageMagick-*
./configure --prefix=/usr --with-bzlib=yes --with-fontconfig=yes --with-freetype=yes --with-gslib=yes --with-gvc=yes --with-jpeg=yes --with-jp2=yes --with-png=yes --with-tiff=yes
make clean
make install
cd ..

# get rubygems
wget http://rubyforge.org/frs/download.php/57643/rubygems-1.3.4.tgz
tar xvzf rubygems-1.3.4.tgz
cd rubygems-1.3.4
ruby setup.rb
cd ..

# install rails
gem install rails --no-rdoc --no-ri

# install rmagick
gem install rmagick --no-rdoc --no-ri

# install mysql gem
gem install mysql --with-mysql-config=/usr/bin/mysql_config --no-ri --no-rdoc

# install passenger and apache module
gem install passenger --no-ri --no-rdoc

DISCLAIMER: This worked great for me on a Linode 360 running CentOS 5.2. If you have any additions or changes, leave them in the comments and I'll be sure to add them. You can also get the entire script in a gist here.

Tagged: centosscript