s

Posts Tagged with "ruby"

Using God Gem to Monitor mySQL

Oct 02, 2011  -  Comments

I recently ran into a problem on a client server where the msyqld process died, and neither of us noticed it for almost 24 hours. That meant that the site was completely useless that entire time. Needless to say, the client wasn't thrilled about this.

I told him that I'd look into setting up a monitoring service to, at the very least, notify me if msyqld ever crashed. During my research, I stumbled upon the god gem (which I'd used in the past to monitor mongrel processes back in the days before Passenger). I didn't realize it could be used to monitor other processes as well, such as msyqld.

God is a rubygem, so it's easy to install on any *nix operating system. It's config files are also written in Ruby, so that makes it even better.

What you'll need to get the script running:

  1. Install the god gem (as root).
    sudo gem install god
  2. Find the location of your msyqld pid file.
    mysqladmin -u root -p variables | grep pid_file
  3. Determine the command to start/stop/restart the msyqld service. Usually /etc/init.d/mysqld

Here is the config file I wrote to watch my mysqld process:

  # run in non-daemonized mode (so you can monitor it) with `god -c /path/to/mysql.god -D`
  # run normally with `god -c /path/to/mysql.god`

  # Settings for email notifications (optional)
  God::Contacts::Email.defaults do |d|
    d.from_email = 'god@my-app.com'
    d.from_name = 'God'
    d.delivery_method = :smtp # this can also be :sendmail
    d.server_host = 'smtp.myapp.com'
    d.server_port = 25
    d.server_auth = true
    d.server_domain = 'myapp.com'
    d.server_user = 'smtp_user@myapp.com'
    d.server_password = 'password'
  end

  # you can create as many email entries as you'd like
  God.contact(:email) do |c|
    c.name = 'me'
    c.to_email = 'me@email.com'
  end

  God.watch do |w|
    # you can name this whatever you want
    w.name = "mySQL Server"

    # polling interval
    w.interval = 30.seconds

    # command to start service
    w.start = "/etc/initd/mysqld_start_ampamp_/etc/initd/httpd_restart/index.html"

    # command to stop service
    w.stop = "/etc/initd/mysqld_stop/index.html"

    # command to restart service
    w.restart = "/etc/initd/mysqld_restart_ampamp_/etc/initd/httpd_restart/index.html"

    # how long to wait after starting service before monitoring resumes
    w.start_grace = 20.seconds

    # how long to wait after restarting service before monitoring resumes
    w.restart_grace = 20.seconds

    # location of pid file
    w.pid_file = "/var/run/mysqld/mysqld.pid"

    # tell god to delete the pid file when mysqld crashes
    w.behavior(:clean_pid_file)

    # determine the state on startup
    w.transition(:init, { true => :up, false => :start }) do |on|
      on.condition(:process_running) do |c|
        c.running = true
      end
    end

    # determine when process has finished starting
    w.transition([:start, :restart], :up) do |on|
      on.condition(:process_running) do |c|
        c.running = true
      end
      # failsafe
      on.condition(:tries) do |c|
        c.times = 8
        c.within = 2.minutes
        c.transition = :start
      end
    end

    # start if process is not running
    w.transition(:up, :start) do |on|
      on.condition(:process_exits) do |c|
        # send an email to me to notify me that the service has crashed
        c.notify = 'me'
      end
    end

    # lifecycle
    w.lifecycle do |on|
      # If the service keeps triggering a restart over and over, it is considered to be "flapping".
      on.condition(:flapping) do |c|
        c.to_state = [:start, :restart]
        c.times = 5
        c.within = 1.minute
        c.transition = :unmonitored
        # If the service is flapping, wait 10 minutes, then try to start/restart again.
        c.retry_in = 10.minutes
        c.retry_times = 5
        c.retry_within = 2.hours
      end
    end
  end

A few things to note about my script:

  1. In my start and restart commands, I also restart Apache. Running my site with Passenger, I would get a Rack error when mysqld restarted, but apache hadn't. YMMV.
  2. I'm using an SMTP server for my email notification. You can also use sendmail, or you can exclude the email notifications altogether.
  3. When running the script for the first time, it's a good idea to add the -D flag. This runs god in non-daemonized mode so all output is piped to STDOUT. That way, you can watch what it's doing to ensure everything is working correctly.

For more information about the god gem, see their site: god.rubyforge.org

Tagged: rubymysqlgod

Download Files with Ruby

Feb 17, 2010  -  Comments

This weekend, I remembered an old podcast I used to listen to that I wanted to hear again. It wasn't available on iTunes anymore, so I did a Google search for it. I found the show's Web site, and saw that they had a complete archive of all four seasons of the podcast (20 episodes per season). The problem was, each episode was on it's own page, with a separate page to access the download link. I started to manually try to download each episode, but that got old REALLY quickly. Why not write a simple script to do the work for me?

Here's the Ruby script I whipped up in about 15 minutes:

require 'net/http'

1.upto(4) do |season|
  1.upto(20) do |episode|
    episode = '%02d' % episode
    puts "Downloading Season ##{season}, Episode ##{episode}"
    Net::HTTP.start("website.com") do |http|
      resp = http.get("/episode_archive/s#{season}ep#{episode}.mp3")
      open("s#{season}ep#{episode}.mp3", "w") do |file|
        file.write(resp.body)
      end
    end
    puts "Episode downloaded!"
    puts
  end
end

puts "All files downloaded!!"

An explanation of what's going on here:

Lines #3-4: There are 4 seasons and 20 episodes per season, so iterate through them.
Line #5: We need to left pad the episode numbers, because that's how the files are named.
Line #7: Connect to the host.
Line #8: Get the episode (named like "s1ep05.mp3").
Lines #9-11: Save the file using the same name.

Tagged: rubyscript

Ruby Script to Add Apache Virtual Host Entry

Aug 10, 2009  -  Comments

When you're working with Rails, you never really have to add a virtual host entry for development (unless you use Passenger). You can always just fire up script/server and navigate to http://localhost:3000.

At my new job, I'm doing a lot of work on PHP and Drupal sites, which require you to add an entry to your host file and add a virtual host conf file for Apache. After only going through this process twice, I was already tired of it. I wrote the following script to automate the process for me.

#!/usr/bin/ruby

########################################
##### VARIABLES YOU NEED TO CHANGE #####
########################################

host_dir  = '/etc/hosts'            # path to your hosts file
sites_dir = '/Library/Webserver'    # path to the directory where you keep your sites (NO TRAILING SLASH!!!)
conf_dir  = '/etc/apache2/sites'    # path to directory where named conf files live

########################################

unless ARGV[0]
  puts "Usage: add_site sitename [hostname_for_url]"
  puts "Example: add_site sample sample.dev"
  exit
end

name = ARGV[0].strip
hostname = ARGV[1].nil? ? ARGV[0] : ARGV[1].strip

# first things first: make sure named conf file doesn't exist already
if File.exists?("#{conf_dir}/#{name}.conf")
  puts "Conf file named #{name}.conf already exists!"
  exit
end

# check to make sure host file exists
if File.exists?(host_dir)
  
  puts "Adding entry to #{host_dir}."
  File.open(host_dir, 'a') do |host_file|
    # append host entry to end of file
    host_file.puts "127.0.0.1\t#{hostname}"
  end
  puts "Host entry added!"
  
  puts "Adding named conf file."
  File.open("#{conf_dir}/#{name}.conf", 'a') do |host_file|
    # add entry
    host_file.puts <<EOF
<VirtualHost *:80>
  ServerName #{hostname}

  DocumentRoot "#{sites_dir}/#{name}"
	DirectoryIndex index.php
	<Directory "#{sites_dir}/#{name}">
		Options FollowSymLinks MultiViews Includes
		AllowOverride All
		Order allow,deny
		Allow from all
	</Directory>
</VirtualHost>
EOF
  end
  puts "Conf entry added!\n"
  
  puts "Restarting apache.\n"
  system "apachectl graceful"
  puts "Done!"
  
end

What you need to do to get this to work:

  1. Change the variables at the top of the file (path to your hosts file, path to the folder where you keep your development site, and path to the directory where you want to keep your named conf files).
  2. Rename the file to add_site(with no extension) and move to your /usr/bin directory.
  3. Chmod the file to be executable.

Now you can run the add_site command and provide it with the name of the site folder and optionally the name of the local domain you'd like to use.

Tagged: rubyscriptapache

Twitter Gem Examples

Apr 22, 2009  -  Comments

I recently set up a Twitter account for a monthly bill and task tracking application that I built a few months back. My intent was to try and drive traffic to my site (which had been sitting unused by the general public). To do this, I decided to mass-follow around 350 accounts in hopes of having them follow me back and checking out the site. It worked pretty well, and I even had quite a few users cold follow the account. At first, I would follow the users that followed me when I got the notification from Twitter. After a few days, I got a little behind and the followers started to build up. I figured this would be a good time to check out the Twitter gem to see if I could automate some of my tasks. The gem had exactly what I needed: a way to talk to Twitter via Ruby. I've included below two of the tasks that I created to work with my Twitter account.

First things first, I needed to set up my authentication. To do this, I just created a YAML file in my home directory called .twitter that contains my user email and password. The . means that it's a hidden file (I'm on a Mac). The YAML file is extremely simple, and looks like this:

email: my_twitter_email
password: my_twitter_password

Now, I could use this YAML file for any of the scripts that I wrote.

Task #1: Follow Users Who Follow Me

I wanted to get a list of all my followers and check to see if I'm already following them. If I'm not, I want to create a friendship with them.

#!/usr/bin/env ruby
require 'rubygems'
require 'twitter'

config = YAML::load(open("#{ENV['HOME']}/.twitter"))

httpauth = Twitter::HTTPAuth.new(config['email'], config['password'])
base = Twitter::Base.new(httpauth)

base.followers.each do |follower|
  if !follower.following
    # make sure to rescue in case there is anything wrong with the account
    base.friendship_create(follower.id, true) rescue next
    puts "Created friendship with #{follower.screen_name}"
  end
end

Task #2: Stop Following Users Who Aren't Following Me

I followed about 350 accounts initially, and after about a week, I figured that if they weren't following me by then, they'd probably never follow me. So, since I'm all about reciprocation, I decided to stop following them.

#!/usr/bin/env ruby
require 'rubygems'
require 'twitter'

config = YAML::load(open("#{ENV['HOME']}/.twitter"))

httpauth = Twitter::HTTPAuth.new(config['email'], config['password'])
base = Twitter::Base.new(httpauth)

base.friends.each do |friend|
  if !base.friendship_exists?(friend.screen_name, 'listode')
    base.friendship_destroy(friend.id)
    puts "Destroyed friendship with #{friend.screen_name}"
  end
end

A Quick Note

Keep in mind that, unless you've been white-listed, your account is limited to 100 API calls per hour. That shouldn't be an issue with the first script, since it only makes one call to get the list and one call for each friend creation. You should stay below the cap (unless you have more than 100 followers who you aren't following).

The second script is a different story. It makes one call to get the list of friends, one call for each friend to check following, and one call to destroy the friendship if they aren't following. This can easily burn up the API limit if you have more than 100 friends. I haven't figured out a way to reduce the number of API calls for that script. If you have any tips, leave them in the comments.

Tagged: twitterrubytutorial