Redline Software Inc. - Winnipeg's Leader in Ruby on Rails Development

Using God to Automatically Monitor Mongrels

Here is a sample god configuration file for monitoring all the mongrels on your system. This script assumes that your mongrel configs are all in /etc/mongrel_cluster and that your pid_file attribute in your config files point to an absolute path. I lifted the majority of this script from the example in thin. (Thin is awesome FYI)

I have a few mongrel config files in /etc/mongrel_cluster that look similar to:

1
2
3
4
5
6
7
8
9

---
user: deploy
group: www-data
cwd: /var/www/apps/sampleapp/current
port: "8200"
environment: production
address: 127.0.0.1
pid_file: /var/www/apps/sampleapp/shared/pids/mongrel.pid
servers: 2

When I want to deploy another application on the same machine, I add the config file there and then restart god. It will then begin automatically monitoring the new application.

My /etc/god.conf:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84

require 'yaml'
config_path = "/etc/mongrel_cluster"

Dir[config_path + "/*.yml"].each do |file|
  config = YAML.load_file(file)
  num_servers = config["servers"] ||= 1

  (0..num_servers-1).each do |i|
    number = config['port'].to_i + i

    God.watch do |w|
      w.group = "mongrel-" + File.basename(file, ".yml")
      w.name = w.group + "-#{number}"
      w.interval = 30.seconds

      w.uid = config["user"]
      w.gid = config["group"]

      w.start = "mongrel_rails cluster::start -C #{file} --only #{number}"
      w.start_grace = 10.seconds
      w.stop = "mongrel_rails cluster::stop -C #{file} --only #{number}"
      w.stop_grace = 10.seconds
      w.restart = "mongrel_rails cluster::restart -C #{file} --only #{number}"

      # assemble the pid file, pid files look like mongrel.8000.pid, mongrel.8001.pid etc
      pid_path = config["pid_file"]
      ext = File.extname(pid_path)
      w.pid_file = pid_path.gsub(/#{ext}$/, ".#{number}#{ext}")
      w.behavior(:clean_pid_file)

      w.start_if do |start|
        start.condition(:process_running) do |c|
          c.interval = 5.seconds
          c.running = false
          c.notify = 'team'
        end
      end

      w.restart_if do |restart|
        restart.condition(:memory_usage) do |c|
          c.above = 150.megabytes
          c.times = [3, 5] # 3 out of 5 intervals
          c.notify = 'team'
        end

        restart.condition(:cpu_usage) do |c|
          c.above = 50.percent
          c.times = 5
          c.notify = 'team'
        end
      end

      # lifecycle
      w.lifecycle do |on|
        on.condition(:flapping) do |c|
          c.to_state = [:start, :restart]
          c.times = 5
          c.within = 5.minute
          c.transition = :unmonitored
          c.retry_in = 10.minutes
          c.retry_times = 5
          c.retry_within = 2.hours
          c.notify = 'team'
        end
      end
    end

  end
end

God::Contacts::Email.message_settings = {
  :from => 'god@example.com'
}

God::Contacts::Email.server_settings = {
  :address => "localhost",
  :port => 25,
  :domain => "example.com"
}

God.contact(:email) do |c|
  c.name = 'team'
  c.email = 'team@example.com'
end

One thing that the god install doesn’t do is create a nice init.d script, but you can find one here which I found from this post about using God.

The God website has an example with explanations of what each of those sections mean. Check it out.

Comments