Agile Web Development

Build it. Launch it. Love it.

Bj

Backgroundjob (Bj) is a brain dead simple zero admin background priority queue for Rails. Bj is robust, platform independent (including windows), and supports internal or external manangement of the background runner process.

Jobs can be submitted to the queue directly using the api or from the command line using the ./script/bj:

  api:
    Bj.submit 'cat /etc/password'

  command line:
    bj submit cat /etc/password

Bj’s priority queue lives in the database and is therefore durable - your jobs will live across an app crash or machine reboot. The job management is comprehensive capturing stdout, stderr, exit_status, and temporal statistics about each job:

  jobs = Bj.submit array_of_commands, :priority => 42

  ...

  jobs.each do |job|
    if job.finished?
      p job.stdout
      p job.stderr
      p job.exit_status
      p job.started_at
      p job.finished_at
    end
  end

In addition the background runner process logs all commands run and their exit_status to a log named using the following convention:

  rails_root/log/bj.#{ HOSTNAME }.#{ RAILS_ENV }.log

Bj allows you to submit jobs to multiple databases; for instance, if your application is running in development mode you may do:

  Bj.in :production do
    Bj.submit 'my_job.exe'
  end

Bj manages the ever growing list of jobs ran by automatically archiving them into another table (by default jobs > 24 hrs old are archived) to prevent the jobs table from becoming bloated and huge.

All Bj’s tables are namespaced and accessible via the Bj module:

  Bj.table.job.find(:all)         # jobs table
  Bj.table.job_archive.find(:all) # archived jobs
  Bj.table.config.find(:all)      # configuration and runner state

Bj always arranges for submitted jobs to run with a current working directory of RAILS_ROOT and with the correct RAILS_ENV setting. For example, if you submit a job in production it will have ENV[‘RAILS_ENV’] == ‘production’.

When Bj manages the background runner it will never outlive the rails application - it is started and stopped on demand as the rails app is started and stopped. This is also true for ./script/console - Bj will automatically fire off the background runner to process jobs submitted using the console.

Bj ensures that only one background process is running for your application - firing up three mongrels or fcgi processes will result in only one background runner being started. Note that the number of background runners does not determine throughput - that is determined primarily by the nature of the jobs themselves and how much work they perform per process.

Vitals

Home http://codeforpeople.com/lib/ruby/bj/
Repository git://github.com/github/bj.git
License Ruby's
Rating (12 votes)
Created 29 December 2007

Comments

  • Avatar
    MB
    25 July 2008

    I had some trouble getting the run method of runner to find new jobs. Using MySQL. I modified the AR conditions at line 189 to:

    :conditions => ["state = ? and curdate() <= submitted_at", "pending"]

    Works like a charm.

  • Avatar
    3 October 2008
    1. When submitting a job using rails 2.1+, Backgroundjob, using Active Record, converts the submittedat value to UTC. However, when reading the bjjobs table, the bj daemon does not convert the server's current time to UTC. Thus, scheduled jobs will be delayed by (server timezone - UTC) hours.

    2. You need to explicitly set --log=/path/to/file when starting the bj daemon.

    3. The bj script does not work when submitting jobs from the command-line. Instead, you should create a ruby script that calls Bj#submit.

  • Avatar
    26 November 2008

    It looks like this code has migrated.

    The repo URL above http://codeforpeople.rubyforge.org/svn/rails/plugins/bj/ appears to be pre-historic.

    Ara's website & rubyforge contain versions up to 1.0.1 at http://codeforpeople.com/lib/ruby/bj/ and http://rubyforge.org/projects/codeforpeople

    Github contains what appears to be a blessed migration (version number currently 1.0.3) plus some forks. http://github.com/search?q=bj http://github.com/github/bj/tree/master

    Hopefully this will help people trying to track down the "latest version".

  • Avatar
    5 March 2009

    I just installed BJ for the first time and noticed the following error in my log after the script/bj command ran migrations: <pre>"Mysql::Error: BLOB/TEXT column 'hostname' used in key specification without a key length: CREATE UNIQUE INDEX index_bj_config_on_hostname_and_key ON bj_config (hostname, key)".</pre> Rails version = 2.2.2, mysql-5.0.67-osx10.5-x86. <br/> Is there any particular reason all the string columns in the database are 'text' not 'varchar' (String)?

Add a comment