Err the Blog Atom Feed Icon
Err the Blog
Rubyisms and Railities
  • “Vendor Everything”
    – Chris on March 27, 2007

    Seriously. You’ll thank me later.

    What do I mean, exactly? Well, let’s say you’re working on a small Rails team. You decide to start using test/spec. Be dee dee. As usual, you gem install test-spec then Pistonize the plugin. You start writing specs and begin converting existing tests to specs. You’re on a tear. Nothing can stop you. Behavior is king. You commit your changes.

    You break the build.

    What? Why? Well, all your new specs depend on the test-spec gem, a gem your comrades and continuous integration builder do not have.

    Quickly: fix it! Tell everyone to install the gem locally. Install the gem on your staging server. Carefully install the gem on your production server. Phew. Everyone’s got the same version, right? Right. Well, maybe. (At least the build works.)

    See, there’s something wrong with this scenario: it’s not very DRY. Why should only our code be DRY? Why not our environment, too? It should be.

    The solution we’ve come up with is to throw every Ruby dependency in vendor. Everything. Savvy? Everyone is always on the same page: we don’t have to worry about who has what version of which gem. (we know) We don’t have to worry about getting everyone to update a gem. (we just do it once) We don’t have to worry about breaking the build with our libraries. (we leave that up to the internet entrepreneurs)

    Adding Depended Sees

    Alright, alright, let’s see how we’d vendor the test-spec gem. First, the vendor directory: 
    $ ls -1 vendor/
bin
data
gems
plugins
rails

    We obviously added a few directories, namely gems, data, and bin. The rails and plugins directories should be familiar, one hopes.

    Let’s focus on the the vendor/gems directory: 
    $ ls -1 vendor/gems/
RedCloth-3.0.4
RubyInline-3.6.2
crypt-1.1.4
image_science-1.1.1
memcache-client-1.3.0
session-2.4.0
sphinx-0.9.7-rc2
    We’ll cd in there and then use the handy gem unpack command to, erm, unpack the contents of our test-spec gem: 
    $ gem unpack test-spec
Unpacked gem: 'test-spec-0.3.0'

    Cool. Now we need to dive into our config/environment.rb file to ensure Rails knows to look in vendor/gems/test-spec-0.3.0/lib when we try to require ‘test/spec’. It’s easier than you think.

    Add this to your Rails::Initializer.run block:

    config.load_paths += Dir["#{RAILS_ROOT}/vendor/gems/**"].map do |dir|
  File.directory?(lib = "#{dir}/lib") ? lib : dir
end

    Now all the libraries in vendor/gems will automatically be included in your load path, complete with a lib check for libraries like RubyInline and crypt (which don’t come with lib directories).

    Want to only include some libraries in a specific environment? Maybe you don’t want RubyInline or image_science in development mode. Put this under the above snippet o’ code:

    if %w(development test).include? RAILS_ENV
  config.load_paths.delete_if { |f| f =~ /RubyInline|image_science/ }
end

    Good to go.

    It should be noted that this trick will not auto-require the gems for you. You still need to do that in your config file, or in your gems, or wherever. Maybe with a line like %w(crypt/blowfish redcloth).each { |f| require f } in your config/environment.rb if it pleases you.

    Other Approaches

    For one, Dr Nic has something cool: this patch. It lets you run tasks from within vendor/gems right from your RAILS_ROOT. Nifty. He’s also got his gemsonrails plugin which is a similar (but different) approach than we illustrate here.

    Jay Fields has his own method for autoloading gems in vendor. Worth a look, and a listen.

    Classic Railer topfunky, back in the day, rolled a rake task to scratch the same itch.

    Then there’s this thread on Rails-Core about adding some kinda gemy-ness to Core which, unfortunately, hasn’t yet transpired. The gem in question, the one to metaly manage other gems, lives in technoweenie’s repository.

    I even do this (for better or worse) with cache_fu, btw. I am starting to really like not being dependent on the environment.

    All Set!

    The goal here is simple: always get everyone, especially your production environment, on the same page. You don’t want to guess at which gems everyone does and does not have. Right.

    There’s another point lurking subtlety in the background: once all your gems are under version control, you can (probably) get your app up and running at any point of its existence without fuss. You can also see, quite easily, which versions of what gems you were using when. A real history.

  • Ben, about 4 hours later:

    Nice write-up… thanks.

    You never said what /vendor/bin and /vendor/data were for…

  • Dr Nic, about 6 hours later:

    Amen.

  • mathie, about 7 hours later:

    Great idea! We did start out this way on the project I’m currently most involved in (pretty much exactly as you’ve described, though we chose not to put the version numbers in the gem’s directory name) and it worked well … to a degree.

    How do you deal with extension-based gems that need C code to be built?

  • Val Aleksenko, about 11 hours later:

    You can just set GEM_HOME in you config/boot.rb like this: ENV[‘GEM_HOME’] = File.join(RAILS_ROOT, ‘vendor’, ‘gems’)

    and it would give you all the advantages without any additional coding.

    We use the similar approach only when need to lock down out app to specific (tested) gems versions – http://revolutiononrails.blogspot.com/2007/02/locking-down-deployed-application.html

  • Ned Baldessin, about 11 hours later:

    One problem this creates, is that doing a deploy with capistrano takes ages, because you’re exporting a huge number of files from SVN. One solution to this is to not put them into version control and symlink the vendor/ directory after deploy.

  • Jason Young, about 11 hours later:

    No.

    Please don’t. Please for the love of others having to deal with your code, don’t put core dependencies in vendor (not if you control your servers, or your team/company does). Having a little developer communication about the gems that you as a team are going to decide to use is a whole lot better down the line then having dozens of different copies of a gem lying around – when you realistically only need one.

    That way when the next “Upgrade your Rails NOW NOW NOW” occurs, you can deal with the upgrades once, and not dozens of times for applications that you’ve probably forgotten that you ever wrote, much less deployed – at old contract jobs left long ago.

  • Chris, about 12 hours later:

    mathie: So far it hasn’t been a problem. Our vendor/bin directory houses a subdirectory for each arch (i686-darwin8.8.2, powerpc-darwin8.6.0, x86_64-linux) so we throw some compiled code in there. We have a tmp/inline directory created on deploy for RubyInline, and we don’t actually vendor mongrel—so that pretty much covers it all.

  • Chris, about 12 hours later:

    Val: Nice! That’s a neat trick. How would you exclude gems on a per-environment basis, though?

  • Val Aleksenko, about 12 hours later:

    To Chris: The same way you’d normally do with regular gems – you only require those gems unless RAILS_ENV == ‘production’ or but putting requires in corresponding config/environments/*.rb

  • Chris, about 12 hours later:

    Ned: Or you can have cap run svn up. One of the big advantages of this method is being able to deploy and roll back gems in case of an emergency. Version control is a huge win here.

  • PJ, about 13 hours later:

    Jason, in theory that makes sense, but when you’re working in larger teams and you don’t want to explain to your CSS guy for the fifth time how to install/update their gems, you’ll be happy knowing that all of the dependencies are in vendor.

    As far as the slow-going deploys with everything unpacked in vendor, that’s such a huge misnomer. We’re talking about a 5-10 second difference. (At least for our servers)

  • Adam Sanderson, about 14 hours later:

    Hey that looks like a really neat solution. Always glad to see what you cook up here.

  • Eduardo Rocha, about 15 hours later:

    What about a Maven approach? Maybe we could have config/gems.yml, and always have this checked to see if all gems and theirs versions where installed.

  • Eduardo Rocha, about 15 hours later:

    And have them installed automatically if not there… :)

  • Val Aleksenko, about 16 hours later:

    To Eduardo: We use similar approach @Revolution

    Each application or component has its own config defining gem dependencies with versions. It is being used at runtime to load specific gems. An example: 
    :dependencies:
  - ['rails', '1.1.6']
  - ['json', '~> 0.4']

    The same config is used for packaging to a gem so the dependencies are translated to gem dependencies. It means that when you ‘gem install’ a gem created that way, you also pick up all the dependencies declared. Works us for us pretty good.

  • nicholas a. evans, about 19 hours later:

    I’ve been slowly pushing my team in this direction as well. I really really like having as much as possible in version control, and I find that the benefits (time travel, instantly deployed dependencies) far outweigh the costs (wasted disk space, slightly longer checkouts). For applications where the deployment team wants tight control over what gems are installed where, how, etc… or if other trade-offs are in play, I can see the approach that Val uses at Revolution Health being very useful.

    Chris, would you mind giving a bit more detail about how you do this for a gem with C extensions (rMagick, for example) and how you handle the bin directory? Do the contents of your bin directory come from gems, or is it only for homegrown compiled binaries?

  • Jason Young, about 24 hours later:

    PJ – I’d hope that the design-o that managed to get Locomotive installed and working and groks subversion (or at least textmate bundles that shell out to svn) well enough to check their CSS changes into a repository could learn to handle gem update/install.

    Chris, I do love you man – in that totally platonic developer sort-a-way. The blog post wasn’t all about you – it was mainly for your legion of adoring admirers that don’t think for themselves.

    But I agree, CAPS LOCK TEXT IS LOVE :-)

  • Rich Collins, 24 days later:

    I was wondering about compiled binaries as well. Do you have an easy way to recompile between platforms?

  • Lori Olson, 28 days later:

    +1 for the “giving a bit more detail about how you do this for a gem with C extensions (rMagick, for example) and how you handle the bin directory”

    With a web designer/CSS guy on Windows, developers on Intel and PPC Macs, build, test, and production servers on Linux, this is a non-trivial issue for our team.

  • Shane Vitarana, 2 months later:

    I’m totally for the vendor everything approach, but 

    gem unpack
    just copies the source files and doesn’t build anything. As Lori mentions, some gems with C extensions need to be compiled first, such as ruby-xslt. In this case you still have to run 
    gem install
    .

    I guess one solution is to use 

    gem install
    with the 
    -i
    option for your vendor/gems directory. The load paths would have to be changed to reflect this.

  • Kyle Maxwell, 2 months later:

    I wrote a simple Rails plugin to build your C extensions in the vendor everything approach. You’ll prolly want include it into your Capistrano workflow, although that is left as an exercise for the user ;).

    http://svn.kylemaxwell.com/rails_plugins/vendor_everything_extensions/

    Usage: rake vendor:build_extensions

  • Jodi, 2 months later:

    I know I’m coming to this party late, but I thought I’d throw this in the fridge in case anyone’s still thirsty.

    lovin this !

    a correct to Val’s otherwise great solution: “GEM_HOME for writing, GEM_PATH for reading!” from http://wiki.rubyonrails.org/rails/pages/HowToUseMultipleGemRepositories

    so you’ll want to: ENV[‘GEM_PATH’] = File.join(RAILS_ROOT, ‘vendor’, ‘gems’)

    (not gem home as suggested).

    cheers,Jodi

  • Chris Kampmeier, 2 months later:

    Ned/Chris/PJ: the speed of your deploys shouldn’t be a factor once you move over to Capistrano 2.0, because we’ve got this now:

    set :deploy_via, :remote_cache

    This creates a cached copy of your code in the shared directory, and just runs an ‘svn up’ on that when you deploy. This code then gets copied to the releases directory as usual. Nice and speedy.

  • Bala Paranj, 10 months later:

    How did you get the vendor gems directory? I don’t have that in my vendor directory.

  • roupen n., 10 months later:

    Chris, what about \vendor\data what’s that for? you talked about \vendor\bin

  • Mike Herrick, 11 months later:

    I’m trying to get this set up with a Rails 2.0 based application.

    Any updates on this approach. I’m probably missing something obvious, but while the application works fine via ruby script/server I can’t run any rake tasks (fail looking for gems used by rake tasks). I am guessing I have to set the load path somewhere, but don’t know where. I had hoped that the config/environment.rb settings would get picked up and it would just work, but doesn’t seem to be the case.

    Any ideas?

    Thanks.

  • Barrett, about 1 year later:

    For RMagick we just copied the necessary file(s) into the vendor/gem/rmagick… directory. There is a RMagick2.so (Rmagick2.bundle on Mac) file that is required.

    Same deal with our Sybase adapter.

    Good to go. So far.

  • Twenty-seven people have commented.
    Chime in.
    Sorry, no more comments :(
This is Err, the weblog of PJ Hyett and Chris Wanstrath.
All original content copyright ©2006-2008 the aforementioned.