Vendor Everything — err.the

“Vendor Everything”

– Chris on March 27, 2007

Advertisement
Seriously. You’ll thank me later.

What do I mean, exactly? Well, let’s say you’re working on a small Rails team. You decide to start using test/spec. Be dee dee. As usual, you gem install test-spec then Pistonize the plugin. You start writing specs and begin converting existing tests to specs. You’re on a tear. Nothing can stop you. Behavior is king. You commit your changes.

You break the build.

What? Why? Well, all your new specs depend on the test-spec gem, a gem your comrades and continuous integration builder do not have.

Quickly: fix it! Tell everyone to install the gem locally. Install the gem on your staging server. Carefully install the gem on your production server. Phew. Everyone’s got the same version, right? Right. Well, maybe. (At least the build works.)

See, there’s something wrong with this scenario: it’s not very DRY. Why should only our code be DRY? Why not our environment, too? It should be.

The solution we’ve come up with is to throw every Ruby dependency in vendor. Everything. Savvy? Everyone is always on the same page: we don’t have to worry about who has what version of which gem. (we know) We don’t have to worry about getting everyone to update a gem. (we just do it once) We don’t have to worry about breaking the build with our libraries. (we leave that up to the internet entrepreneurs)

Adding Depended Sees
Alright, alright, let’s see how we’d vendor the test-spec gem. First, the vendor directory:
```
$ ls -1 vendor/
bin
data
gems
plugins
rails
```
We obviously added a few directories, namely gems, data, and bin. The rails and plugins directories should be familiar, one hopes.
Let’s focus on the the vendor/gems directory:
```
$ ls -1 vendor/gems/
RedCloth-3.0.4
RubyInline-3.6.2
crypt-1.1.4
image_science-1.1.1
memcache-client-1.3.0
session-2.4.0
sphinx-0.9.7-rc2
```
We’ll cd in there and then use the handy gem unpack command to, erm, unpack the contents of our test-spec gem:
```
$ gem unpack test-spec
Unpacked gem: 'test-spec-0.3.0'
```
Cool. Now we need to dive into our config/environment.rb file to ensure Rails knows to look in vendor/gems/test-spec-0.3.0/lib when we try to require ‘test/spec’. It’s easier than you think.

Add this to your Rails::Initializer.run block:
```
config.load_paths += Dir["#{RAILS_ROOT}/vendor/gems/**"].map do |dir| 
  File.directory?(lib = "#{dir}/lib") ? lib : dir
end
```
Now all the libraries in vendor/gems will automatically be included in your load path, complete with a lib check for libraries like RubyInline and crypt (which don’t come with lib directories).

Want to only include some libraries in a specific environment? Maybe you don’t want RubyInline or image_science in development mode. Put this under the above snippet o’ code:
```
if %w(development test).include? RAILS_ENV
  config.load_paths.delete_if { |f| f =~ /RubyInline|image_science/ }
end  
```
Good to go.

It should be noted that this trick will not auto-require the gems for you. You still need to do that in your config file, or in your gems, or wherever. Maybe with a line like %w(crypt/blowfish redcloth).each { |f| require f } in your config/environment.rb if it pleases you.

Other Approaches

For one, Dr Nic has something cool: this patch. It lets you run tasks from within vendor/gems right from your RAILS_ROOT. Nifty. He’s also got his gemsonrails plugin which is a similar (but different) approach than we illustrate here.

Jay Fields has his own method for autoloading gems in vendor. Worth a look, and a listen.

Classic Railer topfunky, back in the day, rolled a rake task to scratch the same itch.

Then there’s this thread on Rails-Core about adding some kinda gemy-ness to Core which, unfortunately, hasn’t yet transpired. The gem in question, the one to metaly manage other gems, lives in technoweenie’s repository.

I even do this (for better or worse) with cache_fu, btw. I am starting to really like not being dependent on the environment.

All Set!

The goal here is simple: always get everyone, especially your production environment, on the same page. You don’t want to guess at which gems everyone does and does not have. Right.

There’s another point lurking subtlety in the background: once all your gems are under version control, you can (probably) get your app up and running at any point of its existence without fuss. You can also see, quite easily, which versions of what gems you were using when. A real history.

Nice write-up… thanks.

You never said what /vendor/bin and /vendor/data were for…

Amen.

Great idea! We did start out this way on the project I’m currently most involved in (pretty much exactly as you’ve described, though we chose not to put the version numbers in the gem’s directory name) and it worked well … to a degree.

How do you deal with extension-based gems that need C code to be built?

You can just set GEM_HOME in you config/boot.rb like this: ENV[‘GEM_HOME’] = File.join(RAILS_ROOT, ‘vendor’, ‘gems’)

and it would give you all the advantages without any additional coding.

We use the similar approach only when need to lock down out app to specific (tested) gems versions – http://revolutiononrails.blogspot.com/2007/02/locking-down-deployed-application.html

One problem this creates, is that doing a deploy with capistrano takes ages, because you’re exporting a huge number of files from SVN. One solution to this is to not put them into version control and symlink the vendor/ directory after deploy.

No.

Please don’t. Please for the love of others having to deal with your code, don’t put core dependencies in vendor (not if you control your servers, or your team/company does). Having a little developer communication about the gems that you as a team are going to decide to use is a whole lot better down the line then having dozens of different copies of a gem lying around – when you realistically only need one.

That way when the next “Upgrade your Rails NOW NOW NOW” occurs, you can deal with the upgrades once, and not dozens of times for applications that you’ve probably forgotten that you ever wrote, much less deployed – at old contract jobs left long ago.

mathie: So far it hasn’t been a problem. Our vendor/bin directory houses a subdirectory for each arch (i686-darwin8.8.2, powerpc-darwin8.6.0, x86_64-linux) so we throw some compiled code in there. We have a tmp/inline directory created on deploy for RubyInline, and we don’t actually vendor mongrel—so that pretty much covers it all.

Val: Nice! That’s a neat trick. How would you exclude gems on a per-environment basis, though?

To Chris: The same way you’d normally do with regular gems – you only require those gems unless RAILS_ENV == ‘production’ or but putting requires in corresponding config/environments/*.rb

Ned: Or you can have cap run svn up. One of the big advantages of this method is being able to deploy and roll back gems in case of an emergency. Version control is a huge win here.

Jason, in theory that makes sense, but when you’re working in larger teams and you don’t want to explain to your CSS guy for the fifth time how to install/update their gems, you’ll be happy knowing that all of the dependencies are in vendor.

As far as the slow-going deploys with everything unpacked in vendor, that’s such a huge misnomer. We’re talking about a 5-10 second difference. (At least for our servers)

Hey that looks like a really neat solution. Always glad to see what you cook up here.

What about a Maven approach? Maybe we could have config/gems.yml, and always have this checked to see if all gems and theirs versions where installed.

And have them installed automatically if not there… :)

To Eduardo: We use similar approach @Revolution

Each application or component has its own config defining gem dependencies with versions. It is being used at runtime to load specific gems. An example:

:dependencies:
  - ['rails', '1.1.6']
  - ['json', '~> 0.4']

The same config is used for packaging to a gem so the dependencies are translated to gem dependencies. It means that when you ‘gem install’ a gem created that way, you also pick up all the dependencies declared. Works us for us pretty good.

I’ve been slowly pushing my team in this direction as well. I really really like having as much as possible in version control, and I find that the benefits (time travel, instantly deployed dependencies) far outweigh the costs (wasted disk space, slightly longer checkouts). For applications where the deployment team wants tight control over what gems are installed where, how, etc… or if other trade-offs are in play, I can see the approach that Val uses at Revolution Health being very useful.

Chris, would you mind giving a bit more detail about how you do this for a gem with C extensions (rMagick, for example) and how you handle the bin directory? Do the contents of your bin directory come from gems, or is it only for homegrown compiled binaries?

PJ – I’d hope that the design-o that managed to get Locomotive installed and working and groks subversion (or at least textmate bundles that shell out to svn) well enough to check their CSS changes into a repository could learn to handle gem update/install.

Chris, I do love you man – in that totally platonic developer sort-a-way. The blog post wasn’t all about you – it was mainly for your legion of adoring admirers that don’t think for themselves.

But I agree, CAPS LOCK TEXT IS LOVE :-)

I was wondering about compiled binaries as well. Do you have an easy way to recompile between platforms?

+1 for the “giving a bit more detail about how you do this for a gem with C extensions (rMagick, for example) and how you handle the bin directory”

With a web designer/CSS guy on Windows, developers on Intel and PPC Macs, build, test, and production servers on Linux, this is a non-trivial issue for our team.

I’m totally for the vendor everything approach, but

gem unpack

just copies the source files and doesn’t build anything. As Lori mentions, some gems with C extensions need to be compiled first, such as ruby-xslt. In this case you still have to run

gem install

I guess one solution is to use

gem install

with the

-i

option for your vendor/gems directory. The load paths would have to be changed to reflect this.

I wrote a simple Rails plugin to build your C extensions in the vendor everything approach. You’ll prolly want include it into your Capistrano workflow, although that is left as an exercise for the user ;).

http://svn.kylemaxwell.com/rails_plugins/vendor_everything_extensions/

Usage: rake vendor:build_extensions

I know I’m coming to this party late, but I thought I’d throw this in the fridge in case anyone’s still thirsty.

lovin this !

a correct to Val’s otherwise great solution: “GEM_HOME for writing, GEM_PATH for reading!” from http://wiki.rubyonrails.org/rails/pages/HowToUseMultipleGemRepositories

so you’ll want to: ENV[‘GEM_PATH’] = File.join(RAILS_ROOT, ‘vendor’, ‘gems’)

(not gem home as suggested).

cheers,Jodi

Ned/Chris/PJ: the speed of your deploys shouldn’t be a factor once you move over to Capistrano 2.0, because we’ve got this now:

set :deploy_via, :remote_cache

This creates a cached copy of your code in the shared directory, and just runs an ‘svn up’ on that when you deploy. This code then gets copied to the releases directory as usual. Nice and speedy.

How did you get the vendor gems directory? I don’t have that in my vendor directory.

Chris, what about \vendor\data what’s that for? you talked about \vendor\bin

I’m trying to get this set up with a Rails 2.0 based application.

Any updates on this approach. I’m probably missing something obvious, but while the application works fine via ruby script/server I can’t run any rake tasks (fail looking for gems used by rake tasks). I am guessing I have to set the load path somewhere, but don’t know where. I had hoped that the config/environment.rb settings would get picked up and it would just work, but doesn’t seem to be the case.

Any ideas?

Thanks.

For RMagick we just copied the necessary file(s) into the vendor/gem/rmagick… directory. There is a RMagick2.so (Rmagick2.bundle on Mac) file that is required.

Same deal with our Sybase adapter.

Good to go. So far.

The best article, thanks to the author. good jokes, funny jokes

Tattoos Celtic Tattoos Lower Back Tattoos Dragon Tattoos Skull Tattoos Heart Tattoos Butterfly Tattoos Chinese Tattoos Layered Hair Styles Layered Hairstyles Emo Hairstyles Medium Hairstyles Bang Hairstyle Prom Hairstyle Hair Coloring Heart Lower Back Tattoos

How to download and convert the FLV files on Mac OS? Here are the easy-to-use and top-user-rating [url=http://www.downloadyoutubeformac.com/free-youtube-downloader-for-mac.html]Free YouTube Downloader for Mac[/url] and FLV converter tools. It is safe to install and there are no spyware or adware.

How to download YouTube videos for mac and convert the .FLV files on Mac OS? Here are the easy-to-use and top-user-rating YouTube Video Grabber Mac and Mac FLV converter tools. It is safe to install and no spyware or adware guarenteed.

With the YouTube Converter for Mac application, you can easily download youtube for mac and convert YouTube videos on your Mac OS.

I suggest 2 FLV Converter for Mac software. One is powerful in download YouTube Videos and convert downloaded FLV files; the other is professional in video converting, which can not only convert .flv video to all media formats, but also can convert all kinds of video formats to .flv, plus popular videos.

For anyone with old or elderly loved ones, it is important to take care of your care for the elderly at home residential care facility for the elderly residential care facilities for the elderly care for the elderly at home nursing care for the elderly residential care for the elderly care services for the elderly social care for the elderly care homes for the elderly care home for the elderly community care for the elderly in hopes making sure you or your old / elderly loved ones are taken care of with the best nursing care available.