Making Rails packaging easier - Part1

One step at a time. Today: Requires

You might have seen that we package rails apps for a while now. We have discourse, gitlab-ce, errbit and our own open build service. I probably forgot an app or 2 here. Anyway if you look at all the spec files e.g. this one. You notice we have to specify the Requires manually.

BuildRequires:  %{rubygem rails:5 >= 5.1}
Requires:       %{rubygem rails:5 >= 5.1}

This comes with a few problems actually.

1. our rpm allows updating the rails gem from 5.1.1 to 5.1.2, but our rails app will definitely complain about as the version is more tight in the Gemfile.lock
2. We have to make sure that those 2 lines are kept in sync. In the way the spec file is written currently, this is not much of a problem. But if you would apply spec-cleaner to the file it would group up all BuildRequires and further down in the file all Requires. Then it would get easy to forget this.

So lets fix that!

Fixing it

The OBS team actually had a solution already. They have a little find-requires.sh in their package, which they call in obs-server.spec. While this is a step into the right direction, it has the ruby version hardcoded for their needs and rpm actually has a better way to hook it up in the spec file. If you look at our ruby-common you might have noticed the rubygems.attr file and its companion rubygemsdeps.rb. The rubygems.attr file tells rpm which files we actually want to process with rubygemsdeps.rb. The ruby script then does the heavy lifting and generates Requires and Provides using the rubygems library. So something like that would be nice.

First step is to learn how we can ask the bundler API to give us the list of all required gems. We actually have 2 questions for bundler. Give us everything that is just in the “Gemfile”. And as second step: Give me all the gems you actually used for the “Gemfile.lock”. During my tinkering Henne pointed me to their find-requires.sh script and that helped with part 2. But at that moment I was still fighting with just parsing “Gemfile”. More on this later.

So now we had some basic code to get the names of all used gems + their versions the next question was … how do we get the ruby abi information to our requires generator? Setting environment variables in %install does not work. So I settled for generating a small “.appinfo.yml” file in the app directory. This actually solved a 2nd problem for me “How to handle multiple Gemfile.lock?”. No ‘.appinfo.yml’ means i can ignore that “Gemfile.lock”. Neat!

So i hacked up some dummy code in the test package to generate the file and voila it worked!

But hacking this code in reminded me of a long standing issue … there is alot of duplicated code between all those spec files. Time to clean that up while we are at it. One code block after the other ended up in /etc/rpm/macros.rails. I see this file growing as we simplify the rails packaging more.

At the moment we have 4 macros ready for use:

1. %rails_save_gemfile should be called in %prep directly after unpacking the tarball so we can see what we changed exactly with patches and after regenerating the Gemfile.lock.
2. %rails_regen_gemfile_lock should be called in %build and does exactly what the name says.
3. %rails_write_appinfo should be called in %build and will write our small .appinfo.yml for us.
4. %rails_fix_ruby_suffix should be called in %build and will patch all the service files and helper scripts with the correct ruby suffix. It will walk over all files in the rpm SOURCES dir that contain @RUBY_SUFFIX@ and replace it with the value of the %{rb_suffix}. We need something similar to fix all the shebang lines in scripts that come with the rails apps.

So with all those things in place we end up with this discourse.spec. I applied the same to errbit and gitlab-ce.

After I stuffed everything into our ruby-common package, I noticed some gems started building for ruby 2.1 again, even if they shouldn’t. This happened as I had ruby-common require bundler now and in the default case it meant the system version of bundler was pulled. All the rails specific things are now in a ruby-common-rails sub package. You have to BuildRequire that in your rails app.

What’s next

So I already wrote a script named “bundler-dumpdeps” to generate the “BuildRequires” lines. But I need to clean that up a bit and find out how to convince “bundler install –local –without=‘development test’” to actually skip those 2 groups. Right now we always need to patch the Gemfile to comment them out. But that is for the next round.