This week we upgraded a couple of our applications to Ruby 2.7 and Bundler 2.1.4 and one of the changes that we noticed was that Bundler was complaining about not being able to write to the /opt/local directory.
Turns out this problem shows up because the account that we use to run our application is a system account that does not have a home folder.
This is how the problems shows up:
$ su - system_account $ pwd /opt/local $ mkdir test_app $ cd test_app $ pwd /opt/local/test_app $ gem install bundler -v 2.1.4 $ bundler --version `/opt/local` is not writable. Bundler will use `/tmp/bundler20200731-59360-174h3lz59360' as your home directory temporarily. Bundler version 2.1.4
Notice that Bundler complains about the /opt/local directory not being writable, that’s because we don’t have home for this user, in fact env $HOME outputs /opt/local rather than the typical /home/username.
Although Bundler is smart enough to use a temporary folder instead and continue, the net result of this is that if we set a configuration value for Bundler in one execution and try to use that configuration value in the next execution Bundler won’t be able to find the value that we set in the first execution (my guess is because the value was saved in a temporary folder.)
Below is an example of this. Notice how we set the path value to vendor/bundle in the first command, but then when we inspect the configuration in the second command the configuration does not report the value that we just set:
# First - set the path value
$ bundle config set path 'vendor/bundle'
`/opt/local` is not writable.
Bundler will use `/tmp/bundler20200731-60203-16okmcg60203' as your home directory temporarily.
# Then - inspect the configuration
$ bundle config
`/opt/local` is not writable.
Bundler will use `/tmp/bundler20200731-60292-1r50oed60292' as your home directory temporarily.
Settings are listed in order of priority. The top value will be used.
Ideally the call to bundle config will report the vendor/bundle path that we set, but it does not in this case. In fact if we run bundle install next Bundler will install the gems in $GEM_PATH rather than using the custom vendor/bundle directory that we indicated.
Working around the issue
One way to work around this issue is to tell Bundler that the HOME directory is the one from where we are running bundler (i.e. /opt/local/test_app) in our case.
# First - set the path value # (no warning is reported) $ HOME=/opt/local/test_app/ bundle config set path 'vendor/bundle' # Then - inspect the configuration $ bundle config `/opt/local` is not writable. Bundler will use `/tmp/bundler20200731-63230-11dmgcb63230' as your home directory temporarily. Settings are listed in order of priority. The top value will be used. path Set for your local app (/opt/local/test_app/.bundle/config): "vendor/bundle"
Notice that we didn’t get a warning in the first command (since we indicated a HOME directory) and then, even though we didn’t pass a HOME directory to the second command, our value was picked up and shows the correct value for the path setting (vendor/bundle).
So it seems to me that when HOME is set to a non-writable directory (/opt/local in our case) Bundler picks up the values from ./bundle/config if it is available even as it complains about /opt/local not being writable.
If we were to run bundle install now it will install the gems in our local vendor/bundle directory. This is good for us, Bundler is using the value that we configured for the path setting (even though it still complains that it cannot write to /opt/local.)
We could avoid the warning in the second command if we pass the HOME value here too:
$ HOME=/opt/local/test-app/ bundle config Settings are listed in order of priority. The top value will be used. path Set for your local app (/opt/local/test-app/.bundle/config): "vendor/bundle"
But the fact the Bundler picks up the correct values from ./bundle/config when HOME is set to a non-writable directory was important for us because it meant that when the app runs under Apache/Passenger it will also work. This is more or less how the configuration for our apps in http.conf looks like, notice that we are not setting the HOME value.
<Location /> PassengerBaseURI /test-app PassengerUser system_account PassengerRuby /opt/local/rubies/ruby-2.7.1/bin/ruby PassengerAppRoot /opt/local/test-app SetEnv GEM_PATH /opt/local/.gem/ruby/2.7.1/ </Location>
Some final thoughts
Perhaps a better solution would be to set a HOME directory for our system_account, but we have not tried that, we didn’t want to make such a wide reaching change to our environment just to please Bundler. Plus this might be problematic in our development servers where we share the same system_account for multiple applications (this is not a problem in our production servers)
We have no idea when this change took effect in Bundler. We went from Bundler 1.17.1 (released in October/2018) to Bundler 2.1.4 (released in January/2020) and there were many releases in between. Perhaps this was documented somewhere and we missed it.
In our particular situation we noticed this issue because one of our gems needed very specific parameters to be built during bundle install. We set those values via a call to bundle config build.mysql2 --with-mysql-dir=xxx mysql-lib=yyy and those values were lost by the time we ran bundle install and the installation kept failing. Luckily we found a work around and were able to install the gem with the specific parameters.