Or said in another way: How to run openSUSE on Hetzner dedicated servers.

TL;DR:

1. Install any other linux distribution from their installer.
2. Follow the SDB article about remote installation
3. Install openSUSE via SSH.

Why not build an image for their installimage tool? I have that too. But last time I used that, the install tool logged more things to their logging server than I liked. And it did so via plain HTTP.

This whole blog post was born from the idea to clarify how to nicely force a newer compiler on older distros. But of course we have to start with the basics.

RPM conditionals basics

RPM supports two main forms of conditionals %if and %ifarch. The last one is easy … it lets us do architecture specific things:

%ifarch x86_64 aarch64
[snip]
%endif

The %if is a bit more flexible. We can use it to compare strings.

It is actually relatively simple

1. install the php imagick extension with libheif support
2. enable the preview providers

For openSUSE Tumbleweed the php8-imagick package is already built with libheif support. Though the library in the distribution lacks H.256 support. We can easily remedy that situation by using the libheif1 package from packman instead.

# cat /etc/zypp/repos.d/isv-packman.repo
[isv-packman]
enabled=1
autorefresh=1
baseurl=https://ftp.halifax.rwth-aachen.de/packman/suse/openSUSE_Tumbleweed/
type=rpm-md
gpgcheck=1

Then we can install the new libheif1 with

zypper in --from isv-packman libheif1

Afterwards you have to restart Apache or php-fpm depending on what you use.

So the OBS supports packaging Debian packages for a while now. Is it perfect? No. But it works well enough. Though one problem I recently ran into …

How to get past lintian?

The obvious answer is of course: Fix the errors it spits out. And I agree. I have done that. For all but one …

bad-distribution-in-changes-file

For many of the debian packages, I helped to start, we used:

pkgname (1.2.3-1.1) unstable; urgency=medium

Back when i started using Let’s Encrypt I wanted a lightweight client. Danimo recommended dehydrated. Well it still had a different name back then. But details. So for years I had been a very happy camper with it. Then I learned that many daemons now support that you can run ECDSA and RSA certs at the same time and the server picks the right cert depending on the client. So we will solve this first.

TL;DR

Operating systems let you customize the sensitivity of the microphone allowing you to suppress environmental noise such as keyboard strokes. Video conferences software unfortunately tends to override such settings using autogain making your customization void. While some video conferencing tools let you control autogain via the settings, Jitsi does not have such an option.

However, you can use https://your.jitsi.server/someroom#config.disableAGC=true when joining your Jitsi meetings to disable autogain and have more control over your microphone.

Usually your AppArmor profile would look something like this:

#include <tunables/global>

/usr/lib/colord {
  ...
}

Now if we check if our profile is actually used:

$ ps axZ | grep '^/usr/lib/colord'
/usr/lib/colord (enforce)  4466  ?  Ssl  0:00 /usr/lib/colord

The Z parameter makes ps print the AppArmor profile name, or in the case of selinux the context of the program. /usr/lib/colord (enforce) tells us what AppArmor profile is used and in which mode it is. Another option you can see there is complain. With long binary paths this can become ugly very quickly.

I haven’t been using apache for a few years now … oh wow maybe like 15 years now.

There was one feature I really liked with apache and apparmor though … mod_changehat. The module allows me to assign different apparmor scopes to apache scopes. So you could limit that your wordpress vhost can not access the files of your nextcloud vhost even though they are running in the same apache.

So I am working on a Nextcloud package at the moment. Something that does not have such brain dead default permissions as the docs give you. To give all users who need access to the files, I used ACLs. This worked quite fine. Until during one update … suddenly all my files got executable permissions.

I ran rpm --setperms nextcloud to reset the permissions of all files and then I reran my nextcloud-fix-permissions. Nothing. All was good again. During the next update … broken again. I fixed the permissions manually by removing the x bit and then rerunning my script … bugged again. Ok time to dig.

When you are hear 2 coworkers complaining about breaking each other’s changes to a package, you wonder what is going on. And this time it turns out they were complaining about a rubygem package. So yeah time to find a nice solution to this problem finally.

The conflict

rubygem-annotate and gd. Both provide a /usr/bin/annotate. In the case of our gem based packaging this is of course only a symlink handled by update-alternative. Though the code handling this is split over 2 places - the script that handles installing each gem and our template for the subpackages.

We already mentioned Instantiated Services a few times. However, there is one question we did not yet cover:

How to hook up those instances with our package manager

For a normal service file this is pretty easy. Such a service can be handled as follow in the spec file:

%pre
%service_add_pre superduper.service

%post
%service_add_post superduper.service

%preun
%service_del_preun superduper.service

%postun
%service_del_postun superduper.service

That is it. It will do all the things we want.

After all the fun we had in part 1 of our FreeIPA + DNS series, we now come to a new chapter.

During the move of openSUSE to Let’s Encrypt we used the excellent tool from ssllabs to verify each server after we changed the certificate. One of our coworkers noted “Hey, why don’t you have a CAA record. It would give you an even better score”. CAA records allow the domain owner to specify which certificate authorities are allowed to create certificates for this domain. Which seems to be a nice benefit for little work. For a longer explanation see this blog post from Qualys.

Requests are one of the staples for collaboration in the OBS. You can review via the webui or with OSC.

WebUI

Lets take the request listing for openSUSE:Factory. Your normal workflow will probably end up as

1. middle mouse click on the little magnifying glass icon on the right.
2. review the request in a new tab
3. close the new tab
4. go back step 1

My issues with the WebUI

1. no advancing to the next request in my current list.
2. I have to manually unfold/fold many diff chunks for a proper review.
3. In the latest version of the WebUI: “We truncated the diff of some files because they were too big. If you want to see the full diff for every file, click here.” But even then I have to unfold every change myself again.

OSC - The normal way

1. one terminal: osc rq list -t submit -s new openSUSE:Factory
2. 2nd terminal: osc rq show -d ID - the ID is taken from the first listing.
3. 2nd or in worst case 3rd terminal: osc rq youraction ID
4. go back to step 2

My issues

1. all manually copy pasting of IDs

OSC interactive mode

My config:

One step at a time. Today: Requires

You might have seen that we package rails apps for a while now. We have discourse, gitlab-ce, errbit and our own open build service. I probably forgot an app or 2 here. Anyway if you look at all the spec files e.g. this one. You notice we have to specify the Requires manually.

BuildRequires:  %{rubygem rails:5 >= 5.1}
Requires:       %{rubygem rails:5 >= 5.1}

This comes with a few problems actually.

Restic had been on my watch list for a while, but i never really got around to play with it.

After Jan-Piet Mens blogged about it [1, 2], I thought it was finally time to give it a go. Of course this means, first we need to package it.

This is done now. You can find the packages in home:darix:restic. After reading the first blog post I did restic and rest-server. Happy camper I was until i noticed to next post. Minio was another item on my “Lets play with that” list. So one stone, two birds. Added 2 packages to my collection, because you know just packaging the server without the client is a no go.

I have been a haproxy user for quite a few years now. Even using snapshots in production for a very long time. Especially after support was added to terminate SSL connections directly in haproxy. Getting rid of stunnel was so nice.

For a very long time i was doing really well with this setup. But over time more and more services were put behind haproxy and the connections/s and the total amount of connections went up. We started to see some performance issues. Which at first sounds weird … if you look at the benchmarks on the haproxy website they can do thousands if not hundred thousands of connections per seconds.

The Plan

+---------+    +------------------------+    +---------------------------+
| FreeIPA | -> | upstream hidden master | -> | public facing dns servers |
+---------+    +------------------------+    +---------------------------+

Sounds simple enough right? Well …

The Fun

Let’s get right too it … FreeIPA only sends out notifications to the NS records listed in the zone. But our hidden master is not reachable from the outside and should not be listed as an NS.

‘But bind has “also-notify” just use that.’ you might say now. Which is correct. So a quick check on the ldap scheme reveals there is no setting in the LDAP tree for it. Ok… the nice solution is dead.

As i regularly look it up from my mailbox. lets post it here.

osc co yourproject/yourpackage
cd yourproject/yourpackage
quilt setup -v *spec
cd yourpackage-*/
quilt push -a # apply old patches
quilt new yourpackage-version_fixbuild.patch
quilt edit src/foo.c
quilt refresh

foo-fixbuild.patch will automatically be created in the parent dir. if you work on a package which doesnt have a patch yet. you have to remember to copy the patch from the patch directory to your package directory.

A very simple frontend/backend definition in haproxy can look like this:

frontend http
  bind 0.0.0.0:80    tfo
  bind 0.0.0.0:443   tfo ssl npn http/1.1,http/1.0 no-sslv3 crt /etc/ssl/services/
  default_backend somebackend

backend somebackend
  option forwardfor
  option httpchk HEAD /check.txt HTTP/1.1\r\nHost:\ somehost
  server somebackend1 172.16.0.1 check port 80

$ curl -I http://haproxy.example.com/

Works.

$ curl -I https://haproxy.example.com/

Does not work.

The log snippet also looks ok:

http~ somebackend/somebackend1 0/3004/-1/-1/3005 503 3090 - - SC-- 529/126/0/0/3 0/0 "GET / HTTP/1.1"

But why does it report 503?

By default the epmd on (open)SUSE only listens on 127.0.0.1. Which in itself is a sane default policy. We don’t expose any services directly to everyone just because we install and run them. But if you want to cluster erlang services you need it to listen on more addresses.

$ export EDITOR=vim
$ systemctl edit epmd.socket

Add the following content:

[Socket]
# unset all ports defined in the global file, in our case this is 127.0.0.1:4369
ListenStream=
# add our new ports
ListenStream=[::]:4369

Then run:

If you ever discussed packaging in person with me, you might have noticed i normally don’t call “osc build” but a little shell thingie. The reason is i don’t want to maintain a long command line all the time.

So in my shell RC file i have the following:

osc_build() {
  pkg_dir=$1
  shift
  repo=$1
  shift
  osc build -x vim -k ../rpms/$pkg_dir -p ../rpms/$pkg_dir $repo x86_64 $@
}

ob-15.1() {
  osc_build 15.1 openSUSE_Leap_15.1 $@
}

ob-15.2() {
  osc_build 15.0 openSUSE_Leap_15.0 $@
}

ob-tw() {
  osc_build tw openSUSE_Tumbleweed $@
}
ob-twf() {
  osc_build tw openSUSE_Factory $@
}

-x vim

So i can edit inside the build environment. There is actually really a nice helper to get into the build env:

FFI is a method to interface with native libraries, which is becoming more and more popular in many scripting languages. Unlike with native extensions though, we have nothing that links the shared library. As such our requires generator for shared libraries doesn’t kick in and we get no requires on the shared library package.

To make matters worse with the shared library policy the soversion is dependent on against which distro you build against.

Disclaimer: I am aware that python does not allow easy installation of 2 libraries in parallel. It is just an example.

Let’s say you have an upstream requirement "needs pyfoo >= 1.6.0 but smaller than 2". Sounds easy right? Our naive solution will be:

Requires: python-pyfoo >= 1.6.0
Requires: python-pyfoo < 2

so far … so good. If your repository has only one version of pyfoo available we will probably get the package you want. now lets say we have more than one version available.

Since we have staging for Factory/Tumbleweed, it became surprisingly stable. Major breakages at least for core packages are really rare now. But even for leaf packages it is not a common sight.

One thing was still annoying: for upgrading you should use zypper dist-upgrade or short zypper dup normally. zypper dup will upgrade all the packages to the latest version it finds but also downgrade/uninstall packages, if required. Something a zypper up will not do. This is all fine when you only use the core distribution repositories. If you start using e.g. packman or any other repository from the build service, zypper did switch around the package to the distribution package and a few days later back to the OBS based repository. Depending on what was newer. It is allowed to do that because zypper dup allows also vendor changes.

After I was asked again today how to properly patch a rubygems package, here is a quick how to.

First things first we branch the package.

darix@tengu ~/osc/tmp % osc bco devel:languages:ruby:extensions/rubygem-net-ssh-session
A    home:darix:branches:devel:languages:ruby:extensions
A    home:darix:branches:devel:languages:ruby:extensions/rubygem-net-ssh-session
A    home:darix:branches:devel:languages:ruby:extensions/rubygem-net-ssh-session/net-ssh-session-0.1.6.gem
A    home:darix:branches:devel:languages:ruby:extensions/rubygem-net-ssh-session/rubygem-net-ssh-session.changes
A    home:darix:branches:devel:languages:ruby:extensions/rubygem-net-ssh-session/rubygem-net-ssh-session.spec
At revision 9edd872bc9121e1c68a71d057dfa607f.
Note: You can use "osc delete" or "osc submitpac" when done.

darix@tengu ~/osc/tmp % cd home:darix:branches:devel:languages:ruby:extensions/rubygem-net-ssh-session

So far nothing unexpected a gem file and spec+changes.

darix@tengu rubygem-net-ssh-session % l
total 28
drwxr-xr-x 3 darix users   122 Jul 23 12:21 ./
drwxr-xr-x 4 darix users    47 Jul 23 12:21 ../
-rw-r--r-- 1 darix users 13824 Jul 23 09:54 net-ssh-session-0.1.6.gem
drwxr-xr-x 2 darix users  4096 Jul 23 12:21 .osc/
-rw-r--r-- 1 darix users   143 Jul 23 10:09 rubygem-net-ssh-session.changes
-rw-r--r-- 1 darix users  1472 Jul 23 09:54 rubygem-net-ssh-session.spec

I prefer quilt for my patch handling but quilt setup doesn’t know how to handle our spec files yet. So some manual work required. This btw is also the reason why I include the series file in the source package.

This is just a quick post but imho still pretty neat. If you are a postgresql user you probably know about the commands with \ to inspect your database server e.g. .

mirrorbrain=> \l mirro*
                                  List of databases
    Name     |    Owner    | Encoding |   Collate   |    Ctype    | Access privileges 
-------------+-------------+----------+-------------+-------------+-------------------
 mirrorbrain | mirrorbrain | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
(1 row)

mirrorbrain=> \dt
           List of relations
 Schema |  Name   | Type  |    Owner    
--------+---------+-------+-------------
 public | country | table | mirrorbrain
 public | file    | table | mirrorbrain
 public | hash    | table | mirrorbrain
 public | marker  | table | mirrorbrain
 public | mirror  | table | mirrorbrain
 public | pfx2asn | table | mirrorbrain
 public | region  | table | mirrorbrain
 public | server  | table | mirrorbrain
 public | version | table | mirrorbrain
(9 rows)

mirrorbrain=> 

So far … nothing new … right?

In the good old days of sysvinit, if you wanted to run some extra code during start up, you were either (ab)using the sysconfig file or patching the init script itself. The init scripts in many packages are not as marked as configuration, which means in those cases your edits were lost. If it was marked as configuration then you have to merge your changes with those fixes manually. In either case … not ideal.

Getting started

Let’s say you want to have 3 instances of redis. One for each redis using app. Getting 3 instances app with systemd’s nifty Instantiated Services is easy.

[Unit]
Description=Redis
After=network.target
PartOf=redis.target

[Service]
Type=simple
User=redis
Group=redis
PrivateTmp=true
PIDFile=/var/run/redis/%i.pid
ExecStart=/usr/sbin/redis-server /etc/redis/%i.conf
Restart=on-failure

[Install]
WantedBy=multi-user.target redis.target

“systemctl start redis@discourse” will launch us an instance with “discourse.conf”. Other systemctl commands just work as fine. “systemctl enable redis@discourse” , “systemctl status redis@discourse” . Now to stop all redis instances? “systemctl stop redis.target”

Especially when using irssi via your smartphone it can be highly annoying to switch windows. Yes the /win $number thing will work. But who can remember all those numbers when you go past 20 windows? That’s where window_switcher.pl comes into play.

Setup

First download the file and place it into ~/.irssi/scripts/autorun/. Then switch back to irssi and set up window_switcher. We will bind ctrl+o to window_switcher. The status bar item is placed in the front of the status bar, so we an still see it on small displays.

General

Official Install Documentation

Repositories

1. use the repositories from devel:languages:ruby and home:darix:apps
2. zypper in –from home:darix:apps redis gitlab-ce gitlab-shell

config redis:

See also: /usr/share/doc/packages/redis/README.SUSE

$ cd /etc/redis
$ cp default.conf.example gitlab.conf
$ vi gitlab.conf

* daemonize no
* pidfile /var/run/redis/gitlab.pid
* logfile /var/log/redis/gitlab.log
* dir /var/lib/redis/gitlab/

$ install -d -m 0750 -o redis -g redis /var/lib/redis/gitlab/
$ chown root:redis gitlab.conf
$ sc start redis@gitlab
$ sc enable redis@gitlab

GitLab Shell

1. cd /usr/share/gitlab/shell/; cp config.yml{.example,}
2. set up redis socket matching /etc/redis/gitlab.conf
3. maybe enable audit_usernames (but see warning)

config rails app

1. cd /srv/www/vhosts/gitlab-ce/
2. configure database.yml (based on postgresql example)
3. cp config/gitlab.yml.{example,}

• host, port, https, email_from in gitlab section
• optionally ldap settings

4. cp config/resque.yml.{example,}

• adapt socket for redis. should match /etc/redis/gitlab.conf

5. export RAILS_ENV="${RAILS_ENV:=production}"
6. gitlab-ce-update
7. rake db:seed_fu
8. check for new files which are now owned by root:root (e.g. .gitlab_shell_secret) they should be owned by root:gitlab with permissions u=rw,g=r,o=
9. make sure you have the symlink .gitlab_shell_secret /srv/www/vhosts/gitlab-ce/.gitlab_shell_secret
10. Configure git for gitlab user as seen in official docs
11. sc start gitlab-ce-unicorn gitlab-ce-sidekiq gitlab-workhorse
12. sc enable gitlab-ce-unicorn gitlab-ce-sidekiq gitlab-workhorse

Testing things

1. ssh access

ssh -T gitlab@gitlab.suse.de
Welcome to GitLab, $yourusername!