Notes on installing openSUSE remotely
Or said in another way: How to run openSUSE on Hetzner dedicated servers.
TL;DR:
- Install any other linux distribution from their installer.
- Follow the SDB article about remote installation
- Install openSUSE via SSH.
Why not build an image for their installimage tool? I have that too. But last time I used that, the install tool logged more things to their logging server than I liked. And it did so via plain HTTP.
…RPM Spec files conditionals and forcing package versions
This whole blog post was born from the idea to clarify how to nicely force a newer compiler on older distros. But of course we have to start with the basics.
RPM conditionals basics
RPM supports two main forms of conditionals %if
and %ifarch
. The last one is easy … it lets us do architecture specific things:
%ifarch x86_64 aarch64
[snip]
%endif
The %if
is a bit more flexible. We can use it to compare strings.
Enabling HEIF support in Nextcloud
It is actually relatively simple
- install the php imagick extension with libheif support
- enable the preview providers
For openSUSE Tumbleweed the php8-imagick package is already built with libheif support. Though the library in the distribution lacks H.256 support.
We can easily remedy that situation by using the libheif1
package from packman instead.
# cat /etc/zypp/repos.d/isv-packman.repo
[isv-packman]
enabled=1
autorefresh=1
baseurl=https://ftp.halifax.rwth-aachen.de/packman/suse/openSUSE_Tumbleweed/
type=rpm-md
gpgcheck=1
Then we can install the new libheif1
with
zypper in --from isv-packman libheif1
Afterwards you have to restart Apache or php-fpm depending on what you use.
…The Debian changelog and OBS
So the OBS supports packaging Debian packages for a while now. Is it perfect? No. But it works well enough. Though one problem I recently ran into …
How to get past lintian?
The obvious answer is of course: Fix the errors it spits out. And I agree. I have done that. For all but one …
bad-distribution-in-changes-file
For many of the debian packages, I helped to start, we used:
pkgname (1.2.3-1.1) unstable; urgency=medium
Advanced dehydration
Back when i started using Let’s Encrypt I wanted a lightweight client. Danimo recommended dehydrated. Well it still had a different name back then. But details. So for years I had been a very happy camper with it. Then I learned that many daemons now support that you can run ECDSA and RSA certs at the same time and the server picks the right cert depending on the client. So we will solve this first.
…Jitsi: Turning off Automatic Gain Control
TL;DR
Operating systems let you customize the sensitivity of the microphone allowing you to suppress environmental noise such as keyboard strokes. Video conferences software unfortunately tends to override such settings using autogain making your customization void. While some video conferencing tools let you control autogain via the settings, Jitsi does not have such an option.
However, you can use https://your.jitsi.server/someroom#config.disableAGC=true
when joining your Jitsi meetings to disable autogain and have more control over your microphone.
Nicer AppArmor profiles
Usually your AppArmor profile would look something like this:
#include <tunables/global>
/usr/lib/colord {
...
}
Now if we check if our profile is actually used:
$ ps axZ | grep '^/usr/lib/colord'
/usr/lib/colord (enforce) 4466 ? Ssl 0:00 /usr/lib/colord
The Z
parameter makes ps
print the AppArmor profile name, or in the case of selinux the context of the program.
/usr/lib/colord (enforce)
tells us what AppArmor profile is used and in which mode it is.
Another option you can see there is complain
. With long binary paths this can become ugly very quickly.
PHP FPM Apparmor
I haven’t been using apache for a few years now … oh wow maybe like 15 years now.
There was one feature I really liked with apache and apparmor though … mod_changehat
.
The module allows me to assign different apparmor scopes to apache scopes. So
you could limit that your wordpress vhost can not access the files of your
nextcloud vhost even though they are running in the same apache.
Linux ACL Masks Fun
So I am working on a Nextcloud package at the moment. Something that does not have such brain dead default permissions as the docs give you. To give all users who need access to the files, I used ACLs. This worked quite fine. Until during one update … suddenly all my files got executable permissions.
I ran rpm --setperms nextcloud
to reset the permissions of all files and then I reran my nextcloud-fix-permissions
. Nothing. All was good again. During the next update … broken again. I fixed the permissions manually by removing the x bit and then rerunning my script … bugged again. Ok time to dig.
Resolving a conflict
When you are hear 2 coworkers complaining about breaking each other’s changes to a package, you wonder what is going on. And this time it turns out they were complaining about a rubygem package. So yeah time to find a nice solution to this problem finally.
The conflict
rubygem-annotate and gd. Both provide a /usr/bin/annotate
. In the case of our gem based packaging this is of course only a symlink handled by update-alternative
.
Though the code handling this is split over 2 places - the script that handles installing each gem and our template for the subpackages.
Hooking up instantiated services with RPM
We already mentioned Instantiated Services a few times. However, there is one question we did not yet cover:
How to hook up those instances with our package manager
For a normal service file this is pretty easy. Such a service can be handled as follow in the spec file:
%pre
%service_add_pre superduper.service
%post
%service_add_post superduper.service
%preun
%service_del_preun superduper.service
%postun
%service_del_postun superduper.service
That is it. It will do all the things we want.
…More fun with Freeipa and DNS: Funky records
After all the fun we had in part 1 of our FreeIPA + DNS series, we now come to a new chapter.
During the move of openSUSE to Let’s Encrypt we used the excellent tool from ssllabs to verify each server after we changed the certificate. One of our coworkers noted “Hey, why don’t you have a CAA record. It would give you an even better score”. CAA records allow the domain owner to specify which certificate authorities are allowed to create certificates for this domain. Which seems to be a nice benefit for little work. For a longer explanation see this blog post from Qualys.
…OSC interactive review
Requests are one of the staples for collaboration in the OBS. You can review via the webui or with OSC.
WebUI
Lets take the request listing for openSUSE:Factory. Your normal workflow will probably end up as
- middle mouse click on the little magnifying glass icon on the right.
- review the request in a new tab
- close the new tab
- go back step 1
My issues with the WebUI
- no advancing to the next request in my current list.
- I have to manually unfold/fold many diff chunks for a proper review.
- In the latest version of the WebUI: “We truncated the diff of some files because they were too big. If you want to see the full diff for every file, click here.” But even then I have to unfold every change myself again.
OSC - The normal way
- one terminal:
osc rq list -t submit -s new openSUSE:Factory
- 2nd terminal:
osc rq show -d ID
- the ID is taken from the first listing. - 2nd or in worst case 3rd terminal:
osc rq youraction ID
- go back to step 2
My issues
- all manually copy pasting of IDs
OSC interactive mode
My config:
…Making Rails packaging easier - Part1
One step at a time. Today: Requires
You might have seen that we package rails apps for a while now. We have discourse, gitlab-ce, errbit and our own open build service. I probably forgot an app or 2 here. Anyway if you look at all the spec files e.g. this one. You notice we have to specify the Requires manually.
BuildRequires: %{rubygem rails:5 >= 5.1}
Requires: %{rubygem rails:5 >= 5.1}
This comes with a few problems actually.
…Restic
Restic had been on my watch list for a while, but i never really got around to play with it.
After Jan-Piet Mens blogged about it [1, 2], I thought it was finally time to give it a go. Of course this means, first we need to package it.
This is done now. You can find the packages in home:darix:restic. After reading the first blog post I did restic and rest-server. Happy camper I was until i noticed to next post. Minio was another item on my “Lets play with that” list. So one stone, two birds. Added 2 packages to my collection, because you know just packaging the server without the client is a no go.
…HAProxy - Speeding up SSL
I have been a haproxy user for quite a few years now. Even using snapshots in production for a very long time. Especially after support was added to terminate SSL connections directly in haproxy. Getting rid of stunnel was so nice.
For a very long time i was doing really well with this setup. But over time more and more services were put behind haproxy and the connections/s and the total amount of connections went up. We started to see some performance issues. Which at first sounds weird … if you look at the benchmarks on the haproxy website they can do thousands if not hundred thousands of connections per seconds.
…Fun with FreeIPA and a slightly more complex DNS setup
The Plan
+---------+ +------------------------+ +---------------------------+
| FreeIPA | -> | upstream hidden master | -> | public facing dns servers |
+---------+ +------------------------+ +---------------------------+
Sounds simple enough right? Well …
The Fun
Let’s get right too it … FreeIPA only sends out notifications to the NS records listed in the zone. But our hidden master is not reachable from the outside and should not be listed as an NS.
‘But bind has “also-notify” just use that.’ you might say now. Which is correct. So a quick check on the ldap scheme reveals there is no setting in the LDAP tree for it. Ok… the nice solution is dead.
…Quilt Quick How-to
As i regularly look it up from my mailbox. lets post it here.
osc co yourproject/yourpackage
cd yourproject/yourpackage
quilt setup -v *spec
cd yourpackage-*/
quilt push -a # apply old patches
quilt new yourpackage-version_fixbuild.patch
quilt edit src/foo.c
quilt refresh
foo-fixbuild.patch will automatically be created in the parent dir. if you work on a package which doesnt have a patch yet. you have to remember to copy the patch from the patch directory to your package directory.
…Haproxy - frontend vs backend ports
A very simple frontend/backend definition in haproxy can look like this:
frontend http
bind 0.0.0.0:80 tfo
bind 0.0.0.0:443 tfo ssl npn http/1.1,http/1.0 no-sslv3 crt /etc/ssl/services/
default_backend somebackend
backend somebackend
option forwardfor
option httpchk HEAD /check.txt HTTP/1.1\r\nHost:\ somehost
server somebackend1 172.16.0.1 check port 80
$ curl -I http://haproxy.example.com/
Works.
$ curl -I https://haproxy.example.com/
Does not work.
The log snippet also looks ok:
http~ somebackend/somebackend1 0/3004/-1/-1/3005 503 3090 - - SC-- 529/126/0/0/3 0/0 "GET / HTTP/1.1"
But why does it report 503?
…Epmd and systemd
By default the epmd on (open)SUSE only listens on 127.0.0.1. Which in itself is a sane default policy. We don’t expose any services directly to everyone just because we install and run them. But if you want to cluster erlang services you need it to listen on more addresses.
$ export EDITOR=vim
$ systemctl edit epmd.socket
Add the following content:
[Socket]
# unset all ports defined in the global file, in our case this is 127.0.0.1:4369
ListenStream=
# add our new ports
ListenStream=[::]:4369
Then run:
…OSC helpers for lazy packagers
If you ever discussed packaging in person with me, you might have noticed i normally don’t call “osc build” but a little shell thingie. The reason is i don’t want to maintain a long command line all the time.
So in my shell RC file i have the following:
osc_build() {
pkg_dir=$1
shift
repo=$1
shift
osc build -x vim -k ../rpms/$pkg_dir -p ../rpms/$pkg_dir $repo x86_64 $@
}
ob-15.1() {
osc_build 15.1 openSUSE_Leap_15.1 $@
}
ob-15.2() {
osc_build 15.0 openSUSE_Leap_15.0 $@
}
ob-tw() {
osc_build tw openSUSE_Tumbleweed $@
}
ob-twf() {
osc_build tw openSUSE_Factory $@
}
-x vim
So i can edit inside the build environment. There is actually really a nice helper to get into the build env:
…FFI and Requires
FFI is a method to interface with native libraries, which is becoming more and more popular in many scripting languages. Unlike with native extensions though, we have nothing that links the shared library. As such our requires generator for shared libraries doesn’t kick in and we get no requires on the shared library package.
To make matters worse with the shared library policy the soversion is dependent on against which distro you build against.
…A dive into requires with ranges
Disclaimer: I am aware that python does not allow easy installation of 2 libraries in parallel. It is just an example.
Let’s say you have an upstream requirement "needs pyfoo >= 1.6.0 but smaller than 2"
. Sounds easy right?
Our naive solution will be:
Requires: python-pyfoo >= 1.6.0
Requires: python-pyfoo < 2
so far … so good. If your repository has only one version of pyfoo available we will probably get the package you want. now lets say we have more than one version available.
…Living on Tumbleweed
Since we have staging for Factory/Tumbleweed, it became surprisingly stable. Major breakages at least for core packages are really rare now. But even for leaf packages it is not a common sight.
One thing was still annoying: for upgrading you should use zypper dist-upgrade
or short zypper dup
normally. zypper dup
will upgrade all the packages to the latest version it finds but also downgrade/uninstall packages, if required. Something a zypper up
will not do. This is all fine when you only use the core distribution repositories. If you start using e.g. packman or any other repository from the build service, zypper did switch around the package to the distribution package and a few days later back to the OBS based repository. Depending on what was newer. It is allowed to do that because zypper dup
allows also vendor changes.
Howto patch a rubygem based package?
After I was asked again today how to properly patch a rubygems package, here is a quick how to.
First things first we branch the package.
darix@tengu ~/osc/tmp % osc bco devel:languages:ruby:extensions/rubygem-net-ssh-session
A home:darix:branches:devel:languages:ruby:extensions
A home:darix:branches:devel:languages:ruby:extensions/rubygem-net-ssh-session
A home:darix:branches:devel:languages:ruby:extensions/rubygem-net-ssh-session/net-ssh-session-0.1.6.gem
A home:darix:branches:devel:languages:ruby:extensions/rubygem-net-ssh-session/rubygem-net-ssh-session.changes
A home:darix:branches:devel:languages:ruby:extensions/rubygem-net-ssh-session/rubygem-net-ssh-session.spec
At revision 9edd872bc9121e1c68a71d057dfa607f.
Note: You can use "osc delete" or "osc submitpac" when done.
darix@tengu ~/osc/tmp % cd home:darix:branches:devel:languages:ruby:extensions/rubygem-net-ssh-session
So far nothing unexpected a gem file and spec+changes.
darix@tengu rubygem-net-ssh-session % l
total 28
drwxr-xr-x 3 darix users 122 Jul 23 12:21 ./
drwxr-xr-x 4 darix users 47 Jul 23 12:21 ../
-rw-r--r-- 1 darix users 13824 Jul 23 09:54 net-ssh-session-0.1.6.gem
drwxr-xr-x 2 darix users 4096 Jul 23 12:21 .osc/
-rw-r--r-- 1 darix users 143 Jul 23 10:09 rubygem-net-ssh-session.changes
-rw-r--r-- 1 darix users 1472 Jul 23 09:54 rubygem-net-ssh-session.spec
I prefer quilt for my patch handling but quilt setup doesn’t know how to handle our spec files yet. So some manual work required. This btw is also the reason why I include the series file in the source package.
…It is all about the ... plus
This is just a quick post but imho still pretty neat. If you are a postgresql user you probably know about the commands with \ to inspect your database server e.g. .
mirrorbrain=> \l mirro*
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-------------+-------------+----------+-------------+-------------+-------------------
mirrorbrain | mirrorbrain | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
(1 row)
mirrorbrain=> \dt
List of relations
Schema | Name | Type | Owner
--------+---------+-------+-------------
public | country | table | mirrorbrain
public | file | table | mirrorbrain
public | hash | table | mirrorbrain
public | marker | table | mirrorbrain
public | mirror | table | mirrorbrain
public | pfx2asn | table | mirrorbrain
public | region | table | mirrorbrain
public | server | table | mirrorbrain
public | version | table | mirrorbrain
(9 rows)
mirrorbrain=>
So far … nothing new … right?
…Per service ulimits
In the good old days of sysvinit, if you wanted to run some extra code during start up, you were either (ab)using the sysconfig file or patching the init script itself. The init scripts in many packages are not as marked as configuration, which means in those cases your edits were lost. If it was marked as configuration then you have to merge your changes with those fixes manually. In either case … not ideal.
…Apparmor and multiple instances of the same service
Getting started
Let’s say you want to have 3 instances of redis. One for each redis using app. Getting 3 instances app with systemd’s nifty Instantiated Services is easy.
[Unit]
Description=Redis
After=network.target
PartOf=redis.target
[Service]
Type=simple
User=redis
Group=redis
PrivateTmp=true
PIDFile=/var/run/redis/%i.pid
ExecStart=/usr/sbin/redis-server /etc/redis/%i.conf
Restart=on-failure
[Install]
WantedBy=multi-user.target redis.target
“systemctl start redis@discourse” will launch us an instance with “discourse.conf”. Other systemctl commands just work as fine. “systemctl enable redis@discourse” , “systemctl status redis@discourse” . Now to stop all redis instances? “systemctl stop redis.target”
…Irssi window_switcher.pl
Especially when using irssi via your smartphone it can be highly annoying to switch windows. Yes the /win $number
thing will work. But who can remember all those numbers when you go past 20 windows? That’s where window_switcher.pl comes into play.
Setup
First download the file and place it into ~/.irssi/scripts/autorun/
.
Then switch back to irssi and set up window_switcher. We will bind ctrl+o to window_switcher. The status bar item is placed in the front of the status bar, so we an still see it on small displays.
GitLab CE Quick install for our rpm
General
Official Install Documentation
Repositories
- use the repositories from devel:languages:ruby and home:darix:apps
- zypper in –from home:darix:apps redis gitlab-ce gitlab-shell
config redis:
See also: /usr/share/doc/packages/redis/README.SUSE
$ cd /etc/redis
$ cp default.conf.example gitlab.conf
$ vi gitlab.conf
* daemonize no
* pidfile /var/run/redis/gitlab.pid
* logfile /var/log/redis/gitlab.log
* dir /var/lib/redis/gitlab/
$ install -d -m 0750 -o redis -g redis /var/lib/redis/gitlab/
$ chown root:redis gitlab.conf
$ sc start redis@gitlab
$ sc enable redis@gitlab
GitLab Shell
cd /usr/share/gitlab/shell/; cp config.yml{.example,}
- set up redis socket matching /etc/redis/gitlab.conf
- maybe enable
audit_usernames
(but see warning)
config rails app
- cd /srv/www/vhosts/gitlab-ce/
- configure database.yml (based on postgresql example)
- cp config/gitlab.yml.{example,}
- host, port, https, email_from in gitlab section
- optionally ldap settings
- cp config/resque.yml.{example,}
- adapt socket for redis. should match /etc/redis/gitlab.conf
export RAILS_ENV="${RAILS_ENV:=production}"
gitlab-ce-update
rake db:seed_fu
- check for new files which are now owned by root:root (e.g. .gitlab_shell_secret) they should be owned by root:gitlab with permissions
u=rw,g=r,o=
- make sure you have the symlink
.gitlab_shell_secret /srv/www/vhosts/gitlab-ce/.gitlab_shell_secret
- Configure git for gitlab user as seen in official docs
sc start gitlab-ce-unicorn gitlab-ce-sidekiq gitlab-workhorse
sc enable gitlab-ce-unicorn gitlab-ce-sidekiq gitlab-workhorse
Testing things
- ssh access
ssh -T gitlab@gitlab.suse.de
Welcome to GitLab, $yourusername!
…