This should be a very short blog post, but long enough to justify a blog post instead of a 'tweet' : I had myself a small issue with mitogen plugin in our Ansible infra.

To cut a long story short, everybody knows that ansible relies on ssh as transport. So one can use traditional ~/.ssh/config tuning to declare ProxyJump for some hosts, etc

But when you use mitogen (we do), in the official doc there is a mention of specific parameter for connection delegation : mitogen_via

The simple example on the webpage seems trivial and if you have multiple hosts that need to be configured from remote ansible+mitogen combo, using mitogen would speed things up as it would know about the host topology.

That's what I thought when having a look at the simple inventory on that web page:

[dc2]
web1.dc2
web2.dc2
web3.dc2

[dc2:vars]
mitogen_via = bastion.dc2

Sounds easy but when I tried quickly to use mitogen_via , something that I thought would be obvious in fact wasn't. My understanding was that mitogen would automatically force agent forwarding when going through the bastion host. A simple ansible -m ping (let's assume web1.dc2 in their example) returned …

This week I had to work on a PoC to deploy OpenShift in Virtual Machines instead of bare-metal, like we did recently for the CentOS CI infra

Why in Virtual Machines (KVM guests) and not on bare-metal ? Well, there are cases where you have powerful/beefy machines, but not enough to meet the minimum number of nodes (at least 3 etcd nodes, and not even counting the real workers, at least 2 so 5 in total for bare minimum), while these nodes would perfectly (both at cpu/memory and storage) support the whole infra (assuming that you don't deploy all etcd/control planes nodes on the same physical node of course, and same for workers)

If you have a look at the official openshift documentation, you'll see that while all major cloud providers (AWS, Azure, GCP) are listed, there are also ways to deploy on bare-metal (what we did for CI infra), but also on RHEV, vSphere and Openstack too .. but nothing for plain KVM hypervisors (managed by libvirt in our cases).

But a VM is more or less like a bare-metal install, so what about we treat the VMs as bare-metal ? problem solved, right ? For our bare-metal deployment, we …

Recently in the CentOS Infra, we got a new sponsor giving us access to a server that has a HBA needing a kernel module that was deprecated in the RHEL8 (and thus CentOS 8) kernel by default.

What can you do in such situation ? Answer is easy : Elrepo ! They provide (for years now) kernel modules ready to go for network cards, raid/hba controllers, wifi nics, etc, and for various versions of RHEL/CentOS and other rebuilds using same kernel.

I wanted to give it a try on a node I have at least remote KVM/ipmi access, to reset the node in case of problem. Let's use the following ~8y old IBM blade for this example , that has the following network interface card and also hba :

# lspci |egrep -i 'ethernet|Serial'
0c:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2004 PCI-Express Fusion-MPT SAS-2 [Spitfire] (rev 03)
16:00.0 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 03)
# lspci -n |egrep '0c:00.0|16:00.0'
0c:00.0 0107: 1000:0070 (rev 03)
16:00.0 0200: 19a2:0710 (rev 03)
# ethtool -i eth0|grep driver
driver: be2net …

Recently I had to work with one of my colleagues (David) on something that was new to me : Openshift. I never really looked at OpenShift but knew the basic concepts, at least on OKD 3.x.

With 4.x, OCP is completely different as instead of deploying "normal" Linux distro (like CentOS in our case), it's now using RHCOS (so CoreOS) as it's foundation. The goal of this blog post is not to dive into all the technical steps required to deploy/bootstrap the openshift cluster, but to discuss of one particular 'issue' that I found myself annoying while deploying: how to disable dhcp on the CoreOS provisioned nodes.

To cut a long story short, you can read the basic steps needed to deploy Openshift on bare-metal in the official doc

Have you read it ? Good, now we can move forward :)

After we had configured our install-config.yaml (with our needed values) and also generated the manifests with openshift-install create manifests --dir=/path/ we thought that it would be just deploying with the ignition files built by the openshift-install create ignition-configs --dir=/path step (see in the above doc for all details)

It's true that we ended up with some …

It's usually always a good thing to receive a newer laptop, as usually that means shiny new hardware, better performances and also better battery life. I was really pleased with previous Lenovo Thinkpad t460s and so the normal choice was its successor, also because default model following company standard, and so the t490s

When I received the laptop, I was a little bit surprized (had no real time to review/analyze in advance) by some choices :

• No SD card reader anymore (useful when having to "dd" some image for armhfp tests)
• Old docking style is gone and you have to connect through usb-c/thunderbolt
• Embedded gigabit ethernet in the t490s (Intel Corporation Ethernet Connection (6) I219-LM (rev 30)) isn't used at all when docked, but going through usb-net device

Installing CentOS Stream (so running kernel 4.18.0-147.6.el8.x86_64 when writing this post) was a breeze, after I turned on SecureBoot (useful also because you can also use fwupd to get LVFS firmware updates automagically as I did for my t460s)

But quickly I realized a huge difference between my previous t460s and the new t490s : heat/temperature and so fan usage. To a point where it was …

Puppet CA/puppetmasterd cert renewal

While we're still converting our puppet controlled infra to Ansible, we still have some nodes "controlled" by puppet, as converting some roles isn't something that can be done in just one or two days. Add to that other items in your backlog that all have priority set to #1 and then time is flying, until you realize this for your existing legacy puppet environment (assuming false FQDN here, but you'll get the idea):

Warning: Certificate 'Puppet CA: puppetmasterd.domain.com' will expire on 2019-05-06T12:12:56UTC
Warning: Certificate 'puppetmasterd.domain.com' will expire on 2019-05-06T12:12:56UTC

So, as long as your PKI setup for puppet is still valid, you can act in advance, resign/extend CA and puppetmasterd and distribute newer CA certs to agents, and go forward with other items in your backlog, while still converting from puppet to Ansible (at least for us)

Puppetmasterd/CA

Before anything else, (in case you don't backup this, but you should), let's take a backup on the Puppet CA (in our case, it's a Foreman driven puppetmasterd, so foreman host is where all this will happen, YMMV)

tar …

While I have to admit that I'm using Zabbix since the 1.8.x era, I also have to admit that I'm not an expert, and that one can learn new things every day. I recently had to implement a new template for a custom service, that is multi-instances aware, and so can be started multiple times with various configurations, and so with its own set of settings, like tcp port on which to listen, etc .. , but also the number of instances running as it can be different from one node to the next one.

I was thinking about the best way to implement this through Zabbix, and my initial idea was to just have one template per possible instance type, that would though use macros defined at the host level, to know which port to check, etc .. so in fact backporting into zabbix what configuration management (Ansible in our case) already has to know to deploy such app instance.

But parallel to that, I always liked the fact that Zabbix itself has some internal tools to auto-discover items and so triggers for those : That's called Low-level Discovery (LLD in short).

By default, if you use (or have modified) some …

Recently I had to update the existing code running behind mirrorlist.centos.org (the service that returns you a list of validated mirrors for yum, see the /etc/yum.repos.d/CentOS*.repo file) as it was still using the Maxmind GeoIP Legacy country database. As you can probably know, Maxmind announced that they're discontinuing the Legacy DB, so that was one reason to update the code. Switching to GeoLite2 , with python2-geoip2 package was really easy to do and so was done already and pushed last month.

But that's when I discussed with Anssi (if you don't know him, he's maintaining the CentOS external mirrors DB up2date, including through the centos-mirror list ) that we thought about not only doing that change there, but in the whole chain (so on our "mirror crawler" node, and also for the isoredirect.centos.org service), and random chat like these are good because suddenly we don't only want to "fix" one thing, but also take time on enhancing it and so adding more new features.

The previous code was already supporting both IPv4 and IPv6, but it was consuming different data sources (as external mirrors were validated differently for ipv4 vs ipv6 connnectivity). So …

One thing that one has to like with Entreprise distribution is the same stable api/abi during the distro lifetime. If you have one application that works, you'll know that it will continue to work.

But in parallel, one can't always decide the application to run on that distro, with the built-in components. I was personally faced with this recently, when I was in a need to migrate our Bug Tracker to a new version. Let's so use that example to see how we can use "newer" php pkgs distributed through the distro itself.

The application that we use for https://bugs.centos.org is MantisBT, and by reading their requirements list it was clear than a CentOS 7 default setup would not work : as a reminder the default php pkg for .el7 is 5.4.16 , so not supported anymore by "modern" application[s].

That's where SCLs come to the rescue ! With such "collections", one can install those, without overwriting the base pkgs, and so can even run multiple parallel instances of such "stack", based on configuration.

Let's just start simple with our MantisBT example : forget about the traditional php-* packages (including "php" which provides the mod_php for Apache …

Yesterday, I got some alerts for some nodes in the CentOS Infra from both our monitoring system, but also confirmed by some folks reporting errors directly in our #centos-devel irc channel on Freenode.

The impacted nodes were the nodes we use for mirrorlist service. For people not knowing what they are used for, here is a quick overview of what happens when you run "yum update" on your CentOS node :

• yum analyzes the .repo files contained under /etc/yum.repos.d/
• for CentOS repositories, it knows that it has to use a list of mirrors provided by a server hosted within the centos infra (mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates&infra=$infra )
• yum then contacts one of the server behind "mirrorlist.centos.org" (we have 4 nodes so far : two in Europe and two in USA, all available over IPv4 and IPv6)
• mirrorlist checks the src ip and sends back a list of current/up2date mirrors in the country (some GeoIP checks are done)
• yum then opens connection to those validated mirrors

We monitor the response time for those services, and average response time is usually < 1sec (with some exceptions, mostly due to network latency …