Modifying Anaconda behaviour without rebuilding the whole install media
One thing that I had to have a look at (during CentOS 6 QA), is the way anaconda (the Red Hat/Fedora/CentOS installer) pre-defines some 'tasks' . People used to those kind of install know what I'm talking about : the "Mininal", "Desktop", "Basic Server" and other choices you have during setup. From that first selection, you can decide (or not) to customize the software selection which then leads you to a screen containing categories / groups / packages defined in the comps.xml file present under /repodata on the tree/install media.
If you don't 'see' which screen i'm talking about, a small screenshot of the upcoming CentOS 6 will explain better than words :
Those pre-defined tasks aren't defined in the comps.xml file but rather at build time within anaconda. Fine but how can you 'modify' anaconda behaviour and test it without having to patch anaconda SRPM, rebuild it and launch a new build to generate the tree and install medias ? Easy , thanks to a simple file on the tree !
People wanting to modify anaconda behaviour at install time without having to regenerate the whole tree can just create a small file (updates.img) , put it in the /images directory in the tree. Anaconda (when installing over the network, http/ftp/nfs) always try to see if an updates.img file exists, and if so, use it. Fine, so I could easily try to "patch" it without having to modify the whole tree.
Creating that updates.img (it's just a ext2 filesystem on top) is really easy :
dd if=/dev/zero of=/tmp/updates.img bs=1k count=1440
losetup `losetup -f` /tmp/updates.img
losetup -a|grep updates.img
mkfs.ext2 /dev/loop3 # was loop3 in my case
mkdir /mnt/loop ; mount -o loop /tmp/updates.img /mnt/loop/ ; ll /mnt/loop
drwx------. 2 root root 12288 Jun 11 15:43 lost+found
From now, it's just a matter of putting the new files that you want to test and that will "overwrite" at run-time the defaults anaconda ones.
(in our current example, it was the installclasses/rhel.py that needed to be modified, so I just had to create a installclasses dir and drop my version of rhel.py in there on the loop device)
When you're done, umount the updates.img, copy it to /path/to/your/install/tree/images , restart a http install (verify that permissions and selinux contexts are of course correct !) and enjoy !
Easier and faster. Thanks to the Anaconda team which decided to permit modifying the anaconda behaviour at run-time with a simple file 
IPV6 world day !
It seems quite a lot of people blogged about IPV6 day . It's true that it's always a good idea to speak about IPV6. I'm using IPV6 natively on my server hosted at Hetzner (they offer a /64 IPV6 subnet, which is more than enough for a CentOS server hosting several xen domU Virtual Machines). At home, that's another story. I use a HE.net free tunnel to be able to reach ipv6 hosts. Yes, even in 2011, you still have to use tunnels to use IPV6 ! Why ? that's indeed a good question. Even if my CentOS ipv6 tunnel end-point/router/radvd at home is working correctly, I decided to ask my belgian provider if they had plans on implementing native IPV6. Well, not for my home connection, as I already know that Belgacom (the biggest provider in belgium) doesn't support IPV6 on their BBOX2 modems that they give to customers when ordering a DSL connection at home (while i'm talking about Belgacom, please stop sending me direct advertisement to my mailbox - the real one and not the electronic one - with your invoices about a service - VDSL2/BelgacomTV - that you *can't* offer to all your customers ... thanks) . So I decided to ask their 'professional services' because we have two 'professional and business' lines that we used at $work. Long story short (to avoid explaining how much emails/cases I had to send/open to have an answer) : "no, even on the business lines we can't support IPV6 and we have no plans (*sic*, I hope that guy was just kidding or probably doesn't know the real answer ..) nor dates about future implementation of the IPV6 services/connectivity " ..
Nice .. now /me goes back to CentOS QA mode ...
What do you want to see ? CentOS 5.6 or CentOS 6.0 ?
As you probably know (if you are interested in the Enterprise Linux market), Red Hat released earlier today 5.6 . So automatically some CentOS QA team members started to discuss about that in the appropriate IRC channel. As CentOS 6.0 isn't (yet) released nor ready, the discussion was about putting 5.6 build & release as priority number one or not. Karanbir on his side asked on Twitter about thoughts on the matter, and a discussion was started too on the centos-devel list about that topic. My personal opinion (and shared by some people too) seems to give 5.6 the priority for quite some reasons :
- The centos 5.x install base is there while there is (obviously) no centos 6 install base.
- So those people having machines in production, faced to the net (, etc, etc, ...) would prefer having their machines patched and up2date (security first !)
- People running CentOS 5.x on servers and willing to install php53 packages, now officially included
- On the build side, the el5 build process is clearly identified and known since 2007 : packages with branding issues are already identified and patches/artwork is already there, meaning that it will be probably (no, surely !) faster to have 5.6 out of the door than 6
- Same rule for the QA process : people from the QA team can "blindly" focus on their previous tests, and just have a look eventually at some newer packages (a few, like php53 but not that much in comparison with el6)
Please notice that it's still my personal opinion on that question and isn't the (to be defined) official CentOS position.
CentOS team @ Fosdem 2011
Some members of the CentOS team will be present at the Fosdem . Feel free to come at our booth just to discuss ...
More informations on our wiki and on the Fosdem website
Enabling IPv6 for guests on an Hetzner CentOS 5.5 xen dom0
I was playing with IPv6 in the last days (started to use a tunnel from he.net as my current ISP doesn't support native IPv6 and doesn't plan to support it in a short time) and wanted to add IPv6 to some of my CentOS Xen domU's running on a Hetzner box. This part was a little bit more difficult than for a standard network. Due to their internal network design, Hetzner only allow 'routed' xen networks and not standard 'bridged' ones. What I used for IPv4 was just binding the public IPs on the dom0 and configured all my iptables rules there to forward/SNAT/DNAT to the appropriate domU. But you know that NAT is gone with IPv6 so normally it's supposed to be easier, right ? Well, yes and no, depending on your network layout. Even after having enabled ipv6 forwarding (net.ipv6.conf.all.forwarding=1 ), I was just able to ping the dom0 but not the guests behind. Hmm, that reminds me the proxy ARP that was used for IPv4 but not existing anymore for IPv6 (gone too ...) . ARP was (more or less, not technically correct but read the RFCs if you enough time) replaced by NDP but I don't see such option for IPv6. Well, a kernel feature called proxy_ndp (net.ipv6.conf.all.proxy_ndp=1) exists on newer kernels (like for example the 2.6.32.x that is used on RHEL6 , and so in CentOS 6) but not on CentOS 5.5 (using a 2.6.18.x) kernel .. Hmmm ...
On the other side, I was searching for a 'workaround' probably given by libvirt, but the version included in RHEL5/CentOS5 doesn't know what to do with IPv6. Okay so let's have a look at the Xen and kernel side at the same time. If the proxy_ndp kernel feature is not present on my CentOS 5.5 dom0, I can still 'advertise' my neighbors with the ip command : yes, it supports it : " ip -6 neighbor add proxy your:ipv6:long:address::1 dev eth0"
So we just need to create a modified vif-route script (in fact I decided to call it vif-route6) that will be used for ipv6 guests :
#!/bin/bash
#============================================================================
# /etc/xen/scripts/vif-route6
# Script for configuring a vif in routed mode for IPv6 only
# Based on existing vif-route script in /etc/xen/scripts and adapted for ipv6
#============================================================================dir=$(dirname "$0")
. "$dir/vif-common.sh"main_ip=$(dom0_ip)
main_ip6=$(ip -6 addr show eth0|grep 'scope global'|sort|head -n 1|awk '{print $2}'|cut -f 1 -d '/')case "$command" in
online)
ifconfig ${vif} ${main_ip} netmask 255.255.255.255 up
ip -6 addr add ${main_ip6} dev ${vif}
ipcmd='add'
cmdprefix=''
;;
offline)
do_without_error ifdown ${vif}
ipcmd='del'
cmdprefix='do_without_error'
;;
esacif [ "${ip}" ] ; then
# If we've been given a list of IP addresses, then add routes from dom0 to
# the guest using those addresses.
for addr in ${ip} ; do
${cmdprefix} ip -6 neighbor ${ipcmd} proxy ${addr} dev ${netdev:-eth0} 2>&1
result=`${cmdprefix} ip -6 route ${ipcmd} ${addr} dev ${vif} src ${main_ip6} 2>&1`
done
fihandle_iptable
log debug "Successful vif-route $command for $vif."
if [ "$command" = "online" ]
then
success
fi
Ok, so we have just now to modify our xen domU's config to add a vif that will use that specific script and give it the IPv6 address that we'll assign to that domU (from /etc/xen/your-domU-name):
vif = [ <snip of the first vif> , "mac=00:16:36:38:31:b8,vifname=test.ipv6,script=vif-route6,ip=2a01:4f8:100:4363::dead" ]
You can now start your domU and configure it normally for IPv6 (using obviously that 2a01:4f8:100:4363::dead IPv6 address and choosing the dom0 main IPv6 address as gateway ...
Hope it will help some people in the same situation (using a routed and not a bridged network layout for xen)
Zabbix crashes when using IPMI checks
Working for an IBM Business Partner for quite some years, I was used to deploy and configure (and even teach for IBM) IBM Director as a monitoring solution (for both hardware/operating systems/snmp devices/etc/etc ...). Now that I work as a sysadmin, I have to maintain one IBM director 5.20.3 setup I had myself installed and configured quite some time ago (as a consultant then). But I didn't want to update to 6.2 because it simply kills the machine on which it runs .. needs too much processor, too much memory .. and just to give you an idea : it's a Websphere/java thing that you have to install now ... I wanted to go the opensource way instead, but with something that can still monitor Linux/Windows/snmp devices and IPMI devices (we have quite some IBM servers and/or BladeCenter).
I tested Zabbix and directly felt in love with it : the agent memory footprint is really small (in comparison with that java-based agent on the Director side) and the way to build Items and Triggers is really great. I deployed it in our environment but focused first on the OS/services side (as the 'other' monitoring solution was still there for the hardware layer monitoring). I wanted then to use the integrated IPMI features of Zabbix and started to poll data from our IBM servers ... until .. crash !
From the zabbix_server.log :
2774:20101217:100001.893 IPMI Host [my.host.name]: first network error, wait for 15 seconds
2774:20101217:100002.894 Got signal [signal:11(SIGSEGV),reason:2,refaddr:0x34a3f52a38]. Crashing ...
Hmm, not good when the monitoring application crashes itself. I disabled all my IPMI checks and then the server was back without any issue. I repeated the above steps vice and versa to proove that it was really IPMI related and it's the case. Browsing the Zabbix support website returned me quite some interesting answers, including that one (ZBX-2898) and surely that one (ZBX-633) . Ok so that confirms that IPMI checks have to be disabled now and let's wait for Zabbix 1.8.4 to appear .. In the meantime I'll write some scripts (type External Check) to return values in Zabbix that can be used to create Triggers ... that's also one of the advantages in Zabbix : you can still write many plugins/scripts to do the same things
RPMforge el6 ppc builds …
Following Dag's post about packages now being built for el6 (and landing in the el6 repository for x86_64 and i386) I have to say that the ppc builds are delayed for some reasons.
First is the (already existing) problem with the build arch. RHEL4/5 and 6 aren't build to work on Mac ppc hardware. I was able to build the el4/el5 packages with a minimal mock environment (using the official RHEL tree, but reduced to contain only the ppc and noarch packages, obviously because the ppc64 packages coudn't be installed in the chroot environment). It was even harder with the glibc package from RHEL5 because it contains specific patches that require Power4 or above processor for the ppc arch. I was able to reduild it without those patches, meaning that the buildroot isn't even 100% equal to the real RHEL 5.x tree.
Now that el6 landed, i'll have a look at all these problems and try to chase them one by one. I think that my 10+ years old mac G4 will suffer from all these tests but that's still the machine that i use to build the RPMforge ppc builds.
So my first plan is to try to have a minimal buildroot that can be initiliazed on that old mac (as i've still no better hardware at my disposal ...) and once that i'll be able to have a mock buildroot initiliazed correctly with the RHEL6 ppc only packages (if that's possible, still something to determine), i'll process the whole RPMforge svn tree, meaning several days/weeks for the first run (and no, i refuse to launch a createrepo on the produced tree after each successful build )
If you're a RHEL ppc user and that there are some packages you really want first (the only requests i've received directly for the el5 land were clamav for example), feel free to ask them directly on the RPMforge list.
Thanks for your comprehension : that's hard to produce such packages on a platform not supported upstream 
ProxyCommand to the rescue !
I discussed today with a web developper who needed to reach a machine through ssh but not directly accessible from the wild Internet. In fact, she told me that she takes a shell on each hop with ssh agent forwarding and so from that shell launch another ssh session. Well, of course that works but my question was "Why don't you just simply use a ProxyCommand in your ~/.ssh/config for that host ?". I discussed with quite some people in the last months not knowing that ProxyCommand feature in OpenSSH so once again it was time to at least blog about it
From `man ssh_config` :
ProxyCommand
Specifies the command to use to connect to the server ...
The man page has an example but what I do is using ssh itself as a ProxyCommand. Just an example : suppose you need to reach HostB (not reachable from where you are) but that you can reach HostA (and that HostA can reach HostB). You can configure your ~/.ssh/config like this :
Host HostB
Hostname the.known.fqdn.as.resolvable.by.HostA
User arrfab
ForwardAgent yes
Port 22
ProxyCommand ssh remoteuser@HostA.with.ssh.access nc %h %p
And what if you need to reach HostC, which itself is only reachable by HostB ? Let's just define a new Host section in the ~/.ssh/config and another ProxyCommand !
Host HostC
Hostname the.known.fqdn.as.resolvable.by.HostB
User arrfab
ForwardAgent yes
Port 22
ProxyCommand ssh remoteuser@HostB nc %h %p
You can now directly use the `ssh HostC` from your laptop/workstation and have a direct shell on HostC even if it has to open a connection to HostA and from
there to HostB to finish to HostC.That works also for scp/sftp so you can directly copy/retrieve files to/from HostC instead of copy from one host to the next hop. More informations about those features and the correct syntax in `man ssh_config`.
Hope that you can find that useful if you didn't know that already
To automate ? or not ?
Well, this is a good question and most of us will likely answer 'yes of course'. Indeed, as a SysAdmin you want regular tasks to be automated and act the same way on a bunch of systems. But what if you need a custom task (that you only need once) to be launched on some machines at the same time ? A lot of solutions exist of course to "push" commands to be executed by clients machines. Some will use RHN/Satellite, some will prefer using something like CFengine or Puppet . I've even discussed with some admin pushing some 'encrypted' commands on a Twitter feed followed by some clients able to understand the commands and process them ... Multiple solutions exists, and your imagination is probably the limit.
But what if you just have to manage a *very* small amount of servers at the same time for different customers/environments. For example, as an IT consultant, you probably have a bunch of customers running different solutions and sometimes with only 10 or 20 servers, right ? Will you install a satellite proxy server just to push commands for those 15 machines ? or sometimes even less ? IMHO, ssh is the solution, especially if you want interactive output/processing on all the machines. I discussed with a friend of mine who said that ssh was the solution but taking a shell on 15 servers "one at a time" was time consuming. Of course it is. So why not using a shell multiplexer or distributed shells ? I was astonished to see how many people I had the chance to speak with don't even know that you can launch interactively or in batch the same commands on multiple systems at the same time ! So I thought that maybe it was time to write (like many others did) about ssh based solutions !
I've tested and continue to use on a daily basis different programs (that use ssh in the background of course). Depending on the situation (you'd have to test them to find the one that fits your needs), I like to use cluster-ssh, mussh and shmux. Others are available, like pdsh, etc .. but the first ones are the ones i'm more comfortable with and that I personnally use.
FrOSCon 2010 is over .. waiting now for 2011 :-)
It was the first time that I was at FrOSCon and I admit I enjoyed it. Not only because I can always see in real life some other CentOS contributors (thanks again Andreas, Sarah, Didi and Christoph), but also because I can see some other people really happy with CentOS. Last year (even if i was not there), CentOS was used to 'power' some TFT screens at the entrance. So we did the same this year and just because Dag asked it, we took a picture this year too
http://www.centos.org