Arrfab’s Blog

For my new job I have to learn how to deal with Citrix XenServer (yeah, because of a mixed workload of CentOS domU’s and Windows TSE servers, for which XenServer has been optimized). I liked the fact that I’m directly feeling “like home” , as Citrix XenServer dom0 is based on CentOS (still 5.3 at this time though). One of the things i had to do was to extend a Storage Repository served from an IBM DS3200 through dual HBAs, and using mpp/rdac (the default on XenServer 5.5 when it sees a rdac disk storage backend). Great, I’ve never had problem doing this on plain RHEL or CentOS machines, so after having extended the LUN on the IBM DS3200, I was back on the XenServer side. I always like to read the official documention before doing something (and it’s even faster when you know what you’re searching for) and I found this on the Citrix XenServer documentation : “How to resize a Storage repository after changing the size of an LVM-base storage” . Hmmm, WTF ? Their recipe is : “live migrate the guests, restart the host and proceed for each host”! . No, it has to work without a reboot, we’re not Windows admins, right ? Here is what i did : (that was tested on a test machine !)

We have first to list the current status/size :

[root@xen1 ~]# xe sr-list
uuid ( RO)                : c945d1bb-2432-36ac-2766-ebd2bc7f2e81
name-label ( RW): Hardware HBA virtual disk storage
name-description ( RW): Hardware HBA SR [IBM - /dev/sdb]
host ( RO): xen1
type ( RO): lvmohba
content-type ( RO):
[root@xen1 ~]# xe sr-param-list uuid=c945d1bb-2432-36ac-2766-ebd2bc7f2e81|grep physical-size
physical-size ( RO): 85886763008
[root@xen1 ~]# pvscan|grep c945d1bb-2432-36ac-2766-ebd2bc7f2e81
PV /dev/sdb    VG VG_XenStorage-c945d1bb-2432-36ac-2766-ebd2bc7f2e81   lvm2 [79.99 GB / 16.12 GB free]

Now we’ll extend with the IBM DS StorageManager script editor : “set logicalDrive ["XenPool1"] addcapacity=139 GB;”

Back on the xen host we have to rescan for the new size (using a MPP device presented as /dev/sdb on the xen host) and confirm with dmesg|tail

[root@xen1 device]# echo 1 >/sys/block/sdb/device/rescan ; dmesg|tail

sdb: detected capacity change from 85899345920 to 235149459456

[root@xen1 device]# pvresize /dev/sdb
Physical volume “/dev/sdb” changed
1 physical volume(s) resized / 0 physical volume(s) not resized
[root@sicxen1 device]# pvscan
PV /dev/sdb    VG VG_XenStorage-c945d1bb-2432-36ac-2766-ebd2bc7f2e81   lvm2 [218.99 GB / 155.12 GB free]
PV /dev/sda3   VG VG_XenStorage-9c1e7a2a-2fc0-83eb-3e32-7cea2c9e9d93   lvm2 [60.59 GB / 60.59 GB free]
Total: 2 [279.58 GB] / in use: 2 [279.58 GB] / in no VG: 0 [0   ]

Rescan now that SR :

[root@xen1 device]# xe sr-scan uuid=c945d1bb-2432-36ac-2766-ebd2bc7f2e81
[root@xen1 device]# xe sr-param-list uuid=c945d1bb-2432-36ac-2766-ebd2bc7f2e81|grep physical-size
physical-size ( RO): 235136876544

Done ! and i confirm that the CentOS domU’s were still running after that …

PS : while talking about Citrix XenServer, I have to add that I used only ssh/xe to manage it, as their XenCenter gui app is a Windows only GUI (relying on .Net). But I found several days ago an interesting GPL project: OpenXenCenter, something to keep an eye on as it’s still alpha but quickly involving …

Recently I received (through logwatch) several weekly reports about a mismatch in synchronized block on my md0 (/boot) device :

WARNING: mismatch_cnt is not 0 on /dev/md0

`cat /proc/mdstat` was normal though.

A `echo repair >/sys/block/md0/md/sync_action` followed by a `echo check >/sys/block/md0/md/sync_action` seems to have corrected it. Now `cat /sys/block/md0/md/mismatch_cnt` returns 0 …

Some member of the CentOS team will be present at the Fosdem . Feel free to come at our booth just to discuss …

More informations on our wiki and on the Fosdem website

I was a little bit late to build latest rpm packages from spec files commited in the RPMforge svn tree. I had to deal with some external stuff and also fixing the fact that rpm-macros-rpmforge has to be installed in the chroot prior to try to build the prepared SRPM that my script/wrapper created. Now that it has been fixed and it’s working (and newer rpmforge-release package to reflect all current arches), it was time to update the tree i’m building against/for . No problem for RHEL 4.8 PPC as it was ok but i updated the el5 tree to reflect RHEL 5.4 ppc. And then the problem : Mock dies completely on a bunch of errors but the first one seems obvious :

/usr/sbin/glibc_post_upgrade: While trying to execute /usr/sbin/iconvconfig.ppc child terminated abnormally
error: %post(glibc-2.5-42.ppc) scriptlet failed, exit status 115

Grr. Looking at  the`rpm -qp –changelog glibc-2.5-42.ppc.rpm` to see the differences among the glibc.ppc releases from RHEL 5.0 to 5.4 gave me some pointers :

build ppc and ppc64 base shared libraries with -mcpu=power4,
i.e. only support power4 and newer CPUs, *.a and *.o in
glibc-devel should still work on any powerpc CPU (#241003)

First thing : I always like when I consider reading such bugzilla report but can’t read it. And the machine I  use to build the RPMforge PPC packages for RHEL PPC is quite old (a mac G4 from year 2K with a 7400, altivec supported cpu @ 400MHz). That one, of course, isn’t at the required level compared to the real IBM PPC Power line .. :/

So that means that :

* either I need to find a Power4 (or above) machine to build the RPMforge ppc packages for RHEL5.4 ppc target (someone ?)

* or I need to recompile glibc.ppc with different flags, and all the dependencies … which in the background means producing a CentOS 5 PPC (which is already a slowly but ongoing process) , but I already hear my small machine screaming at me “Welcome to the Hell of dependencies [TM]” …

Maybe some of you have already noticed but the standard Spamassassin rules don’t like 2010. As explained in the SA bug 6269 , the FH_DATE_PAST_20XX rule of course matches every incoming mail starting from today .. Ouch. Time to update your rules or change your score for that rule .. I guess that an update of that rule will be available soon (i hope so) and will be fetched by a simple `sa-update` . In the meantime, time for you to fix it manually !

What can you do when the company you work for (or should I say the people who manage the Internal Network) has decided to switch from Lotus Domino to M$ Exchange 2007 ? Ouch … I can’t say that i’m personnally a great Lotus Domino supporter but it’s stable system and a native client exists for all the current platforms (packaged in .rpm and .deb for Linux,as well as in Java installshied wizard for linux distros not using either rpm nor deb packages) .. But what when you have to switch to Exchange backend ? Up to now I always managed to have my professional laptop installed with CentOS and I surely don’t want Windows on my laptop that i use for my day-to-day work

I had a quick look at the Exchange plugin that you can find for Evolution, but unfortunately that one (that uses OWA in the backend) can only be used against Exchange 2K or 2K3 but is incompatible with 2K7. Then i heard about rumours regarding a new Exchange/Mapi plugin (that requires a newer Evolution/gnome than the one provided in el5). I can’t test it as it requires direct mapi access to the Exchange server and i’m forced (up to now) to use RPC over HTTPS . Damn. It seemed that the only solution was then to install Outlook with Wine on my CentOS laptop .. until i found DavMail : it uses OWA in the backend (and is compatible with Exchange 2k7 OWA) and acts as a IMAP/Caldav/LDAP gateway. Cool, so i can use my MUA of choice (tested now with Thunderbird but i want to test Mutt as well) to read my mails, consult/update my calendar and search/uses the Exchange Addressbook without having to install any M$ component ..

So far, so Good … thanks DavMail !

While i’ve used (up to now) the IBM/LSI-logic solution (aka RDAC) to support multiple paths to an IBM storage solution (aka DS4xxx and DS3xxx), it was a pain because each time you wanted to install a new kernel the procedure implied to remove the old/previous rdac module, boot with the new kernel (without mpp), rebuild mpp/rdac and creating a new initrd and then another reboot (with the new initrd containing the correct module).

I’ve now switched to dm-multipath instead. The basic and provided /etc/multipath.conf normally works quite ok, but if you want to tune it to support more storage vendors/solutions you really have to read the multipath documentation. Jim already blogged about the DS4700 FC backend storage .

Here is the version for the DS3200 (SAS connections) :

devices {
device {
vendor                  “IBM”
product                 “1726-2xx  FAStT”
getuid_callout          “/sbin/scsi_id -g -u -s /block/%n”
prio_callout            “/sbin/mpath_prio_rdac /dev/%n”
features                “0″
hardware_handler        “1 rdac”
path_grouping_policy    group_by_prio
failback                immediate
rr_weight               uniform
no_path_retry           300
rr_min_io               1000
path_checker            rdac
}
}

The other day I had to configure a box that had to fetch some files from another machine and transfer those files from the DMZ to an external bank. While I usually use SFTP for that, in that specific case i had no choice and had to use FTP/SSL. First thing that hurted me is that to fetch the certificate/private key that the bank created for me, I had to use Internet Explorer on a Windows machine ! Ouch … (yeah, they use activex on the page you have to login to for the certificate request, you *can’t* use openssl yourself to send them the CSR …) bad, bad .. and also funny that they point you to an https website to read the documentation, in which they say how to import they Root CA (which obvsiouly you had to import yourself first to read the same manual …) .. From that time i knew i’d have troubles ..

Okay, exporting the SSL certificate/private key from Internet Exploder, using openssl to convert to PEM and i had those ready to be used on my CentOS 5.4 VM. Lftp seems good for such task and supports ssl too .. After having configured my ~/.lftprc with the correct value (like ssl:key-file and ssl:cert-file) I wasn’t able to connect : the message was : “Fatal error: gnutls_handshake: A TLS fatal alert has been received” . Hmm, strange. I decided to test with the RPMforge version (which is built against OpenSSL and not Gnutls) and that one worked correctly (without having changed the conf files). Okay it’s now working but does that mean that the lftp package from 5.x doesn’t work in ssl mode with a client certificate ? I’ve downgraded the package to the one present in the 5.x branch (before the 5.4) : lftp-3.5.1-2.fc6 instead of lftp-3.7.11-4.el5 and it worked perfectly with the same config files too. Sounds like a bug to me and not a config issue so i opened an bug upstream and on the CentOS mantis system. Let’s see how it goes. In the meantime (if you have the same issue) you can either downgrade to the lftp version you’ll find in the 5.3 tree or update to the version from RPMforge.