I was advised by several people to give Foreman a try and it's really simple to install. Their wiki covers basic installation and there is even a yum repo that can be used (Epel has to be enabled too). As i have a small network to manage, I decided to setup Foreman on the same host as puppetmaster. Configuring /etc/foreman/* is easy and missing parts can be configured just by looking at the Foreman website wiki/FAQ. But troubles came when I enabled reports : puppetmasterd config was changed to include :

[master]
reports = store, foreman

and the foreman.rb script (copied and modified from /usr/share/foreman/extras/puppet/foreman/templates/foreman-report.rb.erb) integrated in the correct /usr/lib/ruby/site_ruby/1.8/puppet/reports dir. (Note : don't forget to update $foreman_url).

But no reports were coming in Foreman. hmmm .... error message was :

Report foreman failed: Could not send report to Foreman at http://puppetmaster.mybeautifuldomain.com:3000/reports/create?format=yml: Permission denied - connect(2)

That was not an iptables issue, but selinux one :

type=AVC msg=audit(1329830711.788:28372): avc:  denied  { name_connect } for  pid=13144 comm="puppetmasterd" dest=3000 scontext=unconfined_u:system_r:puppetmaster_t:s0 tcontext=system_u:object_r:ntop_port_t:s0 tclass=tcp_socket

Here is my locally generated selinux for Foreman :

module foreman 1.0;

require {
type puppetmaster_t;
type http_port_t;
type ntop_port_t;
class tcp_socket name_connect;
}

#============= puppetmaster_t ==============
allow puppetmaster_t http_port_t:tcp_socket name_connect;
allow puppetmaster_t ntop_port_t:tcp_socket name_connect;

Things work really better after I added my foreman.pp selinux module on that host. If you don't know how to compile selinux custom policies, please read the nice Selinux page on the CentOS wiki, and especially the "Manually customizing selinux policies" section. Tools like sealert (from setroubleshoot-server package) and audit2allow are really helpful when there is no pre-defined selinux boolean that can be used.

Hope this helps .. and now going back enjoying reports, including error reports by mail (nice feature)

So during Fosdem 2011, I had a meeting with Karanbir to see how we could solve that issue and put automation in the QA loop. We dedicated some (old) machines to be used only for QA, and in a separate VLAN. Basically, here are the steps from the built bits to the QA reports.

• The CentOS buildfarm (using the newly build system called 'reimzul' and using beanstalkd as a queuing system) pushes automatically each new tree to the dedicated QA hardware
• There is a rsync post-xfer script that is launched from there that also uses beanstalkd and some workers (so we can scale out easily if we add machines)
• Each built and pushed tree/ISOs set has its own BuildTag (that is used to identify what was tested and when)
• Some tools (hosted in an internal Git repository) are then used to deploy some Virtual Machines (actually a mix of BareMetal and VMs : blade/Virtual Box/Xen/KVM) and send a report if the "deploy VM step" failed (VMs are installed through ISO/pxe boot/virt-install through http/ftp/nfs methods)
• A test suite (that we call the t_functional stack) is then copied from the local git repo to those newly deployed machines and each test is then ran. From that point a report is then automatically sent to the QA mailing-list so that people can see the results, while the full log is available on QA head node.

The fact that we use two separate git repositories (one for the deploy/provisioniong functions and another one for the tests themselves) was really a good thing, as it permitted some people to include their tests in the t_functional stack. For example , Athmane did a great job writing/fixing some tests used for 6.1 and 6.2.

More informations to come later about how you (yes, *you*) can participate and contribute such CentOS QA auto-tests !