As some of you may know, I’m putting on a workshop, called System Administrator Study Group. If you haven’t heard about this workshop, listen up!

SysadminSG as it’s called, is to get Fedora contributors prepared for certain certification exams, specifically the Red Hat Certified System Administrator (RHCSA) and the Red Hat Certified Engineer (RHCE).

This workshop is not a place where you can come and earn either of these certifications. It is also not a course where you can get free instruction. Instead, this workshop is a hands-on place to sit down with others of like mind and work through the exam requirements as specified by Red Hat. If you have ever desired to get your RHCE certification, or just want to bone up on some of the qualifying skills, this is the place for you.

A Test Run – 18:00 to 21:00 UTC – December 27, 2011

My hope is to have everything working on ec2 instances for participants in the workshop. In that sense, I’ll be setting some things up in the morning tomorrow to make certain things work. It is clear to me that I’ll need help from a few folks to test the setup I’ve created.

If you have an hour or so tomorrow during the above time frame, I’d love your help testing these instances. Please find me in #fedora-cloud on freenode (I’m herlo) and I’ll set you up with your own 24-hour instance. I’ll probably be in there asking gholms and others for help getting ec2 policies working, among other minor issues.

Cheers,

Herlo

Last night, Mike (shalkie) and I had a nice discussion about the process we are now using to build / import packages. This is in preparation for now and the future. A lot of this discussion is thanks to Mathieu Bridon (bochecha).

I have inserted a few notes in the process below for clarification.

02:03 < herlo> k, so you should know that you first ‘skein request’ a package
02:03 < herlo> it can be done with either the –path or –name option, but one must be provided
02:04 < herlo> normally, one shouldn’t grant their own package. I should be around some tomorrow to grant packages in bulk for you as needed
02:04 < herlo> but
02:04 < herlo> if you do need to grant a package
02:04 < herlo> it should be something like
02:05 < herlo> skein grant -k <koji owner> -g <github owner> issue_number name

Package repos and koji tags are what get created with skein grant. However *only* an admin can grant a new package into our build environment. This is to make sure we are not putting packages into the process that are not part of the upstream and/or for a good review
process. In the future, I suspect quite a few people will be admins and this will be less problematic.

02:05 < herlo> once granted, the package can be imported with
02:05 < herlo> skein import /path/to/srpm
02:06 < herlo> usually though, I do take one extra manual step.
02:06 < herlo> I visit the repo page on github and add a service hook
02:06 < herlo> if you look at this page: https://github.com/gooselinux/libgnomecanvasmm26/admin/hooks
02:07 < herlo> you can see the ‘Post-Receive URLs (1)’
02:07 < herlo> if you click on it, you’ll see the url you should add for any new repo
02:07 < herlo> this post receive hook will automatically launch builds upon a commit

If you have admin access to a repository (and if a package has been ‘granted’ to you, then you do), add the following in the Post-Receive
URLs if it’s not already there:

http://roman.gooselinux.org:8080/add

02:08 < herlo> once you have that in place, run the import command described above
02:08 < herlo> at that point, everything should be pretty automatic.
02:08 < shalkie> k
02:08 < herlo> within 10 minutes, the build should automatically launch
02:08 < herlo> shalkie: seem pretty straightforward?
02:09 < shalkie> Yeah it does.
02:09 < herlo> k, so now it should also be clear that you may (and probably already do) have failed builds
02:09 < shalkie> The stack of email certainly suggests that.
02:09 < herlo> lol
02:10 < herlo> shalkie: a good portion of those can be run through again, though I have been working on fixing a few of mine
02:10 < herlo> shalkie: as you look through the failed builds, you may just need to run them again
02:11 < herlo> to do that, you can use
02:11 < herlo> skein build
02:11 < herlo> the way I usually do that is ‘skein build –nowait dist-gl6 <pkg_name>’
02:11 < herlo> only an admin can perform this task (you are an admin)
02:12 < herlo> you can watch the tasks at http://koji.gooselinux.org/koji as you always have, or go out for a
few hours and check the build statuses when you return

02:13 < herlo> BUT WAIT! There’s more!
02:13 < herlo> If for some reason, you determine something has to change in the spec file, like another dependency or something
02:13 < herlo> you’ll need to do two things
02:14 < herlo> first, clone the branch if not already there and then create a vanilla branch
02:16 < herlo> git checkout –track -b  <local branch> <remote>/<tracked branch>
02:16 < herlo> eg git checkout –track -b vanilla origin/vanilla
02:16 < herlo> then push that vanilla branch
02:16 < herlo> git push origin vanilla
02:17 < herlo> this is so we can keep the original state of the package before we make any changes. I’ll be working on  automating this later
02:17 < herlo> when vanilla is pushed, we’ll go back to master and make our changes
02:17 < herlo> git checkout master
02:17 < herlo> <make changes>
02:17 < herlo> git add <files changed>
02:17 < herlo> git commit -m “message regarding changes”
02:18 < herlo> git push origin master
02:18 < herlo> again, when you push to master, a build will be launched within 10 minutes
02:18 < herlo> sit back, enjoy coffee and wait for the build to return

At the moment, the ‘master’ branch is currently what is getting built. Vanilla never gets built, but serves as a reference point to what the upstream did at certain points along the way. The vanilla branch also provides a way to merge in upstream changes with our changes. Git will
make this easy for us, because merge conflicts will be much easier to resolve across branches.

I hope this little conversation is a bit clearer on the process we’re trying to use. Look for further improvements coming to skein 2.1 (I already have quite the feature list) in the next few months.

Cheers,

Clint

PS – I likely won’t be online when this post is published as I’ll literally be enjoying some GoOSe, prepared by my lovely wife. She is an amazing cook / chef and is attempting this for the first time ever!

As of this morning, it appears we only have around 40 failed builds! We’re getting very close! I think we could make an alpha by the end of next week with a good bit of sprinting! As I understand it, one is planned for next week, maybe we should sit down and figure out those details?

Cheers,

Clint

Build statistics from 2011-11-01 through 2011-12-09

It appears that a bit more than 1/3 of the builds failed and notifications are going out to the package owners. This means we can work through each of them and resubmit them. Odds are that most of them are due to dependency issues and only a few for other reasons.

We’re preparing for Pass #2 this weekend. We’ve discovered some interesting things, however, so here’s something we’ll be putting in place first (hopefully):

A repository to track BANNED and EXCLUDED (for now, other names may arise) SRPMs.

At the moment, there are SRPMs we cannot build because we don’t support an architecture (s390, for example) There is no reason to build packages that won’t work. There are also some packages that contain proprietary content we need to scrub, even though the license states they are free software. These packages will have to wait unless they affect a critical path of getting GoOSe 6.0 composed in coming weeks.

The plan here is to create a repository to track these banned / excluded SRPMs. Skein will query this git repository for the BANNED and EXCLUDED files to make sure any request, grant, or build actions first check to make sure the package is not in either list. If the package is in the list, skein will refuse to act upon the package and the requester will have to resolve the issue before attempting again.

Watch for the updated skein v2.1 with this fix in place in the very near future.

Cheers,

Herlo

The best part about this tool is, while I have been working on it, almost all of the remaining SRPMS have been imported and the majority of them have been built on our koji server. Right now, we’re on the last batch and I expect it to finish sometime late Friday or early Saturday (12/10/2011). This ends our first major hurdle to getting an Alpha release of GoOSe Linux 6.0 out the door.

Another huge tool in this process is Grapple, written by python master Nafai, (aka Travis Hartwell). He spent a good bit of time getting this github hook in place that will record all git commits and help to process them automatically. I spent a bit of time and added a client script to grab the recorded commits and send them on to koji automatically with the proper dist value and url. No more guessing! This is the ‘automagic’ corollary to skein build.

Next up, I plan to rebuild the failed builds and push them through koji again, using skein build. Possibly doing this a few more times until we get things pared down to the packages that don’t just have dependency problems. I hope to have most of those fixed up and in place by Monday or Tuesday next week (12/12/2011).I’ve also noticed a few builds that will never build for our arches, so I will have to figure out what to do with them for now.  I don’t want to waste koji resources trying to rebuild them when I know they will never work.

The first Alpha ISO is getting close enough to taste it now! We’ll have to make sure to have a grand celebration upon our first Golden GoOSe Release :)

Cheers,

Herlo

I have now RPMs built for Fedora and EPEL6 (EPEL5 requires some outside help). They are available from my personal repository on fedorapeople.org. If you are interested in trying them out, you can just grab the repo definition for your system.

A review request

Salt development moves very fast. In the past 2 weeks or so, I’ve posted 2 new releases for salt in my review request. I’d really like to get salt into the Fedora and EPEL repositories, but I need this review to be completed. I will happily do a review swap!!

Cheers,

Herlo

As I have been working on getting the import functionality into skein pre2.0, there are a few features that have come to mind as I go along. It’s very interesting to see these features appear once the tool is out in the wild. Some of them have come from other users/contributors and some from my brain, but here’s a dump of what I have been considering adding to skein:

• skein revoke – (this one is from imak, and quite a good idea) It’s possible to accidentally request a repo that you didn’t mean to request. skein revoke will allow you to specify that you erred and will allow you to close the request with a nice message you provide.
• skein request from an SRPM – the above request brought forth something I’d been thinking about doing, but hadn’t really formalized. I suspect this feature will be a good bit of our automation in our second push to finish GoOSe 6.0 by year’s end.
• skein update – providing this feature will probably happen after 2.0. The plan would be to allow an already existing repo to be updated to a new release or version of an SRPM. In addition, it’s possible the update will be to a new branch, which update should be able to handle.

As for the rest of the features, the code for export was just finished today. I expect to be able to finish push, upload and import by the end of this coming week. Then it will be on to importing the rest of the upstream tree and building GoOSe 6.0

Cheers,

Herlo

Skein’s goal is provide easy functionality for rebuilding SRPMS from upstream and import them into github, where they can be built using our koji instance. The process is actually easier than one might think:

• skein request – Request a particular repository be setup on our github organization for the specified package. The package itself should be able to be requested from the SRPM, but that feature is not yet available.
• skein query – To verify the request has been placed, this shows the open queries (by default).
• skein show – For a particular request, show the detail of who requested and the purpose of such a request.
• skein grant – Only an admin can grant the repository. Only members of the admin team on our github organization can grant the repository.
• skein extract – Once granted, the SRPM can be extracted and placed into two basic directories:
• /path/to/package/lookaside/ – contains the archive from the SRPM, usually a tar, tar.gz or zip file. The contents of this directory can then be pushed to the lookaside cache.
• /path/to/package/git/ – contains the spec file, any patches and other sources that are not archives. A Makefile and sources file are also generated along with a .gitignore to provide useful functionality during the koji build

Other functionality is currently under development:

• skein push – Once extracted and committed to the git repository, this pushes the git commits to github.
• skein upload – Once extracted, this uploads the content of the lookaside directory to the lookaside cache at pkgs.gooselinux.org.
• skein import – A combination of skein extract, push and upload, since that seems fairly logical.
• skein build – Albeit mostly complete, it will need to be tested with an SRPM that has been run through this process.

I took a few minutes the other day and created a video of the completed process. I post this here for others to use for their benefit, but also so I can have it in another place besides my laptop.

Skein video

Cheers,

Herlo

Recently, Jonathan Nadaeu and I sat down and discussed Fedora, GoOSe and a few other fun things on This Week in Fedora. The majority of the interview was about the GoOSe project and it covers a good bit of why we’re doing what we’re doing, how it works and how to get involved. Have a listen, and come hang out and help out the project!

In addition, I was asked to do a presentation this week on how to build RPMs at the Ogden Area Linux User Group (OALUG). While I teach this in my day job as a Red Hat Trainer, I’ve never given my own presentation. I’m really excited to share some very useful tips and tricks on how to build an RPMs and set them up in a custom repository for your company or for others to consume. It’s very cool and very easy to do most of this work.

So come on up to OALUG tomorrow night if you can.  I’ll make sure to follow-up here with the useful links for those interested.

Cheers,

Herlo

The project I’ve been working on does require a bit of thought around enterprise Linux versions run by a community. There is the ever amazing CentOS, Scientific Linux and a few others who have been around the block a time or two. The work that they have done has been immense and very helpful to many, including me.

For my project, the work was about building a fully binary compatible, enterprise-ready, community version of Linux, very similar to what CentOS and others have done. The question always comes ‘why?’ which will be addressed later on in future posts. Suffice it to say, the work we’ve been doing has paid off in both a individual and community sense.

In the beginning of this project, it was clear that we needed some tools to make things work the way we wanted. Luckily, there were tools out there to do a good portion of our work. Tools like koji, mock and of course Linux to bring it all together. But other tools seemed to be missing and I went on a quest…

The first tool that seemed to be missing was a way to import src.rpms from the most popular upstream vendor. These packages needed to be rebuilt by koji in some fashion, but just taking the srpms and rebuilding them had been done before, and seems to be the preferred way to date. In my mind however, it seemed that we were missing a step. Enter skein.

While skein is still very green and will need quite a bit more work, it accomplishes the goal of extracting the srpms into two parts. This tool basically sets up two things; a git repository on github.com (for now) along with a location and verifiable way to store the archive stored inside the srpm, called a lookaside cache. If one looks at the way the Fedora Project maintains their source, this process is very similar.

Once the srpm is imported with skein, it can be built with koji. At the moment, this process is fairly manual, but the plan is to improve skein to also allow building from the repositories. However, another bit more automatic way to build would be to use a git hook. Luckily, github provides several ways to accomplish this, including a custom URL to which an HTTP POST can be sent. At which time, koji would download the spec file and source from the appropriate locations and build a srpm.

Koji completes its task by building the binary RPM(s) and appropriately tagging the successful builds. Once complete, mash can be used to generate custom repositories to prepare for composing actual iso images. Mash is a command-line tool, again used by Fedora.

Once the repositories are generated by mash, pungi takes over. The process of building an iso is actually very simple, just a kickstart file, some repositories and pungi are used to create a fully installable DVD or multi-CD iso image.

Here’s a bit of my excellent artwork to better describe the process.

A couple things to note about this process is that while it is starting to become clear how to build an OS from an upstream vendor, there are parts that still haven’t been addressed. Currently, we can import with skein and rebuild the SRPM and build the binary RPMs with koji. We yet to have enough binary RPMs to actually construct a buildroot, but we are getting very close.

Automating the builds with git hooks and a skein build process is a nice big step toward making our own Operating System possible.

The other big piece of the puzzle is dependency resolution. Now this has been mostly resolved by tools and apis like Yum and RPM, but I still feel very much like a n00b when working with them. My hope is to figure out that process in the next week or so, and update skein to make building faster and easier overall.

Cheers,

herlo