When is this happening you ask? Well it’s very simple and very soon, in fact it’s happening July 27-28, 2012. Yes, that’s Tomorrow and Saturday. It’s a bit unexpected for some of you, to be sure, but if you can make it, we’d love to have you for any time possible. If you can’t make it, take some time this next week and work on the GoOSe. The goal is to get the Golden GoOSe 6.0 release out by the end of August, so we really need to start cranking!

What’s left, you ask? Quite a bit actually, but the biggest piece still remaining is our Quality Assurance testing and reporting components. We’re working on them in our qa repository on github, and you can most definitely fork it and get an idea of what we’re doing. We need some serious help with ABI compliance, validation of RPMS and overall validation that all packages have been built.

Wait! There’s more! We also need help with documentation, code and usability testing of the isos. The documentation is all based upon our
‘How to Cook a Goose’ wiki page on github, the skein code needs to be
updated to the new github v3 api and the Isos can be downloaded and
tested.

Some of this is more self-explanatory and some is just plain confusing. This is where we need your help! If you have ever wanted to define how things work, you can jump in now and help us identify things and help us solve problems. That’s the fun part about free and open source software! A good for instance, what is the usability testing requirements for our Isos? Well at the moment, we don’t really have a good handle on this, but you could step in and define some requirements. If you would prefer to help write code, you could clean up things that might look a bit messy. There’s plenty to do!

Please come and help us out this weekend or anytime next week! We’ll be in #gooseproject on irc.freenode.net, or you can work by email, github or ESP. We’re excited to be moving forward and look for your help making the the ‘Quest for the Golden GoOSe’ 6.0 release a reality!

Cheers,

Clint

Last night, Mike (shalkie) and I had a nice discussion about the process we are now using to build / import packages. This is in preparation for now and the future. A lot of this discussion is thanks to Mathieu Bridon (bochecha).

I have inserted a few notes in the process below for clarification.

02:03 < herlo> k, so you should know that you first ‘skein request’ a package
02:03 < herlo> it can be done with either the –path or –name option, but one must be provided
02:04 < herlo> normally, one shouldn’t grant their own package. I should be around some tomorrow to grant packages in bulk for you as needed
02:04 < herlo> but
02:04 < herlo> if you do need to grant a package
02:04 < herlo> it should be something like
02:05 < herlo> skein grant -k <koji owner> -g <github owner> issue_number name

Package repos and koji tags are what get created with skein grant. However *only* an admin can grant a new package into our build environment. This is to make sure we are not putting packages into the process that are not part of the upstream and/or for a good review
process. In the future, I suspect quite a few people will be admins and this will be less problematic.

02:05 < herlo> once granted, the package can be imported with
02:05 < herlo> skein import /path/to/srpm
02:06 < herlo> usually though, I do take one extra manual step.
02:06 < herlo> I visit the repo page on github and add a service hook
02:06 < herlo> if you look at this page: https://github.com/gooselinux/libgnomecanvasmm26/admin/hooks
02:07 < herlo> you can see the ‘Post-Receive URLs (1)’
02:07 < herlo> if you click on it, you’ll see the url you should add for any new repo
02:07 < herlo> this post receive hook will automatically launch builds upon a commit

If you have admin access to a repository (and if a package has been ‘granted’ to you, then you do), add the following in the Post-Receive
URLs if it’s not already there:

http://roman.gooselinux.org:8080/add

02:08 < herlo> once you have that in place, run the import command described above
02:08 < herlo> at that point, everything should be pretty automatic.
02:08 < shalkie> k
02:08 < herlo> within 10 minutes, the build should automatically launch
02:08 < herlo> shalkie: seem pretty straightforward?
02:09 < shalkie> Yeah it does.
02:09 < herlo> k, so now it should also be clear that you may (and probably already do) have failed builds
02:09 < shalkie> The stack of email certainly suggests that.
02:09 < herlo> lol
02:10 < herlo> shalkie: a good portion of those can be run through again, though I have been working on fixing a few of mine
02:10 < herlo> shalkie: as you look through the failed builds, you may just need to run them again
02:11 < herlo> to do that, you can use
02:11 < herlo> skein build
02:11 < herlo> the way I usually do that is ‘skein build –nowait dist-gl6 <pkg_name>’
02:11 < herlo> only an admin can perform this task (you are an admin)
02:12 < herlo> you can watch the tasks at http://koji.gooselinux.org/koji as you always have, or go out for a
few hours and check the build statuses when you return

02:13 < herlo> BUT WAIT! There’s more!
02:13 < herlo> If for some reason, you determine something has to change in the spec file, like another dependency or something
02:13 < herlo> you’ll need to do two things
02:14 < herlo> first, clone the branch if not already there and then create a vanilla branch
02:16 < herlo> git checkout –track -b  <local branch> <remote>/<tracked branch>
02:16 < herlo> eg git checkout –track -b vanilla origin/vanilla
02:16 < herlo> then push that vanilla branch
02:16 < herlo> git push origin vanilla
02:17 < herlo> this is so we can keep the original state of the package before we make any changes. I’ll be working on  automating this later
02:17 < herlo> when vanilla is pushed, we’ll go back to master and make our changes
02:17 < herlo> git checkout master
02:17 < herlo> <make changes>
02:17 < herlo> git add <files changed>
02:17 < herlo> git commit -m “message regarding changes”
02:18 < herlo> git push origin master
02:18 < herlo> again, when you push to master, a build will be launched within 10 minutes
02:18 < herlo> sit back, enjoy coffee and wait for the build to return

At the moment, the ‘master’ branch is currently what is getting built. Vanilla never gets built, but serves as a reference point to what the upstream did at certain points along the way. The vanilla branch also provides a way to merge in upstream changes with our changes. Git will
make this easy for us, because merge conflicts will be much easier to resolve across branches.

I hope this little conversation is a bit clearer on the process we’re trying to use. Look for further improvements coming to skein 2.1 (I already have quite the feature list) in the next few months.

Cheers,

Clint

PS – I likely won’t be online when this post is published as I’ll literally be enjoying some GoOSe, prepared by my lovely wife. She is an amazing cook / chef and is attempting this for the first time ever!

Build statistics from 2011-11-01 through 2011-12-09

It appears that a bit more than 1/3 of the builds failed and notifications are going out to the package owners. This means we can work through each of them and resubmit them. Odds are that most of them are due to dependency issues and only a few for other reasons.

We’re preparing for Pass #2 this weekend. We’ve discovered some interesting things, however, so here’s something we’ll be putting in place first (hopefully):

A repository to track BANNED and EXCLUDED (for now, other names may arise) SRPMs.

At the moment, there are SRPMs we cannot build because we don’t support an architecture (s390, for example) There is no reason to build packages that won’t work. There are also some packages that contain proprietary content we need to scrub, even though the license states they are free software. These packages will have to wait unless they affect a critical path of getting GoOSe 6.0 composed in coming weeks.

The plan here is to create a repository to track these banned / excluded SRPMs. Skein will query this git repository for the BANNED and EXCLUDED files to make sure any request, grant, or build actions first check to make sure the package is not in either list. If the package is in the list, skein will refuse to act upon the package and the requester will have to resolve the issue before attempting again.

Watch for the updated skein v2.1 with this fix in place in the very near future.

Cheers,

Herlo

The best part about this tool is, while I have been working on it, almost all of the remaining SRPMS have been imported and the majority of them have been built on our koji server. Right now, we’re on the last batch and I expect it to finish sometime late Friday or early Saturday (12/10/2011). This ends our first major hurdle to getting an Alpha release of GoOSe Linux 6.0 out the door.

Another huge tool in this process is Grapple, written by python master Nafai, (aka Travis Hartwell). He spent a good bit of time getting this github hook in place that will record all git commits and help to process them automatically. I spent a bit of time and added a client script to grab the recorded commits and send them on to koji automatically with the proper dist value and url. No more guessing! This is the ‘automagic’ corollary to skein build.

Next up, I plan to rebuild the failed builds and push them through koji again, using skein build. Possibly doing this a few more times until we get things pared down to the packages that don’t just have dependency problems. I hope to have most of those fixed up and in place by Monday or Tuesday next week (12/12/2011).I’ve also noticed a few builds that will never build for our arches, so I will have to figure out what to do with them for now.  I don’t want to waste koji resources trying to rebuild them when I know they will never work.

The first Alpha ISO is getting close enough to taste it now! We’ll have to make sure to have a grand celebration upon our first Golden GoOSe Release :)

Cheers,

Herlo

As I have been working on getting the import functionality into skein pre2.0, there are a few features that have come to mind as I go along. It’s very interesting to see these features appear once the tool is out in the wild. Some of them have come from other users/contributors and some from my brain, but here’s a dump of what I have been considering adding to skein:

• skein revoke – (this one is from imak, and quite a good idea) It’s possible to accidentally request a repo that you didn’t mean to request. skein revoke will allow you to specify that you erred and will allow you to close the request with a nice message you provide.
• skein request from an SRPM – the above request brought forth something I’d been thinking about doing, but hadn’t really formalized. I suspect this feature will be a good bit of our automation in our second push to finish GoOSe 6.0 by year’s end.
• skein update – providing this feature will probably happen after 2.0. The plan would be to allow an already existing repo to be updated to a new release or version of an SRPM. In addition, it’s possible the update will be to a new branch, which update should be able to handle.

As for the rest of the features, the code for export was just finished today. I expect to be able to finish push, upload and import by the end of this coming week. Then it will be on to importing the rest of the upstream tree and building GoOSe 6.0

Cheers,

Herlo

Skein’s goal is provide easy functionality for rebuilding SRPMS from upstream and import them into github, where they can be built using our koji instance. The process is actually easier than one might think:

• skein request – Request a particular repository be setup on our github organization for the specified package. The package itself should be able to be requested from the SRPM, but that feature is not yet available.
• skein query – To verify the request has been placed, this shows the open queries (by default).
• skein show – For a particular request, show the detail of who requested and the purpose of such a request.
• skein grant – Only an admin can grant the repository. Only members of the admin team on our github organization can grant the repository.
• skein extract – Once granted, the SRPM can be extracted and placed into two basic directories:
• /path/to/package/lookaside/ – contains the archive from the SRPM, usually a tar, tar.gz or zip file. The contents of this directory can then be pushed to the lookaside cache.
• /path/to/package/git/ – contains the spec file, any patches and other sources that are not archives. A Makefile and sources file are also generated along with a .gitignore to provide useful functionality during the koji build

Other functionality is currently under development:

• skein push – Once extracted and committed to the git repository, this pushes the git commits to github.
• skein upload – Once extracted, this uploads the content of the lookaside directory to the lookaside cache at pkgs.gooselinux.org.
• skein import – A combination of skein extract, push and upload, since that seems fairly logical.
• skein build – Albeit mostly complete, it will need to be tested with an SRPM that has been run through this process.

I took a few minutes the other day and created a video of the completed process. I post this here for others to use for their benefit, but also so I can have it in another place besides my laptop.

Skein video

Cheers,

Herlo

Well, I have been meaning to write this post every day this week, only to be bombarded with work or other projects getting in the way. However, I am currently sitting at the Beans & Brews at 45 East 11400 South in Sandy, Utah participating in the very first GoOSe Sprint.

My goal today is very simple, get skein 2.0 out the door for testing. Most of contributors will be using skein to build the upstream packages necessary to make the distribution work, so I need to finish this work.

There are other tasks being performed today, including documentation, package building, automating tasks and finishing up the new website.

If you have any interest in seeing how an enterprise quality distribution can be rebuilt using free software, we’d love to have you stop by #gooseproject on irc.freenode.net. The GoOSe mailing list is another great place to read about what we’re doing.

Or if you like physical space, come on by the Beans & Brews today from 10am until 2pm. There should be at least a few of us working on something all day long. Just look for the nerds in the corner getting all excited while discussing phrases like ‘build process’ and ‘skein grant’ :P

Update: We’re going to be moving down to the Beans and Brews at 155 East 13800 South at approximately 12:45pm

Cheers,

Herlo

The project I’ve been working on does require a bit of thought around enterprise Linux versions run by a community. There is the ever amazing CentOS, Scientific Linux and a few others who have been around the block a time or two. The work that they have done has been immense and very helpful to many, including me.

For my project, the work was about building a fully binary compatible, enterprise-ready, community version of Linux, very similar to what CentOS and others have done. The question always comes ‘why?’ which will be addressed later on in future posts. Suffice it to say, the work we’ve been doing has paid off in both a individual and community sense.

In the beginning of this project, it was clear that we needed some tools to make things work the way we wanted. Luckily, there were tools out there to do a good portion of our work. Tools like koji, mock and of course Linux to bring it all together. But other tools seemed to be missing and I went on a quest…

The first tool that seemed to be missing was a way to import src.rpms from the most popular upstream vendor. These packages needed to be rebuilt by koji in some fashion, but just taking the srpms and rebuilding them had been done before, and seems to be the preferred way to date. In my mind however, it seemed that we were missing a step. Enter skein.

While skein is still very green and will need quite a bit more work, it accomplishes the goal of extracting the srpms into two parts. This tool basically sets up two things; a git repository on github.com (for now) along with a location and verifiable way to store the archive stored inside the srpm, called a lookaside cache. If one looks at the way the Fedora Project maintains their source, this process is very similar.

Once the srpm is imported with skein, it can be built with koji. At the moment, this process is fairly manual, but the plan is to improve skein to also allow building from the repositories. However, another bit more automatic way to build would be to use a git hook. Luckily, github provides several ways to accomplish this, including a custom URL to which an HTTP POST can be sent. At which time, koji would download the spec file and source from the appropriate locations and build a srpm.

Koji completes its task by building the binary RPM(s) and appropriately tagging the successful builds. Once complete, mash can be used to generate custom repositories to prepare for composing actual iso images. Mash is a command-line tool, again used by Fedora.

Once the repositories are generated by mash, pungi takes over. The process of building an iso is actually very simple, just a kickstart file, some repositories and pungi are used to create a fully installable DVD or multi-CD iso image.

Here’s a bit of my excellent artwork to better describe the process.

A couple things to note about this process is that while it is starting to become clear how to build an OS from an upstream vendor, there are parts that still haven’t been addressed. Currently, we can import with skein and rebuild the SRPM and build the binary RPMs with koji. We yet to have enough binary RPMs to actually construct a buildroot, but we are getting very close.

Automating the builds with git hooks and a skein build process is a nice big step toward making our own Operating System possible.

The other big piece of the puzzle is dependency resolution. Now this has been mostly resolved by tools and apis like Yum and RPM, but I still feel very much like a n00b when working with them. My hope is to figure out that process in the next week or so, and update skein to make building faster and easier overall.

Cheers,

herlo