Today at SCaLE10x: System Administration Study Group

If you haven’t heard my ranting and raving about the SysadminSG last weekend at FUDCon, it went pretty well. Turns out, this weekend is another run of the event. The goal is to help Fedora contributors (and others) prepare for the tough world of Linux system administration.

If you attend the event, you will receive two study guides and an ec2 instance for about 24-hours. The idea is that you will spend the day today studying with others to improve your system administration skills. There are no instructors, but I will be there to get the ec2 instances setup and help with any non-study related issues.

If you are interested in the sysadminSG,  come visit us at SCaLE10x today:

Southern California Linux Expo 10x (SCaLE10x)
Los Angeles Airport Hilton – Catalina C
9am – 5pm

Cheers,

Herlo

FUDCon Blacksburg: sysadminSG, Fedora Infrastructure, pam_otp and more!

Well, FUDCon Blacksburg has come and gone. I believe it was the most productive I’ve been at a FUDCon EVER! I was able to get quite a bit accomplished and much more than I had planned!

What was it that I did, you ask? Well let me tell you all of the tasks I was able to complete.

System Administration Study Group

Over the past year, I’ve been working on a way to do a regular free study group for those who want to improve their system administration skills, get certified, or just brush up on something they didn’t quite understand.

Enter sysadminSG, a self-guided tour of many of the standard tasks any system administrator should know. The main goal was to create an instance of a Fedora 14 machine for studiers to use.

Some of the tasks on the study sheets indicate certain issues that would need to be resolved which can’t be resolved without a ‘master’ instance. I was able to recruit a couple of excellent individuals to help get this further than I thought it would. Jon Stanley and Ivan Makfinsky helped put together many of the pieces which will help studiers get more done. Specifically, an ldap server for centralized authentication and iscsi target luns for use with LUKS encryption, LVM and disk partitioning.

Fedora Infrastructure

I attended the Fedora Infrastructure session, where we covered two major things of concern. One was removing the puppet staging branch and moving everything into ‘Production’. This is a mental shift for many, but makes sense because in truth, everything maintained from an infrastructure perspective is production.

Additionally, a longer term plan of being able to spin up ‘staging’ environments for any new things that will go into production looks to be the direction we’ll be going. Many of the applications we have in Fedora require some work to get us to this point, mostly so they can be more atomic pieces. I think a proof of concept environment will provide us with an good idea of how much work will be involved.

pam_otp

After the Fedora Infrastructure hackfest, there was a two-factor authentication hackfest. We discussed using Yubikeys and a unique PIN together for authentication within Fedora. The initial goal was determined to setup sudo authentication for sysadmin-main group with two-factor authentication.

Nathaniel McCallum found the pam_otp library and was able to get it to compile. He passed it on to me to test and document it, which I was able to get working after a drunk night by the fire. I was then able to document the usage of it and have a test rpm that we’ll be using.

All in all, it was a very productive weekend and more work will be getting done over the next few weeks as well. It’s all very exciting and fun, a really good reason for attending FUDCon!

Cheers,

Herlo

FUDCon Blacksburg 2012: Three days to the Sysadmin Study Group

This Saturday, January 14, 2012 at approximately 1pm Eastern Standard Time, I will be running the System Administration Study Group (SysadminSG) at the Fedora Users and Developers Conference being held in Blacksburg, Virginia.

We did a successful version of the SysadminSG at the Southern California Linux Expo (SCaLE 9x) last year.  And thanks to Ben Williams prodding several months ago, I agreed to run this event again. Note: We will also be running the SysadminSG again at SCaLE 10x on January 21, 2012 as the Fedora Activity Day (FAD).

What is the System Administration Study Group (SysadminSG)?

The System Administration Study Group or SysadminSG is intended as a gathering of individuals with a common purpose, studying and preparation for the Red Hat Certifications. The two main certifications of focus are the Red Hat Certified System Administrator (RHCSA) and the Red Hat Ceritfied Engineer (RHCE).

This is a self-guided workshop for those who gather, it does not have any instructors. However, proctors will be around to assist as needed.

The SysadminSG could be used for many other reasons. The Fedora Infrastructure team is always looking for skilled system administrators. The study group is a great way to bone up on the technologies used by Fedora Project administrators every day.

Why should I participate in SysadminSG?

Because it’s cool, and other reasons, too!

Okay, while it is true that participation is the cool thing to do, it is also very possible you may wish to improve your system administration skills. Doing so with others in the room to bounce questions off can be very helpful. Studying by yourself is challenging and while IRC is nice, there really is nothing like real live people in a room, with similar goals.

So I am interested. What do I need to bring/provide to get me started?

To participate in this event, please bring a laptop, PC or some other computer with a current version of Fedora pre-installed. YOU DO NOT WANT TO INSTALL THE DAY OF THE STUDY SESSION!!

Amazon ec2 instances will be used where possible, so that things can be portable. The instances should be up and running for around 24-hours, which means if you don’t finish during the study session, it’s possible to continue for a short time after the workshop. To access an ec2 instance, it will need to be setup with an ssh public key, so come prepared with one.

Note: If the machine you bring has the capability for Virtual Machines using KVM or VMWare, we can work with that as well, but you may receive mixed results.

Resources! Pig for Wheat!

Okay, stop screaming about it! There will be plenty to do during the 4-hour session, but if you want to get a head start, read up on these guides.

Contributions encouraged and accepted

One of the main goals of the study group is to come up with tasks that would be representative of each of the objectives listed. It is my hope that with a bit of contribution during the session, we can discover and improve the study group to make it a regularly occurring workshop. I would like to see these study guides go from big sections of blank to something more tangible. It’s possible each objective could have two or three tasks for a future SysadminSG workshop.

See you all at FUDCon this weekend!

Cheers,

herlo

Presenting at PLUG Tomorrow: GoOSe Linux – Rebuilding Enterprise Linux the Community Way

Well, this has been a long time coming. It’s taken over 6 months of hard work by our community. Tomorrow night, January 11, 2012, I will stand in front of the Provo Linux User Group (PLUG) and talk about what we have been working toward.

GoOSe Linux – Rebuilding Enterprise Linux the Community Way

Yes, GoOSe Linux is almost here and we’re ready to discuss the process and the goals of our little community. If you have been hearing me rant about GoOSe on the Utah Open Source Planet, Google Plus or Facebook and want to hear more. Or if you are just plain bored tomorrow night with nothing better to do, come down to the Provo Linux User Group. Learn more about how the Enterprise Linux Rebuild community is working together to make a better ecosystem.

If you can’t make it, or want to preview the slides, you can get them on my speakerdeck.com page. I look forward to seeing you all there.

Date: January 11th, 2012
Time: 7:30 PM
Location: C7 Data Centers (Lindon)

For more information, check out the PLUG announcement.

Cheers,

herlo

More Sysadmin Study Group than you can handle! FUDCon Blacksburg and SCaLE 10x

As mentioned previously, there will be a Sysadmin Study Group workshop at FUDCon.

Now we’re announcing the Sysadmin Study Group will be the Fedora Activity Day at SCaLE 10x!

So Exciting!

Well, it’s been a week since the first run of my ec2 instances for the SysadminSG (or System Administrators Study Group).Things look promising. The folks over in #fedora-cloud and #boxgrinder have been especially helpful! Thanks a ton msavy and gholms and rbergeron too!

The plan here is to grant every user in a room access to an ec2 instance for 24-hours. It may be shorter or longer, depending on a few unknowns at this time. After some additional testing tonight, I think just a bit more work and the workshop will ready. We still need to update the Study Guide to match the RHEL6 RHCSA exam and RHCE exam requirements, so if anyone wants to take that on, just let me know and I’ll point you in the right direction.

If you are looking to do some RHCE prep with others, come on down to the Sysadmin SG workshop at either FUDCon Blacksburg or SCaLE 10x! We look forward to seeing you there!

Cheers,

Herlo

FUDCon Blacksburg: System Administrator Study Group

FUDCon Blacksburg is a little less than 3 weeks away and I can’t believe it’s almost here! So much to do, so little time, it will be a great event!

As some of you may know, I’m putting on a workshop, called System Administrator Study Group. If you haven’t heard about this workshop, listen up!

SysadminSG as it’s called, is to get Fedora contributors prepared for certain certification exams, specifically the Red Hat Certified System Administrator (RHCSA) and the Red Hat Certified Engineer (RHCE).

This workshop is not a place where you can come and earn either of these certifications. It is also not a course where you can get free instruction. Instead, this workshop is a hands-on place to sit down with others of like mind and work through the exam requirements as specified by Red Hat. If you have ever desired to get your RHCE certification, or just want to bone up on some of the qualifying skills, this is the place for you.

A Test Run – 18:00 to 21:00 UTC – December 27, 2011

My hope is to have everything working on ec2 instances for participants in the workshop. In that sense, I’ll be setting some things up in the morning tomorrow to make certain things work. It is clear to me that I’ll need help from a few folks to test the setup I’ve created.

If you have an hour or so tomorrow during the above time frame, I’d love your help testing these instances. Please find me in #fedora-cloud on freenode (I’m herlo) and I’ll set you up with your own 24-hour instance. I’ll probably be in there asking gholms and others for help getting ec2 policies working, among other minor issues.

Cheers,

Herlo

 

Skein: Updating the build process (somewhat)

The original post is in the GoOSe mailing list. I am reprinting here for a wider audience.

Last night, Mike (shalkie) and I had a nice discussion about the process we are now using to build / import packages. This is in preparation for now and the future. A lot of this discussion is thanks to Mathieu Bridon (bochecha).

I have inserted a few notes in the process below for clarification.

02:03 < herlo> k, so you should know that you first ‘skein request’ a package
02:03 < herlo> it can be done with either the –path or –name option, but one must be provided
02:04 < herlo> normally, one shouldn’t grant their own package. I should be around some tomorrow to grant packages in bulk for you as needed
02:04 < herlo> but
02:04 < herlo> if you do need to grant a package
02:04 < herlo> it should be something like
02:05 < herlo> skein grant -k <koji owner> -g <github owner> issue_number name

Package repos and koji tags are what get created with skein grant. However *only* an admin can grant a new package into our build environment. This is to make sure we are not putting packages into the process that are not part of the upstream and/or for a good review
process. In the future, I suspect quite a few people will be admins and this will be less problematic.

02:05 < herlo> once granted, the package can be imported with
02:05 < herlo> skein import /path/to/srpm
02:06 < herlo> usually though, I do take one extra manual step.
02:06 < herlo> I visit the repo page on github and add a service hook
02:06 < herlo> if you look at this page: https://github.com/gooselinux/libgnomecanvasmm26/admin/hooks
02:07 < herlo> you can see the ‘Post-Receive URLs (1)’
02:07 < herlo> if you click on it, you’ll see the url you should add for any new repo
02:07 < herlo> this post receive hook will automatically launch builds upon a commit

If you have admin access to a repository (and if a package has been ‘granted’ to you, then you do), add the following in the Post-Receive
URLs if it’s not already there:

http://roman.gooselinux.org:8080/add

02:08 < herlo> once you have that in place, run the import command described above
02:08 < herlo> at that point, everything should be pretty automatic.
02:08 < shalkie> k
02:08 < herlo> within 10 minutes, the build should automatically launch
02:08 < herlo> shalkie: seem pretty straightforward?
02:09 < shalkie> Yeah it does.
02:09 < herlo> k, so now it should also be clear that you may (and probably already do) have failed builds
02:09 < shalkie> The stack of email certainly suggests that.
02:09 < herlo> lol
02:10 < herlo> shalkie: a good portion of those can be run through again, though I have been working on fixing a few of mine
02:10 < herlo> shalkie: as you look through the failed builds, you may just need to run them again
02:11 < herlo> to do that, you can use
02:11 < herlo> skein build
02:11 < herlo> the way I usually do that is ‘skein build –nowait dist-gl6 <pkg_name>’
02:11 < herlo> only an admin can perform this task (you are an admin)
02:12 < herlo> you can watch the tasks at http://koji.gooselinux.org/koji as you always have, or go out for a
few hours and check the build statuses when you return

02:13 < herlo> BUT WAIT! There’s more!
02:13 < herlo> If for some reason, you determine something has to change in the spec file, like another dependency or something
02:13 < herlo> you’ll need to do two things
02:14 < herlo> first, clone the branch if not already there and then create a vanilla branch
02:16 < herlo> git checkout –track -b  <local branch> <remote>/<tracked branch>
02:16 < herlo> eg git checkout –track -b vanilla origin/vanilla
02:16 < herlo> then push that vanilla branch
02:16 < herlo> git push origin vanilla
02:17 < herlo> this is so we can keep the original state of the package before we make any changes. I’ll be working on  automating this later
02:17 < herlo> when vanilla is pushed, we’ll go back to master and make our changes
02:17 < herlo> git checkout master
02:17 < herlo> <make changes>
02:17 < herlo> git add <files changed>
02:17 < herlo> git commit -m “message regarding changes”
02:18 < herlo> git push origin master
02:18 < herlo> again, when you push to master, a build will be launched within 10 minutes
02:18 < herlo> sit back, enjoy coffee and wait for the build to return

At the moment, the ‘master’ branch is currently what is getting built. Vanilla never gets built, but serves as a reference point to what the upstream did at certain points along the way. The vanilla branch also provides a way to merge in upstream changes with our changes. Git will
make this easy for us, because merge conflicts will be much easier to resolve across branches.

I hope this little conversation is a bit clearer on the process we’re trying to use. Look for further improvements coming to skein 2.1 (I already have quite the feature list) in the next few months.

Cheers,

Clint

PS – I likely won’t be online when this post is published as I’ll literally be enjoying some GoOSe, prepared by my lovely wife. She is an amazing cook / chef and is attempting this for the first time ever!

The GoOSe is getting cooked during the holiday!

The original post is in the GoOSe mailing list. I am reprinting here for a wider audience.

As of this morning, it appears we only have around 40 failed builds! We’re getting very close! I think we could make an alpha by the end of next week with a good bit of sprinting! As I understand it, one is planned for next week, maybe we should sit down and figure out those details?

Cheers,

Clint

Flying through RPM Builds with Koji: Pass #1 Results

Well, it looks like we’re just about finished with Pass #1 to build all of the imported SRPMs we can from upstream. The GoOSe Project results are looking good.

Build statistics from 2011-11-01 through 2011-12-09

Completed Builds: 658
Failed Builds: 296

It appears that a bit more than 1/3 of the builds failed and notifications are going out to the package owners. This means we can work through each of them and resubmit them. Odds are that most of them are due to dependency issues and only a few for other reasons.

We’re preparing for Pass #2 this weekend. We’ve discovered some interesting things, however, so here’s something we’ll be putting in place first (hopefully):

A repository to track BANNED and EXCLUDED (for now, other names may arise) SRPMs.

At the moment, there are SRPMs we cannot build because we don’t support an architecture (s390, for example) There is no reason to build packages that won’t work. There are also some packages that contain proprietary content we need to scrub, even though the license states they are free software. These packages will have to wait unless they affect a critical path of getting GoOSe 6.0 composed in coming weeks.

The plan here is to create a repository to track these banned / excluded SRPMs. Skein will query this git repository for the BANNED and EXCLUDED files to make sure any request, grant, or build actions first check to make sure the package is not in either list. If the package is in the list, skein will refuse to act upon the package and the requester will have to resolve the issue before attempting again.

Watch for the updated skein v2.1 with this fix in place in the very near future.

Cheers,

Herlo

Releasing skein 2.0 into the wild – Grapple gets a client function

The GoOSe Project has been very busy over the past two weeks. This evening, after dinner, I will be releasing skein 2.0. It’s major functionality will be documented and placed in the RELEASE.rst file on github. Essentially, the request, query, grant, import and build functionality is what makes this 2.0 and ready to use.

The best part about this tool is, while I have been working on it, almost all of the remaining SRPMS have been imported and the majority of them have been built on our koji server. Right now, we’re on the last batch and I expect it to finish sometime late Friday or early Saturday (12/10/2011). This ends our first major hurdle to getting an Alpha release of GoOSe Linux 6.0 out the door.

Another huge tool in this process is Grapple, written by python master Nafai, (aka Travis Hartwell). He spent a good bit of time getting this github hook in place that will record all git commits and help to process them automatically. I spent a bit of time and added a client script to grab the recorded commits and send them on to koji automatically with the proper dist value and url. No more guessing! This is the ‘automagic’ corollary to skein build.

Next up, I plan to rebuild the failed builds and push them through koji again, using skein build. Possibly doing this a few more times until we get things pared down to the packages that don’t just have dependency problems. I hope to have most of those fixed up and in place by Monday or Tuesday next week (12/12/2011).I’ve also noticed a few builds that will never build for our arches, so I will have to figure out what to do with them for now.  I don’t want to waste koji resources trying to rebuild them when I know they will never work.

The first Alpha ISO is getting close enough to taste it now! We’ll have to make sure to have a grand celebration upon our first Golden GoOSe Release :)

Cheers,

Herlo