Interview with Soren Jacobsen

A couple of weeks ago, Guillaume Lasmayous and I threw the idea of interviewing NetBSD developers through our website, NetBSDfr, to promote the NetBSD Project, and to make their work known to the widest possible audience.

Today, we are discussing with Soren Jacobsen, snj@, NetBSD 5.0 release engineer.

NetBSDfr: First of all, let me thank you for this interview. We plan to run
several interviews of NetBSD developers, I think one every one or 2
months or so. You've been chosen as the first one. Thanks for accepting
to answer our questions.

snj: Thanks for asking me, and thanks for your efforts to strengthen the
community.  It's great to see efforts like this.
Sorry it took so long to get back to you.  I wanted to take some time to
think over my answers.  Please let me know if you think anything needs
clarification.

NetBSDfr: For the readers who don't know you, can you shortly introduce
yourself ?

snj: My name is Soren Jacobsen and I am a NetBSD developer and
release engineer.  I've been part of releng since late 2005, and was the
lead release engineer for the 5.0 release.

NetBSDfr: Why did you choose to run NetBSD ? How long have you been
using it ?

snj: My first experience with NetBSD was 1.5.3, which I guess was
around the second half of 2002.  Compared to a lot of people, I was a bit
late to the party.  At the time, I was frustrated with Linux in a number of
ways, and NetBSD was a solid operating system with a philosophy that
 made sense to me.  There wasn't any one particular thing about
NetBSD that grabbed my attention.  It was more that there was nothing
that bothered me.  It just seemed right.

NetBSDfr: Do you have an idea of the time you spend working on the
NetBSD project daily, weekly, monthly ?

snj: Like most volunteers, the amount I'm able to spend on NetBSD
depends on real life and my motivation level.  A year ago, at most I was
doing 8 hours a week.  More recently, 8 hours a day wasn't uncommon.
The week leading up to the release was especially hectic; I put in a
couple 16 hour days there.

NetBSDfr: How did you become a NetBSD developer ?

snj: When I started using NetBSD full time as my primary operating
system, I became heavily dependent on pkgsrc, our framework for
third-party packages.  There were a number of packages I wanted to
see updated, and I started sending patches against pkgsrc and various
small things in the base system.  At the end of 2003, I was offered an
account, and became a developer in January of 2004.

To anyone out there who is interested in helping the project: pkgsrc
is a great way to get involved.  There are so many packages out there,
and only a finite number of developers to import and maintain them.
Consider submitting PRs against pkgsrc and joining the pkgsrc-wip
project.  We can use all the help we can get in this area.

NetBSDfr: How did you become member of NetBSD release engineering
team ?
Are the members of the releng team elected ? chosen among
developers ? Does this require specific skills ?

snj: The release engineering team is rather informal.  There is no
election process, and no strictly defined leader of the group, although
people do step up to fill this position.  When we feel understaffed, we
 put out a call for volunteers, and usually a person or two will join.  

Release engineering is not as glorious as a lot of other activities, so we
don't normally have people begging to join releng.
It's hard to say about specific skills.  Like in most other areas, having
sound judgment and a good sense of perspective is essential.  These are
the only strictly required skills.  Plenty of others are useful, though:
patience, caution, attention to detail, etc.  Pretty basic stuff, for the
most part.  Of course, the more technical knowledge, the better, but this
is not as big of a deal as it might seem.  I'll explain below.

NetBSDfr: Can you explain us more in details how does NetBSD release
engineering process work ?

snj: There are two basic tasks:
- Pulling up changes into branches.
- Releasing.

The first one is rather straightforward, and it is something we do
constantly.  Developers request that certain changes be pulled up to a
given branch.  If we agree that the changes should be pulled up, we commit
them to the branch.  To help track the status of a branch, we maintain a
record of every change made to a branch over its lifetime.

Putting together a release is a much more complicated task requiring
coordination with a large number of people.  The main task is to decide
the point at which the remaining bugs can safely be ignored.  There are
lots of other tasks involved here, of course: managing the build cluster,
pulling up changes, preparing information about the release, testing,
making sure that important fixes find their way to the branch, begging
people to work on certain things, paying very close attention to the
mailing lists, etc.

NetBSDfr: How does the quality assurance process works ? Who decides
to include this or that patch to a branch ?  This is also linked I think to
the following question: how does the releng team decide how many
betas, RCs versions are required before declaring a branch is stable ?

snj: No one person can have thorough technical knowledge in all areas
of the source tree.  Therefore, it's sometimes hard for a person (or
small group) to be able to say for sure whether a particular change
should be pulled up to the release branch.  This is where one of the
most important qualities of a release engineer comes into place: the
ability to get sound advice from the right people.  To some extent, we
trust that developers submitting changes for pullup have carefully
considered the implications of the change.  But at the same time, we
try to be very cautious, and being able to effectively consult with other
developers is of vital importance. 

We start the release process when we think the tree is in pretty good
shape.  At this point, the branch is known as, e.g., 5.0_BETA.  At this
point, more users start running the code, and we are constantly watching
for feedback (positive and negative).  Intrusive changes are avoided
during all stages of the release process, but they are most acceptable at
this point.  The general rule is that the further along in the release
process we are, the more restrictive we are about changes.  Unfortunately,
it is impossible to have many hard rules about this.  If big nasty bug
comes up, you're faced with a choice between ignoring the problem or
taking a risk on an intrusive change.  Sometimes we take the risk,
sometimes we don't.  This is another case where no one particular person
can have all the answers.  To avoid one person making a bad judgment call,
members of releng discuss these issues with each other and with other
developers.

A release candidate, by definition, occurs when it is felt that the tree
is good enough to release.  In practice, this rarely happens.  There is
almost always a good reason to do at least one more release candidate.  In
some respects, they're more psychological than anything else.  I admit I
rushed a couple release candidates out with full knowledge that 5.0 could
not be released without further fixes.  But when important last-minute
changes are made, putting out a release candidate is a great way to get
those changes tested.  Take 5.0_RC3, for example.  At the time, we were
aware of a few show-stopping bugs, but there were significant WAPBL
changes that needed wider testing, so we felt that RC3 was justified.

In the end, deciding when to declare a release stable comes down to
compromise.  You have a hard list of requirements that the release must
meet.  Beyond that, you just have to get to a point where you feel that
the remaining bugs are outweighed by all the positives in the new release.
This is a hard thing to do, because when you've been tracking every
problem raised for months, you tend to be focused on negatives and can
easily forget about all the people who are having wonderful experiences.
Eventually you have to let go and realize that you'll never have a
bug-free release.  When you get to that point, you tag the final release
and hope that you haven't made a horrible mistake  :) 

NetBSDfr: As a conclusion, can you tell us how you imagine the future
of NetBSD?

snj: We've made a massive jump with 5.0, and 6.0 should continue this
trend.
NetBSD has been ignored by many people for years, but this is changing.
As for more specific predictions, I think we're going to see NetBSD usage
increase greatly in the server and embedded worlds.  I don't envision any
radical shifts in direction, but this is not a problem.  NetBSD has always
been a great operating system; we just need to get out there and introduce
more people to it.

Many thanks to Guillaume for his hard work on preparing, conducting and translating this interview and rendez-vous in a few weeks for the next one. In the meantime, any names of developers to interview, or questions you'd like to ask, are more than welcome.

Original post at NetBSDfr.