Wednesday, July 6, 2016

Remembering Roger Faulkner, UNIX Legend

Roger Faulkner, UNIX engineer since 1976, SunOS/Solaris developer since 1990, creator of /procfs, passed away this past weekend.
Photo by Sherry Q. Moore, 2010. 

Roger Faulkner, or raf as his co-workers knew him, was intelligent and had no patience for fools. He was always happy to share history of UNIX, libc, /proc or any other kernel internals, or his opinion on how things should continue to be improved. If you broke the gate in any way shape or form, he'd let you know within a few hours - and if you blocked his project with this breakage, he would not hesitate to let everyone know. He was an amazing colleague with a wry sense of humor and will be missed.   Most folks remember him as that really smart guy that was tough on the outside and sweet, gentle and kind on the inside.  (and seeing that sweet inside wasn't so hard :-)

Roger did not care how senior the engineer (or management) was - he would not let them get away with things that would hurt UNIX or Solaris. He was also always willing to answer questions, do a code review, or help debug an interesting kernel dump. I learned a great deal from him - how to be a good engineer, how to do an excellent root cause analysis, how to know when a bug is really, truly fixed. 

UPDATE July 7, 2016: Obituary is posted here online.  There is a tribute page there as well, where you can leave notes for his family and friends.

UPDATE: July 20, 2016: Roger's Memorial will be livestreamed Saturday 7-23-16, 1p EDT/10a PST/6p BDT.   Streaming will begin 30 mins prior to the memorial service. Youtube channel memorial broadcast:  Memorial Service.

Roger's more extensive bio, courtesy of his manager, Rob Stephens:

Roger grew up in North Carolina and earned a BS in Physics from North Carolina State University in 1963 and a PhD in Physics from Princeton University in 1968. He became involved with UNIX in 1976 when he helped set up and enhance a UNIX service at Bell Labs, Naperville, IL. Roger returned to Bell Labs, Murray Hill, NJ, in 1979 where he continued to work on UNIX development for two years. He moved to New York City in 1981 to do something entirely different for four years, but he couldn't stay away from UNIX. Roger worked at Unix Systems Laboratory 1986-1988 attempting to develop an application debugger for System V Release 3. The result was the first /proc file system for System V and the truss(1) utility for tracing/displaying application-level system calls.

Roger joined Sun Microsystems in 1990 to work on the merge of AT&T's SVr3 and SunOS4.x to create UNIX SVr4 (a.k.a. Solaris 2.0 at Sun). He then concerned himself with the definition, exposure, and maintenance of the Solaris/UNIX process model, with emphasis on visibility into and support for debugging application programs:

From 1990-1993 he extended the ioctl-based /proc interface from being a single-threaded process model to being a multi-threaded process model with lightweight processes within the traditional process.
Photo by Sherry Q. Moore, 2010

In Solaris 2.6 (1995-1996) Roger created the structured /proc file system, with each entry under /proc being a directory rather than a file, each pid directory under /proc containing individual files and other directories reflecting the full process model for both inspection and control. Programming interfaces defined by the proc(4) manual pages.

In Solaris 8 (1997-1998) Roger created the alternate libthread as a better support library for multi-threading. It is a one-to-one thread/lwp interface rather than the old N-to-M thread/lwp interface implemented in the original Solaris libthread. The alternate libthread become the only threading library in Solaris 9.

In Solaris 10 Roger created the unified process model in which all threading support is folded into libc. All processes became multi-threaded, in principle, eliminating the confusion of having three separate process models as was the case previously. Eliminated static linking of the system libraries; all processes are dynamically linked.

Roger then implemented system changes to enable Solaris 11 to conform to the latest POSIX standard (UNIX V7).

[Solaris 12 work redacted, but let it be known, he's done a lot.]

Roger also lent his expertise to countless Solaris projects and was generous with his time and knowledge as he helped many engineers develop their own expertise about all things UNIX. Roger's dry sense of humor, his chuckle, his irreverence for management, his passion for UNIX, and his inspiration will be missed by everyone who had the privilege to work with him.  Please feel free to share this as it is impossible to include everyone Roger worked with over his many years.

I think Sherry Q. Moore really summed him up in her Facebook post:
 What I learned from Roger:
- You can be brilliant and kind.
- "If you don't have time to do it right, when will you have time to do it over?"
- You can be creative and productive for as long as you want.
- "When you are about to do a putback, if your heart is not pounding, palms not sweating, you shouldn't be doing this (be a kernel engineer) any more."
Meem (Peter Memishian) shared the following (Note: the below source comment can be viewed freely online in context):
Indeed.  Today I lost one of my professional heroes.  As those on PSARC
well know, Roger cast a shadow far beyond his truly immense technical
contributions to UNIX (and Solaris in particular).  His curmudgeonly
outwardness belied a remarkably gentle and caring internal character.
Despite having forgotten more about UNIX than most of us could ever know,
he was as grounded as they come, with a unique style that left indelible
memories on so many of us, and altered the DNA of our engineering culture.

Speaking personally, I've always admired those who prioritize doing over
talking.  Roger was one who quietly moved mountains -- as Bryan captured
in the approval of Roger's RTI which put the final nail in the coffin of
the M-to-N threading model:  

And of course, Roger wasn't afraid to speak his mind when necessary --
as captured in this gem above cv_wait_stop():

  * Same as cv_wait(), but wakes up (after wakeup_time milliseconds) to check
  * for requests to stop, like cv_wait_sig() but without dealing with signals.
  * This is a horrible kludge.  It is evil.  It is vile.  It is swill.
  * If your code has to call this function then your code is the same.

Finally, I'd like to share this mail from many moons ago on the history of
the name "truss", which embodied the soul that he infused into his work.

 | From: "Roger A. Faulkner" <Roger.Faulkner@Eng>
 | To: meem@Eng
 | Subject: Re: curiosity: truss?
 | Date: Wed, 27 Jan 1999 23:34:47 -0800 (PST)
 | For your edification, this is the geneaology of the name "truss"
 | (taken from some mail dated Sep 26, 1988)
 | This was when Ron Gomes and I were jointly developing the first
 | /proc for SVR4 at USL.
 | -----------------------------------------------------------------
 | We considered, and discarded, several alternative names for truss(1),
 | including "trace", before settling on "truss".  The objection to
 | "trace" is that it's too generic a term and shouldn't be co-opted
 | for a specific use like this; there are lots of other things that
 | one might trace.  Among the alternate names we considered were:
 | "ptrace"  (but this incorrectly implies a connection with ptrace(2)),
 | "strace"  (but this is already used for some streams tracing thing),
 | "tss"     for "trace syscalls and signals" (but this is certainly bad),
 | "sst"     a permutation of "tss" (but this implies it's blinding fast),
 | "trss"    another variation of "tss" (but this is unpronouncable).
 | Adding the obvious vowel gave us "truss", which can be construed
 | to mean "TRace Unix Syscalls and Signals".
 | "truss" seems to have the right combination of mnemonic value
 | and disrespect for authority ("If your program doesn't work, put
 | it in a truss.")  It conjures up a mental image which is fairly
 | accurate, considering what the program does.

Rest in Peace, Roger.  May all your RTIs be promptly approved. 
Tim Foster did an in memoriam integration into the ON gate for Roger. He will live in Solaris forevermore.

Below are some tweets I saw passing by.... Please share your own thoughts below, or send to me and I will share them here.

Do you have any memories of raf? Please share in the comments or in your own space.

We are collecting pictures for his family and friends in the Roger Flickr group. Please add your own there.

Don't worry Roger, someone will approve your RTI.