madhadron

Q&A for The Programmer's Hangout Discord

I was invited to do a question and answer session for The Programmer’s Hangout Discord server. They kindly let me reproduce the results here.

Question: If you could go back in time give young Madhadron a single piece of career advice, what would it be?

Be kinder.

I was a pretty harsh human being for many years, to the point were coworkers didn’t want to give talks in front of me. I uttered the phrase “I’m not paid to be nice, I’m paid to be right” too many times. But there was no reason for it in most cases. Everything I was aiming for could have been accomplished more easily and more permanently with kindness.

Question: What is the biggest change of mind you have ever got?

When I started university (in physics), I was focused on “how do I derive this from first principles?” It took a lot of years of study before I realized that was really backwards. The experiments are the fundamental thing. The mathematical structures are summaries of those, images cast in the mirror.

So when I was asking how to derive this from first principles, looking for axioms, it was the wrong question. The right one was what is the collection of experimental data this is summarizing and what idealizations have we made to get our mathematical shadow of it.

Question: Why did you decide to leave science and go into pure software development?

Ah, this one. So I actually started out in software before university, and then went into physics. I spent a decade training to be a scientist. It took me that long to realize that I am, by temperament, an engineer, not a scientist. When I’ve cobbled an experiment together to get a result, my first thought isn’t, “Right, next experiment.” It’s “how do I make this repeatable at the press of the button for the next hundred people?” This is great if you’re a software engineer, not so good if you’re a scientist.

Question: You mention that coaching was an enormous help to you. How do you tell a good coach from a bad coach, specifically in computer science?

Oh, this is a good one. Umm…so the three people I directly dealt with that coached or mentored me in CS in my teens were my old boss (now deceased), a professional software engineer who specialized in enterprise IT systems on Windows and a grad student in programming languages. The latter two were folks I knew from my IRC Dungeons & Dragons groups. I’ve never met them in person, never knew their real names.

I would say that a good coach is one that points you in directions that change your thinking in ways that make you nontrivially more powerful, that makes you feel excited about the subject, and that can concisely orient you in the landscape that you’re groping around. It needn’t be purely technical things. I have had young engineers claim that a simple explanation about incentives and company politics changed their lives, or explaining cash flow and businesses. Models that let them interpret previously vague or invisible parts of their world.

Question: Were there any books / resources that you would consider particularly important in your development as a programmer?

Lots and lots. I spent a lot of time with both Alan Kay and EW Dijkstra’s works. They’re diametric opposites, but there’s a character to their thinking (Kay called it “playing it grand”) that both share, and working through the mental dissonance of these two utterly different approaches was revealing.

In terms of books, I would say Leo Brodie’s Starting Forth and Thinking Forth and Ullman’s Elements of Standard ML were the most influential books that I remember. I also spent a lot of time looking into the history of the field, reading accounts from people who designed or worked on Burroughs machines or NeWS at Sun or lots of other stuff. That kind of oral history contains most of the “why” of our field, and I think it’s vastly underappreciated.

Question: When you took a dive into the world of cosmosdb, did you have any existing specialties in databases or did you get deeper into that kind of knowledge “on the job”?

Ha! I went from being the owner of Facebook’s main ads database to CosmosDB. I had written Facebook’s large scale systems class for new hires and was regarded as part of the MySQL core team in all but official org chart. So, yes, databases were a thing for me.

Databases are one of those things that I’ve been doing in some form since the 1990’s, so I was quite proximate when that’s what needed diving into and I became a specialist. Now, I don’t know any of these databases the way that some of Facebook’s guys knew MySQL. They were amazing, could tell you every lock taken in the code and diagnose things that were like black magic even to me.

Question: Other than Discord, what are some mediums you use for teaching programming? (for example: Stack Exchange, HackHands, etc..)

Aside from being everyone’s crazy uncle on Discord, I mostly teach one on one. Right now I’m using USE Engine and VSCode to teach a friend’s daughter. Otherwise it tends to be in person. If it weren’t for covid, I’d be teaching my current student in person, too. I write for my website a lot. I guess that’s the other major medium.

Question: I also received a lot of invaluable coaching via IRC in the 90’s and early 2000’s, do you feel that discord or other chat mediums provide in the same way that IRC used to?

I have found that the medium is less important than the individual cultures and milieus within the medium. There are Discords that are dead or are horrible. There are super functional ones, such as TPH and a wonderful writer’s chat I know. The same was true for IRC. My IRC life centered around a very functional, close knit #adnd on Dalnet back in the day (long gone now). I’ve also found incredibly functional cultures on email lists, and completely broken ones. The biggest thing is having a critical mass of core people who share expectations about how things operate and ops/admins who can enforce those.

Question: with all your experience in various languages, and architecture, how do you feel about rust? Do you think it will take over from Java/C++(eventually)?

I truly don’t know. If it gets traction as a major language inside somewhere like Facebook, perhaps it could. I have observed that programming languages tend to prosper or disappear for reasons that are largely incidental to their design.

Question: What formulation (i.e. interpretation) of quantum mechanics has your highest baysian prior (i.e. most likely correct)?

Well, I’m convinced that it is nonlocal and quantum states are ontic. I have yet to find a collapse scheme that has seemed like an utter hack. So I seem to be falling back to Everettian more and more just because nothing in it bothers me.

Question: What lead you to joining this server?

A degree of loneliness. I had been very involved in a martial arts mailing list for some years, and was kind of wandering around looking for something to fill that gap of a virtual community.

Question: What kind of non-technical skills have helped you as your time in programming/software development?

Planning and organization, certainly. I’ve been operating an implementation of David Allen’s ‘Getting Things Done’ since 2002, which makes me one of its longest running practitioners, as far as I know. The ability to lighten a conversation, to make people around me smile has been critical, I think. Speaking and writing well are useful. Being able to read critically and accurately is essential. And just being able to listen to someone and lead them out until I understand what they’re saying, and then trying to help them as well as I can.

Question: What is your favorite mathematical object/field of study?

I’m not going to say spinors, because SO(3) is like crack cocaine to my brain. I know I’m in a bad place if I pull those out. I feel like my favorite ones are probably semigroups and semilattices for the sheer amount of organization of my world I get from them.

Question: Out of all the languages you’ve worked with, which one is your favourite?

None of the above. I categorize languages by what I loathe about each of them.

Question: What personal characteristics do you believe are helpful for success in your field?

I have seen so many different people and so many different paths that I rather hesitate to try to guess. We all have different paths, sometimes very different. I have seen assholes thrive and those who care deeply for those around them. I’ve seen vague, big picture thinkers throwing code at the wall do amazing things and people with utterly precise minds. I guess the sole thing is the ability to communicate and collaborate. We’re all in over our heads. Our brains weren’t evolved for this and we need help.

Question: What is most rewarding about your job? What is the most difficult thing?

So this is a bit nuanced. I have avoided becoming a manager, because the day to day production of code is a source of base contentment. Feynman described something similar how he didn’t want a position without teaching because you had something that you were supposed to show up and do and you could succeed at even if your other work was a giant burning mess. I don’t like spending all my time down a coding rabbit hole anymore, and I know it’s not healthy for my mental state (again, people vary). After that I take enormous satisfaction in mentoring and teaching and helping my colleagues talk through things. I am, to the shock of many that know me, a people person.

Question: What’s one piece, or two, of advice you can offer a new grad who lacks experience; whether the advice is technical, personality, anything. Perhaps better phrasing would be, what are one or two things (not limited of ofc) you believe has helped you throughout your career and in life for that matter?

I really, really hesitate to offer advice to new grads in software because I never was one. I never studied CS, never had a programming class, and didn’t take a software job until I was skilled enough where I was being handed independent projects and half project managing to start. Maybe, don’t be afraid to ask stupid questions? If you don’t understand it, probably someone else in the room doesn’t either.

Question: Do you regret not focusing on programming as your career instead of working in a such wide range of fields?

Absolutely not! Even now I work on a huge range of stuff in my own time. I’ve got books queued up on Italian medieval history, developmental biology in the Cambrian explosion, studies of short form in English poetry… I have made much of my career off what I jokingly call “intellectual arbitrage.” And when you get bogged down in one field and it’s not fun, you just go pick something else up. There are lots of people like me. David Epstein’s book ‘Range’ pointed that out, which was nice. You also get a sense of what is fundamental and what is not as you dig into a field. A feel for “this is fluff and ephemeral” versus “this will last me at least a decade.” And you learn to avoid the former.

Question: You have experience with a lot of…atypical languages. It’s pretty to common to read that “learning (FP/stack-based/other less common paradigm) really changed how I think about problem solving on a regular basis”. Do you find that’s personally true, and if so, could you give a concrete example?

I’m a terrible, terrible person to ask that question of, because I learned C, SQL, Forth, and Standard ML at around the same time. So I have designed data structures in Hindley-Milnor type systems like Standard ML and Haskell since I was a teenager and translated them into other languages. I’ve modelled data relationally and then recast it into objects or vice versa. I think the biggest impact is that I reach for what I think the most incisive tool I can for a task is, ignoring whatever else is there, and then translate back out of it. My classmates in physics used to refer to this as “math hammers” because I would bring out crazy stuff to demolish integrals and the like.

Question: Do you think starting and being successful in the software field requires starting < 30yrs of age?

No. I have a friend who is an amazing applied mathematician. He left Boeing and did some time at Tableau in his early 40’s as a software engineer. Then went and did other things.

Question: Since you’ve mentioned Baroque violin, can you give a little summary of what that part of your career was like?

So I’ve played violin since I was four years old. My parents played mostly early music recordings around the house (Christopher Hogwood and the like), so I grew up with Baroque as my native musical idiom. When I got to university I found that I didn’t much like the orchestra. Too regimented, and I didn’t like romantic music much. But I found a home in the early music ensemble. We used to joke that it was where you went if you were too weird for the orchestra. A lot of physics and math majors… We did our concerts at the school, and we would sometimes go out and fill in for professionals in the area since there isn’t exactly a huge early music scene in that part of the country. When I started grad school that all fell by the wayside, though I continued to do some historic dance reconstruction.