Notes on the Go translation of Reposurgeon

The problem that obtruded on me first was quite unexpected: absence of keyword arguments. In Python one can write a function signature like this

and then call it like this:

I used keyword arguments extensively in the Python, especially in object-creation functions where it is often required to pass in multiple parameters that won’t fit neatly into a single data structure.

Go presently has no such feature. This is probably the single most serious readability hit my translation took; it got significantly more difficult to grasp what was going on at all those callsites.

Translating Python map() calls and comprehensions produces code that is ugly and bulky, forcing the declaration of dummy variables that shouldn’t need to exist.

If one graded possible Go point extensions by a figure of merit in which the numerator is "how much Python expressiveness this keeps" and the denominator is "how simple and self-contained the Go feature would be", I think this one would be top of list.

So: map as a functional builtin takes two arguments, one x = []T and a second f = func(T)T. The expression map(x, f) yields a new slice in which for each element of x, f(x) is appended.

This proposal can be discarded if generics are implemented, as any reasonable implementation of generics would make it trivial to implement in Go itself.

Inability to apply const to variables with structure, map, or slice initializers is annoying in these ways:

In Python one can often get a similar effect by using tuples. I used this as a form of internal documentation hint in the original Python. I want it back in Go.

Any extension in the scope of const, even a relatively conservative one like only allowing const structures with compile-time constant members, would have significant benefits.

This is a small point problem, easily fixed, that was far more annoying in practice than it should have been in theory.

Python regexps have both positive and negative lookbehind clauses. The following expression looks for possible Subversion revision designators in comments, excluding bug references:

Go translation reveals that it is remarkably unpleasant, verging on "too painful to be worth it" to do that filtering without lookbehinds.

This is the only real problem I have identified in moving from Python regexps to Go ones. Take that "only" seriously, because regexps are a Swiss-army knife I use heavily; Go regexps are doing well to have no limits that are more annoying.

I have read issue #19412 and am aware of the objections to adding sum types to Go.

Nevertheless, I found their absence was something of a pain point in my translation. Because reposurgeon events can have any one of a set of types (Blob, Tag, Commit, Callout, Passthrough, Reset) I found myself writing a lot of stupid boilerplate code like this:

Besides being inelegant, the requirement for a runtime check to exhaust all cases is a defect attractor. It’s way too easy to forget to write the default case and wind up with silent errors.

Thus, absence of discriminated-sum types is an actual hole in the language that compromises its goal of enforcing strong invariants through type safety checked at compile time.

This will especially tend to become an issue when translating from a language like Python with fully dynamic typing.

I don’t have a concrete proposal to fix this yet. If these notes are well received I may write one.

Though I revised it significantly on completion, much of this report was originally written at about the 12% point of the translation. By twice that far in, 23%, another problem about which I had not originally been intending to complain became obtrusive. That is absence of a general facility for structured exceptions.

Yes, I’m familiar with all the reasons throw/catch wasn’t included in Go 1. Including the laudable goal of forcing programmers to be explicit about error handling and how they propagate errors up their call stack. And I understand that defer/recover was an attempt to provide a tractable subset of catchable exceptions that would minimize the temptation to sin.

Because I broadly agree with this set of goals, I was actively intending when I started this translation not to complain about the lack of general catchable exceptions, or ship any related RFEs, in spite of having a presentiment that they would be a problem. That is, until I hit a wall in the real world and had to rethink.

Here’s my use case. Reposurgeon is an interpreter for a DSL. Situations in which I can tolerate panic-out and die are rare and mostly occur at initialization time. Usually what I want to do instead of panicking on error is throw control back to the read/eval loop, executing some kind of local cleanup hook on the way out. Analogous situations will frequently occur in, for example, network servers.

In a language with labeled throw/catch, or class-valued exceptions, I can address this by explicitly target an exception to some level of the call stack above the point it’s raised. In reposurgeon, for example, there are usually two levels of interest. One is the DSL’s read-eval loop. The other is the outermost scope; if an exception gets there I want to call hooks to gracefully remove working directories (blob storage associated with the repository-history structures being edited) before exiting the program.

In Go, I didn’t seem to have a clean option for this. Which was a problem on two levels…​.

Having Python iterators go missing is really annoying for reposurgeon, in which lazy evaluation of very long lists is a frequent requirement.

Here’s the type example. I have in my repository representation a list of possibly hundreds of thousands of events. A subset of these events is Commit objects. I would like to be able to write

In Python it is easy and natural to write commits() as an iterator which lazily walks the repository event list looking for Commit objects. Each time it is called it either returns with "yield", handing back the next commit, or actually returns — which is a signal that the for loop should terminate.

I can’t do this in Go; I have to write commits() to return an entire constructed slice made by filtering the event list. Which is annoying for long lists, especially when it might well terminate early.

Sure, there’s an alternative. It looks like this…​

…​and about which what I have to say is "Ugh!". That code does not say "walk through all commits", it says "walk through all events and do something to the ones that happen to be commits". I don’t want to wander into event-land here; that type-assertion/cast pair looks altogether too much like a defect attractor. Also, unnecessary eyeball friction.

I had no good idea what could be done about this. I read Ewen Cheslack-Postava’s excellent discussion of iterator patterns in Go and agreed with him that none of them are really satisfactory.

Annoyingly, the iterator pattern he suggests is almost the right thing — except for the part where early break from a channel-based iterator leaves its goroutine running and some uncollectible garbage.

Then, on my second reading, I had a brainstorm. I found a trivial Go extension that would give iterators with no new syntax, no hidden magic, and no yield/return distinction:

New evaluation rule on how to interpret for loops when the range operand is a callable: the loop runs as a generator, yielding each value in succession, until the callable returns the zero value of its type.

So, with that I could write a Repository method like this:

…​and there I have it. An iterator, with exactly the same lifetime as the for loop.

Then I thought it might be best to make this properly parallel to the way iteration via range works.

With this form the iterator could pass back zero values without terminating, terminating only when the second return value from the function-valued range argument goes to false.

I suggest that one of these be adopted for a future release of Go. Small, easy new evaluation rule, big gain in expressiveness.

Figuring out how to do type-safe polymorphism in the event list was more difficult than it should have been. The problem here wasn’t the Go language, it was the official (and unofficial) documentation.

There are two problems here, one of organization and one of style.

The organization problem is that there isn’t one. The official Go documentation seems to center on the library API docs, the specification, the Tour, and a couple of "official" essays written for it. It also includes a corona of white papers and blog posts. Often these are valuable deep dives into specific aspects of the language even when they are notionally obsolete. Some of them are outside the boundaries of the official documentation site.

For example, I got substantial help understanding interfaces from an old blog post by Ian Lance Taylor (one of the Go devs) that was offsite, dated from 2009, and contained obsolete implementation details.

The high-level problem is that while the Go devs have done a praiseworthy and unusually effective job of documenting their language considering the usual limitations of documentation-by-developers, finding things in the corona is hard. And knowing what’s current is hard.

The documentation is (dis)organized in such a way that it’s difficult to know what you still don’t know after reading a Tour page or blog entry or white paper. There should be more "But see here for a dangerous detail" links, in particular to the language specification.

Style. Go has a problem that is common to new languages with opinionated developers (this is part of "the usual limitations" above). There are one or two exceptions, but the documentation is predominantly written in a terse, hieratic style that implicitly assumes the reader already inhabits the mindset of a Go developer.

The documentation is not very good at providing an entry path into that mindset. Not even for me, and I’m an extreme case of the sort of person for whom it should do an effective job if it can do that for anyone.

There is a fix for both problems. It is not magic, but it is doable.

The Go dev team should bring in a documentation specialist with no initial knowledge of Go and a directive to try to maintain an outside-in view of the language as he or she learns. That specialist needs to be full-time on the following tasks:

(1) Edit for accessibility — a less hieratic style

(2) Maintain a documentation portal that attempts to provide a reasonable map of where everything is and how to find it.

(3) Curate links to third-party documents (for example notable Stack Overflow postings), with dates and attached notes on what parts might be obsolete and when the document was last reviewed for correctness.

(4) Bring the very best third-party stuff inside, onto https://golang.org/doc/.

Note: After writing this, I had an even worse time digging up and fixing in my mind all the details of how defer/panic/recover works. It’s almost all documented somewhere, though Peter Seebach and I ended up writing a FAQ entry on how to set local variables from a defer clause to clear up minor confusion. There’s a very helpful blog post on the general topic. But the blog post leaves out the crucial detail that recover returns interface {}, not error; this tripped me up when I was writing my kludge, and I ended up on IRC getting referred to the formal Go specification.

This is all too typical. Everything makes sense once you know it, but before you know it critical details are often lurking in places you have no way of knowing you should look.

Attention to the problem and a good technical writer/editor can fix this.