ongoing by Tim Bray

ongoing fragmented essay by Tim Bray

Luxury media 21 Sep 2022, 9:00 pm

I was cruising through the supermarket when a shopping-cart traffic jam stalled me in front of the “newsstand”. That word is in quotes because most things on offer aren’t news and in fact aren’t magazines in the traditional sense, but single-issue glossies about Princess Diana or Bruce Springsteen or Michelle Obama or Christmas. Anyhow, an awesome cover photo of an owl on the The Atlantic caught my eye; that cover also touted an article about Jack White, whose music I enjoy, so I impulse-bought it. Now I’m thinking about the quality-publishing business model.

Three Atlantic covers Three Atlantic Covers

Recent Atlantic issue covers.

I got a couple hours’ reading pleasure for the handful of bucks I spent on dead trees and ink. Here are some of the advantages of a paper magazine:

  1. It’s as light as any product with “Air” in its name.

  2. You can read it on the bus or in the bath or out in bright sunshine.

  3. The color, resolution, and presentation of images is wonderful. Yes, screens and graphics hardware get better and better, but wow, those pictures on that high-quality paper.

  4. The typography is exquisite. Once again, we make progress on screen presentation, but still.

  5. The pages aren’t visually busy. Yes, there are ads around the edges of some and quite a few odd-numbered pages are full-page ads, once again with fabulous composition and color. But those columns of text, they and their surrounding white space, thoughtfully designed, provide a serene reading experience.

  6. You can fold it up and slip it in a vest pocket. Then you take the folded version out again with one hand and read while you use the other to support yourself on a commuter train.

  7. If you’re operating one-handed you can turn the page by sticking your thumb under the next one and performing a graceful wrist-flip.

  8. The text and images stand still! There are no late-displaying ads or graphics pushing the words around while you’re trying to read them.

  9. Nothing asked me whether it could send notifications.

  10. Nothing suggested that I switch to the app.

  11. In fact: No. Popups. Ever.

  12. Drop it all you want, then step on it even, no biggie. Hell, drive over it too!

  13. No battery, no charger, no wires, no plugs.

  14. The speed at which the text refreshes when you turn the page is remarkable.

  15. There were a few ads suggesting tastefully that I subscribe, but none of the raw sweaty desperation of the contemporary Web.

  16. Nobody was tracking me. Well, I guess the credit-card company reported my grocery purchases to, well, someone, but nobody knows which articles and ads I looked at.

  17. No venture-capitalist libertarians are being enriched.

  18. When you’re done, you toss it in the recycling bin.

Wait, am I saying that the Web was a mistake, that we should all go back to dead trees? Not at all. I couldn’t tap on the pretty-decent Jack White article and watch a YouTube of the songs. I couldn’t share a particularly tasty written morsel on Twitter. I had to go to a store to get it. I couldn’t read it in bed without turning a lamp on. I probably could have paid for a couple months’ online subscription to The Atlantic for the newsstand price.

But on balance I was left thinking “This feels like a luxury product.” I can’t think of an obvious analogy… Perhaps the enveloping, focusing hands-on experience of putting a record on the turntable as opposed to a streaming service on the earbuds? Except for, magazines are more convenient, once you’ve gone to the trouble of going to the store for them. Hmmm, I hear rumors you can arrange to have them delivered to your home at regular intervals; must check that option out.

I’m thinking purveyors of print ought seriously to consider luxury-product business strategies.

Hello, Ruler 2 Sep 2022, 9:00 pm

Hey, look what’s been open-sourced: AWS Event Ruler! Check out the announcement blog. I built v1.0 of this Java library while I was at AWS, and wrote about it in Filtering Lessons. Tl;dr: It offers APIs for declaring pattern-matching Rules, as many as you like, then for presenting data records called Events and finding out which Rules each Event matches, very quickly. It’s in production in multiple Amazon (not just AWS) services, notably EventBridge. Also see: Content-based Filtering.

I wrote the Quamina library because I liked the Ruler concept so much that I wanted a do-over in another language. The Quamina Diary blog series has lots more on algorithms and data structures; the most compact how-it-works intro is in Meet Quamina.

Before I go on, I want to thank David and Usman and Rishi and Shawn and the others who succumbed to my low-key but persistent lobbying and blessed this release.


Ruler’s only runtime dependency is Jackson, which brings along basically zero sub-dependencies. Ruler has a Maven POM, which I pointed at when I opened it in a new IntelliJ project, and was running the unit tests within minutes. Java 8 is good enough and there’s not that much code, 15K or so lines between implementation and test. Unit-test coverage is over 90%.

“Event Ruler”?

Inside AWS it was just “Ruler”. That name was obvious back when the software was written, for V1.0 of what was originally called “CloudWatch Events” and is now EventBridge. The central function of that service is to post rules against event streams and route matching events to useful destinations. The APIs have names like addRule and rulesForEvent so “Ruler” wasn’t exactly a creative triumph. But it’s short and memorable.

The processes inside AWS that led to this announcement yielded the name “Event Ruler” and there’s nothing wrong with that. But it’ll always be just Ruler to me.

Why it matters

The software is widely used inside AWS. Will it be useful outside the cloud-infrastructure world? My bet is yes, because more and more apps use loosely-coupled event-driven interconnections.

For example, I think there are probably a lot of Kafka applications where the consumers could be made more efficient by the application of this sort of high-performance declarative filtering. Yes, I know there are already predicates, but still.

Second, the Rule syntax used for pattern-matching is unconventional and the teams who use it seem to like it. Yes, there’s the occasional Couldn’t it be SQL? but I do think there’s a place for this flavor of structured query.

Finally, AWS has benefited hugely from the use of open-source. So it’s nice to see them giving something back, something built from scratch, something that doesn’t particularly advantage AWS.

It’s worth noting that the GitHub project has four committers (I’m one), only two of whom are Amazon employees.


No nontrivial software is bug-free. But by my arithmetic, in its AWS life Ruler has been invoked on the order of 1016 times, including lots of millions of times while you’ve been reading this. On extremely heterogeneous data. With an extremely miscellaneous selection of Rules. Which is to say, you’re fairly unlikely to get a nasty surprise from it.

Shredding automata

[There’s a point to this anecdote, stay with me.]

It’s a rare EventBridge customer that has more than a few dozen different rules compiled together to match Events. But there was this service that really needed a fast event matcher and, for complicated reasons, needed to run huge amounts of traffic through a single server. One of them asked me “Suppose we needed to post a million rules?” I said “You’re kidding, right?” They weren’t, and they did. To my joy and astonishment, it worked fine.

But then they said “We can’t afford to rebuild that automaton at run-time, so we need to be able to delete rules.” My initial implementation of Ruler never considered the idea of doing this because just thinking about it made my head hurt (still does). So I said “Can’t be done, what’s your Plan B?”

At which point this dude Long Zhang, then in the requesting organization, just went ahead and built it. It works great.

Long became a primary contributor; he and I worked together on lots of features and he added some on his own. He knows more about Java concurrency and its memory model than anyone, and is a fun guy to work with. At this point, I bet he’s got more lines of code in the project than I do.

I remember one time we were having trouble agreeing on how to build anything-but matching and I couldn’t understand his explanation. We ended up emailing set-arithmetic expressions back and forth until we converged. To this day, Long and I have never actually been in the same room. Isn’t the future weird?

After all that was working, the big-single-machine service came back to us and said “Ruler is using too much memory.” Yes, Dear Reader, in the guts of cloud infrastructure, memory really matters. So Long and I dug into that problem. I’m telling this story to help explain…

The hard bits

Conceptually, Ruler is simple. In practice, the code starting at ByteMachine is, uh, non-obvious in places. One reason is supporting the deleteRule API. Which means reaching into a compiled state machine and ripping out all the bits that match some Rule, without breaking anything. While the state machine is being used to process a huge number of queries per second. If you think that sounds hard, you’re correct.

Then, there’s that RAM issue. There is fairly deep weirdness, notably in the inheritance and implementation hierarchy around ByteMachine, aimed at shrinking the memory footprint.

Another hairy bit that I mention mostly for amusement value is building numeric-range-match state machines. If you want a giggle, scan the comments in addRangePattern starting here.

Just JSON?

Out of the box, Ruler can filter data presented as JSON texts, as parsed JSON nodes, and as “flattened” lists of field name/value pairs. It’d be nice if there were “flattener” plug-ins for Avro and CBOR and Protobufs and so on that turned data blobs into field lists. Jackson, which Ruler uses for tokenization, already knows some of these formats, which should make things easier. Hint, hint.

Ruler and Quamina

Obviously, since they’re library code and in different programming languages, neither of Ruler nor Quamina can be a direct replacement for the other. But I’m going to try really hard to make the the syntax for what Quamina calls “Patterns” be 100% compatible with what Ruler calls “Rules”, and to make them as feature-compatible as possible.


Anyone out there think they might want to use this? Feel free to ping me, happy to offer moderate amounts of free advice.

Long Links 31 Aug 2022, 9:00 pm

Hi everyone, once again I invite you to a curation of links to long-form pieces that I have time for but you may not. The hope is that one or two of these will enrich your life. This month, featuring quite a bit of music, computer keyboard obsession, politics as usual, and potatoes.

But let’s start with music… recently, I’ve been writing less. Reasons include: Interesting things I’m working on that are non-public. And a software bug I haven’t figured out that’s blocking a software project. Also, I’ve been playing quite a bit of Diablo Immortal. And (the reason I’m writing this) I’ve been falling down musical rat-holes on YouTube.

I don’t know how YouTube’s algorithms and ML models compare to Spotify’s or Apple’s or anyone else’s because I don’t use those. I do know that YouTube’s suggestions about what to watch next are mostly competent and quite regularly brilliant, with the effect that I ask it to play some song and then it’s three hours later and I need to crash.

It probably helps that I was on Google Music for years, and uploaded ten thousand or so songs into it, and listen quite a lot — most often while I’m driving or doing housework.

While I’m not dogmatic, what I mostly want to watch is live performances. That’s where the musical magic is. Recently, the music I’ve been listening to is almost entirely traditional forms, and mostly performed by women. Probably what pushed this over the edge was my discovery of Sierra Ferrell, about whom I’ve written more than enough; but if her tour is coming anywhere near you, buy tickets. Trust me on this.

Since this outing is going to feature mostly women, let’s start with a guy: Billy Strings, who is a beyond-awesome bluegrass singer and guitar-picker. He has loads and loads of superb performances on YouTube, but I’m going to offer you Dust In A Baggie from Lollapalooza 2022. One of the interesting things about Billy is that he and his parents were mired in meth and related problems, but have managed to escape that trap. Dust In A Baggie covers the territory (“I used my only phone call to contact my daddy / I got 20 long years for some dust in a baggie”).

Let’s stay traditional but switch continents and listen to the Habibti Ensemble performing Baghdad. Don’t know much about them; they’re Israeli but the music has a nearly pure Arabic sound to my ears. Also “habibti” is an Arabic word meaning, more or less, “sweetie”. It’s a lovely tone and tune. Staying in that corner of the globe, check out Light in Babylon with Hinech Yafa. This is a capture of musicians busking on the streets of Istanbul. Everything’s in Turkish and that’s all I can tell you. But check out that woman on the hand drum. Heading way, way south, here’s a concert by Sona Jobarteh & Band. Ms Jobarteh is British-Gambian and said to be the world’s first professional Kora player. This concert flows effortlessly like water down a hill, the tones gentle and the rhythms quietly fascinating.

Let’s return to the New World and visit with the Tedeschi Trucks Band. They’re a big outfit, a dozen or more musicians on-stage at any one point. I’ve always liked their music, though I’m more a fan of Ms Tedeschi than Mr Trucks. Which is why I’m suggesting you take in I Pity The Fool by Bobby Blue Bland. Normally, in a T-T performance, Susan does most of the lead vocals and takes a guitar break here and there, with plenty of vocal support from the band’s other excellent singers and extended guitar outings from Derek Trucks. Not this time: Susan runs this one end to end, with a fabulous break that crescendos for chorus after chorus, then a white-hot explosion of song. One assumes the local fire department was spraying the outside walls to keep her from burning the place down. I particularly enjoy the grins on the other musicians’ faces as they more or less duck for cover and get out of her way.

OK, let’s let some males back in the room. Could anything be more American than a bluegrass band covering When Doves Cry? I’m actually not sure this quite works, but it’s conceptually amazing.

A final musical note without music notes: Rage Against The Machine’s Tom Morello on the Music of His Life. Cool guy, interesting takes.

Shifting abruptly over into the nerd lane, from the New Yorker there’s The Obsessive Pleasures of Mechanical-Keyboard Tinkerers. Most computer geeks already know what this is about, but it’s a sub-sub-sub-culture open to anyone who spends a lot of time banging their fingers against keys like I am at this moment.

More geek stuff: From Seph, 3 tribes of programming. I don’t agree with all of it but I enjoyed reading it.

Back to policy and politics. Oh, and history. I’d never previously encountered but followed some link to The best books on The Great Divergence. What “Great Divergence”, you ask? The one that led to the peoples of Europe, sometime starting in the eighteenth century, becoming richer and more powerful than plenty of other civilizations where the people were just as smart and inventive. There is no explanation for this that enjoys consensus support among historians; this article offers a useful survey and left me wanting to read some of the books they discuss. By the way, in 2017 I wrote up a book by Joel Mokyr, one of the authors here. That book covered basically the same question and if you want to know what I think about it, go read that.

If you’re interested in history and in beer you’ll like On beer, or, why chicks rock by Medieval Historian Eleanor Janega. With pictures!

Obviously the climate catastrophe is the central political problem of our age, and while we have to hand many of the tools we need to reduce the pain, the question of energy storage remains vexed. The reliably-excellent Sabine Hossenfelder offers a super-useful survey in No Sun, No Wind, Now What? Renewable Energy Storage. If you care about this issue (and you should) read it.

Depending on how you parse the word “problem”, you could also argue that we have none more exigent than the recent repeated outbursts of craziness on the right wing of the political spectrum. I’m a leftist but think we need intelligent conservatives. But (in America and Britain at least) there aren’t any. For a pretty deep investigation of how we got here, check out Tanner Greer on The Problem Of The New Right. I sure learned plenty. Relatedly, on a much less theoretical level, John Ganz’s The Enigma of Peter Thiel is useful. It’s subtitled “There Is No Enigma — He's a Fascist.”

Matt Stoller is a full-time anti-monopoly activist and offers an interesting look at a proposed mega-merger in the big-publishing world, characterizing it as Free Speech on Trial. I think he’s 100% right.

I’m not a Marxist but am something of a class reductionist. I think the facts are on my side that class issues, i.e. “who gets the money”, is at the center of everything, and Shareholder Power and the Decline of Labor suggests that I’m right.

Now for something completely different: David Simon Unravels the Moral Twists of Paths of Glory. Simon is best-known as The Wire showrunner, and Paths of Glory is a 1957 WW1 movie. This isn’t text, it’s a film with movie footage and Simon’s head talking, a kind of thing that I rarely link to. But this hit me pretty hard, given that as I write we’ve got another land war going on in Europe. Anything we can do to strip aside any notions of “glory” and “honor” around War is a good thing; this is that.

I’ll end on a hopeful note. Two, even. First, Dave Pawson, a fellow geek, writes about a home-solar project; It’s full of practical information about how doing this can work. Good stuff! Except for it’s a Facebook link  —  it still (sort of) works in a browser that’s not logged into FB.

Finally, How Peru’s potato museum could stave off world food crisis. Damn, I love a well-cooked potato.

On Faith 24 Aug 2022, 9:00 pm

What is “faith”, anyhow? The answers can become abstract, but concretely it’s what gets Salman Rushdie stabbed in the face. Oh wait, is that statement anti-Islamic? Guilty as charged; but then I’m also anti-Christian, anti-Hindu, anti-Buddhist, and, well, there are too many organized religions to list them all. Herewith too many words on Faith and Truth, albeit with pretty pictures. I do find positive things to say, but at the end of the day, well, no.

I’m not a moderate on this issue. I don’t believe, in general, that any supernatural event has ever occurred or, in particular, that any prayer has ever been answered. In my lifetime the outputs of organized religion have been mostly war, sexual abuse, and political support for venal rightist hypocrites.

Does it even matter?

Maybe religion has become irrelevant to your life, a common experience these days. It’s worth studying anyhow, as an example of the human propensity to believe things that are not just untrue, but wildly unlikely. Things that entirely lack supporting evidence. Things that make you shake your head.

Images of the Virgin Mary and child for sale

I may not partake in faith but I’m a keen amateur student of religions. Perhaps it comes of having grown up in Lebanon, where a whole lot of ‘em run up against each other with a mostly disastrous impact on the civic fabric. I’ve visited Jerusalem and Damascus and Chartres and Stonehenge and Avebury and Carnac and Kamakura and Izumo and Beijing’s Temple of Heaven, and some of the world’s great forests.

Faith is real

I think that in my youth I maybe knew a saint. No, really; Father Leonard Guay, a Jesuit architect and astronomer. He built a university in Baghdad which was taken over by the Ba’ath, so he built another in Aleppo; the same thing happened. He offered no regrets. An American Midwesterner, he was in a combination monastery, winery, and observatory in rural Lebanon when we knew him. He loved coming over to talk English, eat corn-on-the-cob, and swap crossword-puzzle books; he took regular puzzles and did them diagramless, with just the clues. He loved kids, knew a million utterly lame jokes, and enjoyed telling us about the current research in the observatory, apologizing for using pagan language, as in the names of constellations and Zodiac signs. His faith glowed within him and around him.

This is the challenge to my personal aversion to the supernatural. I observe empirically that faith exists and that it’s real and that it appears to be good for some people. But my mind recoils at all the crazily baroque apparatus that is inextricably attached to every organized religion. I believe in belief and have no faith in faith.

But I went to Sunday School and my Dad even taught it in his youth. Once, when I boarded for a semester with family friends, I attended a Southern Baptist congregation which even at the age of twelve struck me as pretty looney-tunes. In my own mainstream-Protestant scientist family, the religious pressure was more or less zero.

Buddhist temple in Lahaina

Jodo Buddhist Mission in Lahaina, Maui

What I’m not buying

So I understand pretty well what it is. I’m not buying a deity who hardens Pharaoh’s heart, provoking plague and child slaughter across a whole population. Nor, in general, gods that purport to exhibit gender. Nor any who thrive on praise and consider it essential from their devotees. Hindutva smells to me like old-fashioned ethnofascism with Pujas. I scoff at a putative Savior who lectures that lust is morally equivalent to adultery. I’m not down with Wahhabi support for autocratic murdering princes nor with Crusading nor with throwing settler garbage down into lower Hebron. And the phrase “blasphemy law” makes me shiver with anger.

Faith and humans

But religion comes naturally to Homo sapiens. Perhaps the first big reason is that for much of humanity’s passage across time we were manifestly not in control of our destiny, just ephemeral sparks of life blown about by whims of climate and disease and geography and the population dynamics of our prey and predators, those sparks often snuffed out with no warning. It would have been comforting to think that Someone was in control. Maybe the improved ability to steer one’s own life, enjoyed today by those in developed societies and with an education, is partly responsible for the fading of faith?

Second, worship is a human built-in. We are small after all, regularly confronted by things much greater than ourselves: Our starfield, away from light pollution. The meeting of the Eastern Pacific with the Western Americas. Canyons and waterfalls and great ancient trees.

Candles in Chartres cathedral

Candles in Chartres cathedral.

Of course, we build some of the things we worship. I know of two people who say they acquired faith following on a visit to Chartres, and I believe them. If you can enter that great stone poem without your sense of worship activating, I think you’re weird. The first time I walked in, it felt like a giant hand round my chest, interfering with my breathing.

But I dunno, I’ve had the same feeling at concerts by Laurie Anderson and Slava Rostropovich and The Clash and a host of other artists. Is my gratitude for being alive at the same time as these exceptional people “worship”?

Temperate rain-forest trees


My family has the great good fortune to own a cabin on a small island in Howe Sound near Vancouver, where I regularly experience worship of that Pacific perimeter, and especially the island’s great evergreens and bigleaf maples that tower into the filtered forest light, never still, wind never entirely absent in the greenery. I keep telling my kids they should shut up and listen to the damn trees and they’ll learn things and I’m right but they don’t, usually.

Feeling reverent around trees has the advantage that they’re not avatars of anything that is said to be twitchily concerned about how and with whom you deploy your genitals, or whose intercedents will require your cash to support their lifestyles.


I think it probable that religion will continue to decline, to the extent that its concerns are absent from public discourse. In my own civic landscape it already has.

That’s not entirely a good thing; you can admire aspects of religion without actually believing it. One of them is ritual, prescribed and choreographed public actions. It’s a thing that a high proportion of humans once experienced on a weekly basis; but no longer. I think we miss it. Military services retain rituals, as do the centers of government – consider America’s State of the Union address, or the opening of various nations’ parliaments. Weddings and funerals retain a ritual dimension but are infrequent. While I have no patience for Catholic dogma I often tune in their midnight Christmas mass for its own sake – the singing and chanting, the inner space of St Peter’s basilica, and priestly processions carrying the Host. I love the opening and closing of each Olympic games.

Sacred texts

Every faith has them. While I decline to honor their claims, it’s good to believe that written-down words are important, because they are. The use of language defines what’s special about our species as much as anything else does and I believe the single greatest cultural shift in humanity’s story came when it could be written, and lessons could outlast the storage provided by a human skull.

The Shinto shrine at Izumo

Shinto shrine at Izumo

The worlds of sci-fi, fantasy, and computer games are full of powerful and magical texts; obviously this notion speaks to many people. Religious texts are also historically important because they were replicated a lot and are thus well-represented among the fragments of language that have survived the ravages of centuries. Some of the books and the verses and words are very beautiful.

In fact, the Christian Bible, particularly in its seventeenth-century “King James” embodiment, has been at the center of the cultural experience of my own ethnic group to the extent that I think it probably deserves routine study at some point in the standard curriculum. A whole lot of our ancestral history and much wonderful literature and art is going to elude understanding without at least a basic grasp of its scriptural embedding.

If you want to get inside the head of someone who really held close to those values, go listen to Hildegard von Bingen’s O vis aeternitatis (“The Power of Eternity), probably written around 1150. It’s wonderful music! The world Hildegard inhabited, of faith made real in cloisters and their communities, is as remote from mine as that lived by the characters in the sci-fi I enjoy reading.

Christian wall decoration at the Abbey of  Montserrat, near Barcelona

At the Abbey of Montserrat, near Barcelona

The flavor of truth

Of course, since sacred texts are said to express eternal absolutes, they must necessarily be immutable. Which seems boring and just wrong. It is a core value of scientists and engineers and philosophers and reference publishers that truth is contingent and dynamic; always capable of being better-expressed or deepened or falsified. On top of which, the language we use to express truths grows and mutates across the centuries. I’m not holding my breath waiting for Christendom to convene a Third Council of Nicaea and revise the doctrine of the Trinity. Still, we should respect and preserve and study the sacred texts because they are full of lessons about the people who wrote them and believe them.


Around those scriptural mountains are the rolling hills of exegesis; works of commentary and analysis, for example the Hadith and the Talmud. Christian exegesis is unimaginably vast in scale although it lacks a single named center.

I got lost in the exegetical maze during a failed youthful attempt to write a novel about the rise and wreck of the Tower of Babel, when I tried to understand what that crazy story might really be about.

Exegesis is fun to read! Intellectually challenging on its own terms, and if you have any familiarity at all with the Bible, the depth of meaning apparently waiting to be uncovered in the crevasses between adjacent words is astonishing. Next time you’re in a good library, I recommend looking up “Anchor Bible” in the catalog and poking around the stacks where the call-number takes you. If you’re like me your mind will be boggled at the vastness and complexity of the collection.

On this subject, if you ever find yourself in a colloquy of theologians or bibliophiles or antiquarians, even a brief mention of “The Church Fathers” will get you nods and smiles. They constitute the first few waves of Christian theological writing, there were really a lot of them (no Church Mothers, though), and they wrote an incredible number of books, many beautiful in form and content.

Worshipper at Jing’an temple, Shanghai

At Jing’an Temple in Shanghai

What a certain number of these theologians are trying to do is very similar to the goals of Physics theorists: An explanation of the universe from pure logical principles, showing how it really couldn’t possibly be any way other than the way it is. Christian theologians assert that this must be the Best of All Possible Worlds because what else could God have made?

They want the necessary outcome, via pure logical reasoning, to include an omnipotent omniscient male-gendered Creator and also a Savior, a single instance of God-made-flesh, plus a really hard to understand “Holy Spirit”. Which is to say, they have a heavier lift than physicists do. But to this day, Proofs of the Existence of God remain an amusing sub-sub-domain of theology and exegesis.

[I think you were supposed to be writing about the redeeming features? -Ed.] [Oh, right, thanks. -T.] Finally, it would be unfair to consider religion without acknowledging its leading role in philanthropy across the generations and continents.

Faith why?

None of which means we need to believe what the religiosos claim is true. But why, in the 21st century, do they still believe it? I really don’t have much to add to the two points I made above: The feeling that Someone must be in charge, and our built-in capacity for worship.

Let me offer an incredibly cynical but kind of entertaining take on the subject from Edward Gibbon, in his monumental Decline and Fall of the Roman Empire, six massive volumes dating from the late 1700s. [You didn’t actually buy it and read it, did you? -Ed.] [Sometime around 1984 I joined the “Book of the Month Club” and these great-looking books were the sign-up bonus. I read a lot of it, but got bored around 1000AD in Vol. 5, all that endless Byzantine treachery. -T.]

In a garden across from a Catholic school

In a garden across the street from a Catholic school

Gibbon is discussing the rise of Christianity in the Empire, which he argues contributed to its fall, but that’s neither here nor there. He includes a sprawling survey of the religious landscape and, while discussing the Jews, notes that unlike many other faiths of the time, they weren’t prepared to go along and get along, host an occasional sacrifice to one Caesar or another in their temple; they resisted militantly and to the death. So, quoting from Chapter XV, Part I:

But the devout and even scrupulous attachment to the Mosaic religion, so conspicuous among the Jews who lived under the second temple, becomes still more surprising, if it is compared with the stubborn incredulity of their forefathers. When the law was given in thunder from Mount Sinai, when the tides of the ocean and the course of the planets were suspended for the convenience of the Israelites, and when temporal rewards and punishments were the immediate consequences of their piety or disobedience, they perpetually relapsed into rebellion against the visible majesty of their Divine King, placed the idols of the nations in the sanctuary of Jehovah, and imitated every fantastic ceremony that was practised in the tents of the Arabs, or in the cities of Phœnicia.10 As the protection of Heaven was deservedly withdrawn from the ungrateful race, their faith acquired a proportionable degree of vigor and purity. The contemporaries of Moses and Joshua had beheld with careless indifference the most amazing miracles. Under the pressure of every calamity, the belief of those miracles has preserved the Jews of a later period from the universal contagion of idolatry; and in contradiction to every known principle of the human mind, that singular people seems to have yielded a stronger and more ready assent to the traditions of their remote ancestors, than to the evidence of their own senses.11

10 For the enumeration of the Syrian and Arabian deities, it may be observed that Milton has comprised, in one hundred and thirty very beautiful lines, the two large and learned syntagmas which Selden had composed on that abstruse subject.

11 “How long will this people provoke me? And how long will it be ere they believe me, for all the signs which I have shewn among them?” (Numbers xiv 11). It would be easy, but it would be unbecoming, to justify the complaint of the Deity, from the whole tenor of the Mosaic history.

(Gibbons’ footnotes, included just for fun.)

Um, is that anti-Semitic? Maybe… and some other things Gibbon said definitely were. But he was also anti-Muslim and arguably anti-Christian. And his scoffing seems more aimed at theologies than ethnicities. In fact, the only religion to get many kind words was Rome’s indigenous paganism, because of its tolerance.

While his text is loaded with nods to Christianity being The Right Answer because of Its Divine Provenance, those passages glisten with cynicism (see above) and he was frequently attacked as an enemy of the faith. Gibbon is fun to read.

A miracle

Finally, let’s consider what miracles are: Things that happen for which there is no conventional explanation in our physical understanding of the universe. At least one miracle has happened. The universe, including its cosmic background radiation, its galaxy clusters, its black holes, me and my thread of consciousness, you and yours, and the text I’m writing and you’re reading: They all exist. There’s no explanation at hand as to why anything at all should. Miraculous!

However, check out Why Does the World Exist?: An Existential Detective Story, by Jim Holt (Norton, 2012), a serious but entertaining tour through metaphysics and religion looking for an answer to the question in the title. Spoiler alert: While Holt’s best attempts are stimulating, I was still left thinking of the existence of anything and everything as a miracle.

Former Lutheran church

Former Lutheran church,
property soon to be filled with condos.


You could think of religion as a pathology of society as a whole, consequent on ignorance, fear, and certain built-in features of the human mind. I don’t think it’s going away, although a faith-free world would probably be a kinder, more humane place.

With these words, I may have offended some who partake in faith. I can’t honestly apologize, because an apology is at some level a promise to Stop Doing That.

I really, really just don’t buy it.

Slow Travel 16 Aug 2022, 9:00 pm

Suppose you believe, as I do, that we have to slash our carbon emissions drastically and urgently to ameliorate the worst effects of the climate catastrophe that is beginning to engulf us. And that you believe, as I do, that travel is a good thing for individuals and societies, and that in principle every human should enjoy the opportunity to visit every neighborhood around our fair planet. Can these beliefs be reconciled?

I used to joke, when people looked askance at my refusal to leave Vancouver, that it was the center of the world: “Ten hours to Heathrow, ten hours to Narita!” I was right, but…

  1. Earth can’t afford for everyone to routinely hop on a jet-fueled aluminium cylinder and fly ten hours at a time.

  2. Narita and Heathrow both suck. Arriving at either of them after one of those long flights is usually pretty awful. I speak from painful experience.

The unpleasantness of Narita and Heathrow is not entirely their own fault. After you’ve flown ten hours east or west, when the plane comes down you’ve descended into a nasty fog of jet-lag. Words cannot express my loathing for this condition  — tossing and turning in an unfamiliar hotel bed during the wee hours, fighting off nap attacks in the middle of business meetings, feeling paralyzed after a single glass of wine with dinner.

The lame old anecdote applies here: “Doctor, it hurts when I do this!” “So, stop doing that.” No kidding, and you might help save the planet too.

I have read that recovery from jet-lag takes on the order of one day per timezone crossed. Thus, our current mode of travel, which crosses roughly one time zone per hour while spewing CO2 to poison our children’s futures is simply a bad thing and we should Stop Doing That.

Disclosure: I have been an egregious lifelong travel offender. I’ve enjoyed Damascus and Antibes and São Paolo and Tokyo and Memphis and, well, the list goes on. A world in which those destination options are no longer open to anyone would be a sad place.

What are you suggesting?

That we travel more slowly, which is to say more humanely, and which will enable us to cut down on the greenhouse gas per unit of distance.

Concretely, that for every trip we want to take, we maximize the distance that is covered by train, and minimize those legs that require becoming airborne.

For example

So let’s suppose I want to sample the delights of the Côte d'Azur one more time. Starting, of course, from Vancouver.

The big problem, of course, is the Atlantic ocean. Water travel is unacceptably slow and not notably energy-efficient. So let’s concede that it has to be done by air.

Today, I’d fly to Heathrow (sigh), 7,574km. Then I’d spend unpleasant airport time waiting for my connection to Nice or Cannes or somewhere, another 1000km or so. And I’d arrive feeling like deep-fried shit, wouldn’t really be in the swing of things for another couple of days.

(By the way, I’ve done this, back in the Nineties. There were two immigration lines in Cannes and when we got to the front of ours I noticed that each was being served by a young woman and both of them were flushing and giggling. I glanced left and there was Andre Agassi arriving in France to play in the Monaco open.)

Let’s do this more humanely. By consulting Google Earth I observe that the minimum as-the-plane-flies distance between the North American mainland and Europe is probably Portland, Maine to Brest, France. So our trip becomes three-legged:

  1. Train: Vancouver to Portland, ME: 4,023km.

  2. Fly: Portland to Brest: 4,944km.

  3. Train: Brest to Cannes: 1,034k.

How long is this going to take? Let’s assume that all train travel averages 300km/h. Don’t tell me that’s crazy, I traveled 2000km from Hong Kong to Beijing that way in 2019, averaging 306km/h. It can and should be done.

Modern aircraft cruise at somewhere around 900km/h. Thus:

  1. Vancouver to Portland: 13.4hrs.

  2. Portland to Brest: 5.5hrs

  3. Brest to Cannes: 3.4hrs

Realistically, I probably would have to go to Seattle to catch this imagined future train. But anyhow, the idea is that you wouldn’t do all three legs as a single 22.3hr marathon. You’d start the first leg early and get to Portland late, and stay overnight in a hotel with real beds and real breakfasts. Presumably since Portland is now a major rail hub the locals would have recognized a business opportunity and made it attractive to spend a day there taking the sights in, eating lobster, catching up on minor jetlag, and spending another night before you took off for Brest.

I’ve been to Brest, but only the train station on a trip to the countryside of Brittany. It’s well-worth a day or two’s visit, poking around taking in the standing stones and ciders and country cooking. Why not do that while you recover from the trans-Atlantic jet-lag? Remember that a 5000km leg will hurt less than the currently-typical 7500km jolt.

And then that last leg is a doddle, although France being France you probably have to go through Paris to get from Brest to the Riviera. Why not stop? Yes, you’re a tourist so Parisians hate you, but it’s still a cool place. And I guarantee that with four nights off, you’ll show up at the edge of the Mediterranean in a much better condition to enjoy it.

But wait, there’s more!

The thing that bothers me about this trip is still that trans-Atlantic leg. Yes, we’ve chopped a third off the carbon cost by shortening the route, but it’s still damn expensive, measured in CO2. Is there an alternative?

Maybe so. The airships being manufactured by Luftschiffbau Zeppelin cruise at 115km/h. So the leg from Portland to Brest would take about 43 hours. And to be realistic, the Zeppelin products do not have the scale or range to do this economically.

But let’s throw this challenge in the face of the world’s aircraft designers: Build us a way to travel 5000km or so that spews less carbon and provides a reasonably pleasant experience. We don’t require the 900km/h velocity of the current product, but we’d like it to be faster than 115km/h.

The profession built the current nasty-experience climate-destroying product line because they could, and because we decided incorrectly that we needed to travel one time-zone per hour. Let’s just drop that last constraint, and rejoice in doing so.

The real cost

It’s time, of course. Instead of waking up in Vancouver and going to bed in the South of France, it’s taken us the best part of a week to get there. That’s awful!

No it isn’t, it’s civilized. We’ve seen interesting places, we’ve eaten good meals, we’ve arrived in a decent condition to enjoy ourselves, and we’ve avoided pissing on our children’s future.

Of course, I’ve just ruined everyone’s vacation plans because they don’t get enough time off work for this kind of extravaganza. Well, that’s a bug too. As is the notion that it’s ever a good idea to travel at one time-zone per hour.

False Creek Friend 27 Jul 2022, 9:00 pm

I want to share a project I’ve been helping out with for the last couple of years; the False Creek Friends Society. I haven’t wanted to write about it before now because it was just big-ideas talk. But there’s some science starting up and if you’re nearby you might want to get involved.

False Creek

Courtesy of OpenStreetMap.

What’s a “False Creek”?

Wikipedia has the facts. It’s a little piece of the Pacific stabbing into the belly of Vancouver, surrounded by condos, marinas, a cement maker, museums, festivals, Granville Island, and a really nice seawall. I’m writing this in our family boat, my home office several days a week, tied up at one of those marinas. I bicycle on the seawall. It’s a unique, special place and I care a lot about it.

View of Eastern False creek

Near the east end. There are human and legal stories to be told about all those tied-up boats, in another entry.


I feel a pretty deep connection with False Creek and can’t help noticing a few real problems:

  1. The quality of the water; there are regular no-swim notices from the Health people. There also may be industrial-chemical pollutants left from when it was surrounded by sawmills and factories. But, we don’t actually know much about the nature of the problems. The science just hasn’t been done.

  2. False Creek contains several locations that were central to the lives of the peoples of the Squamish, Musqueam, and Tsleil-Waututh nations that our ancestors stole the territory from. But there are few manifestations of Indigenous culture to be seen.

  3. It’s not a place, it’s a hole in the map with lots of interesting stuff around it.

Foggy day on False Creek

A foggy day under the Burrard bridge.

What are Friends for?

I was contacted in late 2020 by Zaida Schneider, retired journalist and mariner, who mostly lives in his lovely tugboat at another of the marinas, and like me cares about the place. Since then, we’ve talked to a whole lot of people and registered a nonprofit and assembled a seven-strong Board of Directors (I’m one) and launched collaborations. What, exactly, are we trying to achieve?

Well, we have a tactical To-Do List. I strongly suggest that if you are in the neighborhood, you read it. But, we have bigger ideas.

Spring flowers beside False Creek

A spring morning.

How about a park? Canada has National Parks, Vancouver has lots of urban parks, and the province of BC has marine parks. We even have a National Urban Park.

Those Marine Parks are just fabulous, lovely places, but they’re (by design) a long way from where everybody is and to enjoy them, you have to be well-off enough to access a boat.

Why shouldn’t Canada have a National Urban Marine Park, and why shouldn’t it be False Creek, where a half-million people can walk to it and it’s an easy trip for another 2½ million?

Canada has Indigenous Protected and Conserved Areas, which tend to be remote and fairly unspoiled. You’ll never unspoil False Creek, but we could protect it from further damage and heal it where it’s diseased, and prove that a city can live in better harmony with the earth that upholds it and the water that surrounds it. Vancouver has a large Indigenous population; why shouldn’t they lead the protection and conservation?

I would totally love for this to be a hot-spot for Indigenous culture and employment. And, after all, Vancouver is a tourist town, we could educate and delight not only the locals but people from everywhere.

There’s a little problem in that apparently there was never actually an Indigenous-language name for the whole of False Creek. But, you know, that could be fixed.

View of False Creek looking east

Big clouds over the condo towers.

Latest news

Starting in spring, we worked with the Hakai Institute on their Light-Trap program, gathering data on what crustaceans are living under the not-terribly-clear waters of False Creek. We think Dungeness crab may be coming back.

In September, there’s a BioBlitz, sort of the Marine-Biologist equivalent of a hackathon, where in the course of a few days scientists try to build a snapshot inventory of what’s living here.

Come on down!

Lots more projects are brewing. If you’re one of the very many people who either live in sight of False Creek or visit regularly, and our dream of making it something special resonates with you, come out for one of these projects and maybe join the Society!

And anyhow, if you’re in a place where you can visit False Creek, you should do that. And if you’re planning a visit to Vancouver, do drop by. It’s just a really good place. But it could be so much better.

Dark Matter Fun 18 Jul 2022, 9:00 pm

Anyone who’s read one or two of my Long Links curations knows that I’m fascinated by the ongoing conundra in Astrophysics and Cosmology on the subject of Dark Matter. There are a couple of recent developments that are worth noting.


Astrophysicists were kind of forced into concluding that Dark Matter exists for several reasons, but the most obvious is that galaxies spin and the stars on the outer edges are orbiting too fast.

M33 rotation curve

Too fast, that is, given two assumptions:

  1. The only force that matters at galactic scale is gravity.

  2. The action of gravity is (classically, not relativistically) described by Isaac Newton’s Greatest Hit:

    F = G m 1 m 2 r 2

If you’re using a browser that supports MathML (e.g. Safari or Firefox)
you’ll be seeing the famous F=G(m1m2/r2) looking like real math.
Chrome doesn’t think the Web needs math.

Most experts in the field now believe that galaxies are surrounded by huge halos of Dark Matter; arrange it right and then ol’ Isaac’s formula has the stars moving just the way we observe them. It seems like regular matter is a fifth or less of the total amount of gravitational mass in the universe. Dark Matter usefully explains other things about the structure of the Cosmic Microwave Background and the layout of matter in space generally.

The fact that we can’t see it through telescopes or with particle detectors probably means it must be a new form of matter. Which makes physicists’ eyes light up, and they’ve been trying to detect the stuff for decades. And failing, and failing, and failing. They’ve had multiple perfectly reasonable theories for what kind of particles might make up Dark Matter and devised insanely clever experiments to catch them, but no luck.

Eventually someone said, “Uh, about those assumptions…” In particular the second, which is that Newton’s law applies universally, both in the relatively matter-dense neighborhood that I and you, Dear Reader, inhabit, and out in the millions-of-times-less-crowded space at galaxies’ edges.

Thus, Modified Newtonian Dynamics (“MOND” for short). That Wikipedia link has a nice little introduction and the math in the overview isn’t too scary. It introduces a new fundamental constant a0, the (very, very low) level of acceleration where gravity maybe starts to act differently.

MOND isn’t an actual physical theory, it’s just a hypothesis that predicts the observed behavior of most galaxies very nicely without the help of Dark Matter. As noted above, most physicists still think the Dark Matter is out there, we just haven’t figured out how to detect it. But not all physicists.

The Elephant in the Universe by Govert Schilling

What’s new?

Two things. First, there’s a new book giving an overview of the problem: The Elephant in the Universe: Our Hundred-Year Search for Dark Matter, by Govert Schilling, who seems an accomplished science writer. It was given a mostly-positive review in the WSJ by Katie Mack, one of my heroes, so of course I bought it. [Warning: If you follow that link and buy it too, not only will you be sending money to Amazon, I get a cut too.]

It’s a big subject but it’s not a big book; I plowed through it in a single evening. He introduces you to the people leading the Dark-Matter charge and they’re interesting people. It’s even got pictures, and he keeps the math under control. I don’t pretend to deep understanding of Real Astrophysics Math but the author did a good job (in my opinion as the holder of a little-used decades-old math degree) of giving the essential flavors of the problem.

Amusingly, his chapter on the folk interested in MOND is entitled “The Heretics”.


This is currently the largest and most advanced working dark-matter detection experiment, situated in an old mine a little over a mile under the soil of South Dakota. It has just been spun up and the results of its first sixty days’ real science were released on July 7 as a published paper and a presentation: slides and video.

At the center of the detector is seven tonnes of liquid xenon, surrounded by really a whole lot of other stuff. The flavor of Dark Matter they’re looking for consists of Weakly Interacting Massive Particles, which physicists call WIMPs. (Another theorized but not observed flavor of Dark Matter particles is referred to as MACHOs, for Massive Compact Halo Objects.)

Inside the LZ detector

Inside the LZ detector. The real action is in the white central cylinder behind the enclosing transparent one.

I watched the video on YouTube and enjoyed it immensely. The detector is an absolutely fabulous construct, full of raw steampunk beauty, and their description of the problems they’ve had to solve to get the thing to work was absolutely compelling.

So, did they find any WIMPs? They did not, in sixty days’ worth of data. But they plan to run for thousands, and then we’ll see. As with previous Dark-Matter-hunting experiments, they did lower the ceiling on the hypothetical WIMP’s cross section to 6x10-48 cm2. There are two other experiments of a comparable scale spinning up, and the research groups have already got together, decided they need a still-bigger detector, and founded the XLZD consortium to drive that.

Will they find their WIMP?

Nobody knows; that’s the point. But they’re routing millions and millions in research funding at the problem, and literally thousands of researcher-years of effort.

The people who find MOND interesting are dubious. One example would be Stacy McGaugh, a professor at Case Western University, who has a well-constructed and highly readable blog called Triton Station. He discusses the LUX-ZEPLN results in LZ: another non-detection.

He is polite but fairly cutting: “As remarkable a technological accomplishment as experiments like LZ are, they are becoming the definition of insanity: repeating the same action but expecting a different result.”

For their part, the (large majority) who think the Dark Matter is out there and we just have to measure harder are serenely confident.

Isn’t it fun?

Living With Models 13 Jul 2022, 9:00 pm

More people, more often, are finding themselves dealing with the output of Machine-Learning (ML) software. With even a little practice you can spot when it’s happening. It makes me wonder what the future of human/ML interaction looks like. Herewith a few personal experiences.

Gmail spam

One of the world’s largest software systems with an ML component? Spam used to be such a major irritant but now mostly it isn’t except when it is. Obviously, Gmail judges itself by how well it avoids errors both positive (branding legit email as spam) and negative (letting it through to get in my face).

It veers back and forth as they evolve the model. Occasionally some class of spamster figures out how to fool it — I remember when it was translators, most recently it’s subject “Invoice” with a Geek-squad logo in the message body.

But the positive errors are more damaging, because I don’t wanna visit the spam folder ever really, but I feel like sometimes I have to. And when the model goes off the rails in the positive direction, it’s baffling: Posts to mailing lists where I often speak up, notes from teachers about a child’s school issues, things that I really care about.

Protip: Type “/in:spam” into Gmail.

As I write this, Gmail has been running smoothly down the right side of the track in recent weeks, which to say admitting a few spooky spams but filtering out nothing of any value. I bet it doesn’t last.

And I confess to enjoying certain rare classes of spam: Residential real-estate in Mozambique and Mongolia, used Heidelberg presses for sale, lurid love offerings in hilariously broken English.

Jaguar wipers

I mean the ones on my car’s windshield. On an extended drive in variable precipitation, they get dialed in and are amazingly great, flipping the blades back and forth at just the right speed for conditions between drizzle and downpour.

But boy, do they get confused when they’re initially trying to get the picture, flailing furiously at an occasional speckle on the glass, or steadfastly refusing to move when I’m looking through considerable snowflake impact.

Protip: Hit the wiper-fluid button. Which apparently causes it to flush all its caches and begin learning from scratch; almost always yields a sensible state.

Streaming smarts

Back to Google, which is probably OK because they’ve exposed as many humans to ML-model output as any other organization on the planet.

It’s taken literally years, but I have successfully battered YouTube Music’s Your Supermix stream into sanity. I signed up when Google Music kicked the bucket but offered a promise, on its dying breath, to pass along my personal library of twelve thousand or so songs.

It asked me, while getting started, to name some musicians I liked, but then literally crashed when I picked eighteen or so from its alphabetical list. Maybe the problem was they represented too many unique genres?

Anyhow, it eventually got the idea that I liked soft, dreamy stuff and besieged me with Bohren & der Club of Gore and Skinshape and Emancipator which, well OK, all of them, but could we rock out or hear a pop tune occasionally? Yeah, now we can. Here’s a random sampling of my personal “Supermix” on this Wednesday evening in July 2022:

Tim Bray’s YouTube Music “Supermix”

A lot of these are good.

The nice thing is that on another evening it’ll bulk up on Coltrane and Muddy Waters and Brenda Lee. Also, it seems to be sensitive, in a good way, to time of day.

It’s not perfect. In particular, it’s completely fucking useless on classical music.

Protip: Be aggressive with the Thumbs Up/Down buttons.

This model is unlike any other in my experience in that adapts to input stimuli over a time-frame measured in months-to-years.

The future is full of silent struggle

The above are just anecdata. But just as thirtysomethings are the first generation to have grown up never without Internet, today’s very young will be those who will grow up never not interacting with ML models. And remember, the ML model is never on your side; its primary agenda is the agenda of whoever paid to have it built.

I think people younger than I will become skilled at influencing ML models (a skill they mostly won’t even notice they have) and that this skill will increase more or less in parallel with the models’ builders’ skill in meeting the builders’ undisclosed internal business goals.

I retain some shreds of optimism that the humans will stay ahead, at least for a while.

Long Links 30 Jun 2022, 9:00 pm

Dear Readers, once again a selection of links to long-form pieces which I, as a semi-retired person, have time for, but people with normal lives probably don’t. The hope is that one or two will enrich your life. Housekeeping note: I have said “Yes” too often recently and Long-Linking opportunities will probably shrink. Featured this time out: American roots music, BC beer, how wealth vanishes, and the Apple M2.

Bill Strings band with Molly Tuttle

Let’s start with the music. My streamer has noticed that I really like Sierra Ferrell and has been feeding me increasing doses of acoustic American flavors. While I’m not giving giving up on Bach or alt-metal, I’ve enjoyed it. I particularly recommend Billy Strings and Molly Tuttle charging full-throttle through Little Maggie, and AJ Lee and the Brothers Comatose gracefully covering Neil Young’s Harvest Moon.

This one is specific to my part of the world, but hey, we’re a tourist destination and if you’re going to visit us, I think a close study would reward your vacation planning. Local CBC reportor Justin McElroy, who has found a lovely niche in ranking things (neighborhoods, parks, welcome signs) has ranked all the breweries in the bottom left corner of Canada. A few of these were already among my faves and I surely plan to visit more, particularly those within bicycle range.

For your enjoyment, a piece of excellent propaganda. Recently I accidentally studied the Demerara Slave Rebellion of 1823 (even wrote about it). This is a big link in the chain of events and activities that eventually led to the British abolition of slavery. One of the activists’ tools were the 19th-century equivalent of blogs, namely leaflets. Courtesy of the Internet Archive, here is Cushoo: A Dialogue.

Cushoo: An anti-slavery pamphlet

It’s short and snappy (and racist of course, in the manner of its time) but well-done. We know this was effective because the colonial plantation-owners lobbied ferociously to get it banned and, above all, to keep it from circulating among their own slaves. They failed.

There’s no geek-out like a linguistics geek-out. The Economist covers the strange reverse-transatlantic journey of the subjunctive case in modern English. Fun!

Way back when the Net was young, Kevin Kelly suggested that you could make a living with one thousand true fans. Which turned out to mostly not work. But, maybe it’s starting to?

Humans rely on tools. Here’s a convincing curation of nice ones.

OK, this has been a pretty cheery long-link curation so far. But it’s 2022 and There Are Politics and They Are Not Good. Elephant in the Room addresses something that’s going on but is really hard to talk about: The fact that many progressive organizations have been ineffective in recent years because they’ve been tearing themselves apart in an effort to live the progressive ideas they exist to promote. Empirically, we are really bad both at avoiding institutionalized racist/sexist/homophobic culture and at constructively working to get out from under that shadow. This piece is long and (I think) balanced and (I think) important. I have experienced, indirectly, some of the heat and churn being discussed here. We really need to get better at this stuff.

It is no secret that the Big Techs actively try to “manage out” their least effective employees. It’s easy to splash a broad brush of condemnation on this but on the other hand, when an organization is growing by 25%+/year, you’re going to screw up some of that hiring; what to do about it? I sure don’t know. Anyhow, the conversation continues, and I thought “Standing Up For Us Plebs.” Amazon Leaders Reject Policy To Push Employees Out, about an Amazon division going rogue on this issue, sheds illuminating light.

Eric Alterman is a really good writer on politics and popular music. There are a lot of terrible things in the world, and the continuing bloodshed and brutality in Israel/Palestine has been one of them for pretty well my entire life. Alterman surveys the (terrible) state of affairs in 2022 in Israel and Palestine and the Absence of a Solution.

There’s a lot of financial doom-and-gloom going on, you hear stories about how a trillion dollars worth of cryptocurrency value vanished, or how Bezos or Musk or whoever “lost billions”. It’s reasonable to ask “Well, where did that money go?” Noah Smith explains: Where does the wealth go when asset prices go down? It vanishes into nothingness. While I’m as anti-billionaire as the next progressive, I was sort of annoyed, in the recent stock-market run-up, at the feverish headlines about how Bezos was making a million dollars a minute. In fact, wealth due to ownership of financial instruments was being created out of thin air by equity-market machinations; and now it’s being un-created. Poor Jeff has probably lost more money in 2022 than any individual human in history has ever lost before. Not feeling sorry for him. And anyhow it’ll probably all come back. Or maybe not. Whatever. Oh, and the basic politico-financial infrastructure that lets this happen is broken, obviously.

This wouldn’t be a Long Links without an anti-cryptocurrency screed. These days, the stinkiness has become so obvious that it’s kind of hard to find anything new to say. Kyle E. Mitchell, whose blog is called /dev/lawyer, has a new angle in The Work, the Tech, and the Crime. As you’d expect, the viewpoint is that of a working lawyer. Let me pull a paragraph for flavor: “I don’t support cracking down on blockchain people just for being blockchain people. A scene being riven with datajackers, confidence men, and self-taught, emoji-adept bucket shop jockeys does not condemn others not so involved by abstract association, be they merely fools or holdout true believers. But neither do I support special accommodations for those in denial or indifference to the unwelcome company they keep.” Good stuff.

Let’s finish up with a hard-core tech geek-out. (Civilians can close the tab now, my feelings won’t be hurt.) Those who actually care about the nuts and bolts of CPUs and how they fit into the computers in your pocket and on your desk will enjoy Apple M2 Die Shot and Architecture Analysis – Big Cost Increase And A15 Based IP. I’m particularly impressed by the M2 design’s apparent focus on RAM bandwidth and latency; all CPUs wait for memory at the same speed.

Until next time, whenever that is.

Small Tables 25 Jun 2022, 9:00 pm

Computer programs organize bits and bytes into “data structures”. In software of any import, the data structures are usually more interesting than the code around them. This part of the Quamina Diary takes a close look at a very simple data structure that I have greatly enjoyed using to build finite automata, and which I think has lessons to teach; it’s called smallTable.

The problem

As described in a (pretty short) previous episode of this Diary, Quamina matches Patterns to Events, which are flattened into field-name/value pairs for the purpose. We try to match field names and values from Events to those offered in a Pattern. Matching names is easy, they’re immutable strings both in Events and Patterns, and thus the following suffices.

transitions map[string]*valueMatcher

Now, inside that valueMatcher is a harder problem: How to match the value of a field, because patterns can include incantations like *.jpg and "anything-but": ["foo"] and so on. Clearly a map keyed by a string isn’t going to do the job. In fact, we need another flavor of finite automaton that works at the level of characters within fields.

Characters or bytes?

If you want to run an automaton over Unicode text, you have to decide whether you’re going to transition on code points, of which there are potentially 1,114,112 (although only 144,697 are in use as I write this), or on UTF-8 bytes, for which there are only 245 distinct values. Yes, a byte has eight bits, but the values 0XF5-0XFF can’t appear in correctly constructed UTF-8. Which is better, code points or bytes?

On the one hand, here are generally fewer code points than UTF-8 bytes in a string, which means fewer steps to get through your automaton. On the other, the Events coming in over the wire are almost certainly already in UTF-8, and anyhow, a lot of things you’re going to match will be ASCII (for example, numbers), where the number of bytes and characters is the same. For Quamina’s predecessor, which I wrote while at AWS, I picked bytes and never regretted it. And given how easy Go makes it to work with []byte, I wasn’t tempted to do anything else.

Goals and constraints

Let’s say we have a step type that represents a transition between automaton states in response to a UTF-8 byte. Here are two simple Go idioms that represent that kind of state table: map[byte]step and [245]step. Neither of these made me happy, based on prior experience.

First of all, I know that size will be a problem; adding a million Patterns to a Quamina instance is something that people will want to do, then they’ll complain about memory burn.

The second lesson came from gathering statistics on state machines in the wild. Many of the UTF-8 states have only one or two branches, and even in big complicated automata, the average number of out-transitions from a state remains well below 10.

Third, when you’re building automata, it’s common to want to express “transition on a range of byte values.” For example, to match a wildcard construct like *a you transition to a new state when you see a, and loop back on any other value. Similarly, you might be interested in transitioning on anything that’s a decimal digit while matching numbers, or on hex digits while matching IPv6 addresses.

Both the Go map and array fragments above look like they’re going to use way more memory than needed, and in particular Go’s map construct is not simple and not free to initialize. It’s complicated enough that, even after considerable searching, I couldn’t figure out how it actually implements map[byte]. And neither simple option is very appealing for storing byte ranges. So…

Meet smallTable

You can look on GitHub, but here it is:

type smallTable[S comparable] struct {
    ceilings []byte
    steps    []S

It’s generic and the S stands for “step”; the steps are different for deterministic and nondeterministic automata, but the logic for processing them is the same.

First, here’s the code for “find the step that corresponds to a UTF-8 byte value”.

// step finds the member of "steps" in the smallTable that corresponds to a UTF-8 byte value. It may return nil.
func (t *smallTable[S]) step(utf8Byte byte) S {
    for index, ceiling := range t.ceilings {
        if utf8Byte < ceiling {
            return t.steps[index]
    panic("Malformed smallTable")

“Wait,” you say, “that’s going to run off the end of the array!” Here’s why it doesn’t.

const byteCeiling int = 0xf6

// newSmallTable mostly exists to enforce the constraint that every smallTable has a byteCeiling entry at
// the end, which smallTable.step totally depends on.
func newSmallTable[S comparable]() *smallTable[S] {
    var sNil S // declared but not assigned, thus serves as nil
    return &smallTable[S]{
        ceilings: []byte{byte(byteCeiling)},
        steps:    []S{sNil},

Here’s an explanation-by-example of how smallTable works. Suppose we want to model a table where byte values 3 and 4 map to ss1 and 0x34 maps to ss2. Then the smallTable would look like:


Suppose we wanted the same thing, but to map all values other than 3, 4, and 0x34 to ss3. Then we’d have:


Invariant: The last element of ceilings is always byteCeiling. [The attentive reader may be wondering why byteCeiling is 0xf6 rather than 0xf5. We’ll get to that in a subsequent article.]


Everybody knows that dereferencing a hash table (as in map) is O(1), so why would I want to search sequentially? Bear in mind that I’m pulling in a byte array, so I get eight or more ceiling values for each memory fetch, and we know that the size of ceilings is usually less then ten. So I figure that I can run (on average) halfway through the byte array by the time the hashtable code has done a couple of memory fetches and computed a hash. (All computers wait for memory at the same speed.) That’s why ceilings and steps are separate arrays, as opposed to an array of ceiling/step structures. It also helps that the code itself is very compact and should be cache-friendly.

Speculation aside, it used to be a hash table and it didn’t slow down when I switched in smallTable. Plus, it used way less memory. Plus, it represents matches to ranges of bytes very nicely.

It’s not perfect

There’s a lot to like about smallTable. And, one big problem: It’s hard to update. A couple of times I’ve tried to write the code to change an arbitrary byte mapping in an existing smallTable and pretty soon my head started to hurt. So what the code’s doing now is unpacking the smallTable into a unpackedTable:

// unpackedTable replicates the data in the smallTable ceilings and steps arrays.  It's quite hard to
// update the list structure in a smallTable, but trivial in an unpackedTable.  The idea is that to update
// a smallDfaTable you unpack it, update, then re-pack it.  Not gonna be the most efficient thing so at some future point…
// TODO: Figure out how to update a smallDfaTable in place
type unpackedTable[S comparable] [byteCeiling]S

The support routines look like this; the body of code isn’t that interesting:

func unpackTable[S comparable](t *smallTable[S]) *unpackedTable[S]
func (t *smallTable[S]) pack(u *unpackedTable[S])

The profiler shows that in situations where you’re adding a lot of patterns to a Quamina instance, this approach is expensive, especially that pack call. I’m pretty sure that with more head-scratching and a bit of inspiration, I’ll find a faster way to update these things.

smallTable makes me happy. I think that if you’re building an automaton driven by a range of small numbers with modest state fanout, you probably need something much like it.

Demerara Rebellion 12 Jun 2022, 9:00 pm

I accidentally ended up learning a lot about the Demerara Slave Rebellion of 1823. It’s a fascinating set of stories, worth sharing, worth learning about.


I’d always liked the name “Demerara” partly because my Dad loved molasses (on pancakes, toast, whatever) and so my childhood memories are rich with images of a big sticky black pot of Demerara Molasses on the breakfast table. I’ve recently released a software project I’ve been working on for a few months and thought I’d use that name. But I did a web search for “demerara” and thus discovered the 1823 Rebellion (shame on me for not having known about it). Anyhow, I quickly ran across multiple statements that the Rebellion’s leader was Quamina, and loved that name so much that I stole it for my project. Here it is on this blog and on GitHub.

Apologies and triggers

I regard myself as reasonably historically literate but somehow had never learned much about slavery: Its culture or economics or the experience of the enslaved. Again, shame on me.

Now, trigger warnings, and I mean all of them: This story is full of appalling racism and sexual abuse and sordid death and sickening hypocrisy. Blecch.

The sources

There’s been less published on this than you might think. I recommend starting, as I did, with Account of an insurrection of the negro slaves in the colony of Demerara by Joshua Bryant, who was an accountant in Georgetown, the colony capital, and a member of the militia that battled the rebels. The link above is to the Internet Archive, where you can read this for free. It was published the year after the rebellion and is thus very fresh. It’s much more fair than one might expect. Pretty well just the facts and, while the whole thing is implicitly totally racist, I have trouble recollecting anything explicitly racist or mean-spirited from Bryant.

The typography is terrible and the whole book devolves into a series of footnotes that drive the supposed main text into a pathetic couple of lines across the top of the page. But the story is strong and the images are cool.

Retreat of Lt Brady

“Retreat of Lt Brady” from Bryant’s Account.

Plate 9

“Plate 9” (Marine Battalion at Cummingsburgh) from Bryant’s Account. Bryant was a member of this Battalion.

The Gallows

“The Gallows; Five of the culprits in chains, as they appeared on the 20th of September, 1823”. From Bryant’s Account.

The second book is Crowns of Glory, Tears of Blood: The Demerara Slave Rebellion of 1823 by Emília Viotti da Costa, published in 1994. It’s a serious modern work of history, competently if not beautifully written, impeccably sourced, usefully illustrated. I may read more on the Rebellion but, after Bryant and da Costa, I don’t feel I really need to to get the picture.

Below is a picture of a page from da Costa’s book, which I borrowed from the public library. More people should take books out of libraries. You should too.

A page of da Costa’s “Crowns of Glory, Tears of Blood”

For books that I want to take notes on and write about, I’ve become very accustomed to Kindle, which makes this easy. But I was sitting up late working on this one with the help of sticky notes and enjoying the penciled annotations of a previous reader whose identity I’ll never know, and thinking “maybe there’s something to this physical-book idea.”

The story

Demerara produced cotton, coffee, and sugar on plantations which were long and narrow, one of the narrow ends up against the sea or a huge river. The work was labor-intensive, and the labor was slave labor. The slaves were hideously oppressed. Some of the English and Dutch owners were onsite but many were absentees, represented by a local slave-driver (the first occasion I’ve ever had to use that term literally).

Around 1800 the London Missionary Society started a well-organized and well-funded effort to bring Christianity to the slaves; teach them to read, to know Scripture, and to join the church. Many of the slaves took to this eagerly and the congregations were large. The plantation-owner class hated the missionaries, resisted their arrival, and tried in every way to undermine and circumscribe them. They believed (correctly) that slaves who could read, and who regarded their souls as the equals of free people’s, would be less tractable.

The slave trade was outlawed in 1808 (not slavery, just the extraction of human flesh and blood from Africa). This, and the missionary outreach, were part of a growing anti-slavery movement in Britain, led by Wilberforce.

The movement had not been successful in outlawing slavery, but in 1823 they managed to get an Order In Council which forbade the flogging of female slaves and limited working hours to nine per day.

The colonial administration and the planters hated this and dragged their feet on implementing or even announcing it. But word got out among the slaves, many of whom came to believe they had been given their freedom by the King and that the local administration was keeping it from them.

The revolt broke out on the 18th of August, 1823, and was joined by over ten thousand slaves. The level of violence was low; few of the white planters and administrators were killed or even injured. The Governor declared martial law, supplemented his existing military forces by conscripting basically every white male, and sent them out to fight, well-armed and reasonably well-organized.

It was all over in less than a week. Many of the slaves who were perceived as leaders were unceremoniously hanged on the spot, and then there was a series of ludicrously unfair show trials at which basically every slave brought forth was briskly condemned to death. The executions were public and conducted with high ceremony, as a propaganda exercise.

While the Rebellion failed, it may have changed the course of history. But before I come back to that, let’s meet the people involved.

Dramatis personae

The central figure is Jack Gladstone, the slave who actually led the rebellion. He was Quamina’s son, born a slave. As a cooper, a skilled worker, he had considerable scope for travel among the plantations. He was tall and charismatic and passionate, convinced that the slave-masters were withholding the King’s freedom. I’m going to call him just Jack because “Gladstone” was the name of the absentee plantation owner, which slaves inherited by default.

He lost his wife Susanna when she was taken as mistress by a white plantation manager.

Jack was captured a few days into the rebellion and, weeks later, given an unusually extended and ceremonious trial with, unlike the other slaves, assistance from an attorney. Many witnesses, both colonialists and slaves, were called. Bryant’s book includes a lot of material from this trial. The testimony made it clear that once the Rebellion was ablaze, Jack had taken the lead on multiple occasions in protecting the whites from slaves that had extremely well-founded motivations for inflicting extreme violence on them.

Nonetheless, he was sentenced to hang. Not wanting to create a martyr, the colonial Governor applied to the King for mercy for Jack and a few other convicted slaves, on the grounds of his protection of the whites. The mercy was granted, Jack was sold to another slave-owner in Santa Lucia, sent there, and vanishes from history.

John Wray was one of the first missionaries in Demerara, arriving in January 1809. He was doggedly determined in the face of the planters’ resistance, and got considerable support from the administration back in Britain. Eventually he was successful in reaching out to the slaves and built large and thriving congregations across multiple plantations.

His message was not revolutionary, and heavily emphasized the “Christian duty” of obedience and loyalty to the masters. Whatever he said, the message of the equality of souls was implicit in the Scripture the slaves had learned to read, and they apparently drew their own highly reasonable conclusions about justice and morality. Furthermore, Wray himself treated the slaves with respect as humans, and several of them took up roles as church deacons and catechism instructors. He was proactive in trying to mediate disputes between slaves and masters.

He also preached Christian sexual morality, which was being violently contravened every day by the plantation owners, who flagrantly feasted sexually on the infinitely-vulnerable Black women they owned. Once again, the slaves could and did draw the obvious conclusion.

Finally, he fiercely defended the Sabbath, and the right of slaves to attend Sunday service. This was another irritant to the slave-owners, who felt entitled to seven days of work from humans they saw as property.

In 1816, under pressure from the colonists and following on the death of the pious plantation owner who had supported his arrival and residence, John Wray accepted another position in a nearby colony.

Rev. John Smith

His replacement John Smith arrived in early 1817. Smith took up where Wray left off, enlarging still further the slaves’ enthusiasm for and their practice of religion. He is a truly interesting character; a poor simple Englishman who perhaps joined the Missionary cause mostly in search of a living, a man who became possessed by his mission, a man who was a living lesson that Black slaves could and should be treated like humans, a man who became (in private) revolted at the existence of slavery.

Smith may have known that the Rebellion was about to break out, but was apparently consistent in advising the slaves against it. On the other hand, he angrily refused to join the white militia that was putting the Rebellion down.

Finally, Quamina. He was born in West Africa and kidnapped away to Demerara with his mother while very young. He was a skilled worker, a carpenter. An early and enthusiastic adopter of Wray’s missionary message, he became literate, was baptized in 1808, and was elected Deacon in Wray’s church. Later, he worked closely with John Smith and his wife. Both Wray and Smith regarded him highly, praising him in their diaries

Statue of Quamina i Georgetown

Statue of Quamina in Georgetown, Guyana. It’s unlikely the artist had any reliable sources for what he looked like, but the cross and the cutlass capture an important part of the message.

When Quamina’s long-time partner Peggy became ill, his slavemaster refused him time off to care for her; one day after work he found her dead.

Quamina must have known about the planned rebellion. But when it broke out, he strongly advised the slaves to wait, wait for the King’s freedom to be proclaimed. They didn’t.

We don’t know if he participated in the Rebellion, although the colonial government repeatedly proclaimed he was the ringleader. We do know that he swore he would never be taken alive by the whites, and that on September 16, he was tracked down by a colonial military party, ignored their orders to stop, and was shot dead.

The aftermath

Jack Gladstone’s trial was not the main legal event. John Smith was arrested and charged with having promoted “discontent and dissatisfaction, in the minds of the negro slaves toward their lawful masters, managers, and overseers” and various other charges. He was brought to trial before a court martial in October 1823.

Smith represented himself and consciously turned the trial into a piece of political theater. It’s a hell of a story, nicely told by da Costa.

The court martial found Smith guilty and issued the usual death sentence, but an appeal for mercy was sent to the King across the sea. Mercy was granted, but Smith died of consumption in prison before the mercy arrived.

Which led to a huge scandal back in Britain, where Smith was presented as a saint martyred by the cruel, grasping plantation owners. The narrative had the advantage of being pretty well the truth.

While you can argue about details and might-have-beens, historians seem to feel that that the white-hot scandal around the trial significantly advanced the anti-slavery cause, leading inexorably to the Slavery Abolition Act ten years later. So Quamina and Wray and Smith and especially Jack may not have won their battle, but pretty clearly they helped their side win the war.

Things I learned

Reading old books is good! But aside from that…

  • The abolition of the slave trade (well before that of slavery) was a big deal economically because, for a slave-owner, it was much cheaper to buy a slave off the boat than it was to raise a baby slave into adult servitude.

    Also, a lot of the female slaves were unenthusiastic about bringing children into their life of slavery. So without a fresh crop of victims from Africa, the labor force tended to decline.

    So, of course, there was plenty of illegal slave smuggling.

  • Manumission — the freeing of slaves — was not terribly uncommon, but became much rarer after the abolition of the slave trade. Still, it’s unsurprising that the population of free Black people in Demerara grew vigorously. But “free” didn’t work with the plantation economy.

  • You probably already have a bad opinion of colonial plantation slave-owners. The more you learn about them, they more you’ll loathe them. I absolutely cannot begin to understand how people could live in that role and look at their own faces in the mirror. The abuse, violence, exploitation, and rape was continuous and egregious and probably worse than a hyperoverprivileged person like me can imagine.

  • The white-hot screeds against the missionaries and the evils that they would inflict by teaching humans to read, and exposing them to Christian scripture, are astonishing. I quote: “Our colonies are now inundated with canting hypocritical tailors, carpenters, tinkers, cobblers, etc. who, too lazy to work for an honest livelihood in the Mother Country, and charmed with the idea of living in ease and luxury abroad, found it very convenient to become converts to the new light and volunteer to teach the Gospel without the ability to spell one of its verses.” Sounds like a bottom-feeding GOP politician snarling at Wokeness.

  • The planters made a lot of money… for a while. Then they became victim to falling commodity prices, financial chicanery, and the fact that the money and power were in London, held by people who didn’t care about them. Predictably, when their businesses turned sour, the suffering went mostly to the slaves.

  • Working in the plantation fields wasn’t the worst place a slave could find themself. Their owners regularly hired their slaves out in “task gangs” to other colonials. This was generally a terrible experience because the task-gang customer had no incentives to have a good relationship with the rented people, and typically worked them unmercifully.

  • The slaves were not completely without rights; there was an official called the “fiscal” whose job it was to investigate mistreatment and illegal oppression. Having nothing to lose, the slaves were extremely litigious, bombarding the fiscal with an endless stream of complaints. Most, of course, were quickly dismissed, and often the complainers were punished for speaking up. But people in desperate places will do desperate things in the pursuit of justice.

  • The slaves had an internal economy; they had allotments of land to grow produce to farm and to sell in the Georgetown market. These plots of land were called “provision grounds” and there were regular disputes when the slaves’ owners did not grant them the time to cultivate to which they felt entitled or, worse, seized the provision-ground land to grow their own crops on. I find that term “provision grounds”, to be evocative and sad. It haunts me somewhat.

  • The women among the slaves had it way worse than the men. There was a culture of casual brutality against them, the slave-owners telling each other that the women were trouble-makers, recalcitrant, needed “firm discipline”. I can’t even bring myself to write about the sexual predation and the treatment of these women’s children; it’s beyond horrible.

  • Demerara was a long way from London, and its rulers found it convenient to ignore many of the edicts from the Mother Country urging better treatment of the slaves. Those rulers, however, were unsuccessful in trying to prevent the word about these edicts getting out. This pattern of behavior was a key factor in provoking the sense of grievance that led to the Rebellion.

  • The slaves came from several different African ethnic groups, and obviously (for at least the first few generations) retained some of the African languages and cultural patterns. There is evidence that the leadership of the Rebellion was mostly slaves coming from the Akan people, called “Coromantee” by the colonialists. But of these patterns and structures among the slaves, we know almost nothing, because their owners didn’t care and they had no incentive to share. The missionaries, who were eager to tell the slaves’ side of the story, ignored those stories. Sadly, they seem to have been lost.

    After my reading, I feel I know Wray and Smith pretty well. I have much less insight into the real protagonists, Quamina and Jack.

  • The British post-Rebellion political struggles focused around the person of John Smith. The slaves were given no agency by either side. It is shocking but true that it took the imprisonment and death of one not-very-distinguished white man to launch the forces into motion that, in the end, led to a good outcome.

It’s a dark, terrible story. But there are rays of light in it. I am profoundly grateful that I live in an age where my children are at risk neither of becoming slaves nor slavers. Obviously, in the twenty-first century, we are still trying to find ways to heal the wounds inflicted on our society and culture by slavery. Anti-racism is the price of entry to the community of people trying to heal the wounds. But we’re still not very good at it.

Making Code Faster 10 Jun 2022, 9:00 pm

I’ve enjoyed writing software for 40+ years now. Lots of activities fall into that “writing software” basket, and here’s my favorite: When you have a body of code with a decent unit-test suite and you need to make it go faster. This part of the Quamina diary is a case study of making a piece of the library faster.

How to think about it

When you’re sitting down to create software, you probably start with preconceptions about whether and where there are likely to be performance issues. If you’re writing Ruby code to turn your blog drafts into HTML, it doesn’t much matter if republishing takes a few extra seconds. So, write the first cut of that software in the simplest and clearest possible way, without sweating the milliseconds.

But for code that’s on the critical path of a service back-end running on a big sharded fleet with lots of zeroes in the “per second” numbers, it makes sense to think hard about performance at design time.

At this point, I want to call out to Nelson Elhage’s Reflections on software performance. Nelson approvingly quotes the following pieces of conventional wisdom:

  • premature optimization is the root of all evil.

  • Make it work, then make it right, then make it fast.

  • CPU time is always cheaper than an engineer’s time.

I approve too! But… Sometimes it just has to be fast, and sometimes that means the performance has to be designed in. Nelson agrees and has smart things to say on the subject.

And then there’s the background against which I wrote the Quamina code: It’s a successor to similar library I wrote v1.0 of at AWS, and which is in production in multiple high-volume services, running hot. So I didn’t have to speculate about the kinds of stress that would be applied, nor about what kinds of design goofs would produce bottlenecks.

On profiling

Nobody is smart enough figure out what’s making a piece of code slow by looking at its inputs, outputs, and structure. That’s why we have profilers. Fortunately for me, Quamina is written in Go, which means the profiler is built-in, helpful, and doesn’t even slow your benchmarks down much. The other two languages I’ve used mostly in recent decades are Java and Ruby and the profiler situation is for both those languages is kind of shitty. I had to pay real money to get the Java profiler I used at AWS and while it worked, it was klunky, not fun to use.

Whatever language you’re in, have a look around and find a good profiler you can live with and keep it handy. And if you’re in Go-land, go read up on the profiler (avoid the official docs Google turns up, they’re outdated and useless).

The Go profiler has a very decent command line that will show you readouts in a local HTTP server or textual tables or exported as PDF. I find the latter easiest to use but YMMV. But what I actually use is the built-in profiler module in GoLand, JetBrains’ Go IDE. There’s a “run with profiler” button and when you hit that the first thing you get is an IDE pane full of flame graph. Which takes a little while to figure out but once you do it’s brilliant. It might not always show you what your problem is but it sure as hell shows you where to look. Here’s one of those.

GoLand flame-graph output from the Go profiler

The middle bump is loading the test data, the right third is profiling overhead (I think), the app logic is in the left third.
It suggests we have a close look at
storeArrayElementField and oh, we will.

The GoLand profiler module lets you walk around the call graph and seek out hot methods and so on; I never wanted it to do anything it couldn’t.

I’m not going do a how-to on the profiler; it’s not hard to learn, and you should learn it. Anyhow, most of you kids are in VS Code these days, which I assume has good profiler support. And having said all that, the graphics below are the PDF output from the Go profiler, no IDE involved.

Gotta have those tests

So, you understand what the performance problem is, and you have a profiler setup that you know how to use. But just like it says in the first paragraph, good unit tests are totally essential. Because the profiler’s going to show you a problem, and you’re going to refactor and rewrite to fix the problem. You can do that fearlessly if you have a high-coverage unit-test suite that you can run with a single key combo and comes back in ten seconds or less. Otherwise, sucks to be you.

Because when you refactor and rewrite, you will break things! Guaranteed. But if you’ve got that unit-test backstop, you don’t need to worry, because you’ll find out right away, and you can iterate till it passes and find out if your work helped.

If you need to speed up code and it doesn’t have decent unit-test coverage… well, then, writing the unit tests is the price of admission. Which, at the end of the day, is just another reason to never let anyone check code in that doesn’t have those tests in good shape.

Benchmark choices

With a profiler and unit tests, you have two legs of the tripod. The third is a nice data-driven repeatable benchmark. I use the San Francisco Parcels JSON file, a couple of hundred thousand JSON objects comprising 190MB or so. Big enough to get meaningful measurement; small enough that Quamina can run pattern sets on it in a second and change.

The San Francisco JSON objects have several large arrays full of 19-digit floating-point numbers, which is pretty well a worst-case scenario for the flattener, so most people will see better performance than we do while we’re testing. Finding a good benchmark data set is a good investment of your time.

Back Story

V1.0 of Quamina’s ancestor at AWS (I’ve been calling it “The Matcher”) was about fast enough for EventBridge, so it didn’t really get any optimization love. Then when other services started picking it up, we started hearing the occasional gripe. In particular some team down in the bowels of CloudWatch wanted to use it for something and benchmarked it against a few other options that looked slow to me. The Matcher won, but not by very much, at which point it occurred to me to profile it. Did I ever get a surprise.

As discussed elsewhere in this Diary, matching Patterns to Events is a two-step process: First you have to “flatten” the event into a list of pathname/value fields, then you run the matching automaton. When I fired up the profiler, I found that he Matcher was spending like 90% of its time flattening events. Ouch!

I’d been using a popular JSON library to turn the event into a tree structure, then running around the tree structure to peel out the fields. Which requires a whole lot of memory allocation. BZZZZT! Wrong! Anyone with significant optimization experience will tell you to keep memory management off your critical path; some of my biggest optimization wins over the years have come from pre-allocating buffers. At AWS, I juiced up the flattening code but can’t remember most details. Except for, I used a streaming JSON parser.

Cheap JSON

Well, I wasn’t going to do the same dumb thing twice, so my first cut at Quamina’s JSON flattener used Go’s json package’s Tokenizer streaming API. Thus I was pretty disappointed the first time I profiled a big Quamina run and saw the flattener burning 90% of the time, and and then again when the method burning most of the time was some internal thing with Malloc in its name.

A couple of minutes’ research revealed that yeah, that API is known to be slow and yeah, it’s because it gobbles memory like a hungry teenager. There are a couple of alternate community-built tokenizers; I looked at them but for some reason nothing really appealed. Also I’d like for Quamina to have the minimum in dependencies, ideally zero, just because that makes it easier to adopt.

Also, I realized, I had a couple of advantages over someone writing a general-purpose tokenizer. As I’ve mentioned before, one of Quamina’s designed-in optimizations is that once you’ve added a bunch of Patterns to an instance, you know which fields you need to look at while matching and, crucially, which you can ignore. So while you’re flattening, you can skip over lots of fields and for that, the memory cost should be zero.

Second, to match field values, Quamina uses a byte-driven automaton to support things like * and numerics. So there’s no need to turn the field values into strings or numeric data types, everything can be a []byte. And since a message coming from outside is also a big []byte slice, there’s really no need to allocate anything except for sub-slices, and that only for the fields you care about.

Thus was born flattenJSON, a complete (but not general-purpose) JSON tokenizer in 750 lines of Go. It’s a collection of state-machine mini-parsers; one at the top level, then sub-parsers for objects, arrays, numbers, string values, and so on. As of now, it still takes more time to flatten than match an event, but it’s close. Which means the whole thing sped up by a factor of about five.

What next?

Time to declare victory? Well, let’s break out that profiler again. Here are reports on CPU and memory utilization.

CPU profile of the Quamina flattener Memory allocation profile of the Quamina flattener.

Above: CPU profile. Below: Memory profile.
You can enlarge them, but if you want a quality view
there are PDFs for CPU and memory.

For people who are learning the Go profiler, this command produced files named and with the profile data:

go test -cpuprofile -memprofile  -v -run=TestFlatteningProfile

And here are the command lines that generated the PDFs:

go tool pprof -pdf -show_from flattenJSON
go tool pprof -pdf -show_from flattenJSON

So, what is the profiler trying to tell us? The CPU profile makes it pretty clear that the flattener is spending the bulk of its time managing memory, names like memmove and refillAllocCache are a giveaway. In an earlier revision, the heavy-hitting methods had GC in their names, so at least I’m now avoiding the garbage collector.

If we look what called the heavy hitters, or alternatively just glance at the memory profile, it’s pretty obvious that storeArrayElementField is our prime suspect. Let’s have a look.

// storeArrayElementField adds a field to be returned to the Flatten caller, straightforward except for the field needs its
//  own snapshot of the array-trail data, because it'll be different for each array element
//  NOTE: The profiler says this is the most expensive function in the whole matchesForJSONEvent universe, presumably
//   because of the necessity to construct a new arrayTrail for each element.
func (fj *flattenJSON) storeArrayElementField(path []byte, val []byte) {
	f := Field{Path: path, ArrayTrail: make([]ArrayPos, len(fj.arrayTrail)), Val: val}
	copy(f.ArrayTrail, fj.arrayTrail)
	fj.fields = append(fj.fields, f)

Yep, it makes a new Field struct and then gives it a copy of that array of ArrayPos structs, and then appends the new Field to a list called fields.

It’s that array-copy that’s getting us. Can you think of anything that might reduce the pain here?

I can, but only because I know what this thing is all about. To match correctly, you need to know about the arrays in the element you’re matching so that so that you don’t match fields from different array elements. For example, if you have an array of objects representing the Beatles, you don’t want a match of
{"FirstName":["Paul"], "LastName":["Lennon"]}
to work.

Anyhow, as you run through a long array, it turns out that each Field only needs an ArrayTrail copy in case it has child nodes that will append more stuff to the trail. So if you made the ArrayTrail segmented and Fields could share segments, you could probably re-use the path-so-far segment and avoid a lot of the copying. Or you could think about of having an array of *ArrayPos rather than ArrayPos, so you could share steps and only copy pointers.

Both options would introduce complexity and the second might not even be a big performance win, because later in the matching process you have to run through array and look at each element and you’ll lose a lot of memory locality as you fetch them.

Will I go after storeArrayElementField at some future point? Maybe, if I can think of something simpler. Got any good ideas? Quamina’s already pretty fast.


Test. Benchmark. Refactor. Iterate. It’s not fancy. It’s fun. I have been known to whoop out loud with glee when some little move knocks the runtime down significantly. How often do you do that at work?

Dangerous Gift 2 Jun 2022, 9:00 pm

A friend of mine pinged me online, said “Hey, remember that thing we were talking about doing a couple years ago? I still have one of the domain names, gonna let it go unless you want it.”

I said “sure, I’ll take it”.

Him: “Let me look up how to transfer it.”

Me: “I’ll look up how to receive it. There’s always some bureaucracy and ceremony. But all my domains are at R these days and they’re pretty good at making easy stuff easy.” (“R” refers to a well-known top-25 registrar, whose name I’m withholding to protect the guilty.)

Him: “WAIT. STOP.”

Me: “?”

Him: “I’m at R too. Hold on… this looks easy.” [A minute of silence…] “OK, look and see if you got it.”

So I sign into R and list all my domains. “Holy crap, there it is. That was easy.”

So I told him thanks. But that evening, the transaction kept rattling around in my mind, and I was getting less and less comfortable.

Because I was thinking, maybe a bad actor could use this to SWAT me. Suppose the bad actor has an account at R, held by some anonymous tax-shell company in a remote jurisdiction, and they own plenty of domain names, maybe innocuous, maybe horrific, suggesting torture, suffering children, revenge video, death camps… Suppose they posted truly horrific (and violently illegal) stuff at some IP address on a “bullet-proof” overseas server, pointed one of their names at it, transferred the name to me, and then tipped off law enforcement about this horrific abuse being hosted by some guy named Tim. How long till my front door gets broken down?

That evening, I mentioned it to my spouse who is also my business partner and she said “Oh yeah, I wondered what that was about, I got an email from R saying your buddy had transferred a domain name to you.”

I inquired if they’d asked her to do anything to accept the transfer and she said “No, but it did have a number to call if this wasn’t kosher.”

Which might help avert the nightmare SWAT scenario, assuming you are the kind of person who diligently keeps up on your email inbasket and promptly reads bureaucratic-sounding emails from domain shops.

And anyhow it seemed too obvious; surely there must be some policy or regulation in place to keep this kind of awfulness from happening?

Well, I hang around the IETF (I’m currently co-chairing a very minor working group.) And in the IETF are people who know people who Really Know Their Stuff about how domain names work and are regulated, in practice. So I found one of those people.

I told him the story and asked “Is what happened there legal, and could a bad actor make it look like I operated a bad domain?”

Him: “Dunno about ‘legal’ because I learned what IANAL means about 35 years ago. But, yeah, a bad actor could make you look bad because when the police look at the WHOIS data for the domain name, your info would be there, and it would be assured by your registrar, who is also the registrar for other domains that you own.”

Me: “So, are there any regulations or policies or, you know, laws, that apply here?”

Him: “There are definitely contractual agreements between ICANN and every registrar. Looking at ICANN’s official transfer policy, what happened seems to fall well outside of Section II.C.1.2, which clearly says that R must ‘Obtain confirmation of the Change of Registrant request from the New Registrant, or a Designated Agent of the New Registrant’ before R ‘process[es] a Change of Registrant from the Prior Registrant to a New Registrant’. Sending a message afterwards doesn’t seem to pass the sniff test for that, at least to this non-lawyer.”

Me: “Practically speaking, what do you think might happen if the bad guy made this move and tipped off the right law-enforcement agency?”

Him: “It really depends on how savvy that LEA is. These days, one would hope LEA officers would at least look at who owns the domain name, but you just said that the registrar transferred it to you and changed the WHOIS data to use your full name and address. I don’t see how they could distinguish this from you registering it yourself unless they notice that the name had transferred recently.”

Me: “So, what should we do about this?”

Him: “Tell ICANN. They’ve got a compliance department who deals with registrars and registries who don’t follow the contracts. You could instead just tell R, but I can’t really imagine a scenario where even a great tech support person would both understand the problem and be able to get it to the right people on their legal team in an reliable fashion.”

“Me: OK, will do. It seems likely that if R is doing this, some of the other thousands of registrars are too. Hey, there’s a blog piece in this, and maybe another when it gets resolved.”


This is actually co-authored by me and one of the friends who appears in the conversations above, who prefers to remain un-named.

Field-Value Automata 19 May 2022, 9:00 pm

When I introduced Quamina, I described the core trick: You prepare an arbitrary JSON blob for automaton-based matching by “Flattening” it into a list of name/value pairs, then sorting them in name order. Today, a closer look at how you work with the flattened data.

This is one of a series of essays on programming topics motivated by my work on eventing services at AWS and, since then, on Quamina, a content-filtering library implemented in Go (GitHub).

By example

Consider the following Pattern:

{"x": ["a"], "y": [1, 2]}

It should match any JSON blob containing a top-level x field whose value is the string "a" and a top-level y field whose value is 1 or 2.

Since we sort the fields of incoming events in order of name, we know how to build the automaton. There are two things that aren’t obvious:

  1. Matching each field is a two-step process: First, see if the field name matches, then the value.

  2. This pattern doesn’t care about any fields except for the top-level x and y. So the automaton has to bypass all the others.

Given that, here’s the little automaton that could:

simple finite automaton


There are two kinds of things in the picture: field matchers and value matchers, labeled in the diagram with names beginning fm and vm. In Go code, they’re implemented by types named fieldMatcher and valueMatcher.

The matchers transition straightforwardly, looking for an x field with value "a" and then a y field with value 1 or 2. But, each of these matchers have a * representing “anything else” that loops back to the field-matcher state. This is how fields that don’t appear in the pattern are ignored. The automaton loops in fm0 until it sees an x field, then moves along to fm1 if the field’s value is "a", otherwise looping back to fm0. If it does get to fm1, it’ll loop there forever until it sees a y field with value 1 or 2, transitioning to fm2 or fm3 respectively and bypassing any other fields we don’t care about.

The big red !T on fm2 and fm3 says reaching either state means you’ve matched the pattern T that was added in the code sample above.


I’d used finite automata more than once before this, but on every other occasion I was parsing a programming language or Internet data format, which have none of this “skip any fields you’re not looking for” crap. Which meant that the state machine I drew on the whiteboard in my AWS office looked like the one just above, but without those loopbacks labeled *. Yeah, even though they’re implicit in the design. What can I say, I was in a hurry.

So when I sat down to write the code to traverse the automaton, I just couldn’t figure out how to make it work. I felt, as I often feel, that I’d gotten in over my head.

Since I wasn’t smart enough to write down the correct automaton, I had to figure out how to code around the one I had. Sitting on the living-room couch in my Mom’s over-full house at Christmas 2014, I came up with this. Go check it out if you like reading code, but let me try to explain it…

In English

You have an automaton that looks like the picture above, only without the * back-links. It has a start state. You don’t know which fields in the event (if any) are going to match the pattern. You do know that any match has to begin with the start state.

So, you make a little type called a Proposal, which says, state S might match field F. And you have a pool of Proposals to work on. To start with, that pool contains one proposal for each field, suggesting that the start state might match it.

Then you turn a loop loose that runs as long as there are any proposals in the pool. It reads a proposal and tries matching its state to its field. If it works, which means the field name/value combo transitions to another state, you toss proposals for that state and all the following fields back into the pool. Let’s work a quick example

Suppose you have 3 fields (F0, F1, F2), so you load the pool with proposals for F0/Start-State, F1/Start-State, and F2/Start-State. Let’s say that neither F1 nor F2 matches the Start-State, but F0 does, transitioning to State-X. So you toss proposals for F1/State-X and F2/State-X into the pool. F1 doesn’t match State-X but F2 does, transitioning to State-Y. State-Y has an annotation that you’ve matched a pattern, so you have something to return to your caller. The pool is now empty and there are no more fields after F2 to build new proposals, you’re done.

The fact that the fields are sorted by name really matters here; as you work your way through the automaton, you never have to worry about transferring back to a previous state.

It’s all in less than fifty lines of code (once again, starting here), which I’m not going to try to squeeze into this skinny blog column. If you’re looking at that code, please ignore (for the moment) the static about “exists:false matches” and “Array conflict”; but both are maybe interesting enough to get a write-up later in this series.

I don’t know if this approach to traversing automata (a) has been investigated, (b) has a name, or (c) is any good. I do know that it worked really well in practice, handling many millions of events per second in multiple AWS services.

I’ve done a bit of pen-and-paper analysis and don’t think the amount of work is meaningfully different from a conventional traversal. But it did occur to me that in principle this approach could be made multithreaded; you could process multiple proposals in parallel on different cores. But anyhow, the profiler says this part of traversing the automaton is hardly visible as part of the total compute. So I left it this way in Quamina for sentimental reasons.


In the first cut, the field matcher was just map[string]*valueMatcher and the value-matcher was map[string]*fieldMatcher. It worked OK and the fieldMatcher is still like that but, for reasons I’ll write about later, that’s a bad choice for the valueMatcher.

Smarter than me

At some level, Quamina is. One symptom is places like this in the code, distinguished by extended verbose comments. These are where I got stuck and bashed my head across the wall until I got something that worked, and knew I’d have no chance of understanding it later (nor would any subsequent visitor) unless I could squeeze out a coherent English explanation. As Prof. Feynman said, if you can’t explain something in simple language, you don’t really understand it. The observation that computer programmers can build executable abstractions that work but they then have trouble understanding is not new and not surprising. Lots of our code is smarter than we are.

The code is also smart because at AWS I had extremely talented collaborators who added things that I didn’t think were possible until they worked. The proportion of the useful ideas in here that are actually mine is probably less than 50% now.

Finally, we had the insane luxury of running this in production against millions-per-second event flows and watching what broke. And of hearing from other teams using it about what they had managed to break. I guarantee: Nobody is smart enough to predict the behavior of software under this kind of stress without experiencing it.

So if you are face-to-face with a piece of production software that’s new to you, and find yourself feeling stupid, don’t worry about it. That software represents a collection of flashes of peak-energy inspiration from people more or less as smart as you, and there were probably quite a few of those people, and it’s been enhanced with the wisdom that comes only from being used at length for real work.

If course it’s smarter than you. That’s OK, you can still improve it.


As of late May, Quamina has picked up a couple of collaborators with way more GitHub expertise than me, and its repo is growing all sorts of bells and whistles, mostly on the CI/CD front. Which means that I hope to do a release next month and see if anyone actually wants to use this. Also, more stuff to write about!

Golang Diaries: Generics 14 May 2022, 9:00 pm

My coding time this year has been invested in Quamina, an event-filtering library implemented in Go. Just in the last couple of weeks I spotted an opportunity to bring Go’s shiny new generics to bear, and not just anywhere, but for a central data structure. I got good results, but the process was sort of painful; I kept trying things that looked like they ought to work but didn’t. I’m sharing this tour through that experience because I’m a reasonably competent programmer with mainstream tastes and if I hit these bumps in the road others probably will too.

[The title of this piece suggests it’s a continuation of two Golang Diaries entries I wrote back in 2013 when I was first discovering the language. Why not? I have at least one more in mind.]

Why generics?

Quamina is all about building and running finite automata. To do this you need a table-like data structure that represents states and the transitions between them. Quamina uses both deterministic and non-deterministic finite automata, DFAs and NFAs for short. The only real difference is that a transition from a DFA state is always to one other state; in an NFA you can have transitions to multiple others, in practice to a list of states.

I noticed that the NFA and DFA tables were almost identical, as was the code that that built them and stepped through them while matching. I had recently read When To Use Generics on the Go Blog, which has sections entitled:

  • When using language-defined container types,

  • General purpose data structures, and

  • Implementing a common method.

Check, check, and check. Then in the final section, One simple guideline, it says “If you find yourself writing the exact same code multiple times, where the only difference between the copies is that the code uses different types, consider whether you can use a type parameter.” Check. So, off I went. Later that week it was all working. But along the way I yelled at the computer more than I’d have expected.

I don’t recommend diving into the Quamina source at this point, there’s too much else to explain. So…

Back to basics

To help illustrate my bumpy road, I made a toy GitHub project with a file called typing.go. It defines a genericized container type:

type container[S comparable] struct {
  label string
  list  []S

And one associated function:

func (c *container[S]) size() int {
  return len(c.list)

Now I’m going to define a boring struct type:

type fish struct {
  species string

Strong typing oops

Here’s where things started going off the rails. I offered the following and neither the IDE nor the compiler complained:

type fishTank container[fish]

“Wow”, I told myself, “now I have a strongly-typed container, so I can write nice clean code without any generics cruft!” On that basis, I defined another function:

func (t *fishTank) fishCount() string {
  return fmt.Sprintf("How many fishies? %d!", t.size())

Ouch! The Goland IDE said Unresolved reference 'size' and the compiler (via go build) said t.size undefined (type *fishTank has no field or method size).

It seems to me like fishTank should have a size() method? After all, it’s a container, and the compiler seemed to be OK with me saying so, and container has size().

Note: Dear reader, as I proceed through this narrative, if you are Go-erudite it will doubtless occur to you that:

  1. There’s a better (and supported) way to do what I’m trying to do.

  2. I shouldn’t want to do what I’m trying to do.

I welcome the first type of remark; one goal of writing this piece is to provoke them. As for the second, I really don’t care what what you think I should want. As I said above, my hypothesis is that enough others will also want to do this type of thing that the issue’s worth addressing.

Update, the morning after (see #1 above)

So there is a way to accomplish what I want, like so:

type fishTank struct {

Thanks to the commenters below and over at YCombinator for the guidance. It’s not obvious to me that this is “better”, though. Um, would it be rude to say that it’s counter-intuitive and requires understanding things that ordinary folks in the code mines shouldn’t have to? But OK, it works, and I’m going to go back and make some pieces of Quamina a little more readable.

Fat fish

Let’s enrich that fish type a bit:

type dataRichFish struct {
  species string
  datum   [8]float64

Now, suppose this: First of all, not all fish have that datum material available. Second, this app goes into production and we start to track a whole lot of fish, millions and millions. And people start yelling at us to use less memory. Well, Go’s interface mechanism should make this easy:

type flexibleFish interface {
  getSpecies() string
  getDatum()   [8]float64
func (f *dataRichFish) getSpecies() string {
  return f.species
func (f *dataRichFish) getDatum() [8]float64 {
  return f.datum

var dataNotProvided [8]float64
type stingyFish struct {
  species string
func (f *stingyFish) getSpecies() string {
  return f.species  
func (f *stingyFish) getDatum() [8]float64 {
  return dataNotProvided

See how it works? A flexibleFish can either have an attached datum or not and if it doesn’t, it signals the absence by returning the static global dataNotProvided.

Now let’s put those flexibleFish in the tank…

type flexibleTank container[flexibleFish]

Ouch! Quoth the compiler: flexibleFish does not implement comparable.

So interfaces aren’t comparable? Hmm, but I can make(map[interface{}]int) and don’t you have to be comparable to be a map key? I guess an interface could be implemented by something whose type was []int, which isn’t comparable. OK, the shallowness of my understanding is showing here.

But, suppose there was a way to promise that a type was comparable. It looks like there is, at least in Go 1.18. Let’s try it.

type comparableFlexibleFish interface {
  getSpecies() string
  getDatum()   [8]float64

The compiler seems to be OK with that. So, one last step…

type comparableFlexibleFishTank container[comparableFlexibleFish]

Ouch! Goland: interface includes constraint elements, can only be used in type parameters. Compiler: interface is (or embeds) comparable. I understand them about equally well. [Narrator: He’s equally baffled.]

I guess I am skating pretty far outside the lines here? But I think the intent of that comparable in the type declaration is perfectly clear. This thing implements the listed methods and promises to be comparable. So if I try to claim something is a comparableFlexibleFish and it’s implemented the right methods but isn’t comparable, the compiler can stop me with a helpful error.

Once again, maybe I shouldn’t want to do these things, but I do.


The performance was great, no detectable slowdown. [Note this is not always true and if you care, you should really follow that link. But get a large cup of coffee first.]

And I threw away a whole bunch of duplicative code, which made me happy.

When generics arrived in Java, I hated them. I thought the ratio of pain removed to complexity added was way too low. I don’t hate Go’s generics, which is actually pretty strong testimony given that I spent the best part of a couple of days fighting all the stuff described above.

But, my guess is the Go generics story is isn’t finished yet.

Meet Quamina 12 May 2022, 9:00 pm

No parent will name a favorite among their children. But I do have one among my brainchildren, my software contributions over the decades: The event-streaming code I helped build at AWS. After rage-quitting I missed it so much that over the last few months, I wrote a library (in Go) called Quamina (GitHub) that does some of the same things. This is about that.

Quamina offers an API to a construct called a “Matcher”. You add one or a hundred or a million “Patterns” to a Matcher then feed “Events” (data objects with possibly-nested fields and values) to it, and it will return you an array (possibly empty) of the Patterns each Event matches. Both Patterns and Events are represented by JSON objects (but it should be easy to support other Event encodings).

Quamina (and here I beg pardon for a bit of chest-pounding) is really freaking fast. But what’s more interesting is that its speed doesn’t depend much on the number of Patterns that have been added. (Not strictly speaking O(1), but pretty close.) Concretely: On my 2019 Intel MacBook Pro, it can pattern-match several hundred thousand Events per second without much regard for how many Patterns are loaded into the Matcher. The benchmark I use most for optimization has maliciously-constructed rules that force the Matcher to look at basically all the fields in 1KB-ish Events, where the values are arrays of long floating-point numbers. I try to keep this worst-case running at 150K/second or better.


Back in 2014-15 I wrote v1.0 of a conceptually similar library operating inside AWS, which now implements pattern-matching capabilities for EventBridge and quite a few other teams around Amazon, filtering millions and millions of events per second so they can be accepted, rejected, or routed. Lots of super-talented collaborators emerged over the years and I bet that these days, my code isn’t even a majority of the thing.

I’ve been lobbying AWS to open-source it and that conversation continues, it might happen. But I missed working on it so much that I built Quamina. Also, the AWS version isn’t in Go and I thought a Go library might be nice.

I plan to write a few pieces about Quamina. Whether or not its features interest you, it illustrates lessons in dealing with data in motion at a scale that not very many people are lucky enough to be given a chance to code for.


Quamina is based on finite state machines a.k.a. finite automata. There is a rich body of academic investigation into the discrete-math theory of automata, almost none of which I’ve studied. But once I got the basic idea, state machines have been the first tool I’ve reached for across the decades when faced with hard programming problems. I’m not going to introduce the concept here because anyone who’s either got a CS degree or written much software knows the basics.

I like using automata because they have very, very predictable performance. If you’re processing a row of data items you only have to look at each item once, which is a pretty strong constraint on the time you’re going to spend. Also, the typical unit of work in stepping through an automaton is small. Thus, fast and predictable. Assuming I write more on Quamina, there’ll be lots of digging into details of how to make automata useful.

Quamina by example

This is going to be easier if we have an example to work with. Here’s a sample Event:

  "Image": {
    "Width":  800,
    "Height": 600,
    "Title":  "View from 15th Floor",
    "Thumbnail": {
      "Url":    "",
      "Height": 125,
      "Width":  100
    "Animated" : false,
    "IDs": [116, 943, 234, 38793]

(This is actually the canonical example of a JSON object from RFC8259.)

Here are two Patterns that would match the Event:

{"Image": {"Width": [800]}}
  "Image": {
    "Animated": [ false ],
    "Thumbnail": {
      "Height": [ 125 ]
    "IDs": [ 943, 811 ]

For more examples and explanation, see the README.

This doesn’t look much like a typical query or pattern-matching language; the Pattern echoes the “shape” of the Event you’re trying to match. This turns out to have pleased a few developers, one or two of whom have said nice things to me.

Unfortunately I have no recollection of the thought processes that got us there in the white heat of the early design sprints for what became EventBridge. I remember a meeting in Seattle where I got up and sketched a few of these on the whiteboard and people said “It’s not SQL should it be?” and then eventually “Yeah, looks OK”, then we moved on. There must have been some discussion before that?

The trick

The idea of compiling together a bunch of automata and using them to match data is not exactly new; I think the “|” operator was introduced to grep while I was still in high school. The idea of using it on data that takes the form of JSON objects is a little less obvious. Here’s how it’s done:

  1. First, you flatten the Event fields into name/value pairs. Here are a few of the fields in flattened form:

    Image.Width: 800
    Image.Thumbnail.Width: 100
    Image.IDs: [116, 943]
  2. Then you sort them by pathname. This is perfectly OK, because the order of fields in a JSON object doesn’t matter.

    Image.IDs: [116, 943]
    Image.Thumbnail.Width: 100
    Image.Width: 800
  3. Then you do the same trick with the Pattern. For example, let’s use the second one above:

    Image.Animated: false
    Image.IDs: 943
    Image.Thumbnail.Height: 125
  4. Then you make a state machine based on the flattened sorted Pattern and run it against the flattened sorted Event, and away you go.

What next?

Quamina has enough features to be useful and the unit test coverage isn’t terrible. I’d like people to look at it and see if solves any problems they have, or alternately if they see a reason why it’s All Wrong. Bear in mind that there’ll probably be another (more feature-rich) version in another popular programming language coming from AWS later this summer.

It isn’t at 1.0 yet and has no CI/CD, which means I reserve the right to change the APIs. In particular, should the MatchesForJsonEvent() be made a Go generic? (Quamina already has generics internally). I couldn’t stop anyone putting it into production, but I probably wouldn’t if I were you. But if that idea even crosses your mind, please get in touch.

Long Links 6 May 2022, 9:00 pm

Once again, a selection of long-form pieces for those who (unlike me) have jobs and won’t have time for all of, but might benefit from ingesting one or two. This month we have lots of musical offerings, along with politics, advice on how to walk, and 5G.

Let’s start weird. I first ran across the writing of John Michael Greer when his site was called The Archdruid Report because Greer was in fact an Archdruid, which is to say the leader of a niche sect with “Druid” in its name. He’s irritating as hell because his writing is bathed in an atmosphere of “I’m so smart, I can see all sorts of things nobody else can and I’m doing you the favor of sharing my wisdom.” But I still occasionally read him because once you get past that, he does say lots of interesting things. In Whispers of the Fall, he takes up the argument that we are members of a falling civilization. Grim stuff, but good reading.

Hania Rani

These L-L aggregations are always going to have some geeky software stuff. But before I scare away the tech civilians, let’s take our first dip into music, with the amazingly lovely work of Hania Rani, a Polish composer, keyboardist, and singer. I’d put her somewhere on a musical spectrum with Phil Glass and Enya at the ends. A good place to start is Live from Studio S2, but type her name into YouTube and you’ll see lots of great stuff. Several of her performances are collaborations with cellist Dobrawa Czocher, and they’re exquisite.

I’m a little nervous about saying this, but: I just can’t take my eyes off the screen when Ms Rani is on it. She is neither glamorous nor conventionally pretty, but (particularly in that Studio S2 video) she somehow, when performing, radiates effortless beauty so extreme that (for me at least) it lifts up the music.

Now let’s read about fast software. Nelson Elhage’s Reflections on software performance takes up the vexed question about how and when we should think about making software performant. He acknowledges the following three guidelines, which I generally believe in and have intoned at younger programmers more than once:

  1. Premature optimization is the root of all evil.

  2. Make it work, then make it right, then make it fast.

  3. CPU time is always cheaper than an engineer’s time.

And yet, and yet… as Elhage explains eloquently, performance is important, and sometimes you just can’t add it in later. This strikes close to my heart because during my years at AWS my primary coding project was an event-filtering library that was obsessively concerned with performance. I have hope that AWS will open-source it, but in recent months I’ve put in a lot of hours rebuilding it in another programming language, and with this thing, performance is sort of the point. So I can see both sides of this one.

While we’re doing tech, let’s have a look at 5G, about which I recently wrote a highly skeptical blog piece. I remain unconvinced that the huge capital outlay 5G requires will buy much that’s useful. Anyhow, AWS has a product called “Wavelength” that is supposed to deliver 5G’s benefits by locating compute out in the cellular-network relays. AWS just announced New AWS Wavelength Zone in Toronto – The First in Canada. That link is to a blog about putting 5G to work, which I think is valuable because it really leans into the practicalities.

What’s the application that’s going to benefit from 5G wonderfulness? A little knee-high rolling robot used to make small food deliveries, for example a single cup of coffee, within a 2km radius. The benefit is a factor of ten cost reduction, which is nice I guess. But, uh…

Now let’s veer gently away from technology into politics by way of Josh Marshall’s On The Elon Musk Razzmatazz. I’ve been viscerally unhappy about the Musk Twitter takeover, but unable to sort out why. Mr Marshall did. Here’s a quote:

That’s what this is and where we are: an extremely powerful and wealthy jackass on an ego trip. You can take the bro out of the frat house but you can’t take the frat house out of the bro. In fact when you’re worth hundreds of billions of dollars (for now…) you don’t even have to leave the frat house. You can bring it with you. This is a guy whose ideas about speech and also the construction of syllogisms apparently culminated two joints in at a dorm room bull session in sophomore year…

It’s mostly not quite that sharp-edged; in fact, a carefully-considered piece that anyone who cares about Twitter, or about the quality of online discourse, should read.

EROS, by Boris and Endon

Back to music, namely a brand-new release on Bandcamp: BORIS + ENDON / EROS. Boris is a Japanese experimental-metal band that I’ve blogged about twice in Photos of Wata of Boris and Live Metal Is Better. This is a collaboration with Japanese even-more-experimental metallistas Endon, whom I once saw live opening for Boris and sort of dissed in that second blog piece. But here we have 28 minutes or so of thoughtful and well-executed metal which is intermittently quite beautiful. If you like it, you can buy it off Bandcamp and a noticeable proportion of that money goes straight to the artists, so go ahead and do that.

Back to civic policy now, and an issue that’s white-hot where I live and I think in most big interesting cities: Is it OK to build big fancy high-rises in the middle of un-fancy neighborhoods? Density is good, right, but gentrification is, um, mostly bad I guess? Anyhow, the NYTimes has numbers to report: A Luxury Apartment Rises in a Poor Neighborhood. What Happens Next? The numbers aren’t that satisfying, but the take-away is that it’s not obvious that the high-rise does any particular damage. Which isn’t going to make this issue any less vexed, but if you read it you’ll at least have seen some data being thought about.

NYT on guitar soloing

While we’re visiting the Times, let’s veer back into music via a piece that’s pretty well pure fun: Why We Can’t Quit the Guitar Solo. It’s a triumph of fancy Web presentation technology, with cool guitar riffs bursting out at you as you scroll, many of which you will have heard. Only a fool would want a musical life entirely without shredding, so the target here is pretty soft, but the author and Web geeks here have great fun skewering it, and you’ll probably have fun scrolling through it.

Now let’s step sideways by way of Interview: Ramez Naam, futurist, author, and investor. I loathe the term “futurist” so, despite this having been recommended by a Smart Person, went in hostile. But I was won over. I certainly didn’t agree with everything Mr Naam said, but I was never bored, and felt like I’d learned one or two useful things. Perhaps I was drawn in by his arguments for something I profoundly believe: Assuming we don’t all die in the climate catastrophe, there is going to be a freaking huge amount of money made building out the low-carbon energy services as we transfer from now to the future. So if you have some money, finding a way to invest in this project might help save a couple billion lives and also pay back pretty well.

In the last Long Links I enthused about the writing of Chris Arnade. He goes for long walks in interesting parts of the world, and then in parts that most people wouldn’t think interesting, and takes pictures, and writes them up, and it’s never not interesting. Recently he published How to Walk (12 miles a day). Walking for extended distances is, I believe, one of the core competences of Homo sapiens and if you don’t believe me go read Bruce Chatwin’s The Songlines then come back to argue the point; but you won’t. We are almost all city livers now, and thus it’s hard to overvalue good practical advice on how to excel at this essential human behavior in the context that we actually inhabit.

There is, however, a point that is troublesome: Arnade points out that in many long-walk contexts, you just can’t beat the comfort and practicality of socks-under-sandals (good socks and good sandals, of course). But I have been advised more than once by the members of the demographic that I would like to have sex with that I can forget it if I ever appear in that combination. So I don’t, which doesn’t mean that he’s wrong.

Long Time Coming Sierra Ferrell

And now for our musical grand finale: Sierra Ferrell. I listen to her music more or less every day and caught her latest tour during a Covid downcurve. I have rarely been so sure about anything as I am in predicting: Gonna be a big star.

What happened was, she ran away from a broken West-Virginia home to be a hobo — no really, hopping trains and living in camps. Then a few people noticed the talent, then she got onto the tiny-venue acoustic circuit, then a few of her YouTubes started getting millions of views, then the tour I attended (still in progress as I write) is selling out most of its dates.

So, why? Probably most important, the songs are wonderful. Pretty well a pure country aesthetic, but with melodies that you’ll wake up humming and lyrics that go to surprising but truthful places. Then, the voice; she is the kind of singer that arrives only every other generation, if we’re lucky.

Then there’s that beauty thing that I mentioned reluctantly above. I don’t feel in the slightest reluctant to talk about this because she works so hard at it. She never appears twice in the same output, and every outfit is fabulous. As is her facial jewelry, and tattoo, and hairstyle. The night we saw her she wore this floor-length dress that was pure white and covered with reflective sequins so that she sparkled like diamonds, as if there weren’t enough other reasons to watch her. If you drop by her Wikipedia entry, the only picture just now is by me and it’s not very good, only faintly suggesting the visual effect of That Dress. I have no idea if she’s actually conventionally pretty (although I’m pretty sure, to quote Blondie, that her hair is beautiful) because onstage, she has that Star Power that you just can’t look away from.

Ms Ferrell deeply understands how to do online video, so the best way to check her out is to drop by YouTube and type “Sierra Ferrell” into the search bar and watch whatever comes up. One hint: The stuff she’s been posting recently is quite a bit better than those millions-of-views videos from a few years ago.

No, wait, what did I just say? The best way to check her out is to see if the tour’s coming anywhere near you and if it is, drop everything, postpone grandma’s funeral if necessary, and go see the show. You’ll thank me.

4,000,000m Lessons 18 Apr 2022, 9:00 pm

The odometer on my e-bike clicked over to 4K and, rather than a general-purpose “e-bikes are great” rave, I thought I’d assemble a few concrete arguments for them, suitable for re-use with friends and loved ones in the (likely) case that you’re already convinced. With pictures.

4001km on the e-bike odometer

… is prime.

It’s good for you

The skeptics are prone to say that if you want fitness, you should ride a real bike, dammit. They are wrong. Let me end this argument right here and now with Health benefits of electrically-assisted cycling: a systematic review and Riding Electric Bicycles Boon To Health And Not Cheating, Confirms Literature Review and The effect of cycling on cognitive function and well-being in older adults and Metabolic and Cardiovascular Responses to a Simulated Commute on an E-Bike. Or if you want non-academic, there’s a nice NYTimes piece: E-Bikes Can Provide a Good Workout.

It’s important to be clear that by “e-bike” I mean the ones with electric assist; i.e. no throttle, you have to pedal to go, and pedal harder to go faster. Also, you have to keep the boost set at a moderate level. The Bosch electric system that appears on a whole lot of popular e-bikes has four levels: Eco, Tour, Sport, and Turbo. I keep it on Eco and switch to Tour for challenging hills. I’ve never really used the top two levels, nor felt the need.

It’s fast

I find that my average e-bike cruise is something like 25km/h, and I get around town astonishingly fast (note also the numbers in that NYTimes piece above). It helps a lot that Vancouver has a pretty decent bike-route network. And obviously, so does no looking for parking.

Less obviously, when your bike path runs along the side of a road with cars on it that has stop lights and intersections, you realize how agonizingly slow these things are. The endless halts behind someone waiting to turn left or for pedestrians to cross, and behind people inching along looking for parking, you don’t even notice them when you’re driving, but a lot of it you can just breeze through on a bike-path while staying perfectly safe.

For navigating across the city anywhere less than 5km or so, the bike is not going to get me there significantly later than my car. And, I’ll be able to park right in front of where I’m going. And, the parking is free.

It’s never boring

Closely related to the previous point, sitting in traffic look at another car’s ass end, or even worse, stuck behind a truck so you can’t even see ahead, is boring. On a bike, you’re always moving, you’re way more intimate with the scenery, and if you come to a nice view you can stop on impulse and take a picture.

Nice view along a Vancouver e-bike commute

It’s cheap

The bikes themselves are aren’t cheap. For the purposes of this piece, I poked around the landscape to pick out something that’s mid-range, well-reviewed, and from a manufacturer I have respect for: the Trek Verve+ 3. US$3,300 isn’t cheap for a bike but it’s insanely cheaper than anything with four wheels that you’d want to drive, and the running costs are really too low to be worth measuring. I’ve had my bike for three years and a bit and the service charges, including repairs after a pretty bad accident (see below) and a couple upgrades, add up to less than $1,000.

Trek Verve+ 3

It’s good for the city

Look, if you don’t see that a city with more bikes and less four-wheelers is a better city to live and work, there’s nothing I can say that’ll help you. But I will say this: Nobody wants to live in a house on a main road, but a house on one of our city’s main cycle paths would gain value.

It’s good for the planet

This is hardly in doubt, but I stumbled across a good quantitative write-up on the subject, from Britain: How green is cycling? Riding, walking, ebikes and driving ranked. I’m going to reproduce four of their summary bullet points, which widened my eyes, and encourage you to go read the whole piece.

  • Cycling has a carbon footprint of about 21g of CO2 per kilometre. That’s less than walking or getting the bus and less than a tenth the emissions of driving

  • About three-quarters of cycling’s greenhouse gas emissions occur when producing the extra food required to “fuel” cycling, while the rest comes from manufacturing the bicycle

  • Electric bikes have an even lower carbon footprint than conventional bikes because fewer calories are burned per kilometre, despite the emissions from battery manufacturing and electricity use

  • If cycling’s popularity in Britain increased six-fold (equivalent to returning to 1940s levels) and all this pedalling replaced driving, this could make a net reduction of 7.7-million tons of CO2 annually, equivalent to 6% of the UK’s transport emissions

Tell me a couple of those didn’t surprise you.

The infrastructure is human-scale

Nobody loves taking their car in to get fixed. The world of bikes is less monopolized, less exploitive, and generally nicer. By way of evidence I offer this:

Velostar coffee and bikes in Vancouver

At Velo Star, the coffee is good and the baristas are cute. And they replaced my chain for $70 Canadian, tax and labor in.

It’s practical

Once you have an e-bike, you really, really need to spend a little more and get a good pannier. They’ll try to sell you a pair, which I have but only use one. Because I have no trouble getting my laptop, my raingear, and the groceries I picked up on the way home in mine, which doesn’t even look that big.

Did I mention raingear? A decent raincoat and rainpants make all the difference in the world.

Now, let’s be honest. When you’re going to drive away, you just get in the car, fasten the seatbelt, and you’re off. On a bike, you have to get your helmet (and maybe raingear) on, unlock it, and stow your pannier before you’re in motion. When you’re just getting started biking, this feels onerous. But as with most things in life, practice makes perfect. A couple of months in and your muscle memory will have all this stuff — especially the lock/unlock dance — filed away and it’ll happen quickly without even thinking about it.

What about safety?

Yeah, it’s an issue, as you can see below. And I think e-bikes are possibly more dangerous than regular ones. That’s what the numbers say (at least they’re safer than e-scooters).

The author after a bike accident

It’s pretty simple: There are a lot of situations where an e-bike makes it easy to go way faster than is safe. So… don’t go that fast! For example, I got my face decorated, as you see above, when I cut the corner on a T-section in the bike-path, banking through a patch of muddy ground, and BANG! I was on my side on the ground, bleeding.

Before I go on, I want to point out that my helmet saved my life on that occasion. I recommend Bontrager helmets, another product of Trek. And let me confess: When I’m just going five blocks to pick up parmesan and red wine because we’re having pasta, I often don’t wear one. And then I don’t go fast at all.

But yeah, in general, slow down. In particular, I want to point out that bicycle brakes really, really suck on wet pavement, so if you’re going downhill on a rainy road, dial it way the hell back.

My take-away: If you’re careful and if you have a decent bike-route system, the benefits of e-biking overbalance any risk, particular if you don’t bomb along at maximum speed. Now, I do bomb along at maximum speed in places where it’s safe, and damn that’s fun.

E-biking is fun. And good for you. And your city. And your planet. Give it a try.

Three Perfect Tools 13 Apr 2022, 9:00 pm

We live in a world of things that have Product Managers and Design Teams and Documentation Architecture and often vast ambition. And yet, so many of them are so bad. And worse, some which are good then become bad for no good reason. There is a particular joy in a product that just does what you need done, in about the way you expect or (thrillingly) better, and isn’t hard to figure out, and doesn’t change unnecessarily. Here are three to learn from.


At our cabin, we have reasonably modern electric heat that keeps things comfy, but for our occasional winter visits we also have a nice efficient wood-stove. Because we are surrounded by 100+-year-old forest, some of which falls down (or, rarely, is cut if it threatens to fall in the wrong place), there’s plenty of wood to burn. But it comes in big pieces which have to be sawed down to short sections then split into pieces small enough to fit into the stove.

Normally, splitting wood is a joyous form of light exercise — hit the end of the fragment with a decent axe, put a little oomph in it, and the wood springs apart. Very satisfying. But certain pieces of wood are some combination of hard and knotty and twisted and big. Even a reasonably skilled wood-splitter can be driven to distraction.

Spiral splitting wedge

For these, what you need is this Spiral Splitting Wedge, which I bought from Lee Valley. You position it judiciously against the end of the fragment, and pound away with a hammer; I use a nice little one-hand 4lb sledge, which seems about right. You don’t have to hit it that hard, just enough for the blade to bite, then the combination of the wedge’s spread and its spiral twist inexorably rips apart the awfulest, gnarliest firewood. Not quite as satisfying as an axe on a clean straight piece, but then few things are. Just Works. Is cheap.

Then you curl up on the sofa facing the stove, feeling virtuous while the rain lashes the windows and metal roof, and watch the kids toast marshmallows.

I’m pretty sure the design of this product hasn’t changed in my lifetime. And I live without fear that it will change, because the one we have will definitely be working long after I’m gone.


Our cabin barbecue died as a consequence of being left out in the rain too much during the current renovation. It was at least a decade old, and we replaced it with the current equivalent model, a Weber Q 3200 (link is to the Canadian site) because it’s compact, easy to roll around, and can feed 8 people with a little cooking forethought and ingenuity.

Weber Q 3200 barbecue

Compared to the model it’d just replaced, the changes were subtle and thoughtful: A slightly slicker tank retainer, a better spark mechanism, and an elegant little light placed just right for those times you’re cooking after dark.

Why am I so excited about this decent but pedestrian product? Partly because it hasn’t changed; the design is pretty well correct and no hyperoverentitled Product Manager has been allowed to introduce any Bold New Paradigms.

But the great thing, the truly great thing, is the assembly directions. There are a quite a few pieces, but they all fit together in only one way, which is the right way, and they snap into place, and the directions illustrate each step in a way that no-one, no matter how clueless, can misinterpret. You go from flat-pack to working BBQ in an astonishingly short time with not a single good reason to swear.

This appeals to a lot of my core values.

And should you get an invitation to our cabin, I’ll demonstrate the machine’s excellence with a hearty serving of Eastern-Mediterranean chicken kebabs (Arabic: شيش طاووق; Hebrew: שישליק עוף; Turkish: tavuk şiş). They have reduced strong men to tears of joy.

Music writer

My music teacher sent me a Rumba score we were going to work on, which meant picking up the MuseScore app. I hadn’t used it in years, but as soon as I opened it I smiled, because it’s truly great software. Now, you have to know how to read music and the names of the glyphs and markers that appear in it. But if you do, you can teach yourself in almost no time to produce a pretty damn decent-looking score and do so quickly.

I don’t remember ever opening a tutorial or reading a HOWTO; the first time I wanted to capture some music, I just opened it up, dived in, and figured it out.

When was that first time? Last century when I was studying cello, my teacher, a couple times a year, held friends-and-family recitals that filled most of a medium-sized church; they were fun!

One time I performed a duet with my wife Lauren, who has a light, soaring soprano; an arrangement I made of Simon and Garfunkel’s Scarborough Fair/Canticle. For the first part I just played chordal pizzicato background to her voice. For the middle part, with the two voices, the cello took the second, with lots of legato. I think the voice/cello trade-off sounded intense and lovely, and we got a nice round of applause. Here’s a sample of the main and middle sections.

Scarborough Fair/Canticle for voice/cello; main part Scarborough Fair/Canticle for voice/cello; middle part

In case anyone out there plays the cello (or any other alto-to-bass register instrument) and wants to take a run at this, here’s the music. It doesn’t include full lyrics or the vocal line but that’s OK because I bet most singers already know it and will be comfy in E minor.

At the event, I used a hand-written score, where by “written” I mean “scribbled”; my handwriting is terrible and that includes music. A couple of fellow students asked for copies so I downloaded MuseScore and within a couple of hours, I had it. When I fooled with that Rumba just recently, I found it similarly effortless.

But wait, there’s more!

The MuseScore app is open-source, free, standards-compliant, well-documented and seems to have a lively community around it. I suspect this is because whenever you’re in a roomful of serious software practitioners, the proportion that are competent amateur musicians is going to be way higher than in the general population. So the number of people who combine programming talent and music literacy isn’t small.

It’s 2022 and the world is terrible but still has good things in it. We need more like these, please.

Page processed in 2.556 seconds.

Powered by SimplePie 1.3.1, Build 20210120101038. Run the SimplePie Compatibility Test. SimplePie is © 2004–2022, Ryan Parman and Geoffrey Sneddon, and licensed under the BSD License.