Exactly Once Delivery

At Remind I helped build the announcement and chat message delivery platform. With thousands of messages streaming through a system that bridged Heroku and our AWS VPC, network partition and other failures were real occurrences, which led to some creative work ensuring messages were only delivered once to each recipient. So I was interested to see two articles posted within a day of each other talking about “exactly once delivery”.

TL;DR: It’s really hard, a lot of the systems wind up looking alike (ie, two-phase ack), and the one we built probably wasn’t as bullet proof as we thought. The interesting question wasn’t “Are we doubling up?” or “Are we dropping things?”, but “How will we know when either happens?”

Most Popular Erroneous Post

There are a few posts on my site that get the lion’s share of traffic. One of those is a one-off project I did to convert Maildir to Mbox. Today someone left a comment, saying that it didn’t work. And then I realized that somewhere along the line sys.argv[-1] had been converted to sys.argv[1]. Not really the same thing.

I’ve update the page, but it’s a little embarrassing to only realize 18 months (or more) after originally posting it that you’ve been giving out bum advice.

Eventually, I’ll be right.

date:2010-02-03 21:41:14
wordpress_id:1454
layout:post
slug:most-popular-erroneous-post
comments:
category:aside
tags:analytics, mistake

We called it “magic”

Just under ten years ago I started working at Canterbury School doing a variety of things. One thing I wound up doing was building the new Intro to Computer curriculum, based on Python. When Vern and I presented our approach at PyCon in 2003, we were asked what advantages we thought Python had over its predecessor in the curriculum, Java. The first answer was always, “Magic; a lack thereof.” There was less boilerplate, fewer incantations, a much shorter list of things you have to wave your hands about and say, “Don’t worry, we’ll talk about this later in the semester. For right now, it’s magic, just do it.” Magic distracts students, and makes them wonder what you’re hiding.

Seeing a comparison between Java and Clojure (albeit one you can read as more about succinctness than clarity), I was reminded that this lack of magic — boilerplate, ceremony, whatever — is still important.

date:2010-01-06 19:04:13
wordpress_id:1317
layout:post
slug:we-called-it-magic
comments:
category:aside, development
tags:magic, python

Passing the buck

Let’s say, just hypothetically, of course, that one day your laptop, which is less than six months old, suddenly has multiple failures: the backlight, optical drive and touch pad all die. So you call the unnamed manufacturer [1] and arrange for service. They assure you that it will be repaired in 3-5 business days and overnight you a package.

Of course, when you look at the status on the web, the estimated return date is actually something like 7 business days, but no matter. And then the estimated return date comes and goes, and no laptop appears. So you call “Max” in lovely Bangalore, and he assures you that the backlight is being repaired and it will ship in 2-3 days, “maximum, sir[2]_“.

Four days later you call. Two days after that. And finally you’re told that they’re waiting on a backlight, and that no, they don’t have one in, and that it won’t be in until the end of the month. This, of course, stretches your repair window from “3-5 days” to 30-40 days. And of course, they understand how frustrating it is, but really, what are they supposed to do? But miracle of miracles, you receive an email two weeks early saying “your laptop has shipped.”

Of course, in HP Mind-Fuck-Your-Customer Land, “shipped” doesn’t mean “shipped” exactly. It means “we told FedEx to pick it up but they didn’t. So it’ll be there in a few days. Hopefully.” This is essentially what I was told tonite when I called again. Let’s think about the logic:

  • First, we told FedEx to pick it up.
  • But they didn’t.
  • So we told them again.
  • And we assume they’ll pick it up now.

And when I had the temerity to suggest to my helpful customer service rep that FedEx not picking up something was actually HP’s problem[3]_, he said “No, sir, we told FedEx to pick it up. You should call them to complain.” Seriously?

Hey Carly, wasn’t that merger supposed to make the company better at serving customers?

[1]What the hell, it’s HP.
[2]I should note that this is why I don’t worry about my job when I hear tales of outsourcing; I’ve yet to have a customer support interaction with one of the many “Max”, “Bob”, “Sam”, or “Andy“s that seem to populate the Indian sub-continent that actually left me feeling like that company cared to keep my business. That’s no way to treat a customer, let alone run a business.
[3]Since, well, I’m not FedEx’s customer, I’m HP’s, and seeing as this is my second HP notebook, I’m the most valuable kind: repeat. Well, I was the most valuable kind.
date:2007-02-08 22:52:54
wordpress_id:474
layout:post
slug:passing-the-buck
comments:
category:aside

Get Out!

Today is National Coming Out Day. If you’re in the closet, I hope you’re trying on your mother’s shoes and lipstick. Since I’m already out (and have been for nearly 10 years now), I’ll celebrate by mentioning how amusing I find it that Wonkette is referring to the Foley-scandal-du-jour as the Cocktober Suprise Tee-hee.

date:2006-10-11 16:59:30
wordpress_id:452
layout:post
slug:get-out
comments:
category:aside, my life

Web 2.0…

Struggle as you may to define Web 2.0, the folks over at Tagly have figured it out.

Tagly Screenshot

Tagly Screenshot

It’s all about ripping off Flickr‘s visual identity.

date:2006-09-24 18:09:59
wordpress_id:444
layout:post
slug:web-20
comments:
category:aside, General