10 stories
·
0 followers

JMAP is on the home straight

1 Share

In 2014 we wrote about JMAP, the open protocol for modern email which had grown out of the web interface we’ve had since 2011. We believe in open standards, both as a contribution back to the community from what we’ve learned, and as a bulwark against the growth of walled gardens.

JMAP in 2014 was a first draft of a protocol and an idea. JMAP in 2018 is almost finished the standard process, and has solid real-world use.

Let’s have a look at what’s happened over the past 4 years.

Building community engagement

We started the JMAP process in 2014, by talking to everyone we could find — at conferences, online, even visiting companies directly and chatting to our counterparts in their email teams! With these discussions we validated that there was interest in a modern, open standard for email which could give a user experience as good as or better than proprietary alternatives. One year after announcing, we wrote another blog post detailing some of the people building on our first JMAP proposal.

Key features of JMAP were significantly better than the protocols available at the time: push updates for immediate notification of changes, batched commands to reduce latency, and a single authentication and endpoint for related data types to eliminate the support headache of partial authentication failures.

The use of JSON and HTTP as the basis of JMAP was always a key point — it means that people wanting to build something on top of email don’t have to re-implement complex parsers or find a software library in order to get started.

We found plenty of interest, but staff at many of the bigger players said “this sounds great, but we won’t be allowed to build it into our products unless it’s a standard”. Fair enough. We were happy to self-publish it as a standard over at https://jmap.io/, but there was clear demand that it be standardised properly.

JMAP at the IETF

The IETF (Internet Engineering Task Force) is the premier standards body for the internet. We first proposed JMAP to the IETF in late 2016, and a new working group was founded to work on it in early 2017. I wrote last year about our work there so far and the chartering process. In the first year, major changes were made to JMAP. Authentication was removed entirely from the JMAP spec proper, because it’s more general than just one protocol.

In general, the IETF likes separation of concerns. In the same way that an office building might have individual tenancies fitted out frequently, entire floors redone on a less frequent basis, and yet the skeleton of the building outlast all those changes, IETF protocols are built with infrastructure at different levels expecting to have different lifetimes. It is likely that authentication protocols will evolve, and even possibly that transport protocols will change, and the core object structure of JMAP will remain relevant. So it made sense to remove authentication.

Other major areas of JMAP have been completely rewritten. The original structure of setMessages, getCalendarEvents etc gave way to a Noun/verb syntax. The fact that “Message” meant too many things led to it being renamed as “Email”, so we now have Email/set and CalendarEvent/get. Originally, email was sent by adding the mailbox with the special role “Outbox”. We now have an EmailSubmission object which can provide an interface to things like delayed send, undo send, and message delivery status if the underlying system supports them.

The entire Email object was redesigned to allow a middle ground between the really basic client that doesn’t care about email structure at all, and a full service client which needs to understand the entire MIME structure. By adding the middle ground, it’s possible to handle structured email sensibly, use bandwidth efficiently, and barely ever have to care about the raw email.

Finally, there was a separation into a core protocol document and an email specific document. This also allowed us to remove a ton of duplication throughout the specifications, and will make future JMAP extensions easier to write. We expect the next extension to add calendaring, and soon after that a contacts/addressbook module. The delay on addressbook is due to there not yet being a fully specified standard JSON data object for contact records.

The IETF process has definitely made JMAP better — more regular, more general, and more easily implemented in more than just our environment. We appreciate all the effort that members of the working group have put into improving the protocol.

JMAP Core and JMAP Mail are currently at Last Call within the IETF, the working group effort having been finished. We expect them to be published early in 2019.

JMAP at CalConnect

CalConnect is an industry group involved in all things calendaring and contacts. While JMAP itself isn’t being worked on directly at CalConnect, a JSON data model for calendars was already underway. We joined that effort, and then brought the work to the IETF, where it is also close to standardisation in the calendaring extensions working group.

CalConnect is also concerned with the next generation of addressbook formats, and we expect the work on the contacts data model to continue in both venues.

We actually started at CalConnect before going to IETF — we’ve been a member of CalConnect since 2014, and we were aware of them for years before that due to Ken from the Cyrus development team being involved via Carnegie Mellon University.

JMAP at FastMail

It’s all very well to talk about JMAP in the abstract, but we were also keen to make sure it worked in the real world. We employ developers who work on the Cyrus IMAP open-source email server, which we run ourselves. The Cyrus team, particularly Robert, have fully implemented JMAP as an open source server. Our Perl developers also built Ix, a Perl JMAP server framework done properly, and I slapped together the JMAP Proxy (which is now kinda out-of-date — I hope to finish updating it early in the new year).

Finally, we’ve been working hard on a test suite for JMAP, to help others implement their servers.

But enough about the software, how about the experience! When we created our brand new Topicbox product, we built directly on top of JMAP for the email. We also used JMAP-inspired APIs for the rest of the product experience, so Topicbox’s early users have been on JMAP for over a year now.

For the FastMail product, we started rolling out a limited beta to our users a few months ago. We had hoped to have it deployed to all users, but there are a couple of final bugs that aren’t ironed out — in particular the non-conversation mode has issues with the fact that an email in JMAP can exist in multiple folders at the same time (which can be used to support a label mode), so the counts are wrong in some edge cases. Still, 30% of our customers have been converted to JMAP already. All new customers are on JMAP automatically, and our new mobile applications which are fully based on JMAP will come out early in the new year.

The future of JMAP

We expect the core and mail specifications to be standards very soon. The JMAP working group at IETF plan to extend our charter to include contacts and calendars, and perhaps notes and files as well. Inside FastMail, we’ll keep building both our internal and external protocols in the JMAP way, and publishing specifications once we’re happy with them so we can be more of a platform.

If you’re interested in being involved in JMAP, the IETF mailing list is currently the best place to discuss the protocol. Once the specs are in the field, we will find a better place for a user community to build around the protocol. We expect existing users of earlier JMAP drafts will update their implementations once the standard is published, and the large email providers will see the benefit of implementing JMAP gateways to their mail stores.

We will continue to improve the test suite and evangelise JMAP as the best email protocol out there!

Read the whole story
hananc
2146 days ago
reply
Share this story
Delete

Quality Speakings

1 Share

Unfortunately my suite of annoying verbal tics – um right um right um, which I continue to treat like Victor Borge’s phonetic punctuation – are on full display here, but I guess we’ll have to live with that. Here’s a talk I gave at the GTA Linux User Group on “The State Of Mozilla”, split into the main talk and the Q&A sections. I could probably have cut a quarter of that talk out by just managing those twitches better, but I guess that’s a project for 2019. In the meantime:





The preview on that second one is certainly unflattering. It ends on a note I’m pretty proud of, though, around the 35 minute mark.

I should go back make a note of all the “ums” and “rights” in this video and graph them out. I bet it’s some sort of morse-coded left-brain cry for help.

Read the whole story
hananc
2219 days ago
reply
Share this story
Delete

On Cash

1 Share

What happened was, I was walking through a train station on the way to work with the buds playing randomly on shuffle and a knifeblade of Real Music came in from outside so I had to pull them out of my ears, there was this young dude getting great electric sound out of a teeny amp doing a slow take on Little Wing with a really good voice: When I’m sad, she comes to me… plenty room between the notes and lots of soul in them. He looked a little hard-pressed; I put my hand in my pocket for some coin but there was no coin there because I live on plastic these days and that busker went unpaid that morning. Another reason for a thing I’m thinking of: Going back to cash.

Used to be I had to hit the bank most weeks to arrange brass-in-pocket, but no longer. [Dear reader: The following is for Americans, who are a little behind on this stuff.] For a while now, almost anywhere I went and wanted to buy whatever for less than $100, I just tapped the paymachine with my credit card. No PIN no waiting no fuss no muss. And more recently, because who wants to dig around in their pocket for their credit card like a filthy savage, I can just hold my phone up to the machine because of course I’m already holding it like you’re holding yours mostly, and Something Something Pay makes this ecstatic little Internet Chiming Sound to let me know that I can take my latte or jeans or flowers or beers and say “Thanks, have a good one” and walk away, that relationship is over.

So, why should I bother dealing with cash?

  1. Routing everything through my credit card is feeding the global payments cartel, who extract a tariff measured in freaking dollars for almost every routine transaction. I’ve seen so many great business plans go begging because they could have done something wonderful in the world if you could only do micropayments, which you can’t, did I mention that cartel that basically has it locked up and tied down?

  2. Obviously when you pay for everything with plastic (even if it’s your mobile pretending to be plastic) then The Man Knows Who You Are And What You’re Buying. Well, The Man also knows where anyone who carries a mobile phone around (which means everyone) is all the time, but let’s not let the perfect be the enemy of the good. I mean, right now I do a single Amazon search for Resonant Cavity Extractors and the freaking cavity extractors follow me around the Internet for weeks; I’m pretty sure the Big Vision guys in the payments cartel have this notion where after I fill That Prescription at the local drugstore, I’m gonna get pervasive popups for [Stop right there, Tim -Ed.]

  3. There’s no good way to lay a couple bucks on the guy playing Little Wing.

I haven’t got this worked out along Microeconomics-theory lines, but I’m starting to hear an internal voice saying “Cash is good.” So I think I’m going to start dropping by the ATM more often, and making a clinking sound when I walk. And laying a few bucks on buskers, because if nobody does that there won’t be any.

Read the whole story
hananc
2222 days ago
reply
Share this story
Delete

Social Media Announcement

6 Comments and 11 Shares
Why I'm Moving Most of My Social Activity to Slack, Then Creating a Second Slack to Avoid the People in the First One, Then Giving Up on Social Interaction Completely, Then Going Back to Texting
Read the whole story
hananc
2252 days ago
reply
I am moving to USENET
Share this story
Delete
5 public comments
satadru
2252 days ago
reply
More threading in newsblur comments please! :)
New York, NY
Covarr
2252 days ago
reply
Why I'm joining Twitter and setting up a bot that just links to my Facebook posts: A tech-unsavvy manifesto
East Helena, MT
synchromesh
2253 days ago
reply
"Mastadon"?
Norfolk, England
thelem
2253 days ago
http://lmgtfy.com/?s=d&q;=Mastadon
bootsofdoom
2253 days ago
cf. Mastodon
thelem
2253 days ago
Good point @deadlock989, DDG found it anyway so I didn't even notice the incorrect spelling. Knowing Randall I'm going to assume that's a deliberate mistake referencing something I don't know about.
nanciejk
2252 days ago
Anytime I'm not sure of something Randall's referenced, I check out http://www.explainxkcd.com/wiki/index.php/Main_Page
jth
2250 days ago
0/2 @thelem - Here's a link for the service: https://mastodon.social/ and the image I'm seeing has the typo corrected now (spelled correctly as Mastodon).
alt_text_at_your_service
2253 days ago
reply
Why I'm Moving Most of My Social Activity to Slack, Then Creating a Second Slack to Avoid the People in the First One, Then Giving Up on Social Interaction Completely, Then Going Back to Texting
fxer
2252 days ago
you'll always be my favorite for no reason at all, other alt-text bot
alt_text_bot
2253 days ago
reply
Why I'm Moving Most of My Social Activity to Slack, Then Creating a Second Slack to Avoid the People in the First One, Then Giving Up on Social Interaction Completely, Then Going Back to Texting
alexjurkiewicz
2253 days ago
i missed you, second alt text bot ❤️
firetech
2252 days ago
Technically, first alt text bot. It was just down a while, before alt_text_at_your_service popped up. ;)
Aissen
2252 days ago
@alexjurkiewicz it didn't go anywhere, it's been up non-stop for months now, but maybe you didn't see the posts ? Newsblur has a weird Pulic Comments selection algorithm.

Meteorite Identification

7 Comments and 11 Shares
Click for an actual flowchart for identifying a meteorite. My favorite part is how 'Did someone see it fall? -> Yes' points to 'NOT A METEORITE.' This is not a mistake.
Read the whole story
hananc
3003 days ago
reply
https://en.m.wiktionary.org/wiki/regmaglypt
Share this story
Delete
4 public comments
chrisrosa
3002 days ago
reply
how to identify a meteorite (be sure to check the mouse-over).
San Francisco, CA
manderay
3003 days ago
reply
I used to work in a lab in the Department of Earth and Planetary Sciences at Washington U, therefore I am (extremely) tangentially related to this xkcd and feeling very pleased with myself. (also I wonder if they've figured out why their site is crashing this morning...)
Chicago
thelem
3003 days ago
reply
The site it links to is down, but the flowchart is accessible at https://web.archive.org/web/20160110210954/http://meteorites.wustl.edu/check-list.htm
Brighton, UK
lahosken
3003 days ago
Thanks to this, I can see that "Did someone see it fall?" has no "No" option. This is obviously part of the conspiracy to hide the existence of aliens.
alt_text_bot
3004 days ago
reply
Click for an actual flowchart for identifying a meteorite. My favorite part is how 'Did someone see it fall? -> Yes' points to 'NOT A METEORITE.' This is not a mistake.

Now

13 Comments and 28 Shares
This image stays roughly in sync with the day (assuming the Earth continues spinning). Shortcut: xkcd.com/now
Read the whole story
hananc
3912 days ago
reply
DST?
Share this story
Delete
10 public comments
ddc0660
3906 days ago
reply
Pretty cool.
jth
3909 days ago
reply
I want a wall-sized real-life version of this where the time circle rotates and I can put a dot on my current location. Also, a puppy.
Saint Paul, MN, USA
ooter37
3907 days ago
I have a clock like that.
jepler
3911 days ago
reply
all these comments and no criticism of the projection used?
Earth, Sol system, Western spiral arm
llucax
3911 days ago
I also thought about that...
rclatterbuck
3911 days ago
Why would there be? The Azimuthal Equidistant Projection is perfect for displaying approximate timezones. I guess some people might have a problem with it being centered on the South Pole, but they are just showing a Northern Hemisphere bias.
llucax
3911 days ago
On the countrary, being centered in the South Pole is a Nothern Hemisphere bias. It shows Europe bigger than South America and as big as Africa... Intersting insight about projections in The West Wing: http://www.youtube.com/watch?v=vVX-PrBRtTY
rclatterbuck
3911 days ago
It also shows them ridiculously distorted in a manner unusual to those familiar with the typical cylindrical and pseudo-cylindrical projections. Size isn't everything. What is easier to pick out? Uruguay or Germany?
AlexHogan
3910 days ago
This is the TIME CUBE
Dadster
3909 days ago
Bottom-up is always good....
Michdevilish
3911 days ago
reply
Like
Canada
JayM
3912 days ago
reply
.
Atlanta, GA
wffurr
3912 days ago
Alt text: " This image stays roughly in sync with the day (assuming the Earth continues spinning). Shortcut: xkcd.com/now"
MourningDragon
3912 days ago
reply
There should be an award for great ideas like this!
yashimii
3912 days ago
:) Ausdrucken
MourningDragon
3911 days ago
Kann ich Deinen ePaper Drucker haben ;-) ...oder ich klebe es an den Stundenzeiger. Cool, jetzt bekomme ich einen Award!
bogorad
3912 days ago
reply
Да, мощно.
Barcelona, Catalonia, Spain
growler
3912 days ago
reply
brilliant
kazriko
3912 days ago
reply
This post appears right when I'm still at work after midnight. x_x
Colorado Plateau
miah
3912 days ago
reply
Why is there not an app for this that looks like this?
Denver, CO
Ludwig
3912 days ago
It's called a "globe."
Next Page of Stories