We are now, simply, Redis
O a frabjous day it was, the best and first ever Redis Developers Day.
A month or so ago antirez had tweeted about putting together this awesome event. I immediately let the Redis team know that we were going on a field trip, so they should get their passports in order. Then the waiting began – 28 days that seemed to go at the speed of a multi-join poorly-indexed SQL query.
Yiftach, Yossi, Yoav and I arrived at Heathrow the day before and made our way to the hotel (while minding the gaps and looking left) before heading off to the Redis London Meetup hosted at the GoCardless offices with the help of the Rackspace UK folks. With food and drinks aplenty, talk was fast and loose and people seemed to be enjoying themselves (I certainly was). Then we had a great talk by Mad Scientist Bill Anderson about Redis Sentinel and his latest brainchild, RedSkull (soon to be open sourced and pushed to GitHub). I could have danced all night but eventually it was back to the hotel for some shuteye before the big day.
The next morning, we arrived promptly at Pivotal’s London offices where the Redis Developers Day was held. I was busy making myself a fancy Cappuccino and trying to identify the attendees – there were people I’d met the night before, those I’ve been following and corresponding with online and some who were unfamiliar to me (BTW, no girls, what a shame) – when suddenly I felt a tap on my shoulder and a voice said: “Hi, I’m Salvatore.”
I turned around to meet the Creator of Redis, a remarkable piece of software that I’ve been living and breathing for the last couple years. The first thought I had was: “Really? That’s @antirez? He doesn’t look anything like his online avatars!” 🙂 It’s not that I was expecting a Robert Downy Jr. or an alley cat, but meeting him face-to-face was still jarring. He stood before me – average only in height with an honest smile and intelligent eyes – and we shook hands. We had a great, albeit short, discussion about distributed systems until it was time for the day to start in earnest.
And, of course, it started with a presentation by Antirez:
Salvatore’s opening session reflected the day’s theme – it was all about taking Redis, and specifically its development, to the next level. From a development project’s perspective, there’s no doubt that the current model has limitations when it comes to scalability (a fact that was admitted openly by Salvatore and Matt). With limited resources and multiple tasks, they find it hard to advance effectively on all fronts while balancing their different efforts. There was definite agreement among all participants that more should be done to pull additional developers into the project and facilitate easier contributions. For example, new ideas should be discussed in the mailing list to get more exposure and opinions. Since the current mailing list serves two purposes – developers and Q&A – we’d suggest splitting it into two separate lists for a step in that direction.
The upcoming Redis Cluster (RC1 expected to be released tomorrow) was discussed as well. It is, perhaps, the biggest step the Redis project can take to evolve and increase its usefulness and applicability. Listening to Salvatore, I became convinced more than ever that the community is also moving forward, itself evolving side by side with the project. The transition from a “one man’s show” to the next phase is a trip through uncharted waters that is anything but easy, but Salvatore seemed honestly committed to making it happen.
Matt Stancliff followed with a session that, with a healthy dose of humor, covered his views of similar challenges. His inspiring work on a plugin system (plus his geocoding and JSON implementations with it) exemplifies the need for more modularity in Redis, and an easy way for other developers to extend it. Personally, I can’t wait for this addition to make it into the mainstream source, but there is concern that plugins will introduce instability to Redis deployments (with great power comes great responsibility – Uncle Ben, but stupidity is infinite – Albert Einstein).
Another modularity aspect that was discussed was how to make Redis itself more reusable by splitting it to formal sub-components such as the event loop, networking, librdb, existing data types and so on. Despite the code refactoring needed, this would potentially allow developers to not only extend Redis easily and in a maintainable way (think new data types and encodings), but to also to adopt Redis’ internals for building upon and increase contributions back from the community.
As the day progressed, we had several more presentations and roundtable discussions.
The Redis team presented this slides deck, which includes several of our ideas for development that address the operational challenges that we’re dealing with. As a Redis-as-a-Service provider, we’re not the typical Redis user because we operate dense multi-tenant Redis deployment across cloud platforms, but we feel that improvements like RDB streaming, efficient Lua scripts replication and better memory management are very applicable to the general public. These contributions to Redis’ development are already being used in production by our Redis Cloud service and some are open sourced from our Redis fork at GitHub. In addition, we unveiled a project that we’ve been working on for quite some time now – using flash memory as an extension to a server’s RAM. You’ll be hearing more from us about this in the near future, as we’re quite proud of it and plan to open source it.
The roundtable discussions were a real kicker – a lot of ideas were put on the table for feedback. Among these were Bloom filters, reliable PubSub mechanisms and changes to the RDB file format. I expect both Salvatore and Matt to provide their own writeups of the event in the near future, so stay tuned for more. In the meantime, here’s the list of urgent issues that Antirez had put together based on the discussions that took place that day: https://github.com/antirez/redis/issues/2045.
It is pretty amazing to be part of the Redis project, not just because of the awesome technology, but because of the people that we get to meet and work with: my personal hero Marc Gravell (he answers questions on SO in his spare time! :)), the friendly competition, other Redis developers and users, and of course Salvatore and Matt #theincredibleduo. I personally found the experience absolutely gratifying, and as a team, Redis is 100% committed to continue sharing and being a part of the Redis community.
To sign off, I’m borrowing this tweet from Salvatore: “returning home with many ideas and interest in making new stuff happen… thank you”
O frabjous day! Callooh! Callay!