Relocation Redis’ community establishes home in the /r/redis subreddit

Much to the chagrin of email practitioners, Salvatore Sanfilippo @antirez had decided to move the community’s home from its long-time residence at the redis-db Google Groups mailing list to /r/redis. The rationale behind the decision boils down to evolving and revitalizing the community’s channels of communication. Presently, a few weeks into the journey, both redis-db and /r/redis seem to be thriving and while dubbed as an “experiment” that may fail or succeed, I find it fun (and a good excuse for surfing reddit on work time!) – how about voicing your own opinion in this discussion?

Extra bits

• The announcement in the redis-db mailing list
• Updates from /r/redis on Twitter via @RedditRedis

Minor release Redis 3.0.5 and 2.8.23 released (3 minutes to read)

ICYMI a minor Redis release with moderate severity issues resolved.

Migration Migrating Redis Data (6 minutes to read)

Parker Selbert @sorentwo reviews, compares and contrasts the ways you can take to migrating your Redis data. I would call this #RecommendedReading in heartbeat if it wasn’t for him quoting me – quoting me is the mark of a wise man or big fool (and you can quote me on that).

OH Marcus Smith @roomnoise > Redis is the duct tape of modern distributed engineering.

PSA Storing Data with Redis (5 minutes to read)

An educational piece from Mike Perham @mperham, creator of the OSS Sidekiq and Inspeqtor, on “the drawbacks of stuffing everything into one Redis database”. #RecommendedReading both because of the lessons (donts: namespacing & numbered databases, dos: dedicated instances) but also because of this gem of a sentence: “If you are using Redis for caching (and you should probably use memcached instead), …”

Clarifications about Redis and Memcached (6:37 minutes to read)

Salvatore Sanfilippo @antirez‘s retort to the above summarizes the real differences between the two technologies, namely Redis’ memory and allocator efficiencies, modifiable data, persistence, replication and Lua. I/O multi-threadedness is also mentioned and it is likely that Redis will get this capability the not-too-distant future given the work done on lazy delete (note: multithreaded I/O was also discussed during RDD15).

NEW! Indexing data for range queries in Redis

If there’s one key concept that really baffles many newcomers to the Redisverse it is most likely the fact that data is accessed by its primary (and only) key. Based on both personal experience as a long-time RDBMS person as well as upon my interactions with the Redis community, I believe that the crux of the matter lies in the pampering that we’ve grown so accustomed to when working with relational databases. In case you forgot, just like Redis, primary key lookup is also what RDBMSs excel at. However, a respectable RDBMS will let you get away with any syntactically-correct query and will return the data to you even if it’ll take it a lifetime to do so. To fetch any reply to a query that isn’t based on primary keys, an RDBMS will default to a full scan (it is, in fact, optimized for that) unless it is set up with relevant indices.

The robustness of modern relational query parsers and the ease with which one can CREATE INDEX on practically anything have made us expect the database to respond to all types of queries and at most “help” it by giving it a few friendly hints. And if you’re working in a relational model these expectations and modus operandi are perfectly acceptable. But Redis isn’t a relational database and won’t provide you with answers to arbitrary queries – you have to prepare the data according to the way you plan to retrieve it and that’s it. That basically means that you need to build your own 2ndary indices in Redis if you need an efficient non-primary-key-based way to access your data.

So how do you go about creating an index in Redis? That depends. Depending on your data and query, there are different ways to do that (see, for example, chapter 7 on searching in Redis in Action by Josiah Carlson @dr_josiah or my very own Redis Indices preso on the same). An advanced technique such as Composite Score Sorted Set Indexing (CSSSI) is very useful when you want to use two scores for your data, but even that is limited in precision to ~53bits and becomes practically unusable with higher dimensions.

For me the obsession with range queries in Redis had started when I was a young Redis advocate after stumbling upon this question from Stack Overflow: Search a key inside a block of keys with redis. I was haunted by a compelling need to answer but since then I was never too happy with my result (admittedly, it was written being tipsy at a wedding party of one of the Redis team members). Since then, range queries kept popping in all kinds of places, and while a good solution was always possible on a per-case basis, a more generic approach seemed to be called for. The final straw (in the haystack?) of the first act was yet another Stack Overflow question: Intersection of two or more sorted sets

Second act: as RDD15 was getting nearer, I advocated the need for range queries to Redis’ developers and provided the numbers from the Quadtree PoC to jumpstart the discussion. First, I pitched it internally to the lead and the team of open source Redis developers and afterwards at RDD itself. A detailed accounting of the ensuing discussions is outside the scope of a newsletter item that’s already getting far too long as it is, so I’ll stop blogging and get to the immediate outcomes.

Documentation: A new Secondary indexing with Redis page has been added to the corpus by Salvatore Sanfilippo @antirez per the “good software means good documentation” principle that he practices (reminder: if you want to contribute to the documentation that’s simple – it’s all open source).

Redimension: Also from Salvatore Sanfilippo @antirez is the Redimension #Ruby library that has two purposes. Primarily, it takes away the pain from using lexicographical ranges to perform range queries on multi-dimensional elements. On top of that, it is an experiment for exploring an indexing (indicing?) API that could be added to Redis regardless of the underlying data structures (i.e.g. btree, Quadtree, kd-tree, R-tree or others). My contribution was to port and implement Redimension in Redis-compliant #Lua that can be used from anywhere.

I’m pretty confident that this isn’t the last time you’ll be hearing about indexing, ranges, trees and the rest of that good stuff in Redis.

Power to the People Recent improvements to Redis #Lua scripting (timeless)

TL;DR also post-RDD, a debugger for Redis Lua scripts that is contained in Redis and effects replication for Lua scripts from the purist, fixed-width article writer, somewhat-groggy-in-the-morning-until-the-coffee-kicks-in video overview-and-demo (33:40 minutes to watch) recorder and a big time cheater who develops Redis 🙂 Also, mark your calendars and get ready for a v3.2 RC scheduled for this December.

Honorable Mention: Marijan from Tikoder @trikoder had recently considerably ramped up his workflow for debugging Redis Lua scripts to near perfection – http://www.trikoder.net/blog/redis-and-lua-scripts-new-workflow-89

OH Sergei Egorov @bsideup > Damn it! Redis is so fast so I spent 30 minutes to figure out that “0ms” length of the trace is not a bug >_< /cc @antirez @Redis

OH Ossie Okeke @ossieokeke > @antirez ….does Redis have a psychic feature? I think my request just came back a few milliseconds before i made it!

Security A few things about Redis security (1337)

You should know better than leaving your servers wide open to the internets, but if you need a reminder of just how dangerous it is to let people access an uprotected server here you go. Now go and make sure that the requirepass configuration directive is enabled on your server and then read the Redis Security documentation page.

Secure Redis #Security #howto

While on the topic of security, José Coelho @ZeCoffee had put together a detailed guide on setting up highly available Redis for those who are bothered with compliance issues.