If you are storing purely normalized numerical data, then Redis can very efficiently store time series data in a bitfield. To store data, we first must determine an arbitrary epoch time, resolution, and number format. Let’s take our example of capturing temperature data. Let’s assume we want to take a temperature once every minute and we’ll set the epoch at midnight of a single day. Let’s assume we’re measuring indoor ambient temperatures in celsius.
That being said we can structure our data like this:
This will yield a day’s worth of data in around 1.44KB
We can record the temperature with the BITFIELD command:
> BITFIELD bit-ts SET u8 #0 22 1) (integer) 0
In this case we recorded the temperature in the key bit-ts in an unsigned 8-bit (u8) at midnight (#0) and with the temperature of 22. Bitfields aren’t limited to unsigned 8-bit values, in fact you can manipulate a single bit through to 63 bits unsigned or 64 bits signed. Note the hash prior to the offset (#0), this means that you want it aligned to the data type specified. For example if you specify #79, that would mean 79 bytes in – leaving off the hash would mean the 79th bit.
The offset can is aligned to the type of number being stored starting with 0. For example, if we want to record 1 A.M., accounting for zero based slots, we would use an offset of #59 or noon for an offset of #719.
> BITFIELD bit-ts SET u8 #59 23 SET u8 #719 25 1) (integer) 0 2) (integer) 0
This illustrates the BITFIELD is variadic, you can add multiple values at one time.
Let’s add a few more values to our time-series:
> BITFIELD bit-ts SET u8 #60 21 SET u8 #61 20 1) (integer) 0 2) (integer) 0
Now, let retrieve those values:
> BITFIELD bit-ts GET u8 #59 GET u8 #60 GET u8 #61 1) (integer) 23 2) (integer) 21 3) (integer) 20
BITFIELD’s sub-command GET is formatted just like SET except it doesn’t accept a value as its third argument.
This is fine if we know all the indexes we need to get, but sometimes we need a range of values and it would be too much to specify each byte individually. We can use the GETRANGE command. Normally, this is used to retrieve bytes from string, but BITFIELDs are just a different way of addressing the same underlying data.
> GETRANGE bit-ts 59 61 "x17x15x14"
This has returned bytes 59 to 61 in hex form (23, 21, and 20 in decimal). Language clients do a better job of handling binary data than redis-cli and can usually retrieve your languages’ idiomatic byte array.
In our example we’ve manipulated bytes 0, 59-61 and 719. What happens if we request a byte that we haven’t set yet? Let’s see:
> BITFIELD bit-ts GET u8 #40 1) (integer) 0 > BITFIELD bit-ts GET u8 #750 1) (integer) 0 > GETRANGE bit-ts 30 50 "x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00x00"
We can see that Redis treats unassigned bytes as hex 00 / decimal 0. This can be tricky when dealing with time series data – your application logic needs to make some differentiation between a value of 0 and unset. This may involve some sort of artificial floor and skipping values set as 0, especially when using signed integers as this might be a legitimate value in the middle of your range.
The actual stored length of the time series is actually dependent on the last byte set. So, in our example, the last byte stored was byte #719 (zero-based), so the data is 720 bytes long. We can confirm this with STRLEN:
> STRLEN bit-ts (integer) 720
BITFIELD-based time-series are a powerful and compact pattern for storing numerical or binary data. However, this solution doesn’t fit all use-cases and should be carefully evaluated if it fits your particular needs.
