Most database migration tools available today are offline in nature. Migrating data from AWS ElastiCache to Redis Cloud for example means backing up your Elasticache data to an AWS S3 bucket and importing it into Redis Cloud using its UI.This implies some downtime and might result in data loss. Other available techniques include creating point-in-time snapshots of the source Redis server & applying the changes to the destination servers to keep both servers in sync. It might sound like a good approach but can be challenging when you have to maintain dozens of scripts to implement the migration strategy.
RIOT Redis is a migration tool that allows for seamless live replication between two Redis databases.
1. Getting Started#
Download the latest release and unzip the archive.
Launch the bin/riot-redis script and follow the usage information provided.
2. Build and Run#
git clone https://github.com/redis-developer/riot.git
cd riot/riot-redis
./riot-redis3. Install via Homebrew (macOS)#
brew install jruaux/tap/riot-redis`Usage#
❯ riot-redis
Usage: {app} [OPTIONS] [COMMAND]
--help Show this help message and exit.
-V, --version Print version information and exit.
-q, --quiet Log errors only
-d, --debug Log in debug mode (includes normal stacktrace)
-i, --info Set log level to infoYou can use --help on any subcommand:
❯ riot-redis --help
❯ riot-redis import --help
❯ riot-redis import .. hset --helpRedis connection options are the same as redis-cli:
-h, --hostname=<host> Server hostname (default: 127.0.0.1)
-p, --port=<port> Server port (default: 6379)
-s, --socket=<socket> Server socket (overrides hostname and port)
--user=<username> Used to send ACL style 'AUTH username pass'. Needs password.
-a, --pass[=<password>] Password to use when connecting to the server
-u, --uri=<uri> Server URI
-o, --timeout=<sec> Redis command timeout (default: 60)
-n, --db=<int> Database number (default: 0)
-c, --cluster Enable cluster mode
-t, --tls Establish a secure TLS connection
-l, --latency Show latency metrics
-m, --pool=<int> Max pool connections (default: 8)Redis URI syntax is described here.
4. Example#
Here is an example of a live replication from a source Redis running on localhost and port 6379, to a target Redis running on localhost and port 6380:
❯ riot-redis -h source -p 6379 replicate --idle-timeout 500 -h target -p 6380 --live5. Verification#
Once replication is complete RIOT Redis will perform a verification step to compare values and TTLs between source and target databases. The output looks like this:
OK:1000 V:0 >:0 <:0 T:0- •OK: # identical values
- •V: # mismatched values
- •: # keys only present in source database
- •<: # keys only present in target database
- •T: # keys with TTL difference greater than tolerance
6. Architecture#
RIOT Redis implements client-side replication using a producer/consumer approach:
- •the producer is connected to the source Redis (e.g. ElastiCache) and iterates over keys to read their corresponding values
- •the consumer is connected to the target Redis (e.g. Redis Cloud) and writes the key/value tuples previously created
- 1.Key reader: initiates a SCAN and optionally calls SUBSCRIBE to listen for keyspace notifications (live replication).
- 2.Value reader: takes the keys and calls DUMP and TTL.
- 3.Key/Value writer: takes key/value/ttl tuples and calls RESTORE and EXPIRE.
NOTE
Live replication makes use of keyspace notifications. Make sure the source Redis database has keyspace notifications enabled using notify-keyspace-events = KA in redis.conf or via CONFIG SET.
NOTE
The live replication mechanism does not guarantee data consistency. Redis sends keyspace notifications over pub/sub which does not provide guaranteed delivery. It is possible that RIOT Redis can miss some notifications in case of network failures for example.