Redis Migration With Zero Downtime to Memorystore
In this post we are going to discuss Redis Migration With Zero Downtime from One Server to another server or to Memorystore. There are several ways to migrate from one Redis Server to another Redis Server or MemoryStore but none of them supports zero downtime. Transferring data between systems poses significant risks for businesses. It maybe potential downtime, the risk of data loss, and the possibility of data corruption, all of which are just a few examples of potential complications that can arise.
While the process of migrating data from older or obsolete databases to newer ones carries inherent risks, businesses must embrace innovation to ensure their survival. Modern databases come with advanced features and capabilities that can enhance business operations and maintain competitiveness. For instance, customers transitioning from self-managed databases to fully-managed databases provided by cloud providers can give numerous benefits.
Setup Source Redis Instance in a VM
Installing and Configuring Redis Instance
You can deploy the Redis server by utilizing the APT package manager sourced directly from the official Ubuntu repositories.
Start by updating your local APT package cache:
sudo apt-get update
Install Redis by using the following command
sudo apt install redis-server
The above command will download and install all dependencies for Redis Server, we need to change several options in the Redis Configuration for migration, below is the command to edit the configuration file
sudo nano /etc/redis/redis.conf
Now that we have installed and setup source redis server for migration, before using it let’s check if it is live
Testing Redis Instance
use the following command to check the installed redis server status
sudo systemctl status redis-server
If it is running without issue then below output will come.
This output indicates that Redis is up and running and is ready to boot whenever the server starts. Let’s connect to the Redis Server with redis-cli command
If you enter like below then redis server is running fine.
you will receive the response like below
this confirms that the redis server is running fine just enter exit it will exit from the redis server
Loading Data into Redis Source Server
Load data to the source Redis Server if you have or use the following sample data
Redis Movie sample dataset that available for download from Github repository, Redis Movie Database Sample Dataset.
This sample dataset contains movies stored as Redis Hash. The movie hashes contain the following fields:
movie:id : The key of the hash title : The title of the movie plot : A summary of the movie genre : The genre of the movie, for now a movie will only have one single genre release_year : The year the movie has been released as a numerical value rating : The ratings from the public numerical value votes : Number of votes poster : Link to the movie poster imdb_id : id of the movie in the IMDB database
Importing the dataset
Copy and run the following command to import and insert in Redis Source Server
wget https://github.com/redis-developer/redis-datasets/blob/master/movie-database/import_movies.redis redis-cli -h 172.16.0.64 -p 6379 < ./import_movies.redis
you can check some data using redis-cli with below command:
HMGET "movie:343" title release_year genre
1) "Spider-Man" 2) "2002" 3) "Action"
we are going to use this dataset as the source for our migration to Memorystore Redis.
Note : Here I’m going to use Memorystore as target you can also use any other Redis Server as target just by changing the ip and other details in the migration process.
Setup Memorystore Redis instance
create a Memorystore Redis instance, you can use the Google Cloud console, the gcloud CLI, or the Memorystore API
Using the gcloud CLI:
- Open the Cloud Shell from GCP Console
- Set the project you want to create your instance in as the default project in gcloud by entering the following command:
gcloud config set core/project PROJECT_ID
- Enter the following command to create a 1 GiB Basic Tier Redis instance in the us-central1 region with Redis version 7 name redis-prod:
gcloud redis instances create redis-prod --size=1 --region=us-central1 --redis-version=redis_7_0
- After the instance is created, enter the describe command to get the IP address and port of the instance:
gcloud redis instances describe redis-prod --region=us-central1
- Check Redis functionality similar to the Source Redis Server setup
Deploy RIOT for Live Replication of Redis Instance
Most of the Redis Migration tools that are available today or offline in nature means need to take backup from the source and restore on the target. But here RIOT implements client-side replication using dump & restore or type-based read & write. Both snapshot and live replication modes are supported.
In live replication mode, RIOT monitors the source database for any changes using keyspace notifications. Whenever a key undergoes modification, RIOT retrieves the associated value and transmits that modification to the target database.
Live replication depends on keyspace notifications. We have to make sure that the source database has keyspace notifications enabled using notify-keyspace-events = KA in redis.conf (or via CONFIG SET). For more details see Redis Keyspace Notifications.
Also note that this live replication mechanism does not guarantee data consistency. It is also possible that RIOT may miss some notifications incase of network failure of source server.
Moreover, depending on factors such as data structure type, size, and the rate of changes in the source database, RIOT may encounter challenges in keeping pace with the stream of changes.
RIOT operates by processing data in batches, where a fixed number of records (referred to as a batch or chunk) are read, processed, and written at a time. This process is repeated until all data from the source has been handled.
Setup RIOT instance
We need a VM to install RIOT tool for the migration, we can also use the source VM for installing the Tool but to avoid any disruption in the prod environment we are going to setup this tool in another VM.
Download and Unzip the RIOT tool
wget https://github.com/redis-developer/riot/releases/download/v3.1.5/riot-standalone-3.1.5-linux-x86_64.zip unzip riot-standalone-3.1.5-linux-x86_64 cd riot-standalone-3.1.5-linux-x86_64/bin
Run the Replication
Below is the command to start migration or replication
riot -h <ip-address> -p <port> replicate -h <ip-address> -p <port> –batch <integer> --mode <snapshot|live|compare>
-h, --hostname Server hostname -p, --port Server port
There are many options and tasks that RIOT can handle such as transforms and applying filters. Details of the command and options for the migration can be found in the official documentation here
Let’s start the live migration using the simple following command
./riot -h 172.16.0.64 -p 6379 replicate -h 10.185.99.203 -p 6379 --mode live
Check the Replication Data
check the replication by inserting new data in our Redis source using below command:
redis-cli -h 172.16.0.64 -p 6379
HSET "movie:1200" title "Guardians of the Galaxy" genre "Action" votes 704613
After the insert, let’s check the new data in our target Redis instance:
redis-cli -h 10.185.99.203 -p 6379
HMGET "movie:1200" title release_year genre
1) “Guardians of the Galaxy”
the new data has been replicated successfully to our target VM or Memorystore Redis instance.
Once the replication is completed successfully you can switch the traffic from your old redis instance to new instance by changing the application properties.
Tags : Redis Migration With Zero Downtime, Redis to Memorystore Migration without downtime, Redis live migration ,Redis Server to another Redis Server zero downtime
Please leave your feedback or comments below.