Architecture Secret: Move flowers and wood. Simulate redis using mysql

The article was reproduced from the public number: Little Sister Taste

This year, what you see is not necessarily what you think. Behind a mysql protocol, it could be tidb behind a linux machine, it might be a streamlined docker; you think xjjdog is a woman, but maybe you don't know much about it yourself; and when you're yelling php, it's probably the developer and you who're kidding, rewriting the suffix, and the back end using java.

We all know that redis is fast, but its capacity is related to memory capacity and can easily reach bottlenecks. S ome Internet companies use redis directly as a back-end database (in the following admiration). W hen business volumes skyrocket, there is a trade-off between redis capacity and price. It's too late to change the business code to simulate some of redis' data structures with some persistent storage.

redis supports nearly ten data types, five of which are most commonly used. string hash zset set list etc. This article explores the simulation implementation of common operations for several common data structures.

In fact, what we need to develop is a redis proxy proxy redis clients, after connecting to our agent, are resolved by protocol. The parsed commands are simulated and then positioned in the appropriate mysql based on the configured route.

That is, the redis you use actually use mysql to store the data. There is no rdb and there is no aof

Redis is a text protocol

redis is a text protocol with a protocol name called RESP RESP is a shorthand for the Redis serialization protocol. It is an intuitive text protocol with the advantage that it is exceptionally simple and has excellent parsing performance.

As shown, the Redis protocol transfers structural data that can be summed up into five minimum cell types. At the end of each cell, the carriage return line break symbol is added uniformly.

Here are a few rules:

单行字符串 以 + 开头；
多行字符串 以 $ 开头，后跟字符串长度；
整数值 以 : 开头，后跟整数的字符串形式；
错误消息 以 - 符号开头；
数组 以 * 号开头，后跟数组的长度；

For example, here's an array of messages.

*3\r\n:9\r\n:9\r\n:6\r\n

So the resolution and assembly of this protocol is very simple. In netty case, we have codec-redis modules for us to use.

Implementation: Data structure design

In the design of the data table, we found that kv and hash are no different in efficiency because they can be located directly according to key

zset on the other hand, results in a number of operations that are not performed as efficiently as possible due to the ability to sort.

In addition, because of our different data structures, we use different tables for storage. So delete the operation and do it again on each table.

kv design

kv or string is the most basic type of data in redis One key to one value and the value of the string type can store up to 512MB.

Design a dedicated database table rstore_kv where rkey is the primary key.

rkey        varchar
val     varchar
lastTime    bigint

Set operation

insert into rstore_kv("rkey","val","lastTime") values($1,$2,$3)
on duplicate key update set "val"=$2,"lastTime"=$3

Get action

select val from rstore_kv where "rkey" = $1

del operation

delete from rstore_kv where "rkey" = $1

the exists operation

select count(*) as n from rstore_kv where  "rkey" = $1

ttl operation

select lastTIme from rstore_kv  where  "rkey" = $1

Hash design

hash is a set of key values (key->value). hash is ideal for storing objects.

Design a dedicated database table rstore_hash where rkey and hkey are the joint primary keys.

rkey        varchar
hkey        varchar
val     varchar
lastTime    bigint

hset operation

insert into rstore_hash("rkey","hkey","val","lastTime") values($1,$2,$3,$4)
on duplicate key update set "val"=$3,"lastTime"=$4

hget operation

select val from rstore_hash where "rkey" = $1 and "hkey" = $2

hgetall operation

select hkey,val from rstore_hash where "rkey" = $1

hdel operation

delete from rstore_hash where "rkey" = $1 and "hkey" = $2

del operation

delete from rstore_hash where "rkey" = $1

hlen, hexist operation

select count(*) as num from rstore_hash where "rkey" = $1

ttl operation

select max(lastTIme) from rstore_hash  where  "rkey" = $1

zset design

Redis zset like set is a collection of string type elements and does not allow duplicate members. T he difference is that each element is associated with a double score. redis uses fractions to sort members of a collection from small to large. Its underlying structure is a jump table, which is particularly efficient but takes up a lot of memory.

Design a dedicated database table rstore_zset where rkey and member are the joint primary keys.

rkey        varchar
member        varchar
score     double
lastTime    bigint

Zadd operation

insert into rstore_zset("rkey","member","score","lastTime") values($1,$2,$3,$4) on duplicate key update update set "score"=$3,"lastTime"=$4

zscore operation

select score from rstore_zset where "rkey" = $1 and "member" = $2

zrem operation

delete from rstore_zset where "rkey" = $1 and "member" = $2"

zcard, exists operation

select count(*) as num from rstore_zset where "rkey" = $1

zcount operation

select count(*) as num from rstore_zset where "rkey" = $1 and score>=$2 and score<=$3

zremrangebyscore operation

delete from rstore_zset where "rkey" = $1 and score>=$2 and score<=$3

zrangebyscore operation

select member,score from rstore_zset
where "rkey" = $1 and score>=$2 and score<=$3 order by score asc,member asc

zrange operation

select member,score from rstore_zset
where "rkey" = $1 order by score asc offset $2 limit $3

zrank operation

select rank from (select member,rank() over (order by "score" asc, "lastTime" asc) as rank from rstore_zset where "rkey" = $1 ) m where m."member"= $2;

ttl operation

select max(lastTIme) from rstore_zset  where  "rkey" = $1

del operation

delete from rstore_zset where "rkey" = $1

Set design

Redis Set string type is a disordered collection.

Design a dedicated database table rstore_set where rkey and member are the joint primary keys.

rkey        varchar
member        varchar
lastTime    bigint

Sadd operation

insert into rstore_set("rkey","member","lastTime") values($1,$2,$3)
on duplicate key update update set "lastTime"=$3

Scard operation

select count(*) as num from rstore_set where "rkey" = $1

Sismember operation

select member from rstore_set where "rkey" = $1 and "member" = $2

smembers operation

select member from rstore_set where "rkey" = $1

srem operation

delete from rstore_set where "rkey" = $1 and "member" = $2

del operation

delete from rstore_set where "rkey" = $1

ttl operation

select max(lastTIme) from rstore_set  where  "rkey" = $1

End

This article simulates only the most common features of the most commonly used data structures, many, many of which are not supported, notably distributed lock setnx and so on. So the development of this proxy layer, to do ok, is not so simple.

At the same time, let's take a look at the data structure of redis from a simulated perspective, in relational databases. In this way, we can deepen our understanding of redis and understand the value of its existence.

Here's W3Cschool编程狮 about architecture secrets: moving flowers and wood. I hope to help you with the introduction of mysql simulation redis.