May 31, 2021 Article blog
This article comes from the public number: Java Geek Technology Author: Duck Blood Fans
Hey, Recently a Powder Has Committed Another Double.
Here's the thing, some time ago A powder company production transactions were occasionally misreported, and the ultimate reason for this was because redis commanded a timeout.
It is puzzling, however, that production transactions use only the simple command redis set, which makes sense that it is not possible to execute so slowly.
So what's causing this problem?
To find out, we looked at the recent slow logs of Redis and found that the command that took more time was
keys XX*
Seeing the prefix of the key of this command operation, A powder only found that this is the application for which he is responsible.
But a powder check, although their own code did not take the initiative to use
keys
but the underlying usage framework is being used indirectly, so there is today's problem.
A powder-responsible application is a management background app, and permission management uses the Shiro framework, where Redis is used to store Session information because there are multiple nodes that require distributed Sessions.
Since Shiro did not provide the Redis Storage Session component directly, A powder had to use a github open source component, shiro-redis.
Because
the Shiro
framework needs to periodically verify that Session is valid,
the Shiro
underlying layer will call
SessionDAO#getActiveSessions
to get all the Session information.
shiro-redis
on the other arm,
SessionDAO
interface, using
keys
command to find all of Redis's stored
Session
keys.
public Set<byte[]> keys(byte[] pattern){
checkAndInit();
Set<byte[]> keys = null;
Jedis jedis = jedisPool.getResource();
try{
keys = jedis.keys(pattern);
}finally{
jedis.close();
}
return keys;
}
Finding the cause of the problem is simpler, finding a solution on github, and upgrading
shiro-redis
to the latest version.
In this release,
shiro-redis
fixes this problem by replacing
keys
with the
scan
command.
public Set<byte[]> keys(byte[] pattern) {
Set<byte[]> keys = null;
Jedis jedis = jedisPool.getResource();
try{
keys = new HashSet<byte[]>();
ScanParams params = new ScanParams();
params.count(count);
params.match(pattern);
byte[] cursor = ScanParams.SCAN_POINTER_START_BINARY;
ScanResult<byte[]> scanResult;
do{
scanResult = jedis.scan(cursor,params);
keys.addAll(scanResult.getResult());
cursor = scanResult.getCursorAsBytes();
}while(scanResult.getStringCursor().compareTo(ScanParams.SCAN_POINTER_START) > 0);
}finally{
jedis.close();
}
return keys;
}
Although the problem was successfully solved, but A powder heart is still a little puzzled.
Why do
keys
slow down other command execution?
Why is
Keys
instruction query so slow?
Why is
Scan
directive okay?
First, let's look at the first question, why does
keys
directive cause other commands to execute more slowly?
To answer this question, let's first look at how the Redis client executes a command:
Standing from the client's perspective, executing a command is divided into three steps:
But this is only what the client thinks of itself, but in fact at the same time, there may be many clients sending commands to Redis, which we all know uses a single-threaded model.
In order to handle all client request commands at the same time, Redis takes the form of queues internally, queued to execute.
So the client actually takes four steps to execute a command:
Because Redis single-threaded execution commands can only be performed sequentially from the queue to start the task.
As long as the 3 process executes commands too slowly and other tasks in the queue have to wait, redis appears to external clients to be blocked and has not been responded to.
Therefore, the Use Redis procedure must not execute instructions that require a long run, which can cause Redis to block and affect the execution of other instructions.
Let's start by answering the second question, why is
Keys
instruction query so slow?
Before you answer this question, think back to the underlying storage structure of Redis.
It doesn't matter if you don't know your friends, you can look back at the previous article "Ali Interviewer: HashMap familiar?" O kay, let's talk about the Redis dictionary! 」。
Here A Powder CopieS The Previous Article, Redis Underlying Uses The DictionarY Structure, Which Is Similar To The Java HasHMaP Base.
The keys command
keys
to return all Redis mid-keys that conform to the given pattern
pattern
and for this purpose, Redis has to traverse the dictionary's underlying array of
ht[0]
tables, which is
"O(N)"
(N is the number of keys in Redis).
If the number of keys in Redis is small, this execution is still very fast. By the time the number of Redis keys slowly increases, to millions, millions, or even hundreds of millions, then this execution will be slow and slow.
Here's an experiment done locally, using lua scripts to add 10W keys to Redis, and then using keys to query all
keys
which can block for about a dozen seconds.
eval "for i=1,100000 do redis.call('set',i,i+1) end" 0
Here A powder deploys Redis using Docker, and performance may be slightly worse.
Finally, let's look at the third question, why is
scan
instruction okay?
This is because
scan
command uses a black technology -
a cursor-based iterator.
Each time the
scan
command is called, Redis returns a new cursor and a certain number of keys to the user.
The next time you want to continue getting the remaining keys, you need to pass the cursor into the scan command to continue the previous iteration.
Simply put, the
scan
command uses paginated queries for redis.
Here is an example of an iterative process of a scan command:
scan
command uses cursors to subtly split a full query into multiple times, reducing query complexity.
Although
scan
command is as time-complex as
keys
it is
"O(N)",
but because
scan
command only needs to return a small amount of key, it executes quickly.
Finally, while the
scan
command
keys
there are other drawbacks that are introduced:
These defects need to be considered in our development.
In addition to
scan
redis has several other incremental iteration commands:
sscan
Used to iterate on database keys in the current database to resolve blocking issues that
smembers
may be causing
hscan
command is used to iterate on key-value pairs in hash keys to resolve blocking issues that
hgetall
can cause.
zscan
Commands are used to iterate on elements in an ordered collection, including element members and element scores, to generate
zrange
that can cause blocking problems.
Redis uses a single thread to execute the action command, all clients send over the command, Redis is now placed in the queue, and then taken out of the queue in order to execute the corresponding command.
If either task is performed too slowly, it affects the other tasks in the queue, making it seem blocked for the external client to be slow to get a response from Redis.
So don't execute blocking instructions like
keys
smembers
hgetall
zrange
in production, and if you really need to, you can use the appropriate
scan
command to step through them to effectively prevent blocking problems.
That's
W3Cschool编程狮
says about
wanting to do things in production?
So try these Redis commands
and hope to help.