How do you look at these memory consumption data of redis, mainly depending on which one indicates t

First of all, you need to understand that Redis is a database Redis is an in-memory database, and all data basically exists in memory, and will be periodically refreshed to the hard disk in the form of appends or snapshots. Since Redis is an in-memory database, the speed of read and write is very fast, so it is often used for caching data, pages, etc.

Nowadays it is used in more and more systemsredisAs a caching system, but with the growth of the number of users, the business data continues to increase, the memory space of the Redis server may reach a bottleneck, timely observe how much memory occupied by various keys in Redis, it will be convenient for us to evaluate when to upgrade the Redis server specifications, and whether it is necessary to optimize the program to design a reasonable storage structure, the following will introduce two tools to yourdrwithredis-rdb-tools, which can meet our needs very well.

1.First of all, I have to go to the Redis server side to copy the RDB file to the local computer, for convenience, I put the RDB file in the directory where the RDR tool is located.

Here, let's talk about how to find the Redis RDB file, by looking at the Redis server configuration fileYou can quickly locate the configuration and follow the path to find the Redis RDB backup file.

2.Open a command-line window in the path where the RDR tool is located and execute the command.

After the command is executed, the server is started locally and listens on port 8080

3.Open a browser and visit http: localhost:8080 to see a detailed memory usage data report, including the number of keys, different data types, element counts, etc.

However, there is a small problem in the data report through the web version, there is no obvious distinction between the different databases in Redis, and they are mixed together, which is not too clear.

For more information, please refer to the official repository.

1.Install the python environment, I installed it here.

2.Install redis-rdb-tools via pip

I didn't install the python-lzf library here, but it doesn't affect the actual use, this library is to speed up the RDB file parsing speed

3.After the installation is complete, it can be used in the command line, and the input instruction is used to generate a memory report file.

After waiting for a period of time, after the command blocking is executed, the corresponding file will be generated in the path specified by the -f parameter.

Open the generated CSV file in Excel to see detailed statistical results, including the memory usage of all keys in all databases

redis-rdb-tools also comes with a very useful command that can help us directly query the memory usage of a single key, in the following format.

After the command is executed, the memory usage corresponding to the specified key is directly displayed.

Note that this operation should be used with caution in the production environment, and the execution may be blocked for a long time depending on the size of the key, and the result may be calculated

After the client connects to the server via redis-cli, enter info memory

The command will display the statistics of the Redis system about storage, and this article focuses on the three terms used memory, used memory RSS, and MEM fragmentation ratio to make a detailed explanation, and everyone is welcome to criticize and discuss.

Used memory is allocated by a Redis memory allocator (e.g. jemalloc).The total amount of memory, which is mainly used to store the data generated by the actual runtime of Redis. Note that the total amount of memory mentioned here includesMemorywithVirtual memory

Used memory RSS is the memory occupied by the Redis process in the operating system, so in addition to the memory allocated by the allocator, it also includes the memory required for the process itself to run, memory fragmentation, etc., howeverVirtual memory is not included

From the description of used memory and used memory rss, it is difficult to determine which is greater or less between them, and the reasons are as follows:

1.If the used memory uses virtual memory, if the virtual memory usage exceedsThe Redis process itself occupies a large amount of memoryProgram memory fragmentation, the Used Memory value may be greater than the Used Memory RSS.

If the used memory is not occupied or occupies a small amount of virtual memory, the used memory RSS value may be greater than the used memory, contrary to the previous article.

The relationship between the two can reflect the running state of the Redis system from the side, which leads to the third term, MEM fragmentation ratio.

MEM fragmentation ratio is the ratio of used memory to used memory RSS used memory. Based on the previous description of what used memory rss and used memory mean, it is easy to draw the following conclusions:

Use redis-cli to log in to the Redis server, execute the info command, where'# memory'The paragraph section is the usage of Redis memory, for example:

This is in the configuration file:

Redis supports running times to dynamically modify the memory size via commands.

Since you can set the maximum memory size of Redis, there will be a time when the configured memory will run out. When the memory runs out, doesn't it mean that there will be no memory available if you continue to add data to Redis?

In fact, Redis defines several strategies to deal with this situation:

One is written in the configuration file, :

The other is to use commands.

As mentioned above, the maximum memory available for Redis is used up, and the LRU algorithm can be used to eliminate the memory, so what is the LRU algorithm?

Approximate LRU algorithm

Redis uses an approximate LRU algorithm, which is not quite the same as the regular LRU algorithm. The approximate LRU algorithm uses the random sampling method to eliminate data, randomly selecting 5 (default) keys each time, and eliminating the least recently used keys.

In order to implement the approximate LRU algorithm, Redis adds an additional 24-bit field to each key to store the last time the key was accessed.

Optimization of approximate LRUs

Some optimizations have been made to the approximate LRU algorithm. The new algorithm maintains a candidate pool (size 16), the data in the pool is sorted according to the access time, the first randomly selected key will be put into the pool, and each subsequent random key will be put into the pool only if the access time is less than the minimum time in the pool, until the candidate pool is full. When it is full, if there is a new key that needs to be added, the last access time (the most recent access) in the pool will be removed.

When you need to eliminate the key, you can directly select the key with the least recent access time (the longest time not to be accessed) from the pool and eliminate it.

The LFU algorithm is a new elimination strategy. Its full name is least frequently used, and its core idea is to eliminate the most recently visited frequency of the key, and the rarely visited ones are eliminated first, and the most visited ones are retained.

The LFU algorithm can better represent the popularity of a key being accessed. If you are using the LRU algorithm, and a key has not been accessed for a long time, but only once in a while, then it is considered hot data and will not be eliminated, and some keys that are likely to be accessed in the future will be eliminated. This is not the case if the LFU algorithm is used, because using it once does not make a key hotspot.

There are two strategies for LFU: