The Nginx event module

Event module

The type and functionality of event

Nginx is a module based on the event processing model. I t abstracts the event module to support cross-platform. The types of event processing it supports are: AIO (Asynchronous IO), /dev/poll (Solaris and Unix- specific), epoll (Linux-specific), eventport (Solaris 10-specific), kqueue (BSD-specific), poll, rtsig (real-time signal), select, etc.

The main function of the event module is to listen for connections established after accept and add and remove read and write events. T he event processing model is used in conjunction with Nginx's non-blocking IO model. When the IO is readable and writeable, the corresponding read and write event is awakened, and the callback function handles the event.

Especially for Linux, Nginx most events use the epoll EPOLLET (edge trigger) method to trigger events, and only the listen port read event is EPOLLLT (horizontal trigger). For edge triggers, if a readable event occurs, it must be handled in a timely manner, otherwise a read event may no longer fire and the connection will starve to death.

typedef struct {
        /* 添加删除事件 */
        ngx_int_t  (*add)(ngx_event_t *ev, ngx_int_t event, ngx_uint_t flags);
        ngx_int_t  (*del)(ngx_event_t *ev, ngx_int_t event, ngx_uint_t flags);

        ngx_int_t  (*enable)(ngx_event_t *ev, ngx_int_t event, ngx_uint_t flags);
        ngx_int_t  (*disable)(ngx_event_t *ev, ngx_int_t event, ngx_uint_t flags);

        /* 添加删除连接，会同时监听读写事件 */
        ngx_int_t  (*add_conn)(ngx_connection_t *c);
        ngx_int_t  (*del_conn)(ngx_connection_t *c, ngx_uint_t flags);

        ngx_int_t  (*process_changes)(ngx_cycle_t *cycle, ngx_uint_t nowait);
        /* 处理事件的函数 */
        ngx_int_t  (*process_events)(ngx_cycle_t *cycle, ngx_msec_t timer,
                                   ngx_uint_t flags);

        ngx_int_t  (*init)(ngx_cycle_t *cycle, ngx_msec_t timer);
        void       (*done)(ngx_cycle_t *cycle);
} ngx_event_actions_t;

These are the key structures abstracted by event processing, and you can see that each event processing model requires some functionality. The most critical is the add and del functions, which are the most basic functions for adding and deleting events.

Accept lock

Nginx is a multi-process program, 80 ports are shared by processes, multi-process simultaneous listen 80 ports, there is bound to be competition, but also the so-called "surprise" effect. W hen the kernel accepts a connection, it wakes up all the waiting processes, but in fact only one process gets the connection, and the other processes are invalidly awakened. S o Nginx uses its own set of accept-locking mechanisms to prevent multiple processes from calling accept at the same time. N ginx multi-process locks are implemented at the bottom by default through CPU spin locks. If the operating system does not support spin locks, file locks are used.

The entry function for Nginx event processing is ngx_process_events_and_timers(), and here's part of the code that you can see about its locking process:

if (ngx_use_accept_mutex) {
        if (ngx_accept_disabled > 0) {
                ngx_accept_disabled--;

        } else {
                if (ngx_trylock_accept_mutex(cycle) == NGX_ERROR) {
                        return;
                }

                if (ngx_accept_mutex_held) {
                        flags |= NGX_POST_EVENTS;

                } else {
                        if (timer == NGX_TIMER_INFINITE
                                || timer > ngx_accept_mutex_delay)
                        {
                                timer = ngx_accept_mutex_delay;
                        }
                }
        }
}

In ngx_trylock_accept_mutex() function, if the lock is obtained, Nginx adds listen's port read event to event processing, which can be accepted when a new connection comes in. N ote that the accept operation is a normal read event. The following code illustrates this point:

(void) ngx_process_events(cycle, timer, flags);

if (ngx_posted_accept_events) {
        ngx_event_process_posted(cycle, &ngx_posted_accept_events);
}

if (ngx_accept_mutex_held) {
        ngx_shmtx_unlock(&ngx_accept_mutex);
}

ngx_process_events () function is the entry point for all event processing, and it traverses all events. T he process that grabs the accept lock is slightly different from the normal process in that it is marked with NGX_POST_EVENTS, which means that only events are accepted and not processed within the ngx_process_events() function and added to the post_events queue. S pecific ngx_accept_mutex not processed until the lock is removed. W hy is that? Because ngx_accept_mutex is a global lock, this minimizes the time from the beginning to the end of the process after the process grabs the lock, so that other processes continue to receive new connections and increase throughput.

ngx_posted_accept_events and ngx_posted_events are accept deferred event queues and normal deferred event queues, respectively. Y ou can ngx_posted_accept_events or put ngx_accept_mutex in the lock. The queue handles the accept event, which brings in all the connections waiting in the kernel backlog in one go and registers them for the read-write event.

The ngx_posted_events is a normal delay event queue. I n general, what kind of events are placed in this normal delay queue? M y understanding is that those CPUs that take a lot of time can be put in. Because Nginx event processing is handled in turn in a large loop according to the trigger order, because Nginx can only handle one event at a time, some time-consuming events delay the processing of all future events.

In addition to locking, Nginx also optimizes the balance of request processing for each process, which means that if the process grabs too many locks at high load, the process is prohibited from accepting requests for a period of time.

For example, in ngx_event_accept function, there is a similar code:

ngx_accept_disabled = ngx_cycle->connection_n / 8
              - ngx_cycle->free_connection_n;

ngx_cycle-gt;connection_n is the total number of connections that a process can allocate, and ngx_cycle-gt;free_connection_n is the number of idle processes. The above example shows that if the number of idle processes for the current process is less than 1/8, it will be blocked from accepting for some time.

Timer

Nginx uses a timer mechanism when timeouts are required. F or example, those read and write timeouts after the connection is established. Nginx uses a red-black tree to construct a recurring device, an orderly binary balanced tree with the complexity of finding insertions and deletions as O (logn), so it is an ideal binary tree.

The mechanism of the timer is that the value of the binary tree is its time-out, and each time the minimum value of the binary tree is found, if the minimum value has expired, the node is deleted, and then continues to look until all time-out nodes are deleted.