Nginx daily logs

One of the reasonably important diagnostics tools in server maintenance are access logs. Now depending on which server you use the logs will be slightly different but you'll have them.

When it comes to Nginx the easiest way to store logs to a specific place is by adding the access_log and error_log directives to the Nginx config file:

server {
    # rest of the config 
	
    # specify the logging paths
    access_log /home/ubuntu/logs/nginx/access.log;
    error_log /home/ubuntu/logs/nginx/errors.log;
}

Side note: Above I use absolute paths. You can use relative paths as far as I know, but the moment you move the position of your file, you break all the paths. My personal preference is to always use absolute paths. I shoot my self less often. If you are more masochistic use relative ones.

The issue with this approach is that the access.log and error.log file will grow with time and balloon in size. But the truth is any log older than a day is entirely useless on the production server and is taking only unnecessary space.

So the idea is: How do we split Nginx logs into daily files?

We want a file structure of the form:

~/logs/
    nginx/
        access/
            2020-05-10.log
            2020-05-11.log
            2020-05-12.log
            ...
        errors/
            2020-05-10.log
            2020-05-11.log
            2020-05-12.log
            ...

Make sure to create the ~/logs/nginx/access and ~/logs/nginx/errors folders.

Daily log files

We could use the option of parameters in Nginx and write to a new log file for each day. Right but error_log directive doesn't allow parameters in the log path.

Nginx developers decided it's of the significant importance that the path to error logs should always exist so once the Nginx configuration is running, we can't change that path. Well ...

So the only way for us to convince Nginx to split logs into separate days is to "rewrite" the Nginx config file each day with a cron job. What could possibly go wrong in this process 🙂

Now I'm not going to argue that this is the cleanest approach, but it's the best I've got till now. This solution is far from easy and requires a bit of understanding of how Linux works in the guts.

At the end of this guide we'll have added something like this to our server:

/etc/
    nginx/
        server.conf
        log_by_date.inc
    systemd/
        system/
            nginx_log_update.service
            nginx_log_update.timer
/home/
    ubuntu/
        nginx_log_update.sh
        logs/
            nginx/
                ...

Nginx config file

First, let's rewrite the Nginx config file by adding one include and removing the access_log and error_log directives:

server {
    # rest of the config 

    # include the file with daily log paths
    include log_by_date.inc;

}

Then for each day, we want the log_by_date.inc look like this:

ccess_log /home/ubuntu/logs/nginx/access/2020-10-10.log;
error_log /home/ubuntu/logs/nginx/error/2020-10-10.log;

The log_by_date.inc has to be in the same folder as nginx config file. So in my case at /etc/nginx/.

We'll use bash to update log_by_date.inc daily and run it with the Linux service manager.

Daily path update

Now we won't use cron to overwrite the log_by_date.inc file but the systemd service manager since it's a bit more robust and we use it to run Nginx service anyway. If you don't have an Ubuntu server, I assume you know which service manager you use. Otherwise Google 😉

We'll need to create three scripts:

• nginx_log_update.sh actual bash script that will rewrite the logging configuration log_by_date.inc
• nginx_log_update.service service that will run the bash script nginx_log_update.sh
• and nginx_log_update.timer the timer that will run the service nginx_log_update.service each day.

Yes, I know its a mess, but we need three separate scripts for this.

bash script

First, we want the date with year-month-day format. We get this with:

date=$(date -Id)

Create a variable for the location of the log files:

home_folder="/home/ubuntu"
nginx_logs="$home_folder/logs/nginx"

Now we need to regenerate the config file. First, we create the file in a home folder since we can't update it in /etc directly using echo even with sudo privileges.

Create a new file by using operator >:

echo "access_log $nginx_logs/access/${date}.log;" > $home_folder/log_by_date.inc

now append to the file with operator >>:

echo "error_log $nginx_logs/errors/${date}.log;" > $home_folder/log_by_date.inc

copy the new daily log file over to /etc:

sudo cp $home_folder/log_by_date.inc /etc/nginx

restart Nginx to start using the new log files:

sudo systemctl restart nginx

and the full script:

#! /bin/bash

date=$(date -Id)

home_folder="/home/ubuntu"
nginx_logs="$home_folder/logs/nginx"

echo "access_log $nginx_logs/access/${date}.log;" > $home_folder/log_by_date.inc
echo "error_log $nginx_logs/errors/${date}.log;" > $home_folder/log_by_date.inc

sudo cp $home_folder/log_by_date.inc /etc/nginx

sudo systemctl restart nginx

Place the bash script into home_folder and give the script proper permissions 744 (read, write, execute):

sudo chmod 744 /home/ubuntu/nginx_log_update.sh

Now run the script for the first time to actually generate the log files specification (we need it before we can start Nginx):

source /home/ubuntu/nginx_log_update.sh

Now test that the new Nginx config is actually valid:

sudo nginx -t

and enable the Nginx if you didn't before and restart it:

sudo systemctl enable nginx
sudo systemctl restart nginx

Alright, we have the bash part done. Now we need to create a systemd service that will actually run the bash script.

service

This one is actually pretty easy to create a file named nginx_log_update.service with the following content:

[Unit]
Description=Execute Nginx log update every day at midnight.

[Service]
ExecStart=/home/ubuntu/nginx_log_update.sh

[Unit] part is to tell the systemd what's going on. OK, actually for us to know what's going on once we check the logs.

The [Service] part and ExecStart= tell the systemd which script to run when it starts the "service".

Copy the nginx_log_update.service file over to /etc/systemd/service and set 644 permisions (owner can read, write, but can't execute):

sudo chmod 644 /etc/systemd/system/nginx_log_update.service

You don't need to start this service the timer will do that for you. Let's write it.

service timer

Basically, we need to add the [Timer] section:

[Unit]
Description=Execute Nginx log update every day at midnight.

[Timer]
OnBootSec=0min
OnCalendar=*-*-* 00:05:00
Unit=nginx_log_update.service

[Install]
WantedBy=multi-user.target

The OnCalendar=--* 00:05:00 tells the systemd to run the service every day five minutes after midnight. The Unit=nginx_log_update.service tells the timer which service to run "at midnight".

Now start the timer:

sudo systemctl enable nginx_log_update.timer
sudo systemctl start nginx_log_update.timer

That's it.

Write permissions

Now depending on where you store your logs on the server, you might have to specify the correct permissions. On AWS Ubuntu server you should be good to go if you store logs directly on the instance disk. If you're storing to the separate storage to keep the instance clean be a grown-up and figure it out 😛

Recap

That's it. We now have a daily log file for Nginx access and error requests.

If you have the home folder at /home/ubuntu you should have the following folder structure created:

/etc/
    nginx/
        server.conf
        log_by_date.inc
    systemd/
        system/
            nginx_log_update.service
            nginx_log_update.timer
/home/
    ubuntu/
        nginx_log_update.sh
        logs/
            nginx/
                ...

Plus of course the rest of your server configuration.