We had an odd situation where disk space was getting swallowed up on a Digital Ocean server running Laravel. The issue started happening after an upgrade from Laravel 7 to Laravel 8 (though it could have been there before unnoticed). The database size had been growing, so my first though was just to resize the droplet. But when an additional 100GB was eaten up over the weekend I realized there must be something else going on.

It turned out we had deleted log files that were still being kept open by a system process, and therefore could not be fully deleted, but were also continuing to grow:

We found these with the command:

lsof +L1

Which showed:

COMMAND  PID  USER   FD   TYPE DEVICE  SIZE/OFF NLINK    NODE NAME
php7.4  1543 forge   15w   REG  252,1 44241528600     0 1809826 /home/forge/data.example.com/storage/logs/laravel.log.1 (deleted)
php7.4  1544 forge   15w   REG  252,1 44242240400     0 1809826 /home/forge/data.example.com/storage/logs/laravel.log.1 (deleted)
php7.4  1545 forge   15w   REG  252,1 44242240400    0 1809826 /home/forge/data.example.com/storage/logs/laravel.log.1 (deleted)

This StackExchange post has a little more information about tracing down the issue, and this one has a more technical explanation for why it happens.

To resolve the issue, we truncated the deleted file using the method described here, which did free up the space:

:>/proc/1543/fd/15

Then we killed all the processes:

forge@data-service:~/data.example.com$ kill -SIGTERM 1543
forge@data-service:~/data.example.com$ kill -SIGTERM 1544
forge@data-service:~/data.example.com$ kill -SIGTERM 1545

This appeared to resolve the issue for us.