Awasu » Deleting large files from an ext3 file system
Sunday 25th October 2015 2:43 AM [Tutorial]

Just a quick follow up on my recent epic tome on setting up a Banana Pi as a file server. I mentioned that I configured my disks to use the ext3 file system, and while it's generally fine, it does have one weakness: it is very slow deleting large files[1]Since I use my NAS for back ups, some of my files are well over 100GB.. Even worse, it locks up the file system, impacting other activity and causing stalls, which kinda sucks if you're watching a movie at the time 🙁

depesz took a very detailed look at the problem and some possible solutions, the TL;DR being that it's better to progressively shrink the file until it's all gone rather than asking the operating system to delete it as a file.

For the benefit of anyone having problems with this, here's a script that I wrote that implements this idea:

#!/bin/bash

# parse the command-line arguments
if [ $# -lt 1 ] ; then
    echo "$(basename $0) file1 file2 file3 ..."
    echo "  Delete file(s) slowly."
    exit 1
fi

chunk_size=100000000
for fname in "$@"; do
    # check if we were given a directory
    if [ -d "$fname" ]; then
        # yup - process each file, then remove the directory
        #echo "Deleting directory: $fname"
        find "$fname" -type f -exec $(readlink -e "$0") \{\} \;
        rm -rf "$fname" 
        continue
    fi
    # check if we were given a file
    if [ ! -f "$fname" ]; then
        # nope - ignore it
        continue ;
    fi
    # yup - delete the file slowly
    #echo "Deleting file: $fname"
    while true; do
        # get the current size of the file
        fsize=$(ls -l "$fname" | cut -d' ' -f 5)    
        if [ $fsize -lt $chunk_size ] ; then
            # the file is small enough to just delete
            #echo "- Deleting file."
            rm -f "$fname"
            break
        fi
        # truncate the file, then loop back
        #echo "- Truncating file: $fsize"
        truncate -c -s -$chunk_size "$fname" || exit
        sleep 0.25
    done
done

Pass in a list of files and/or directories and it will slow-delete them. It will take longer to run, but will have far less impact on the rest of the system.

Nothing to do with Awasu, but hopefully someone out there in Internet-land will find it useful... 🙂

   [ + ]

1. Since I use my NAS for back ups, some of my files are well over 100GB.
Have your say