Fuser Detects FTP Completion

At work, we have legacy systems with problems which no one had taken the time to fix. One such legacy problem involved an FTP server. Client applications would FTP files up to the server for processing. That part worked fine. What didn’t work was knowing when the FTP was complete so we could start processing the data.

Recently, I decided to fix this problem.

Many people have written on the subject. One of the approaches advised, “watching the file and when the file size stops changing, you can use it.” I didn’t like that one. For so many reasons. Another recommended using lsof. Hummmmmmm. I didn’t get a warm fuzzy feeling with that one either.

A coworker suggested I try fuser. That did the trick as fuser allowed me to monitor an incoming file and determine when it was no longer being used by the ftp (or any other) process.

#!/bin/bash
DELAY="10"

while true; do
  for FILENAME in `ls -A1 /var/ftp/incoming/*.gz 2>/dev/null`; do
    if [[ `fuser $FILENAME | wc -c` -eq 0 ]]
      then

        # do something interesting with the data here
        mv  $FILENAME ./backup/.

    fi
  done
  sleep $DELAY
done

All our incoming FTP files arrive in a single directory. The above script loops through the list of files once every 10 seconds. Normally, ls complains when you ask for a file listing and there are no files to list. The 2>/dev/null code fragment will send the complaint quietly to the bit bucket.

For each file found, fuser lists all processes that are using the file. I simply count the number of characters, wc -c, in the response from fuser. If the file is not being used by any processes, fuser returns nothing and the character count is zero. At that point, I can safely process the file.

update 2008-12-25: I should have added that this server runs Centos 5.1. I did the development on OS X (10.5.5) and that fuser behaves a little differently on the two platforms.

Red Leopard