As a continuation of our series of Bash tutorials I thought it would be good to discuss lock files. The most common use of lock files is to ensure a second instance of a script does not run until the first is completed. Anytime you have multiple processes working with shared resources, using lock files will help avoid conflicts and race conditions.
It is important to point out that a lot of programming languages use lock files. For this tutorial we will be discussing their use for shell scripting. However, we believe that this information will be easily transferable to other languages.
Here is a real world example of how lock files could be used. A colleague of mine wrote a script that uses rsync over ssh to synchronize his workstation home directory to a shared directory on a server. He set the cron job to run this script every hour. This was fine for the first month or so, but one day he was copying a bunch of logs and several large ISO files to his home directory. This influx of data caused the rsync script to run longer than one hour. Before it could complete cron kicked off a second instance of the script. This slowed down the copy job which then caused a third instance to kick off, then a fourth, and so on. Before he knew it his workstation was unusable. Using a lock file could have help avoid this scenario.
In this tutorial we will discuss the basics of using lock files. We will discuss creating a basic lock files as well as using the flock utility to manage lock files and queuing.
Table of Contents
Creating a Basic Lock File
A basic lock file is when you create a file to be used as a lock. The existence of the file will be a signal to any subsequent instance of the script not to run. When the script exits, you delete the file. The next instance of the script will check for the existence of the file to determine if it is safe to execute.
Here is some example code:
#!/bin/bash
# Check is Lock File exists, if not create it and set trap on exit
if { set -C; 2>/dev/null >~/manlocktest.lock; }; then
trap "rm -f ~/manlocktest.lock" EXIT
else
echo "Lock file exists… exiting"
exit
fi
# Do Something, Main script work here…
echo "I am a script and I am doing something… anything…"
sleep 30
Thanks to David for the updated code.
In the above example, we are using a simple if statement to check if a file exists. If the file exist, the script exits (cannot get lock). If the file does NOT exist, it creates the file and a trap to delete the file on exit then continues.
Here is a our script in action. I will kick off the script and then send it to the background. I will then attempt to run the script again, at which point it should exit because the lock file exists.
There you have a simple lock. If the script runs a second time and sees the file is created, it will exit.
NOTE: In my opinion it is important to use a trap to delete the file instead of using a simple "rm <filename>" at the end of the script. We spoke in depth about the advantages of using traps in "Using Trap to Exit Bash Scripts Cleanly".
Using flock to Create Lock Files
Flock is the name of both a kernel level system call and a command line utility. The latter is simply a way to manage the systems calls from shell scripts or the command line.
The flock command works on file descriptors, not files. A file descriptor is a unique number used to access a data resource. The most popular file descriptors are 1 (standard input), 2 (standard output) and 3 (standard error). Which were described in "Introduction to Linux IO, Standard Streams, and Redirection".
In Unix and related computer operating systems, a file descriptor (FD, less frequently fildes) is an abstract indicator (handle) used to access a file or other input/output resource, such as a pipe or network socket. File descriptors form part of the POSIX application programming interface.
- WikiPedia
Using flock to Queue Jobs
One of the major advantages of using flock over a simple lock file is the ability to queue, or wait for a lock. By default flock will wait indefinitely to get a lock unless you use the -n (nonblock) option.
Here is the code we will use as our test script.
#!/bin/bash
exec 100>/var/tmp/testlock.lock || exit 1
flock 100 || exit 1
echo "Doing some stuff…"
echo "Sleeping for 30 seconds…"
sleep 30
Let's take a deeper look at the two important lines that make up the lock.
exec 100>/var/tmp/testlock.lock || exit 1
This redirects the file descriptor 100 to the desired lock file. Using exec ensures the redirect is available for the life of the shell. The "|| exit 1" instructs the script to exit if the exec command fails.
flock 100 || exit 1
The above line obtains a lock on file descriptor 100. Basically holding a lock on the file until the shell closes. Since scripts run in a sub-shell, the file will be closed (or unlocked) when the script exits. The "|| exit 1" instructs the script to exit if a lock cannot be obtained.
That's it, we just created a file lock using flock. It's pretty straight forward once you wrap your head around it.
Using Wait or Timeout for Queued Jobs
You also have the option of waiting x number of seconds to acquire a lock before failing.
Let's take this example.
#!/bin/bash
exec 100>/var/tmp/testlock.lock || exit 1
flock -w 10 100 || exit 1
echo "Doing some stuff…"
echo "Sleeping for 5 seconds…"
sleep 5
Here we are using the -w (wait) option for flock and giving it a value of 10. This instructed flock to fail if the lock cannot be acquired in 10 seconds. Since the script should only take 5-6 seconds to run, we should be able to acquire a lock before the timeout. Let's test it out.
As you can see in the example above, flock waited until the script released the lock. Then the second instance of the script acquired the lock and executed.
Timeouts are a good way to ensure you don't have a bunch of scripts waiting for some hung up resource to drop a lock.
Do Not Queue Jobs While Using flock
As we mentioned above, the default behavior of flock is to wait indefinitely to acquire a lock. We also discussed using the -w (timeout) option to set a time limit on how long it will wait. Conversely, you can use the -n (nonblock) option to instantly fail if it cannot immediately acquire a lock.
Example code:
#!/bin/bash
exec 100>/var/tmp/testlock.lock || exit 1
flock -n 100 || exit 1
echo "Doing some stuff…"
echo "Sleeping for 10 seconds…"
sleep 10
Below you will see that I background the first instance of the test.sh script. I then try to run a second instance and it fails instantly instead of waiting for a lock.
Additionally, the -E (conflict-exit-code) can be set to use a custom exit code. This option is only used with the nonblock option or the timeout option, since flock will otherwise wait indefinitely for a lock.
Exclusive Lock vs Shared Lock
There are two kinds of locks, exclusive (AKA write lock) and shared (AKA read lock).
- If an exclusive lock is taken, no other process can take a lock, exclusive or shared, on that file.
- If a shared lock is taken, other processes may take a shared lock on the same file, but no processes may take an exclusive lock.
The flock utility in Linux will take an exclusive lock by default. You must specify your desire for shared lock by using the -s (shared) option.
Cleaning Up flock Lock Files
Deleting the lock files created when using flock is not necessary since flock doesn't use the existence of the file as an indicator. However it is probably a good idea to keep a file system tidy.
You can use a trap to delete lock files, just as we did with the basic lock file.
#!/bin/bash
exec 100>/var/tmp/testlock.lock || exit 1
flock -n 100 || exit 1
trap 'rm -f /var/tmp/testlock.lock' EXIT
# Main script actions here
Conclusion
In this article we touched on all the basics of creating a lock file and how to use them in shell scripts. We discussed creating a basic lock file and locking files with flock. This should be enough information to get you started effectively using lock files. I will link to the pertinent man pages and some good bash scripting resources below for additional information.
If you enjoyed this article please consider showing your support by following us on Twitter, Facebook and subscribing to our Newsletter.
Resources
Leave a Reply Cancel reply
This site uses Akismet to reduce spam. Learn how your comment data is processed.
9 Comments
Join Our Newsletter
Categories
- Bash Scripting (17)
- Basic Commands (51)
- Featured (7)
- Just for Fun (5)
- Linux Quick Tips (98)
- Linux Tutorials (65)
- Miscellaneous (15)
- Network Tools (6)
- Reviews (2)
- Security (32)
After the first example I was expecting a sentence like: "This is the naive approach, how you should never do it!", but it never came. The first thing one should learn about locks, is that they have to be checked and created at the same time in one atomic operation. Everything else is asking for RCs at the most inconvenience of times. In bash it's a common theme to use the nonclobber (set -C) option for this purpose, which ensures a redirect target is only created when file doesn't exist yet. So your example can be rewritten in an RC free way like this:
if { set -C; 2>/dev/null >manlocktest.lock; }; then
trap "rm -f ~/manlocktest.lock" EXIT
else
echo "Lock file exists… exiting"
exit
fi
I've no idea how this gets formatted here, but I hope the idea is clear.
The solution with flock is interesting and surely more elegant. But the question is how portable is it? It seems it's pretty Linux specific, so this is more about Linux than about bash scripting. So my concern is many people will stick with the first example and another generation of badly written scripts will be the result.
I tried all the different code examples on this page and they all work. your comment is interesting but I cant get the code to work.
./test.sh: line 2: trap: -f: invalid signal specification
./test.sh: line 2: trap: /home/jbaran/manlocktest.lock”: invalid signal specification
./test.sh: line 1: “rm: command not found
David has a typo in his original code, the first line redirect should be >~/manlocktest.lock instead of >manlocktest.lock
The "Post Comment" functionality converts quotes like " into smart (Unicode) quotes. You have to replace them with the original ASCII quotes.
David, thanks for your input. I have updated the article.
Also, you are correct, flock lacks portability.
Thanks, I appreciate it! But you also missed replacing two smart quotes. While it works, since they're harmless for an echo, they will be printed literally, which is surely not what's intended.
Good catch, corrected.
thanks
The following command is not a good practice:
exec 100>filename
It is better to let bash chose a free filedescriptor and then pass it to flock:
exec {fd}>filename
flock $fd