Dealing with filenames that contain spaces or special characters is something everyone who uses Linux has to learn. They pose special problems and need to be dealt with in different ways. In this Linux quick tip we will discuss using a for loop on a file with spaces in the lines.

Before we go into the fix, let's try to understand the problem.

For this experiment, let's assume we have the following text file.

This line has spaces
these
do
not

Now we want to use a for loop to iterate through this file and use each line as input for some action. When we create our for loop, the shell sees the spaces on the first line and assumes each is a different iteration.

$ for i in $(cat test); do echo $i; done  This  line  has  spaces  these  do  not

The IFS (internal field separator) controls this behavior. The shell uses the IFS for word splitting. It defines the character (or character set) used as a delimiter when splitting words, or splitting lines into words.

For many command line interpreters (“shell”) of Unix operating systems, the internal field separator (abbreviated IFS) refers to a variable which defines the character or characters used to separate a pattern into tokens for some operations

-Wikipedia

By default the IFS (field separator) is set to <space><tab><newline>. So when then the shell sees the spaces in the first line, it divides the line into four tokens (four words).

Changing IFS for Our Needs

IFS is an environmental variable. Before changing environmental variables it is best practices to save their content. This allows them to be easily set back to their default value.

Let's start by saving the IFS variable to OLDIFS.

OLDIFS=$IFS

We can manually set IFS to whatever we want. In this case we need the field separator to be a newline. We can set IFS just like we would any other variable.

IFS=<our value>

To set IFS to a newline, we can use the command substitution to grab a newline output from the echo command.

IFS=`echo -e "\n"`

The output of echo -e "\n" is a newline. Wrapping it in back ticks tells the shell to use the output of the command as the variable IFS.

Now when we do the same for loop, the shell looks for a newline to separate the text.

$ for i in $(cat test); do echo $i; done
 This line has spaces
 these
 do
 not

Now we can easily set the IFS back to default like so:

IFS=$OLDIFS

Conclusion

Now you can use the internal field separator to control how words are separated. This is just one simple use case for setting a custom IFS.

Resources and Links