Pipes vs Xargs: Which One To Use When Writing Bash Scripts In Linux

Are you using Linux to manage your servers?

Do you find yourself frequently copying the output on the terminal and pasting it into different bash commands?

If you answered “Yes”, then you should definitely start using the Pipes and xargs command in Linux. 

These are extremely helpful Linux utilities that will save you time and effort when working with the command line. Using these opens up so many possibilities for manipulating and processing data in the command line. 

If you want to perform complex actions in the Linux terminal and grow your Linux skills past the basic level, you will definitely need to learn how to use these commands.

In this article, you will learn exactly what xargs is, how it is different from pipes, and when to use which tool.

Let’s get started!

What Are Pipes in Linux?

Pipes are a way of connecting the standard output of one command to the standard input of another command in Linux. This allows you to chain commands together and process data in a stream.

To understand pipes, imagine two people talking on a ‘tin can telephone’. Here the first person is passing some information and the second person can only hear the information being passed. In this example, the telephone is acting as a pipe and the second person cannot jump to the end of the conversation without listening to the entire conversation first.

Similarly, pipes allow you to pass along information between processes and help you manipulate and process data in various ways. The syntax of pipes in pretty simple, just put a | sign between two commands.

first | second

In the above example, the output of the first will be used as the input for the second. This is pretty useful when working in the CLI.

For example, if you want to count the number of files and directories in the current directory, you can use a pipe like this:

ls | wc -l

This command will list all files and directories in the current directory, and then pipe the output to the wc command, which will count the number of lines. The pipe character | is used to redirect the output of command to the other. The command on the left side of the pipe sends its output to the command on the right side of the pipe, which receives it as its input.

Pipes are useful for performing operations on multiple items of data without having to write a loop or a script. You can also use multiple pipes to create a pipeline of commands, such as:

find . -name "*.txt" | grep "hello" | wc -l

The above command will find all files in the current directory and its subdirectories that have the “.txt” extension, and then pass them to the grep command, which will filter only the files that contain the word “hello”. The output of the grep command will then be piped to the wc command, which will count the number of lines. The result of this command is the number of files that have the “.txt” extension and the word “hello” in them.

What is the xargs Command?

The xargs utility in Linux reads data from standard input and converts it into command line arguments for another command. This allows you to perform operations on large amounts of data without having to write a loop or a script.

For most part, this is pretty similar to the pipe operator that we discussed above, but with one major difference. The pipe operator redirects its output to the standard input of the second command, whereas the xargs utility will create a list of all the data and then pass it as command line arguments to the second command.

To understand this, imagine the same person from above writing the information on a piece of paper and then passing it along to the second person. In this example, the paper is the xargs utility and the second person can see all the information at once.

Having the ability to pass along information via the command line arguments opens up many possibilities. For example, if you want to delete all files that have the word “temp” in their name, you can use xargs like this:

find . -name "*temp*" | xargs rm

This command will find all files in the current directory and its subdirectories that have “temp” in their name, and then pass them as arguments to the rm command, which will delete them.

Xargs or Pipes: Which One to Use?

You may have already spotted that these two commands are basically the same apart from one key detail – pipes are useful for processing data in a stream.

The main difference between pipes and xargs is that pipes pass data as standard input, while xargs passes data as command line arguments.

This means that pipes will only work with commands that accept standard input, while xargs can only work with commands that accept command line arguments.

What Are Standard Input and Command Line Arguments?

Standard input and command line arguments are two different ways of passing data to a program in Linux.

• Standard input is a stream of data that the program reads from its standard input file descriptor, usually the keyboard or a pipe.
• Command line arguments are strings of text that the program receives as parameters when it is invoked from the shell.

For example, if you run the command ls -l /home, the program ls will receive two command line arguments: -l and /home. The program can access these arguments using the argc and argv variables in C, or equivalent mechanisms in other languages.

Although many software tools accept both command line arguments and standard input, not all do. As a software user, we often do not have the option to change how the program processes input. 

For example, the command echo will print whatever you type after it, but it will not read from standard input. If you try to pipe the output of ls to echo, you will get nothing:

ls | echo

This is where xargs comes in handy – xargs will take the data from standard input and convert it into arguments for the command you specify. For example, if you want to print the names of all files and directories in the current directory, you can use xargs with echo like this:

ls | xargs echo

This command will list all files and directories in the current directory, and then pass them as arguments to the echo command, which will print them.

The choice of using pipes or xargs depends on the situation and the commands you want to use. In general, pipes are more efficient and elegant, as they avoid creating intermediate files or processes.

However, xargs can be more flexible and powerful, as it allows you to customize the arguments and options for the command you want to execute. It also has many options that allow you to handle different scenarios, such as limiting the number of arguments, replacing placeholders, prompting the user, or dealing with filenames that contain spaces or special characters. Let’s see some of these use cases in the next section.

How to Use xargs like a Pro in Linux

xargs has many options that allow you to customize its behavior and handle different situations. Here are some of the most common and useful ones:

Limit the Number of Arguments

The -n option specifies the maximum number of arguments to pass to each invocation of the command. For example, if you want to print the names of all files and directories in the current directory in groups of three, you can use xargs with -n 3 like this:

ls | xargs -n 3 echo

This command will list all files and directories in the current directory, and then pass them as arguments to the echo command in groups of three. Then the echo command will be executed multiple times, once for each group of three arguments that it received.

Replace Placeholder Text

When working in the command line, you will often encounter cases where you need to run many simple commands, each with a tiny change. The -I option allows you to specify a placeholder that will be replaced by each argument in the command.

For example, if you want to create many files in your server which follow a common pattern, then you can use xargs with -I {} like this:

seq 12 | xargs -I {} touch runcloud_{}.txt

The above command will create a sequence of twelve numbers, then use those numbers to create twelve files that follow the pattern runcloud_#.txt.

Prompt User For Confirmation

When generating commands on the fly, it is pretty easy to do irreversible damage to your server. The smart way to use xargs is by letting it generate all the commands and then manually confirming it before it can be executed.

To do this, use the -p flag which prompts the user before executing each command and waits for confirmation. For example, if you want to delete all files that start with the word “runcloud”, but you want to confirm each deletion in batches of four, you can use xargs with -p like this:

find . -name "runcloud*" | xargs -p -n 4 rm

This command will find all files in the current directory and its subdirectories that have “runcloud” in their name, and then pass them as arguments to the rm command, which will prompt the user before deleting each file and wait for a “yes” or “no” answer.

In the above example, we can see that first we created twelve files. Then we used the find function to find any files that match our criteria and then passed them along in batches of four. The resulting rm command is displayed in the terminal and we can press y or n to confirm or reject the action.

After the command has been executed, we can see that the files in the second batch of deletion remain untouched because we pressed n when we were asked to confirm the deletion, whereas all the other files that matched the criteria were removed.

Process Files with Weird Names

The xargs utility is not secure, because it can execute arbitrary commands if the input contains malicious or unexpected data. For example, if the input contains spaces, quotes, or other special characters, xargs may interpret them as part of the command or the arguments, and cause unwanted or harmful effects. 

If you’re working with data that contains spaces, special characters, or even foreign language text, then it is always a good idea to terminate the output of a line using null characters to avoid unexpected output.

The -0 or –null flag expects the input to be null-terminated, meaning that each argument is separated by a null character instead of a space or a newline. For example, if you want to delete all files that have the word “temp” in their name, but some of them have spaces in their name, you can use xargs with -0 like this:

find . -name "runcloud*" -print0 | xargs -0 rm

This command will find all files in the current directory and its subdirectories that start with “runcloud” and then print them with a null character after each name. xargs will then read the input as null-terminated and pass them as arguments to the rm command, which will delete them.

In the above example, we can see that terminating the output with a null character allows the program to gracefully execute the given command. If we had omitted the null termination step, we would have encountered different errors.

After Action Report

Both pipe and xargs are powerful and versatile utilities that can help you perform different operations on data in the command line. There is a character limit to xargs, but it depends on the system and the options used.

By default, it tries to pass as many arguments as possible to the command, without exceeding the maximum length of the command line. To see the limits of xargs on your system, you can use the --show-limits option.

Normally, xargs runs the second command once, even if there is no input (output from the first command). This option is useful to avoid errors or unwanted actions when the input is empty. To suppress this, you can specify --no-run-if-empty option.

These are just some of the options that xargs offers. You can find more information and examples by typing man xargs or xargs --help in the command line.

It is clear that xargs is a must-learn tool for any Linux user who wants to go beyond the basics and explore the possibilities of the command line. 

However, if you don’t want to become a master in Linux command line utilities, but still want to manage your Linux server yourself, then you should check out RunCloud.

RunCloud is a cloud-based server management platform that lets you deploy, configure, and monitor your web applications with ease. If you’re an advanced user, you can take advantage of the CLI  and API, as RunCloud gives you full control over your server. 

If you’re not comfortable with using the CLI or the API, don’t worry. RunCloud also offers an easy to navigate dashboard that can deploy applications in a breeze. You can also use RunCloud to set up SSL certificates, domains, cron jobs, firewalls, and more.

RunCloud is a versatile and reliable tool that can help you manage your server on your own, without the hassle and cost of hiring a system administrator. Don’t miss this opportunity to take your server management to the next level. Sign up for RunCloud today and enjoy the benefits of a cloud-based server management platform that is fast, secure, and easy to use.