Bash Scripting Tutorial: A Comprehensive Guide to Piping and Grep

Contents

1. Introduction
2. What are Pipes and Grep?
3. Combining Pipes and Grep
4. Go Beyond with Grep Flags
5. Piping: The Multi-Tool You Never Knew You Needed
6. Example Use Case: Finding Open Ports on a Host
7. Wrap Up

Introduction

Today, we're stepping up our game and venturing deeper into the command-line labyrinth. Today we're moving a little beyond the basics, and diving into two indispensable Linux tools: Piping (|) and Grep.

This is getting into more advanced territory, so if you haven't already, go check out the Linux Basics: How to Navigate the Terminal and What is Linux?

You're probably thinking "Piping? I thought this was about Linux, not plumbing! And what even is GREP?"

Well, I'm so glad you asked! Let's get into it!

What are Pipes and Grep?

Before we deep dive, let's get the definitions straight. Pipes (|) in Linux are used to chain multiple commands together. In simpler terms, they take the output of one command and use it as the input for another command.

On the other hand, 'grep' (Global Regular Expression Print) is a powerful utility used to search and filter text. Whether it's a single file or an entire directory, 'grep' is the Swiss Army Knife you need for your text processing tasks.

Combining Pipes and Grep

Now, let's look at the magic that happens when we combine these two powerhouses.

Let's assume you want to search for the word 'Linux' in a file named 'sample.txt'. With 'grep', it's as easy as:

grep 'Linux' sample.txt

But what if you want to search for 'Linux' in a multitude of files or, say, in the output of another command? Here's where piping comes in handy.

For instance, assume you have a directory and when you use the ls command, this is the output you get:

LinuxGuide.txt
WindowsGuide.txt
LinuxCommands.pdf
MacOSGuide.txt
LinuxTutorial.txt

Now, if you pipe (|) this list to grep 'Linux', it will filter out only the lines that contain the word 'Linux'. So the output of ls | grep 'Linux' would be:

LinuxGuide.txt
LinuxCommands.pdf
LinuxTutorial.txt

Here, ls lists all the files, and grep filters this list to display only those file names that contain 'Linux'.

Go Beyond with Grep Flags

Just like our friend ls, grep comes with a slew of flags to make your life easier.

For instance, the -i flag ignores case (so 'Linux', 'linux', and 'LINUX' are all the same), and -r allows you to search recursively in a directory.

Usage:

grep -i 'linux' sample.txt grep -r 'linux' directory/

Remember, the man command is always your friend when you want to explore more flags!

man grep

Piping: The Multi-Tool You Never Knew You Needed

The power of pipes isn't just limited to grep. You can combine it with a host of other commands to customise your output.

Ever needed to view the top five long-running processes in your system? With pipes and the ps and head commands, it's as easy as:

ps aux --sort=-%mem | head -n 5

This command lists all processes (ps aux), sorts them based on memory usage (--sort=-%mem), and finally uses head -n 5 to display the top five entries.

The possibilities with pipes are virtually limitless. They're an essential tool in your Linux arsenal, allowing you to build complex command sequences and manipulate data in many versatile ways.

Example Use Case: Finding Open Ports on a Host

Imagine your task was to find all the open ports running on an host.

A typical nmap scan output can look something like this

Starting Nmap 7.91 ( https://nmap.org ) at 2023-07-02 13:00 PDT
Nmap scan report for 192.168.1.1
Host is up (0.0011s latency).
Not shown: 995 closed ports
PORT     STATE SERVICE
21/tcp   open  ftp
22/tcp   open  ssh
80/tcp   open  http
443/tcp  open  https
3389/tcp open  ms-wbt-server

Nmap done: 1 IP address (1 host up) scanned in 1.42 seconds

You're objective is to make a list of all the open ports and save them in a text file.

You can run a command like this:

nmap -p- -oG - 192.168.1.1 | grep open | awk '{print $1, $2, $3, $5}' > nmap_output.txt

• nmap -p- -oG - 192.168.1.1: This command runs nmap on the target 192.168.1.1, scans all ports (-p-), and outputs the results in 'grepable' format (-oG -).
• grep open: This command filters the nmap output to only include lines with the word 'open'.
• awk '{print $1, $2, $3, $5}': This command extracts the first, second, third, and fifth columns from each line. These correspond to the hostname/IP, the port number, the state (open/closed/filtered), and the service running on the port, respectively.
• > nmap_output.txt: This command redirects the output into a file named nmap_output.txt.

When you cat the resulting text file, nmap_output.txt, you should see something like this:

192.168.1.1 21/open tcp ftp
192.168.1.1 22/open tcp ssh
192.168.1.1 80/open tcp http
192.168.1.1 443/open tcp https
192.168.1.1 3389/open tcp ms-wbt-server

Don't worry if the above is hard to follow. I'm just illustrating what a high-level use case looks like.

Wrap Up

Mastering the art of piping and grep commands in Linux will open a new world of efficient and effective data management. In essence:

• Use | to pipe output from one command as input to another.
• Utilise grep to filter and search text data.
• Enhance your data manipulation by combining both pipe and grep together.

It is crucial to remember that practice is key. The more you experiment with different commands and scenarios, the more you'll discover and learn. Don't be afraid of mistakes - they are just stepping stones on your path to mastering Linux.

Unless you make those mistakes as root, then it might be time to buy a new computer...

Seriously, the number of times I've accidentally locked myself out of a Linux virtual machine in the cloud because I deleted the virtual network interface is unreal...

From simple tasks such as listing and searching files, to complex operations such as network analysis and system monitoring, the power of piping and grep commands is limitless. With time, you'll be chaining commands like a pro!

Till next time!