General Information about Pipelines
Case scenario:
In the supermarkets, you would like to buy plastic bottles of water. The plastic bottles have to be produced first. The way how that they are produced, is shown in figure 1 below. In figure 1, you can see oil on the left and plastic bottles on the right.
Figure 1: Plastic bottles produced out of oil.
It is possible to make those plastic bottles out of the oil. The question is: How?!
As you can imagine, several steps - or processes - must be gone through. In every step/process, another task is done. The first process in this example could be making the oil more solid. The second process could be making the more solid oil transparent. The third process could be extracting polluting molecules from this oil; so this oil cannot contaminate the drinking water inside the bottle. The final process could be making the plastic bottles out of the transparent oil.
All these processes have to be put after one another. Then, you get a simple pipeline for making those plastic bottles out of the oil.
As you can imagine, several steps - or processes - must be gone through. In every step/process, another task is done. The first process in this example could be making the oil more solid. The second process could be making the more solid oil transparent. The third process could be extracting polluting molecules from this oil; so this oil cannot contaminate the drinking water inside the bottle. The final process could be making the plastic bottles out of the transparent oil.
All these processes have to be put after one another. Then, you get a simple pipeline for making those plastic bottles out of the oil.
Pipelines:
So, a pipeline is used to make plastic bottles out of that oil in this case scenario above. The making of these bottles will happen in phases/processes. Each phase/process has its own task.
A pipeline can be compared to pipes that are connected after each other. 2 pictures in figure 2 below show what is meant.
On the left, you see pipes. Each pipe represents a specific task that is supposed to accomplish. The output of one pipe will be input for the next one. Etcetera. The only problem is that each pipe is separate. The solution for that problem is to connect all those pipes together after one another. The end result is one long pipeline. That is shown in the picture on the right in figure 2.
A pipeline can be compared to pipes that are connected after each other. 2 pictures in figure 2 below show what is meant.
On the left, you see pipes. Each pipe represents a specific task that is supposed to accomplish. The output of one pipe will be input for the next one. Etcetera. The only problem is that each pipe is separate. The solution for that problem is to connect all those pipes together after one another. The end result is one long pipeline. That is shown in the picture on the right in figure 2.
Figure 2: Building of a pipeline from separate pipes.
The picture of the pipeline in figure 2 is of course only a representation. In software engineering, each pipe is a process and all processes are connected to each other in a pipeline.
Pipelines in Software Engineering:
The definition for a pipeline is as follows:
A chain of processing elements, arranged so that the output of each element is the input of the next element.
A simple pipeline is shown figure 3.
A chain of processing elements, arranged so that the output of each element is the input of the next element.
A simple pipeline is shown figure 3.
Figure 3: A simple pipeline.
This simple pipeline consists of 5 processes. Each process has its own specific task. The name of each process shows what the function of that process is. Furthermore, the output of each process will be the input for the next process.
In this pipeline example, the process 'Input Reader' get its input from the start, but its output will be transported to the process 'Filter' as input. As process 'Filter' has filtered the data, its output will be processed in the process 'Processor'. Etcetera. Finally, all the data is going through the processes 'Formatter' and 'Output Writer'. The process 'Output Writer' will finalize the pipeline to save all the output data.
In this pipeline example, the process 'Input Reader' get its input from the start, but its output will be transported to the process 'Filter' as input. As process 'Filter' has filtered the data, its output will be processed in the process 'Processor'. Etcetera. Finally, all the data is going through the processes 'Formatter' and 'Output Writer'. The process 'Output Writer' will finalize the pipeline to save all the output data.
Types of Pipelines:
There are 2 types of pipelines:
- Single pipelines
Single pipeline is a pipeline that runs on only 1 computer. - Multiple pipelines
Multiple pipeline is a pipeline that generates other pipelines as jobs for other computers in a supercomputer or in a gridcluster of computers, where each job gets its specific task depending on the data it has to process.
Instruction movie about pipelines:
In the next instruction movie, there will be explained about pipelines.
Instruction movie 1: Explanation about pipelines.