Organized Pipeline Scripting
in Single Pipelines
in Single Pipelines
Maybe, you think: "Oh, I want to build my own single pipeline to analyze and explore my data. I program some scripts. I connect them together to call them after one another and it works." That sounds easy, but that is not the truth unfortunately. When you decide to build a single pipeline according to Majops, you have to follow some rules. You will have to program your single pipeline very organized.
Here, you read the rules how you have to be build a single pipeline very organized according to Majops.
Bash is the highway of a single pipeline:
First of all, what is very important, is that scripts and programs of your single pipeline always have to be called from the programming language Bash. All programs and scripts are called from the Linux environment. The programming language for Linux is Bash. If you would call from a different environment, that would be insufficient. For example, If you would call your scripts and programs from Python, what actually happens, is that the Python environment is left and the environment changes to Bash. After that, the script or program is called and the environment changes back to the original environment of Python. This happens each time with every call. Because this is carried out many times, this slows down your single pipeline as you can imagine.
To prevent these problems, you always have to use a so-called highway bash script as starting point. This highway bash script can be downloaded on the page of Majops Skeleton Script Downloads in Single Pipelines.
Below in figure 5, you can see how this highway bash script actually works schematically.
Figure 5: Overview of how a single pipeline works according to Majops.
The highway of a single pipeline according to Majops is always in the programming language Bash. From Bash, other scripts or programs are called.
The highway of a single pipeline according to Majops is always in the programming language Bash. From Bash, other scripts or programs are called.
The overview in figure 5 shows the highway in the middle in the programming language Bash. On the left and on the right next to that highway, you see other processes that can be called from the highway. Each process is in a different programming language. There are many programming languages, but the most well known are put in this figure. You also see in the left down corner that other programs can be used as well instead of programming a new script of your own. N.B.: Always use an existing program, if a specific process in your single pipeline already exists in that program!
Instruction movie about the explanation of the highway in overview to code:
The overview in figure 5 may not mean anything to you. That is why this overview will be shown in code how it works in the next instruction movie. This instruction movie can be seen below.
Instruction movie 3: Instruction movie of how the overview of highway is in code.