Majops Skeleton Scripts
for Major Pipelines in Multiple Pipelines
for Major Pipelines in Multiple Pipelines
If you would like to build a script in the major pipeline of your multiple pipeline according to Majops, you always have to make use of one of Majops Skeleton Scripts.
There are 4 types of Majops Skeleton Scripts for a major pipeline:
- Jobscript maker scripts
- Process scripts
- Highway bash script
- Start bash script
Each type of Majops Skeleton Script will be explained below.
Jobscript maker scripts:
The jobscript maker scripts are the most important scripts and are used for creating the scripts for the minor pipelines that the major pipeline generates. The jobscript maker scripts are called from a highway bash script. The jobscript maker scripts are in the programming languages Python2 and Python3. These are not in Bash or another language, because Python2 and Python3 are very fast in generating those scripts for the minor pipelines. In Python2 and Python3, it is possible to save a whole script at once, which increases the speed very much. Saving whole scripts at once are not possible in Bash.
Structure of a Majops Skeleton Jobscript Maker Script:
Each Majops Skeleton Jobscript Maker Script has the same structure for each programming language. Its structure could be divided in 8 parts:
- Imports of libraries
- Declarations of global variables
- The function/method showUsageInformation()
- The function getHeadersJobscript()
This function is only necessary for building the highway bash script of a minor pipeline. - The function makeNewJobscript()
- All the other functions/methods for the script itself
- The function main()
- Additional information of the script itself for:
- Remarks
- Descriptions how the script works
- Possible future improvements
The Majops Skeleton Jobscript Maker Scripts can be downloaded for the programming languages Python2 and Python3.
Click here to go to the download page for the Majops Skeleton Scripts for a major pipeline.
Click here to go to the download page for the Majops Skeleton Scripts for a major pipeline.
Process scripts:
Process scripts are used for processing the data in a major pipeline. These process scripts will be called from the highway bash script. The process scripts are in several programming languages. The choice of which programming language is chosen depends on the task to be carried out in that process.
Structure of a Majops Skeleton Process Script:
Each Majops Skeleton Process Script has the same structure for each programming language. Its structure could be divided in 6 parts:
- Imports of libraries
- Declarations of global variables
- The function/method showUsageInformation()
- All the other functions/methods for the script itself
- The function main()
- Additional information of the script itself for:
- Remarks
- Descriptions how the script works
- Possible future improvements
The Majops Skeleton Process Scripts can be downloaded for several programming languages.
Click here to go to the download page for the Majops Skeleton Scripts for a major pipeline.
Click here to go to the download page for the Majops Skeleton Scripts for a major pipeline.
Instruction movie about the process scripts:
In the next instruction movie, there will be explained about the process scripts. Directly after that, a process script is programmed as example.
Instruction movie ...: Explanation about the process scripts.
In this instruction movie, there will be explained about the process scripts. Directly after that, a process script is programmed as example.
In this instruction movie, there will be explained about the process scripts. Directly after that, a process script is programmed as example.
Highway bash script:
In the highway bash script, jobscript maker scripts, process scripts and/or normal programs are called after one another. The highway bash script can be called by a start bash script.
It is also possible that in a highway bash script another pipeline(s) is/are called.
Click here to go to the download page for the Majops Skeleton Scripts for a major pipeline.
It is also possible that in a highway bash script another pipeline(s) is/are called.
Click here to go to the download page for the Majops Skeleton Scripts for a major pipeline.
Start bash script:
Each major pipeline has 1 start bash script. This start bash script is only used to start the major pipeline. The start bash script generates the right command line arguments for the highway bash script it starts, depending on the answers of the user.
It is always in the programming language Bash and it should always be built at the last moment of building a major pipeline.
Click here to go to the download page for the Majops Skeleton Scripts for a major pipeline.
It is always in the programming language Bash and it should always be built at the last moment of building a major pipeline.
Click here to go to the download page for the Majops Skeleton Scripts for a major pipeline.
Structure of the Majops Skeleton Start Bash Script for a major pipeline:
The Majops Skeleton Start Bash Script has the following structure:
- Additional information of the major pipeline as comments
- An if statement for the Show Usage Information, in which information about the major pipeline is given.
- Code that will show the possible input files and/or input directories to the user, when he starts the major pipeline with start bash script.
- Code that will show questions to the user, in which he has to answer those for generating the right input for the highway script of the major pipeline.
- An alphanumeric overview of all the scripts and/or files that should be present in the major pipeline (zip file).
These files are checked, if each of them is present. If one is absent, the major pipeline will be aborted and a list of missing scripts and/or files will be shown on the screen to the user. - The highway bash script of the major pipeline is called with the right generated command line arguments. After that, the major pipeline starts running.
Reasons for the existence of the start bash script:
There are several reasons why the start bash script has been introduced in Majops. The first reason is that a user can recognize very easily and find out how a major pipeline should be started.
The second reason is that a user can see immediately, which highway bash script starts the major pipeline. The highway of a major pipeline is mostly recognized by a bash script that ends with "...Pipeline.sh" in its file name. If you have one highway bash script in your major pipeline, nothing is happening. But problems will arise, if you have more highway bash scripts in your major pipeline. A user cannot see anymore which highway bash script is the one that will start the major pipeline. By using a start bash script, that problem is solved.
The third reason is that a user does not have to find out what command line arguments he has to type in for the highway bash script, that is first called. Because the start bash script asks questions to the user, which he has to answer, the start bash script generates the right command line arguments for the user for the highway bash script.
The fourth reason is that the start bash script can check the presence of every file. If one is absent, the major pipeline will not continue.
The second reason is that a user can see immediately, which highway bash script starts the major pipeline. The highway of a major pipeline is mostly recognized by a bash script that ends with "...Pipeline.sh" in its file name. If you have one highway bash script in your major pipeline, nothing is happening. But problems will arise, if you have more highway bash scripts in your major pipeline. A user cannot see anymore which highway bash script is the one that will start the major pipeline. By using a start bash script, that problem is solved.
The third reason is that a user does not have to find out what command line arguments he has to type in for the highway bash script, that is first called. Because the start bash script asks questions to the user, which he has to answer, the start bash script generates the right command line arguments for the user for the highway bash script.
The fourth reason is that the start bash script can check the presence of every file. If one is absent, the major pipeline will not continue.
Working of the start bash script:
Figure ... shows how the start bash script really works. You can see that when the start bash script is started, the user has to answer questions, after which right command line arguments are generated for the highway bash script for starting the major pipeline.
Figure ...: Working of the start bash script of a major pipeline.
Here is shown how the start bash script of a major pipeline works. When the start bash script is started, the user has to answer questions, after which right command line arguments are generated for the highway bash script for starting the pipeline. These are given to the highway bash script of the major pipeline. As you can see, the jobscript makers are most abundant in that highway bash script. The command line arguments will mostly be used for those jobscript makers to make the scripts for the minor pipelines of the multiple pipeline.
Here is shown how the start bash script of a major pipeline works. When the start bash script is started, the user has to answer questions, after which right command line arguments are generated for the highway bash script for starting the pipeline. These are given to the highway bash script of the major pipeline. As you can see, the jobscript makers are most abundant in that highway bash script. The command line arguments will mostly be used for those jobscript makers to make the scripts for the minor pipelines of the multiple pipeline.