Protocol of Building a Major Pipeline
in a Multiple Pipeline
in a Multiple Pipeline
On this page, you can read how you can build a major pipeline in a multiple pipeline.
The following steps need to be followed for building a major pipeline very successfully.
Steps of the protocol of Building a major pipeline in a multiple pipeline:
Step 1: Determining the processes between the input data and the final results in the minor pipelines:
Before you can start building the major pipeline, you first have to investigate what the input data are and what the proposed final results will be out of those. These data will be processed by the minor pipelines. In other words, you first have to build the minor pipelines themselves at skeletal level. Therefore, you must read the page of Protocol of Building Minor Pipelines in a Multiple Pipeline. After that, you can continue this protocol.
Before you can start building the major pipeline, you first have to investigate what the input data are and what the proposed final results will be out of those. These data will be processed by the minor pipelines. In other words, you first have to build the minor pipelines themselves at skeletal level. Therefore, you must read the page of Protocol of Building Minor Pipelines in a Multiple Pipeline. After that, you can continue this protocol.
Step 2: Building the Jobscript Maker:
This is the most important part of building the major pipeline. Now you have built the skeleton scripts for the minor pipelines of your multiple pipelines in step 2, you have to use these for building the jobscript maker of your major pipeline. The jobscript maker is a script - mostly in Python - that builds the minor pipelines for each specific data set. The jobscript maker will scan all the data sets that have to be processed. For each specific data set, it will create the scripts for the right processing those data with the help of the skeleton scripts of the minor pipelines. For more information about how to build the jobscript maker, you have to go to the page of ....
This is the most important part of building the major pipeline. Now you have built the skeleton scripts for the minor pipelines of your multiple pipelines in step 2, you have to use these for building the jobscript maker of your major pipeline. The jobscript maker is a script - mostly in Python - that builds the minor pipelines for each specific data set. The jobscript maker will scan all the data sets that have to be processed. For each specific data set, it will create the scripts for the right processing those data with the help of the skeleton scripts of the minor pipelines. For more information about how to build the jobscript maker, you have to go to the page of ....
Step 3: Testing the scripts and major pipeline:
After you have programmed all the scripts, you can test your major pipeline. But before you can do that, you have to be sure that your minor pipelines work correctly. But if that's not the case, you can test your major pipeline. If your major pipeline works, you have finished.
After you have programmed all the scripts, you can test your major pipeline. But before you can do that, you have to be sure that your minor pipelines work correctly. But if that's not the case, you can test your major pipeline. If your major pipeline works, you have finished.
Step 4: Finalizing the scripts:
When your major pipeline works perfectly, you have to carry out the following steps in each script:
When your major pipeline works perfectly, you have to carry out the following steps in each script:
- You have to comment your code.
- You need to remove all dead code.
- You have to write down the Show Usage Information's:
In the Show Usage Information, the following has to be explained about the script in the right order from the top to the bottom:- Function of the script
- Way of usage (of the command line arguments)
- Example of starting the script
Step 5: Making the start bash script:
All your scripts have been programmed beautifully now, but to start the major pipeline itself a start bash script must be programmed.
You always have to use the Start Majops Skeleton Script Bash Pipeline as start for it.
In that start bash script, the following items have to programmed in the right order from top to the bottom:
All your scripts have been programmed beautifully now, but to start the major pipeline itself a start bash script must be programmed.
You always have to use the Start Majops Skeleton Script Bash Pipeline as start for it.
In that start bash script, the following items have to programmed in the right order from top to the bottom:
- You have to write down the Show Usage Information:
In the Show Usage Information, the following have to be explained about the pipeline in the right order from the top to the bottom:- Function of the pipeline
- Way of usage (of the command line arguments)
- Example of starting the pipeline with the start bash script
- Put all the scripts and programs that are called and used, when your major pipeline is running in the file check list in alpha numerical order.
- Write down the questions for the user of the major pipeline that he has to answer for generating the right command line arguments of your major pipeline.
- Program the code for the generation of the right command line arguments for your major pipeline according to the answers of the user.
- At the end of the start bash script, your major pipeline will be called with the major highway bash script by giving the right command line arguments to it.
N.B.: This is only one line of code!
Step 6: Making a flowchart of your multiple pipeline:
You have to make a flowchart of your majoe pipeline. Therefore, it is best to use a flowchart program. A good free program for that is Dia. When you make the flowchart, all the scripts must be mentioned on that and you have to draw the flowchart from the left to the right or from the top to the down.
Save the flowchart as a PNG picture and name the PNG file in the following way:
Flowchart_[ pipeline name ].png
Example:
Flowchart_PlasticBottlesCreatorPipeline.png
You have to make a flowchart of your majoe pipeline. Therefore, it is best to use a flowchart program. A good free program for that is Dia. When you make the flowchart, all the scripts must be mentioned on that and you have to draw the flowchart from the left to the right or from the top to the down.
Save the flowchart as a PNG picture and name the PNG file in the following way:
Flowchart_[ pipeline name ].png
Example:
Flowchart_PlasticBottlesCreatorPipeline.png
Step 7: Finalizing and archiving your multiple pipeline:
You have done all the steps before. Now you can finalize and archive your multiple pipeline for other users in the future.
All the scripts, including the start bash script and the minor pipelines, and the flowchart need to be compressed in 1 zip file. This zip file will get the same name as the pipeline name.
[ pipeline name ].zip
Example:
PlasticBottlesCreatorPipeline.zip
You have done all the steps before. Now you can finalize and archive your multiple pipeline for other users in the future.
All the scripts, including the start bash script and the minor pipelines, and the flowchart need to be compressed in 1 zip file. This zip file will get the same name as the pipeline name.
[ pipeline name ].zip
Example:
PlasticBottlesCreatorPipeline.zip