Evolution of Prioritized Dynamic Scheduling Algorithm for HPC

Open to public: 8/1/2019
Last updated: 8/7/2019 16:56 KST
Link style designed by HTML5 UP. Authorized under CCA 3.0 license.

Introduction
Download Source
Release details
Technical Requirements
How to install
How to run

The prioritized dynamic approach of high-performance computer scheduling algorithm is a type of HPC scheduling algorithm ("PYSA") which considers the priority of each job requests waiting for execution in an HPC (cluster). Compare with traditional HPC scheduling algorithms and classical PYSA algorithms, our evolution approach cares and actually uses the job priority when generating the job execution sequence. As a result, the job request execution sequence may be out of the pushing order but will have the HPC (cluster) to performance a 50% improvement on overall utility rate and have an overall 21% time-saving.

All of our approaches go into two branches, FE and LTE. FE stands for Future Evolution, which is the version supports three parameters from the request. However, the LTE version, Long-term Evolution, not the 4G LTE, supports a flexible amount of request's description parameters. In the FE approach, users do not have to define their size of parameter and name of them. The reason is, in FE, we pre-defined that each parameter should contain four general parameters and three technical parameters. General classifiers are RID (Request ID), Pushing Time, Request Priority and Execution Length, and technical variables are CPU, GPU, and Memory. However, in LTE approach, besides four general parameters, users are required to define how many parameters and what they are. The method of definition will be explained below.

Other than just the algorithm, we are also providing an algorithm simulation and evaluation method to help developers and future researchers to better understand their newly designed scheduling algorithms by making comparisons between current existing scheduling algorithms. Our simulator and evaluation service can run algorithms coded in any language, no matter if it is C, Java, C++, Python, or even PHP. As long as the operating system does support such language, we are ok with it.

PYSA-LTE Server Oriented Source
PYSA-LTE Request Oriented Source
EVAS-FE Source
EVAS-LTE Source

GitHub source is still private for future researching purpose.

In the current release, we are providing several things. For PYSA, we only offer the LTE version, while the FE edition has archived as a history. We are providing two branches of PYSA-LTE. These two branches are both designed base on the idea of PYSA but are slightly differentiated on how they are doing the placement task.

For the simulation and evaluation service, we provide both FE and LTE versions. Don't worry, even if PYSA-FE is not available anymore, EVAS (the evaluation service)-FE has been modified to be adapt with PYSA-LTE.

Both PYSA-LTE and EVAS family REQUIRES a Unix or Linux system to proper run as expected. In another word, our designs are NOT compatible with Windows family systems, Windows NT, MS-DOS, and any other non-Unix or non-Linux systems. However, for windows users, they can run this program by installing the Windows Subsystem for Linux on their eligible Windows 10 devices.

Both PYSA-LTE and EVAS family are optimized for Apple OS X (a.k.a. Apple macOS), Apple Darwin, and Ubuntu because we implemented these programs under OS X and Ubuntu. To be clarified, Apple OS X systems are certified Unix operating systems.

To run our programs, we do REQUIRE user's computer to have pre-installed the latest version of GCC, G++, clang, C RT std=c99, C++ RT std=c++11, Python 3, pip3 by Python 3 and Java 10 (Java is recommended but is not required for LTE. Java is required for FE). Programs do have a hardware requirement to run correctly, but we believe current modern personal computers and servers are strong enough to run, so we will not specify the hardware requirements.

After unpacking the archive, run ./make.sh to freshly compile the program. Then use ./a.out to execute the program. Please note that the program maker and complier DO NOT verify your system runtime environment, please make sure your system is eligible to compile and run programs before you run make.

In this tutorial, we assume all of your executables are called a.out

PYSA Family executables

Command-line format

For all PYSA evolutions, we are expecting you to give an address of a configuration file. The file address is required, or the program will throw an error. Run program as ./a.out <yourConfFile>. For example, you can run your program as ./a.out ./input.conf, if your destinated configuration file is ./input.conf.

Input file syntax

As the inputting configuration file, we are expecting it to have the syntax as shown below:

para0	...	paraN
para0ClusterCapacity	...	paraNClusterCapacity
req0Para0Var	...	req0ParaNVar
...
...
reqNPara0Var	...	reqNParaNVar
					

Do not have empty line unless it is the end of the configuration file. Use one (1) tab in between parameters. DO NOT have spaces or tabs before the first parameter. In FE mode, also applied to EVAS, N MUST be greater or equals to 2, for parameter count.

For example, a valid configuration file can be:

CPU	GPU	MEM	NTWK
100	100	100	100	123 //The parameter size can be greater than header, but cannot be fewer
1	1	1	1 //Here you defined your first request
2	2	2	2
				

EVAS Family executables

EVAS is running based on flag, that means, you will need to use flags and options to have the program to work for you.

Command Syntax General

We have a universal command syntax to run the program. Simply, use ./a.out -[flag] [option 0] ... [option N], or ./a.out --[flagLong] [option 0] ... [option N] to use long flags. Please note that, same as all other command-line tools such as gdb, git, and valgrind, in between each element of the command, please use one (1) space to separate. For flags and options, use one (1) dash for short command or two (2) dashes for long command. For example, a valid command can be ./a.out -l or ./a.out --list.

Register an algorithm

Syntax: ./a.out -a [algorithmName] [executableDirectory] [executableFileName] or ./a.out --add [algorithmName] [executableDirectory] [executableFileName]. DO NOT use ./ prefix in the executable file name section. Algorithms will be captured and took a picture by EVAS, so you are free to move the algorithm to another directory, even remove it from the disk, without affecting the EVAS recognization.
Example: ./a.out -a testAlgorithm ./algorithm alg.out, which means I want to register an algorithm whose name is testAlgorithm, the executable file is located at ./algorithm, and the executable is called alg.out.

Remove a registered algorithm from the list

Syntax: ./a.out -r [algorithmName] or ./a.out --remove [algorithmName].
Example: ./a.out -r testAlgorithm, which means I want to remove an algorithm which name is testAlgorithm from the list of registered algorithm list.

Update a registration

Syntax: ./a.out -u [algorithmName] [executableDirectory] [executableFileName] or ./a.out --add [algorithmName] [executableDirectory] [executableFileName]. DO NOT use ./ prefix in the executable file name section. The to be updated algorithm must be already registered in EVAS. Please use the exact spelling of the registred algorithm, or the program will throw an error. To rename a registered algorithm, please remove it from the list first by using -r flag, and then use -a flag to re-register the algorithm with the new name.
Example: ./a.out -u testAlgorithm ./algorithm_new alg_new.out, which means I want to update an algorithm registration which name is testAlgorithm, with the new executable file located at ./algorithm_new, and the executable is called alg_new.out.

View the registration list

Exact command: ./a.out -l or ./a.out --list

Comparison

The default mode of the comparator will do the comparison and print out only the minimum information, which is essential for developers to understand the simulation execution status and the evaluation result. To get more info regarding the simulation and evaluation, please use an option to re-run the program. Without a specified inputting configuration, EVAS will automatically generate testing cases, but these testing cases are FE-type description only. Users can use multiple options regardless of the sequence. In comparison, please run command:./a.out -c [option 0] ... [option N]. The following section will only talk about options.

Default mode
No options specification required.

Timing mode
-t or --time. The timing mode will also print out the timing statics to show EVAS took how long time to do what process. For example, it can say, "Input conf process used 0.001388 seconds," which means it cost EVAS 0.001388 seconds to capture and process the inputting configuration.

Ranking mode
-r or --rank. This mode will prints out all registered algorithms with their evaluation result, which is the score and ranking of each of the listed algorithms. For example, EVAS can say "Position 1: 19AGA3540729150245413 (a.k.a. s1): 57.000000."

Detailed mode
-d or --detail. In detailed mode, EVAS will print out all of the internal status and temporary objects. i.e., original request list, algorithm execution status map according to its output, and algorithms alter IDs. In the FE version, EVAS can print more things, for example, the original algorithm output.

Specifying an input source
-f=[fileAddress] or --file=[fileAddress]. By specifying a configuration file, EVAS will use the input instead of an auto-generated configuration. By specifying a source, developers can further optimize their algorithms by focusing on their special computing need.

At the end

This is the end of the tutorial. Please contact us if you have any comments, questions or complaints. Thank you, and enjoy your use.