MOEAFramework Setup Guide

MOEAFramework is a Java library for numerical optimization using  multiobjective evolutionary algorithms (MOEA). This document will focus on using the framework to optimize a user-defined problem. It will also discuss more advanced topics such as submitting multiple runs to a cluster, generating a reference set of optimal solutions, and measuring algorithm performance. For additional information about the concepts behind MOEAs and recommended references, please refer to the technical report “Basic MOEA Concepts and Reading” and browse the Reed Group Blog. The MOEAFramework website also contains extensive documentation for the framework, as well as an API reference of each of the functions available in the library.

Since MOEAframework is written in Java, it is easy to write a simple class to interface with the library.  The user can easily switch between multiple algorithms, sample problems, and post-processing analysis techniques to measure algorithm performance.

Software Requirements

There are several types of software needed for this activity.

If you’re using a Linux computing cluster, there are some additional requirements.  If you are new to remote Linux/UNIX systems, you may want to read about Setup and Basic Commands.

Getting Started with MOEAFramework

At http://www.moeaframework.org/examples.html you’ll find six example files of MOEAframework’s functionality.  The website explains the purpose of these six example files. All six files should run without modification.

Runs NSGA-II on UF1 the test problem.

Runs the UF1 test problem using three separate algorithms.

Collects runtime dynamics for the UF1 test problem running with the NSGA-II algorithm.

Demonstrates how to add a new problem (the DTLZ2 test function) in Java code.

Demonstrates how to add a new problem (the DTLZ2 test function) in C code.

A GUI that will run your problem in real time and print out runtime dynamics for test functions. Start here if you are a beginner to the MOEA Framework or MOEA’s in general.

These examples will help you understand how to interface with the library. The GUI (Example 6) can be opened on Windows simply by double-clicking the *.jar file. Example 5 describes how to connect a user-defined problem to the framework, which will be the focus of the following sections.

Connecting External Problems

Optimizing user-defined, “external” problems  is a common research need across many disciplines, and is readily addressed using MOEAframework.  There are two major steps to connect an external problem to the framework:

  1. Modify your C/C++ program to communicate with MOEAFramework
  2. Create a Java class (wrapper) to define and run your problem 

Once you learn the format of the Java wrapper class, you will also be able to perform simpler optimizations of native problems included in the library.

MOEAFramework optimizes external problems by passing decision variables to the user-defined executable via sockets. The user-defined problem reads the decision variables passed in, performs its calculations, and returns values of objectives and constraints back to the framework. This process continues until the maximum number of function evaluations is reached, at which point the external problem terminates and MOEAFramework performs any post-processing steps needed.

Step #1: Modify Your C/C++ Program 

The information in this section is also explained in a video: External problems in MOEAFramework.

Modify the main() function of your program as follows:

int main(int argc, char **argv) {

//Define your variables, objectives, and constraints

        Objs = new double[num_objectives];

        Vars = new double[num_variables];

        Cons = new double[num_constraints];

        //If required - Initialize your problem

        //Intialize the MOEA Framework with your objectives

        MOEA_Init_socket(num_objectives, num_constraints, NULL);

        while (MOEA_Next_solution() == MOEA_SUCCESS)

        {

        

                MOEA_Read_doubles(num_variables, Vars);

        

                //Calculate your problem here using the decision variables (Vars)

                evaluate_my_problem(Vars);

        

MOEA_Write(Objs, Cons); // objectives and constraints

}

 

MOEA_Terminate();

return EXIT_SUCCESS;

}

Compile the C/C++ executable

        gcc -Wall -O3 -o moeaframework.o -c moeaframework.c

        gcc -Wall -O3 -o DTLZ2.exe DTLZ2.c moeaframework.o

Step #2: Create a Java Class to Define and Run Your Problem

The information in this section is also explained in a video: External Problems: Java Class.

//Example of Java wrapper class for connecting an external problem

//This is only part of Example5.java. Use the full file for your template.

public static class MyDTLZ2 extends ExternalProblem {

                public MyDTLZ2() throws IOException {
                        super("./auxiliary/c/dtlz2_stdio.exe");
                }

       //Note: if your executable requires command-line, arguments, do:

       //super(“./path/to/executable.exe”, “arg1”, “arg2” .. )

                @Override
                public String getName() {
                        return "DTLZ2";
                }

                @Override
                public int getNumberOfVariables() {
                        return 11;
                }

                @Override
                public int getNumberOfObjectives() {
                        return 2;
                }

                @Override
                public int getNumberOfConstraints() {
                        return 0;
                }

                @Override
                public Solution newSolution() {
                        Solution solution = new Solution(getNumberOfVariables(),
                                        getNumberOfObjectives());

                        for (int i = 0; i < getNumberOfVariables(); i++) {
                                solution.setVariable(i, new RealVariable(0.0, 1.0));

                    //set your decision variables ranges here.

                    //this example sets every variable to a 0-1 range
                        }

                        return solution;
                }


        }

//This is only part of Example5.java. Use the full file for your template.

public static void main(String[] args) {
                //check if the executable exists
                File file = new File("./auxiliary/c/dtlz2_stdio.exe");
                                
                if (!file.exists()) {
                        System.err.println("Executable file not found");
                        return;
                }
                
                //configure and run the DTLZ2 function
                NondominatedPopulation result = new Executor()
                                .withProblemClass(MyDTLZ2.class)
                                .withAlgorithm("NSGAII")
                                .withMaxEvaluations(10000)
                                .run();
                                
                //display the results (this problem has two objectives)
                for (Solution solution : result) {
                        System.out.print(solution.getObjective(0));
                        System.out.print(' ');
                        System.out.println(solution.getObjective(1));
                }
        }

Some common problems to avoid:

Compile your Java class

javac -classpath MOEAFramework-1.15-Executable.jar:. Example5.java

java -Xmx128m -classpath MOEAFramework-1.15-Executable.jar:. Example5

Outputting Optimal Decision Variables and Objective Values

                for (Solution solution : result) {

                        for (int i = 0; i < 8; i++)

                        {

                                System.out.print(solution.getVariable(i));

                                System.out.print("  ");

                        }

                        for (int i = 0; i < 5; i++)

                        {

                                System.out.print(solution.getObjective(i));

                                if (i < 4)

                                {

                                        System.out.print("  ");

                                }

                                else

                                {

                                        System.out.println();

                                }

                        }

                        

                }

Congratulations, you now have the optimization working! To use a different algorithm, or to change the maximum of function evaluations, you just need to modify your Java file and re-compile it. This summarizes the process of connecting a user-defined problem in C/C++ to the MOEAFramework and configuring the two programs to communicate with each other. We will now discuss more advanced topics such as running multiple random seeds, generating and visualizing a reference set, and calculating statistics to measure algorithm performance.

Advanced Topics

Running Multiple Random Seeds

In the example above, you ran a single optimization for a set number of function evaluations. We might expect that for a sufficient number of function evaluations, the algorithms will converge to the global Pareto front (or at least we hope). But because the algorithms work by stochastically sampling and mutating solutions, solution quality can also depend on the random seed used. An ideal algorithm would not have this dependence, since it would converge to the global Pareto optimal front regardless of random seed. However, random seeds do matter in reality, and it is usually a good idea to run multiple random seeds to ensure robustness. Another benefit of random seeds is reproducibility—optimizing with the same random seed will produce the same output every time.

In MOEAFramework, there are two ways to run multiple seeds. If your function evaluations are very fast and you would like to run multiple seeds in a single job, modify the Executor statement in Example5 above from Executor.run() to Executor.runSeeds(10) to run ten different random seeds consecutively.

The limitation with this approach is that it is often not feasible to run multiple random seeds in a single job, particularly for more complex models and/or high numbers of function evaluations.   Furthermore, it is often helpful to use pseudorandom numbers that are “repeatable” (i.e., if you run seed 1 twice, you get the same results each time).  We’ll modify the Java class to use a predictable random seed, and pass the random seed to the Java class via the command line. This process is detailed in the following steps.

import org.moeaframework.core.PRNG;

        public static void main(String[] args) {

                long seed;

                if (args.length > 0) {

                        try {

                                seed = Long.parseLong(args[0]);

                                PRNG.setSeed(seed);

                        } catch (NumberFormatException e) {

                                System.err.println("Argument must be a number");

                                System.exit(1);

                        }

                }

//...Other Commands from the Java file follow here: Executor, etc.

        }

Submitting Multiple Jobs from BASH

Once you are running multiple random seeds, the process of submitting runs becomes tedious in short order. We would like to automate the submission process to loop through a list of random seeds and send out cluster jobs for us, rather than submitting them manually. We will do this using BASH (Bourne-Again Shell), which is the default command-line language on most Linux-based systems. This example will assume that you are submitting jobs to a Linux cluster based on PBS (Portable Batch System), but you should be able to tweak the script to run on other types of systems.

The example script below performs the process of automated job submission. It loops through an array of random seeds. Each time through the loop, it builds a string to specify the commands for that job, and sends it to the queuing system using the qsub command.

#!/bin/bash

NSEEDS=50 # How many random seed runs do you want to submit?

SEEDS=$(seq 1 ${NSEEDS}) # Create an array of seed values from 1 to NSEEDS

JAVA_ARGS="-Xmx256m -classpath MOEAFramework-1.15-Executable.jar:."

JAVA_CLASS="Example5" # What is the name of your Java class?

NAME_BASE="Example5" # The string to begin your output filenames

for SEED in ${SEEDS} # Loop through the different seed values and submit a run for each

do

        # the NAME here is used for the command “qsub” and for renaming your

        # output files. It’s different than the name of the java program

        NAME=${NAME_BASE}_${SEED}

        echo "Submitting: ${NAME}"

        

        # Build a string called PBS which contains the instructions for your run

        # This is basically writing your *.sh files for you to submit to the cluster

        # Important: how much walltime do you want to request?

        # This also specifies folders for output files and error files

        PBS="#!/bin/bash\n\

        #PBS -N ${NAME}\n\

        #PBS -l nodes=1\n\

        #PBS -l walltime=24:00:00\n\

        #PBS -o output/${NAME}.out\n\

        #PBS -e error/${NAME}.err\n\

        cd \$PBS_O_WORKDIR\n\

        java ${JAVA_ARGS} ${JAVA_CLASS} ${SEED}"

        

        # This last line ("java ...") is the command to run your problem

        # The -o flag specifies where to print the output stream.

# The -e flag specifies where to print the error stream.

        # Echo the string PBS to the function qsub, which submits it as a cluster job

# A small delay is included ("sleep") so we don't overload the submission process

        echo -e ${PBS} | qsub

        sleep 0.5

done

Some notes about this script:

./submit50Seeds.sh

chmod +x submit50Seeds.sh

qstat –u insertYourUsernameHere

qdel <job#>     

To delete all jobs running under your username, run:

qkill ${qselect -u <yourUsername>}

Generating a Reference Set

MOEAFramework contains a number of post-processing tools to analyze your results. One of the first analysis steps is typically to generate a reference set, which includes all of the non-dominated solutions across your optimization runs. (This could include multiple random seeds, multiple algorithms, etc.). The following script shows an example of how to generate a reference set from the approximation sets you just created:

#!/bin/bash

PROBLEM=Example5

JAVA_ARGS="-classpath MOEAFramework-1.15-Executable.jar:."

# Generate the reference set from all combined approximation sets

# Note that you will need to point this command at the output sets that you generated

# The arguments are " -o (reference set to create) (names of your set files)

echo -n "Generating reference set..."

java ${JAVA_ARGS} org.moeaframework.util.ReferenceSetMerger -e 0.01,0.01 -o ${PROBLEM}.reference ./output/*${PROBLEM}*.out

echo "done."

  

Removing unwanted values from approximation set files

#!/bin/bash

         

# Cut out only the objective function values from the Borg output files.

 

# MYPATH - the folder where the source files are “./” is the current directory

MYPATH=./output/

#INPUT_NAME_BASE – the string that your input files start with

INPUT_NAME_BASE=Borg_myProblem_

#OUTPUT_NAME_ADDENDUM – what addendum do you want to add to the current files?

OUTPUT_NAME_ADDENDUM=_ObjOnly

#What extension do your files have?

EXTENSION=.out

#What columns do your objective values start and end with?

START_COLUMN=9

FINISH_COLUMN=13

#How many seeds need the addendum change?

NUM_SEEDS=50

         

echo "Beginning..."

for ((I=1; I<=$NUM_SEEDS; I++));

do

  echo "Processing $I"

    cat ${MYPATH}${INPUT_NAME_BASE}${I}${EXTENSION} | cut -d ' ' -f ${START_COLUMN}-${FINISH_COLUMN} > ${MYPATH}${INPUT_NAME_BASE}${I}${OUTPUT_NAME_ADDENDUM}${EXTENSION}

  done

echo "Totally done."

Runtime Dynamics

A more advanced post-processing step is to calculate the performance of the algorithm during runtime using one or more performance metrics. More information about multi-objective optimization metrics is available in the MOEAFramework documentation as well as the references listed at the beginning of this document. Runtime performance is measured using the Instrumenter class. The general idea is to measure the performance of the algorithm relative to the reference set, since the latter is the best-known set of solutions. To set up the Instrumenter class, insert the following into the main() function of your Java file, just above the block defining the Executor:

                //Create and configure Instrumenter object        

                Instrumenter instrumenter = new Instrumenter()

                            .withReferenceSet(new File("DTLZ2.reference"))

                            .withProblemClass(DTLZ2.class)

                        .withEpsilon(0.01,0.01)

                        .attachPopulationSizeCollector()

                        .withFrequency(1000)

                            .attachElapsedTimeCollector()

                            .attachGenerationalDistanceCollector()

                        .attachHypervolumeCollector()

                        .attachAdditiveEpsilonIndicatorCollector();

                //configure and run the DTLZ2 optimization

                NondominatedPopulation result = new Executor()

                                .withProblemClass(DTLZ2.class)

                                .withAlgorithm("NSGAII")

                                .withMaxEvaluations(10000)

                                .withInstrumenter(instrumenter)

                                .run();

        Accumulator accumulator = instrumenter.getLastAccumulator();

                

                for (int j=0; j<accumulator.size("NFE"); j++) {

                  System.out.println(accumulator.get("NFE", j) + "\t" +

                                      accumulator.get("Elapsed Time", j) + "\t" +

                                      accumulator.get("Population Size", j) + "\t" +

                                accumulator.get("GenerationalDistance", j) + "\t" +  

                                    accumulator.get("AdditiveEpsilonIndicator", j) );

                }

                    for(String ss : accumulator.keySet()) {

                                    System.out.println(ss);

                    }

Editing Reference Sets for Aerovis (Aerovis Header)

AeroVis is a software package within the group that is used to view multidimensional datasets.  In general, you need a tab-delimited text file of objective function values, where each solution is printed in a separate row, to view the reference set in AeroVis.  Additionally, you may have decision variable values, and other parameters, in different columns as well.

AeroVis requires a set of header lines in the text file in order to work properly.  The following steps discuss how to convert the MOEAframework output to the proper format needed by AeroVis.

# This file contains the final non-dominated population.

# <GEN_HEADER> NA

# <DATA_HEADER> obj1, obj2, obj3, obj4

#

-999

#

REFERENCE SET OF FOUR OBJECTIVE VALUES GOES HERE

#