Friday, May 14, 2010

Running Experiments and Using Scripts

One of the huge advantages of using a computer simulation platform such as PS-I lies in its ability to generate large sample sizes for experiments in a relatively short amount of time as well as being able to easily store and process large amounts of data.  However, taking advantage of these capabilities requires a basic understanding of Tcl/Tk, a programming language that can be used to give a series of instructions to PS-I.  Tcl/Tk can be used to produce scripts (.scp files) which you can load in PS-I the same way you would load a model file (.mdl).  Scripts have a wide range of functionality, but for the purposes of this post we will be using scripts to a) Run a model multiple times while collecting statistics for each run and b) Change model parameters during the course of a run.

You can download the script we will be working with here and be sure to have SimpleC.mdl as well.  Scripts are simply text files, saved with a .scp filename to work properly with PS-I, so they can be edited using Notepad or any other text editing software.  We recommend using Notepad++ because of its increased functionality (tabbed windows, better find and replace capabilities, compare function, etc.), but Notepad works fine as well.  If you open the script file, you’ll see about 25 lines of code, much of which is comprised of comments explaining what each line of code means.  These comments will always appear with a "#" in front of them which instructs PS-I to disregard that line in the script. This script is intended to automatically run SimpleC.mdl five times while recording the statistics for each run.  Below I’ll briefly explain each section of the code technically and give some examples of why that part of the script is useful. 

The first section of the code looks like this:
for {set i 1} {$i < 6} {incr i} {
This code designates i as a variable with an initial value of 1, and loops the script that follows this line 5 times, increasing the value of i by one each time the loop completes and ending when i=6.  Everything after the open bracket at the end of this line are the instructions that will be looped to PS-I.
load_file  "SimpleC.mdl"
This section of code tells PS-I to load the file called SimpleC.mdl.  Since this is the beginning of the loop, this model will be loaded 5 times before the script finishes running.
set_routine_code_by_name "bias_seed" "$i"
 This code instructs PS-I to find the "bias_seed" parameter and set its value to the current value of the variable i.  We defined i in the first line of the script, its value begins at 1 and increases by 1 each time the loop starts over.  This means that each of the 5 times that SimpleC.mdl is loaded, it will be assigned a different bias seed.  Changing the bias seed for each of the five different runs means that the random, exogenous shocks to the landscape will be different each time, producing wider variability and ultimately more robust findings in experiments.
file delete "SimpleC_$i.csv"
set_filename "SimpleC_$i.csv"
Here the script is instructing PS-I to start collecting and saving all of the statistics specified in the model to a .csv file named SimpleC_X.csv where X is the current value of our variable i (and also corresponds to the run number).  The "file delete" line exists to ensure that if you happen to already have a file called SimpleC_1.csv in your PS-I folder, PS-I deletes that file then recreates it instead of simply amending new statistics to the end of the pre-existing file.
 new_field_viewer
This section of code instructs PS-I to open the field viewer window so the user can watch the model run.  This is not necessary in order for the model to run, but it can be helpful, especially to new users.
for {set k 0} {$k < 50} {incr k 1} {step}
This code sets a new variable k with an initial value of 0 and increases it by one until it reaches 49.  Each time this happen, the model moves forward one timestep, meaning that after this instruction SimpleC.mdl will have moved from timestep 0 to timestep 50.
apply_effect "landscape" "basic" {"Influence_level" "2"}
Here we have an example of how a script is capable of manipulating a model during the course of a run.  At this point the model has moved forward 50 timesteps and now we are applying an effect to the our landscape, more specifically all the basic agents in the landscape.  The effect we are applying sets the Influence_level attribute for these basic agents to 2 (In SimpleC.mdl, the default influence level for basic agents is 1).
for {set k 0} {$k < 50} {incr k 1} {step}
Again we step the model forward 50 time steps.
save_state_to_file "SimpleC_t50_$i.snp"
Then we save a snapshot (.snp file) of the end-condition of this run of SimpleC.mdl.  This snapshot will save the exact layout of the model landscape at the time, including all of the activated and subscribed identities for each agent, and can be opened in PS-I the same way a .mdl file can be.
}
Lastly, a close bracket is needed to tell PS-I that this iteration of the loop has finished.  The instructions will then be repeated four more times before the script finishes.

While this may seem like a complicated process, this scripting technology that allows us to run any number of runs and manipulate the models in any way during those runs without needing to manually operate the PS-I user interface is crucial when it comes to running experiments.  Two requirements for a well-designed experiment are a statistically significant sample size and independent variables that can be altered during different model runs; scripting allows for a PS-I user to easily meet both of these requirements.  If you want to test the script out yourself, put both SimpleC.mdl and SimpleC.scp into your root PS-I folder, open PS-I, go to File then Load, select PS-I Scripts (.scp) under "File Types" and load SimpleC.scp.  If you're curious, after the script finishes you can open up the .csv files for any of the runs in Excel (or a similar program) and explore how PS-I outputs statistics.  I'm sure there will be a post in the near future dealing with how a PS-I user can determine what statistics to collect and how to use tools like Excel macros to explore the statistics outputs at a distributional level.  More advanced scripting options and the code associated with them can be found in the Appendices of the PS-I manual, here.

No comments:

Post a Comment