Trees | Indices | Help |
|
---|
|
Example LSM python modules and packages.
mpirun
The Lattice Solid Model implementation takes advantage of the MPI
(Message Passing Interface http://www.mpi-forum.org/) to provide parallel
execution. MPI programs are executed in an MPI environment, usually
by invoking the mpirun
command.
Unfortunately, different implementations of MPI, (eg LAM, SGI's
MPT, MPICH, etc) do not necessarily share the same command-line
options for the mpirun
command. To make running LSM
simulations a little easier for users, the ESyS-Particle software
provides some wrappers scripts which invoke the appropriate
mpirun
command.
The folling sub-sections explain the details of running LSM
simulations on the ess.esscc.uq.edu.au, an SGI Altix-3700
supercluster. In the examples, running the
esys.lsm.examples.waveprop.WaveSim
particle-wave-propagation python-module is used to illustrate command
syntax.
On the Altix 3700, ess.esscc.uq.edu.au, Environment
Modules http://modules.sourceforge.net/ is used for
setting up user environments for various locally installed
software. To define the assorted module
commands and
make the various packages available, at a bash prompt execute:
user@ess$ source /opt/modules/default/init/bash user@ess$ export MODULEPATH=$MODULEPATH:/raid2/tools/modulefiles
Generally, it is a good idea to put these two commands into a
~/.bashrc
file as follows:
# # Initialise the environment-modules stuff # if [ -f /opt/modules/default/init/bash ]; then source /opt/modules/default/init/bash export MODULEPATH=$MODULEPATH:/raid2/tools/modulefiles # # Load environments for various favourite modules. # module load gnuplot-4.0.0 module load pbspro5.4 fi
To set up the environment for running LSM simulations, execute:
user@ess$ module load lsm/sgimpi
If the environment has been successfully loaded, executing:
user@ess$ which mptrunLsm
should return a valid path to the mptrunLsm
script.
To run the esys.lsm.examples.waveprop.WaveSim python module using the SGI's MPT (Message Passing Toolkit), execute the following at the command prompt:
user@ess$ mkdir MyWaveSim user@ess$ cd MyWaveSim user@ess$ nice -n 15 mptrunLsm esys.lsm.examples.waveprop.WaveSim
This command will begin executing a wave propagation simulation,
generating some output to stdout and creating various data files in
the current working directory. Notice the use of nice
,
to lower the execution priority of the simulation, this is a
courtesy to other users when running longer multi-cpu jobs in the
interactive CPU set on multi-user systems. The
mptrunLsm
script simply examines the first command
line argument (treating it as a python module name) and
exec's the associated python-script
($LSMDIR/lib/python2.3/site-packages/esys/lsm/examples/waveprop/WaveSim.py)
as an MPI python process with appropriate mpirun
command line arguments.
The esys.lsm.examples.waveprop.WaveSim module accepts several
command line options which control various parameters of the
simulation. The --help
option:
user@ess$ mptrunLsm esys.lsm.examples.waveprop.WaveSim --help
causes the listing of the available command line options including descriptions of options and default values.
The wave-propagation simulation writes particle position and
displacement data to particle_*.txt
text files.
Two-dimensional wave propagation snap-shots can be
visualised using the esys.lsm.examples.DisplacementPlotter module.
To visualise the displacement data in file
particle_9.txt
, execute the following at the command
prompt:
user@ess$ lsmpython esys.lsm.examples.DisplacementPlotter particle_9.txt
This will display a VTK (Visualisation ToolKit) window with the particle displacement magnitude plotted as a 3D surface. The window is interactive: spin the camera by dragging with left mouse button, translate camera by dragging with middle mouse-button and zoom the camera by dragging the right mouse-button and close the window by typing 'q'.
Here, the lsmpython
script is used to run the
DisplacementPlotter
python module. This module does
not rely on an MPI implementation so can be run directly under
python. The lsmpython
script is provided as a
convenience for running modules specified by module-name
(esys.lsm.examples.DisplacementPlotter
) as opposed to
being specified by module-script-filename
($LSMDIR/lib/python2.3/site-packages/esys/lsm/examplesDisplacementPlotter.py)
- less typing and running a python-module is independent of the
install location.
The esys.lsm.examples.DisplacementPlotter accepts command line options:
user@ess$ lsmpython esys.lsm.examples.DisplacementPlotter --help
will print valid command line options, descriptions and default values.
A esys.lsm.examples.waveprop.WaveSim simulation also produces
seismograph data files. This seismograph data can be visualised
with gnuplot
http://www.gnuplot.info/. For example, running
the WaveSim module with default parameters produces the seismograph
data file srcToCorner_265.000_229.631_0.000.txt
. This
file contains data for a seismograph placed at coordinate (265.000,229.631,0.000) (this seismograph is a
member of a group of seismographs which are positioned on a line
extending from the source disturbance to a corner of the particle
block, hence the "srcToCorner_" file name prefix). Each
line of the file contains 10 values separated by whitespace: time,
displacement (dx,dy,dz), velocity (vx,vy,vz) and acceleration
(ax,ay,az). The x component of the displacement
can be plotted using:
user@ess$ module load gnuplot-4.0.0 user@ess$ gnuplot . . . Terminal type set to 'x11' gnuplot> plot 'srcToCorner_265.000_229.631_0.000.txt' using 1:2 with line
which produces a line plot of x-displacement (column 2 of data file) versus time (column 1 of the data file). To gnuplot the displacement magnitude sqrt(δx*δx + δy*δy+δz*δz):
gnuplot> plot 'srcToCorner_265.000_229.631_0.000.txt' using 1:(sqrt($2*$2+$3*$3+$4*$4)) with line
Another dataset produced during the simulation is
"record-section" data. Again, this is seismograph
displacement, velocity and acceleration data, but it is for
multiple seismographs. The x-displacement
record-section data can be displayed in gnuplot
using:
gnuplot> plot 'srcToCorner_record_section_reordered.txt' using 1:($2*0.0004 + $3) with line
This plots multiple seismograph datasets on a single set of axes and allows the visualisation of P-wave and S-wave propagation. The 'srcToCorner_record_section_reordered.txt' file contains 11 columns of data (time, distance-to-source, displacement(dx,dy,dz), velocity(vx,vy,vz), acceleration(ax,ay,az)).
To set up the environment for PBS Pro http://www.pbspro.com, load the
pbspro5.4
module:
module load pbspro5.4
To submit a script through PBS:
qsub qsubScript.sh
An example qsubScript.sh
script for for submitting
a wave propagation simulation to PBS:
#!/bin/bash ### Job name must be <= 15 characters #PBS -N MyWaveSim ### Specify a PBS queue, q1 - 2 to 4 CPU jobs, q2 - 8 to 32 CPU jobs #PBS -q q2 ### Specify required number of CPUs #PBS -l ncpus=18 ### Redirecting Output and Error files #PBS -e /raid2/user/MyWaveSim/out.err #PBS -o /raid2/user/MyWaveSim/out.err ### Mailing ALerts: a -abort, b -begin execution and e -end execution #PBS -m abe #PBS -M user@uq.edu.au # # Setup user paths to run LSM stuff # source /opt/modules/default/init/bash export MODULEPATH=$MODULEPATH:/raid2/tools/modulefiles module load lsm/sgimpi export LSMPYTHONPKGDIR=$LSMDIR/lib/python2.3/site-packages/esys/lsm # # Where we want data to be saved, create $OUTPUTDIR # if it doesn't exist. # export OUTPUTDIR=/raid2/user/MyWaveSim if [ -d $OUTPUTDIR ]; then echo Output dir $OUTPUTDIR exists. else echo Creating $OUTPUTDIR ... mkdir $OUTPUTDIR cd $OUTPUTDIR fi cd $OUTPUTDIR # # Remove any existing data files. # rm srcTo*.txt surf*.txt particle_*.txt out.err out.log # # Use mpirun and dplace to place MPI processes on CPU's. # mpirun -up 128 -np 1 \ `which dplace` -e -c0-31 \ `which mpipython` \ $LSMPYTHONPKGDIR/examples/waveprop/WaveSim.py \ -b[512,320,1] \ -m8000 \ -n16 \ -s0.35 \ -d[4,4,1]
|
|||
|
Trees | Indices | Help |
|
---|
Generated by Epydoc 3.0.1 on Sun Apr 13 05:19:09 2014 | http://epydoc.sourceforge.net |