GeezerVisonThe World Wide Web, restored to 800 pixels of glory. |
| Home|About|Gaming cabinet|Archives|How it works|Starter sites |
GeezerVison is showing github.com — PRMS-Python/prms_python/simulation.py at master · PRMS-Python/PRMS-Python Expand file tree/ Copy pathsimulation.pyMore file actions 411 lines (330 loc) · 15 KB / Copy pathsimulation.pyFile metadata and controls411 lines (330 loc) · 15 KB 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 # -*- coding: utf-8 -*- """ simulation.py -- Contains ``Simulation`` and ``SimulationSeries`` classes and associated functions for managing PRMS simulations at a low level. """ from __future__ import print_function import glob import multiprocessing as mp import os import shutil import subprocess import time from .data import Data from .parameters import Parameters from .util import load_statvar OPJ = os.path.join class SimulationSeries(object): ''' Series of simulations all to be run through a common interface. Utilizes :class:`multiprocessing.Pool` class to parallelize the execution of series of PRMS simulations. SimulationSeries also allows the user to define the PRMS executable command which is set to "prms" as default. It is best to add the prms executable to your $PATH environment variable. Each simulation that is run through ``SimulationSeries`` will follow the strict file structure as defined by :func:`Simulation.run()`. This class is useful particularly for creating new programatic workflows not provided yet by PRMS-Python. Arguments: simulations (list or tuple): list of :class:`Simulation` objects to be run. Example: Lets say you have already created a series of PRMS models by modifying the input climatic forcing data, e.g. you have 100 *data* files and you want to run each using the same *control* and *parameters* file. For simplicity lets say there is a directory that contains all 100 *data* files e.g. data1, data2, ... or whatever they are named and nothing else. This example also assumes that you want each simulation to be run and stored in directories named after the *data* files as shown. >>> data_dir = 'dir_that_contains_all_data_files' >>> params = Parameters('path_to_parameter_file') >>> control_path = 'path_to_control' >>> # a list comprehension to make multiple simulations with >>> # different data files, alternatively you could use a for loop >>> sims = [ Simulation.from_data ( Data(data_file), params, control_path, simulation_dir='sim_{}'.format(data_file) ) for data_file in os.listdir(data_dir) ] Next we can use ``SimulationSeries`` to run all of these simulations in parrallel. For example we may use 8 logical cores on a common desktop computer. >>> sim_series = SimulationSeries(sims) >>> sim_series.run(nprocs=8) The ``SimulationSeries.run()`` method will run all 100 simulations where chunks of 8 at a time will be run in parrallel. Inputs and outputs of each simulation will be sent to each simulation's ``simulation_dir`` following the file structure of :func:`Simulation.run()`. Note: The :class:`Simulation` and :class:`SimulationSeries` classes are low-level in that they alone do not create metadata for PRMS simulation scenarios. In other words they do not produce any additional files that may help the user know what differs between individual simulations. ''' def __init__(self, simulations): self.series = list(simulations) def run(self, prms_exec='prms', nproc=None): """ Method to run multiple :class:`Simulation` objects in parrallel. Keyword Arguments: prms_exec (str): name of PRMS executable on $PATH or path to executable nproc (int or None): number of logical or physical processors for parrallel execution of PRMS simulations. Example: see :class:`SimulationSeries` Note: If ``nproc`` is not assigned the deault action is to use half of the available processecors on the machine using the Python :mod:`multiprocessing` module. """ if not nproc: nproc = mp.cpu_count() // 2 pool = mp.Pool(processes=nproc) pool.map(_simulation_runner, self.series) pool.close() return self def outputs_iter(self): ''' Return a :class:`generator` of directories with the path to the ``simulation_dir`` as well as paths to the *statvar.dat* output file, and *data* and *parameters* input files used in the simulation. Yields: :obj:`dict`: dictionary of paths to simulation directory, input, and output files. Example: >>> ser = SimulationSeries(simulations) >>> ser.run() >>> g = ser.outputs_iter() Would return something like >>> print(g.next()) { 'simulation_dir': 'path/to/sim/', 'statvar': 'path/to/statvar', 'data': 'path/to/data', 'parameters': 'path/to/parameters' } ''' dirs = list(s.simulation_dir for s in self.series) print(dirs) return ( { 'simulation_dir': d, 'statvar': OPJ(d, 'outputs', 'statvar.dat'), 'data': OPJ(d, 'inputs', 'data'), 'parameters': OPJ(d, 'inputs', 'parameters') } for d in dirs ) def __len__(self): return len(list(self.outputs_iter())) def _simulation_runner(sim): sim.run(prms_exec='prms') class Simulation(object): """ Class that runs and manages file structure for a single PRMS simulation. The ``Simulation`` class provides low-level managment of a PRMS simulation by copying model input files from ``input_dir`` argument to an output dir ``simulation_dir``. The file stucture for an individual simulation after calling the ``run`` method is simple, two subdirectories "inputs" and "outputs" are created under ``simulation_dir`` and the respective input and output files from the current PRMS simulation are transfered there after the ``Simulation.run()`` method is called which executes the PRMS model, (see examples below in :func:`Simulation.run`). A ``Simulation`` instance checks that all required PRMS inputs (control, parameters, data) exist in the expected locations. If simulation_dir is provided and does not exist, it will be created. If it does exist it will be overwritten. Keyword Arguments: input_dir (str): path to directory that contains control, parameter, and data files for the simulation simulation_dir (str): directory path to bundle inputs and outputs Example: see :func:`Simulation.run()` Raises: RuntimeError: if input directory does not contain a PRMS *data*, *parameters*, and *control* file. """ def __init__(self, input_dir=None, simulation_dir=None): # check if model input paths exist idir = input_dir self.input_dir = idir self.simulation_dir = simulation_dir if idir is not None: self.control_path = os.path.join(idir, 'control') self.parameters_path = os.path.join(idir, 'parameters') self.data_path = os.path.join(idir, 'data') if not os.path.exists(self.control_path): raise RuntimeError('Control file missing from ' + idir) if not os.path.exists(self.parameters_path): raise RuntimeError('Parameter file missing from ' + idir) if not os.path.exists(self.data_path): raise RuntimeError('Data file missing from ' + idir) # build output (simulation_dir) and move input files there if simulation_dir is not None: self.simulation_dir = simulation_dir if simulation_dir and simulation_dir != input_dir: if os.path.exists(simulation_dir): shutil.rmtree(simulation_dir) os.mkdir(simulation_dir) shutil.copy(self.control_path, simulation_dir) shutil.copy(self.data_path, simulation_dir) shutil.copy(self.parameters_path, simulation_dir) self.control_path = os.path.join(simulation_dir, 'control') self.parameters_path = os.path.join(simulation_dir, 'parameters') self.data_path = os.path.join(simulation_dir, 'data') else: self.control_path = None self.parameters_path = None self.data_path = None self.simulation_dir = None self.has_run = False @classmethod def from_data(cls, data, parameters, control_path, simulation_dir): ''' Create a ``Simulation`` from a :class:`Data` and :class:`Parameter` object, plus a path to the *control* file, and providing a ``simulation_dir`` where the simulation should be run. Arguments: data (:class:`Data`): ``Data`` object for simulation parameters (:class:`Parameters`): ``Parameters`` object for simulation control_path (str): path to control file simulation_dir (str): path to directory where simulations will be run and where input and output will be stored. If it exists it will be overwritten. Returns: :class:`Simulation` ready to be run using ``simulation_dir`` for inputs and outputs Example: >>> d = Data('path_to_data_file') >>> p = Parameters('path_to_parameters_file') >>> c = 'path_to_control_file' >>> sim_dir = 'path_to_create_simulation' >>> sim = Simulation.from_data(d, p, c, sim_dir) >>> sim.run() Raises: TypeError: if ``data`` and ``parameters`` arguments are not of type :class:`Data` and :class:`Parameters` ''' if not isinstance(data, Data): raise TypeError('data must be instance of Data') if not isinstance(parameters, Parameters): raise TypeError('parameters must be instance of Parameters, not '\ + str(type(parameters))) if os.path.exists(simulation_dir): shutil.rmtree(simulation_dir) os.makedirs(simulation_dir) sim = cls() sim.simulation_dir = simulation_dir sd = simulation_dir data_path = OPJ(sd, 'data') data.write(data_path) params_path = OPJ(sd, 'parameters') parameters.write(params_path) shutil.copy(control_path, OPJ(sd, 'control')) return sim def run(self, prms_exec='prms'): """ Run a ``Simulation`` instance using PRMS input files from ``input_dir`` and copy to the ``Simulation`` file structure under ``simulation_dir`` if given, otherwise leave PRMS input output unstructured and in ``input_dir`` This method runs a single PRMS simulation from a ``Simulation`` instance, waits until the process has completed and then transfers model input and output files to respective newly created directories. See example of the file structure that is created under different workflows of the ``run`` method below. Keyword Arguments: prms_exec (str): name of PRMS executable on $PATH or path to executable Examples: If we create a :class:`Simulation` instance by only assigning the ``input_dir`` argument and call its ``run`` method the model will be run in the ``input_dir`` and all model input and output files will remain in ``input_dir``, >>> import os >>> input_dir = os.path.join( 'PRMS-Python', 'prms_python', 'models', 'lbcd' ) >>> os.listdir(input_dir) ['data', 'data_3deg_upshift', 'parameters', 'parameters_adjusted', 'control'] >>> sim = Simulation(input_dir) >>> sim.run() >>> os.listdir(input_dir) # all input and outputs in input_dir ['data', 'data_3deg_upshift', 'parameters', 'parameters_adjusted', 'control', 'statvar.dat', 'prms_ic.out', 'prms.out' ] Instead if we assigned a path for ``simulation_dir`` keyword argument and then called ``run``, i.e. >>> sim = Simulation(input_dir, 'path_simulation') >>> sim.run() the files structure for the PRMS simulation created by ``Simulation.run()`` would be:: path_simulation ├── inputs │ ├── control │ ├── data │ └── parameters └── outputs ├── data_3deg_upshift ├── parameters_adjusted ├── prms_ic.out ├── prms.out └── statvar.dat Note: As shown in the last example, currently the ``Simulation.run`` routine only recognizes the *data*, *parameters*. and *control* file as PRMS inputs, all other files found in ``input_dir`` before *and* after normal completion of the PRMS simulation will be transferred to ``simulation_dir/outputs/``. """ cwd = os.getcwd() if self.simulation_dir: os.chdir(self.simulation_dir) else: os.chdir(self.input_dir) p = subprocess.Popen( prms_exec + ' control', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE ) prms_finished = False checked_once = False while not prms_finished: if not checked_once: p.communicate() checked_once = True poll = p.poll() prms_finished = poll >= 0 self.has_run = True # avoid too many files open error p.stdout.close() p.stderr.close() if self.simulation_dir: os.mkdir('inputs') os.mkdir('outputs') shutil.move('data', 'inputs') shutil.move('parameters', 'inputs') shutil.move('control', 'inputs') # all remaining files are outputs for g in glob.glob('*'): if not os.path.isdir(g): shutil.move(g, 'outputs') os.chdir(cwd) |
| You are visitor 01402503 — GeezerVison does not store the pages you look up. Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required. |