Page tree
Skip to end of metadata
Go to start of metadata

Running Calculations

In this section, you will learn:

  • How to create and run a simulation
  • How to start the EMPro calculation engine
  • The main factors to consider before beginning a calculation

After the EMPro project setup is complete, it is time to run calculations on the geometry. The Simulations workspace window stores the project simulation(s). From this window, the user creates, queues and runs the simulations.

The actual electromagnetic calculations are not made by the EMPro GUI. Rather, the electromagnetic calculations are run by a separate program called EMPro FDTD once the finished project file has been saved. Usually this process is run from the EMPro GUI, which calls EMPro FDTD as needed. However, the user may also run EMPro FDTD directly from the command line, or from a remote computer. Once the simulations have been run and the desired calculations have completed, the results can be viewed within the EMPro GUI.

Simulations Workspace Window

The Simulations workspace window provides the interface to queue projects to be run with EMPro FDTD, the calculation engine.

The Simulations workspace window

The Simulations workspace window lists the name of every simulation that has been created for the project. Its Status column shows whether the simulation has been created, queued, or completed.

This window also provides the user with the ability to choose how to run the simulation on their computer. Under the Queue drop-down list, specify whether to run the simulation on the CPU or on the hardware card with Acceleware hardware acceleration.

New simulations are created by pressing the New Simulation button in the upper-left corner of this window. The associated definitions are described below.

Creating a New Simulation

Choosing a Source

There are two main options for choosing a source in in EMPro, as seen in the figure below. When a voltage or current source is used as an input, check Use Discrete Sources. Alternatively, the user may also Select External Excitation to use as the source. Use No Sources should be selected in special cases with the Static Solver. Discrete sources and external excitation sources are briefly described below.

Note

For more information on discrete sources, refer to Circuit Component Definition Editor. For more information on external excitations, refer to External Excitation Editor.

Choosing a source for a simulation

Discrete Source

A discrete source is a cell edge on which the electric field is modified by the addition of some type of input waveform. The cell edge can be modified to behave like a voltage or current source. All calculations with discrete source input are performed in total field. Antenna and microwave circuit computations are examples of calculations that may be performed using discrete sources.

Plane Wave

An incident Plane Wave source is assumed to be infinitely far away so that the constant field surfaces are planar and normal to the direction of propagation. All calculations with this plane wave source are performed in scattered-field. Total field values may be saved and are typically more desirable than scattered-field plane wave. Calculations of radar cross section or scattering may be performed using this input.

Note

For important considerations when choosing between scattered and total-field plane wave sources, refer to Plane Wave Excitations.

Gaussian Beam

This choice allows for a focused Gaussian beam source in which the incident electric field has a two-dimensional, radially-symmetric Gaussian distribution in planes normal to the incident direction and converges to maximum intensity at the focus point. As with the plane wave source, all calculations with a Gaussian beam source are performed in scattered field, though total field values may also be saved and displayed also. Unlike the plane wave and discrete sources, the Gaussian beam source requires that the source waveform be sinusoidal. Examples where this type of source is useful include structures used at optical frequencies and situations where it is desired to illuminate only a portion of the geometry.

Parameter Sweep Setup

A parameter sweep can be set up so that a script will loop through a particular parameter multiple times and run a calculation at each iteration. For example, if the parameter is antenna length, a script can loop through various antenna lengths, and run a calculation at each length. The Sweep Type is defined by the values it will collect (Start, Incr., Count values, or Start, End, Count values), in order to generate the Ending Value or Increment values, respectively. A third option is to select Explicit Values for the parameter sweep, which do not necessarily have to be at evenly spaced intervals.

Note

For more information on scripting and to see a sample script for a simple parameter sweep, refer to Scripting Workspace Window.

Setting up a parameter sweep

S-parameters Simulation Setup

To calculate a S-parameter for a project with multiple ports defined, the simulation will consider one port at a time to be the active port. The computation with that source active will provide data for a column of the S-parameter matrix. For example, if port one is excited in a three-port circuit, EMPro will determine S11, S21 and S31. If different ports are to be excited, a separate calculation must be performed with each port active. For example, if the full S-parameter matrix for a two-port problem is desired, two calculations must be performed with a different port active in each. EMPro will save the S-parameters for each run in separate files, differentiated by the active port number.

Note

For more information about the data saved with port sensors, refer to Sensor Tools.

If a parameter sweep is specified in the Setup Parameter Sweep tab for multiple parameters in addition to specifying multiple ports within the Setup S-Parameters tab, the parameter sweep will be performed for each individual port.

Setting up S-parameters

Frequencies of Interest

Within this section, the Frequencies tab specifies whether the simulation is a broadband (transient) or steady-state calculation. For broadband calculations, uncheck the Collect Steady-State Data box at the top of this tab. For steady-state calculations, check this box and choose whether the calculation is to only Use Waveform Frequency or to Use Specified Frequencies of interest. By specifying more than one frequency of interest, the calculation engine will essentially run a separate calculation at each discrete frequency by running DFT, saving each as its own run. This will therefore increase the calculation time in comparison to using only the waveform frequency.

In the Data Storage tab, the user has the ability to specify whether to save temporary data In Memory or On Disk. Saving the data in memory will speed up the calculation because there is no file saving or loading from disk, but it increases the memory requirements.

Also use this tab to designate which data to save for steady-state far zone post-processing. For a broadband excitation, the user has several options. Checking Normalize Fields will match calculated values to a sinusoidal run. Checking Compute Dissipated Power will calculate dissipated power based on electric field and magnetic field samplings. It is recommended to leave this box unchecked unless there is specific interest in dissipated power, because it can increase run time significantly due to sampling data over the entire geometry. Sampling Interval specifies how often to sample a data type. A sampling interval of one provides the most accurate results because it reduces the effects of aliasing.

Note

For more information on viewing far zone post-processing results, refer to Post Processing.

Setting up Frequencies of Interest under the Frequencies and Data Storage tabs

Specifying Field Formulation

The Specify Total/Scattered Field Interfaces definitions are only applicable with Plane Wave sources and are dependent on boundary condition specifications. They regulate how to perform the calculation in certain regions, specifically where a region of total-field is surrounded by a region of scattered-field. The interface between the two regions must be free space. Scattered fields cannot be sampled inside the total-field region and vice versa. Calculations of radar cross section or scattering are based on fields inside the scattered-field region. For non-periodic boundaries, the six sides of the total-field region are defined as eight cells into the FDTD mesh. For these conditions, there is no option to turn the interface off.

Setting up Total/Scattered Field Formulation

There are two sets of X, Y, and Z boundaries listed within this dialog. The checkboxes listed in the Desired dialog are available for users to indicate which interfaces will be turned on. Depending on the boundary conditions, however, the selected Desired definitions may not be applicable or may have to be applied in conjunction with other definitions. Thus, the Effective dialog displays the actual definitions that will be applied during the calculation.

When Periodic boundaries are specified, certain sides for the interface may be turned off and the total-field region may extend to the boundary using this definition. Periodic boundaries may be useful for applications such as optics where small geometries are repeated over and over again. The figure below illustrates an example in which the outer boundaries have been set to Periodic in the the Y and Z directions, and the total/scattered field interface has been turned off (unchecked) in the lower X , upper and lower Y , and upper and lower Z directions.

The interface may be turned off for problems that use periodic boundary conditions

Specifying Termination Criteria

Convergence and stability are essential in determining whether a calculation will yield usable results. Convergence in a broadband calculation is met when all electromagnetic energy has dissipated to essentially zero. There are several options in EMPro to define termination criteria to ensure proper convergence had been reached.

Selecting termination criteria within the Simulations workspace window

The most basic method of ending a calculation is defining a value in the Maximum Timesteps definition. Once the defined number of timesteps has completed, the calculation will stop. It is important to note that the calculation will terminate regardless of whether or not convergence has been met, so setting this definition to a proper value is important. If it is too low, results will be of no use.

Selecting the Detect Convergence check-box will automatically stop the calculation if slow convergence is detected, regardless of whether the number of maximum timesteps have completed. Due to numerical noise in the calculation, there may be a trivial amount of electromagnetic energy, even after the calculation has converged. The value defined in the Threshold dialog dictates when the calculation has reached an acceptable value to assume convergence.

Note

A general rule of thumb is that the values should have diminished by at least 30 dB or 1/1000th from the peak values.

For sinusoidally-excited problems, typical values for this setting range from -55 dB to -25 dB depending on the level of accuracy versus runtime desired. For instance, if high accuracy S-parameters are the goal, then the convergence threshold should be set to lower than -30 dB. If however, the user wants to view antenna patterns, -30 dB to -35 dB is suitable. The trade-off here is run time for accuracy. In general, for sinusoidally excited problems, using automatic convergence with a threshold of about -35 dB will produce very accurate results and will run in the shortest possible time to reach this level.

If a calculation is finished but convergence has not been reached, the output from most calculations will be meaningless. The only option is to decrease the convergence Threshold or increase the number of Maximum Timesteps in the calculation and run it again. If a resonance is occurring at a frequency beyond the range of interest, and a broadband input is used, the input waveform can be modified to limit the frequency content and eliminate the resonance. If the resonance is in band, or a sinusoidal input is used, then this is not applicable and more timesteps must be run or a lower convergence threshold must be chosen.

Advanced Options

There are several options available in the Advanced button of this window that allow for more specific termination criteria.

Selecting the Flatline Detection check-box will stop the calculation if a slow convergence is detected. This may occur if the user sets the convergence threshold to a very low value (e.g. < -50 dB). In this case the calculation may converge but to a level higher than specified.

Note

To prevent false convergence, "slow" convergence can only be detected once the convergence level has reached at least -40 dB.

For a steady-state calculation, convergence is reached when near-zone data shows a constant amplitude sine wave - when all transients have died down and the only variation left is sinusoidal. In this case "convergence" is tested on the average electric field in the space for its deviation from a pure sine wave. If Detect Convergence is turned on, EMPro automatically places points throughout the space for this purpose. It is particularly important to monitor the results inside high permittivity dielectrics since the field propagation in these materials is much slower than in free space.

To ensure that steady-state calculations converge, EMPro will enable you to control the Sampling Interval and Sampling Density of the sample points. The temporal Sampling Interval definition is used to control the interval for which convergence is tested during calculations with broadband (pulse) excitations.

Note

Setting this value to 100 or 200 timesteps is typical. Setting this value to much less than that increases the computational overhead a small amount.

Sampling Density is used to control how many spatial samples are used to determine convergence. The sample points are equally spaced in all three dimensions of the grid. Low density samples every 4th point in each dimension, while High density samples every point in each dimension. For very low frequency problems or where the number of timesteps per RF cycle is greater than 200 (e.g., very small cells with a low frequency excitation), this should be set to LOW. For moderate frequencies or where the number of timesteps per RF cycle is less than 200 but greater than 100 this should be set to Medium. For high frequency problems where the number of timesteps per RF cycle is less than 100, a High setting gives the best accuracy. This setting is for both broadband as well as sinusoidal excitations.

Notes

The Notes section is simply a tool for users to add any notes to be attached to a project. The notes will be available in the Simulations workspace window after the simulation is created.

Queuing and Running Simulations

After defining the necessary components of the simulation in the New Simulation dialog described above, select Create And Queue Simulation and the main window will show all of the queued and completed simulations as seen in the figure below. Only one simulation can be run at one time, so as soon as a calculation is complete, another queued simulation will begin.

Running a calculation

Below this dialog are three tabs:

  • Summary, where a basic summary of the calculation is provided, as seen in below.
  • Notes, which simply documents any notes that were added by the user in the New Simulation dialog.
  • Output, which provides the output generated by EMPrFDTD. Statistics such as the percentage of completion, current timestep, convergence, time elapsed, etc, are listed for every simulation in this tab (including output information for every parameter swept).
    The calculation Summary tab

Starting the Calculation Engine

There are two ways to start the EMPrFDTD engine. The first way is to launch it from EMPro in the Simulations workspace window. The second way is to start the calculation from a command line. This is preferable for calculations that require a large amount of memory in order to free memory used by the interface.

Running EMPrFDTD Within the EMPro Interface

Calculations launched from EMPro are sent to the EMPrFDTD engine from the Simulations workspace window. A list of every simulation that has been created within the New Simulation dialog is listed in the main window. When a simulation is ready to run, send it to EMPrFDTD by selecting the simulation and clicking the Add To Queue button, located to the right of Selected Simulation at the top of the window. Although multiple simulations can be queued at once, only one simulation is run in EMPrFDTD at a time. Once the Play button is pressed, each simulation will run one at a time until all of the queued simulations have terminated.

Note

For more on creating a new simulation, refer to Creating a New Simulation.

Running EMPrFDTD From the Command Line

In some cases, it is useful to run EMPrFDTDfrom a command line. This will prevent EMPro from overloading since it will not be allocating memory for EMPrFDTD in addition to its normal memory requirements. This is not an issue for most calculations, but it may cause problems when the memory required to run a calculation approaches the limitations of computer memory. In this case, closing EMPro and running EMPrFDTD from the command line will free up any memory used by EMPro.

To run EMPrFDTD, navigate to the appropriate project folder and run the command:

Windows: emprfdtd.exe [options]

Mac OS X/Linux: emprfdtd [options]

For normal operation no [options] need be specified, but the table below lists several options that may be of interest.

Command Line Options

option

Function

v

Verbose mode, prints progress of calculation.

proc N

Sets the number of processors to use for a calculation with the multi-processor module (optional).

loadstatice

Loads fields saved by the static voltage solver.

savestaticeonly

Saves fields by the static voltage solver and terminates before time-stepping.

savestatice

Saves fields by the static voltage solver and continues timestepping.

hardware

Uses Acceleware hardware acceleration; in the case that an error occurs initializing Acceleware, the engine uses the software to run calculation.

hardwareonly

Uses Acceleware hardware acceleration; in the case that an error occurs initializing Acceleware, the calculation is terminated.

forcetemporarydatatomemory

Forces steady-state data to be stored in memory rather than in an external file.

saronly

Calculation skips time-stepping and only performs post-processing of SAR data.

Running the Calculation Remotely

Running a remote calculation is an alternative way to potentially speed up calculation time and free local computer resources for other operations. This section will explain how to copy files and launch the calc engine on a remote computer.

Note

If your EMPro project is stored on a file system which is remotely accessible, you can avoid the process of copying files back and forth from the remote machine. Simply log into the remote machine, navigate to the simulation directory and launch the calc engine.

Running Simulations Remotely

  1. Create the simulation. After creating the simulation from the Simulations workspace window, click the Create Simulation Only button to save the simulation to the Simulations folder where your EMPro project is stored.

    You can find the path to this directory in the first line of text under the Summary tab


    Note

    For more information about setting up a simulation in EMPro, refer to Simulations Workspace Window.

  2. Copy files to the remote machine. For simplicity, you can copy the whole Simulations directory (you'll need to copy it recursively, since it contains subdirectories). However, only a few files in this directory are required by the calculation engine:

    project.xsim
    Run*/geometry.input
    Run*/mesh.input
    Run*/project.input


    Note

    The * in the file name represents the simulation number, (e.g., Run0001/geometry.input, Run0002/geometry.input, etc).


    For Windows users, it may be easiest to zip each file individually and transfer them to the remote machine over an ftp connection.
    For Mac OS X users, you can save these files in a compressed archive with the following command:

    tar -czf inputFilesForCalcEngine.tar.gz project.xsim Run*/geometry.input Run*/mesh.input Run*/project.input

    Then copy the inputFilesForCalcEngine.tar.gz file to the remote machine and extract it using:

    tar -xzf inputFilesForCalcEngine.tar.gz
  3. Run the simulation Log in to the remote machine, change directories to the Simulations folder, and run EMPrFDTD for the project.

    Note

    For more information on running the calc engine from the command line, refer to Running EMPrFDTD From the Command Line.


  4. Copy files back to local machine. In order to view your results from the EMPro GUI, you'll need to copy all of the Simulations files from the remote machine to your local machine, with the exception of the files listed above. That is, after running the simulation on the remote machine, all new files should be copied back into the Simulations folder on your local machine. The tar utility makes this easy. On the remote machine, change directories to the Simulations directory and run

    tar -czf outputFilesFromCalcEngine.tar.gz -exclude=project.xsim -exclude=Run*/geometry.input -exclude=Run*/mesh.input -exclude=Run*/project.input *

    For Windows users, zip the outputFilesFromCalcEngine.tar.gz file and send it back to your local machine over the ftp connection. You can then extract the files using Winzip. Overwrite any old files with newer versions of the same file.
    For Mac OS X users, copy the outputFilesFromCalcEngine.tar.gz file to your local machine, change directories and extract it using:

    tar -xzf outputFilesFromCalcEngine.tar.gz

  5. Refresh results. To make your results available from the EMPro GUI, click Refresh in the Results workspace window.

Remote SAR Post-processing

  1. Setting up post-processing. After requesting post-processing from the RESULTS workspace window, in the subsequent dialog box, tell the application to not run the calculation right away.
  2. Copy raw SAR data, calc engine input files, and SAR request files. The following files are required for post-processing in the Simulations folder on the remote machine:

    project.xsim
    Run*/geometry.input
    Run*/mesh.input
    Run*/project.input
    Run*/SteadyStateOutput//SAR_Raw_Sensor..sar.gz
    Run*/SteadyStateOutput/*/SAR_Raw_Sensor.stats
    Run*/request.sar

    The * in the file name represents the simulation number, (e.g., Run0001/geometry.input, Run0002/geometry.input, etc).


    For Windows users, it may be easiest to zip each file individually and transfer them to the remote machine over an ftp connection.

    For Mac OS X users, change the directory to Simulations and run the following command on your local machine:

    tar -czf sarRawData.tar.gz project.xsim Run*/geometry.input Run*/mesh.input Run*/project.input Run*/SteadyStateOutput//.sar.gz Run*/request.sar

    Then copy the sarRawData.tar.gz file to the remote machine and extract it using:

    tar -zxf sarRawData.tar.gz

  3. Run SAR averaging on the remote machine. On the remote machine, change directories to the Simulations directory, and execute the calc engine application using the -saronly command line flag.

    Note

    For more information on running the calc engine from the command line, refer to Running EMPrFDTD From the Command Line.


  4. Copy SAR results back to the local project. If you've already copied your other results back to the local project, you'll need to package up your SAR results and copy them back. From the remote machine, copy the following files to the Simulations directory on the local machine:

    Run*/status
    Run*/SteadyStateOutput/*/*gsar.gz
    Run*/SteadyStateOutput//.infosar1g
    Run*/SteadyStateOutput//.infosar10g
    Run*/SteadyStateOutput//.infoseq
    Run*/SteadyStateOutput//.*gssq
    Run*/SteadyStateOutput//.stats

    Then run the following command:

    tar -czf sarResults.tar.gzRun*/status Run*/SteadyStateOutput/*/*gsar.gz
    Run*/SteadyStateOutput//.infosar1g
    Run*/SteadyStateOutput//.infosar10g Run*/SteadyStateOutput//.infoseq
    Run*/SteadyStateOutput//.gssq Run/SteadyStateOutput//.stats

    Note

    Notice that this command sequence retrieves all of the SAR averaging results, including those done with your initial calc engine run.


    For Windows users, zip the sarResults.tar.gz file and send it back to your local machine over the ftp connection. You can then extract the files using Winzip. Overwrite any old files with newer versions of the same file.

    For Mac OS X users, copy the sarResults.tar.gz file to your local machine, change directories and extract it using:

    tar -xzf sarResults.tar.gz

  5. Refresh results. To make your results available from the EMPro GUI, click Refresh in the Results workspace window.

Calculation Considerations

The calculation portion of EMPro may be quite lengthy depending on the application. A few guidelines are provided here for estimating computer resources, monitoring the progress of the calculation, and avoiding calculation instability.

Note

For information about defining proper termination criteria to ensure that the calculation has finished, refer to Creating a New Simulation .

Computer Resources Estimation

EMPrFDTD will give a time estimate while the calculation is running. This is recalculated every time EMPrFDTD updates its status based on how much time passed since the last update and the remaining number of timesteps. It is not a completely accurate estimation since it does not consider data such as near-field samplings which may only be saved during certain portions of the calculation. Also, the estimate does not include any post-processing which may occur. Of special note are the SAR averages since they are computed in post-processing and may require a significant amount of calculation time.

Note

A quick way to estimate the amount of memory that EMPrFDTD will need for a given problem is to multiply the number of cells in the geometry by 27. If magnetic materials are included in the geometry (any material with non-free space permeability), multiply by 30, rather than 27. The resulting number is the approximate number of bytes needed to calculate the project.

Multi-Processing Modules

EMPrFDTD has the ability to do both threaded and Message Passing Interface (MPI) calculations. Both of these capabilities are optional features which may be added to the calculation engine.

Multi-threaded calculations use shared memory and are intended for computers with multiple processors and/or cores on a single motherboard. The overhead for the multi-threading routines can cause very small calculations to run slower when more than one thread is selected.

Note

In general, the multi-threaded option should not be used when the number of FDTD cells in the geometry is less than one million.

The number of cells is computed simply from the X , Y , and Z dimensions (in cells) of the geometry space. The number of threads used for a calculation is defined in the Queue drop-down box of the Simulations workspace window, as seen below.

Specifying calculation engine threads

Specify the number of processors that are to be used for the calculation under Maximum # Of Threads.

To specify the number of threads from the command line, the "-proc N" option should be used, where N is the number of threads.

Note

For a summary of command-line options, refer to Running EMPrFDTD From the Command Line.

MPI calculations can be executed on cluster of computers that are connected by a network. At present, only Linux computers are supported and each computer must have the same MPI libraries installed. To run an MPI calculation, see the documentation for the MPI library that is installed on all machines in the cluster. This documentation will give instructions on how to start an application using the MPI tools.

Acceleware Hardware Acceleration

EMPro has optional Acceleware hardware acceleration. The hardware is available in the form of graphics cards, which can replace existing graphics cards or act from a stand-alone computer. When this hardware is available, it may be enabled within the Simulations workspace window under the Queue drop-down box. To enable it from the command line, the "-hardware" option should be used.

Monitoring Calculation Progress

While EMProFDTD is running, its progress will be updated periodically. When launched from the EMPro interface, the progress of the calculation will be printed in the Output tab located within the Simulations workspace window, as seen in the figure below. When running from a command line, the progress will be printed to the window that was used to start the calculation if the "-v" option is used.

Monitoring calculation progress in the Output tab

Calculation Stability

Improper application of the outer boundaries can lead to unstable calculations. Typically the stability of a calculation depends on a few simple guidelines involving boundaries and source placement. Because an absorbing boundary condition like LIAO tries to simulate free space, it requires that a certain amount of continuity be present in the cells leading to the boundary. The cross section at a boundary must be the same for at least 10 cells in from the boundary.

Note

For many problems, a free-space border of 10-20 cells is the best way to ensure stability and accurate performance of the outer boundary.

Another rule for stability is that no source can be placed within 10 cells of an absorbing boundary. An unstable calculation is easily determined by viewing the line plots of time domain data or by viewing the field snapshots. When automatic convergence is enabled, EMProFDTD can automatically detect an unstable simulation and terminate it.

In some cases instability can be introduced by a frequency-dependent material. If such materials are used in the calculation and an instability results, it may be necessary to change the material parameters or reduce the calculation timestep.

  • No labels