GEMINI

The GEMINI model (Geospace Environment Model of Ion-Neutral Interactions) is a three-dimensional ionospheric fluid-electrodynamic model written (mostly) in object-oriented fortran (2008+ standard). GEMINI is used for various scientific studies including:

• effects of auroras on the terrestrial ionosphere
• natural hazard effects on the space environment
• effects of ionospheric fluid instabilities on radio propagation

The detailed mathematical formulation of GEMINI is included in GEMINI-docs. A subroutine-level set of inline generated documentation describing functions of individual program units is given via source code comments which are rendered as webpages. GEMINI uses generalized orthogonal curvilinear coordinates and has been tested with dipole and Cartesian coordinates.

Generally, the Git main branch has the current development version and is the best place to start, while more thoroughly-tested releases happen regularly. Specific releases corresponding to published results are generally noted in the corresponding journal article.

Bug Reporting

The GEMINI development teams values input from our user community, particulary in the form of reporting of errors. These allow us to insure that the code functions properly for a wider range of conditions, platforms, and use cases than we are otherwise able to directly test.

Please open a GitHub Issue if you experience difficulty with GEMINI. Try to provide as much detail as possible so we can try to reproduce your error.

Platforms

Gemini is intended to be OS / CPU arch / platform / compiler agnostic. Operating system support includes: Linux, MacOS, and Windows. CPU arch support includes: Intel, AMD, ARM, IBM POWER, Cray and more. GEMINI can run on hardware ranging from a Raspberry Pi to laptop to a high-performance computing (HPC) cluster. Generally speaking one can run large 2D or modest resolution 3D simulations (less than 10 million grid points) on a quad-core workstation, with some patience.

For large 3D simulations (many tens-to-hundreds of millions of grid points), GEMINI is best run in a cluster environment or a very "large" multi-core workstation (e.g. 16 or more cores). Runtime depends heavily on the grid spacing used, which determines the time step needed to insure stability, For example we have found that a 20M grid point simulations takes about 4 hours on 72 Xeon E5 cores. 200M grid point simulations can take up to a week on 256 cores. It has generally been found that acceptable performance requires > 1GB memory per core; moreover, a large amount of storage (hundreds of GB to several TB) is needed to store results from large simulations.

Quick start

To build Gemini and run self-tests takes about 10 minutes on a laptop. Gemini3D uses several external libraries that are built as a required one-time procedure. Gemini3D works "offline" that is without internet once initially setup.

Requirements:

• C++17 and Fortran 2008 compiler. See compiler help for optional further details.
  • GCC 8 and newer
  • Clang 7 and newer
  • Intel oneAPI
  • Cray with GCC backend
• Python and/or MATLAB for scripting front- and back-ends
• CMake: if your CMake is too old, download or python -m pip install cmake
• MPI: any of OpenMPI, IntelMPI, MPICH, MS-MPI. See MPI help if needed. Without MPI, Gemini3D uses one CPU core only, which runs much more slowly than with MPI.

Gemini3D setup

Install Gemini3D prerequisite libraries. This is a one-time process used by any Gemini3D builds you do (or other programs). If your Python is too old, it will also install a local Python interpreter.

git clone https://github.com/gemini3d/external.git

cmake -B external/build -S external -DCMAKE_INSTALL_PREFIX=~/libgem
# the libgem name/location is arbitrary

cmake --build external/build

Note: if the external build fails and you don't see the error message, run the build command cmake --build external/build once or twice more see if you can scroll back to it. The build is massively in parallel and so sometimes the error is many screens back up.

Set environment variables CMAKE_PREFIX_PATH and edit PATH environment variable as follows. On Linux add to ~/.bashrc, or on MacOS add to ~/.zshrc:

export CMAKE_PREFIX_PATH=~/libgem
export PATH=$HOME/libgem/bin:$PATH

On Windows from Windows Terminal / Command Prompt:

rundll32 sysdm.cpl,EditEnvironmentVariables

Then add "New" variable CMAKE_PREFIX_PATH with value ~/libgem. "Edit" variable Path with New value %USERPROFILE%\libgem\bin.

Build the Gemini3D code

```sh
git clone https://github.com/gemini3d/gemini3d.git

cd gemini3d

cmake -B build -DCMAKE_PREFIX_PATH=~/libgem

cmake --build build --parallel
```

Non-default build options may be used.

GEMINI has self tests that compare the output from a "known" test problem to a reference output. To verify your GEMINI build, run the self-tests:

ctest --test-dir build

Offline HPC batch CTest

Note: some HPC systems only have internet when on a login node, but cannot run MPI simulations on the login node. Batch sessions, including interactive, may be offline. To run CTest in such an environment, download the data once from the login node:

ctest --test-dir build --preset download

then from an interactive batch session, run the tests:

ctest --test-dir build --preset offline

GEMINI Numerical Library Dependencies

For various numerical solutions Gemini relies on:

• LAPACK
• scalapack
• MUMPS

For file input/output we also use:

• hdf5
• h5fortran
• zlib

Running GEMINI from a Shell Environment

For basic operations the GEMINI main program simply needs to be run from the command line with arguments corresponding to to the number of processes to be used for the simulation, the location where the input files are and where the output files are to be written:

mpiexec -np <number of processors>  build/gemini.bin <output directory>

for example:

mpiexec -np 4 build/gemini.bin ~/mysim3d/arecibo

GEMINI can also be run via scripting frontends, e.g. python -m gemini3d.run -np options.

Advanced Command Line Options

By default, only the current simulation time and a few other messages are shown to keep logs uncluttered. gemini.bin command line options include:

-d | -debug : print verbosely -- could be 100s of megabytes of text on long simulation for advanced debugging.

-nooutput : do not write data to disk. This is for benchmarking file output time, as the simulation output is lost, so this option would rarely be used.

-out_format : normally Gemini reads and writes data files in the same format (HDF5, NetCDF4). This option allow one to read in one format (say NetCDF4) while writing HDF5.

• h5: HDF5 output (most commonly used)
• nc: NetCDF4 output
• dat: raw binary output (not recommended, doesn't support newer features)

-manual_grid <# x2 images> <# x3 images> : forces the code to adopt a specific domain decomposition in x2 and x3 by using the integers given. If not specified the code will attempt to find its own x2,x3 decomposition. The number of grid points in x2 and x3 must be evenly divisible by the number of user-specified images in each direction, respectively.

-dryrun : only run the first time step, do not write any files. This can be useful to diagnose issues not seen in unit tests, particularly issues with gridding. It runs in a few seconds or less than a minute for larger sims, something that can be done before queuing an HPC job.

Running GEMINI through Scripting Environments

If you prefer to issue the GEMINI run command through a scripting environment you may do so (via python) in the following way:

1. make a config.nml with desired parameters for an equilibrium sim.

2. run the equilibrium sim:

   python -m gemini3d.run /path_to/config_eq.nml /path_to/sim_eq/

3. create a new config.nml for the actual simulation and run

   python -m gemini3d.run /path_to/config.nml /path_to/sim_out/

Input file format

See Readme_input for details on how to prepare input data for GEMINI. Generally speaking there are python and MATLAB scripts available in the mat_gemini and pygemini repositories that will save data in the appropriate format once generated.

Loading and plotting output

GEMINI uses Python for essential interfaces, plotting and analysis. Matlab scripts relevant to Gemini to mat_gemini repo.

Only the essential scripts needed to setup a simple example, and plot the results are included in the main GEMINI repository. The Gemini-scripts and Gemini-examples contain scripts used for various published and ongoing analyses.

See Readme_output for a description of how to load the simulation output files and the different variable names, meanings, and units.

Computing Magnetic Field Perturbations

An auxiliary program, magcalc.f90, can be used to compute magnetic field perturbations from a complete disturbance simulation. See Readme_magcalc for a full description of how this program works.

List of other associated Readmes

1. Readme_output - information about data included in the output files of a GEMINI simulation
2. Readme_input - information on how input files should be prepared and formatted.
3. Readme_compilers - details regarding various compilers
4. Readme_cmake - cmake build options
5. Readme_docs - information about model documentation
6. Readme_mpi - help with mpi-related issues
7. Readme_magcalc - some documentation for the magnetic field calculation program
8. Readme_VEGA - information on how to deploy and run GEMINI on ERAU's VEGA HPC system.
9. Readme_prereqs - details on how to install prerequisites on common platforms.

Known limitations and issues of GEMINI

1. Generating equilibrium conditions can be a bit tricky with curvilinear grids. A low-res run can be done, but it will not necessary interpolate properly onto a finer grid due to some issue with the way the grids are made with ghost cells etc. A workaround is to use a slightly narrower (x2) grid in the high-res run (quarter of a degree seems to work most of the time).

2. Magnetic field calculations on an open 2D grid do not appear completely consistent with model prototype results; although there are quite close. This may have been related to sign errors in the FAC calculations - these tests should be retried at some point.

3. Occasionally MUMPS will throw an error because it underestimated the amount of memory needed for a solve. If this happens a workaround is to add this line of code to the potential solver being used for your simulations. If the problem persists try changing the number to 100.

   mumps_par%ICNTL(14)=50

4. There are potentially some issues with the way the stability condition is evaluated, i.e. it is computed before the perp. drifts are solved so it is possible when using input data to overrun this especially if your target CFL number is > 0.8 or so. Some code has been added as of 8/20/2018 to throttle how much dt is allowed to change between time steps and this seems to completely fix this issue, but theoretically it could still happen; however this is probably very unlikely.

5. Occasionally one will see edge artifacts in either the field -aligned currents or other parameters for non-periodic in x3 solves. This may be related to the divergence calculations needed for the parallel current (under EFL formulation) and for compression calculations in the multifluid module, but this needs to be investigated further... This do not appear to affect solutions in the interior of the grid domain and can probably be safely ignored if your region of interest is sufficiently far from the boundary (which is always good practice anyway).

6. Occasionally on Windows you may get a system error code 0xc0000005 when trying to run Gemini. This typically requires rebooting the Windows computer. If this is annoying, please let us know--it happens rarely enough that we're not sure if it's a Microsoft MPI bug or something else.