GooFit / GooFit

• CMake 3.9+
• CUDA 8.0+ (with caveats below)
  • CUDA 8: Supported
  • CUDA 9.2, 10.0: Some warnings from Eigen, supported
  • CUDA 9.0, 9.1: Buggy, see known issues
  • CUDA 10.1, 10.2: Not yet supported due to Thrust 1.8 incompatibility
• An nVidia GPU supporting compute capability at least 3.0 (3.5+ recommended)

• A compiler supporting OpenMP and C++11 (GCC 4.8+, Clang, and Intel 17 tested, GCC 4.7 not supported)
• Note that TBB is also available as a backend, but it still requires OpenMP to be present.
• On macOS, this backend requires brew install libomp or a custom compiler.

• Single threaded builds are available for debugging and development (such as on the default Clang on macOS)

There are also Python Bindings. This requires Python (2 or 3), NumPy, SciKit-Build, and CMake. CUDA 8+ is required if using CUDA. If you want the most recent stable release, use pip install -v goofit (If you have pip 9 or less, you'll need scikit-build and cmake installed beforehand). Python 2 will be removed soon.

Repository method:

You can uses pip install -v . inside the repository. You can also directly force the bindings from a normal build with -DGOOFIT_PYTHON=ON. You can check your install with python -m goofit. You can debug a goofit file named python_script.py with gcc using gdb -ex r --args python python_script.py.

Other python requirements for the examples:

• numpy-1.11.1+
• pandas-0.15.1+
• uncertainties-3.0.2
• matplotlib
• plumbum

Optional:

• numba

• -DGOOFIT_HOST=Auto: This is CPP unless device is OMP, in which case it is also OMP. This changes thrust::host_vector calculations, and is not fully supported when set to a non-default setting.
• -DGOOFIT_TESTS=ON: Build the GooFit tests
• -DGOOFIT_MPI=ON: (OFF/ON. With this feature on, GPU devices are selected automatically). Tested with MVAPICH2/2.2 and OpenMPI.
• You can enable sanitizers on non-CUDA builds with -DSANITIZE_ADDRESS=ON, -DSANITIZE_MEMORY=ON, -DSANITIZE_THREAD=ON or -DSANITIZE_UNDEFINED=ON.
• If clang-tidy is available, it will automatically be used to check the source. If you set -DGOOFIT_TIDY_FIX=ON, fixes will be applied to the GooFit source.
• -DGOOFIT_SPLASH=ON: Controls the unicode splash at the beginning.
• -DGOOFIT_CERNROOT=ON: Allows you to disable the automatic search for ROOT (used by the PIP Python build)
• -DCMAKE_UNITY_BUILD=OFF: Turn on Unity builds in CMake 3.16+. Should be a bit faster (does not speed up CUDA portions of builds).

• Use make VERBOSE=1 to see the commands used to build the files.
• Use cmake .. -LH to list the CMake options with help.
• Use ccmake if available to see a curses (terminal) gui, or cmake-gui for a completely graphical interface.
• Use -G and the name of a generator to use something other than make, like Xcode or Ninja.
• Open the CMakeLists.txt with QtCreator to generate for that IDE.
• Set the release type with -DCMAKE_BUILD_TYPE=Release, RelWithDebInfo, Debug, etc.
• Set up multiple build directories, like build-omp and build-cuda.
• CMake caches your -D option selections in your build directory so you don't have to specify them again.
• CMake reruns when needed when you make unless you add a file that it globs for (like new goofit_projects).
• Use make -j12 to build with 12 cores (for example). You can set this as the MAKEFLAGS environment variable, too.
• Use CMake --build . to build without referring to your specific build tool, like make or ninja.
• If you are using the llvm tool-suite, you can use -DCMAKE_EXPORT_COMPILE_COMMANDS=ON to generate the .json file that the clang-* commands expect.

The examples are designed to be easy to add to. Make a new directory, then add a new CMakeLists.txt in your directory with one or more of the following two lines:

goofit_add_directory()
goofit_add_executible(MyNewExample MyNewExample.cu)

The first line adds your .cu file with GooFit code as an executable, and the second one sets up a symbolic links to the source and dataFiles in the build directory to the source directory. If you prefer to only have some files symbolically linked, use goofit_add_link(filename.ext) explicitly for each file. This happens at configure time. To get the example to build when you build GooFit, add the name of your directory to examples/CMakeLists.txt.

If you are building with separable compilation, you can also use goofit_add_pdf(mypdf.cu) to add a PDF. This will also require that you include any directory that you need with include_directory, as usual.

To add packages, use standard CMake tools. For example (CMake 3.5+), to add Boost 1.49+ filesystem and TTreeReader from ROOT:

set(Boost_USE_STATIC_LIBS OFF)
set(Boost_USE_MULTITHREADED ON)
set(Boost_USE_STATIC_RUNTIME OFF)
find_package(Boost 1.49 REQUIRED COMPONENTS filesystem)

goofit_add_executable(K3Pi K3Pi.cu)
target_link_libraries(MyNewExample Boost::filesystem ROOT::TreePlayer)

External package (BETA)

GooFit now requires separable compilation, so it also now supports "external" packages, much like most other libraries. You can design your package with GooFit included as a subdirectory, and it should just work. You'll also save time by not building examples, python bindings, and tests. The recommended procedure:

git add submodule <url to goofit> goofit
git submodule update --init --recursive

Then, you'll need a CMakeLists that looks something like this:

cmake_minimum_required(VERSION 3.9...3.16)

project(my_external_package LANGUAGES CXX)

add_subdirectory(goofit)
goofit_external_package()

goofit_add_executable(myapp myapp.cpp)

That's it! Just make a build directory and build. The goofit_external_package() command sets up optional CUDA, as well as links all reasonable files into your build directory. You can run goofit_setup_std(), goofit_optional_cuda() and goofit_add_directory() instead if you want.

Classic method

If you'd like to make a separate GooFit project, you can do so. Simply checkout your project inside GooFit, with the name work or goofit_+something. CMake will automatically pick up those directories and build them, and GooFit's git will ignore them. Otherwise, they act just like the example directory. If you add a new directory, you will need to explicitly rerun CMake, as that cannot be picked up by the makefile. The automatic search can be turned off with the GOOFIT_PROJECTS option, or by using GOOFIT_PROJECT_<name> for a specific package. GooFit packages should contain:

goofit_add_package(MyPackageName)

After the package name, you can list ROOT to require that ROOT. The package will be disabled if ROOT is not found.

The following IDEs have been tested. Here $SRC refers to the source directory, and usually is .. or ../GooFit. You may want -DCMAKE_BUILD_TYPE=Debug and/or -DGOOFIT_DEBUG=ON.

The build system underwent a major upgrade in the move to CMake. The folders that were introduced to keep the includes structured require modifications of source code, converting lines like #include "Variable.hh" to #include "GooFit/Variable.h". This modification can be done for you by running the provided script, scripts/ModernizeGooFit.py on your source files (requires Python and Plumbum). You should remove your old Makefiles and use the new CMakeFiles.txt files provided in examples - this should require writing two lines of code instead of the 50 or so previously needed. You should also add a GooFit Application to your code. (2 lines of CMake)

The new GooFit::Application, which is not required but provides GooFit options, like GPU selection and status, as well as MPI support and configurable command line options, is available by adding:

#include "GooFit/Application.h"
using namespace GooFit;

// Place this at the beginning of main
Application app{"Optional description", argc, argv};

// Command line options can be added here.

GOOFIT_PARSE(app);

See CLI11 for more details. The pipipi0 example has an example of a complex set of options.

The other key differences in code are the addition of the GooFit namespace (using namespace GooFit allows fast conversion), and the removal of direct access to members of Variable (using getters/setters, or directly treat the variable like its value).

See Converting to GooFit 2.0, GooFit 2.1, and the Changelog.

Using the MPI version with an appropriate environment setup will allow for multiple GPU's to be used, and/or allow for multiple nodes. To use this feature simply turn the flag on with CMake -DGOOFIT_MPI=ON. This will divide the dataset by the number of processes involved. For instance, if you have two nodes that will be involved in the calculation, the data will be split in half. Currently, each node will load the entire buffer from disk, then load partitioned data it will work on. It is highly recommended not to use more than one process per node for MPI+OpenMP versions.

A few notes about using the MPI version:

• You will need to use the CountingVariable for any event numbers used or referenced within the code, or anything that counts with the events.
• Please call setDataSize after setData. If you do not, setDataSize doesn't have m_iEventsPerTask, which will need to be recalculated.

This advanced option is for GPU devices only. The script scripts/find_optimal.py will search a programmable group and grain space in order to find the optimal configuration for the particular PDFs. This should be run after an example has been developed and tested. Please look at scripts/find_optimal.py to see how to formulate a particular script. Depending on the searchable space, this can take hours to days to compute. The script will loop over the space and configure each parameter, then recompile and run the example a number of times. A spreadsheet is calculated to help notice patterns, and the fastest version is printed to the user.