# 20.10

-- CMake: fix ENABLE_PROFPARSER and ENABLE_SENSEI options (#1428)

-- Remove EB LSCore and levelset, and reimplement signed distance function

capability. (#1429 & #1425)

-- Remove flags setCGVerbose and setCGMaxIter since those are more

accurately called setBottomVerbose and setBottomMaxIter. The

"setCG..." flags are misleading since they actually apply to non-CG

bottom solvers as well. (#1413)

-- Remove GetPosition and SetPosition from ParIter. These were added for

WarpX, but they are not used any more. (#1408)

-- Only check the requested MPI Threading level if AMReX also initialized

MPI. (#1406)

-- DPCPP support for mulitple GPUs (#1392)

-- Random Number Generation on Device (#1363) In order to support RNG on

device with DPC++, we have to change the API because DPC++ does not

support global device variables.

-- Remove redundant particle tests and old/obsolete/non-test tests (#1389

& #1381)

-- Gpu kernel fusing is integrated in MFIter. (#1332)

-- New runtime parameter, amrex.max_gpu_streams. (#1386)

-- Switch to `use mpi` from `include 'mpif.h'` (#1385)

-- Remove reliance on managed memory from

AMReX_ParticleCommunication.H/.cpp (#1380)

-- CMake: re-organize tutorials (#1333)

-- Implement Particle in a way that does not involve UB. (#1337)

-- amrex::EB_average_down(): Fix indexing error. (#1360)

-- Only add one copy of the ghost particle regardless of how many isects

we have (#1359)

-- Add non-EB slopes and extend the generality of EB slope routines in 2D

and 3D (#1350)

-- Async IO: Check Runtime Threading (#1351)

-- Changes to make AMR codes compiled with EB but not building any

Geometry work as if it is all regular. (#1349)

-- Fix mkconfig.py (#1344)

-- Add -Xcudafe --diag_suppress=esa_on_defaulted_function_ignored to

suppress spurious warnings from nvcc. (#1342)

-- EB extend_domain_face (#1321)

-- Add parameter that stores OpenMP support version (#1325)

# 20.09

-- Fix interpolation coefficients for EB stencil in cell-centered linear

solvers when the EB surface at the domain boundary is tilted.

-- Add a parameter to `AMReX_omp_mod` that can be used by Fortran code to

check whether AMReX was built with OpenMP support, and, if yes, what

the version number (_OPENMP) was at the time AMReX was compiled.

-- CMake: re-write genex evaluation functions.

-- Extend the number of unique particles per cpu we can have at once.

-- Add a new method to the `amrex_distromap` type in

F_Interfaces. `get_pmap` fills a caller-owned array of PEs.

-- Fix a long standing bug in GNU make system on the use of HOST and

HOSTNAME.

-- Add Scan::InclusiveSum and ExclusiveSum for CPU to avoid ifdef.

-- Add check for empty probin_file in Amr::restart.

-- Update Sundials interface, documentation, and build to be version

agnostic.

-- Updates for ROCm 3.6 and 3.7.

-- Add section to Make.unknown for intel mpi.

-- CMake: re-factor third party libraries setup.

-- Port TagBoxArray to GPU.

-- Fix a bug in matching of COMP_VERSION to correctly treat CCE > 9.

-- By default, EB outside the domain will be generated by extending from

the domain faces.

-- Fix an issues due to roundoff error in determining whether particles

are inside the domain.

-- Remove Perilla because it's incompatible with our GPU strategy and it's

no longer being maintained.

-- Switch the overset mask convention to that 1 means unknown and 0 means

known. The new convention is more convenient for AMR-Wind.

-- Remove old solvers, Src/LinearSolvers/C_CellMG and C_TensorMG. They

have been superseded by Src/LinearSolvers/MLMG.

-- MacProjector: allow for re-use of the object and enhance multi-level

algorithm.

-- Add asserts for bounds checking to Array1D and Array2D.

-- Optimization of the construction of SFC.

-- Refactoring of particle buffer map.

-- Optimization of FPinfo and complementIn.

-- Move the Amr/Extrapolater used in IAMR/PeleLM to C++/GPU compliant.

-- Less surprising behavior for the ok() method of FabArray. This makes

the `ok()` method of FabArray return `false` instead of crashing if the

`define()` method has yet to be called.

-- Optimization of regrid.

-- Special FillBoundary for Almo.

-- Fix the average_down_faces calls in linear solvers to see periodicity.

-- Make sure the calculation of divu at inflow face in nodal projection

does not use tangential velocities on an inflow face.

# 20.08

-- New average_down_faces functions that take periodic boundary into account.

-- Update the documentation for Nsight.

-- Fix a bug with HDF5 boxcenter data.

-- New runtime parameter, `eb2.extend_domain_face`, that can be used to extraploate EB information

from domain faces.

-- New GNU Make options for GCC and Clang, `WARN_ALL` and `WARN_ERROR`. `WARN_ALL` turns on more

compiler warnings. `WARN_ERROR` turns warnings into errors.

-- Fix various compiler warnings.

-- Fix memory fragmentation issue with communication buffer.

-- Semi-coarsening support in linear solvers.

-- Fix virtual particle issues.

-- Capability to fuse GPU kernel launches.

-- Make ghost particles support soa data.

-- CMake: ability to export build tree.

-- Reset amrex_geometry_module initialization state when finalizing.

-- Remove Long version of Gatherv.

-- Optimization of TagBoxArray::collate.

-- CMake: fix missing PETSc setup in Config file.

-- BndryFunc for nodal data.

-- MLMG Hypre fix for singular problems.

-- Add particle copy filters.

-- Async output support for particles.

-- BackgroundThread class.

# 20.07

-- Nodal hypre solver now supports coarsening before hypre is called.

-- A new ParmParse parameter, `amrex.the_arena_is_managed` is introduced to set the default memory

type of `The_Arena`.

-- MFIter is now OMP thread safe when it is compiled with GPU support.

-- Overset support is added to cell-center and tensor linear solvers.

-- Remove compile time flag, BACKTRACE, for simplicity.

-- Updates on Blueprint and Ascent.

-- HDF5 support in GNU Make and CMake.

-- CMake support for DPC++.

-- Add -DNDEBUG for non-debug build.

-- Support for Windows.

-- Support for ParallelContext in Particles.

-- Non const iterator for neighbor list.

-- Tool for comparing single-level plotfiles of all nodalities.

-- GNU Make: Use -M instead the perl script to generate dependencies.

-- Particle: Remove directly accessing the m_idata and m_rdata structs internally.

-- Extend the makeSFC method to take an additional argument where the user can override the number

of processes a BoxArray is distributed to.

-- Turn tiling off by default for gpu.

# 20.06

-- Set a non-default CMAKE_INSTALL_PREFIX only if AMReX is the

top-level CMake project. Otherwise, the default of a downstream

project using AMReX as subproject will be placed in a quite

confusing, temporary directory.

-- C order support for Array2D and Array3D.

-- Disable unqualified abs function within amrex namespace.

-- Updates for SENSEI 3.2.

-- CMake: link to OpenMP::Fortran only when needed.

-- DPC++ support.

-- Add non-const data() to GpuArray.

-- Fix HIP version of scan.

-- Return immediately from GPU memcpy if size is zero.

-- Documentation on EB linear solvers.

# 20.05

-- AMReX's git history has been rewritten. If you have problems due

to this, follow the instruction here. https://xkcd.com/1597/ We

also have a new CONTRIBUTING.md.

-- EB linear solvers: You now need to specify the location of

diffusion coefficient (face center or face centroid) and the

location of the unknowns (cell center or cell centroid).

-- Optimization of makeKnapSack and makeSFC.

-- Unification of GPU and CPU versions of neighbor particle /

neighbor list implementations.

-- New HostDeviceParallelFor functions that can run on either host

or device depending on the launch region flag.

-- Add CUPTI trace for CUDA kernel timing.

-- Implement ENABLE_FORTRAN (BL_NO_FORT) option in CMake.

-- Fix CUDA build of tools.

-- Fix invalid git version in CMake.

# 20.04

-- BaseFab can no longer be used on device. Many BaseFab functions

now requires RunOn::Device or RunOn::Host template parameter when

GPU support is enabled.

-- FillPatch* funcitons are now templated to support FabArray in

addition to MultiFab.

-- Fix -pthread for nvcc by default in Make.unknown

-- EB now has the option to build coarse level from GeometryShop

when coarsening from fine level fails. This could be used for

hyperbolic system with EB resolved to the finest level

-- CMake: add C++ 17 among valid options for AMREX_CXX_STANDARD

-- Add option to set fixed number of iterations in the Fortran

interface of linear solver

-- Fabs in alias FabArray now carries EB information.

-- Add AMREX_LOOP_3D and 4D macros.

-- Make amrex::Random() return numbers in the same interval for both

CPU and GPU: [0:1)

-- More flux register functionality for flash

-- Chunked partition in particle redistribution to use less memory.

# 20.03

-- AmrCore can now be built without using ParmParse.

-- Improve BL_NO_FORT support.

-- Fix OMP bug in MFIter constructors taking BoxArray and

DistributionMapping.

-- Cray compiler has been re-enabled in CMake.

-- For device only code, the users can now use #if AMREX_DIVICE_COMPILE.

-- Fix an integer overflow issue in TileArray cache.

# 20.02

-- RealVect updated for GPU.

-- Support for restarting from HDF5 particle checkpoint file.

-- Improvement of MacProjector and NodalProjector

# 20.01

-- GPU host device version of swap.

-- Fix an edge case of gpu block reduce where the block size is less

than a warp.

-- Always check MPI buffer size for overflow.

-- CMake support for PETSc.

-- Include C sources in the CUDA proprecessing.

-- It's now safe to construct a zero components Fab.

-- CXXSTD: option to set c++ standard in GNU make system

-- Improve performance of the nodal linear solver on GPU.

# 19.12

-- Fix potential deadlocks in amrex::Random.

-- MultiFab::Swap that does shallow swap if it can, does deep swap

otherwise.

-- FabArray::arena function that returns the Arena used to build.

-- Nodal projection sync residuals are implemented for RAP.

-- GPU support for initializing Fabs to NaNs.

# 19.11

-- AmrCore::regrid will now simply return instead of abort if the

base level is already the maximal level.

-- Add NodalProjector that wraps MLMG for convenience.

-- Enforce in AmrCore that blocking factor must be a power of 2.

-- Add support to write plotfile in HDF5 format.

-- FabArray's Reduce functions can now take lambda working on Array4

in addition of FAB.

-- Add AMREX_REAL_MIN, AMREX_REAL_MAX and AMREX_REAL_LOWEST for

convenience.

-- GPU launch functions and macros are now safe with empty boxes.

-- Remove FabView from BaseFab.

-- Avoid race conditions in GPU reduction code called multiple

times.

-- MLMG tensor solver is fixed for AMR.

# 19.10

-- Add inhomogeneous Neumann BC to MLMG for convenience.

-- Fix a corner case of tensor solver.

-- Various fixes for single precision and particle real.

-- Add amrex::Long that has a size of 8.

-- Add int8_t support in typecheck.

-- C++ user literal _rt for amrex::Real and _prt for

amrex::ParticleReal.

-- EB MAC projector now requires the user to specify whether the

face velocities are defined on face centers or centroids.

# 19.09

-- First pass of Sr/EB on GPU.

-- Optimization of particle redistribution.

-- EB integral: add integral of x*y*z

-- HostDevice::Atomic::Add that does atomic add for both device and

host omp code.

-- makeFineMask: optional arguments specifying coarse and fine flag

values.

-- Optimization of FabArray communication for GPU by merging small

kernels.

# 19.08

-- MacProjector has the option to set maximal coarsening level.

-- OwnerMask and OverlapMask are now built on GPU.

-- Random number generation on GPU no longer requires an call to

initialize seeds. They are done in amrex::Initialize and resize

is not needed when launching with more threads than the size of

seed array.

-- Optimization of NeighborParticles.

# 19.07

-- Geometry/CoordSys: Static class members have been removed.

Functions such as `isPeriodic`, `isAnyPeriodic`, `isAllPeriodic`,

`ProbLo`, `ProbHi`, `IsCartesian`, `IsRZ`, and `IsSPHERICAL` are

now non-static. The default Geometry can been obtained via

`DefaultGeometry()` function.

-- Funciton amrex::Loop and amrex:LoopConcurrent are added to help

looping over cells in a Box.

-- Nodal solver: A bug in nodal owner mask that could affect

covergence was fixed.

-- Fortran amrex_amrcore_module: add amrex_set_boxarray,

amrex_set_distromap, and amrex_set_finest_level.

-- Option to allocate MultiFab data in arenas different from

The_Arena().

-- Option to tag MultiFabs and query their memory usages.

-- AsyncFab has been deleted.

# 19.06

-- Geometry/CoordSys: New constructor and define function are added.

We plan to remove static data from Geometry and its base CoordSys

in the next release. This means most of the static functions in

Geometry class will become non-static. These include

`isPeriodic`, `isAnyPeriodic`, `isAllPeriodic`, `ProbLo`,

`ProbHi`, `IsCartesian`, `IsRZ`, and `IsSPHERICAL`. You can

prepare codes now for the upcoming changes. In almost all cases,

all you have to do is replace `Geometry::function` with

`DefaultGeometry().function`.

-- Add tensor linear solvers for viscous flows with and without EB.

-- Algoim: A modified version of Algoim is added. This removes the

dependence of 3d EB nodal MLMG solver on the official Algoim code

and Blitz.

-- EB: Fix the potential ownership issue is IndexSpace.

-- Add a new FillPatchTwoLevels for EB.

# 19.05

-- Interface of virtual function Interpolater::interp has changed.

-- Add Fortran funcntions for shared memory reductions.

-- EB nodal linear solver: The default bottom solver is now bicgcg.

It starts with bicgstab and switches cg if bicgstab fails.

-- EB nodal linear solver: Finished the implementation of matrix

coarsening with RAP.

-- Particle: Option for adding runtime struct of array components.

-- Add a trilinear interpolation option for 3D.

-- Add more support for non-uniform grow vectors to MultiFab and

FabArray.

-- AmrMesh: n_error_buf is now IntVect. This allows for different

number of error buffer cells in different directions.

-- Using PETSc as bottom solver no longer requires Hypre.

# 19.04

-- SUNDIALS 4 support.

-- Option to disable device sync in MFIter.

-- Functions for prefetch unified memory.

-- GNU Make system: It is now an error if a function with return

type does not return.

-- GNU Make system: Pass C++ flags to nvcc when compiling C codes.

-- Allocate 3/4 of gpu memory upfront in The_Arena.

-- BaseFab now has Allocator template parameter.

-- Random number generation on gpu.

# 19.03

-- Deprecated Fortran framework F_BaseLib and linear solver F_MG

have been removed. A number of tools written in F_BaseLib have

been migrated to Tools/Plotfile/. Note that the Fortran wrappers

to C++ (F_Interfaces) are still there.

-- In addition to launch macros, there are now functions for

lauching, amrex::launch, amrex::For and amrex::ParallelFor.

These improves debuggability.

-- If compiled with BASEFAB_IS_PINNED, pinned memory will be used

for BaseFab meta-data. On systems with nvlink, this improves the

performance a little bit.

-- Performance improvement in particle redistribution.

-- Particle NeighborList has been moved from tutorial to AMReX

proper.

-- Cell-centered MLMG linear solver have been ported to GPU.

# 19.02

-- Array4 class that contains the important information in BaseFab

class. It can be captured by lambda and used on gpu.

-- A number of new macros for launching for loops: AMREX_FOR_1D,

AMREX_FOR_3D, AMREX_FOR_4D, AMREX_PARALLEL_FOR_1D,

AMREX_PARALLEL_FOR_3D, and AMREX_PARALLEL_FOR_4D.

-- Disable cuda aware MPI by default. It can be enabled with

parameter amrex.use_gpu_aware_mpi.

# 19.01

-- Documentation and Tutorials on GPU

-- EB implicit function using spline

-- FabArray::fabPtr() function takes MFIter and returns a device

pointer to BaseFab.

# 18.12

-- More GPU supports

# 18.11

-- GPU support in Base and Particle

-- BaseFab is no longer polymorphic

-- EB now supports ghost cells outside physical domain

# 18.10

-- MacProjector class for MAC projection

-- Support for Dirichlet EB in linear solver

-- Option to use PETSc as bottom solver

-- Multi-level linear solver with EB not crossing coarse/fine

boundary.

-- Sundials 3 support

# 18.09

-- Cell-centered linear solver with homogeneous Neumann embedded

boundary.

-- Hypre as a bottom solver has a IJ matrix interface and supports

homogeneous Neumann boundary condition on embedded boundary.

-- Old embedded boundary codes have been removed.

# 18.08

-- Faster geometry generation is available at Src/EB2.

Tutorials/EB/CNS is now based on EB2.

-- AMReX is now fully compliant with xSDK.

-- Fortran interface for particles. See Tutorials/Amr/Advection_F/

for an example.

# 18.07

-- Pre- and post-interpolation hooks in FillPatch.

-- NDEBUG is no longer defined in xSDK mode.

# 18.06

-- When amrex::Initialize is called, optional std::ostream arguments

can be passed in and their defaults are std::cout and std::cerr.

-- ParmParse inputs files can now contain Fortran namelist that can

be accessed from Fortan (and Frotran only). These Fortran inputs

are stored in amrex_namelist character variable in

amrex_parmparse_module, and can be read with standard Fortran

I/O.

-- There is a new non-blocking parallel copy function in

MultiFab/FabArray called ParallelCopy_nowait. It returns a

CopierHandle object and can later be used to finish the

communication with CopierHandle::finish().

# 18.05

-- FillBoundary and ParallelCopy functions can now take IntVect

arguments as the number of ghost cells. That is the number of

ghost cells could be different for different directions.

-- MultiFab, iMultiFab and FabArray can have different number of

ghost cells in different directions.

-- Fortran module mempool_module has been renamed amrex_mempool_module.

-- Many macros have been renamed.

* USE_PARTICLES and PARTICLES have been renamed AMREX_PARTICLES

* DIMENSION_AGNOSTIC has been renamed AMREX_DIMENSION_AGNOSTIC

* USE_CVODE has been renamed AMREX_USE_CVODE

* DEBUG has been renamed AMREX_DEBUG

-- amrex::Array is now an alias to std::array in stead of amrex::Vector.

# 18.04

-- New BoxList constructors making a given number of Boxes from a

single Box.

-- Optimization of particle redistribution

-- MLMG nodal linear solver: add solvability fix

-- Modification to amrex::Initialize and Finalize to make it

suitable as a library in an application that does not use AMReX

as a framework.

# 18.03

-- STL support for EB

# 18.02

-- Fortran interface: added new functions amrex_multifab_build_alias

and amrex_imultifab_build_alias.

-- Documentation in reST/Sphinx. See Docs/Readme.sphinx for more

details.

-- Src/LinearSolvers/MLMG now supports nodal based data.

-- Fortran module AMReX_fillpatch_module takes a user provided

callback function for filling physical boundary conditions. That

function used to use 0-based index for component (i.e., the last

index). For consistence with 1-based index for component in

other Fortran modules, this has been changed to 1-based index for

component.

# 18.01

-- New linear solver MLMG at Src/LinearSolvers/MLMG/. Currently it

supports cell-centered single level and multi-level composite

solve. Fortran interface is also available at

Src/F_Interfaces/LinearSolvers/. See Tutorials/LinearSolvers for

tutorials.

# 17.12

-- Ported more features (including dynamic scheduling and explicit

tile size) of MFIter from C++ to Fortran.

-- Added procedures to Fortran amrex_multifab_module to add,

subtract, multiply, divide, saxpy and lincomb (linear

combination). Note that we call it saxpy, but it doesn't mean

single precision.

-- The `configure; make; make install` build approach enables linear

solvers by default now.

-- MultiFab and iMultiFab now support move assignment operator.

This allows for swapping two MultiFabs without deep copying. For

instance, std::swap(mf1, mf2);

-- Tutorials/Amr/Advection_AmrCore changes from using

`Vector<unique_ptr<MultiFab>>` to `Vector<MultiFab>`.

# 17.11

-- AMREX_ASSERT_WITH_MESSAGE and AMREX_ALWAYS_ASSERT_WITH_MESSAGE

are new macros for assertion when it fails additional message

will be printed. For example:

AMREX_ASSERT_WITH_MESSAGE(x > y, "reason why x > y must be true");

-- amrex::Array is deprecated and replaced by amrex::Vector.

Currently Array is an alias to Vector. So this should not break

any codes. In the future, we will remove Array completely.

Therefore it is recommended that application codes should start

replacing Array with Vector. There is a script,

amrex/Tools/Migration/amrex_array_to_vector.sh, to help. But use

it with caution. If it is completely gibberish to you, you

probably shouldn't use the script.

-- functions for printing to a file in a parallel and thread-safe manner

have been added. For example:

amrex::AllPrintToFile("output") << "Hello from rank " << rank <<

" and thread " << thread << "\n";

will print out the requested information to separate files for each

rank and thread.

# 17.10

-- Added a load balance approach to Amr/AmrLevel. The runtime

parameters for this is amr.loadbalance_with_workestimates and

amr.loadbalance_level0_int. The latter is for single level

(i.e., amr.max_level=0) only. For multi-level runs, load balance

is done during regrid and thus the load balance interval is

controlled by amr.regrid_int. To use this approach, AmrLevel

derivative must has a StateData for work estimates and its index

is returned by virtual function int WorkEstType().

-- MFIter now supports dynamic scheduling of OpenMP threads. For

example,

for (MFIter mfi(mf, MFItInfo().SetDynamic(true).EnableTiling(tile_size);

mfi.isValid(); ++mfi)

{ ... }

-- Added EBFluxRegister to Src/EB for refluxing and

re-redistribution in EB.

-- amrex.signal_handling = 1 is a new runtime parameter that can be

used to control whether AMReX should handle signals like SIGSEGV

etc.