.. comment
Title:
GPU Acceleration of Geant4 Simulations of
the Daya Bay Anti-Neutrino Detector
Abstract:
Studies of the acceleration of Geant4 based Monte Carlo Simulations
of the Daya Bay anti-neutrino detector using the Chroma GPU
optical photon simulation package are presented.
Original open source developments made to bring the
Geant4 simulation context to the GPU are described.
They include a Geant4 geometry exporter package (G4DAE)
and a bridging package (G4DAEChroma) that integrates
external optical photon propagation with standard
Geant4 simulations. The G4DAE geometry exporter
package is planned to be incorporated with the 2015
release of the Geant4 simulation toolkit.
The export of triangulated geometry data into XML files
following the standard COLLADA 3D specification was
found to allow both massively parallel CUDA based optical
photon propagation and high performance OpenGL
visualizations of geometry and event data.
12 Jan 2015, Monday 17:05
.. raw:: html
.. role:: i
.. role:: alarm
.. role:: warn
.. role:: ok
.. include::
.. s5_background_image::
#
# slide titles and background image urls,
# including server relative urls like /env/geant4/geometry/collada/daeview/20140419-170713.png
# and protocol relative urls like //localhost/env/test/LANS_AD3_CoverGas_Humidity.png
#
# NB1 slide titles here must match those in body precisely,
# NB2 also ensure all slide titles are unique
#
# first slide id is exceptionally: slide0, not the mangled title
slide0
/env/geant4/geometry/collada/g4daeview/20140419-170713.png auto_auto 0px_0px
g4daeview.py : Fast OpenGL 3D viewer for G4DAE files
/env/geant4/geometry/collada/g4daeview/20140419-170713.png
Cerenkov Photons Simulation - Top View
/env/geant4/geometry/collada/g4daeview/20141224-115923.png
Cerenkov Photons Simulation - Side View
/env/geant4/geometry/collada/g4daeview/20141224-115935.png
Scintillation Photons Simulation - Top View
/env/geant4/geometry/collada/g4daeview/20141224-121444.png
Scintillation Photons Simulation - Side View
/env/geant4/geometry/collada/g4daeview/20141224-121435.png
Standard Geant4 Workflow
/env/keynotefigs/G4DAEChroma/G4DAEChroma.001.png
External Photon Simulation Workflow
/env/keynotefigs/G4DAEChroma/G4DAEChroma.002.png
.. comment
Generated Scintillation Photons GPU cf Geant4
/env/g4dae/generated_scintillation_time_wavelength.png
G4/DetSim Generated Cerenkov Wavelength
/env/g4dae/g4_cerenkov_wavelength.png
======================================================================
GPU Accelerated Geant4 Simulation with G4DAE and Chroma
======================================================================
.. class:: small
http://simoncblyth.bitbucket.org/env/presentation/gpu_accelerated_geant4_simulation.html
http://simoncblyth.bitbucket.org/env/presentation/g4dae_geometry_exporter.html
http://simoncblyth.bitbucket.org/env/presentation/gpu_optical_photon_simulation.html
.. sidebar:: Drastic 200x Potential Speedup
Chroma project claims 200x speedup of optical
photon (OP) propagation compared to Geant4.
**A goal worthwhile chasing**.
.. class:: small
* Geometry Model Implications
* Introducing Chroma
* Geant4 <-> Chroma Integration
* G4DAE Geometry Exporter
* Validating GPU Geometry
* G4DAEChroma bridge
* Chroma Forked
* Validating Chroma Generated Photons
* Next Steps
* Visualizations
| Simon C Blyth, National Taiwan University
| **January 2015**
.. comment
Title
00 : GPU Accelerated Geant4 Simulation with G4DAE and Chroma
Geometry Model Implications
01 : implications of geometry model choice
Introducing Chroma
02 : chroma : ultra-fast photon mc, developed by stan seibert, university of pennsylvania
Geant4 <-> Chroma Integration
03 : standard geant4 workflow
04 : geant4 <-> chroma integration
05 : external photon simulation workflow
G4DAE Geometry Exporter
06 : https://bitbucket.org/simoncblyth/g4dae
07 : collada : standard for 3d digital asset exchange (dae)
08 : liberated geometry allows ubiquitous high peformance visualization
09 : g4daeview.py : fast opengl 3d viewer for g4dae files
Validating GPU Geometry
10 : compare dae exports to gdml and vrml2(wrl)
11 : chroma raycasting : exercises geometry intersection
12 : chroma raycast with entire geometry in view
G4DAEChroma bridge
13 : g4daechroma : bridging from geant4 to chroma
14 : g4daechroma : for implementation details...
Chroma Forked
15 : https://bitbucket.org/simoncblyth/chroma forked:
Validating Chroma Generated Photons
16 : chroma generated scintillation photons cf geant4
17 : chroma generated cerenkov photons cf geant4
Next steps
Visualizations
17 : cerenkov photons simulation - side view
18 : cerenkov photons simulation - top view
19 : scintillation photons simulation - top view
20 : scintillation photons simulation - side view
Extras
21 : extra slides
22 : g4 generated cerenkov wavelengths in material categories
23 : refractive index interpolation explains jump at 200nm
24 : comparison of generated scintillation photon distributions
25 : comparison of generated cerenokov photon distributions
13 : g4daeview.py : dayabay chroma photon propagation (1)
:small:`Implications of Geometry Model Choice`
-------------------------------------------------------
:red:`Track > Geometry intersection` typically limits
simulation performance. Geometry model determines
techniques (and hardware) available to accelerate intersection.
:green:`Geant4` Geometry Model (:blue:`solid based`, :red:`locked in`)
*Tree of nested solids composed of materials,
each shape represented by different C++ class*
:green:`Chroma` Geometry Model (:blue:`surface based`, :red:`extremely simple > portable`)
*List of oriented triangles, each representing
boundary between inside and outside materials.*
3D industry focusses on surface models >> frameworks and GPU
hardware designed to work with surface based geometries.
:small:`Chroma : Ultra-fast Photon MC, Developed by Stan Seibert, University of Pennsylvania`
-----------------------------------------------------------------------------------------------
.. class:: small
* http://chroma.bitbucket.org
Chroma tracks photons through a :red:`triangle-mesh detector geometry`,
simulating processes like diffuse and specular reflections,
refraction, Rayleigh scattering and absorption. Using triangle meshes
eliminate geometry code as :red:`just one code path`.
.. sidebar:: Original Motivation for G4DAE exporter
Allow use of Chroma GPU accelerated optical photon propagation
with existing DayaBay Geant4 geometry.
.. class:: small
200x performance claim:
*With a CUDA GPU Chroma has propagated 2.5M photons per second
in a detector with 29k photomultiplier tubes. This is
200x faster than GEANT4.*
.. class:: small
**BUT** Chroma needs : :red:`triangles + inside/outside materials`
.. class:: tiny
http://on-demand.gputechconf.com/gtc/2013/presentations/S3304-Particle-Physics-With-PyCUDA.pdf
https://bitbucket.org/chroma/chroma
https://bitbucket.org/simoncblyth/chroma (my fork)
Standard Geant4 Workflow
--------------------------
.. class:: small
:i:`Some invisible vertical space`
.. class:: large
:i:`.`
:i:`.`
:i:`.`
OP Generated in Cerenkov and Scintillation G4VProcess,
some reach PMTs and form hits.
.. class:: tiny
:i:`.`
:i:`.`
:i:`.`
:i:`.`
:i:`.`
::
// G4VProcess subclass
PostStepDoIt(const G4Track&, const G4Step&)
{
// calculate NumPhotons
aParticleChange.SetNumberOfSecondaries(NumPhotons);
// generate and collect secondaries
for (G4int i = 0; i < NumPhotons; i++)
{
...
aParticleChange.AddSecondary(aSecondaryTrack);
}
}
// G4VSensitiveDetector subclass
bool ProcessHits(G4Step* step, ...)
{
// form hits from step information
}
Geant4 <-> Chroma Integration
----------------------------------------
.. sidebar:: Bridging to external OP simulation
.. class:: small
Optical photons (modulo reemission) are the
leaves of the simulation tree, allowing external
simulation to be integrated rather simply.
.. class:: small
Recreate Geometry info on GPU (:blue:`G4DAE`)
* export triangulated geometry as COLLADA/DAE files
* export PMT identifiers associating sensors with volumes
* import geometry and identity files into Chroma and upload to GPU
:red:`Runtime bridge` via message queue (:blue:`G4DAEChroma`)
* collect generation step parameters and prevent
photon processing by not forming secondaries
* send generation steps to Chroma and receive hits in reply
* include hits into standard Geant4 hit collections,
subsequent simulation proceeds normally
GPU Optical Photon simulation (:blue:`Chroma`)
* receive generation steps from message queue
* copy generation steps to GPU and allocate GPU memory for photons
* invoke generation and propagation CUDA kernel
* copy propagated photons back to CPU
* send reply with hits back to :blue:`G4DAEChroma`
External Photon Simulation Workflow
-------------------------------------
https://bitbucket.org/simoncblyth/g4dae
---------------------------------------------------------------------------
.. sidebar:: Inclusion into Geant4
.. class:: small
Geometry exporter presented at
Sept 2014 Geant4 Collaboration Meeting.
Proposal to contribute was accepted, intended
to be included with 2015 Geant4 release.
.. class:: small
Exports *Geant4* Geometry into COLLADA/DAE standard 3D files,
based on *GDML writer* code, same *XercesC* dependency for XML handling. Export includes:
* volume tree: solid/physical volume/logical volume heirarchy
* geometry: vertices, triangles+quads (triangulation from *G4Polyhedron*)
* materials and surfaces with properties as function of wavelength
(using DAE *extra* XML elements)
.. image:: /env/geant4/geometry/collada/g4dae/g4dae_bitbucket.png
:width: 700px
:align: center
:small:`COLLADA : Standard for 3D Digital Asset Exchange (DAE)`
-----------------------------------------------------------------
.. class:: small
* https://www.khronos.org/collada/
.. image:: /env/graphics/collada/collada_khronos.png
:width: 900px
:align: center
:small:`Liberated Geometry allows Ubiquitous High Peformance Visualization`
-----------------------------------------------------------------------------
.. sidebar:: Geometry Set Free
.. class:: small
Liberating geometry data from Geant4/ROOT
gives free choice of visualization packages.
Many commercial, open source apps/libs
provide high performance visualization of DAE files
using GPU efficient OpenGL techniques.
:red:`=> Shockingly Smooth Visualization performance`
.. class:: small
Commercial:
* *Finder/Quicklook/Preview/Xcode* (standard OSX 10.8+)
* :blue:`Xcode : useful effect tweaking interface`
* :blue:`Preview : set black background for visibility`
* *Sketchup 8* free (OSX) [also Windows]
* *ESKO Studio Viewer* free (iOS on iPad)
Open source:
* *Meshlab* (OSX/Linux), slow DAE loading, OK interface [also Windows]
* *Blender* (OSX), painful GUI [also Windows/Linux]
* *g4daeview* (OSX/Linux), :red:`extremely useful for debugging`
Frameworks:
* *pycollada*, Python COLLADA parsing/construction, :red:`lxml + numpy powered fast parsing`
* *threejs*, Javascript COLLADA loading, renders HTML canvas with WebGL
* *SceneKit*, Objective C Framework used by OSX apps
Many more applications and frameworks listed http://en.wikipedia.org/wiki/COLLADA
.. class:: tiny
http://meshlab.sourceforge.net
https://bitbucket.org/simoncblyth/meshlab (fork improves COLLADA loading speed)
:small:`g4daeview.py : Fast OpenGL 3D viewer for G4DAE files`
-----------------------------------------------------------------------------
.. sidebar:: PyOpenGL Based
.. class:: small
Easy to extend due to fast python development, yet access to
highly efficient OpenGL techniques such as
VBOs, PBOs and GLSL shaders.
.. class:: small
G4DAE visualization [#g4daeview_usage]_ implemented with PyCollada, **PyOpenGL** and Glumpy
* very fast/responsive 3D OpenGL visualization [#vbo]_
* flexible tree/list based partial geometry loading
* intuitive virtual trackball translate/rotate
* parallel or perspective projections
* interactive fov and near/far plane clipping
* persistent viewpoint bookmarks
* animation by interpolation between bookmarks or orbiting
* numerical control via UDP remote messaging
* live Geant4 connection, photon, genstep visualization, single stepping
* easily extensible python implementation [#g4daeview_code]_
* photon picking interface, with property inspection
Also used for testing GPU photon propagation with Chroma project
* interactive raycasting, including animated
* propagation visualization, with time slider
.. class:: tiny
.. [#g4daeview_usage] http://simoncblyth.bitbucket.org/env/notes/geant4/geometry/collada/g4daeview/g4daeview_usage/
.. [#vbo] Implemented with single OpenGL Vertex Buffer Object (VBO) for entire geometry
.. [#g4daeview_code] https://bitbucket.org/simoncblyth/env/src/tip/geant4/geometry/collada/g4daeview/
:small:`Compare DAE Exports to GDML and VRML2(WRL)`
----------------------------------------------------
.. class:: small
Export validation:
* Comparison of all vertices/faces reveals :red:`boolean solids are discrepant`.
* Perfect [#perfect]_ DAE WRL agreement achieved by *cheating* : :blue:`perform triangulation once and reuse`.
* Boolean solid triangulation sensitivity must be kept in mind as a potential systematic
for computational uses
* **DAE not much bigger than GDML**, despite including all triangles/vertices [#not-repeating]_
.. class:: small
==================== ================= ============== ================== =================
Qty DayaBay Lingao Far Juno x0.5
==================== ================= ============== ================== =================
Volumes 12,229 12,229 18,903 25,000
Triangles 2,448,064 2,448,064 4,189,680 21,886,158
Vertices 1,245,996 1,245,996 2,128,208 10,993,079
DAE/GDML/WRL (MB) **6.9**/4.0/98 **6.9**/4.0/96 **8.6**/6.0/167 **6.1**/-/-
==================== ================= ============== ================== =================
.. class:: tiny
``VGDX_20140414`` counts using ``g4daeview.py -g 0: --with-chroma``, Juno geometry truncated
.. class:: tiny
.. [#perfect] Maximum DAE WRL offset < 0.13 mm, after patching VRML2 export precision bug (fixed in current G4).
Details: http://simoncblyth.bitbucket.org/env/notes/geant4/geometry/collada/dae_cf_wrl/
.. [#not-repeating] Geometry is not repeated, instead the COLLADA format uses geometry instancing and a tree of transforms,
just like GDML
:small:`Chroma Raycasting : exercises geometry intersection`
-------------------------------------------------------------
.. class:: small
Raycasting exercises slowest part of optical photon propagation: :red:`geometry intersection`.
.. image:: /env/chroma/chroma_camera/chroma_raycast_illustration.png
:width: 600px
:align: center
.. class:: small
* Shoot rays thru every pixel out into geometry, 1 CUDA thread for each, typically >1M rays
* Find triangle intersections using BVH [#bvh]_ acceleration structure
* Determine color based on ray to triangle normal angle
* Alpha blend nearest 10 surfaces, providing transparency effect
.. class:: tiny
.. [#bvh] Boundary Volume Heirarchy, a tree of bounding boxes with triangles inside leaf nodes
:small:`Chroma Raycast with entire geometry in view`
----------------------------------------------------------------
.. class:: small
Render Split into 3x3 CUDA kernel launches, 1 thread per pixel, ~1.8s for 1.23M pixels, 2.4M tris (with [#hw]_)
.. image:: /env/chroma/chroma_camera/20140423-162109.png
:width: 800px
:align: center
.. class:: tiny
.. [#hw] MacBook Pro (2013), NVIDIA GeForce GT 750M 2048 MB ;
Workstation GPUs such as NVIDIA Kepler K20 expected at least ~5x faster
:small:`G4DAEChroma : Bridging from Geant4 to Chroma`
-----------------------------------------------------------
.. class:: small
https://bitbucket.org/simoncblyth/env/src/tip/chroma/G4DAEChroma/G4DAEChroma/
.. sidebar:: Transport Infrastructure
.. class:: small
Collects generation steps from Geant4, sends to Chroma, receives
hits in reply, integrates into G4 Hit Collections.
:red:`Implemented with core of ~20 C++ classes`
.. class:: small
* :green:`Two detector specific subclasses required eg` *DybG4DAECollector* *DybG4DAEGeometry*
* NumPy array creation from C++, transport via ZeroMQ
* JSON metadata communication C++/Python, *map <-> dict*
* Geant4 hit integration using additional *G4VSensitiveDetector*
that steals hit collection pointers from existing one
* Transform Cache to obtain PMT local coordinates from global hit coordinates
* SQLite DB integation for monitoring/control
.. image:: /env/chroma/G4DAEChroma/G4DAEChroma_bitbucket.png
:width: 700px
:align: center
:small:`G4DAEChroma : For implementation details...`
-----------------------------------------------------------
.. class:: small
http://simoncblyth.bitbucket.org/env/notes/chroma/G4DAEChroma/G4DAEChroma/G4DAEChroma_implementation/
.. image:: /env/chroma/G4DAEChroma/G4DAEChroma_implementation.png
:width: 700px
:align: center
https://bitbucket.org/simoncblyth/chroma :small:`Forked:`
---------------------------------------------------------------------------
.. class:: small
Changes in my fork of Chroma:
* stability + efficiency improvements, enabling mobile development
* OpenGL/CUDA interoperation, using VBOs
* add recording of propagation steps into VBO datastructure
* more efficient Raycasting using 4x4 matrix uniforms and PBOs
* animated photon propagation visualization
* change serialization approach from ROOT/TObject to NumPy (NPY)
* :red:`generation of Cerenkov and Scintillation Photons based
on Geant4 Generation Step inputs`
.. image:: /env/chroma/chroma_fork_jan2015.png
:width: 700px
:align: center
:small:`Chroma Generated Scintillation Photons cf Geant4`
------------------------------------------------------------
.. class:: small
Chroma interpolates all properties in 20nm bins, stair artifacts
in wavelength distribution can be reduced by using eg 10nm property interpolation bins.
.. image:: /env/g4dae/generated_scintillation_time_wavelength.png
:width: 800px
:align: center
:small:`Chroma Generated Cerenkov Photons cf Geant4`
-------------------------------------------------------
.. class:: small
Geant4/DetSim wavelength distribution has a blip at 200nm, corresponding to edge of water
refractive index properties. (see the extra slides)
.. image:: /env/g4dae/generated_cerenkov_time_wavelength.png
:width: 800px
:align: center
Next Steps
------------------
.. class:: small
G4DAE Geometry Exporter
* implement parametrized/replica geometry handling, following GDML
* investigate issue inherited from GDML of a skipped
edge case (when a volume is shared between multiple volume pairs)
resulting in missing *G4LogicalBorderSurface*
* investigate improving default export appearance in common viewers
* test with newer and latest versions of Geant4
* test on more detector geometries
* incorporate into Geant4 codebase
.. class:: small
G4DAEChroma bridge
* refactor to remove duplication in *G4DAEHit* *G4DAEHitList*
* eliminate use of *cnpy* external in favor of *numpy.hpp*
.. class:: small
Chroma
* Port Cerenkov *ApplyWaterQE* to Chroma (expect quick)
* Port *DsPmtSensDet::ProcessHits* QE gymnastics to Chroma
* Implement double sided surfaces, to match *G4LogicalBorderSurface*
* Compare PMT Hit distributions between *NuWa/DetSimChroma* and *NuWa/DetSim*
* :red:`iterate until match achieved or discrepancies explained`
* Find desktop GPU on which can test operation and performance
:i:`Cerenkov Photons Simulation - Side View`
----------------------------------------------
.. sidebar:: Cerenkov Photons
.. class:: tiny
Simulated Cerenkov shock front from an 100 GeV muon travelling
from right to left across Dayabay AD.
Primaries are simulated by Geant4, Cerenkov "steps" of the primaries
are transferred to the GPU where **photons are generated, propagated and PMT hits formed**.
Photon colors indicate reemission (green), absorption(red), specular reflection (magenta), scattering(blue), no history (white).
:i:`Scintillation Photons Simulation - Side View`
--------------------------------------------------
.. sidebar:: Scintillation Photons
.. class:: tiny
Simulated Scintillation photons from an 100 GeV muon travelling
from right to left across Dayabay AD.
Primaries are simulated by Geant4, Scintillation "steps" of the primaries
are transferred to the GPU where **photons are generated, propagated and PMT hits formed**.
Photon colors indicate reemission (green), absorption(red), specular reflection (magenta), scattering(blue), no history (white).
Cerenkov Photons Simulation - Top View
----------------------------------------------
Scintillation Photons Simulation - Top View
----------------------------------------------
Extra Slides
---------------
:small:`g4daeview.py : OpenGL view of Juno Geometry`
-------------------------------------------------------------------------------
.. class:: tiny
External view of Juno geometry with cutaway. The extreme size of the Juno geometry (50 million nodes in Chroma representation)
provides a challenge for development on mobile GPUs.
As my developments operate at the Geant4 level wherever possible it
was relatively straightforward to apply the machinery developed for
Dayabay to the Juno detector. In collaboration with
Juno simulation experts the geometry was exported from
Geant4 and GPU visualized in under a days work.
.. image:: /env/geant4/geometry/collada/g4daeview/20140716-194144.png
:height: 650px
:align: center
:small:`g4daeview.py : Chroma Raycast of Juno Geometry`
-------------------------------------------------------------------------------
.. class:: tiny
External view of Juno geometry. The extreme size of the Juno geometry (50 million nodes in Chroma representation)
provides a challenge for development on mobile GPUs. The black rectangle arises due to aborts to avoid GPU
crashes.
.. image:: /env/geant4/geometry/collada/g4daeview/20140716-191232.png
:height: 650px
:align: center
:small:`G4 Generated Cerenkov Wavelengths in material categories`
------------------------------------------------------------------------
.. class:: small
Blip caused by 200nm edge in water refractive index
.. image:: /env/g4dae/g4_cerenkov_wavelength.png
:width: 700px
:align: center
:small:`Refractive Index Interpolation Explains Jump at 200nm`
----------------------------------------------------------------
.. class:: small
DeadWater, IwsWater, OwsWater have same RINDEX starting from 200nm.
Chroma interpolates properties onto a standard set of wavelengths,
getting rid of the jump.
.. image:: /env/g4dae/plot_refractive_index_comparison.png
:width: 900px
:align: center
:small:`Comparison of Generated Scintillation Photon Distributions`
------------------------------------------------------------------------
.. class:: small
Position, direction, polarization XYZ + time, wavelength, weight
.. image:: /env/g4dae/generated_scintillation_3xyzw.png
:width: 700px
:align: center
:small:`Comparison of Generated Cerenokov Photon Distributions`
------------------------------------------------------------------------
.. class:: small
Position, direction, polarization XYZ + time, wavelength, weight
.. image:: /env/g4dae/generated_cerenkov_3xyzw.png
:width: 700px
:align: center
:small:`g4daeview.py : Dayabay Chroma Photon Propagation (1)`
-------------------------------------------------------------------------------
.. class:: tiny
Chroma GPU photon propagation at 12 nanoseconds. The photons are generated by Geant4
simulation of a 100 GeV muon travelling from right to left.
Photon colors indicate reemission (green), absorption(red),
specular reflection (magenta), scattering(blue), no history (white).
.. image:: /env/geant4/geometry/collada/g4daeview/20140716-161445.png
:height: 600px
:align: center