Opticks : GPU Optical Photon Simulation for Particle Physics with NVIDIA OptiX

Outline

• Context and Problem
  • Jiangmen Underground Neutrino Observatory (JUNO)
  • Optical Photon Simulation Problem...
• Tools to create Solution
  • Optical Photon Simulation ≈ Ray Traced Image Rendering
  • Rasterization and Ray tracing
  • Turing Built for RTX
  • BVH : Bounding Volume Hierarchy
  • NVIDIA OptiX Ray Tracing Engine
• Opticks : The Solution
  • Geant4 + Opticks Hybrid Workflow : External Optical Photon Simulation
  • Opticks : translates G4 optical physics to CUDA/OptiX
  • Opticks : translates G4 geometry to GPU, without approximation
  • CUDA/OptiX Intersection functions for ~10 primitives
  • CUDA/OptiX Intersection functions for Arbitrarily Complex CSG shapes
• Validation and Performance
  • Random Aligned Bi-Simulation -> Direct Array Comparison
  • Perfomance Scanning from 1M to 400M photons
• Overview + Links

JUNO_Intro_1

JUNO_Intro_4

JUNO_Intro_2

JUNO_Intro_3

Optical Photon Simulation Problem...

Optical Photon Simulation ≈ Ray Traced Image Rendering

Not a Photo, a Calculation

http://on-demand.gputechconf.com/siggraph/2013/presentation/SG3106-Building-Ray-Tracing-Applications-OptiX.pdf

Much in common : geometry, light sources, optical physics

• simulation : photon parameters at PMT detectors
• rendering : pixel values at image plane
• both limited by ray geometry intersection, aka ray tracing

Many Applications of ray tracing :

• advertising, design, architecture, films, games,...
• -> huge efforts to improve hw+sw over 30 yrs

August 2018 : Major Ray Tracing Advance

• NVIDIA RTX Platform, Turing GPU
• ray trace dedicated hardware : RT cores

SIGGRAPH_2018_Announcing_Worlds_First_Ray_Tracing_GPU

Ray-tracing vs Rasterization

TURING BUILT FOR RTX 2

NVIDIA RTX Metro Exodus

Spatial Index Acceleration Structure

Hardware Traversal of BVH Spatial Index

NVIDIA® OptiX™ Ray Tracing Engine -- http://developer.nvidia.com/optix

OptiX Raytracing Pipeline

Analogous to OpenGL rasterization pipeline:

OptiX makes GPU ray tracing accessible

• accelerates ray-geometry intersections
• simple : single-ray programming model
• "...free to use within any application..."
• access RT Cores[1] with OptiX 6.0.0+ via RTX™ mode

NVIDIA expertise:

• ~linear scaling up to 4 GPUs
• acceleration structure creation + traversal (Blue)
• instanced sharing of geometry + acceleration structures
• compiler optimized for GPU ray tracing

Opticks provides (Yellow):

• ray generation program
• ray geometry intersection+bbox programs

[1] Turing RTX GPUs

Geant4OpticksWorkflow

Opticks : translates G4 optical physics to CUDA/OptiX

GPU Resident Photons

Seeded on GPU

associate photons -> gensteps (via seed buffer)

Generated on GPU, using genstep param:

• number of photons to generate
• start/end position of step

Propagated on GPU

Only photons hitting PMTs copied to CPU

Thrust: high level C++ access to CUDA

• https://developer.nvidia.com/Thrust

OptiX : single-ray programming model -> line-by-line translation

CUDA Ports of Geant4 classes

• G4Cerenkov (only generation loop)
• G4Scintillation (only generation loop)
• G4OpAbsorption
• G4OpRayleigh
• G4OpBoundaryProcess (only a few surface types)

Modify Cerenkov + Scintillation Processes

• collect genstep, copy to GPU for generation
• avoids copying millions of photons to GPU

Scintillator Reemission

• fraction of bulk absorbed "reborn" within same thread
• wavelength generated by reemission texture lookup

Opticks (OptiX/Thrust GPU interoperation)

• OptiX : upload gensteps
• Thrust : seeding, distribute genstep indices to photons
• OptiX : launch photon generation and propagation
• Thrust : pullback photons that hit PMTs
• Thrust : index photon step sequences (optional)

Opticks : translates G4 geometry to GPU, without approximation

Automated : Geant4 "World" -> Opticks CSG -> CUDA/OptiX

• intersection functions for ~10 primitives
• intersection program for arbitrarily complex CSG shapes

Structure

• repeated geometry instances identified (progeny digests)
• instance transforms used in OptiX/OpenGL geometry
• merge CSG trees into global + instance buffers

Material/Surface/Scintillator properties

• interpolated to standard wavelength domain
• interleaved into "boundary" texture
• "reemission" texture for wavelength generation

CUDA/OptiX Intersection functions for ~10 primitives

• 3D parametric ray : ray(x,y,z;t) = rayOrigin + t * rayDirection
• implicit equation of primitive : f(x,y,z) = 0
• -> polynomial in t , roots: t > t_min -> intersection positions + surface normals

Sphere, Cylinder, Disc, Cone, Convex Polyhedron, Hyperboloid, Torus, ...

CUDA/OptiX Intersection Program for Arbitrarily Complex CSG shapes

Outside/Inside Unions

dot(normal,rayDir) -> Enter/Exit

• A + B boundary not inside other
• A * B boundary inside other

Complete Binary Tree, pick between pairs of nearest intersects:

• Nearest hit intersect algorithm [1] avoids state
  • sometimes Loop : advance t_min , re-intersect both
  • classification shows if inside/outside
• Evaluative [2] implementation emulates recursion:
  • recursion not allowed in OptiX intersect programs
  • bit twiddle traversal of complete binary tree
  • stacks of postorder slices and intersects
• Identical geometry to Geant4
  • solving the same polynomials
  • near perfect intersection match

[1] Ray Tracing CSG Objects Using Single Hit Intersections, Andrew Kensler (2006)

with corrections by author of XRT Raytracer http://xrt.wikidot.com/doc:csg

[2] https://bitbucket.org/simoncblyth/opticks/src/tip/optixrap/cu/csg_intersect_boolean.h

Similar to binary expression tree evaluation using postorder traverse.

j1808_top_rtx

j1808_top_ogl

Validation of Opticks Simulation by Comparison with Geant4

Bi-simulations of all JUNO solids, with millions of photons

mis-aligned histories

mostly < 0.25%, < 0.50% for largest solids

deviant photons within matched history

< 0.05% (500/1M)

Primary sources of problems

• grazing incidence, edge skimmers
• incidence at constituent solid boundaries

Primary cause : float vs double

Geant4 uses double everywhere, Opticks only sparingly (observed double costing 10x slowdown with RTX)

Conclude

• neatly oriented photons more prone to issues than realistic ones
• perfect "technical" matching not feasible
• instead shift validation to more realistic full detector "calibration" situation

Performance : Scanning from 1M to 400M photons

Full JUNO Geometry j1808v5

• "calibration" source genstep at center of scintillator

Production Mode : does the minimum

• only saves hits
• skips : genstep, photon, source, record, sequence, index, ..
• no Geant4 propagation (other than at 1M for extrapolation)

Multi-Event Running : measure interval and launch

interval : avg time between successive launches, including:

• upload gensteps
• launch : avg photon generation + propagation time
• download hits

NVIDIA Quadro RTX 8000 (48G)

scan-pf-1_NHit

scan-pf-1_Interval_over_Launch

scan-pf-1_Opticks_vs_Geant4 2

scan-pf-1_Opticks_Speedup 2

scan-pf-1_RTX_Speedup

Useful Speedup > 1000x : But Why Not Giga Rays/s ? (1 photon ~10 rays)

• NVIDIA claim : 10 Giga Rays/s with RT Core
• -> 1 Billion photons per second
• RT cores : built-in triangle intersect + 1-level of instancing
• flatten scene model to avoid SM<->RT roundtrips ?

OptiX Performance Tools and Tricks, David Hart, NVIDIA https://developer.nvidia.com/siggraph/2019/video/sig915-vid

Where Next for Opticks ?

JUNO+Opticks into Production

• optimize geometry modelling
• full JUNO geometry validation iteration
• offline integration
• optimize GPU cluster throughput:
  • multi-GPU strategy, split/join events to fit VRAM
• support OptiX 7, find multi-GPU load balancing approach

Geant4+Opticks Integration : Work with Geant4 Collaboration

• finalize Geant4+Opticks extended example
  • aiming for Geant4 distrib
• prototype Genstep interface inside Geant4
  • avoid customizing G4Cerenkov G4Scintillation

Expand Community : Webinars, Conference Tutorials ?

• geometry translation help : NEXO, DUNE, LZ
• interest -> usage : SABRE, Baikal GVD, KM3Net, MicroBooNE
• expand interest : scintillator using medical imaging companies

Meeting the Challenge of JUNO Simulation with Opticks

Opticks : state-of-the-art GPU ray tracing applied to optical photon simulation and integrated with Geant4 to eliminate memory and time bottlenecks.

• Drastic speedup -> better detector understanding -> greater precision
  • any simulation limited by optical photons can benefit

geocache_360