Opticks : GPU Optical Photon Simulation for Particle Physics with NVIDIA OptiX

Opticks Benefits

Outline

• Short video
• Problem and approach to solving
  • Optical Photon Simulation problem
  • Hybrid solution : Geant4 + Opticks
  • NVIDIA OptiX Ray Tracing Engine
• Opticks : optical photon simulation with NVIDIA OptiX
  • geometry workflow : geocache, GPU textures
  • large geometry techniques, geometry modelling
  • Geant4/Opticks event workflow
• Validation, performance comparisons
• Summary

Optical Photon Simulation Problem...

JPMT Before Contact 2

Ray Traced Image Synthesis ≈ Optical Photon Simulation

OptiX Pixel Calculation

http://on-demand.gputechconf.com/siggraph/2013/presentation/SG3106-Building-Ray-Tracing-Applications-OptiX.pdf

Geometry, light sources, optical physics ->

• pixel values at image plane
• photon parameters at detectors (eg PMTs)

Ray tracing has many applications :

• advertising, design, entertainment, games,...
• BUT : most ray tracers just render images

NVIDIA OptiX had foresight to be less specific:

• general geometry intersection API
• OptiX is to ray tracing what OpenGL is to rasterization

rasterization

project 3D primitives onto 2D image plane, combine fragments into pixel values

ray tracing

cast rays thru image pixels into scene, recursively reflect/refract with geometry intersected, combine returns into pixel values

NVIDIA OptiX 1

NVIDIA OptiX 2

https://research.nvidia.com/publication/optix-general-purpose-ray-tracing-engine

Opticks : Translate Geant4 Geometry "Context" for GPU

Export Tesselated Geometry

• Geant4 -> G4DAE XML https://bitbucket.org/simoncblyth/g4dae

Load G4DAE XML

• label primitives with boundary indices
• find repeated geometry "instances"
• write NumPy serialization .npy geocache

Geocache -> few seconds startup, instead of few minutes

Load/upload geocache

• load geocache from file
• upload OptiX textures to GPU
• upload geometry buffers to GPU

GPU geometry buffers read by:

• OptiX bounding box and intersection programs
• OpenGL GLSL shaders for visualization

Large Geometry Techniques : Instancing Mandatory

Geometry analysed to find instances

JUNO: ~90M --> 0.1M triangles

• 18k 20" PMTs
• 36k 3" PMTs

OpenGL/OptiX instancing

Multiple Renderers

• global, full instances, bbox instances
• rapid switching controls

Geometry Modelling : Tesselated vs Analytic Photomultiplier Tubes

Analytic : more realistic, faster, less memory, much more effort

For Dayabay PMT:

• partition CSG solids into 12 single primitive parts (instead of 2928 triangles)
• splitting at geometrical intersections avoids implementing general CSG boolean handling
• geometry provided to OptiX in form of ray intersection and bounding box code

Aim : analytic description of geometry on critical optical path, remainder tesselated

Hybrid Geant4/Opticks Event Workflow

GPU Resident Photons

Seeded on GPU

associate photons -> gensteps (via seed buffer)

Generated on GPU, using gensteps:

~Stacks collected before photon generation:

• number of photons to generate
• start/end position of step
• other quantities needed for GPU generation

Propagated on GPU

Only photons hitting PMTs copied to CPU

Thrust: high level C++ access to CUDA

• https://developer.nvidia.com/Thrust

Geant4/Detector Simulation

• modified Scintillation, Cerenkov processes
  • collect genstep
  • skip optical photon generation

Opticks (OptiX/Thrust GPU interoperation)

• OptiX : upload gensteps
• Thrust : seeding, distribute genstep indices to photons
• OptiX : launch photon generation and propagation
• Thrust : pullback photons that hit PMTs
• Thrust : index photon step sequences (optional)

Geant4/Detector Simulation

• populate standard hit collections
• subsequent electronics simulation proceeds unaltered

Multi-event handling

• reuse/resize OptiX buffers for each event

Compare Opticks/Geant4 Simulations with Simple Lights/Geometries

1M Photons -> Water Sphere (S-Polarized)

0.5M Photons -> Dayabay PMT

Photon step records

128 bit per step : highly compressed position, time, wavelength, polarization vector, material/history codes

Photon flag sequence

16x 4-bit step flags recorded in uint64 sequence, indexed using Thrust GPU sort (1M indexed ~0.040s)

Sequence index -> interactive OpenGL selection of photons by flag sequence

1M Rainbow S-Polarized, Comparison Opticks/Geant4

Deviation angle(degrees) of 1M parallel monochromatic photons in disc shaped beam incident on water sphere. Numbered bands are visible range expectations of first 11 rainbows. S-Polarized intersection (E field perpendicular to plane of incidence) arranged by directing polarization radially.

PMT Opticks/Geant4 step distribution comparison TO BT [SD]

Good agreement reached, after several fixes: geometry, total internal reflection, group velocity

position(xyz), time(t)

polarization(abc), radius(r)

PMT Opticks/Geant4 step distribution comparison : chi2/ndf

tconcentric : test geometry configured via boundaries

• Idealized geometry for testing optical physics in situation with no geometry issues.
• Boundaries specify : outer-material/outer-surface/inner-surface/inner-material

tconcentric-testconfig(){
    local test_config=(
        mode=BoxInBox
        analytic=1
        shape=sphere boundary=StainlessSteel///Acrylic         parameters=0,0,0,$(( 5000 + 5 ))
        shape=sphere boundary=Acrylic//RSOilSurface/MineralOil parameters=0,0,0,$(( 5000 - 5 ))
        shape=sphere boundary=MineralOil///Acrylic             parameters=0,0,0,$(( 4000 + 5 ))
        shape=sphere boundary=Acrylic///LiquidScintillator     parameters=0,0,0,$(( 4000 - 5 ))
        shape=sphere boundary=LiquidScintillator///Acrylic     parameters=0,0,0,$(( 3000 + 5 ))
        shape=sphere boundary=Acrylic///GdDopedLS              parameters=0,0,0,$(( 3000 - 5 ))
    )
    echo "$(join _ ${test_config[@]})"
}

https://bitbucket.org/simoncblyth/opticks/src/tip/tests/tconcentric.bash

tconcentric : spherical GdLS/LS/MineralOil

tconcentric : Opticks/Geant4 chi2 comparison

.      seqhis_ana  1:concentric   -1:concentric     c2
.                       1000000   1000000       373.13/356 =  1.05  (pval:0.256 prob:0.744)
0000           8ccccd    669843    670001             0.02  [6 ] TO BT BT BT BT SA
0001               4d     83950     84149             0.24  [2 ] TO AB
0002          8cccc6d     45490     44770             5.74  [7 ] TO SC BT BT BT BT SA
0003           4ccccd     28955     28718             0.97  [6 ] TO BT BT BT BT AB
0004             4ccd     23187     23170             0.01  [4 ] TO BT BT AB
0005          8cccc5d     20238     20140             0.24  [7 ] TO RE BT BT BT BT SA
0006          8cc6ccd     10214     10357             0.99  [7 ] TO BT BT SC BT BT SA
0007          86ccccd     10176     10318             0.98  [7 ] TO BT BT BT BT SC SA
0008          89ccccd      7540      7710             1.90  [7 ] TO BT BT BT BT DR SA
0009         8cccc55d      5976      5934             0.15  [8 ] TO RE RE BT BT BT BT SA
0010              45d      5779      5766             0.01  [3 ] TO RE AB
0011  8cccccccc9ccccd      5339      5269             0.46  [15] TO BT BT BT BT DR BT BT BT BT BT BT BT BT SA
0012          8cc5ccd      5111      4940             2.91  [7 ] TO BT BT RE BT BT SA
0013              46d      4797      4886             0.82  [3 ] TO SC AB
0014      8cccc9ccccd      4494      4469             0.07  [11] TO BT BT BT BT DR BT BT BT BT SA
0015      8cccccc6ccd      3317      3302             0.03  [11] TO BT BT SC BT BT BT BT BT BT SA
0016         8cccc66d      2670      2675             0.00  [8 ] TO SC SC BT BT BT BT SA
0017          49ccccd      2432      2383             0.50  [7 ] TO BT BT BT BT DR AB
0018          4cccc6d      2043      1991             0.67  [7 ] TO SC BT BT BT BT AB
0019            4cc6d      1755      1826             1.41  [5 ] TO SC BT BT AB

Top 20 chart above, (category 100 down to ~100 photons for propagation of 1M photons)

tconcentric : Opticks/Geant4 history counts chi2/df ~ 1.0

Match achieved between two optical simulations, run from single executable

CPU

Daya Bay detsim optical physics updated to G4 10.2

GPU

above ported to CUDA/OptiX

Match achieved only after many bug fixes, including:

• scattering, was comparing different implementations
• diffuse reflection, ported G4 approach
• reemission "rejoining"
• missing optical surfaces
• truncation recording discrepancy
• group velocity of wrong material after refraction (G4 issue 1275)
• interpolation mismatches

Work guided by the next largest chi2 contributor

https://bugzilla-geant4.kek.jp/show_bug.cgi?id=1275

Photon Propagation Times Geant4 cf Opticks

• Opticks > 200X Geant4 with only 384 core mobile GPU[1] (multi-GPU workstation up to 20x more cores)

• Interop uses OpenGL buffers allowing visualization, Compute uses OptiX buffers
• Interop/Compute : perfectly identical results, monitored by digest

[1] MacBook Pro (2013), NVIDIA GeForce GT 750M, 2048 MB, 384 cores

OpticksDocs

import numpy as np
a = np.load("photons.npy")

Summary

Opticks enables Geant4 based simulations to benefit from optical photon simulation taking effectively zero time and zero CPU memory, thanks to massive parallelism made accessible by NVIDIA OptiX.

• The more photons the bigger the overall speedup (99% -> 100x)
• Drastic speedup -> better detector understanding -> greater precision
• Large PMT based neutrino experiments, such as JUNO, can benefit the most

https://bitbucket.org/simoncblyth/opticks

OptiX ray trace of boolean CSG box-sphere, photons approaching

Boolean CSG may allow fully analytic GPU geometry translations

tboolean box-sphere after

Photon propagation step point positions.

YouTube Video

https://www.youtube.com/watch?v=CBpOha4RzIs

Daya Bay Composite Ray Traced geometry and rasterized Cerenkov propagation