Opticks : GPU Optical Photon Simulation for Particle Physics with NVIDIA OptiX

Progress with JUNO + Opticks : GPU Optical Photon Simulation with NVIDIA® OptiX™

Simon C Blyth, IHEP — https://bitbucket.org/simoncblyth/opticks — July 2019, IHEP

Outline

Introduction
- Optical Photon Simulation Problem...
- Rasterization and Ray Tracing
- Ray Traced Rendering ~ Photon Simulation
- NVIDIA TITAN RTX GPU
- NVIDIA OptiX Ray Tracing Engine
- Boundary Volume Heirarchy (BVH) algorithm
- Opticks Primitives, CSG on GPU
- Translate Geometry and Physics
- Geometry changes enabling translation
p15 Updates : Support OptiX 6.0.0, RTX mode, RT Cores
- JUNO-360 benchmark, RTX mode speedup, multi-GPU scaling
p21 Validation : Systematic Scan of 40 JUNO Solids
- Random Aligned Bi-Simulation -> Direct Array Comparison
p34 Performance : Scanning from 1M to 60M photons
- Opticks vs Geant4 performance (test geometry)
Opticks : Aims to Revolutionize JUNO Muon Simulation
(Tao Lin, Simulation Group) Deferred OP Sim using Gensteps
Opticks Links

Optical Photon Simulation Problem...

JPMT Before Contact 2

Ray-tracing vs Rasterization

/env/presentation/nvidia/nv_rasterization.png

/env/presentation/nvidia/nv_raytrace.png

Rendering + Photon Simulation : Limited by Ray Geometry Intersection

Ray Tracing Tools can Help Optical Photon Simulation

industry continuously improving ray trace performance
NVIDIA Turing GPU : raytrace dedicated "RT Cores"
- Up to "11 GigaRays per second" per GPU

https://nvidianews.nvidia.com/news/nvidia-reveals-the-titan-of-turing-titan-rtx

TITAN RTX : 72 Raytrace Dedicated RT Cores, 4608 CUDA Cores, 24GB VRAM, 2500 USD

NVIDIA® OptiX™ Ray Tracing Engine -- http://developer.nvidia.com/optix

OptiX makes GPU ray tracing accessible

accelerates ray-geometry intersections
simple : single-ray programming model
"...free to use within any application..."
access RT Cores[1] with OptiX 6.0.0+ via RTX™ mode

NVIDIA expertise:

~linear scaling up to 4 GPUs
acceleration structure creation + traversal (Blue)
instanced sharing of geometry + acceleration structures
compiler optimized for GPU ray tracing

https://developer.nvidia.com/rtx

User provides (Yellow):

ray generation
geometry bounding box, intersects

[1] Turing+ GPUs eg NVIDIA TITAN RTX

BVH

Opticks : GPU Geometry starts from ray-primitive intersection

3D parametric ray : ray(x,y,z;t) = rayOrigin + t * rayDirection
implicit equation of primitive : f(x,y,z) = 0
-> polynomial in t , roots: t > t_min -> intersection positions + surface normals

/env/presentation/tboolean_parade_sep2017.png

CUDA/OptiX intersection for ~10 primitives -> Exact geometry translation

Ray intersection with general CSG binary trees, on GPU

Pick between pairs of nearest intersects, eg:

UNION tA < tB	Enter B	Exit B	Miss B
Enter A	ReturnA	LoopA	ReturnA
Exit A	ReturnA	ReturnB	ReturnA
Miss A	ReturnB	ReturnB	ReturnMiss

Nearest hit intersect algorithm [1] avoids state
- sometimes Loop : advance t_min , re-intersect both
- classification shows if inside/outside
Evaluative [2] implementation emulates recursion:
- recursion not allowed in OptiX intersect programs
- bit twiddle traversal of complete binary tree
- stacks of postorder slices and intersects
Identical geometry to Geant4
- solving the same polynomials
- near perfect intersection match

[1] Ray Tracing CSG Objects Using Single Hit Intersections, Andrew Kensler (2006): with corrections by author of XRT Raytracer http://xrt.wikidot.com/doc:csg
[2] https://bitbucket.org/simoncblyth/opticks/src/tip/optixrap/cu/csg_intersect_boolean.h: Similar to binary expression tree evaluation using postorder traverse.

Opticks : Translate Geant4 Geometry to GPU, Without Approximation

Direct Geometry : Geant4 "World" -> Opticks CSG -> GPU

much simpler : fully automated geo-management

Material/Surface/Scintillator properties

interpolated to standard wavelength domain
interleaved into "boundary" texture
"reemission" texture for wavelength generation

Structure

repeated geometry instances identified (progeny digests)
instance transforms used in OptiX/OpenGL geometry
merge CSG trees into global + instance buffers
export meshes to glTF 2.0 for 3D visualization

Ease of Use

easy geometry : just handover "World"

[1] G4 primitives used need corresponding Opticks implementations, contributions for: any unsupported geometry are welcome

JUNO Geometry Changes to enable translation to GPU

First 2 change geometry, last 2 change modelling only

Guide Tube Torus : removed (for now): AVOIDED : OptiX 6.0.0 NOT working with torus intersect
PMT_20inch_body : simplified neck: FIXED : "cylinder - torus" -> polycone
PMT_20inch_inner : simplified CSG modelling: FIXED : depth 4 tree (31 nodes) -> 1 primitive
sAirTT : CSG modelling coincidence avoided: FIXED : "box - cylinder" : growing the subtracted

20inch PMT neck : "cylinder-torus" -> polycone

Opticks : Translate Geant4 Optical Physics to GPU (OptiX/CUDA)

OptiX : single-ray programming model -> line-by-line translation

CUDA Ports of Geant4 classes

G4Cerenkov (only generation loop)
G4Scintillation (only generation loop)
G4OpAbsorption
G4OpRayleigh
G4OpBoundaryProcess (only a few surface types)

Modify Cerenkov + Scintillation Processes

collect genstep, copy to GPU for generation
avoids copying millions of photons to GPU

Scintillator Reemission

fraction of bulk absorbed "reborn" within same thread
wavelength generated by reemission texture lookup

Opticks (OptiX/Thrust GPU interoperation)

OptiX : upload gensteps
Thrust : seeding, distribute genstep indices to photons
OptiX : launch photon generation and propagation
Thrust : pullback photons that hit PMTs
Thrust : index photon step sequences (optional)

geocache_360

JUNO-360 benchmark with OptiX 6.0.0, RTX mode

bench_20190526_202537

bench_20190526_202537 2

bench_20190526_143808

bench_20190526_143808 2

Validation : Systematic Scan of All 40 JUNO Solids

Create test geometries using solids, materials, surfaces from base geometry geocache

40 geometries: test containers for all JUNO solids
input photons: feed same initial photons to Opticks and Geant4
input cuRAND randoms to Geant4: same random sequence for Opticks and Geant4
bi-simulation executable: save propagations from Opticks and Geant4 as OpticksEvents (.npy arrays)

Validation : Random Aligned Bi-Simulation -> Direct Array Comparison

NPY serialized arrays

convenient analysis with NumPy
np.load(path, mmap_mode="r") : loading array slices





In [11]: pdv = np.where(dv > 0.0001)[0]
In [12]: ab.dumpline(pdv)
      0   1230 : TO BR SC BT BR BT SA
      1   2413 : TO BT BT SC BT BR BR BT SA
      2   9041 : TO BT SC BR BR BR BR BT SA
      3  14510 : TO SC BT BR BR BT SA
      4  14747 : TO BT SC BR BR BR BR BR BR BR
      5  14747 : TO BT SC BR BR BR BR BR BR BR
    ...

In [20]: ab.b.ox[pdv,0]                                 In [21]: ab.a.ox[pdv,0]
Out[20]:                                                Out[21]:
A()sliced                                               A()sliced
A([    [-191.6262, -240.3634,  450.    ,    5.566 ],    A([    [-191.626 , -240.3634,  450.    ,    5.566 ],
       [ 185.7708, -133.8457,  450.    ,    7.3141],           [ 185.7708, -133.8456,  450.    ,    7.3141],
       [-450.    , -104.4142,  311.143 ,    9.0581],           [-450.    , -104.4142,  311.1431,    9.0581],
       [  83.6955,  208.9171, -450.    ,    5.6188],           [  83.6954,  208.9172, -450.    ,    5.6188],
       [  32.8972,  150.    ,   24.9922,    7.6757],           [  32.8973,  150.    ,   24.992 ,    7.6757],
       [  32.8972,  150.    ,   24.9922,    7.6757],           [  32.8973,  150.    ,   24.992 ,    7.6757],
       [ 450.    , -186.7449,  310.6051,    5.0707],           [ 450.    , -186.7451,  310.605 ,    5.0707],
       [ 299.2227,  318.1443, -450.    ,    4.8717],           [ 299.2229,  318.144 , -450.    ,    4.8717],
 ...

http://bitbucket.com/simoncblyth/opticks/src/tip/notes/issues/tboolean_box_perfect_alignment_small_deviations.rst

tv21_1M_a

tv21_1M_a 2

tv21_1M_c

tv16_Fastener

absmry_1M

absmry_1M 2

ta34_1M

ta34_1M 2

Validation Issue 1 : History Mis-alignment : 0.01-0.50% of photons

Large Range 0.01% 0.50% 3% depending on extent

PMT_20inch_inner2_solid(LV 19) 163/1M

extra BR(boundary reflect) or BT(boundary transmit)

sWorld(LV 39) 29906/1M ~3%

large extent : direct photons travel 360m
many "maxbounce" truncations, lots of scattering


ab.mal (LV 19)
aligned   999837/1000000 : 0.9998 : 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24
maligned     163/1000000 : 0.0002 : 2908,4860,5477,12338,17891,18117,28709,32764,37671,43675,45874,...

      0   2908 : * :                   TO BT BR BR BT SA                 TO BT BR BR BR BT SA
      1   4860 : * :                      TO BT BT BT SA                    TO BT BT BT BT SA
      2   5477 : * :                      TO BT BT BT SA                    TO BT BT BT BT SA
      3  12338 : * :                      TO BT BR BT SA        TO BT BR BR BR BR BR BR BR BR
...
ab.mal (LV 39)
aligned   970094/1000000 : 0.9701 : 0,1,2,4,5,6,7,8,9,10,11,12,13,14,16,17,18,19,20,21,22,23,24,26,27
maligned   29906/1000000 : 0.0299 : 3,15,25,80,132,288,294,329,376,377,379,436,511,536,655,662,...
slice(0, 25, None)
      0      3 : * :        TO BT SC BR BR BR BR BR BR BR        TO BT SC BR BR BR BR BR SC BR
      1     15 : * :        TO BT SC BR BR BR BR BR SC BR        TO BT SC BR BR SC BT SC BT SC
      2     25 : * :                 TO BT SC BR BR BR AB        TO BT SC BR BR BR BR BR BR BR
      3     80 : * :                 TO BT BT SC BT BR AB        TO BT BT SC BT BR BR SC BR AB
...

[1] https://bitbucket.org/simoncblyth/opticks/src/tip/notes/issues/alignment_options_review.rst

Validation Issue 2 : History Aligned but Deviant : Mostly < 0.05% of photons

float vs double precision

Likely cause of deviations

small deviations grow from bounce to bounce
"lever arm" amplification of small errors depending on geometry
Opticks uses double only when unavoidable (eg scattering/absorption distance from log(u) [1])
Geant4 uses double everywhere
double has high performance cost on GPU
OptiX can only partially[2] operate in double precision

Plan

attempt to isolate sources, with double precision tests

[1] https://bitbucket.org/simoncblyth/opticks/src/tip/optixrap/cu/propagate.h

[2] https://devtalk.nvidia.com/default/topic/1036155/optix/double-precision-of-ray-lengths/

Performance : Scanning from 1M to 60M photons

tboolean-box test Geometry + input photons

input photons : convenient for debugging, but high overheads

Production Mode : does the minimum

only saves hits
skips : genstep, photon, source, record, sequence, index, ..
no Geant4 propagation

Multi-Event Running : measure interval and launch

interval : avg time between successive launches (includes all overheads)

upload photons/gensteps
launch : avg photon generation + propagation time
download hits

[1] TCURAND GPU randoms on host https://bitbucket.org/simoncblyth/opticks/src/tip/thrustrap/TCURAND.hh

NHit

Opticks_vs_Geant4

Opticks_vs_Geant4 2

Overheads

Opticks_Speedup

RTX_Speedup

Opticks[1] : Aims to Revolutionize JUNO Muon Simulation

State-of-the-art GPU ray tracing[2] applied to optical simulation

replaces Geant4 optical simulation with GPU equivalent
- translate G4 geometry to GPU without approximation[3]
- port G4 optical physics to CUDA[4]

Optical photons generated+propagated entirely on GPU

only photons hitting PMTs require CPU memory
- optical photon CPU memory --> ~zero
muon workload perfect for GPUs, Opticks >> 1000x Geant4
- optical photon simulation time --> ~zero

Status : validation iterations ongoing

direct comparison of random aligned bi-simulations
geometry changes made at GDML level

[1] Open source project http://bitbucket.org/simoncblyth/opticks

[2] NVIDIA OptiX ray tracing engine

[3] using innovative Constructive Solid Geometry implementation on GPU

[4] scattering, boundary, reemission, absorption

genstep_interface

Opticks Links

https://simoncblyth.bitbucket.io: Opticks presentations and videos
https://juno.ihep.ac.cn/cgi-bin/Dev_DocDB/ShowDocument?docid=3927: Opticks visualization screen capture movie of JUNO
https://groups.io/g/opticks: Opticks mailing list archive
opticks+subscribe@groups.io: send email to this address, to subscribe

https://simoncblyth.bitbucket.io/opticks/index.html: Opticks installation instructions
https://bitbucket.org/simoncblyth/opticks: Opticks code repository