Opticks : GPU Optical Photon Simulation

http://simoncblyth.bitbucket.io/env/presentation/opticks_gpu_optical_photon_simulation.html (Jan 2016) http://simoncblyth.bitbucket.io/env/presentation/optical_photon_simulation_with_nvidia_optix.html (July 2015)

Executive Summary

• Validation comparisons with Geant4 started, rainbow geometry validated
• Performance factors so large they become irrelevant (>> 200x)

Contents

• Brief History of GPU Optical Photon Simulation Development
• Introducing Opticks
• Handling JUNO geometry
• Mesh Fixing with OpenMesh surgery
• Analytic PMT geometry description
• Opticks/Geant4 : Dynamic Test Geometry
• Rainbow Geometry Testing
• Summary

Brief History of GPU Optical Photon Simulation Development

winter 2014 (within Chroma)

• integrate G4DAE geometry exports
• generate Cerenkov/Scintillation photons on GPU

spring 2015 (start transition to Opticks)

• decide lack of multi-GPU is showstopper for Chroma
• discover NVIDIA OptiX ray tracing framework
• create Opticks (using OptiX) to replace Chroma

summer/autumn 2015 (Opticks transition completed)

• infrastructure operational, G4 optical physics ported
• large geometry support added using instancing

autumn/winter 2015 (Validations begin)

• major tesselation bug avoided with OpenMesh
• develop analytic PMT description
• validation machinery created CfG4

Newton published Opticks in 1704

Based on: NVIDIA OptiX Ray Tracing Engine [C++/C/CUDA]

OptiX Tutorial App

Image pixels calculated by recursively bouncing rays doing shadow, reflection, refraction calculations. Runs at interactive speeds with GeForce GT 750M.

OptiX provides: CUDA compiler optimized for Ray Tracing

• ray tracing framework, no rendering assumptions
• state-of-the-art GPU accelerated geometry intersection
• regular releases, improvements, tuning for new GPUs
• shared C++/CUDA context eases development

NVIDIA expertise on efficient GPU/multi-GPU usage

• persistent warps sized to fill machine
• load balancing between warps, GPUs

IHEP Performance Check

Used IHEP 4 GPU workstation to verify near linear performance scaling across multiple GPUs

https://developer.nvidia.com/optix

https://research.nvidia.com/publication/optix-general-purpose-ray-tracing-engine

NVIDIA OptiX In Action

Introducing Opticks : Recreating G4 Context on GPU

Basis packages

• Opticks : definitions, configuration
• NPY : host array handling, persistency, python analysis
• NumpyServer : network IO of NPY arrays

Geometry packages

• AssimpWrap : G4DAE geometry loading with Assimp fork
• GGeo : preparing geometry for GPU
• OpenMeshRap : mesh fixing

GPU library interface packages

• OpticksOp : high level GPU control
• OptiXRap : OptiX control
• ThrustRap : photon indexing
• CUDAWrap : persist cuRAND state
• OGLWrap : OpenGL visualization

Main packages

• GGeoView : visualization, Opticks main
• CFG4 : Geant4 10.2 comparison, separate main

Large Geometry/Event Techniques

Instancing implemented for OpenGL and OptiX

Geometry analysed to find instances

• repeated sub-trees with transforms for each instance
• JUNO: ~90M triangles, instancing reduces to 0.1M

Switchable Rendering

• Multiple renderers: global, full instances, bbox instances
• Rapid switching GUI control

Compute Mode

• Pure OptiX buffers faster (~7X interop)
• Less GPU memory (1)
• Visualize by separate event loading

Ideas to investigate

• Analytic PMT definition expected to improve OptiX efficiency
• Culling non-visible geometry to improve OpenGL performance
• Level-of-detail : change geometry detail based on distance

1. Interop forced to duplicate data as OpenGL/OptiX geometry sharing not operational

JUNO Geometry Instance Rendering Control

JPMT Inside Wide

http://simoncblyth.bitbucket.io/env/graphics/ggeoview/jpmt-inside-wide_half.png

Mesh Fixing with OpenMesh surgery

After fix

Color indicates material assigned. Prior to fix, assignments incorrect for all photons heading towards Dayabay AD top lid.

For topologically correct meshes, without holes:

V - E + F = 2 (Eulers Characteristic)

Intersection boundaries determines photon material, bad meshes cause incorrect material assignments

G4Polyhedron Tesselation Bug

• Some union solids become cleaved meshes
• close parallel faces cause flickering OpenGL render
• ~25/250 Dayabay meshes have issues
• two critical ones fixed: IAV, OAV.
• TODO: check/apply to JUNO geometry

OpenMeshRap finds/fixes cleaved meshes

• based on open source project: OpenMesh
• extracts real topological meshes
• finds close parallel faces between the meshes
• deletes the extra faces
• stitches the split mesh together with added triangles

Analytic PMT geometry : more realistic, faster, less memory

• parse geometry XML to give CSG tree : solids with boolean intersections, unions
• partition 5 solids into 12 single primitive parts
• splitting at geometrical intersections avoids implementing general CSG boolean handling

Analytic PMT in 12 parts instead of 2928 triangles

Partitioned PMT

Geometry provided to OptiX in form of ray intersection code

PMT intersection by comparison with part intersections: cylinder and partial sphere intersection from quadratic roots

Volume to Surface translation

Volume heirarchy: Pyrex/Vacuum/Bialkali puts photocathode inside vacuum but as coincident boundaries it makes no difference for volume description.

Coincident boundaries do not work for surface description, must adopt correct heirarchy: Pyrex/Bialkali/Vacuum

TODO: single PMT testing vs G4 to check implications

Sphere intersection only 2 cases, Cylinder 10 cases (axial, walls, endcap, outside/inside)

OptiX Ray Traced Analytic PMT geometry

Near clipped, orthographic projection.

Analytic PMTs together with triangulated geometry

Dayabay Ray Trace: PMTs analytic, the rest triangulated

Opticks/Geant4 : Dynamic Test Geometry

Commandline arguments parsed into geometry/material/surface description. Each test shape requires:

• OptiX: analytic intersection code
• OpenGL: tesselation for visualization
• Geant4: geometry construction code in CfG4 package

Compare Opticks/Geant4 propagations with simple test geometries with:

     ggv.sh        --test --testconfig "..."  --torch --torchconfig "..."   # Opticks
     ggv.sh --cfg4 --test --testconfig "..."  --torch --torchconfig "..."   # Geant4

Opticks Absolute Reflection compared to Fresnel expectation

Comparison of simulated absolute reflection of S and P polarized single events against expectation from Fresnel formula. Using uniform planar incident cyclindrically directed light.

Multiple Rainbows from a Single Drop of Water

Caustic bunching at least deviation causes rainbow

1. Primary bow, single reflection : ray f at bow angle
2. Secondary bow, double reflection : ray g at bow angle
3. Deviation angles 0:180 degrees hemisphere opposite to incident rays

Jearl D. Walker, 1975, Multiple rainbows from single drops of water and other liquids http://patarnott.com/atms749/pdf/MultipleRainbowsSingleDrops.pdf

Disc beam 1M Photons incident on Water Sphere (S-Pol)

Photons shown by lines with color representing polarization direction. S-Polarized (perpendicular to plane of incidence) intersection by disc radially directed polarization. Geodesic icosahedron tesselation just for OpenGL visualization, actual OptiX geometry is perfect sphere.

2ns later, Several Bows Apparent

Opticks/Geant4 Photon Step Sequence Comparison

• BT/BR: boundary transmit/reflect
• TO/SC/SA: torch/scatter/surface absorb

1M flag sequences indexed using CUDA Thrust, 0.040 s

 64-bit uint  Opticks    Geant4    chi2                                      (tag:5,-5)

        8ccd   819160    819654    0.15  [4 ] TO BT BT SA                    (cross droplet)
         8bd   102087    101615    1.09  [3 ] TO BR SA                       (external reflect)
       8cbcd    61869     61890    0.00  [5 ] TO BT BR BT SA                 (bow 1)
      8cbbcd     9618      9577    0.09  [6 ] TO BT BR BR BT SA              (bow 2)
     8cbbbcd     2604      2687    1.30  [7 ] TO BT BR BR BR BT SA           (bow 3)
    8cbbbbcd     1056      1030    0.32  [8 ] TO BT BR BR BR BR BT SA        (bow 4)
       86ccd     1014      1000    0.10  [5 ] TO BT BT SC SA
   8cbbbbbcd      472       516    1.96  [9 ] TO BT BR BR BR BR BR BT SA     (bow 5)
         86d      498       473    0.64  [3 ] TO SC SA
  bbbbbbbbcd      304       294    0.17  [10] TO BT BR BR BR BR BR BR BR BR  (bow 8+ truncated)
  8cbbbbbbcd      272       247    1.20  [10] TO BT BR BR BR BR BR BR BT SA  (bow 6)
  cbbbbbbbcd      183       161    1.41  [10] TO BT BR BR BR BR BR BR BR BT  (bow 7 truncated)
         4cd      161       139    1.61  [3 ] TO BT AB
       8c6cd      153       106    8.53  [5 ] TO BT SC BT SA
        86bd      138       142    0.06  [4 ] TO BR SC SA
        4ccd      100       117    1.33  [4 ] TO BT BT AB

Rainbow deviation angles

Deviation angle distribution of all 3M photons. Photon wavelengths from Plankian 6500K blackbody spectrum (implemented with inverse CDF GPU texture lookup). Simulated "images" obtained from wavelength spectrum of each bin using CIEXYZ weighting functions converted into sRGB/D65 colorspace. The two images exposed for luminance (CIE-Y) of max bin of 1st and 2nd bows.

Rainbow Spectrum for 1st six bows

Spectra obtained by selecting photons by internal reflection counts. Colors obtained from spectra of each bin using CIEXYZ weighting functions converted into sRGB/D65 colorspace. Exposures by normalizing to bin with maximum luminance (CIE-Y) of each bow. White lines indicate geometric optics prediction of deviation angle ranges of the visible range 380-780nm. 180-360 degrees signifies exit on same side of droplet as incidence.

1M Rainbow S-Polarized, Comparison Opticks/Geant4

Deviation angle(degrees) of 1M parallel monochromatic photons in disc shaped beam incident on water sphere. Numbered bands are visible range expectations of first 11 rainbows. S-Polarized intersection (E field perpendicular to plane of incidence) arranged by directing polarization radially.

1M Rainbow P-Polarized, Comparison Opticks/Geant4

Deviation angle(degrees) of 1M parallel monochromatic photons in disc shaped beam incident on water sphere. Numbered bands are visible range expectations of first 11 rainbows. P-Polarized intersection (E field within plane of incidence) arranged by directing polarization tangentially.

Performance Comparison Opticks/Geant4

Rainbow 1M S-Pol

Average Propagate Time for 1M photons

MacBook Pro 2013, NVIDIA GeForce GT 750M 2GB (384 cores)

• Opticks ~200X Geant4 with only 384 core mobile GPU
• multi-GPU workstation up to 20x more cores
• photon propagation time will become effectively zero

Opticks Interop/Compute Modes

• perfectly identical results, monitored by digest
• Interop: uses OpenGL buffers allowing visualization
• Compute: uses OptiX buffers
• Compute (and G4) propagated events can be visualized by loading into interop mode viewer

Summary

Opticks Transition Complete

• squeezed JUNO geometry onto GPU with instancing
• devised mesh fixing workaround for G4Polyhedron bug
• developed analytic PMT representation
• photons fully GPU resident, only copy PMT hits to CPU

Opticks Validation

• developed validation machinery CfG4
• simple rainbow geometry matches Geant4

Next Tests

• Prism, Lens, Fresnel Rhomb (2x TIR)
• Single Analytic PMT
• Full geometries

JPMT Wide

http://simoncblyth.bitbucket.io/env/graphics/ggeoview/jpmt-wide_half.png

JPMT Before Contact

http://simoncblyth.bitbucket.io/env/graphics/ggeoview/jpmt-before-contact_half.png

JPMT After Contact

http://simoncblyth.bitbucket.io/env/graphics/ggeoview/jpmt-after-contact_half.png

JPMT Approach

http://simoncblyth.bitbucket.io/env/graphics/ggeoview/jpmt-approach_half.png

JPMT Arrival

http://simoncblyth.bitbucket.io/env/graphics/ggeoview/jpmt-arrival_half.png

JPMT Inside Outside

http://simoncblyth.bitbucket.io/env/graphics/ggeoview/jpmt-inside-outside_half.png

JPMT Headview

http://simoncblyth.bitbucket.io/env/graphics/ggeoview/jpmt-headview_half.png

JPMT Backview

http://simoncblyth.bitbucket.io/env/graphics/ggeoview/jpmt-backview_half.png

EXTRAS : Details and Tests in progress

Details

• Optical Photons now fully GPU resident
• Volume to Surface Translation details need validation
• OptiX Geometry Experience

Tests requiring G4 comparison

• Prism S-Polarized
• Prism P-Polarized
• Prism Deviation vs Incident angles for 10 wavelengths
• Lens focussing

Optical Photons now fully GPU resident

All photon operations now done on GPU:

• seeded (assigned gensteps)
• generated
• propagated
• indexed material/interaction histories

Only PMT hits need to be copied back to CPU

Thrust/CUDA/OptiX interop used to implement

Volume to Surface Translation details need validation

Zoomed view of PMT edge (mm): showing 3mm Pyrex,

Volume based geometry can get away with coincident boundaries, Surface based geometry cannot

OptiX Geometry Experience

Geometry Issues Fixed/Avoidable

• triangulated geometry leaks when shoot millions of photons at cracks (1)
• bounding boxes that touch geometry cause leaks, slightly enlarging the boxes avoids leak (<1 in 3M)

1. more of a problem for testing than actual usage

Prism S-Polarized

Quadrant of cylindrically directed S-polarized photons at 10 wavelengths (from 100 to 800 nm) incident from left.

S-polarization: perpendicular to incident plane

Prism P-Polarized

Gap in reflection at Brewsters angle is apparent where the P-polarized photons can only be transmitted.

P-polarization: not perpendicular to incident plane

Prism Deviation vs Incident angles for 10 wavelengths

Prism geometry and Snell's law at two refractions allows deviation angle vs incident angle to be predicted. Comparison of simulation results with expectations for 10 wavelengths using refractive index of Schott F2 Flint Glass.

Lens focussing

Lens constructed from intersection of two spheres. Disc parallel beam incident from left. Color represents polarization direction. 2nd reflections are apparent.