Monitoring JUNOSW + Opticks : GPU ray trace accelerated optical photon simulation

Outline

• Optical Photon Simulation Context
  • (JUNO) Optical Photon Simulation Problem...
  • Four NVIDIA RTX Gen : Ray Tracing : ~2x every ~2 yrs
  • NVIDIA OptiX : Ray Tracing Engine -- Accessible GPU Ray Tracing
  • Geant4 + Opticks + NVIDIA OptiX : Hybrid Workflow
• Opticks + JUNOSW (OJ) : Testing as Geometry and Models continue to Evolve
  • Opticks+JUNOSW Monitoring
  • gitlab-ci in OJ repository
  • Opticks Grafana Dashboard
  • Goal of Monitoring
• Preparations for Three Species LS model
  • MC Method : Sampling using Cumulative Distribution Function (CDF)
  • Monte Carlo Method : Modelling Scintillator Re-emission
  • Scint wavelength generation in triplicate (LAB, bisMSB, PPO)
  • One billion wavelengths Chi2 check
• Geant4 + Opticks + NVIDIA OptiX : Production Scaling ?
  • Geant4 + Opticks + NVIDIA OptiX : Monolith x4, x16
  • OpticksClients + OpticksService : Share GPUs
  • Opticks "Optical Core" => Server, "Periphery" => Client
  • NP_CURL.h : Array transport via HTTP POST
  • Opticks Server Prototype : python + FastAPI + nanobind + CSGOptiX
  • NVIDIA Triton Inference Server (aka Dynamo-Triton)
• Summary + Links

(JUNO) Optical Photon Simulation Problem...

Opticks solves this using GPU ray tracing via NVIDIA OptiX

Four NVIDIA RTX Gen : Ray Tracing : ~2x every ~2 yrs

• RT Cores: fixed-function ASICs dedicated to ray tracing : bbox intersection, ray-triangle intersection, BVH traversal, ...

[1] baseline : 2018 "World's First Ray-Tracing GPU -- 10 Gigarays/sec"

NVIDIA® OptiX™ Ray Tracing Engine -- Accessible GPU Ray Tracing

Flexible Ray Tracing Pipeline

Green: User Programs, Grey: Fixed function/HW

Analogous to OpenGL rasterization pipeline

OptiX makes GPU ray tracing accessible

• Programmable GPU-accelerated Ray-Tracing Pipeline
• Single-ray shader programming model using CUDA
• ray tracing acceleration using RT Cores (RTX GPUs)
• "...free to use within any application..."

OptiX features

• stateless, multi-threaded and asynchronous
• built-in triangles, user-defined custom primitives
• acceleration structure creation + traversal (eg BVH)
• instanced geometry + acceleration structures
• compiler optimized for GPU ray tracing

User provides (Green):

• ray generation, geometry bounding boxes
• custom primitive intersect functions
• instance transforms

Latest Release : NVIDIA® OptiX™ 9.1.0 (Dec 2025)

• NVIDIA R590+ driver
• OptiX 6 support removed from drivers R590+

Geant4 + Opticks + NVIDIA OptiX : Hybrid Workflow

Opticks enables Geant4 based simulation to offload optical photon simulation to the GPU

NVIDIA GPU ray tracing of billions[1] of rays per second applied to optical simulation

[1] Actual performance depends on geometry and its modelling, JUNO optical simulation speedups > 1000x Geant4 have been measured

Opticks+JUNOSW (OJ) Monitoring

Opticks : fast replacement for Geant4 optical sim, BUT FRAGILE:

• model/geometry change often needs CUDA development
  • eg: new three species LS model with state transitions
• any change can break correctness + performance
  • => REQUIRES CONTINUOUS MONITORING

Accelerate dev-cycle via automated : Validation + Performance Mon.

OJ + Opticks Monitoring : https://code.ihep.ac.cn/blyth/oj/

• gitlab-ci nightly tests - (gitlab-ci coordination, Grafana presentation)
• GPU Workstation: gitlab-runner + Grafana server
• Opticks: summarize test performance/status into SQLite3 DB tables
• visualize using Grafana Dashboard
  • interactive plotting via SQL queries => dynamic plots

"Pipeline" works, NEXT: extend coverage to all important tests

opticks_monitoring_gitlab_ci_20260716_141838.png

opticks_monitoring_grafana_20260715_160444.png

Opticks Grafana Dashboard

• 1. interactive time window
• 2. direct links from gitlab-ci

opticks_monitoring_grafana_20260715_155458.png

Web Interface -> Edit Grafana viz of "cxs_min.sh" - GPU time vs photon

• Viz based on SQL queries from SQLite3 DB tables
• Grafana very popular => AI very helpful

Opticks+JUNOSW (OJ) Monitoring : Goal

Nightly : DETAILED Validation + Performance Monitoring

Requires, for all important tests:

1. expand DB schema to hold summary results
   • tables : ~/opticks/sysrap/sreportdb.sql
2. add reporting stage that writes summary results into DB tables
   • python OR C++ using ~/np/NSQLite.h
   • insert into tables : ~/opticks/sysrap/sreportdb.h
   • control script : ~/opticks/sysrap/tests/sreportdb.sh
3. add Grafana visualization of results + alerts for deviations
   • use web interface to develop SQL query + configure viz

• currently cxs_min.sh added to develop the "pipeline"
• straightforward to incorporate more tests
• trying json "metadata" field - to avoid too many schema changes
• sqlite3 : json_extract(metadata, '$.photon_counts.total')

Preparations for Three Species LS model

Preparations for Three Species LS model

• MC Method : Sampling using Cumulative Distribution Function (CDF)
• Monte Carlo Method : Modelling Scintillator Re-emission
• Scint wavelength generation in triplicate (LAB, bisMSB, PPO)
• One billion wavelengths Chi2 check

MC Method : Sampling using Cumulative Distribution Function (CDF)

Aim : create sample (a set of values) that follows an analytic PDF

Cumulative distribution function (CDF) : Prob( X < x)

• definite integral PDF f -> CDF F
  • F(b) - F(a) = P( a < X <= b ) = Integral a->b f(x) dx
• monotonically increasing function, CDF(-inf) = 0, CDF(inf) = 1

Map uniform randoms onto probabilities

1. integrate PDF -> CDF (mapping PDF domain onto probability [0,1] )
2. invert the CDF, so domain becomes [0,1]
3. inverted_CDF(uniform random) -> sample value

Intuitively : throw randoms uniformly "vertically"

• Sigmoid CDF : low probability close to zero or 1 corresponding to tails
• greater probability in high PDF areas
• extreme case of delta-function PDF : all probability at one value
  • sigmoid CDF becomes Heaviside step-function

CDF "encodes" shape of PDF in convenient form for sampling

https://en.wikipedia.org/wiki/Inverse_transform_sampling

Monte Carlo Method : Modelling Scintillator Re-emission

reemission_probability(wavelength)

fraction of light absorbed in scintillator re-emitted with different wavelength.

generating re-emitted wavelength

• expected PDF(wavelength) -> inverted_CDF(probability)

Opticks Re-emission model

• fraction of absorbed photons "reincarnated" in same thread
• inverted_CDF(probability) -> GPU "re-emission" texture

float u_reemit = curand_uniform(&rng);
if (u_reemit < reemission_probability )
{
    ...
    p.wavelength = reemission_lookup(curand_uniform(&rng));    # re-emission texture lookup  
    s.flag = BULK_REEMIT ;
    return CONTINUE;
}
else
{
    s.flag = BULK_ABSORB ;
    return BREAK;
}

https://bitbucket.org/simoncblyth/opticks/src/default/optixrap/cu/propagate.h

Scint wavelength generation in triplicate (LAB, bisMSB, PPO)

 0        e-m   e   e+m  ["HD20": e=.05,m=e/10]   -e-m 1-e 1-e+m        1
 |         |    |    |                             |    |    |          |
 +---LHS-------------+                             +----------RHS-------+
 |         +----:----------------MID--------------------:----+          |
 |              :    |        1 - 2*(e-m)          |    :               |
 |              :    |                             |    :               |
 |        u < e :    |     otherwise MID zone      |    : u > 1 - e     |
 |  x=u/(e+m)   :    | x = (u-(e-m))/(1-2*(e-m))   | x=(u-(1-e-m))/(e+m)|
 

• [u: uniform random 0->1] At edge (u < 0.05, u > 0.95) : CDF flat => ICDF steep : need ultra high resolution
• "HD20" 4096 bin, 20x at edge (but only 3x larger) => enables GPU wavelength lookup(float) to match Geant4 calc(double)
• margins(m) prevent Chi2 spikes from float im-precision at zone boundaries

float qscint_three::wavelength_hd20(int species_idx, float u ) const

QScintThree_test_sh_COMP_012_LOGY1_EDGE1_DPI300

1e9 wavelengths - (logscale) G4:Opticks diff not visible - Chi2/ndf < 1.001

QScintThree_test_sh_COMP_012_LOGY0_EDGE1_DPI300

1e9 wavelengths - (linear) G4:Opticks diff not visible - Chi2/ndf < 1.001

QScintThree_test_sh_COMP_012_LOGY1_EDGE1_DPI300_BOFFSET1e7

1e9 wavelengths - (lin+offset) G4:Opticks diff not visible - Chi2/ndf < 1.001

Scaling Opticks

Geant4 + Opticks + NVIDIA OptiX : Production Scaling ?

• hard to make good use of GPU resources, gets harder with each generation
  • under-utilized GPUs
• monolithic scaling Opticks ?
  • GPU idle while : Geant4 init + non-optical simulation : wasteful of GPU resources

=> Client-Server architecture

• separate GPU optical simulation into OpticksServer
• share server between CPU nodes

Geant4 + Opticks + NVIDIA OptiX : Hybrid Workflow 2x2 ?

Geant4 + Opticks + NVIDIA OptiX : Monolith x4 ?

Geant4 + Opticks + NVIDIA OptiX : Hybrid Workflow 4x4 ?

Geant4 + Opticks + NVIDIA OptiX : Monolith x16 ?

"Monolithic" scaling : very inefficient use of scarce GPU resources

OpticksClients + OpticksService : Share GPUs

Client.png

Opticks "Optical Core" => Server, "Periphery" => Client

Client build from common Opticks codebase with OPTICKS_CONFIG=Client

• OpticksClient use just like OpticksFull(monolithic) - simulate via network to GPU node

NP_CURL.h : Array transport via HTTP POST

Basis for Opticks Client - using libcurl 7.76.1+ (2021) - default in many Linux distro

NP.hh : C++ array with NumPy serialization (Opticks numerical base), NP_CURL.h headers:

HTTP 429, 503 : Too Many Requests, Service Unavailable (temporary downtime)

• follows Retry-After: <seconds> header suggestion

https://github.com/simoncblyth/np/blob/master/NP_CURL.h -- https://github.com/simoncblyth/np/blob/master/NP.hh

Opticks Server Prototype : python + FastAPI + nanobind + CSGOptiX

CSGOptiX/tests/CSGOptiXService_FastAPI_test/CSGOptiXService_FastAPI_test.sh

• FastAPI : ASGI python web framework
• nanobind : python <=> C++ (Using uv, pip)
• opticks/CSG : load persisted geometry
• opticks/CSGOptiX : simulate

Prototype client + service operational

• load testing with k6 https://k6.io
• started adding concurrent CUDA support to Opticks
  • WIP: extend async CUDA to all Opticks GPU usage

"Roll your own" Prototype Server : Educational, BUT:

• difficult to robustly serve many clients
  • => unsuitable for MC production campaign (?)
• PLAN : compare with widely used open source server : NVIDIA Triton Inference Server

NVIDIA Triton Inference Server (aka Dynamo-Triton)

Triton[1] : open-source, designed to accelerate AI deployment at scale

• despite "Inference", "AI" : is flexible async tensor server
• NVIDIA maintained open source server software
• dynamic batching : requests queued and combined in batch
• multi-GPU support, concurrency
• widely used : tech, financial, manufacturing companies

Wrap Opticks as Custom C++ Triton Backend "model(s)" ?

Make request load like inference : smaller, more uniform

Make requests during genstep collection :

• => tunable max_slots
• decouple compute from physics

High GPU utilization => concurrency :

• async CUDA Opticks
• async CUDA memory pools [2]

Robust Server (MC campaign):

• static flat VRAM
• backpressure, signal client retry

[1] https://developer.nvidia.com/dynamo-triton

[2] cudaMallocFromPoolAsync

Summary and Links

Opticks : state-of-the-art GPU ray traced optical simulation integrated with Geant4, with automated geometry translation.

GPU-less OpticksClient + OpticksService in development, bringing Opticks everywhere + improving GPU utilization.

• NVIDIA Ray Trace Performance continues rapid progress (2x each gen., every ~2 yrs)
• any simulation limited by optical photons can benefit from Opticks
• more photon limited -> more overall speedup (99% -> ~90x)