Opticks + JUNO : MR180 Timestamp Analysis

Open source, https://bitbucket.org/simoncblyth/opticks

Reminder of junoPMTOpticalModel issues, complex PMT geometry and:
- BUGS: polarization and propagation time within PMT are wrong
- MORE BUGS: unphysical mid-vacuum reflect, refract, absorb, detect, "tunneling"
Reminder of Fix using Custom Boundary POM (PMT Optical Model), MR 180 STILL PENDING
SEvt timestamping
Standalone develop machinery
- stamp plot, no large gaps
- PHO_N : truncation bump, Truncation example
Insitu checks with input photons
- stamp plot : clear tail gaps in A+B, ProcessHits culprit
- profiling : junoSD_PMT_v2::ProcessHits with SProfile.h
- profiling : offender => get_pmtid
Faster get_pmtid, checks
- alternate get_pmtid implementation, SD touchable histories
Yaoguang Observations (Not reproduced by Simon on Workstation)
- 10 evt check on workstation, A vs B
- B has more Calls to ProcessHits : AnaMgr constrained by that ?
PHO_AVG : Photon Time vs step points : Debug and Release
PHO_N : Photon step points

Simon C Blyth, IHEP, CAS — 25 May 2023

junoPMTOpticalModel Issues : All from using FastSim

simple 2 volume (Pyrex+Vacuum) geometry
standard G4OpBoundaryProcess reflect/refract/absorb/TIR
much less code : reuse of standard Geant4
easier Opticks on GPU : simpler, more standard
- no near-degenerate 1e-3 mm boundaries
- allows same geometry on CPU and GPU
- avoid history rewriting for comparisons

BUT : need to change very nasty Geant4 impl:

G4OpBoundaryProcess::PostStepDoIt: ugly monolith, rats nest of nested ifs

Unpleasant to get familiar with
Change palatable, using : C4CustomART.h
Keep change separated, detailed validation vital

Four volume PMT due to FastSim limitation
- Two volumes (Pyrex + Vacuum) would be natural

Fake volumes yield many fake intersects
- Pyrex/Pyrex + Vacuum/Vacuum same material fakes
- complicates Geant4 step point history
- complicates comparison with Opticks

Non-FastSim in PMT propagate at Pyrex(not Vac.) speed
- FastSim->SlowSim transition misses speed fixup

reflected + refracted polarization incorrect
- should follow G4OpBoundaryProcess, depending on S/P

ModelTrigger whereAmI BUG => Unphysical photons
- mid-vacuum A,R,T,D inside PMT
- "tunneling" through dynode/MCP

hamaLogicalPMT_fake.png

Biggest problem with FastSim based POM : forces using unnatural geometry

body hidden under inner1 inner2

body is FastSim envelope volume

CURRENT : Unnatural 4-volume PMT (Pyrex+Pyrex+Vacuum+Vacuum) => Half Fake

hamaLogicalPMT_natural.png

Biggest advantage with Custom Boundary Process POM : enables natural geometry

Natural 2-volume PMT (Pyrex + Vacuum) : NOT COMPATIBLE WITH FastSim

FIX : Pivot to Custom Boundary Process For PMT Optical Model ?

  +---------------pmt-Pyrex----------------+
  |                                        |
  |                                        |
  |                                        |
  |       +~inner~Vacuum~~~~~~~~~~~+       |
  |       !                        !       |
  |       !                        !       |
  |       !                        !       |
  |       !                        !       |
  |       !                        !       |
  |       +                        +       |
  |       |                        |       |
  |       |                        |       |
  |       |                        |       |
  |       |                        |       |
  |       |                        |       |
  |       +------------------------+       |
  |                                        |
  |                                        |
  |                                        |
  +----------------------------------------+

OpSurfaceName[0] == '@'

local_z>0 : does MultiLayer ART calc
!(local_z>0) : standard mirror_opsurf

Custom Boundary Process : Advantages

natural geometry, no fakes
standard Geant4 polarization, propagation, time
less code, simpler code
simpler Geant4 step history (no fakes)
same geometry on GPU+CPU, easier Opticks validation
half the geometry objects to model PMT (4->2)

Current FastSim POM	4 Solid, 4 LV, 4 PV
Custom Boundary POM	2 Solid, 2 LV, 2 PV

Disadvantages

maintain Custom4 C4OpBoundaryProcess
updating Geant4 may require care (depending on changes)

Advantages far outweigh disadvantages

Issue 88, Branch, Merge Request : STILL PENDING

junosw/-/issues/88

Geant4 FastSim based junoPMTOpticalModel implementation has incorrect polarization and propagation time within PMT and also overcomplex 4 volume PMT geometry

junosw/-/tree/blyth-88-pivot-PMT-optical-model-from-FastSim-to-CustomG4OpBoundaryProcess

Pivot Branch

junosw/-/merge_requests/180

Merge Request

SEvt Timestamps : `opticks/sysrap/SEvt.hh`


 In [1]: a.rr
 Out[1]: array([1684168432404887, 1684168434567412],
         dtype=uint64)

 In [2]: np.c_[a.rr.view("datetime64[us]")]
 Out[2]:        ## beginOfRun, endOfRun
 array([['2023-05-15T16:33:52.404887'],
        ['2023-05-15T16:33:54.567412']],
        dtype='datetime64[us]')

 In [5]: np.c_[a.s0.view("datetime64[us]")]
 Out[5]:         ## beginPhoton stamps
 array([['2023-05-15T16:33:53.860482'],
        ['2023-05-15T16:33:53.860451'],
        ['2023-05-15T16:33:53.860421'],
        ...,
        ['2023-05-15T16:33:52.470429'],
        ['2023-05-15T16:33:52.470155'],
        ['2023-05-15T16:33:52.469285']],
        dtype='datetime64[us]')

 In [8]: a.t.shape      ## pointPhoton stamps
 Out[8]: (10000, 32)

sysrap/stamp.h stamp::Now()

Collect uint64_t timestamps (microseconds since epoch (UTC))

beginOfRun
- beginOfEvent
  - beginPhoton
    - pointPhoton : photon step points
    - pointPhoton
    - pointPhoton
    - ...
  - endPhoton
  - ...
- endOfEvent
- ...
endOfRun

STAMP_illus.png

Standalone FewPMT(1) test : No large stamp gaps ([us] microseconds)

TimeStamps : beginPhoton, pointPhoton, ..., endPhoton, OnePMT Standalone : UNIFORM : NO BIG GAPS (NO ProcessHits)

PHO_N_consistent_point_counts.png

A,B Point counts consistent : Bump at 32 from recording truncation

FewPMT_truncated_record_32.png

Point 31 : at propagation time 4.3 ns (photon ends at: 10.9 ns, at [-237.9,0.,-200] -- 6.6ns of propagation not recorded

STAMP_probe_gap.png

SD (+ some SA) photons : last step stamp "gaps" : slow ProcessHits?

Insitu (ntds2_cf), input photons targetting NNVT:0:1000

STAMP_ProcessHits_56us_Hama_0_0.png

STAMP_TT=257400,600 STAMP_ANNO=1 ~/j/ntds/stamp.sh ## inpho => Hama:0:0

(s0) beginPhoton (s1) finalPhoton ( ) pointPhoton (h0) ProcessHits[ (i0) ProcessHits] -- ProcessHits taking ~56us for SD

opticks/sysrap/SProfile.h : Simple Profiling via TimeStamp[us] Collection

#include "SProfile.h"
template<>
std::vector<SProfile<16>> SProfile<16>::RECORD = {}  ;
...
G4bool junoSD_PMT_v2::ProcessHits(G4Step * step,G4TouchableHistory*)
{
 m_profile->stamp(0);
 ...
 m_profile->stamp(1);

 const G4VTouchable* touch = track->GetTouchable();
 int pmtID_1 = touch->GetReplicaNumber(1) ;
 if(pmtID_1 <= 0) pmtID_1 = touch->GetReplicaNumber(2) ;

 m_profile->stamp(2);

 int pmtID_2 = U4Touchable::AncestorReplicaNumber(touch) ;

 m_profile->stamp(3);

 int pmtID = get_pmtid(track); // TAKING 50-56 us  

 m_profile->stamp(4);

Collect TimeStamps to find slow code
NumPy analysis reveals whats taking the time

 #include <cstdint>
 #include <vector>
 #include <chrono>
 #include "NPX.h"

 template<int N>
 struct SProfile
 {
   uint64_t idx ;
   uint64_t t[N] ;

   static std::vector<SProfile<N>> RECORD ;
   void add(){ RECORD.push_back(*this) ; }
   void zero(){ idx = 0 ; for(int i=0 ; i < N ; i++) t[i] = 0 ; }

   static uint64_t Now() // microseconds[us] since UTC epoch
   {
     std::chrono::time_point<std::chrono::system_clock>
         t0 = std::chrono::system_clock::now();
     return
         std::chrono::duration_cast<std::chrono::microseconds>
            (t0.time_since_epoch()).count() ;
   }

   void stamp(int i){ t[i] = Now(); } // stamp collection 

   static constexpr const char* NAME = "SProfile.npy" ;
   static NP* Array(){ return
      NPX::ArrayFromVec<uint64_t,SProfile<N>>(RECORD,1+N); }
   static void Save(const char* dir, const char* reldir=nullptr){
      NP* a = Array(); a->save(dir, reldir, NAME) ; }
 };

Timestamp collection + persisting implemented in the header

Analysis of uint64_t timestamps from ProcessHits, A(N=0) B(N=1)

Select complete SProfile (ProcessHit hit candidates) in A,B:

 // sevt.py : pf = f.junoSD_PMT_v2_SProfile

 In [30]: aa = a.pf[a.pf[:,6]>0,1:]
 In [38]: bb = b.pf[b.pf[:,6]>0,1:]

Subtract first timestamp for each ProcessHits call. Relative to start microseconds [us]:

 In [35]: aa[:,:6] - aa[:,0,np.newaxis]
 Out[35]:
 array([[   0,  551,  551,  571,  678, 1899],
        [   0,    1,    1,    1,   53,   96],
        [   0,    1,    1,    1,   54,   61],
        [   0,    0,    0,    1,   52,   57],
        [   0,    1,    1,    1,   57,   63],
        ...

 In [39]: bb[:,:6] - bb[:,0,np.newaxis]
 Out[39]:
 array([[   0,    0,    0,    1,   53, 1192],
        [   0,    0,    0,    1,   54,   99],
        [   0,    1,    1,    1,   54,   64],
        [   0,    1,    1,    1,   53,   59],
        [   0,    1,    1,    1,   53,   61],
        ...

        ##  0     1     2     3--->4    5

Most (50->56 us) ProcessHits time between stamps 3 and 4 in junoSD_PMT_v2::get_pmtid

Tail "gap" : junoSD_PMT_v2::get_pmtid : ~50us per hit candidate

Only few SD volumes => use simpler approach :

const G4VTouchable* th =
    track->GetTouchable();
int ID(-1) ;
ID = th->GetReplicaNumber(1);
if( ID <= 0)
ID = th->GetReplicaNumber(2);

G4VTouchable::GetReplicaNumber(int depth): depth:0/1/2 : current/parent/grandparent

tested : old_lpmt, new_lpmt, spmt, wpmt
same pmtId result in ~1us (not ~50us)

int junoSD_PMT_v2::get_pmtid(G4Track* track) {
    int ipmt= -1;
    const G4VTouchable* touch= track->GetTouchable();
    int nd= touch->GetHistoryDepth();
    int id=0;
    for (id=0; idGetVolume(id)==track->GetVolume()) {
        int idid=1;
        G4VPhysicalVolume* tmp_pv=NULL;
        for (idid=1; idid < (nd-id); ++idid) {
          tmp_pv = touch->GetVolume(id+idid);

          G4LogicalVolume* mother_vol = tmp_pv->GetLogicalVolume();
          G4LogicalVolume* daughter_vol = touch->GetVolume(id+idid-1)->
              GetLogicalVolume();
          int no_daugh = mother_vol -> GetNoDaughters();
          if (no_daugh > 1) {
              int count = 0;
              for (int i=0; (count<2) &&(i < no_daugh); ++i) {
                  if (daughter_vol->GetName()
                          ==mother_vol->GetDaughter(i)
                             ->GetLogicalVolume()->GetName()) {
                      ++count;
                  }
              }
              if (count > 1) break;
          }
        }
        ipmt= touch->GetReplicaNumber(id+idid-1);
        break;
      }
    }
    return ipmt;
}

Looping over daughters at multiple levels and name matching to find copyNo : NO NEED FOR SUCH GENERALITY

SD Touchable histories for SPMT, LPMT_NEW

 spmt (old/new makes no difference)::

     junoSD_PMT_v2::ProcessHits pmtID 325044 pmtID_1 325044 pmtID_2 325044 pv PMT_3inch_body_phys
     junoSD_PMT_v2::ProcessHits
     C4Touchable::Desc HistoryDepth  7 TouchDepth  0 ReplicaDepth  1 ReplicaNumber 325044
      i  0 cp      0 nd      2 so PMT_3inch_body_solid_ell_ell_helper pv  PMT_3inch_body_phys
      i  1 cp 325044 nd      2 so  PMT_3inch_pmt_solid pv   PMT_3inch_log_phys
      i  2 cp      0 nd  46276 so          sInnerWater pv          pInnerWater
      i  3 cp      0 nd      1 so       sReflectorInCD pv     pCentralDetector
      i  4 cp      0 nd   4521 so      sOuterWaterPool pv      pOuterWaterPool
      i  5 cp      0 nd      1 so          sPoolLining pv          pPoolLining
      i  6 cp      0 nd      1 so          sBottomRock pv             pBtmRock

 LPMT_NEW (same depth to get copynumber as SPMT)::

     junoSD_PMT_v2::ProcessHits pmtID 1425 pmtID_1 1425 pmtID_2 1425 pv NNVTMCPPMT_PMT_20inch_log_phys
     junoSD_PMT_v2::ProcessHits
     C4Touchable::Desc HistoryDepth  7 TouchDepth  0 ReplicaDepth  1 ReplicaNumber   1425
      i  0 cp      0 nd      1 so NNVTMCPPMT_PMT_20inch_pmt_solid_head pv NNVTMCPPMT_PMT_20inch_log_phys
      i  1 cp   1425 nd      3 so NNVTMCPPMTsMask_virtual pv    pLPMT_NNVT_MCPPMT
      i  2 cp      0 nd  46276 so          sInnerWater pv          pInnerWater
      i  3 cp      0 nd      1 so       sReflectorInCD pv     pCentralDetector
      i  4 cp      0 nd   4521 so      sOuterWaterPool pv      pOuterWaterPool
      i  5 cp      0 nd      1 so          sPoolLining pv          pPoolLining
      i  6 cp      0 nd      1 so          sBottomRock pv             pBtmRock

Also checked with input photons targetting Hama:0:0 ## (pmtid == 0)

SD Touchable histories for LPMT_OLD, WPMT

 LPMT_OLD (one more level to get copynumber)::

     junoSD_PMT_v2::ProcessHits pmtID 1425 pmtID_1 1425 pmtID_2 1425 pv NNVTMCPPMT_PMT_20inch_body_phys
     junoSD_PMT_v2::ProcessHits
     C4Touchable::Desc HistoryDepth  8 TouchDepth  0 ReplicaDepth  2 ReplicaNumber   1425
      i  0 cp      0 nd      2 so NNVTMCPPMT_PMT_20inch_body_solid_head pv NNVTMCPPMT_PMT_20inch_body_phys
      i  1 cp      0 nd      1 so NNVTMCPPMT_PMT_20inch_pmt_solid_head pv NNVTMCPPMT_PMT_20inch_log_phys
      i  2 cp   1425 nd      3 so NNVTMCPPMTsMask_virtual pv    pLPMT_NNVT_MCPPMT
      i  3 cp      0 nd  46276 so          sInnerWater pv          pInnerWater
      i  4 cp      0 nd      1 so       sReflectorInCD pv     pCentralDetector

 WPMT (same depth to get copynumber as LPMT_OLD)

    junoSD_PMT_v2::ProcessHits pmtID 31000 pmtID_1 31000 pmtID_2 31000 pv PMT_20inch_veto_body_phys
     junoSD_PMT_v2::ProcessHits
     C4Touchable::Desc HistoryDepth  6 TouchDepth  0 ReplicaDepth  2 ReplicaNumber  31000
      i  0 cp      0 nd      2 so PMT_20inch_veto_body_solid_1_2 pv PMT_20inch_veto_body_phys
      i  1 cp      0 nd      1 so PMT_20inch_veto_pmt_solid_1_2 pv PMT_20inch_veto_log_phys
      i  2 cp  31000 nd      2 so mask_PMT_20inch_vetosMask_virtual pv mask_PMT_20inch_vetolMaskVirtual_phys
      i  3 cp      0 nd   4521 so      sOuterWaterPool pv      pOuterWaterPool
      i  4 cp      0 nd      1 so          sPoolLining pv          pPoolLining
      i  5 cp      0 nd      1 so          sBottomRock pv             pBtmRock

B_V1J008_N1_OIPF_PMT_20inch_veto:0:1000_target_WPMT.png

OPTICKS_INPUT_PHOTON_FRAME=PMT_20inch_veto:0:1000 ntds2

N=1 MODE=3 ~/j/ntds/ntds.sh ana ## checking "get_pmtid_fast" for WPMT

Yaoguang Observations (Not reproduced by Simon on Workstation)

totalPE distributions are almost the same as expected.
But .. the natural geometry is more time-consuming than the unnatural one.
- (this is batch running, presumably with much more memory+IO pressure)

Yaoguang : Event Time from Mean of 100 runs of 500 evt

`--pmt-unnatural-geometry`	1.40s
`--pmt-natural-geometry`	2.04s	(~45% longer with bug fixes ?)

/junofs/users/wangyg/Software/juno-dev/Official/sim_dat

python ${RunPath}/tut_detsim.py
     --evtmax 500 --seed $1  ... (output opts skipped) ...
     --pmt-natural-geometry [ OR --pmt-unnatural-geometry ]
     gun --particles gamma --momentums 2.223
     --momentums-interp KineticEnergy --positions 0 0 0

Simon : Comparing an implementation with many known bugs with one that fixes them

thats an apples-vs-oranges "invalid" comparison
also the implementation from the branch is not the same at the one in main
BUT good to understand whats taking time : hoping for optimizations

10 evt check on workstation (no memory pressure?)

Debug build, with Opticks U4Recorder, only opticks-anamgr

GUN=2 ntds2_cf:            Count       Total(ms)
A : DetSimAlg                10          55232.92090
B : DetSimAlg                10          48998.21484

Debug build, no Opticks, default anamgr

GUN=2 ntds0_cf:           Count       Total(ms)
A : DetSimAlg              10          32182.09277
B : DetSimAlg              10          20433.68774

Release build, no Opticks, default anamgr

GUN=2 ntds0_cf:             Count       Total(ms)
A : DetSimAlg                10          13037.99695
B : DetSimAlg                10          12551.25012

Release build, no Opticks, default anamgr, switch to "get_pmtid_fast" (B: ~6% faster)

GUN=2 ntds0_cf:             Count       Total(ms)
A : DetSimAlg                10          12678.48804
B : DetSimAlg                10          11844.33704

Release build, no Opticks, default anamgr, switch to "get_pmtid_fast" (B: ~6% faster), disable all anamgr

GUN=2 ntds0_cf:             Count       Total(ms)
A : DetSimAlg                10          11973.54700
B : DetSimAlg                10          11426.45898   Workstation : : A,B get closer as memory decreases ?

A vs B : Attempt to Explain Observed Workstation Performance

        A          B
array([[55232.921, 48998.215],
       [32182.093, 20433.688],
       [13037.997, 12551.25 ]])

 a[:,1]/a[:,0]  # B/A
 array([0.887, 0.635, 0.963])

 a[:,0]/a[:,1]  # A/B
 array([1.127, 1.575, 1.039])

Debug build, U4Recorder, only opticks-anamgr
Debug build, no Opticks, default anamgr
Release build, no Opticks, default anamgr

Both slow

recording every photon point is expensive (also fake skipping is expensive)

A,B most diff

default multiple anamgr => memory/IO pressure ?

highest memory pressure

A,B closest

Release eases memory ?

lowest memory pressure

Release running => greatly reduced memory pressure
B has ~30% elevated ProcessHits:false calls from larger sensitive volume
- if AnaMgr sensitive to that could explain decreased B performance in batch jobs

B has ~25% more ProcessHits calls : Is AnaMgr, IO, .. impacted ?

NEVT=5 ./ntds.sh cfmeta

msab.c_itab

array([
[[26082, 25126, 25586, 26383, 26177], # A 0:ProcessHits_count
 [31874, 31477, 31544, 31319, 32353]],# B

[[ 4022,  3967,  3934,  4193,  4137],     1:ProcessHits_true
 [ 3972,  4011,  3941,  4026,  3947]],

[[22060, 21159, 21652, 22190, 22040],     2:ProcessHits_false
 [27902, 27466, 27603, 27293, 28406]],

[[ 4022,  3967,  3934,  4193,  4137],     3:SaveNormHit_count
 [ 3972,  4011,  3941,  4026,  3947]],
...

[[25249, 24311, 24723, 25561, 25368],     8:NEDEP
 [30964, 30552, 30654, 30408, 31459]],

[[    1,     1,     4,     0,     2],     9:NBOUND
 [   99,    92,    94,    93,    81]],

fake volume 1e-3 mm larger than vacuum is Sensitive

full 5mm larger than vacuum is Sensitive

larger sensitive volume than A
=> 30% more ProcessHits:false calls
potential to reduce perf when memory constrained
- depends on the AnaMgr

Ideas for further investigations

When using Opticks U4Recorder, only --opticks-anamgr is enabled:

anamgr(){ cat << EOU
--opticks-anamgr
--no-anamgr-normal
--no-anamgr-genevt
--no-anamgr-edm-v2
--no-anamgr-grdm
--no-anamgr-deposit
--no-anamgr-deposit-tt
--no-anamgr-interesting-process
--no-anamgr-optical-parameter
--no-anamgr-timer
EOU
}

Suggestions to understand futher:

compare with IO disabled
compare with all anamgr disabled
enable anamgr one-by-one
enable IO
memory measurements

which has most impact on A/B performance ?

However : Pointless to spend much effort as A is WRONG:

incorrect propagation time within PMT
incorrect reflect polarization
incorrect refract polarization
tunneling through geometry (*)
unphysical photon : mid-vac D,A,R,T (*)
overcomplex PMT geometry
can still use: --pmt-unnatural-geometry

Opticks + JUNO : MR180 Timestamp Analysis

Opticks + JUNO : MR180 Timestamp Analysis

Opticks + JUNO : MR180 Timestamp Analysis

Open source, https://bitbucket.org/simoncblyth/opticks

Simon C Blyth, IHEP, CAS — 25 May 2023

junoPMTOpticalModel Issues : All from using FastSim

hamaLogicalPMT_fake.png

hamaLogicalPMT_natural.png

FIX : Pivot to Custom Boundary Process For PMT Optical Model ?

Issue 88, Branch, Merge Request : STILL PENDING

SEvt Timestamps : `opticks/sysrap/SEvt.hh`

STAMP_illus.png

PHO_N_consistent_point_counts.png

FewPMT_truncated_record_32.png

STAMP_probe_gap.png

STAMP_ProcessHits_56us_Hama_0_0.png

opticks/sysrap/SProfile.h : Simple Profiling via TimeStamp[us] Collection

Analysis of uint64_t timestamps from ProcessHits, A(N=0) B(N=1)

Tail "gap" : junoSD_PMT_v2::get_pmtid : ~50us per hit candidate

SD Touchable histories for SPMT, LPMT_NEW

SD Touchable histories for LPMT_OLD, WPMT

B_V1J008_N1_OIPF_PMT_20inch_veto:0:1000_target_WPMT.png

Yaoguang Observations (Not reproduced by Simon on Workstation)

10 evt check on workstation (no memory pressure?)

A vs B : Attempt to Explain Observed Workstation Performance

B has ~25% more ProcessHits calls : Is AnaMgr, IO, .. impacted ?

Ideas for further investigations

PHO_AVG_GPFX_V_for_Debug.png

PHO_AVG_GPFX_R_for_Release.png

PHO_N_Release_Count_10.png

A	N=0	--pmt-unnatural-geometry
B	N=1	--pmt-natural-geometry

Opticks + JUNO : MR180 Timestamp Analysis

Opticks + JUNO : MR180 Timestamp Analysis

Simon C Blyth, IHEP, CAS — 25 May 2023

junoPMTOpticalModel Issues : All from using FastSim

hamaLogicalPMT_fake.png

hamaLogicalPMT_natural.png

FIX : Pivot to Custom Boundary Process For PMT Optical Model ?

Issue 88, Branch, Merge Request : STILL PENDING

SEvt Timestamps : opticks/sysrap/SEvt.hh

STAMP_illus.png

PHO_N_consistent_point_counts.png

FewPMT_truncated_record_32.png

STAMP_probe_gap.png

STAMP_ProcessHits_56us_Hama_0_0.png

opticks/sysrap/SProfile.h : Simple Profiling via TimeStamp[us] Collection

Analysis of uint64_t timestamps from ProcessHits, A(N=0) B(N=1)

Tail "gap" : junoSD_PMT_v2::get_pmtid : ~50us per hit candidate

SD Touchable histories for SPMT, LPMT_NEW

SD Touchable histories for LPMT_OLD, WPMT

B_V1J008_N1_OIPF_PMT_20inch_veto:0:1000_target_WPMT.png

Yaoguang Observations (Not reproduced by Simon on Workstation)

10 evt check on workstation (no memory pressure?)

A vs B : Attempt to Explain Observed Workstation Performance

B has ~25% more ProcessHits calls : Is AnaMgr, IO, .. impacted ?

Ideas for further investigations

PHO_AVG_GPFX_V_for_Debug.png

PHO_AVG_GPFX_R_for_Release.png

PHO_N_Release_Count_10.png

SEvt Timestamps : `opticks/sysrap/SEvt.hh`