Once again, we’ll discuss and resolve the various hold-ups with the Run 3 Pipeline Review.
We have three epics relating to this, all with open issues:
Epic: Run 3 Science Inputs - these issues need to be resolved so that the instance catalog code can be run.
Bryce reported that the SL matching is now complete, with an updated lens catalog checked in.
Epic: Instance Catalogs at SLAC - these issues need to be resolved before the Run 3 phoSim pipeline (including the instance catalog generation) can be run.
What order should we run the ObsHistIDs? Rahul: suggests the following sequence: Run 3.1) Repeating the approach we took to Run1, selecting a subset of ~1000 visits that span all years and all filters. Then Run 3.2) do all the WFD visits (that have not yet been done, some 1500 or so). Then, Run 3.3) fill in the gaps to complete Year 3 (~2000 visits). Then, Run 3.4+) do all the rest in chronological order, Year 1, then Year 2, Year 4 etc (around 20,000 total, in several batches).
SL needs WFD visits, primarily - this is where Run 3.2 came from above. Rahul points out this means we need to use proposal option 54 not 56.
Down-selection script is called “VisitDownSelector” - Rahul will write this simple script, with all of the above functionality, so that Tom can divide up the visits into Runs.
Seth: what about high Moon? Runtime is up to 30-40 times longer. Jim: put them in the XXL queue and wait? Turns out they hit a hard batch farm limit. Suggestion: downselector should read the outputs from PhoSimPredictor, and avoid high Moon nights for now. (We don’t want to discard high moon visits completely, as we still hope to one day complete these runs, perhaps when checkpointing is operational). Rahul will work with Seth to make this happen, starting with issuing the downselector work.
Scott will finish off the Sims Recipe ready for the review (see PR #328)
Epic: Run 3 phoSim Workflow - these issues need to be resolved before the Run 3 phoSIm pipeline can be run from the workflow engine.
Since we’ll be running PhoSim 3.5 in Run 3, Rahul needs a test eimage to compare with his instance catalog - Jim, Seth and Tom will make one as per his request in #325
The instance catalogs seem to be taking a long time to generate - Scott says this because FatBoy is having to work hard to get the galaxies out. He and Rahul think they can speed it up by dumping out the galaxies (which don’t change) and checking them into the Twinkles repo, so that the instance catalogs can work from that, and reduce the load on FatBoy. Scott will work on this #346
This is happening in Epic: Run 3 Pipeline Review. It looks as though we really shoudl be able to hold a pipeline review next Thursday, when Tom is back from the CHEP meeting and has had time to pull things together for us to look at.
Materials for review need to be posted in the checklist in the top comment of issue 332 - assignments are there, now we just need to fill in our sections. Phil will re-doodle for a good time to meet on Thursday.