Looking forward to the collaboration meeting in early February, we talked through our roadblocks to producing Run 3 DIAObject light curves, the remaining PhoSim “Long Jobs” and related PhoSim/workflow engine development, and Bryce showed us how we can now run IPython notebooks at NERSC remotely, over the web.
Bryce got this working, and told us all about it. Notes on how to get set up are here: https://github.com/LSSTDESC/Monitor/blob/master/doc/setup_jupyter-dev.md We agreed they could use some high level introduction text, but they are fully tested and Bryce is planning to use this method to analyze the Run 3 light curves. We tried installing a python package from the notebook itself: `pip install --user <package>` worked fine (after kernel restart). We started thinking about how to share results generated by NERSC notebooks, and will investigate shared project area, and storing outputs as html while checking in clean notebooks to the Twinkles repo.
Simon pointed out #422 (config files for old-style coadd creation) and #423 (FITS table header interpretation), and we discussed them. This led to a recollection that we need to get `cmodel` galaxy measurements going for the SN error model: Simon will provide a cookbook update.
Tony reports:
All completed run 3 visits (21951 visits) have been processed through the eImage stage. Only 5 failures:
2x ValueError: cannot convert float NaN to integer (472762, 2441837)
2x RuntimeError: Only 1 sources for calculation of aperture correction for 'base_GaussianFlux'; require at least 2 (472793, 1212518)
1x transient failure perhaps, should be rerun (2215324)
There is nothing preventing coadds/image differencing being run, except perhaps lack of NERSC allocation.
We talked about getting results together in time for the February meeting: Tony and Simon will get the pipeline up and running and try submitting jobs over teh weekend, hopefully taking us into the NERSC allocation red.
Tom reported that the checkpointing mechanism (DMTCP) has been installed and tested, but does not yet have support from the workflow engine. A first support attempt failed but the issue is still on Brian’s agenda. Brian will work on this. We have 19 Run 3 jobs to retry (they timed out) and then ~1000 further Run 3 jobs that are predicted to exceed the 48 hour limit.
Tom also reported on a useful conversation with Glenn S, about how the multithreading is working: there could be some efficiey gains to be made by recoding. Tom will investigate wih others at SLAC (Tony suggested the GEANT4 team), and work with Glenn to improve the PhoSim code (we hope!).
Tony and Tom are meetig with Mustapha at NERSC to discuss these and other issues, and are hopeful of getting things working better at NERSC this year.
We decided to postpone serious Twinkles 2 planning to the collaboration meeting, but noted that some discussion with parties outside the DESC can begin now. Simon suggested that the DM team might be interested in Twinkles 2 as a testbed for Alert production. Scott offered to contact Ashish Mahabal and Federica Bianco in the Transients and Variables science collaboration about getting realistic populations of more transients into Twinkles 2, to provide some fresh inputs to the Broker and Classifier codes that are being developed.