Our Run 3 sprint (milestone #10) is due this week: we will make a plan for executing the pipeline review, after checking in with everyone working on the various pieces.
Epic here (#321) - Bryce, Phil and Rahul to report on the readiness of the sprinkling/SN code to be run at SLAC.
Do we now have reasonable and useful models for the lens galaxies and source AGN?
Source AGN issue #309 - we’ll use 0.1 dex tolerance in both source redshift and flux
Lens Galaxies issue #310 - Bryce and Phil will work on this this afternoon.
Do we now have reasonable and useful abundances and distributions of SNe?
Rahul and Phil will quickly converge via the Design Note, but we’re looking at avoiding multiple SNe per host over 10 years, and using the natural redshift distribution. EmpiriciSN is not yet production ready - we’ll adopt it when it is published, for Twinkles 2.
Last week Rahul reported that construction on UW campus means they are going to have to cut power to the astronomy building for about eight hours on Sunday, October 2nd. UW IT will actually not begin powering down serves (like fatboy) until Saturday, October 1, so fatboy should be up all day on Friday Sept 30. They anticipate everything will be back up and ready to use by 8am PDT on Monday, October 3.
The UW group reported last week that they have a version of the instance catalog generator which is sufficiently ready that it would be worth trying at SLAC. Rahul was going to create a setup script for generating instance catalogs, and also log in to SLAC to test things out - he has submitted a pull request which Jim will review.
Which version of the sims code should we install and use at SLAC? (#319) - Scott will publish a new version this afternoon, for Heather to eups install and build, and then Tom to use tomorrow.
Seth has been investigating PhoSim’s treatment of bright stars, in the DC1 PhoSim Deep run - there are image and runtime issues that we are in the process of understanding.
There are dark patches that seem to be due to crosstalk. Here’s the thread, #315. We decided to turn off crosstalk in PhoSim, which is not trivial (but still straightforward). Jim will issue this, and also report to John P about why we are doing this. We also noted that eimages should not really contain crosstalk, if they are supposed to be above-detector images - maybe worth a PhoSim ticket?
Bright stars lead to high CPUtime in adjacent chips, see here. A bright star can influence simulations for a large fraction of the focal plane; due to effects from ghosting and mirror scratches(!), phosim considers progressively larger (square) regions of influence as the stellar magnitude decreases. 7th magnitude stars such as we have ~2 per field, can each influence 25 CCDs, and the phosim simulations for the ‘outlier’ CCDs are particularly expensive. More of an issue for DC1 PhoSim Deep, but interesting nonetheless. Simon and Tom will continue the conversation with John P about DC1 PhoSim Deep. We noted that the Twinkles assumption is that there are no artifacts from bright stars on nearby chips.
Discussion here (#315), led by Tom.
Given the lack of checkpointing capability, we should go ahead with Run 3 and accept that some jobs will fail for lack of time (as they did in Run 1). The new task is structurally ready and waiting for a few more pieces (e.g., instanceCatalog generation, and validation of instance catalog that is produced)
Regarding validation of the PhoSim instance catalogs, we discussed a few options. Initially, a line by line comparison with an IC made at UW with the same code seems useful - Rahul will supply this. Beyond this, there are some basic checks to do (eg IC is not empty or too small), and then we are into comparing the IC with the PhoSim output image FITS headers, with an “after-burner” validation script. The temporary files made by PhoSim could also be useful, but this would mean calling PhoSim by means other than phosim.py, which, while not discouraged, is not the industry standard way of using the software.
Heather has looked into dmctp as an alternative checkpointing mechanism, and will report on what she has found out. See Heather’s confluence page for gory details. High level summary, testing continues at SLAC using the batch system and simple phosim runs. Checkpoint files generated take up about 1.4 GB gzip or 2.4 GB unzipped. Runtimes are somewhat impacted, very little if we avoid gzipping the checkpoint files while running (~4%) and a larger impact if gzipping occurs (~18%). Currently running some restarted jobs using checkpoints to make sure the output remains the same with and without dmtcp. That should/could be done by end of the week or early next week - and so could be a viable option during Twinkles 3 production.
Should we consider other options, like dropping moon? Rahul made the sensible suggestion that we should consider removing it for visits that we would not otherwise be able to run to completion. However, there are limits to how far we should compromise our principles: Phil thinks we should retain the Moon, because it is so important for the background and we care about the realism of the photometry and its uncertainties.
In issue #323 we have been discussing a pilot error-reducing “pipeline review”. Simon, Seth, Jim and Phil seem willing to take part, with review materials being prepared by Rahul, Scott, Bryce and Tom.
Materials to be reviewed are:
Sims Cookbook Recipe (Author: Scott, Phil)
Input catalog generation code (catalogs include catsim instance, phosim instance, phosim centroid, and so code includes treatment of lens sprinkling, SNe, and OpSim) (Author: Rahul et al, Phil)
PhoSim pipeline tasks and configuration files, including default values (Author: Tom)
DM Level 2 Cookbook Recipe (still without image differencing, for now) (Author: Simon)
DM Level 2 pipeline tasks and configuration files (Author: Tony)
When can all this be prepared by? Simon has limited availability next week, so we will plan out time around him. He will provide a short list of windows, which we will promptly doodle for.
What needs to be done in order for the reviewers to be walked through it? We need a review page where links to each piece of material is provided. Phil and Jim will design this.
How long should we assign to this important review meeting? An hour in person might be enough, especially if some work has been done offline first.
Who needs to be at the review? Rahul, Jim and Tom can represent the coding, Phil, Simon, Jim and Seth can act as reviewers. Scott, Bryce, and Tony can help provide links to the pipeline materials, which must all be up to date and ready to run.