The Legacy Surveys (LS, [REF]) is a combination of three digital ground-based imaging surveys mapping more than 16,000 square degrees in three optical bands (g, r, and z) to a depth 1 to 2 magnitudes deeper than the Sloan Digital Sky Survey (SDSS [REF]). These surveys provide a public dataset of instrumentally calibrated images, source catalogs, cross identifications, time series, a powerful sky viewer, and more. This document describes the initial instrumental calibration of the raw CCD images from the three cameras and telescope. Further calibrations and data processing of these calibrated images are performed by the Legacy Surveys pipelines (LSPs [REF) to produce the LS data products as described in [REF].
The three surveys comprising the LS are the DECam Legacy Survey (DECaLS, [REF]), the Mosaic3 z-band Legacy Survey (MzLS, [REF]), and the Beijing-Arizona Sky Survey (BASS, [REF]). The LS is further augmented with DECam public data from other programs. We call this augmented DECam dataset DECaLS+ since the handling of all the DECam observations are uniformly the same including the instrumental calibration described here.
The Legacy Surveys collected data from three camera/telescope instruments: DECam/Blanco, Mosaic3/Mayall, 90prime/Bok. Each instrument requires a pipeline to remove their, often unique, instrumental signatures. While this document focuses on the techniques used to produce the science ready data -- uniform, linear flux across the CCDs, a flat sky, and accurate astrometry -- this section briefly introduces the three pipelines and the environment they run in.
The three pipelines are called the Dark Energy Camera Community Pipeline (DCP), the Mosaic3 Community Pipeline (MCP), and the BASS Community Pipeline (BCP). Collectively these are the Community Pipelines (CPs). The pipelines were developed and operated at NOAO. To be clear, the DCP evolved over several years from the Dark Energy Survey Data Management Community Pipeline (DESDM-CP [REF]) through a process of replacing the orchestration and calibration library components, restructuring or replacing processing modules, and adding modules for steps not provide by the DESDM-CP. The other pipelines were purely developed at NOAO for the LS but were informed by earlier NOAO instrument pipelines. The drivers for the development process were to maximize commonality with the infrastructure at NOAO for long term maintenance and operator consistency and easy of use. The end goal of the CPs is to provide science ready data where the distinction between which instrument that data came from is largely removed.
The CPs run on the same dedicated cluster of Linux machines at NOAO consisting of processing nodes, a database server, and shared file system servers. They draw the raw data and archive the final data products in the NOAO Science Archive ([REF] using the same interfaces. They are orchestrated to follow the same high performance, data parallel, distributed, coarse-grained, scatter/gather processing model. The orchestrator is the NOAO High Performance Pipeline System (NHPPS [REF]). The coarse-grained modules are structured into functionally similar service oriented sub-pipelines consisting largely of IRAF scripts with some CSH scripts. In many cases the same code modules are used in the different pipelinse. The elements of the DESDM-CP retained are wrapped in CSH scripts. A common calibration library interface and table schema organize calibration information and reference files in pipeline specific data stores. Data products are formated similarly as FITS multiextension files ([REF]) for the flux data, the data quality masks, and the weight maps. The astrometric FITS World Coordinate System (WCS) are represented using the same conventions. They are calibrated against the same Pan-STARRS and Gaia reference catalogs. Operator diagnostics have similiar content of logs and graphics.
Beyond the CPs the Legacy Surveys project applies additional pipelines (LSPs [REF]).
[Any words about portability and availability? Calibration data availability?]
A fundamental capability in pipelines and science analysis is segmenting images into sources and background. Most commonly segmenting sources is for creating science catalogs. But in pipelines the ability to create masks of the sources and identifying sky is also very important. This is the case for the community and Legacy Survey pipelines.
While elements of the DCP remaining from DESDM-CP make use of the widely known Terapix tools (SExtractor and SCAMP [REF]), all the CPs make use of the lesser known IRAF Astronomical Cataloging Environment (ACE, [REF]). This package was designed around producing the various types of segmentation products useful in pipeline algorithms: catalogs, object masks, two types of sky images, and sky variance images. It was also designed to so the products work with seamlessly with other IRAF tools, especially the object masks. This is clearly appropriate since most of the CPs functionality comes from IRAF tools and script modules.
The purpose of this special section is to streamline the descriptions for the various algorithms in the CPs that make use of one or more of the ACE segmentation products. The functions of the catalogs and object masks are fairly obvious. The sky is determined in 64x64 pixel blocks. A histogram is made of unmasked pixels and the mode determined. An iterated standard deviation around the mode is computed and the sky value is the mean in a 3 standard deviation about the mode in the histogram. When a sky value can't be determined in a block the sky value is filled in by interpolation from nearby blocks. The block values can be output as a binned mini-sky image. The sky values and sigma values can also be output at the size of the image by interpolation.
The southern component of the Legacy Surveys consists of observations made with the Dark Energy Camera (DECam [REF]). The bulk of the data was collected by the Dark Energy Camera Legacy Survey (DECaLS [REF]), an NOAO survey program (2014B-0404) on the CTIO Blanco 4m telescope with DECam. The data was augmented with observations from other programs utilizing DECam. All the raw exposures were archived in the NOAO Science Archive. The raw exposures were then instrumentally calibrated at NOAO as the first step of producing the DECaLS+ data set for DESI ([REF]) targeting and community research. These calibrated exposures are archived in the science archive. In this section the various calibrations to remove the instrumental signature of the camera are described. Some additional instrumental calibrations are performed by the Legacy Surveys pipelines to produce the final data release [REF]).
The instrumental calibrations are performed by the DECam Community Pipeline (DCP, [REF]). This is a high performance, data parallel, distributed pipeline running on a dedicated cluster of machines at NOAO. The DCP can be configured in various ways for specific purposes and data products. This document covers the configuration for the DECaLS+ dataset; in essence, the DECaLS+ pipeline. The DECaLS+ configuration skips projection to a standard sky sampling, sky flattening at the CCD level, combining of overlapping exposures, and detections of transients from overlapping exposures. The DCP data products reflecting these additional transformations are, therefore, not produced. These steps are performed by the LSPs.
As is the case with all DCP data products, the DCP evolved over time as some calibrations were improved and some steps were deprecated or significantly modified. The evolution of the pipeline was tracked by data products tagged with a pipeline version. The history of the evolution and the tags is given [here]. As one would expect the calibrations improved with time. DECaLS+ is based on a minimum pipeline version of [Vxxxx]. This means that data taken and processed earlier in the project, both as part of DECaLS and for some of the augmented data, was reprocessed with a version meeting the requirement. In this document the calibrations for the typical version are described with addendums as appropriate. Note that the most common instrumental calibrations did not evolve over time.
The DECaLS+ DCP draws datasets from the archive. For DECaLS the datasets were typically runs. But for for reprocessing and augmenting from other projects a list of exposures, scattered in time, defined the datasets. The use of runs -- a set of (nearly) consecutive nights -- allows use of unregistered stacks of exposures to provide the median systematic pattern of the camera for removing residual structures across the whole array.
The datasets are divided into exposures to be ignored (i.e. bad), zero bias sequences, dome flat sequences, and science exposures. The dome sequences are divided by night. The dome flat and science exposures are also divided by filter and processed independently and in parallel. "Bad" data consists of exposures externally labeled as bad, those corrupted in some way (which were often recovered if the corruption happened during data transport), missing required keywords, and thoses whose titles (OBJECT keyword) indicate they are junk or tests of various types. Another important type of bad data are any exposures with the CCD voltages turned off or turned on in the preceding exposure (the first exposure after the camera is activated are unusable). Typically, activation occurs before dome calibrations and a first exposure to be ignored is part of the routine.
The DCP functionality, calibrations, correction, and products are presented in the following sections. Section 3.1 describes the creation of master dome calibrators that precedes science processing. The core of instrumental calibration -- flux calibration -- is described in section 3.2. Then astrometric calibration mapping pixels to celestial coordinates follows in section 3.3. Determining a magnitude zero is in section 3.4. Flux calibration doesn't necessarily produce a uniform background so section 3.5 describes handling of the sky background. There are various sources of unusable or scientifically suspect pixels which are identified and flagged as described in section 3.6. The last section, 3.7, describes the final data products. This functional organization does not represent the orchestration of the pipeline workflow where many of the steps are intertwined.
As is typical of CCD cameras, DECam observing includes taking sequences of exposures for zero bias and flat field calibration. This is typically done by the observatory staff. Zero bias exposures are taken with the shutter closed and blocking filters using and a commanded exposure time of zero. Flat field exposures are taken of an illuminated "white" screen in a darkened dome. To minimize noise contributions in these calibrators, sequences of consecutive exposures with optimal exposure times are taken and averaged together.
The nightly dome calibration sequences are processed first and in the order of zero bias and dome flats. The sequences are processed into stacked master calibrations. For random reprocessing the dataset lists don't contain dome calibration exposures and the appropriate master calibrations are expected to have been created and available in the DCP calibration library. The order of bias, dome flat, and science processing allows the result of the preceding step to be applied to the data in the next step.
The initial instrumental calibration steps are performed on all exposures -- calibrations and science alike. These operations are header updates (just resetting saturation levels thus far), crosstalk corrections, saturated pixel identifications, electronic (overscan) bias subtraction, and LED masking. Apart from the zero bias exposures, they are also corrected for non-linearity. These operations are described in subsequent sections.
Zero bias sequences for each night are stacked into a master zero bias. The pixels at each point in the mosaic array are averaged with the highest and lowest values excluded. The inverse variance, referred to henceforth as weights, of each pixel is estimated from the pixel statistics and engineering values for the readout noise and gain and stored as a weight map. The master zero files are available from the archive.
The dome flat sequences for each filter are corrected for bias with a master zero bias, scaled by their medians, and stacked as the average with the high and low value excluded. The master bias applied is the chronologically nearest which is almost always from the same night. The median is computed in a central region with bad pixels excluded. The weights are estimated from the pixel statistcs and readout noise and gain values for the amp and include the contribution from the bias subtraction. The weights are saved in a weight map for use in the science calibrations. The master dome flats are available in the archive.
A display of a typical bias and dome flats is shown in figure X. Fine details are not visible due to the size of the field and the differences in gain of the CCDs which dominate in the flat fields.
Figure X: Typical Bias and Dome Flat Master Calibrations from 2019-03-07.
Zero bias from -2 to 5 counts and dome flats from +/- 20% about the global mean normalization. Upper-left is a zero bias, upper-right is g-band, lower-left is r-band, and lower-right is z-band.
The primary purpose of an instrumental pipeline is to transform the raw instrumental pixel values to linear values with uniform proportionality to the flux from the sky. The first steps are to make the transformed values zero when there is no light and then make the proportionlity with flux (known as the gain) be the same for all pixels at all signal levels. Pixels that can't be transformed to scientifically useful values are identified and recorded in a data quality mask as discussed in section 3.6.
Estrada et al. (2010) reported DECam CCD nonlinearity at both low fluxes and near saturation as seen in flat field images. The DCP corrects for this nonlinearity with a amplifier-dependent lookup table that converts bias subtracted counts to linear counts (G. Bernstein et al. 2017, in preparation). The lookup table has not changed during the DECaLS+ processing. Figure X.A shows the linearity behavior as the numerical derivative of the lookup table. The curves for all the amplifiers are superimposed to illustrate the similarities and variation of the linearity. In an ideal linear detector the curves would be at one for all raw count values.
The crosstalk correction scales the signal from one amplifier and subtracts it from another amplifier. The scaling is defined by a set of coefficients (constant up to a level near saturation and then quadratic). Figure X.B plots the coefficients as the excess crosstalk contribution to a victim pixel by a 50,000 count pixel in another amplifier; i.e. the crosstalk coefficient multipled by 50,000. There are three cases in the plot. The white symbols show the excess in amplilfier A when the source pixel is in amplifier B of the same CCD, the red symbols for effect of amplifier B from amplifier A on the same CCD, and the green symbols for the effect on a CCD (in either amplifier) from a different CCD. In general crosstalk is strongest between amplifiers on the same CCD than between CCDs.
Figure X: DECam Linearity and Crosstalk
The left figure shows linearity curves for all amplifiers superimposed as a function of raw counts. These are computed by numerical differentiation of the linearity look-up table.
The right figure shows the crosstalk coefficients represented as the excess crosstalk signal at 50,000 counts as a function of CCD number. The white symbols are for the effect on amplifier A by amplifier B on the same CCD, the red are for the effect on amplifier B by amplifier A on the same CCD, and the green are the effect on either amplifier by amplifiers in other CCDs.
The individual science exposures -- the on-sky exposures with observation type of 'object' or 'standard' -- are calibrated by a master zero bias, a dome flat, a photometric flat (aka 'starflat' or 'photflat'), and possibly a dark sky illumination flat. An example grayscale rendering of the three types of flat field calibrations is shown in fig. X. These are selected as the nearest available in time. For bias and dome the master calibrations are typically from the same night as the exposure. For the photflat these are only generated periodically and, for some of DECaLS+, the only available photflat was from 2013. Illumation flats were generated from runs with many exposures. Use of illumination flats was deprecated when star flats became available at different epochs. The calibrations used are documented in the data product headers.
Dark sky illumination flat fields are a common technique for the instrumental calibration of CCD cameras. These are like dome flats but using the night sky as the source of uniform illumination. The problem is conflating response patterns with sky patterns which, for the wide field of DECam, can be significant. For the first years of the DCP, including a significant portion of the DECaLS+, a standard star flat from 2013, supplied by DESDM-CP, was used as the basic photometric response calibration. This was followed by steps to remove sky patterns (see section X.X) and then any residual patterns were addressed by a dark sky illumination flat. One reason for this was to deal with variations in the camera response with time that were not addressed by updates to the photometric flat fields.
The dark sky illumination flats were constructed when datasets, usually from a several of nights and even programs, had a sufficient number of suitable dithered exposures. The exposures were first dome and photometric flat fielded followed by pupil ghost and large scale background gradient subtraction (see section 3.5). The exposures are combined using source masks and statistical rejection to form the average response to sky. This image is smoothed and normalized to unit mean across the CCDs to form the dark sky illumination flat field. After inspection for non-sky structures -- imprints of any faint halo light of bright sources or insufficient dithering -- the illumination flat field is added to the DCP calibration library. All the illumination flats in the library are then available for application based on proximity to the observation date. An illumination flat is applied multiplictively and the file name is recorded in the image header keyword ILLUMCOR. An example illumination flat is shown in fig. X.C.
A different, somewhat more recent, technique for addressing spatial response variation beyond dome flat fields is based on photometry of stars. These are variously called photometric flats, star flats, or "thousand points of light" calibration. The same stars are dithered spatially (such that the same stars appear on multiple amplifiers and CCDs) and then a response pattern that minimizes the photometric differences, after dome flat fielding, is generated. An example photometric flat is fig. X.B. When the DCP was able to create such photometric flats from special on-sky calibration fields (PLVER=4.8) via the Berstein method ([REF]) the use of illumination flats was deprecated and dark sky templates were no longer produced. The DECaLS+ has a mix of DCP data with and without illumination correction and with epoch dependent star flats applied.
Figure X: Dome Flat, Star Flat, and Illum Flat
These are r-band flat fields. (left) Dome flat +/- 20%, (middle) Star flat +- 20%, (right) Dark sky illumination flat +/- 1%.
Astrometric calibration consists of matching stars detected in an exposure with an astrometric reference catalog and fitting a world coordinate system (WCS) function mapping image pixel coordinates to celestial coordinates for each CCD. The WCS function for a CCD is a 3rd order bidimensional polynomial distortion followed by tangent plane projection in a FITS formalism, called TPV, as described here [link].
The source catalogs for each CCD are generated by SExtractor ([REF]). These are matched against the 2MASS reference catalog (PLVER < X.XX), Gaia DR1 (PLVER between X.XX and X.XX), and Gaia DR2 (PLVER > X.XX). The source catalogs include celestial coordinates evaluated from the instrumental pixel centroids, a predefined library WCS, and the pointing supplied at the telescope. The matching is done by SCAMP ([REF]). A vaild match requires the matching statistics (number of stars, RMS, and Chi^2) to meet certain thresholds.
There are a number of reasons the matching may fail with the most common one being low transparency producing few or no stars. During this and subsequent steps a failure results in a header keyword indicating the failure. The post-DCP DECaLS+ processing ignores the exposures that failed the DCP astrometric solution.
Each CCD has a predetermined WCS function that defines the CCD position and orientation in the mosaic array relative to a tangent point near the center of the field. It also includes a higher order distortion map. The predetermined distortion maps were derived by the DES project [REF]. The same maps were used throughout the DECaLS+ project.
These library WCS functions correct each star to a nominal undistorted astrometric coordinate. The astrometric solution then fits a 3rd order polynomial to adjust the tangent point, orientation, and a constrained large scale distortion across the mosaic array to minimize the errors in the match to the astrometric catalog. The solution is then exported to the CCD WCS in the TPV formalism ([link]). This solution method essentially keeps the predetermined metrology with allowance for a constrained distortion of the focal plane. DECaLS+ calibrations beyond the DCP allow for additional independent small offsets of the CCDs to minimize the positional errors not accounted for by the focal plane constrained astrometric solution of the DCP. This later processing also applies proper motion corrections which is not part of instrumental calibration.
A photometric characterization, meaning a single filter magnitude zero point, is performed using PS1 provided the WCS solution was successful. Another SExtractor source catalog is generated with the new WCS which is then matched to the PS1 sources which is not hard given a successful astrometric calibration. However if this fails (when the astrometric solution appeared successful but was not actually correct), then the photometric and astrometric solutions are marked as failed in the data headers.
A magnitude zero point is determined by an iterated sigma clipped average of the instrumental to reference magnitude differences. The zero point and various associated photometric quantities are recorded in the image headers. A photometric catalog of detected sources with zero point calibrated magnitudes is also produced but is not exported as a data product.
The DECaLS+ post-DCP processing does its own photometric calibrations based on PS1 independent of the DCP characterization. Therefore, for some DECaLS+ reprocessing the DCP is configured to skip this step to save computational time.
The goal of these steps is to produce a constant background across the field by adding or subtracting counts. There are two primary reasons for this; making it easier for later photometry algorithms to measure background and for visual evaluation. The latter is more than just esthetic -- the eye is very good at finding systematic problems in displays of the image, but background gradients make such evaluations less sensitive. Some instrumental pipelines go so far as removing background to a zero level. The DCP maintains the original mean sky level as a potentially useful characteristic of the exposure.
There are two challenges in estimating the sky background. One is that, despite the various gain calibrations, there may still be variations caused by residual gain errors. Disentangling multiplicative and additive effects ultimately requires an assumption that one is negligible compared to the other. We assume the various steps described for gain uniformity, particulary the photometrically derived flat fields, are sufficient at whatever scale the background is to be made uniform. The second problem is with very large scale, low surface brightness light from astronomical sources such as large galaxies and inter-galaxy cluster light. This is guided by the science goals. For the DECaLS+ the goal is the distant extra-galactic population where sources are small. Studies of larger scale, low surface brightness features can be done with targeted re-reductions for fields of special interest.
There is a reflection path between the filter and the dewar window in DECam. This reflection produces a large doughnut pattern, known as a pupil ghost, whose amplitude varies with the filter and amount of light in the field of view. The photometric effect on the data from the dome flat field is accounted for by the star flat. What remains in science data is a background that needs to be subtracted. The methods used to deal with the pupil ghost evolved over time and each have left some low level residuals (a small fraction of the pattern amplitude) in the large scale background.
There are also large scale patterns in the sky background that vary with the observing conditions; primarily from stray and scattered light in the telescope and dome and, particularly, from moon light. Smaller, more defined, patterns are reflections from very bright stars that produce halos of various sizes and distance from the culprit star. The DCP does nothing for the latter case and DECaLS+ will have some artifacts of this type. For the former, the DCP fits a large scale function across the field.
The pupil and stray light patterns are, of course, part of a single background which the DCP addresses in stages; first the pupil ghost and then the stray light. Unfortunately this is also coupled with residual response differences in the CCDs which, as noted previously, are considered to be removed by preceding flat fielding. The remaining amplifier and CCD differences are then attributed to background light color differences.
An ACE mini-sky image is produced for each CCD. These are pasted together into a single representation of the DECam field. Figure X.A shows an example. This single image representation of the full field allows measuring large scale patterns such as the pupil ghost and background gradients.
Figure X: Example Pupil Ghost and Sky Segmentation
The pupil pattern is modeled in each exposure by fitting a function in rings (azimuthally) and then fitting a function across the rings (radially). An inner and outer ring defines a background plane where the fitted pattern goes to zero there. The result is a pupil image in the same 64 binned scale.
For most of DECaLS+ (DR9 and earlier) the pupil image in the ring fitting pupil ghost image is magnified by linear interpolation over the section corresponding to each CCD and subtracted from the exposure. Any question of amplitude scaling is implicit in the fitting since it is done per exposure. Because of beating against the differences in CCD sky response levels and absence of any constraint on the radial shape, there were some ring-like residual artifacts in the background.
An improved algorithm in the most recent CP version, which applies to just a few DECaLS+ DR9 exposures, takes good nights with many exposures and stacks the ring fitting mini-pupil images to make pupil templates for each filter and filter cassette. The cassette in which a filter is located have different distances from the dewar and, hence, different scales. Figure X.B shows an example template. At this grayscale mapping the individual exposure ring fitting approach looks identical. When a template is available an amplitude is determined, weighted by the pupil signal. The mini-sky image of the exposure and the amplitude scaled template is magnified and subtracted rather than the individual ring fitting model. In addition, to deal with the CCD sky response levels, a residual template from a stack of many exposures, after the pupil template and large scale sky subtraction, is produced and applied. The result is an improved background with reduced ring-like residual structures.
After the pupil pattern is removed, figure X.C, the remaining background of the mini-sky image is fit by a bi-dimensional cubic spline with a scale length of approximately two CCDs. This does a good job with smooth gradients but some stray light patterns in bright moonlight are not removed. This is most visible in DECaLS+ z-band exposures which were prioritized during moonlight conditions. The mean of the fitted pattern is removed and then the sky is magnified by linear interpolation for each CCD and subtracted. The result is that the mean background of the exposure retains its counts while gradients and other large patterns are removed.
Bad pixels are primarily defined by instrument scientists as unusable pixels in the CCDs. Saturated pixels are identified based on pixel values after crosstalk correction and electronic (overscan) bias subtraction. The thresholds for saturations are set by instrument scientists. Bleed trails are identified algorithmically. This includes "edge bleeds" produced by extremely bright stars saturating the serial readout register and affecting the image lines near register.
The identification of cosmic rays has evolved over time. An initial cosmic ray algorithm from DESDM was later replaced by a DESDM version of an LSST algorithm. An additional NOAO developed algorithm is based on ACE. ACE catalogs includes a variety of measures including shape related quantities. These catalog quantities are used with a multi-parameter filter to separate sources from cosmic rays and streaks. Care is taken to not identify edge-on disk galaxies as streaks. A challenge in using segmentation catalogs is that long streaks either blend with other sources in the field or break up into short pieces during the threshold detection.
These various transient detection algorithms are designed to be conservative to avoid misidentifying true celestial sources. Therefore, many streaks and some cosmic rays are missed and not labeled in the data quality mask. The later DECaLS+ process of building coadded brick tiles also adds transient identifications through comparison to all data overlapping a given point ([REF]).
An LED masking step applies only during a period when a CCD (N15/#46) developed a variable strength glowing pixel which corrupted calibrations when not masked. The LED step determines the strength of the glow and masks a region around the pixel based on that strength.
The DCP produces a data quality mask. The mask consists of integer arrays with code values for various categories of pixels with suspect values. The data quality mask is updated as the various calibration operations are performed. The code categories are shown in table X.
1 = bad pixels 3 = saturated pixels 4 = bleed trail and edge bleed pixel 5 = cosmic rays 8 = streaks
Note that the data quality identifications are not perfect. During the course of DECaLS+ identifications algorithms changed, were tweaked, and had problems with assignments of the codes. The cosmic ray and streak identifications only identify a fraction of the actual cases. Beyond the DCP the Legacy processing identifies additional pixels with data quality issues. Most significantly by outlier identifications in the multiple images contributing to the LS tiling (aka bricks). The DCP data quality data products are what is in the science archive and the later identifications are within the data release files.
The last steps of the DCP are to create the final data products, archive them, and pass them on to the next stage LS pipeline. The data products are all compressed FITS files ([REF]). When uncompressed they are FITS multi-extension files (MEF [REF]) with a data-less primary header of global keywords applying to all extensions. There are typically 60 or 61 extensions, one for each working CCD. Table X identifies the different DCP data products for the DECaLS+, table X defines the data product filenames as stored in the archive and referenced in the LS data releases, and table X specifies the data quality codes in the data quality masks.
Table X: DCP Data Products
The table shows the data products and how they are categorized. The OBSTYPE column is the observation/calibration type where "object" means an on-sky science exposure. The PROCTYPE and PRODTYPE columns categorize a data product by processing type and product type column. The CODE column is a identifier in the archive filenames reflecting the OBSTYPE, PROCTYPE, and PRODTYPE with single character codes in that order. The DCP codes for the data products are shown in table X and the character code convention and archive filename schema is described in [REF].
OBSTYPE PROCTYPE PRODTYPE CODE DESCRIPTION object InstCal image ooi The instrumentally corrected science flux data with astrometric solution for each exposure. This includes a uniform gain and background sky matching the mean of the raw counts. The pixel values are in units of photoelectrons per second based based on the measured gains and exposure times. object InstCal dqmask ood Data quality mask of integer bad pixel codes as defined in table X. object InstCal wtmap oow Weight map (inverse variance) for sky pixels sampled and smoothed over a scale of 64 pixels. zero MasterCal image zci Master zero bias from a calibration sequence. dome flat MasterCal image fci Master dome flat field from a calibration sequence. fringe MasterCal image hci Fringe template from a night with at least 50 good exposures. illum MasterCal image oci Dark sky template from a night with at least 50 good exposures.
Table X: Data Product Files
The data products are all compressed FITS files ([REF]). When uncompressed they are FITS multi-extension files (MEF [REF]) with a data-less primary header of global keywords applying to all extensions. There are typically 60 or 61 extensions, one for each working CCD. The files in the archive and as referenced in the LS data products have names with the following structure. The naming convention for the archived data products is described in [REF].
[instrument]_[yymmdd_hhmmss]_[code]_[filter]_[version].fits.fz
Field Description [instrument] For the DECaLS+ this is always "c4d" for Cerro Tololo + Blanco 4-m Telescope + DECam. [yymmdd_hhmmss] The UT date/time of the exposure. [code] The data product code as defined in table X. [filter] The abbreviated filter band which is g, r, z for the primary DECaLS+ imaging. This applies to all data products except zero bias for which this field is missing. For the calibration files there is also a letter following the filter for the type of calibration: e.g. F for dome flat. [version] The version of the data product which is distinct from the pipeline version. For the LS DR9 release this is mostly "ls9".
Table X: Data quality codes
The data quality data product uses the integer codes given here for each pixel. The code values are in priority order. For example, if a pixel might lie in a cosmic ray but if it was considered a CCD defect it would code 1 rather than 5.
0 = scientifically good pixels 1 = bad pixels 3 = saturated pixels 4 = bleed trail and edge bleed pixel 5 = cosmic rays 8 = streaks
The Mosaic3 z-band Legacy Survey (MzLS [REF]) was performed with the Mosaic3 camera ([REF]) on the Kitt Peak Mayall 4-m telescope. The telescope was dedicated to the survey after the end of general community use and before the installation of DESI ([REF]). The camera was built specifically for MzLS with CCDs optimized for z-band response. The instrument had some unusual characteristics beyond those of a typical CCD camera. Some of the calibration steps described in this section are about dealing with those characteristics and are covered in more detail than the more typical CCD calibrations.
The Mosaic3 camera provides similar 4 CCD / 16 amplifier raw data to the 90prime camera of the BASSLS camera. The principle difference is a finer pixel scale. This means much of the MCP and BCP share code and a similar workflow.
The raw exposures are archived in the NOAO Science Archive. They were then instrumentally calibrated at NOAO as the first step of producing the LS data set for DESI targeting ([REF]) and community research. The calibrated exposures are archived in the science archive. The instrumental calibrations were performed by the Mosaic3 Community Pipeline (MCP). This is a high performance, data parallel, distributed pipeline running on a dedicated cluster of machines at NOAO. In this section the various calibration methods used to remove the instrumental signatures of the camera are described. Additional calibrations were also performed by Legacy Survey pipelines to produce the final LS data releases [REF]).
The MCP draws data to be processed from the archive for each night. The datasets are divided into exposures to be ignored (i.e. bad), zero bias sequences, dome flat sequences, and science exposures.
The Mosaic3 controller had occasional readout errors which still produced archived raw exposure files. The MCP has heuristics to detect errors which cannot be recovered in the pipeline. These include missing required keywords (e.g. exposure time and filter), the same data read out multiple times, and incomplete pixel counts. Other exposures that are excluded from pipeline processing are those listed in a "bad exposure" list or identified as focus, donut, junk, or pointing by the observer in the title keyword.
Another occasional readout error was dropping the first column in certain amplifiers. This column is not part of the imaging area so the science data can be recovered. The MCP uses a heuristic to identify this situation, adds a dummy column, and notes it in the logs and in an XSHIFT keyword which is only present when the fix was needed.
The MCP functionality, calibrations, corrections, and products are presented in the following sections. Section 4.1 describes the creation of master dome calibrators that precedes the science processing. The core of instrumental calibration -- flux calibration -- is described in section 4.2. The astrometric mapping pixels to celestial coordinates follows in section 4.3. Determining a magnitude zero point is described in section 4.4. Flux calibration doesn't necessarily produce a uniform background so section 4.5 describes the handling of the sky background. There are various sources of unusable or scientifically suspect pixels which are identified and flagged as described in section 4.6. The last section, 4.7, describes the final data products including creation of an inverse variation product for estimating pixel value uncertainties. Nothe that this section organization does not represent the orchestration of the pipeline workflow where many of the steps are intertwined.
As is typical of CCD cameras, the MzLS observing nights included taking sequences of exposures for zero bias and dome flat field calibration. Zero bias exposures are taken with the shutter closed and a zero exposure time. Flat field exposures are taken of an illuminated "white" screen in a darkened dome. To minimize noise contributions in these calibrators, sequences of consecutive exposures with zero or optimal exposure times are taken and averaged together.
The nightly dome calibration sequences are processed by the MCP first and in the order of zero bias then dome flats. The sequences are processed into master calibrations for each night Every exposure in a calibration sequence is corrected for crosstalk, bad pixels, and electronic bias as described in section 4.2. Dome flats exposures also apply the chronologically nearest master zero which is typically from the same night.
After the instrumental corrections are applied to the individual exposures of the sequences they are combined into master calibrations. The combination is a pixel-by-pixel average with a number of low and high values (nlow and nhigh) excluded. The number excluded is given by
nlow = max (1, n - 7) / 2, nhigh = max (1, n - nlow - 5) / 2
where n is the number in the sequence and integer division (rounding down to an integer) is used. The typical MzLS calibration sequences were 11 exposures which means the two highest and two lowest values were excluded and the remaining 7 averaged. Figure X shows a master zero bias (left) and master dome calibration (right) for February 5, 2018. The pattern noise seen in the bias is a characteristic of this camera. It is variable so that it is not removed by the master bias in the science exposures. This is discussed later.
A final correction is applied to the master dome flat field. The Mosaic3 camera has a reflection between the filter and the corrector [correct me if I'm misremembering this light path] which produces a doughnut pattern, commonly called the pupil ghost. If this is not removed from the flat field then it would bias the flat field correction on the science exposures by treating this pattern as aa gain variation. The pattern is removed by fitting a function in rings ( azimuthal fits) and then fitting a function across the rings (radial fits). This is done for each ccd independently and only where the pattern is present. The fits are made to go to zero at the inner and outer edges of the pattern. The fitted pattern is then removed multiplicatively leaving a flat field without the pattern visible. The quality of the pupil ghost removal is demonstrated in fig. X.
Figure X: Typical Bias and Dome Flat Master Calibrations from 2018-02-05.
The primary purpose of an instrumental pipeline is to transform the raw instrumental pixel values to linear values with uniform proportionality to the flux from the sky. The first steps are to make the transformed values zero when there is no light and then make the flux proportionlity (known as the gain) be the same for all pixels. Pixels that can't be transformed to scientifically useful values are identified and recorded in a data quality mask as discussed in section 4.6.
The first step is a correction for wraparound of the digital values from the analog converter when the signal exceeds 65535. MzLS amplifiers 1, 3, and 4 are not corrected. For the other 13 amplifiers the median of each line is computed and pixels with values below 90% of the median are assumed to have wrapped around. For those a value of 65535 is added. Naturally this results in these pixels being identify as saturated as described in section 4.6.
A crosstalk correction is next. The correction scales the signal from one amplifier and subtracts it from another amplifier. The scaling is defined by a set of coefficients for each pair that have a measurable effect. The strength of the crosstalk varied with time so the coefficients were determined at different epochs. Figure X shows the magnitude of the crosstalk when the contributing amplifier has 50,000 counts. The ordinate is the amplifier number (with the bottom bars indicating the CCD groupings) and the absissa is the correction counts. Note that a negative correction means the counts must be increased in the affected amplifier. The Mosaic3 crosstalk was found to only be induced by amplifiers on the same CCD.
Figure X: Mosaic3 Crosstalk @ 50K
The electronic bias for each amplifier is determined for each line in each exposure from extra amplifier measurements immediately following reading out each CCD line. These extra measurements are commonly call overscan pixels. For most exposures the overscan pixels for each line are averaged with the high and low pixels excluded -- i.e. a line-by-line calibration. For a small number of exposures in the survey, those with a very bright star as the last pixels read, a line-by-line correction produced a poorer result. For those cases the line-by-line method was replaced by a high order legendre polynomial fit to all the averaged line overscan values and the fit values for each line are used to subtract the electronic bias. The remaining zero level instrumental signature is removed by subtracting the master zero bias for the night.
The first gain normalization calibration applies a master "white screen" dome flat field; usually from the same night. A predetermined mean gain for the camera (fixed at 1.80 for the whole of the MzLS) and the recorded exposure time is used to convert the pixel values from instrumental units to estimated photoelectrons per second. The MCP data products, therefore, have an effective gain of unity and pixel values of photoelectrons per second. A header keyword, GAINNORM, is recorded to allow conversion between photon rates and detector counts as needed.
A static photometric flat field, refered to as photflat1 to distinguish it from a later, second photometric flat field, follows to normalize the gains to Pan-STARRS photometry. This calibration was based on a large set of early exposures. The general photometry trend found was approximated by a linear radial function from the center of the camera array; i.e. a cone surface. Figures X.1 and X show an image and radial cut of this photometric flat field normalized to 1 at the center of the camera. Some of the trend can be attributed to the optical projection of the sky on the pixels such that pixels near the edge of the field image a smaller patch of sky. These pixels should measure less flux but the dome flat field assumes the pixels should measure the same flux leading to exaggerating the response. The photometric flat field naturally includes a correction for this effect. The red curve in figure X shows the spatial response without this effect. What is seen is that the majority of the photometric calibration is not due to this optical projection effect. What it is due to has not been explained [true?].
Figure X: Photometric star flats for Mosaic3.
The correction is the white line which is a linear function in radius from the center to the edge; i.e. a cone surface. The red curve shows this corrected for decreasing projected pixel area on the sky from the optics. This indicates that the majority of the photometric correction is not due to this optical effect.
(Left) First photometric flat from PS1 photometry - 15% range. (Right) Second photometric flat from repeat imaging of stars - 10% range.
The reason photflat1 is a static calibration over the course of the survey is that it is fairly stable and the MCP later applies further photometrically based corrections. One is a nightly calibration, referred to here as an ampflat correction, and a second is a photometric star flat, referred to as photflat2, derived for each season based on repeat observations of stars after all other corrections have been applied. The seasonal photometric (star) flat field was derived using the calibration procedure of Schlafly+2012. Figure X.2 show an example of a photflat2.
The ampflat correction is applied in an incremental manner. Since the MzLS was a dedicated program, observing night after night in seasons, the correction from the preceding night is usually the one applied after the photometric star flat. This correction, which is different for each amplifier, involves measuring the sky brightness with a masked and sigma clipped mean and evaluating an ampflat function as described below. This stage of calibration, with the preceding night's ampflat correction and with or without the second pass seasonal star flat, is used for the following steps of magnitude zero point measurement, astrometric calibration, pupil pattern subtraction, and sky pattern removal. The result after these steps is then used to update the ampflat function for the current night in the calibration library. The delta from the preceding night's ampflat to the current night is applied to the data and the net correction (previous night plus the delta from the current night) is recorded in the data headers.
As part of the MCP photometric verification against Pan-STARRS it was found that magnitude zero points varied in a way not consistent with the expected atmospheric causes. The variations correlated with the sky brightness and the correlation was different for each amplifier. This could imply an amplifier non-linearity. Considerable engineering effort went into trying to derive a traditional linearity function; i.e. a look-up table or function mapping an input pixel value to a pixel value. This was not successful.
Instead the MCP implemented a gain correction as a function of sky brightness for each amplifier. This is called the "ampflat function". While this is not equivalent to a true linearity correction the final photometry produced by the MzLS, which only applies a magnitude zero point to each CCD in the post MCP calibration, is better than XXX ([REF]) with no indication of a systematic error with magnitude for unsaturated stars. A plausible explanation is that rather than a non-linearity the amplifier gain changes due to temperature changes induced by the amount of charge being read which is dominated by the background level. [What do you think of this speculation?]
The ampflat functions are derived as follows. After all corrections have been applied the stars in the exposure are cataloged and matched to the Pan-STARRS reference catalog. The differences between the zero point computed over all amplilfiers and just for the stars in the amplifer are computed. The sky brightness is also measured for each amplifier. On a night with at least 50 exposures a 2nd order legendre polynomial is fit to the zero point differences for each amplifier as a function of the decimal logarithm of the sky brightness. There are various sanity checks and exclusion measurements with sky levels below 1000 and above 8000 instrumental counts. The fitting includes 5 iterations of 2 sigma clipping. The amp flat function is then evaluated for each amplifier in each exposure and applied as another gain correction. The coefficients are merged with the correction applied to the data from the previous night to produce a new amp flat function for subsequent data.
Figure X illustrates the sky brightness dependency and the correction method. The figure shows the four amps in CCD2 for the night of February 5th, 2018. The zero points, shown as a gain correction, for all exposures compared to Pan-STARRS are plotted. To better illustrate the effect, only the previous night's mean gain correction was applied rather than the sky level dependence. The curves in the figure show the fitted behavior with sky level. The sky level is shown in instrumentally corrected counts.
Figure X: Delta amplifier flat.
4.3. Astrometry
ACE source catalogs for each CCD are matched against the Gaia astrometric reference catalog ([REF]). During the processing of MzLS (for DR9) Gaia DR1 was primarily used. The source catalogs include celestial coordinates evaluated from the instrumental pixel centroids, a library astrometric distortion map, and the telescope supplied pointing. The matching determines a pointing offset, a scale change, and any small rotation. The latter two effects are the result of differential refraction. A vaild match requires a fraction of the source and reference coordinates to match. When a match is found a catalog of measured pixel coordinates and Gaia celestial coordinates is created.
There are a number of reasons the matching may fail with the most common one being low transparency producing few or no stars. During this and subsequent steps a failure is identified by a header keyword indicating the failure. The post-MCP MzLS processing ignores exposures that failed the MCP astrometric solution.
A world coordinate system (WCS) function for each CCD is fit to the matched catalog sources. The function used is a distorted zenithal project (ZPX [REF]) which includes a model of the telescope/camera optical pincushion distortion. The remaining distortions are fit by a bi-quartic polynomial. The RA and DEC quadrature residual rms of the WCS relative to the Gaia coordinates is required to be below a particular value. If any CCD fails then the astrometric solution is considered to have failed.
A successful solution adds the WCS function and some diagnotistic header keywords (ASTMCAT, WCSCAL, WCSDRA, WCSDDEC, WCSRMS) to the exposure and updates the source catalog celestial coordinates. For consistency across the Legacy Survey the fitted ZPX WCS function is recast to a distorted tangent plane WCS (TPV [REF]) in the image headers. The celestial coordinates in the source catalog are updated using the astrometric solution.
Early in the MzLS it was discovered that the Mosaic3 readouts had a clocking error that introduced a 1/3 pixel shift in the field between the lower and upper amplifiers in each CCD. In order to have a continuous astrometric function across the CCD, which is desirable to maximize the number of astrometric reference stars and utilize continuity constraints across the CCD, the MCP shifts the upper CCD amplifier images to put all pixels on the same integer CCD grid. The rebinning uses a sinc interpolation that preserves the noise. Evaluation of the statistics and the astrometric solution were indistinguisable from data after the controller error was corrected. A YSHIFT header keyword is present to document when this correction was applied to data prior to fixing the error.
Figure X show the astrometric errors in right ascention (along colums in the camera orientation) between the continuous WCS astrometric solution and the Gaia coordinates averaged in 64 pixel slices for the 4 CCDs. The distortion function attempts to converge for both lower and upper amplifiers but can't handle the discontinuity near the boundary which causes the jump seen in the figure. Figure X shows a 2D visualization of the astrometric errors averaged over all amplifiers in 64x64 blocks. The left column shows this error pattern from the 1/3 pixel readout problem with and without the digital shift corrections. With the corrections the astrometry shows difference between the lower and upper amplifiers in RA.
Figure X also shows a "bow" pattern. This is stronger along DEC (the middle column) though also with some effect on RA. This pattern is attributed to the way the CCDs are mounted. The amplitude of the coordinate distortion is about 25 milli-arcseconds in each bow but in opposite directions. The MCP does not correct for this in the WCS function.
Figure X: Centroid Shift Correction
The DEC differences between the Gaia reference coordinates and the coordinates from the astrometric solutions applied to the measured instrumental coordinates as a function of CCD line. The measurements and solutions are from all exposures on the night of 2016-03-01 which had the 1/3 pixel readout error. Each point is an average over 64 lines. The discontinuity arises from fitting a single astrometric function over the whole CCD which cannot then respond to the abrupt 1/3 pixel offset boundary. The plots illustrate the magnitude of the error if not fixed and that it is similar in all four CCDs.
Figure X: Astrometric Statistics
The differences between the Gaia reference coordinates and the coordinates from the astrometric solutions applied to the measured instrumental coordinates are visualized as averages in 64x64 regions. The measurements and solutions are from all exposures on the night of 2016-03-01. This is a night before the 1/3 pixel error in the readouts between top and bottom was fixed.
The three columns are for (left) errors in RA, (middle) errors in DEC, and (right) RMS (quadrature combination of RA and DEC standard deviations). The two lines are for (top) without the numerical shift and (bottom) with the correction. The grayscale for the left and middle columns are +/- 100 marcsec and the right panel is 100 to 400 marcsec. Note this combines all CCDs from all exposures. Separating by CCDs shows patterns which are very similar to those shown (e.g. see figure X).
Note that RA varies vertically and DEC horizontally so that the vertical shift problem and its minimization is visible in the right panels. The amplitude of the jump is about 150 marcsec. The middle column, and to a lesser extented the left columm), show a pattern induced by the CCD bonds. The two bows bias the coordinates in opposite directions which results in the bright and dark pattern. The MCP does not currently correct for this. The amplitude (peak to mean) in each bow is about 25 marcsec.
4.4. Photometric Characterization
A photometric characterization is performed using PS1 provided the WCS solution was successful. The updated source catalog is matched to the PS1 stars, which is not hard given a successful astrometric calibration. However this occasionally fails because the astrometric solution reported a successful RMS from random misidentifications in clouds and cosmic rays. When the photometric matching fails then both the astrometric and photometric solutions are marked as failed in the data headers. When the matching succeeds a magnitude zero point is determined by an iterated sigma clipped average of instrumental to reference magnitude. The zero point and various associated photometric quantities are recorded in the image headers. A photometric catalog of detected sources with zero point calibrated magnitudes is produced.
The MzLS post-MCP processing does its own photometric calibrations based on PS1 independent of the MCP characterization. However, the MCP photometric catalogs for each exposure of the night are used at this point to determine the final ampflat corrections as a function of amplifier and sky brightness as described earlier.
4.5. Sky Background Uniformity
The goal of these steps is to produce a constant background by adding or subtracting counts. There are two primary reasons for this; making it easier for later photometry algorithms to measure background and for visual evaluation. The latter is more than just esthetic -- the eye is very good at finding systematic problems in displays of the image, but background gradients make evaluations less sensitive. Some instrumental pipelines go so far as removing background to a zero level. The MCP maintains the original mean sky level as a potentially useful characteristic of the exposure.
There are two challenges in estimating the sky background. One is that, despite the various gain calibrations, there may still be variations caused by residual gain errors. Disentangling multiplicative and additive effects ultimately requires an assumption that one is negligible compared to the other. We assume the various steps described for gain uniformity, particularly the photometrically derived flat fields, are sufficient at whatever scale the background is to be made uniform. The second problem is with very large scale, low surface brightness light from astronomical sources such as large galaxies and inter-galaxy cluster emission. This is guided by the science goals. For the MzLS the goal is the distant extra-galactic population where sources are small. Studies of larger scale, low surface brightness features can be done with targeted re-reductions for fields of special interest.
The converse of the source detection segmentation is the background. The background is determined by statistics in 64x64 pixel blocks. The background statistic is a mean about the mode within a 3-sigma window. The sky values are recorded as images; i.e. it is a 64 times reduced image of the background for each amplifer. All the background mini-images are pasted together to produce a single image of the mosaic. This process preserves differences in amplifier backgrounds and shows background patterns that extend across the field of view. Figure X shows an example with the with the sky statistics blocks visible in the rendering.
The background images are used to remove the background pattern (gradients and amplifier level difference left after photometric flat fielding) by subtracting the mean, magnifying (by linear interpolation) each CCD section, to the full size of the CCD images, and subtracting.
There is one complication with Mosaic3 which is the pupil ghost pattern that is prominent in the binned sky images. Fortunately the pattern is fairly compact and localized to a region in the central four amplifiers (one per CCD). The pattern is removed prior to computing the the residual background around the mean.
This is done first by fitting a linear function of radius to the sky values outside the region affected by the pupil ghost. Then a quadratic function of radius is fit to the sky values within the pupil region with the linear fit, non-pupil background subtracted. The quadratic model is then subtracted from the binned sky image. Note that this is not the actual pupil ghost removal from the data products which is done later in the full sized image data (see fig. X).
Figure X shows an example of a binned mini-sky image before and after the quick pupil removal. The right image is the one that is magnified and subtracted from the exposure. This is the same exposure as in fig. X.
Figure X: First Sky Removal
The next steps involve pixel-space stacks (meaning not celestially registered) of the exposures from a night to make calibrations reflecting the systematic patterns. In addition to requiring a minimum number of exposures, an analysis is done on the mini-sky files of the night. The analysis looks at the exposure times, amount of saturation, sky brightness, magnitude zero points, and rms of the sky images about the mean. These are iterated to remove outliers. The end result is a list of exposures, called the template exposure list, which are typical good exposures in terms of sky signal in non-cloudy conditions.
If the template exposure list from the night is sufficiently large a pupil template image is constructed. The exposures are masked to just the pupil ghost region. Sources overlapping the region are masked by the ACE source masks produced earlier. The masked images are scaled to a common mean sky level and combined by a clipped average. The clipping is +/- 3 times the difference between the median and the 10% percentile (see pclip algorithm in [REF]). Pixels that were masked in all exposures are filled in from surrounding pixels by interpolation. The pupil pattern in the stack is fit in polar coordinates about a predetermined center and edges. Outlier values in the template are clipped. The final pupil template is entered into the calibration library and used subsequently by the chronological proximity of observations to the night. Therefore, nights in which a pupil template is generated use the template generated from the its own observations and nights without sufficient good data to create a template use the closest available night's template. Figure X.x is an example of the derived pupil ghost template.
A pupil template, smoothed by a moving 5x5 block average, is scaled and subtracted from each exposure. The relative amplitude is determined from pixels within the pupil region mask and not in the source masks The scale is computed using the template values as weights so that the statistical weight is strongest where the pupil is strongest. Figure X illustrates the pupil subraction. The left image is a single exposure before the pupil subtraction, the middle image is the pupil template create from the night's data, and the right is after scaling and subtracting to the particular exposure. Only the central region affected by the pupil ghost is shown in this figure. The quality of the pupil ghost removal is generally excellent with the exceptions being with extremely bright stars near the center or bad or saturated data.
Figure X: Pupil Ghost Removal
The next steps are similar to those for the pupil ghost. If the template exposure list from the night is sufficiently large a dark sky template is constructed. The exposures are masked by the source masks, scaled by the sky levels, and stacked. In this case the pixels are the median of the unmasked pixels. Any pixels masked in every exposure are filled in by interpolation from neighbors. Iterated sigma clipping of the stack is used to limit outliers to minimum and maximum values. The stack is then smoothed by variable order functions that observe the amplifier boundaries. This means that higher order functions fit the edges of the CCD and in the interior a lower order behavior is fit. This is done separately for each amplifer in the CCDs. By not smoothing across the amplilfier boundaries this allows amplifier differences in the sky to be included in the sky template. The template is normalized across all CCDs by the average of the means of the CCD templates to maintain the relative difference between CCDs. The template is entered into the calibration library and used for that night's data and for nearby nights where the template exposure list was not adequate for a template from that night. The dark sky template is scaled to match the exposure sky mean and subtracted from each CCD. Figure X illustrates a typical dark sky template.
Figure X: Dark Sky Flat
The dark sky subtraction is based on the systematic sky pattern, i.e. typical of the night. This is after the various gain adjustments and the source detection sky step. Despite these, some sky patterns and amplifier offsets remain. While the most important goal of the instrumental pipeline is to produce photometric uniformity, the visual and digital uniformity of the background is important for the Legacy Survey source detection pipline and sky viewer as well as future archival users.
For Mosaic3 the most critical effect is a strong, rapidly variable bias pattern noise. The variable nature of this noise is such that it is not removed by the overscan and zero bias steps nor the large scale background flattening steps described earlier. Figure X shows the pattern noise in one amplifer of a random exposure. The rapid variation of the bias means it varies during the course of every line. It is different in each amplifier. Therefore, a line oriented algorithm applied separately to each amplifier was required. Also required was that the algorithm not affect the signal from sources; especially the faint sources which are the principle target of the MzLS. Since some amplifiers are affected more than others the MCP allows selecting which amplifiers are corrected. For DR9 processing the choice was simply made to apply the pattern noise removal algorithm to every amplifier.
Figure X: Mosaic3 pattern noise.
Pattern noise in one amplifier of a random exposure. The box is the region shown in figure X at higher resolution.
The challenging aspect of the problem is that no matter how low a threshold above sky is used for source detection there is always light extending beyond the threshold. The ACE source masks are produced with a very low threshold (fractions of the sky sigma, by applying a convolution filter and masking beyond the detection threshold by growing the edges. The complexity of the algorithm described here revolves around excluding source light to separate the pattern noise for modeling and subtracting.
Figure X illustrates some of the steps in the algorithm for a small segment of a line from the example in figures X and X. The first step is to carefully clean the image data of sources and their faintest wings. The object detection masks are expanded along lines by 8 pixels and the masked pixels are replaced by interpolation along the columns (fig. X.2). The masked pixels are then replaced by unmasked data from 32 pixels away in either direction along the line (fig. X.3). The mean and standard deviation of the resulting image is computed and values that are more than 5 standard deviations above the mean and less than 3 standard deviations below the mean are capped a those limits.
The next step is to isolate the background from the pattern noise. The source cleaned image is smoothed by a 64 pixel moving median. Despite the preceding line oriented source cleaning there may still be some weak "lumps" from undetected source light. The median smoothed image is segmented with ACE as if the lumps were sources. The pattern noise is isolated by subtracting the segmented sky and replacing the segmented "sources" by interpolation along lines and then along columns (fig. X.4 which is exaggerated by a factor of 3). In the illustrated example there are no "lumps". The lines in the pattern noise image are smoothed by fitting a cubic spline of 15 segments to form the final pattern noise to be removed (fig. X.5 which is also exaggerated by a factor of 3).
The pattern cleaned result is formed by subtracting the pattern noise image and the ACE segmented sky with the original mean preserved (fig. X.6). This algorithm has two effects. It does a very good job of removing the strong pattern noise and any residual background patterns as demonstrated in figure X. However, for large sources where the underlying pattern noise becomes an extrapolation, there is some weak residual pattern noise and line structure apparent. Because these are bright sources it has little effect on the photometry and is largely esthetic which is well justified by the great improvement of removing the strong pattern noise.
Figure X: Three stages in the Mosaic3 pattern removal.
(Top) Stage before the pattern removal algorithm. This is after all other MCP calibration steps. (Middle) Overlay of source mask. The green line indicates the line illustrated in figure X.(Bottom) Final result of pattern removal.
Figure X: Stages of the pattern removal algorithm.
Various transformations of the line indicated by the green line in figure X are shown. (1) The starting data. (2) After column interpolation of the source mask. (3) After replacement of sources by nearby data. (4) The extracted pattern noise after ACE cleaning exaggerated by a factor of 3. (5) The spline fit of the pattern noise exaggerated by a factor of 3. (6) The final pattern noise subtracted data.
4.6. Bad Data
Bad pixels, regions, and columns in each CCD were determined prior to the start of the MzLS. These static CCD defects are added to the data quality mask produced for each exposure. These bad pixels are replaced by interpolation. Also identified from the raw data values are saturated pixels and pixels within a radius of 2 pixels (center to center). These are not replaced by interpolation in MzLS. The saturation levels for each amplifier are externally determined and referenced through the calibration library for each epoch. The MzLS saturation values were updated during the course of the observations.
The pixels affected by overflow and charge transfer, the so-called bleed trails, are identified by a matched filter of 20 pixels in length above the saturated level. Neighbors of bleed trail pixels within a radius of 2 pixels are also identified as affected by bleed trails to catch the roll off at the edges of the trails.
The amplifier images are segmented to build a source catalog and mask, a background image, and identify cosmic rays and streaks. The segmenting and cataloging is performed using the Astronomical Cataloging Environment (ACE [REF]). The source catalog includes a variety of measures including shape and significance related quantities. These quantities are used to identify cosmic rays and streaks. The pixels thus identified are added to the data quality mask with codes 5 and 8 respectively.
Returning to the point after the initial star and prior to the final amp flat calibration, the amplifiers are checked for excessive saturation. Occasionally the MzLS exposures had sky brightness approaching the defined saturation levels in some amplifiers. To avoid problems with large masked areas in amplifiers the saturation flag in the data quality mask is cleared and data quality keywords in the data headers are used to carry the information that the exposure, at least in some amplifiers, should be considered suspect.
The Mosaic3 CCDs are thick and extremely susceptible to particle events. If the seeing, measured by ACE as a FWHM over the set of detections heuristically labeled as stars, is greater than four pixels then the cataloged sources that fit a multi-parameter filter are identified as cosmic rays. One novel parameter is the ratio of the number of pixels defining the edge of a detected source to those interior. This is particularly effective for "snakey" cosmic rays. In addition, a DESDM/LSST cosmic ray algorithm (immask [REF]) is applied since this might catch more cosmic rays.
Streaks are identified by an algorithm tuned to finding long linear shapes in the source catalog parameters. Care needs to be taken to not identify edge-on disk galaxies as streaks. Another challenge is that long streaks either blend sources in the field or break up into short pieces during the threshold segmentation process. To a lesser extent this segmenation blending and break up applies to the cosmic rays.
These transient detection algorithms are designed to be conservative to avoid misidentifying true celestial sources. Therefore, many streaks and cosmic rays are missed and not labeled in the data quality mask. The later MzLS process of building coadded brick tiles identifies additional transients through comparison to all data overlapping a given point ([REF]). The MCP provides a similar capability but it is not applied because the repeat observations are scattered through the survey and are not available with the night-by-night processing orchestration used for the MzLS.
The final product of the ACE step are various data quality quantities added to the data headers.
4.7. Data Products
The last steps of the MCP are to create the final data products, archive them, and pass them on to the next stage LS pipeline. The data products are all compressed FITS files ([REF]). When uncompressed they are FITS multi-extension files (MEF [REF]) with a data-less primary header of global keywords applying to all extensions. There are four extensions, one for each CCD. Table X identifies the different MCP data products for the MzLS, table X defines the data product filenames as stored in the archive and referenced in the LS data releases, and table X specifies the data quality codes in the data quality masks.
A data product not discussed in other sections, but required for the MzLS, is an inverse variance map known as a weight map. This is used for the The Tractor analysis ([REF]). The weight map gives the statistical uncertainties of each pixel value. Because of the many complex operations performed in transforming the raw exposures to the final instrumentally calibrated data products, the MCP does not attempt to propagate the uncertainies from the raw data using estimated readout noise and gain. Instead it directly estimates the final variance in the sky pixels and leaves the extension to Poisson uncertainties at other signal levels to later MzLS processing ([REF]).
The inverse variance estimates for the sky are a product of an ACE segmentation step on the final flux image. ACE computes the variance from the statistics in 64x64 blocks as the standard deviation about the mode of the bad/source masked pixels, i.e. the sky variance. When blocks are significantly compromised by sources the variance of the block is estimated by interpolation from nearby blocks. The full sized weight map is obtained by spatially interpolating from the estimated weights in the block sampling.
Table X: MCP Data Products
The table shows the data products and how they are categorized. The OBSTYPE column is the observation/calibration type where "object" means an on-sky science exposure. The PROCTYPE and PRODTYPE columns categorize a data product by processing type and product type column. The CODE column is a identifier in the archive filenames reflecting the OBSTYPE, PROCTYPE, and PRODTYPE with single character codes in that order. The MCP codes for the data products are shown in table X and the character code convention and archive filename schema is described in [REF].
OBSTYPE PROCTYPE PRODTYPE CODE DESCRIPTION object InstCal image ooi The instrumentally corrected science flux data with astrometric solution for each exposure. This includes a uniform gain and background sky matching the mean of the raw counts. The pixel values are in units of photoelectrons per second based based on the measured gains and exposure times. object InstCal dqmask ood Data quality mask of integer bad pixel codes as defined in table X. object InstCal wtmap oow Weight map (inverse variance) for sky pixels sampled and smoothed over a scale of 64 pixels. zero MasterCal image zci Master zero bias from a calibration sequence. dome flat MasterCal image fci Master dome flat field from a calibration sequence. fringe MasterCal image hci Fringe template from a night with at least 50 good exposures. illum MasterCal image oci Dark sky template from a night with at least 50 good exposures. Table X: Data Product Files
The data products are all compressed FITS files ([REF]). When uncompressed they are FITS multi-extension files (MEF [REF]) with a data-less primary header of global keywords applying to all extensions. There are four extensions, one for each CCD. The files in the archive and as referenced in the LS data products have names with the following structure. The naming convention for the archived data products is described in [REF].
[instrument]_[yymmdd_hhmmss]_[code]_[filter]_[version].fits.fz
Field Description [instrument] For the MzLS this is always "k4m" for Kitt Peak + Mayall 4-m Telescope + Mosaic3 camera. [yymmdd_hhmmss] The UT date/time of the exposure. [code] The data product code as defined in table X. [filter] The abbreviated filter band which is g or r for the MzLS. This applies to all data products except zero bias for which this field is missing. [version] The version of the data product which is distinct from the pipeline version. For the LS DR9 release this is mostly "ls9". Table X: Data quality codes
The data quality data product uses the integer codes given here for each pixel. The code values are in priority order. For example, if a pixel might lie in a cosmic ray but if it was considered a CCD defect it would code 1 rather than 5.
0 = scientifically good pixels 1 = bad pixels 2 = below sky "holes" 3 = saturated pixels 4 = bleed trail and edge bleed pixel 5 = cosmic rays 8 = streaks 5. BCP: The Beijing-Arizona Sky Survey (BASS)
The Bejing-Arizona Sky Survey (BASS [REF]) of the northern sky was performed with the University of Arizona Bok Telescope and the 90prime camera on Kitt Peak. The collaboration with the Legacy Survey consisted of an exchange of the MzLS z-band data with the BASS g-band and r-band data taken using the same sky coverage and observing strategies. The two survey programs developed independent pipelines for the g, r, and z datasets. The LS instrumental pipeline for the BASS data is called the BASS/Bok Community Pipeline or BCP.
The 90prime camera provides similar 4 CCD / 16 amplifier raw data to the Mosaic3 camera with the principle difference being a larger pixel scale. This means much of the MCP and BCP share code and a similar workflow.
The raw exposures were electronically transferred from the Bok Telescope to NOAO and archived in the NOAO Science Archive. They were then instrumentally calibrated at NOAO as the first step of producing the LS data set for DESI ([REF]) targeting and community research. The calibrated exposures are archived in the science archive.
The BCP is a high performance, data parallel, distributed pipeline running on a dedicated cluster of machines at NOAO. In this section the various calibration methods used to remove the instrumental signature of the camera are described. Additional calibrations were also performed by the Legacy Surveys in producing the LS data releases [REF]).
The BCP draws data to be processed from the archive for each night. The datasets are divided into data to be ignored (i.e. bad), zero bias exposures, dome flat exposures, and science exposures. Dark exposures are ignored. The dome flat and science exposures are divided by filter and processing proceeds independently and in parallel by filter.
The BCP checks the initial data for various conditions and skips those which preclude processing. These include missing required keywords (e.g. exposure time and filter). Other exposures that are excluded are those listed in a "bad exposure" list and those which are identified as test, junk, focus, skyflat and twilight by the observer in their title (OBJECT keyword). Exposures taken in filters other than bokr and g are ignored.
The BCP functionality, calibrations, correction, and products are presented in the following sections. Section 5.1 describes the creation of master dome calibrators that precedes science processing. The core of instrumental calibration, flux calibration, is described in section 5.2. Then astrometric calibration mapping pixels to celestial coordinates follows in section 5.3. Determining a magnitude zero is next in section 5.4. Flux calibration doesn't necessarily produce a uniform background so section 5.5 describes handling of the sky background. There are various sources of unusable or scientifically suspect pixels which are identified and flagged as described in section 5.6. The last section, 5.7, describes the final data products including creation of an inverse variation product for estimating pixel value uncertainties. This section organization does not represent the orchestration of the pipeline workflow where many of the steps are intertwined.
5.1. Master Dome Calibrators
As is typical of CCD cameras, the BASS observing nights included taking sequences of exposures for initial zero bias and flat field calibration. Zero bias exposures are taken with the shutter closed and a zero exposure time. Flat field exposures are taken of an illuminated "white" screen in a darkened dome. To minimize noise contributions in these calibrators, sequences of consecutive exposures with zero or optimal exposure times are taken and averaged together.
The nightly dome calibration sequences are processed by the BCP first and in the order of zero bias then dome flats. The sequences are processed into master calibrations for each night Every exposure in a calibration sequence is corrected for crosstalk, bad pixels, and electronic (overscan) bias as described in section 5.2. The 90prime data also shows a coherent trend with column which can be corrected by post-readout lines. These line postscans were not present in all survey exposures. Dome flats exposures also have the chronologically nearest master zero which, because of the way the BASSLS was processed, is typically from the same night.
After the instrumental corrections are applied to the individual exposures of the sequences they are combined into master calibrations. The combination is a pixel-by-pixel average with a number of low and high values (nlow and nhigh) excluded. The number excluded is given by
where n is the number in the sequence and integer division, rounding down to an integer, is used. The typical BASS calibration sequences were 10 exposures which means the two highest and one lowest values were excluded and the remaining 7 averaged. Figure X shows an typical master zero and g/r-band dome flat from the night of February 20, 2018.
Figure X: Typical Bias and Dome Flat Master Calibrations from 2018-02-20.
5.2. Flux Calibrations
The primary purpose of an instrumental pipeline is to transform the raw instrumental pixel values to linear values with uniform proportionality to the flux from the sky. The first steps are to make the transformed values zero when there is no light and then make the proportionlity with flux (known as the gain) be the same for all pixels. Pixels that can't be transformed to scientifically useful values are identified and recorded in a data quality mask as discussed in section 5.6.
The first operation is a crosstalk correction to the raw pixel values. The 90prime camera has 16 readout amplifer channels which affect each other electronically in a way that depends on the number of electrons being measured. The correction scales the signal from one amplifier and subtracts it from another amplifier for each pixel measured by the controller electronics at the same time. A proportionality coefficient is determined for each pair of amplifiers that have show a measurable crosstalk effect. The strength of the crosstalk varied with time and coefficients were determined for different epochs. Figure X illustrates the magnitude of the crosstalk when an amplifier measure 50,000 counts. The ordinate is the amplifier number (with the bottom bars indicating the CCD groupings) and the absissa is the count correction of the affected amplifier. The white points are crosstalk from an amplifier on the same CCD and the green are from other CCDs.
Figure X: 90prime Crosstalk @ 50K
The electronic bias for each amplifier is determined for each line in each exposure from the overscan pixels (amplifier measurements immediately following reading a CCD line) -- i.e. a line-by-line overscan calibration. The overscan pixels for each line are averaged with the high and low pixels excluded. For a small number of exposures in the survey a line-by-line correction produced a bad result. For those cases the line-by-line method was replaced by a high order legendre polynomial fit to all the averaged line overscan values and fit values for each line are used to subtract the electronic bias.
As mentioned earlier, the 90prime camera can have a variable column bias. In the 2015 to 2017 data 20 postscan lines were included in the raw data. The central 18 lines (which follow all the CCD lines) are averaged and smoothed by a cubic spline of 5 segments. This smoothed postscan line is then subtracted from every line. These overscan and postscan steps apply to all raw exposures including the zero bias and dome flat exposures used to produce the master calibrations.
The remaining zero level instrumental bias signature is removed by subtracting the chronologically nearest master zero bias as described in section 5.1.
These zero signal calibrations are followed by application of the master dome flat field for the night as a first step to normalize the amplifier gains by reference to a white screen. A predetermined mean gain for the camera (fixed at 1.30 for the whole of the BASSLS) and the recorded exposure time are used to convert the pixel values from instrumental units to estimated photoelectrons per second. The final BCP data products are in these units. A header keyword, GAINNORM, is recorded to allow conversion back to instrumental counts as needed.
An initial illumination correction is applied as a first correction for the difference between the dome white screen illumination and the sky. The illumination correction is produced from an unregistered smoothed stack of exposures in a first pass calibration with had no other flat field corrections beyond the dome flat. These were produced up to 2016-07-12. Beyond that further nights used this last initial illumination since a subsequent step corrects for nightly variations. Figure X shows the illumination correction from 2016-07-12.
Figure X: 90prime Illumination and Photometric Flat
After the initial, and largely static illumination correction, a photometric gain adjustment is applied to each amp. This is a single number derived from matching instrumental fluxes from the night to PS1 photometry. This is referred to here as the "amp flat" correction.
The amp flat correction is applied in an incremental manner. Since BASSLS was a primary observatory program observing night after night in seasons, the correction from the preceding night is applied during initial flat fielding. This correction, which is different for each amplifier, involves measuring the sky brightness with a masked and sigma clipped mean and evaluating an amp flat function (described below). This stage of calibration with the amp flat correction of a recent preceding night is used for the following steps of zero point measurement, astrometric calibration, pupil pattern and sky pattern removal. This is then what is used to update the amp flat function for the current night and apply the updated gain correction to the amplifiers. The amp flat corrections values are recorded in the data headers.
The amp flat functions are derived as follows. After all corrections have been applied the stars in the exposure are cataloged and matched to the Pan-STARRS reference catalog. Magnitude zero points for the exposure and the differences from the zero points for each amplifier are computed. The sky brightness is also measured but, unlike the Mosaic3 CP, the zero points do not show a dependence on this parameter. On a night with at least 50 exposures a constant is fit to the zero point magnitude differences for each amplifier at all sky brightnesses; i.e. an average. There are various sanity checks and exclusion of poor measurements. The fitting includes 5 iterations of 2 sigma clipping. The amp flat function is then applied as another gain correction. The fit coefficient (the average) is merged with the correction applied to the data from the previous night to produce a new amp flat correction for subsequent data.
Figure X show an example from the night of 2016-02-11 for CCD1 in the r-band. The points are from individual exposures and the solid line is the night's ampflat values relative to the initial values applied from an earlier night.
Figure X: 90prime Ampflat 
A final photometric (star) flat field calibration is applied. This is derived from repeat imaging of stars in BASSLS using the calibration procedure of Schlafly ([REF+2012]). One photometric flat per filter was derived for each observing year. To obtain this the BASS data was processed twice; first without this photometric flat field and then a second time with the calibration derived from the stars in the first pass results. The distinctions between the star flat and the amp flat calibrations are: 1) the amp flat is a single gain for each amplifier while the star flat provides corrections at a higher (though not individual pixel) scale, 2) the amp flat is measured by night as opposed to by season, and 3) the amp flat is based on an external photometric source (PS1) while the star flat is derived from the internal consistency of the instrumental photometry.
The photometric flat for 2018 is shown in figure X. A feature requiring an explanation is the bright spot in the lower right CCD. This is a region affected by scratches in the dewar window resulting in scattering and a different point spread function from the rest of the exposure.
5.3. Astrometry
ACE source catalogs for each CCD are matched against the Gaia astrometric reference catalog ([REF]). During the processing of BASSLS (for DR9) Gaia DR1 was mostly used. The source catalogs include celestial coordinates evaluated from the instrumental pixel centroids, a library astrometric distortion map, and the telescope supplied pointing. The matching determines a pointing offset, a scale change, and any small rotation. The latter two effects are the result of differential refraction. A vaild match requires a fraction of the source and reference coordinates to match. When a match is found a catalog of measured pixel coordinates and Gaia celestial coordinates is created.
There are a number of reasons the matching may fail with the most common one being low transparency producing few or no stars. During this and subsequent steps a failure results in a header keyword indicating the failure. The post-BCP BASSLS processing ignores the exposures that failed the BCP astrometric solution.
A world coordinate system (WCS) function for each CCD is fit to the matched catalog sources. The function used is a distorted zenithal project (ZPX [REF]) which includes a model of the optical pincushion distortion with remaining distortions fit by bi-quartic polynomials. The quadrature rms of the WCS relative to the Gaia coordinates is required to be below a threshold value. If any CCD fails the threshold then the astrometric solution is considered to have failed.
A successful solution adds the WCS function parameters and some diagnotistic header keywords (ASTMCAT, WCSCAL, WCSDRA, WCSDDEC, WCSRMS) to the image data and updates the source catalog celestial coordinates. For consistency across the Legacy Survey the fitted ZPX WCS function is recast to a distorted tangent plane WCS (TNX [REF]) in the image headers.
There is one unfortunate feature of the 90prime camera affecting the astrometry. The camera shows non-negligible charge transfer efficiency (CTE) in the serial readout that leads to point spread functions with centroids shifted in opposite directions between left and right amplifiers. Implementing a complex modeling and deconvolution algorithm was deemed out of scope for the BCP/BASSLS. Using amplifier independent astrometric solutions rather than a single solution over the four amplifiers of a CCD was also deemed to require significant software challenges for the later stages of the BCP and LS processing. A pragmatic solution, which greatly minized the effect on the astrometric solution, was to digitally shift the image data by a small amount. Since it is a relative effect between the two amplifiers only the right amplifiers are rebinned using a high quality sinc interpolator. While there was some evidence of a variation in the CTE with time the zeroth order correction of a fixed shift per CCD was used for the BASSLS processing. The shifts were 0.36 pixelin CCD1 and 0.16 in the other CCDs.
Figure X show the astrometric errors in declination between the continuous WCS astrometric solution and the Gaia coordinates averaged in 64 pixel slices along the readout axis for the 4 CCDs. The distortion function attempts to converge for both left and right amplifiers but can't handle the discontinuity near the boundary which causes the jump seen in the figure. Figure X shows a 2D visualization of the astrometric errors averaged over all amplifiers in 64x64 blocks. The middle column shows this error pattern with and without the digital shift corrections. With the corrections the astrometry shows minimal patterns at a level of 10s of milli-arcseconds.
Figure X: Centroid Shift Correction
The DEC differences between the Gaia reference coordinates and the coordinates from the astrometric solutions applied to the measured instrumental coordinates as a function of CCD column. The measurements and solutions are from all exposured on the night of 2016-03-01 which had the 1/3 pixel readout error. Each point is an average over 64 lines. The discontinuity arises from fitting a single astrometric function over the whole CCD which cannot then respond to the abrupt 1/3 pixel offset boundary. The plots illustrate the magnitude of the error if not fixed and that it is similar in all four CCDs.
Figure X: Astrometric Statistics
The differences between the Gaia reference coordinates and the coordinates from the astrometric solutions applied to the measured instrumental coordinates are visualized as averages in 64x64 regions. The measurements and solutions are from all r-band exposures on the night of 2016-02-11.
The three columns are for (left) errors in RA, (middle) errors in DEC, and (right) RMS (quadrature combination of RA and DEC standard deviations). The two lines are for (top) without the compensation for the CTE effect (bottom) with the correction. The grayscale for the left and middle columns are +/- 100 marcsec and the right panel is 100 to 800 marcsec. Note this combines all CCDs from all exposures. Separating by CCDs shows patterns which are very similar to those shown (e.g. see figure X).
Note that RA varies vertically and DEC horizontally so that the horizontal CTE problem and its minimization is visible in the middle column. The amplitude of the jump is about 75 marcsec but, as seen in fig. X, the magnitude of the CTE shift varies with CCD. The CTE simple shift correction does well as seen in the lower middle panel. There are no other patterns though the RMS is larger than for Mosaic3; in part because of larger pixels.
5.4. Photometric Characterization
A photometric characterization, meaning a single filter magnitude zero point, is performed using PS1 provided the WCS solution was successful. The updated source catalog is matched to the PS1 sources, which is not hard given a successful astrometric calibration. However if this fails (when the astrometric solution appeared successful but was not actually correct), then the photometric and astrometric solutions are marked as failed in the data headers. A magnitude zero point is determined by a iterated sigma clipped average of the instrumental to reference magnitude differences. The zero point and various associated photometric quantities are recorded in the image headers. A photometric catalog of detected sources with zero point calibrated magnitudes is also produced for the determination of amp flat corrections based on the night's observations.
The BASSLS post-BCP processing does its own photometric calibrations based on PS1 independent of the BCP characterization. However, the BCP photometric catalogs for each exposure of the night are used to determine a new amp flat correction for each amplifier as described earlier.
5.5. Sky Background Uniformity
The goal of these steps is to produce a constant background by adding or subtracting counts. There are two primary reasons for this; making it easier for later photometry algorithms to measure background and for visual evaluation. The latter is more than just esthetic -- the eye is very good at finding systematic problems in displays of the image, but background gradients make evaluations less sensitive. Some instrumental pipelines go so far as removing background to a zero level. The BCP maintains the original mean sky level as a potentially useful characteristic of the exposure.
There are two challenges in estimating the sky background. One is that, despite the various gain calibrations, there may still be variations caused by residual gain errors. Disentangling multiplicative and additive effects ultimately requires an assumption that one is negligible compared to the other. We assume the various steps described for gain uniformity, particulary the photometrically derived flat fields, are sufficient at whatever scale the background is to be made uniform. The second problem is with very large scale, low surface brightness light from astronomical sources such as large galaxies and inter-galaxy cluster light. This is guided by the science goals. For the BASSLS the goal is the distant extra-galactic population where sources are small. Studies of larger scale, low surface brightness features can be done with targeted re-reductions for fields of special interest.
The ACE mini-sky images are used. These were created for each amplifier to preserve differences between them. These are then put together to create an image of the mosaic field. This full field mini-sky image is used to remove background patterns (gradients across the field and amplifier level differences left after photometric flat fielding) by subtracting the mean over the field and then magnifying, by linear interpolation, to the regions and size of the CCD images and subtracting.
The next steps involve unregistered stacks of the exposures from a night to make calibrations reflecting remaining systematic patterns. In addition to requiring a minimum number of exposures, an analysis is done on the reduced sky files of the field. The analysis looks at the exposure times, amount of saturation, sky brightness, magnitude zero points, and rms of the sky images about the mean. These are iterated to remove outliers. The end result is a list of exposures, called the template exposure list, which are typical good exposures in terms of sky signal in non-cloudy conditions.
If the template exposure list from the night is sufficiently large a fringe template image is constructed. This is only done for the r-band observations. Sources are masked by the ACE object detection masks. The masked images are scaled to a common mean sky level and combined by a clipped average. The clipping is +/- 3 times the difference between the median and the 10% percentile (see pclip algorithm in [REF]). Iterated sigma clipping of the stack is used to clip outliers to a minimum and maximum value. The amp differences are removed by averaging 500x500 regions nearest the CCD center in each amplifier and subtracting them. The fringe pattern is obtained by subtracting a moving median with a 200 pixel square window. Pixels which are masked in all the exposures are set to zero in the fringe template. The final fringe template is entered into the calibration library and used subsequently by the chronological proximity of observations to the night. Therefore, nights in which a fringe template is generated use the template generated from the its own observations and nights without sufficient good data to create a template use the closest available night's template.
The fringe template is subtracted from each exposure by masking pixels in its source mask and then fitting an amplitude to match the template to the unmasked pixels. A 3x3 box average filter is used to reduce the noise in the fitting and the statistics are weighted by the template values; i.e. the departure of the fringing from zero carries greater weight. The template is scaled by the computed amplitude and subtracted from the image data to make the fringe correction. Figure X.L is an example fringe template from the end of the survey.
Figure X: 90prime r-band Fringe and Dark Sky Templates
The next steps are similar to those for making a fringe template and removing the fringing. This is done for both filters. If the template exposure list from the night is sufficiently large a dark sky template is constructed. The exposures are masked by the object detection masks, scaled by the sky levels, and stacked. In this case the pixels are the median of the unmasked pixels. Any pixels masked in every exposure are filled in by interpolation from neighbors. Iterated sigma clipping of the stack is used to clip outliers to a minimum and maximum value. The stack is then smoothed by variable order functions that observe the amplifier boundaries. This means that higher order functions fit the edges of the CCD and in the interior lower order behavior is fit and this is done separately for each amplifer in the CCDs. By not smoothing across the amplilfier boundaries this allows amplifier differences in the sky to be included in the sky template. The template is normalized across all CCDs by the average of the means of the CCD templates to maintain relative difference between CCDs. The template is entered into the calibration library and used for that night's data and for nearby nights where the template exposure list was not adequate for a template from that night.
A dark sky template is scaled to match the exposure sky mean and subtracted from each CCD. The dark sky subtraction is based on the systematic sky pattern, i.e. typical of the night. This is after the various gain adjustments and the source detection sky step. Despite these some sky level patterns remain. While the most important goal of the instrumental pipeline is to produce photometric uniformity, the visual and digital uniformity of the background of BASSLS and archival users. In particular, for the Legacy Survey source detection pipleine and the sky viewer as well as future archival users. Figure X.R is an example r-band dark sky template from the same night's data as the fringe template.
5.6. Bad Data
The BCP identifies bad or scientifically suspect pixels as described here. Pixels which are not identified by criteria in this section are considered good.
Detector bad pixels, regions, and columns in each CCD were identified prior to the start of the BASSLS processing. These static CCD defects are transferred to the data quality mask produced for each exposure. The identified pixels are replaced by interpolation from nearby pixels. Also identified from the raw pixel values are saturated pixels and pixels within a radius of 2 pixels (center to center) of those pixels. The saturation levels for each amplifier are externally determined and referenced through the calibration library for each epoch. The BASSLS saturation values were fixed at 55,000 for all amplifiers over the course of the survey processing.
The pixels affected by overflow and charge transfer, the so-called bleed trails, are identified by a filter of 20 pixels column length above the saturated level. Neighbors of bleed trail pixels within a radius of 2 pixels are also identified as affected by bleed trails.
A check is made for cases with exposures pushing sky above saturation by counting the number of pixels flagged as saturated. To avoid problems with large masked areas in amplifiers the saturation flag in the data quality mask is cleared and data quality keywords in the data headers are used to carry the information that the exposure, at least in some amplifiers, should be considered suspect. Saturating the sky was rare in the BASSLS.
The 90prime camera digital values can wraparound to low values for saturated sources. This is heuristically detected as pixel values below the sky level. The heuristic is to compute the median and standard deviation of all unmasked pixels. The computation is iterated (5 times) with sigma clipping (3 sigma below and 2 sigma above). Pixels below 20 standard deviations of the median are flagged as saturated in the data quality mask and the image pixels have 65535 added to their values. Pixels near the identified wraparounds are sometimes observed to also be depressed, though between the sky threshold and saturation, causing dark rings in the transition from the wings of the bright starts to the saturated cores. To address this pixels within 3 pixels (measured between pixels centers) are also treated as wraparound in the same way except the image values are set to 65535 instead of being added.
Another step dealing with below sky regions not due to saturated sources, called "holes", are identified by creating ACE catalogs and object masks. The object masks for "sources" with negative fluxes (i.e. below background) are dilated and flagged in the data quality mask. This complements the wraparound algorithm and is more sensitive to pixels slightly below sky. These differ in that the holes are flagged but the flux image is not altered. These are rare and are more often associated with other bad pixel regions.
ACE catalogs includes a variety of measures including shape related quantities. The shape quantities are used to identify cosmic rays and streaks. The pixels thus identified are added to the data quality mask with codes 5 and 8 respectively.
If the seeing, measured by ACE as a FWHM over the set of detections heuristically labeled as stars, is greater than four pixels then cataloged sources that fit a multi-parameter filter are identified as cosmic rays. One novel parameter is the ratio of the number of pixels defining the edge of a detected source to those interior. This is particularly effective for "snakey" cosmic rays.
Streaks are also identified in ACE catalogs by an algorithm tuned to finding long linear shapes. Care needs to be taken to not identify edge-on disk galaxies as streaks. The other challenge is that long streaks either blend with other sources in the field or break up into short pieces during the threshold segmentation.
These various transient detection algorithms are designed to be conservative to avoid misidentifying true celestial sources. Therefore, many streaks and cosmic rays are missed and not labeled in the data quality mask. The later BASSLS process of building coadded brick tiles also adds transient identifications through comparison to all data overlapping a given point ([REF]).
5.7. Data Products
The last steps of the BCP are to create the final data products, archive them, and pass them on to the next stage LS pipeline. The data products are all compressed FITS files ([REF]). When uncompressed they are FITS multi-extension files (MEF [REF]) with a data-less primary header of global keywords applying to all extensions. There are four extensions, one for each CCD. Table X identifies the different BCP data products for the BASSLS, table X defines the data product filenames as stored in the archive and referenced in the LS data releases, and table X specifies the data quality codes in the data quality masks.
A data product not discussed in other sections, but required for the BASSLS, is an inverse variance map known as a weight map. This is used for the The Tractor analysis ([REF]). The weight map gives the statistical uncertainties of each pixel value. Because of the many complex operations performed in transforming the raw exposures to the final instrumentally calibrated data products, the BCP does not attempt to propagate the uncertainies from the raw data using estimated readout noise and gain. Instead it directly estimates the final variance in the sky pixels and leaves the extension to Poisson uncertainties at other signal levels to later BASSLS processing ([REF]).
The inverse variance estimates for the sky are a product of an ACE segmentation step on the final flux image. ACE computes the variance from the statistics in 64x64 blocks as the standard deviation about the mode of the bad/source masked pixels, i.e. the sky variance. When blocks are significantly compromised by sources the variance of the block is estimated by interpolation from nearby blocks. The full sized weight map is obtained by spatially interpolating from the estimated weights in the block sampling.
Table X: BCP Data Products
The table shows the data products and how they are categorized. The OBSTYPE column is the observation/calibration type where "object" means an on-sky science exposure. The PROCTYPE and PRODTYPE columns categorize a data product by processing type and product type column. The CODE column is a identifier in the archive filenames reflecting the OBSTYPE, PROCTYPE, and PRODTYPE with single character codes in that order. The BCP codes for the data products are shown in table X and the character code convention and archive filename schema is described in [REF].
OBSTYPE PROCTYPE PRODTYPE CODE DESCRIPTION object InstCal image ooi The instrumentally corrected science flux data with astrometric solution for each exposure. This includes a uniform gain and background sky matching the mean of the raw counts. The pixel values are in units of photoelectrons per second based based on the measured gains and exposure times. object InstCal dqmask ood Data quality mask of integer bad pixel codes as defined in table X. object InstCal wtmap oow Weight map (inverse variance) for sky pixels sampled and smoothed over a scale of 64 pixels. zero MasterCal image zci Master zero bias from a calibration sequence. dome flat MasterCal image fci Master dome flat field from a calibration sequence. fringe MasterCal image hci Fringe template from a night with at least 50 good exposures. illum MasterCal image oci Dark sky template from a night with at least 50 good exposures. Table X: Data Product Files
The data products are all compressed FITS files ([REF]). When uncompressed they are FITS multi-extension files (MEF [REF]) with a data-less primary header of global keywords applying to all extensions. There are four extensions, one for each CCD. The files in the archive and as referenced in the LS data products have names with the following structure. The naming convention for the archived data products is described in [REF].
[instrument]_[yymmdd_hhmmss]_[code]_[filter]_[version].fits.fz
Field Description [instrument] For the BASSLS this is always "ksb" for Kitt Peak + Steward Observatory + Bok/90prime. [yymmdd_hhmmss] The UT date/time of the exposure. [code] The data product code as defined in table X. [filter] The abbreviated filter band which is g or r for the BASSLS. This applies to all data products except zero bias for which this field is missing. [version] The version of the data product which is distinct from the pipeline version. For the LS DR9 release this is mostly "ls9". Table X: Data quality codes
The data quality data product uses the integer codes given here for each pixel. The code values are in priority order. For example, if a pixel might lie in a cosmic ray but if it was considered a CCD defect it would code 1 rather than 5.
0 = scientifically good pixels 1 = bad pixels 2 = below sky "holes" 3 = saturated pixels 4 = bleed trail and edge bleed pixel 5 = cosmic rays 8 = streaks