Emily Lakdawalla
Page 30
et al. (2012). Also useful is Alexander and Deen (2015). Two peer-reviewed articles were in preparation as this book was being written: Bell et al. (2017) and Malin et al. (2017). Because Mastcam
shares its electronics, detector, and focal mechanism design with MAHLI, the Edgett et al. (2012)
MAHLI paper is also informative.
7.2 Mastcam 239
away from the cameras, at a spot on the ground 2 meters away from the rover. The detector
is capable of capturing 720p high definition video (1280-by-720 pixels) at a rate of 5
frames per second. Further facts are summarized in Table 7.2.
The Mastcams have the same detectors as MAHLI and MARDI and use the same focus
mechanism as MAHLI. The detector is a Kodak KAI-2020CM Charge-Coupled Device
(CCD), which is 1640 pixels wide by 1200 pixels high. The sides and corners of the
images are partly occluded by the baffle and are affected by vignetting. The vignetting
exists because the filter wheels, and specifically the shapes of their openings, were built before the descope of zoom capability, at a time when Mastcam only planned to produce
1200-by-1200-pixel subframes. Most images taken for science purposes crop away the
sides to an image width of 1344 or 1200 pixels, operationally called a “full frame”
(Figure 7.3). On sol 1589, the Mastcam team switched to using 1328-by-1184-pixel “full frames” for more efficient memory management. 2 The original full-frame image size used 12.3 blocks in flash memory; the slightly smaller subframe uses just under 12 blocks at
virtually no cost to the usefulness of the image.
Figure 7.3. Size of the left Mastcam frame and common subframe areas. Mastcam image 320ML0010520330107781E01. NASA/JPL-Caltech/MSSS/Emily Lakdawalla.
2 Michael Malin, personal communication, email dated April 14, 2017
240 Curiosity’s Science Cameras
7.2.1.2 Color imaging
Unlike most space cameras, the Mastcams, MAHLI, and MARDI take natural color
images like consumer digital cameras. Each pixel is covered with a red, green, or blue
filter in a Bayer pattern. A Bayer pattern is a checkerboard of colored pixels; in every
2-by-2 array of pixels, two corner pixels are covered by green filters, one is covered by a red filter, and one by a blue filter. Color comes from interpolating among the pixels to
generate complete red, green, and blue images. Interpolation can happen onboard the
spacecraft or on Earth.
Each Mastcam eye is equipped with an 8-position filter wheel. It may seem odd to add
a filter wheel to a camera that already has color filters over its detector, but fortunately for spectroscopists, the Bayer color filters on the Mastcam detectors are “leaky” in infrared
wavelengths. During normal color imaging, a broadband filter blocks these infrared wave-
lengths (Figure 7.4). But the Mastcams can operate like other filter-wheel-equipped space cameras in the near-infrared with six narrowband science filters in each eye, used for
spectroscopic imaging (Figure 7.5). The science filters were distributed between the two cameras so that, if one camera fails, the other will still be able to accomplish some of the science objectives. Three of the filters are essentially identical between the two eyes, and three differ, so a total of nine distinct science filters is available for multispectral imaging.
Each eye’s filter wheel also has one filter with a neutral-density coating that blocks most light and permits the Mastcams to directly image the Sun through a blue (right eye) and
infrared (left eye) filter.
Figure 7.4. Mastcam detector Bayer filter bandpasses without and with the “clear” infrared cutoff filter. Dark lines show the quantum efficiency of the optics and detector; at wavelengths beyond about 850 nanometers, all three Bayer filters allow an equal amount of light to pass.
Brighter lines show the normalized transmission of the three Bayer filters with the clear filter in the optical path, which allows only visible wavelengths (420–690 nanometers) to pass.
Data courtesy Jim Bell.
7.2 Mastcam 241
Figure 7.5. Mastcam narrowband filter transmission. Data courtesy Jim Bell.
The narrowband filters are usually named by their filter wheel positions (L0, L1,
etc...) or referred to using the wavelengths that were requested from the Mastcam filter
supplier (440, 525, 675, etc...), but their actual center wavelengths are slightly different from those values. The as-built center wavelengths of the filters on the cameras on Mars
are listed in Table 7.3.
Table 7.3. Mastcam spectral filters and bandpasses as built. Data from Bell et al. (2012 ).
Filter
Left Eye Wavelength±
Right Eye Wavelength ±
Position
Bandwidth (nm)
Nickname
Bandwidth (nm)
Nickname
0
590 ± 88
Clear
575 ± 90
Clear
640 ± 44
Bayer red
638 ± 44
Bayer red
554 ± 38
Bayer green
551 ± 39
Bayer green
495 ± 37
Bayer blue
493 ± 38
Bayer blue
1
527 ± 7
525, green
527 ± 7
525, green
2
445 ± 10
440, blue
447 ± 10
440, blue
3
751 ± 10
750
805 ± 10
800
4
676 ± 10
675, red
908 ± 10
905
5
867 ± 10
865
937 ± 10
935
6
1012 ± 21
1035
1013 ± 21
1035
7
880 ± 10 ND5
880, solar
440 ± 20 ND5
440, solar
As a consequence of the convolution of Bayer and narrowband filters, some narrow-
band images contain less spatial information than others. In particular, an image taken
through L2/R2 (440), L3 (750), R3 (800), L4 (675), or R7 (440 ND) has good signal only
242 Curiosity’s Science Cameras
in one out of every four pixels (the red ones or blue ones), while L1/R1 (525) has signal in only one of every two pixels (the green ones). JPEG-compressing the full-size versions of
these images before transmitting them to Earth would have very strange results. So before
converting the shorter-wavelength narrowband images to JPEG, the camera electronics
throw out data from the relatively unresponsive pixels and do bilinear interpolation to fill in data from the missing pixels. As an example, for the L2/R2 (blue) images, the electronics throw out the data from the red and green pixels and fill in with values interpolated
from the blue pixels. Narrowband filter images that have been JPEG-compressed are
returned to Earth as grayscale, with only the luminance (brightness and darkness) channel;
the chrominance (color variation) information isn’t provided. Because the longer expo-
sures required to take narrowband filter images accentuate the effects of bad pixels and
because the images have intrinsically less spatial information, they tend to look noisier
than the broadband color images.
7.2.1.3 Focus
The Mastcam focal mechanism uses a stepper motor with 16100 discrete motor positions.
To autofocus, a Mastcam starts at a commanded motor position and then takes a set of
images, inc
rementing the motor count by a specified step each time. Usually, the autofocus
images are subframes of the full scene. The camera then JPEG-compresses the photos.
The file size of the photos measures the complexity of the scene; an out-of-focus scene
will be blurrier, so will compress to a smaller file size. The camera considers the motor
count as a function of JPEG file size and fits a parabola to the sizes of the three largest files.
The vertex of the parabola is taken to be the best-focus motor count, and Mastcam moves
the focus to that position and takes one more image.
When a scene has a lot of depth, the autofocus algorithm doesn’t always select the focal
depth that scientists want, so it can be better to specify the focal depth for those observations. To save time when capturing landscape mosaics, the Mastcams can be commanded
to autofocus one frame and then use the same focus setting for subsequent frames that are
expected to be in focus at the same position.
The motor count associated with an in-focus image is a function of the range to the
best-focus features in the image. For the Mastcam-100, the temperature of the instrument
also affects the focus. To determine range from motor count, use the following equations:3
Mastcam 34
-
: range = 363.64 / ( 2427.50 − (motor count) )
Mastcam 100
-
: range = 3322.3
( 3491.9−2 58
. ∗ (instrument temperaturre) − (motor count) ) .
3 Bell et al. (2017)
7.2 Mastcam 243
7.2.1.4 Electronics
Each Mastcam, MAHLI, and MARDI has its own board in the electronics assembly. The
following discussion therefore applies to MARDI and MAHLI as well as each Mastcam.
Each electronics board has a computer, 128 megabytes of SDRAM, and 8 gigabytes of
flash memory for each camera, which can accommodate about 4000 total images. The
electronics assembly determines autofocus and autoexposure parameters and sends this
information to the camera heads. The camera detector captures 12-bit images. After it
acquires an image, a camera head sends the data to the RAM inside the digital electronics
assembly for further processing and storage. For all images, the camera head electronics
create thumbnails 1/8th the size of the originals as they are transferred to the electronics board. (The electronics aren’t capable of downsampling images by any factor other than
8.) Mastcam converts the images from 12- to 8-bit depth to reduce file size. Most com-
monly, the team commands the instrument to use a square-root look-up table to do the
12-to-8-bit conversion. This allots more of the limited 256 values in the 8-bit image to
darker areas, preserving detail in shadows that would otherwise be lost. Images are usually stored raw, without compression (in which case each full-size image takes about 2 megabytes of space, on average).
The main rover computer maintains a list of the files in storage, and copies requested
images to its own memory as commanded before transmitting them to Earth. Thumbnail
images get returned to Earth very soon after acquisition, supplying the Mastcam team a
visual index to the image data collected on the rover. When the rover computer requests an
image from Mastcam, it requests that the image be compressed before transmission, either
losslessly or lossily. The electronics board has a lot of options for lossy compression.
Mastcam can use Bayer interpolation4 to convert pictures to color, then save them in JPEG
format. Usually, the JPEG images are compressed using a method that preserves more
detail in an image’s brightness and darkness (luminance) but downsamples the detail in the
color variation (chrominance) by a factor of two. Such images are referred to as “JPEG
422”, while JPEG-compressed images that preserve full-resolution chrominance informa-
tion are referred to as “JPEG 444”.
Returning space science data in lossy JPEG format is somewhat unusual, although it’s
getting more common as camera detector sizes outstrip our ability to transmit all that data to Earth. Even slight JPEG compression produces large savings in file sizes. A JPEG quality of 90 (measured on a scale from 1, lowest, to 100, highest quality) generally produces images
with less than half the file size of an uncompressed image (Figure 7.6). 5 The team selects less compression (typically JPEG quality 85) for images intended to support science, and
more compression (typically JPEG quality 65) for images taken for documentation
purposes.
The cameras’ large flash memory volume makes it possible to keep raw data onboard and
return science images months or even years after they were originally taken. At times when
the rover is not capable of doing much science (like during holidays, solar conjunctions, or 4 Onboard interpolation uses the Malvar-He-Cutler linear interpolation algorithm
5 Bell et al. (2017)
244 Curiosity’s Science Cameras
Figure 7.6. Relationship between JPEG compression quality and file size for all four Curiosity color cameras. Mastcam-34 images have the largest file sizes because they are usually in focus over most of the image so have high amount of detail, which compresses poorly. MARDI, by contrast, is out of focus, so compresses much more readily. An uncompressed image has 8 bits per pixel.
Compression quality 101 refers to losslessly compressed images. Figure from Bell et al. ( 2017 ), based on analysis by Jason Van Beek and Michael Malin.
anomalies that restrict mobility) but is still capable of sending data to orbiters, daily downlinks can be packed with Mastcam data that has been idling on the rover for months, usually losslessly compressed versions of images that had previously been returned lossily.
7.2.1.5 Artifacts and blemishes
Several types of artifacts can affect the quality of Mastcam images. Some of these are
intrinsic to the camera, some have to do with the way the data are stored or transferred, and some result from how the images are processed either within the camera or on Earth. Each
camera has some (but very few) bad pixels: hot pixels that make bright spots, dead pixels
that make dark spots, and gray pixels that don’t respond as well as others around them to
incoming light. Occasionally, a new hot pixel appears on a camera detector, likely caused
by an energetic particle flying from the MMRTG or from space. One such hot pixel
appeared on the center right of the right Mastcam on sol 392, and had disappeared again
by sol 710. A particularly bright one appeared near the top right of the left Mastcam on sol 834 and has remained ever since (Figure 7.7).
7.2 Mastcam 245
Figure 7.7. How hot pixels, shutter smear, and JPEG compression can reduce Mastcam image quality. Taken as part of a drive direction panorama, this Mastcam image was
returned to Earth with fairly high JPEG compression (quality 65). Mastcam image
1030ML0045010040305530E01. NASA/JPL-Caltech/MSSS/Emily Lakdawalla.
246 Curiosity’s Science Cameras
Images containing particularly bright objects – such as parts of the rover, or hot pixels –
can be affected by shutter smear. The Mastcams have no physical shutters. Instead, once
the camera has exposed the detector for the requested length of time, it shifts the charge
out of the exposed area and into a shielded area called the transfer cell, one line at a time.
All the rest of the lines in the image are shifted upward as each line is moved into the
transfer cell and read out. While this is all happening, the detector is still bei
ng exposed to the scene. As long as the readout of the image happens quickly relative to the exposure
time, it’s hard to notice the effects of shutter smear. But if the scene is bright enough that the exposure time is short – or if the scene contains a very bright pixel – there may be a
vertical bright smear streaking the image, running down from bright pixels. The hot pixel
that appeared on the left Mastcam on sol 834 is bright enough to create such a streak on all left Mastcam images acquired since it appeared (Figure 7.7). Fortunately, the Mastcam CCDs are large enough that, although cosmetically annoying, the blemishes are not harmful to science.
The JPEG compression means that most images have some compression artifacts.
JPEG compression works on 8-by-8-pixel blocks, and the boundaries of those blocks are
often visible. JPEG compression is more effective in places with smooth variations in
color and brightness, but can introduce strange artifacts in areas of high contrast. Areas
where there is high contrast or a lot of variation also tend to be areas of scientific interest –
for instance, in an area of very finely laminated rock layers in alternate Sun and shadow
(Figure 7.7, bottom). Where the compression artifacts make it difficult to interpret the geology, the team can choose to re-transmit the image with less compression, or even
losslessly – as long as the original image is still stored in the camera’s flash memory.
As of early 2017, not quite half of all of the Mastcam data had been returned a second time with lossless compression. At that time, the mission switched to returning all non-time-critical Mastcam science data losslessly, accepting delayed data return in exchange for a
larger proportion of losslessly compressed data and a reduction in the complexity of data
curation.6
7.2.1.6 Calibration target
The Mastcam calibration target is a flight spare of the ones flown on the Mars Exploration
Rovers whose design was modified by the addition of magnets. It is mounted on the rover
deck, 1.2 meters away from the cameras, on top of the box that houses the rover pyro fire
assembly (Figure 7.8).
Unfortunately, because of the location of the hardstop on the azimuth actuator of the
remote sensing mast, it is not possible to image the calibration target through both Mastcam eyes with a single pointing. The mast head has to rotate almost 360° in order to image the