Mastcam-Z

2021-12-23

Mars Helicopter Flights: Caught on Video!

Contributed by Justin Maki, Mark Lemmon, and Jim Bell

Earlier this year the Ingenuity Mars Helicopter became the first aircraft to fly on the surface of another planet.  When the helicopter lifted off of the Martian surface at 7:50 UTC on April 19th, 2021 (Sol 58 of the Perseverance surface mission), it marked the beginning of a new era for space exploration.  With that first flight, the aerial exploration of Mars had begun, occurring almost 45 years after the Viking landing (1976) and 24 years after the Mars Pathfinder Sojourner rover mission (1997).  Humanity could now robotically explore Mars by air.  And Mastcam-Z caught it all on video.

Ingenuity flying across the Martian landscape during flight #4 (Sol 69; April 30th, 2021).

Ingenuity development started back in 2013, and a question that the team was asking at the time was “can we record a video of the helicopter flights with the rover cameras?”.  When the Mastcam-Z instrument was selected by NASA in 2014, that question was answered.  Mastcam-Z inherited a video capability from the Mars Science Laboratory (MSL) Curiosity Mastcam camera design.  The Mastcams were developed on MSL with the MAHLI (Mars Hand Lens Imager) and MARDI (Mars Descent Imager) cameras at Malin Space Science Systems, Inc. (MSSS) in San Diego, CA.  All three of the MSL camera types (“MMM”, for short) shared common base designs.  MSSS also built the Mastcam-Z cameras for Arizona State University, and because of this commonality, the Mastcam-Z cameras have the same video capability that was developed for the MARDI camera.  And while video has been used only sparingly on MSL Mastcam, it was just simpler, more efficient, and more cost effective to keep the video capability in the inherited software design for Mars 2020.  During MARDI development back around 2005, the idea of acquiring video of a helicopter flying around on Mars wasn’t even imagined.  But 16 years later, that video capability would be used to record the historic first flights of Ingenuity.

Single frame from the Mars Science Laboratory (MSL) Curiosity rover’s Mars Descent Imager (MARDI) video sequence during MSL Entry, Descent, and Landing (EDL). Mastcam-Z inherited the MARDI video capability developed for MSL. See the full Curiosity descent movie posted here: https://mars.nasa.gov/resources/20081/dropping-in-on-mars-in-high-res/?site=msl

By 2017, the dimensions and form factor of the helicopter were more or less finalized, and the Mastcam-Z optical design was also nearly complete.  With the basic design parameters in hand, it was possible to generate a notional, simulated view of the helicopter in an image from the MSL telephoto (100-mm focal length) Mastcam camera, to scale (see figure below).  The simulated view assumed that Mastcam-Z would be at full zoom (110-mm focal length), with the helicopter approximately 120 meters away.  It looked plausible that the videos would show some detail, including the landing legs and the rotors. The simulated views were obviously just notional, but as we approached launch (and landing), the idea that we might record the flights with Mastcam-Z video started to gain wide support.  And then it became part of the baseline plan! The first actual videos of Ingenuity were acquired at a closer distance of roughly 70 meters away, so the actual resolution was better than the simulation. 

Predictions of what the Ingenuity helicopter might look like to the Mastcam-Z cameras (left) turned out to be lower resolution than the real images (right), because the team was allowed to park the Perseverance rover closer to Ingenuity than had originally been envisaged.

The Mastcam-Z uplink operations team at MSSS, led by experienced planetary instrument operations engineer Elsa Jensen, has a lot of experience pointing cameras to targets on the surface of Mars as the rover drives from one place to another.  We use various Cartesian and spherical coordinate frames to point the cameras to points of interest, and for the most part the process is fairly straightforward.  But unlike most of the rocks and hillsides, the helicopter moves around.  With this added dimension, the targeting process became a question of not just targeting the right location, but targeting it at the right time.  As easy as that timing challenge sounds, it required a great deal of coordination between the Mastcam-Z operations team, the rover vehicle operations team, and the helicopter operations team (as with many things in space mission operations, it was a multi-team effort). 

Our adopted strategy to cover any residual uncertainties in timing was to start recording images well before the expected flight time, and end recording well after the expected landing time.  We would then downlink and review the thumbnail videos (thumbnails are much smaller, spatially downsampled versions of the larger, original images), along with a small number of full frame images, to determine which particular frames caught the actual flight.  The cameras would be later commanded to copy those particular frames of interest from internal camera storage over to the rover computer for subsequent downlink to Earth.  The video frames were compressed as individual JPEG images at quality value of 50.  Once on the ground, the individual frames were assembled into mpeg-compressed video files using ffmpeg. Mastcam-Z team member and atmospheric science expert Mark Lemmon also did a bunch of post-processing of the videos to bring out both subtle and not-so-subtle details of the ways that the helicopter created an artificial dust-lifting wind as it flew.

These videos are exciting to watch, even months later.  The full set of 10 videos are included here (5 initial demonstration flights x 1 video per camera x 2 cameras). Additionally, specially-processed versions of these videos have been created by Mastcam-Z team member Mark Lemmon, using the same processing methods that were described in his “Dust Devil Photobombers!” blog entry. These processed versions provide dramatic evidence for unique dust-lifting and dust-transport processes that are initiated and sustained by Ingenuity’s “blade wash”.

Mastcam-Z Movies of the Demonstration Flights

Flight #1.  The historic first flight, around 12:30 pm Local Mean Solar Time (LMST) on mission sol 58 (April 19, 2021), was a simple takeoff, followed by a hover at a height of 3 meters above the ground, with a rotation, and a landing.  We captured the full flight at a wide angle (34 mm, 6.5 frames/second (fps)) in the right camera and at full zoom (110 mm, 6.3 fps) in the left camera.  Total flight time was approximately 39 seconds.

The historic first flight of a powered aircraft on another world. Ingenuity lifts off! (Mastcam-Z left camera, 110-mm zoom, April 19, 2021)
Specially-processed flight #1 left camera video, enhancing the effects of dust lifting by Ingenuity (processing by Mark Lemmon)
The historic first flight of a powered aircraft on another world. Ingenuity lifts off! (Mastcam-Z right camera, 34-mm zoom, April 19, 2021)
Specially-processed flight #1 right camera video, enhancing the effects of dust lifting by Ingenuity
(processing by Mark Lemmon)

Flight #2:  The second flight, around 12:30 pm LMST on mission sol 61 (April 22, 2021), hovered at 5 meters above ground, and consisted of a 2 meter lateral move in the direction to and from the rover (4 meters total).  We captured the full flight in the right camera (34 mm, 6.5 fps) and part of the flight in the left camera (48 mm, 6.5 fps).  Total flight time was approximately 52 seconds.

Ingenuity flight #2 (Mastcam-Z left camera, 48-mm zoom, April 22, 2021)
Ingenuity flight #2 (Mastcam-Z right camera, 34-mm zoom, April 22, 2021)
Specially-processed flight #1 left (right panels) and right (upper left panel) camera videos, enhancing the effects of dust lifting by Ingenuity during the helicopter’s lift-off (processing by Mark Lemmon)

Flight #3:  The third flight, around 12:30 pm LMST on mission sol 64 (April 25, 2021), hovered at 5 meters above ground, and flew 50 meters out and back (100 meters total).  The entire flight was captured by both cameras at the widest focal length setting (26 mm, 6.6 fps).  Flight 3 was the only flight captured in stereo video thus far.  The helicopter flies out of view on the right side of the image, and comes back into view from the right, near the end of the video.  Total flight time was approximately 80 seconds.

Ingenuity flight #3 (Mastcam-Z left camera, 26-mm zoom, April 25, 2021)
Ingenuity flight #3 (Mastcam-Z right camera, 26-mm zoom, April 25, 2021)

Grab your red/blue glasses and see https://www.youtube.com/watch?v=Q75-HetU57A for the stereo video!


Flight #4:  The fourth flight, around 12:30 pm LMST on mission sol 69 (April 30, 2021), hovered at 5 meters above ground, and flew 133 meters to the left and back (266 meters total).  For this video the entire flight was captured in the left camera (26 mm, 6.7 fps) and part of the flight in the right camera (26 mm, 6.8 fps).  For this flight, the SuperCam microphone also recorded audio.  Total flight time was approximately 117 seconds.

Ingenuity flight #4 (Mastcam-Z left camera, 26-mm zoom, April 30, 2021), with SuperCam audio!
Specially-processed flight #4 left camera take-off video, enhancing the effects of dust lifting by Ingenuity
(processing by Mark Lemmon)
Specially-processed flight #4 left camera return-to-takeoff-site video, enhanced to show the dramatic amount of dust entrainment by Ingenuity as it came back from dustier areas off screen to the left.
(processing by Mark Lemmon)
Ingenuity flight #4 (Mastcam-Z right camera, 26-mm zoom, April 30, 2021), with SuperCam audio!
Specially-processed flight #4 right camera “out and back” video, enhancing the effects of dust entrainment and transport by Ingenuity
(processing by Mark Lemmon)

Flight #5:  The fifth and final demonstration flight, around 12:30 pm LMST on mission sol 76 (May 7, 2021), was the first time the helicopter flew off to a new landing location.  The landing was 129 meters to the south, flying at a cruising altitude of 5 meters above ground.  Near the end of the flight (above the landing site), the helicopter flew from 5 meters altitude up to height of 10 meters, hovered, then landed.  Because the remote sensing mast (RSM) could not be pointed during any of the helicopter flights (due to potential radio interference with the heli communications link), we decided to point the cameras at the flight #5 landing location and wait for the helicopter to arrive on the scene.  Both cameras captured the landing, and the right camera also caught a number of dust devils before and during the landing.  The right camera video was acquired at 26 mm, 6.8 fps, and the left camera captured the landing at full zoom (110 mm) and at our highest video frame rate so far: 15.4 fps.  The left camera video was subframed vertically to increase the frame rate, allowing a fun “super slo-mo” version of the springy landing to be captured.  The SuperCam microphone audio was also acquired for this flight.  Total flight time was approximately 108 seconds.

Ingenuity flight #5, landing only (Mastcam-Z left camera, 110-mm zoom, May 7, 2021), at highest frame rate yet (15.4 fps)
…and a short and zoomed-in slow-motion replay! Ingenuity sticks the landing!
Ingenuity flight #5, landing, with background dust devil! (Mastcam-Z right camera, 26-mm zoom, May 7, 2021). Ingenuity comes in around 2:10.
Specially-processed flight #5 right camera landing video, enhancing the effects of dust lifting by Ingenuity and showing a cool dust devil in the background behind the helicopter during landing!
(processing by Mark Lemmon)

Flight #13: Mastcam-Z recorded another movie of Flight #13 around noon LMST on sol 193 (Sept. 5, 2021), as part of a cool atmospheric science experiment that used the helicopter as an artificial dust devil maker. The helicopter was about 300 meters from the rover for this flight, so it’s challenging to spot it in the wide-angle left Mastcam-Z view at 26-mm focal length, but much easier to watch in the telephoto right Mastcam-Z view at 110-mm focal length. The total duration of the flight was about 160 seconds, but only part of that was actually caught in the videos. Lots more details about this flight can be found in the NASA/JPL press release at https://www.nasa.gov/feature/jpl/nasa-s-perseverance-captures-challenging-flight-by-mars-helicopter

Ingenuity flight #13 (Mastcam-Z left camera, 26-mm zoom, 6.5 frames/sec; September 5, 2021)
Ingenuity flight #13 (Mastcam-Z right camera, 110-mm zoom, 6.0 frames/sec; September 5, 2021)

As of late December 2021, Ingenuity has now flown a total of 18 flights, with flight numbers 6 and onward being part of an Operational Demonstration and Science Support extended mission for the helicopter. It’s been exciting to see Ingenuity help scout out areas where the rover either hasn’t driven yet or can’t drive into, helping provide the science team with unique new perspectives on the geology of Jezero crater.We’ll try to update this blog with any other Ingenuity movies that Mastcam-Z takes, too, once they are fully downlinked and processed from the rover!