[Ed. note: Introspection is an attribute of a sober, focused mind. With it comes a humbling of the ego and an appreciation of one's innate humanity that Mr. Hale sums up in his final statement in his blog post: "...this is how I found out that we were never really as smart as we thought we were."]
How We Nearly Lost Discovery
by Wayne Hale, posted on April 18, 2012, to his blog.
Now that Discovery is safely delivered to the Smithsonian, I think I can tell the story of how we nearly lost her in July of 2005, and how well intentioned, highly motivated, hard working, smart people can miss the most obvious.
It’s tough to know people who have died. Many of us knew the astronauts on Challenger and Columbia well. We had met with them daily, we had visited in their homes, we knew their families, their children. It is not an easy thing to lose a colleague; especially one who entrusted their safety to you. So don’t question whether we were motivated to prevent another loss.
Discovery was the shuttle return to flight vehicle after the Challenger was lost; two and a half years were spent from January 28, 1986 until Discovery flew in September 1988. Many improvements were made which resulted in a safe space flight. Many improvements were made but safety was not assured.
It was not until Discovery again flew in July of 2006 before she flew safely. That counts as the third “return to flight” mission for Discovery.
You see, we dodged a bullet in 2005. One we should have seen coming but didn’t.
After the loss of Columbia, it became very clear that that the loss of insulating foam off of the external tank caused damage to the heat shield which lead to the loss of the good ship and crew. It was a devastating time. Tens of thousands of experiments, tests, and analyses were performed to discover why foam was lost from the external tank and how to prevent it. One of the most powerful series of test performed used a partial panel, about 2 feet square, of the aluminum skin of the tank covered with foam. This test article was chilled on one side to the cold temperature of the cryogenic hydrogen fuel, and heated on the other side with flames simulating the heat generated by supersonic flight through the lower atmosphere. Adding high velocity airflow to mimic the high speed flight conditions, virtually all the time the foam stayed on the tank. Only when small defects were embedded in the foam would some of that foam pop out under those strenuous conditions.
All the tests and analyses lead to the inexorable conclusion that defects in the application of the foam insulation could cause foam debris to be generated during the early supersonic phase of shuttle flight. We informed the foam technicians at our plant in Michoud Louisiana that they were the cause of the loss of Columbia and then worked them overtime in training with new and exhaustive techniques on how to apply foam with no defects.
Many other safety measures were incorporated during those two and a half years; a new inspection boom with camera was built so that complete inspection could be made before shuttle re-entry; heatshield repair materials were tested and proven to work in the vacuum of space if repairs were needed, plans for safe haven at the International Space Station were formulated so the crew would be safe even if extreme, unrepairable damage were detected. Anybody who followed the shuttle program in those days knows all this.
What you probably don’t know is that a side note in a final briefing before Discovery’s flight pointed out that the large chunk of foam that brought down Columbia could not have been liberated from an internal installation defect. Hmm. After 26 months of work, nobody knew how to address that little statement. Of course we had fixed everything. What else could there be? What else could we do? We were exhausted with study, test, redesign. We decided to fly.
The launch seemed very normal, we had new cameras everywhere including one on the external tank itself. We all breathed a sigh of relieve when Discovery reached orbit seemingly without incident. The members of the Mission Management Team received an early briefing from the video review of no incidents; we boarded our planes from KSC to head back home, that mean Houston for me.
The next morning there was an early MMT briefing from the launch video review team, again nothing major to report; the crew had a busy day on orbit, my assignment was to go to the MMT press briefing where we gave a positive report. It was on the drive home looking forward to dinner and sleep when I got the call.
I think that must have been the worst call of my life. Once earlier I had gotten a call that my child had been in an auto accident and was being taken to the hospital in an ambulance. That was a bad call. This was worse. Think of the worst phone call you have ever gotten. I think this one was worse.
John Muratore, my good friend, fellow flight director, and current head of the shuttle program Systems Engineering and Integration office informed me in very flat terms that he was in the JSC video lab with head photo interpreter Cindy Conrad who had uncovered evidence of a large foam liberation during the critical mach number regime which appeared to have impacted the left wing of Discovery. Just like Columbia.
I made a highly illegal U-turn in the middle of NASA Road 1 and probably exceeded the posted speed limit heading back to JSC and the photo lab. Here is one still frame from the video they showed me: A very large piece of foam coming off the tank heading for the wing.
I thought I would not be able to breath again.
I thought of Eileen and her crew and how we had probably just killed them.
I thought of all the work and time and how wasted it had been.
I had no idea how I could possibly tell the team, the world, what had transpired.
But unlike Columbia, this time the crew had the means to inspect their heat shield. That inspection revealed no damage. The pictures and engineering data showed no damage. Positive and unambiguous evidence showed the heat shield intact and safe.
Then and only then could I start breathing again.
The next day we told the crew, sent them the video, told the entire NASA senior management, and at the press conference showed the video and made the characterization that this was “unsatisfactory”. A pretty bland word for the way I really felt
So what happened? The video wasn’t really clear so either the foam missed the underside of the orbiter completely or struck such a glancing blow that the impact resulted in no damage.
But that is not the real question to be answered. What really happened to the foam? We wouldn’t find out until December.
God gave us a great gift during the previous year. We had ECO sensor problems. At the time that did not look like a gift. We filled the external tank with cryogenic fuel as a test in May and the Engine Cutoff Sensors – little wet/dry indicators at the bottom of the tank – malfunctioned; indicated dry when they were wet. We scrubbed that tank from the first flight and sent it back to the factory to analyze sensors.
Now an external tank is a fragile and valuable thing. On a good day the cost estimate for a completed, flight ready tank was $30 million. On a bad day, more than twice that. We never, ever, had shipped a tank back to the factory. We tested them at the launch site, made occasional repairs, took lots of measurements, but never had shipped one back before that tank.
When we shipped it back – did I forget to mention that Hurricane Katrina devastated New Orleans the month after we launched Discovery? Did I forget to mention that the steering mechanism on the trailer the tank was on failed so we sent another tank back to Michoud before the critical one? Months passed and right before Christmas, we got these X-rays from Michoud.
They showed cracks in the foam. Foam where no defects existed. It turns out that the thermal cycles associated with filling the tank could crack the foam, especially in areas where there were two or more layers of foam.
Finally, this explained the Columbia foam loss. And the Discovery foam loss. And it had nothing to do with the improper installation of the foam.
I flew to New Orleans the next day, and called an all hands meeting where I publicly apologized to the foam technicians. They had not caused the loss of Columbia through poor workmanship. Those guys were reeling from the hurricane’s devastation to their homes and community, and has lived with nearly 3 years of blame. Thin comfort for me to apologize, so late, so little.
We worked feverishly to remove foam on foam wherever we could, minimize it where it could not be eliminated, and the following July we were ready to try again.
Discovery flew on July 4, 2006; no significant foam loss occurred.
I consider that to be the real return to flight for the space shuttle.
So were we stupid? Yes. Can you learn from our mistake? I hope so.
So when you go to the Smithsonian and see Discovery there, think how lucky you are to see her whole, intact, and with her crews safely on the ground.
You see, this is how I found out that we were never really as smart as we thought we were.
Maybe that is a lesson that applies to you, too.
Wayne Hale is retired from NASA after 32 years. In his career he was the Space Shuttle Program Manager or Deputy for 5 years, a Space Shuttle Flight Director for 40 missions, and is currently a consultant and full time grandpa. He is available for speaking engagements through Special Aerospace Services.
