 |
|
|
|
| |
 |
As
Deputy Director at NASA's Goddard Space Flight Center, I was
responsible for overseeing the launch of the Aura spacecraft from atop
a Boeing Delta II rocket out of Vandenburg Air Force Base in July 2004.
Aura is an earth-observing satellite developed to help us study the
quality of the air we breathe. It will look at the state of the ozone
and the atmospheric composition in regards to the Earth's changing
climate.
I headed to California on July 5, 2004. The plan was that the satellite
would launch on the tenth, but we had a few problems getting it off.
This was the fifty-ninth launch of my career, and it was also a little
different than most of my previous launches. Most of the time it's
weather that postpones a launch; there aren't usually that many
technical issues this late in the game. This time, however, we had
several problems, equally split between the launch vehicle and the
spacecraft. I remember a member of the crew asking me, "Is this
normal?" And in my experience, it wasn't.
A Wrench in the Works
We had three significant spacecraft issues during the launch campaign.
These problems, together with the launch vehicle problems, ended up
postponing the launch five days. During that time, the mission
management team met 11 times at all hours of the day and night to try
to sort things out. I myself held four special reviews.
 |
| Launch of the Aura spacecraft from atop a
Boeing Delta II rocket out of Vandenburg Air Force Base in July 2004. |
The first problem was that some tools had been misplaced during final
spacecraft closeout, which could present a problem if they were left on
the spacecraft during launch. A wrench was lost and then found. Then we
realized that we had also lost a flashlight.
The first step to solving this problem was interviewing everyone who
had been involved with getting the spacecraft ready for launch. This
was a massive effort, even extending to people overseas. Then we were
able to make a timeline of activity based on photographic evidence,
which was time-tagged and fairly easy to review. Through these
measures, we were able to limit the possibility of the flashlight's
whereabouts to an area that made up about 10 percent of the spacecraft.
The Likely Story
We were able to determine that the flashlight had last been seen on a
processing table. Something called "scrim," which is a plastic covering
that is taken off the fairing during its installation on the launch
vehicle, had been put on the table. The flashlight was also on the
table, and it was probably swept off with the trash. We even checked
the dump, but the search was futile.
| No one is going to get in trouble for
calling our attention to potential problems. It's the kind of behavior
that we need to encourage. |
Based on all the evidence and interviews, we were able to put together
a story that convinced us that the flashlight was not on the
spacecraft. What was really frustrating, however, is that all of this
investigation could have been avoided. I found out that there was
someone along the way who had noticed immediately that the flashlight
had been misplaced. It was a week before they finally came forward, and
by that time, the trail was cold. The person didn't speak up, because
he was initially afraid to report it. I realized that there needs to be
a clear message sent on this type of thing: No one is going to get in
trouble for calling our attention to potential problems. It's the kind
of behavior that we need to encourage.
Shake, But No Rattle
The second problem had to do with a transistor failure in another
program. A nickel-plated transistor can had been improperly cleaned,
and this created the possibility of particles being generated inside
the can. As long as they were smaller than about two thousandths of an
inch, we wouldn't have a problem. But particles were reported to be
larger than that.
The screening technique for this type of problem is to vibrate the
transistor and listen for particles rattling around. This process is
able to detect particles down to one thousandth of an inch. But the
parts reported as having particles larger than two thousandths of an
inch had passed this test. One of the team members said that the noise
should sound like "an acorn in a coffee cup" if it were that large, but
there was no noise.
I asked to push back the launch date in order to figure out the
problem. It wasn't a popular decision, but I felt it was a necessary
one. I was able to facilitate a discussion between our parts and
materials engineers and those from the program that had the problem. It
turns out that there had been a miscommunication. The "particles" were
actually very quiet flakes. That's why all the parts passed testing
based on an acoustic response -- the flakes didn't make much noise.
We then did a thorough risk assessment for every application of this
part in both the spacecraft and the instruments. Thanks to redundancy,
we convinced ourselves we were OK. Had we just gone ahead with the
launch, it would have been successful. But we wouldn't have known why
it was successful. I felt we needed to take the extra time to figure it
out.
| I wanted to make sure that I took the time
to hear the dissenting opinion. |
The Minority Report
The last problem occurred during the countdown when there was a bit
flip in the solid-state recorder. We had seen this occur occasionally
on Aura's sister spacecraft, Aqua, and it wasn't mission-threatening.
But then it happened again one hour from launch. I asked for a summary
of the situation, and I was basically told that the spacecraft was
ready to fly "as is." I asked if there was anyone who disagreed with
that, and I was told reluctantly that there was one person. His name
was Michael.
I got Michael on the net and asked him to explain his opinion about the
problem and his reservations. It turned out that he had seen the
problem in the test program, but no one was worried because it hadn't
set off any alarms. While he was explaining, there were other people on
the net that kept saying we needed to go ahead with the launch. As the
conversation progressed, I could feel him getting pressure from the
rest of the team and begin to change his mind. I again made the
unpopular decision to delay the launch until the issue could be ironed
out.
 |
| Artist's concept of Aura satellite |
So after speaking to Michael, I went to find out more about the
problem, and to talk to my team about possible solutions. Since the
problem was on both the A and B sides of the recorder, and in the same
word and memory section on each side, we determined that if this became
a frequently recurring problem, we would be able to bypass that section
of memory and work around it. It sounded like a viable option.
But since Michael was the only one who had been concerned about the
problem, I wanted to consult with him before moving forward. He was
working the second shift and was asleep. I got him out of bed to hear
the solution and to see what he thought of it. He believed that it was
airtight. I really felt like this situation was very similar to what
had been outlined in the CAIB report: When minority opinion isn't
valued, people are afraid to speak up, and they end up giving in to
conformance pressure even though they know there's a problem. So I
wanted to make sure that I took the time to hear the dissenting opinion.
And Patience Is Rewarded
In the end, we launched on July 15, 2004. We managed to work around
each of these issues, mainly because we cracked them wide open each
time. There's a personal motto I've adopted from the wine industry, and
I take it with me to each launch. That motto is "No launch before its
time." It's the job of the management not to get caught up in the
"launch fever" that accompanies the last few hours before liftoff. If
there is an issue, no matter how small, it needs to be brought to the
table and dealt with. The problem needs to be investigated, the risk
needs to be evaluated, and sometimes the best decision is to postpone.
Better to hold off five days -- or if necessary, even longer -- and be
sure of success, than be on schedule with failure looming in the
background.
Lessons
- In dealing with potential problems,
it is essential to get to the bottom of technical questions and
understand why things work, not just why they don't work.
- It is important to hear, evaluate,
and respect minority opinion, as well as to protect that minority from
the conformance pressure of the majority.
Question
How can you foster a project environment in which people are not afraid
to speak up immediately when they notice that there is a problem?
Search by lesson to find more on:
 Recently retired from his position as Deputy
Center Director of NASA's Goddard Space Flight Center in Greenbelt,
Maryland, Bill Townsend is now the Vice President
and General Manager of Civil Space Systems of Ball Aerospace and
Technologies Corporation.
Former Goddard Center Director Al Diaz, speaking of his time working
alongside Bill Townsend, says, "I wish everyone who is a cynic about
what it takes to be successful in this business could read this story.
Six and a half years ago when I first became Center Director, we had
the only total failure of a mission in all my time at Goddard. Since
then we delivered 40 spacecraft to orbit or systems to another user.
The difference between then and now is the wisdom that I had to bring
Bill Townsend on board. Bill has been the difference between my having
a balanced life and having none. He has brought a lot of credit to the
Goddard Space Program." |
|
|
|
|
|
|
