A year’s worth of production on Toy Story 2 was nearly lost due to user error and a bad backup. The video “Toy Story 2: The Movie Vanishes” gives a fanciful telling of how this happened, but it’s mostly possible to separate fact from creative elaboration. A tweet from @DisneyPixar confirms the authenticity of its source. It’s also on the Toy Story 2 Blu-Ray DVD combo pack.
Leaving aside the clever but silly stuff about the characters vanishing one by one, we can extract this account: A great many critical files for the production of Toy Story 2 were on a single Linux or Unix computer. Someone who shall remain nameless (and probably jobless) mistakenly entered “rm *” on the directory with these files. An emergency call was made to the sysadmin to just yank the plug out of the wall; this was done about 20 seconds from the time of the entry of the command, most of the files were gone. Worse, the backups for the past month were defective to the point of uselessness. Fortunately, the technical director, Galyn Susman, had been working at home and had all the files there. They physically transported the computer to the studio to copy all the files back.
I’m hoping the video takes serious liberties with what happened for the sake of entertainment. If not, the amount of stupidity that happened is staggering. Let’s assume, for the sake of a lesson in what not to do, that they really did all these things.
First, the Linux/Unix command line is dangerous. rm * will delete all the files from your directory. Think three times before hitting Return after typing it.
Second, if you do enter a mistaken rm *, DON’T UNPLUG THE COMPUTER, YOU IDIOT!! That will just damage the file system and won’t be quick enough to save any files. Hit Control-C. It’s much faster and safer, though even that will probably be too late.
But it took 20 seconds to delete all the files. That says there were a lot of files. It also says they were all in a flat structure with no subdirectories, since rm * doesn’t remove subdirectories. OK, maybe the command was really rm -r *, but the makers of the video were trying to keep things simple and dramatic. If you type rm -r *, think four times. If it’s rm -rf *, make it at least six.
Then, instead of bringing a drive to Galyn’s house and copying the files onto it, they wrapped her computer — the one with the only copy in the world of a year’s worth of work — in blankets and drove it in a car to the studio. Cue Christine Lavin singing “What were you thinking??” But at least they had an offsite backup, even if it was by chance.
OK, I’m not being fair. It’s a video for entertainment, and they really didn’t do quite all of those stupid things, other than not verifying their backups. I hope. But there are still lessons to learn.
1. If you have critical, valuable files, keep a backup of them and occasionally verify that the backups are good. This doesn’t have to be an elaborate check. Just look at the backup drive and list the directory, making sure current backups are there and have plausible file sizes (> 0).
2. Keep an offsite backup. With an operation like Pixar, security considerations apply as well; you don’t want to give backups to just anyone. But by the same token, if you’re Pixar, you can afford to make backups, move them offsite, and give them decent security.
3. If you need to restore from your last offsite backup, treat it extra carefully. If you can’t restore remotely, copy it at the offsite location and restore from the copy.
4. NEVER unplug a running computer to interrupt processing! (Sorry, that one really gets to me.)
Thanks to Mary Ellen Wessels for helping to research the video.