9am Wed.: Corrupt data. Reload. Revised downtime: 36-48 hours.
6pm Wed.: Reload complete. Index re-gen begun.
Shortest re-gen ever! (It pays to buy new servers!)
11pm Re-gen complete.
8am Thurs.: Catalogers review data.
9am Thurs.: Catalogers find some records were not overlayed and lack the 948 tag indicating they’d been processed. ILS vendor records compared against the db clean-up vendor records. The db clean-up vendor records are fine.
Flurry of phone calls ensue between Michael and the ILS vendor. Reports generated to identify how many records missed processing. Hundreds? Thousands? Hundreds of thousands? Michelle refuses to sign off on the project until the ILS vendor fixes the problem.
Noon Thurs.: problem identified. Previous migration back in 1995 had created “?” in first indicators of the 050. Reports indicated that approx. 660 of the 1.4+ million bibs were the problem. ILS vendor fixes and reloads!!
1pm: Michael is ready to send out the “You may resume work on Voyager” message.
Suddenly fire alarms blare in the building. The building manager comes on the loudspeaker: “This is not a drill. Evacuate the building.” Apparently, a pipe had burst in the basement, sending smoke and dust up, and setting off the alarms. Somehow, pressure in other pipes built up, so some leaks occurred – most damage was in the mailroom, and some on the 3rd floor processing area. Could have been much worse! They let us back into the building around 2:00pm.
We went live on Voyager by 2:30. Bunch of happy campers.
Despite the ups and downs of the last 2 days, we came back up within 48 hours. Hallelujah!