The joys of network migration
Today, I helped move the network connection that connects pell, gehenna, and the company I colocate with from a lowly T-1 over to a 20mbs ethernet line. It did not go smoothly.
We'd pre-planned this migration months ago, and thought we'd told the ISP just what our situation was and what hardware we had and what our network was like, but somewhere between the sales engineers we talked to about our requirements and the NOC that actually had to implement those requirements, a few things got lost in translation. A few things, like, um, understanding how routing works. We started to feel a sort of lingering dread when the technicians told us that we needed to fill out a BGP routing modification form (um, hello? One network connection, remember? "Oh, but you need to fill it out anyway"), which intensified when all of the daytime techs just stopped answering email in the middle of the afternoon, and became an ominous cloud when they started asking about the IP address of "a machine that can accept all of the traffic" (um, hello? we don't have one, remember? That's what we TOLD YOU WHEN ME MADE THE PLANS THREE MONTHS AGO!!! Oh, sorry, was I using my outside voice?).
NOC:"I can ping 199.245.177.1 now."
Orc:"I can't ping the gateway, or any machine outside the network."
NOC:"can we put you on hold?" (10 minute delay) "Oh, we assigned that address to our router... Do you have any IP addresses here that you're using?"
Orc:"Yes, all of them, except <first gateway> and <second gateway>."
NOC:<clickity-click> "can we put you on hold?" (10 minute delay, terminated by ping starting to work)
Orc:"Um, I can ping <first gateway> but I can't ping <second gateway>"
NOC:"yes, because <second gateway> is assigned to the (disconnected, and they knew it because we were switching from that T-1 line to this 20mbs ethernet line -- ed) T-1 router, so I picked a different address."
Orc:(no reply, because I was gaping like a fish)
The technicians at the NOC,thankfully, didn't try to brazen it out, and before the cellphone (yes! there's no telephone in the machine room!) we were using ran out of power and disconnected, they used their telephone lifeline to talk to somewhat less-pimply PFYs, who managed to actually glue the gateways onto their big ethernet router, leaving us with a network connection that could wander out into the ISPs network cloud before running into a reef, filling with water, and sinking. And then I had to bail out and go home to do bear-tending, and by the time I got them to sleep the network was just staggering to its feet and, after a 3 hour outage in the busiest time of the day (but I'm not bitter) it came back to life.
At least I got paid for this. It may even be enough to cover the therapy bills.