Content shared with Cosine Earlier:
Just a recap of what happened today over in ALS:
A contractor working for our Infrastructure team had orders to cut and splice the old fiber in ALS today. They had bad information that all old core services were off of the old fiber, and when they cut the service, we lost all services in the building still on the old core.
Core engineering got a phone call from Ag at 12:30 today, stating that their customers were down. We dispatched techs to both Kerr and ALS to check physical connectivity.
In speaking with COSine, they informed us of the contractors being in building, and at that time (1pm) we sent Infrastructure over to help reconstruct a fiber path to get customers back up.
Flexnet/Ag services came up at 2:30pm, and I was just informed by COSine that their services are up, as we got the last link up at 3pm. They could not move customers back onto the old single mode, so your connections are now on the new singlemode fiber bundle (pair documentation below).
On behalf of central IS, I apologize to you all, as well as the customers for the outage. Our management is currently reviewing the events, and will be looking at and correcting our procedures to make sure there’s never a repeat of this kind of outage.
OSU Network Engineering
541-737-5574
