A Link That Passed Every Lab Test Still Failed in the Field
One of the most frustrating networking problems is a failure that disappears when you try to investigate it.
I ran into this on a high-speed Ethernet platform that looked completely healthy in the lab. The link came up normally, traffic passed without issues, and repeated tests gave us no reason to expect trouble.
Then the same hardware moved into a real environment and started showing intermittent link flaps.
The link would stay healthy for long periods, then suddenly drop and recover. Sometimes it happened after hours of normal operation. Sometimes it appeared under hotter conditions. There was no simple pattern and no obvious software failure.
That made the problem difficult to reproduce, but it also gave us an important clue.
The issue was not that the lab testing was wrong. The issue was that the lab could not perfectly reproduce the full range of conditions the hardware would experience in...
Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE
