The Mole Truth

Troubleshooting FXO lines can be like trying to tie your shoe laces with your teeth.  It’s not that it’s impossible, but you know there must be an easier way. Let me tell you, telecom carriers are not here to make this cumbersome job any easier.

Take for instance a recent case in which debugs showed that the carrier was dropping calls coming into an FXO port.  The symptoms were ones I had seen before – basically users hear a half ring when a call comes in and then, try as they might, they can’t answer the call before the line goes dead. Then they hear another tormenting half ring, and again silence when answered.

Having ruled out the notion that someone was playing one fabulously entertaining and clever joke on poor unsuspecting users, I spent quite some time placing test calls and running debugs the first time I encountered this issue. After some quality time with TAC, I turned the issue over to the carrier, confident the problem was on their end.

In the case at hand, I spent the same amount of time placing test calls and debugs, spent a little less time with TAC, and then once again turned the issue over to this customer’s local carrier, confident something was a muck on their end.

At that’s when the games began. Again.

Granted, I am going to be a bit facetious with these, but I know I am not the only one who has had this experience more than a few times…

Rule one of carrier technicians- never admit there is anything wrong.
Here’s how it works:
Assigned technician immediately claims there is no issue at all, everything is just fine.
Uh, so when you call this number the call goes through for you? No. Well, you see, that’s the problem…

Rule two of carrier technicians- make a token appearance on site.
Here’s how it works:
Unbeknownst to you, technician arrives on site, makes bold claims that all things carrier side work fine, and leaves.  Technician calls and tells you about it later.
Wait, are you sure?  Did you get dial-tone up to the demarc?  Oh, you didn’t check for dial-tone…yeah, we need you to go back and confirm that…

Rule three of carrier technicians- make second on site visit, use test equipment, resolve issue.
Here’s how it works:
Technician begrudgingly gets out test equipment and uses it to tone out the line.  Finds moles.
I’m sorry, did you say moles?  Yes, ma’am, moles. A whole family of ’em.

Okay, so maybe finding moles isn’t the rule, but it certainly was an entertaining exception.  You see, the half rings were a symptom in both of my cases of damaged copper wires.  In the first case, the wires had been terminated poorly and exposed.  In the second, a family of moles had decided copper wires make the best houses.

What did I learn from these experiences?  One, tests calls and debugs are your friend.  Two, the carrier technician isn’t.  That is stated a bit harsh- I do have it on good authority that carrier technicians are people too and not in fact spawned from evil alien overlords…BUT a good engineer has to have a willingness to be an advocate on their customer’s behalf. Even when told repeatedly everything is fine- just fine.

In my cases, I had to insist the technician tone out the lines – all other indicators on the carrier side were telling the technician things were fine.  However, once the wires were toned and traced, there was no time wasted getting the lines repaired.

So don’t be afraid to ask the technician to take that second look.  Ask for the results of whatever packet capture that has been done or test that has been performed.  Verify that proper troubleshooting was done – but handle it professionally, because you never know when those debugs, tests, and traces are going to show you missed something.  And then it’s your words you get to eat, so make sure they aren’t sour.

Published 04/12/2012