As the visit room guard, the inmates are probably less familiar with Fred than any other guard besides Adrian. According to the rules, he's not allowed to make small talk with the inmates in the visit room. What's more, some inmates might not have any visitors at all during their entire sentence, so would never even get to see Fred.
Wondering who Adrian is? Have you ever noticed the other guard in the visitors' area of the visit room?