People were more likely to gossip about shirkers to colleagues - who belong to the same group as the shirker and the person spreading the gossip - than to friends.
While certainly more fortunate than just about anyone else you could think of, a shirker he isn't, and frankly, why should we begrudge him a short break now and then?