Accounts of Hermitage say William Douglas, knight of Liddesdale, starved a rival for the sheriffdom in the castle, and that the king then gave Douglas the post. The rival is usually named as Sir Alexander Ramsay, held in 1342 and said to have lasted about seventeen days on grain that dropped through cracks in the floor above. A castle ghost list names Ramsay among the spirits at Hermitage, alongside the claims about Soulis and a visit by Mary, Queen of Scots.
I have only found the ghost claim in roundups and not in any witness account, so the haunting itself is thin. The dungeon story is better established, but I have not checked it against a primary source.
Comments