A Daily Mirror report mentioned crowd trouble at a Forest-Burnley game on 18 April 1964 (which Burnley won 3-1) culminating in some fans trying to attack referee Horner and his linesmen at the end. I'm not sure if that were his final match (the season ended the following weekend) but it wouldn't be surprising if he decided to call it a day after such an experience.
You can see Harold Horner having a few spots of bother in a Tranmere v Chelsea FA Cup tie in the big 62/3 freeze up at