Indeed, using the ball in this way is violent conduct.
A million years ago, the (in)famous Wayne Shaw did something similar in an FA Vase game, although he threw the ball at a spectator who was disgustingly abusing his mother.
The referee sent him off and the club appealed. The Hampshire FA, in their infinite wisdom managed to find him not guilty, despite his admission of the act. They concluded that throwing the ball was not violent conduct - that is what happens when you 250+ years of wisdom on a three man disciplinary committee!!