This is an excellent point. Had the referee shown a yellow for the original incident, and then a second yellow for the foul then Fèr could have had no complaints. I suppose in hindsight the referee wishes he had stopped the game and cautioned Fer for the wrestling, but in my opinion he was correct to allow play to carry on as Wolves had the ball and were about to mount an attack. As discussed on here recently, the referee could have shown two yellows after the trip, making it clear what the first one was for.
Swansea will appeal and I hope they win their appeal, but it all depends on the referee, and whether he will admit that a yellow would have been sufficient punishment.