I've seen guys use a 4 or 6, and while a smaller motor will push it, an 8 or 9.9 will do it easier and will keep your speed more constant on the windy days without constantly messing with the throttle.
Having a little extra HP for the possibility of pushing against strong wind & current is a good thing. I had a 4 HP & the day came where I needed to go upstream in a strong current. It was painfully slow progress.
You can always put a lower pitch prop on to slow it down for trolling.