I had a similar situation as you describe. I've had a series of boats. The 19' bowrider got a LOT of use. Then, I moved up to a Grady White. It also got a lot of use and was very seaworthy and super comfortable. Eventually, it wasn't getting much use, so I sold it.
Eventually, I landed on my current 22' Bay boat with a 150 on the back. Big, wide, fast, roomy, rated to carry 11 people, fuel efficient, easy to tow, etc. Love it! We took it on the lake every summer and wakeboard, tube and ski, grill food up front, hang out, eat, swim, sightsee, etc. And I fish the Bay with it. It has been awesome:
When we moved near Lake Anna, I picked up a 14' tin boat, then moved up to a 16'. Those boats have gotten a LOT of use, when I want to sneak out alone or with the boys to fish. So light and easy to deal with. Here is the latest one:
In 2020, a friend was moving and offered their old pontoon boat to me for $400. I thought I would clean it up and flip it, but took my wife for a ride, and that was it. Now that is HER boat. I play skipper, and she invites friends for evening cruises with grilled shrimp and steak dinners and so on. Really great for family. Sorry, no good pictures I would post on that one.
When we do a lake event, we often bring 2 boats if many or young people are involved. The pontoon acts as the mothership, and people get on and off of one of the smaller boats to wakeboard, tube or ski. We have some great times on the lake.
I hope you can find the right boat or maybe combinations of boats for your needs,