The length factor is for both sides, so your distance is 20' at 15Amps running thru the panel, which calcs out to be 8 - 10 AWG, depending on which chart.
Since your fish finders and cig plug don't need to be on a switch, one alternative (assuming you want to keep the switched fuse panel that you already have) could be:
1. Run your bilge (5A), navs (2A), and courtesy lights (2A) thru your existing switched fuse panel. That's 9A x 20' = 12AWG, so the switch panel's installed 12AWG will work, and
2. Run your finders (2A) and cig plug (4A) thru a separate fuse block, which would be 6A x 20' = 12-14AWG. This would mean you'd have to buy another fuse block and breaker to go with it. If that's not feasible, you could go with in-lines for the finders and cig plug...I really don't like them, but it's been done that way before.
or...you could somehow reduce the run from the breaker to the panel to a total of 10' to make the panel's 12AWG ok.
Oh, and you mentioned that your breaker was 40A...that seems to be too big for the load you're running thru the switch panel, even if you were running everything thru it.