I've been using Seaguar fluoro and it's generally been great. I think birds nests are more a product of the breaks, the spool's tension, and how the line is spooled. Braid, mono, and fluoro can all have birds nests.
I don't have any magic system but I've gotten better at adjusting my setups little by little. It took me a good 20 trips out with a baitcaster to get the feel for everything involved with using it. There are some decent video on YouTube if you search for "baitcaster" or "fishing, birdsnest". (maybe "fishing, backlash")
Best Always,
Vinny