How do I set the tension for specialty threads? I am afraid I will never get it right again.

First, lets look at what tension is. The top tension is not only affected by the tension assembly (or assemblies) on your machine, anything that causes drag or pressure anywhere in the thread path adds tension. Perhaps when using a heavy thread like Sulky 12 wt. Cotton, the dial cannot be backed off enough without completely losing control of the assembly and take-up spring. Look for other ways to reduce drag. Shorten the thread pass or go through fewer holes on the flat guides. Skip the pretension, don't wrap around the guide. You can still maintain control of the thread while reducing the drag so proper adjustment can be made to maintain stitch quality.