Look at what sells higher and pick that. The only thing specialization gives you is an additional cloth when you make the specialty cloth. You'll be able to make all of the patterns.
So, for example. I'm now ebonweave. When I make mooncloth or spellweave, I get one of each. I get two each time I make ebonweave. You know this already...
Sell the extra cloth you make and buy the cloth you need. You'll turn a profit and get geared up in the same amount of time.