I would think wool might develop pulls and wear from too much rough work.

My biggest concern, though, would be cleaning. If you're doing yard work in it, you're going to want to clean it often. For me that would mean I would want something I could throw in the washer with no fuss. Second consideration would be practicality; if you're just mowing the lawn and raking, a kilt would be fine. If you have to be up and down off the ground, bending over, possibly climbing up on things, trousers may be more comfortable.