They're more than twice your price-range, but have you looked into UKs "Spartan"? They definitely qualify as "casual athletic". A little too casual for my taste.

...and I think someone said something about Museum Replicas having some cheap kilts for 30 or 40 dollars, if you just want something for the house.