I see nothing wrong with it at all. Even a matte black sporran with a little bit of silver, like a leather semi-formal sporran would be well suited for formal. It may come off a bit "less-formal" but if done well, I believe it would work.

Edit:

This might be too plain



But the one Bubba suggested is a great option