Personally, I wouldn't wear the sweater to such an event, but thats because I wouldn't want to get too warm.

For a 'sporty-casual' look, if you have a shirt like like this one, I find the go very well with the kilt.

And yea, loose the sporran chain. Either for security reasons or just to prevent the wear and tear on the kilt.