You could've said, "Sorry sir, but I fail to see how that's any of your concern."

; )