'The only man who is really free is the one who can turn down an invitation to dinner without giving any excuse.'
Refusing graciously, without causing offence, is a vital social skill. Whatever the occasion - a date, a dance, an invitation - you have the right to turn the offer down, and on many occasions you will be genuinely unable to accept because of previous commitments.
Whether you are truly already committed, or simply indifferent to the invitation, good manners demand that you offer up some sort of excuse. Simply saying "no, thank you" seems distressingly bald; indicating that you have, for example, a prior (or imminent) engagement softens the blow. Do not, however, make the mistake of buttressing your refusal with elaborate excuses. Less is more, and over-embroidering will instantly rouse suspicion.
If you really don't want to comply (an importunate suitor, for example) a point-blank refusal will certainly get the message across: there will be occasions when you are confronted with a relentless inability to read social signals - and at that point, your good manners may have to be compromised.