INFINITIVES

Not to go to the party now would be a pity.

TO + VERB = INFINITIVE      
 here are 2 to's but 'TO GO' is correct 
 because  party is not   a verb and infinitives are always followed by verbs..