the question can be divided into 2 parts
1. Did you refused the invitation of the victim that night?
2. Did you not refused the invitation of the victim that night?
so, the answer should be "yes, I did refuse" or "no, I did not refuse".