All of us = every one of us (all is a pronoun here), so we should use singular verb.
where = adverb with the meaning of "to what place", so there is no need to use a preposition "to" after the verb "go".
# 查閱日英辭典如下:
all [every one] of us:<釋義> 一同、有志一同、我們一同、我們皆。
例. All of us are equally to blame. (我々みんな同罪だ.)
例. All of us strive for this goal. (それを全員が目指す.)
例. We all have to go.=All of us have to go. (我々はみな行かねばならない.)