There is no definite answer to your question.
Which A4 model are you talking about? What year? I assume it's a 1.8T Avant (station wagon)? BMW has better driving dynamic and feel than the Audi if the Audi is not a Quattro AWD model. A4 probably have better interior assembly, material and built-quality than the 320i.