Does my parents have to live with me to have physical custody?

Generally speaking, yes. But there might be situations, such as boarding school where you wouldn't live with the custodial parent all of the time. 

its probably either one

Physical custody means exactly that, you live with them and they look after you.