Is there any way to stop people from calling my phone to speak with the previous owner?

You can't stop them from calling it, it's just a phone number and anybody can dial it.

After they've called once, you can block the number (no app needed usually) and then they can keep calling it but you'll never see it ring. Exactly how you block the number depends on what phone you have.

Get a new sim and new number

you can just simply block their number using an app.