can I use ”studied” instead of ”had studied” in ?

No, you cannot use 'studied' instead of 'had studied'.

This is a 'if' conditional clause to describe things that were not true in the past.
From the sentence, we know that they did not studied harder in the past. That is why they have not passed.

Use past perfect tence in the conditional clause and 'would + present perfect tense' in the main clause.

PAST PARTICIPLE is used when the action described is 仲耐過 past tense.
We would HAVE passed if we HAD studied harder.
study-->pass
study 先過 pass