English Grammar

You are right that the mistake was made in the past.
However, it is right that present perfect tense is used in this case.
The present perfect tense can be used when an action started in the past and continues up to the present or an action that has already been completed but has a great impact on the present.
In this case, although ths mistake was made in the past, when you look back, you may find that the mistake that you made greatly influences what is happening at present. Therefore, the present perfect tense, which is a tense connecting the past and the present, is used.

It's unspecific time in the past, so, use present perfect tense.

present perfect tense expresses an action done in a (time period) up to the present.(= I have made )
Looking back=when I looked at------It is a past tense.
Looking back=participal phrase,
Hence:-Looking back,I have made a mistake.

If the issue is related to nowadays or it's affecting the present, then present perfect tense should be used.
According to your sentence, it seems the writer wants to talk about something that makes him/her thinking about that moment, therefore, present perfect tense is used.