how do we know that this limit doesn't exist? how to calculate it?

Look at what happens if you plug in values of x near 5. The numerator will be approximately 6, and the denominator will be approximately 0. Specifically, if x is very close to 5 and less than 5, then the fraction will have the form
(number close to 6)/(very small negative number),
so it will be a very large negative number. But if x is very close to 5 and greater than 5, you get a large positive number. This shows that there can't be a limit. You can't even say that the limit is ∞; the limit from one side is +∞ and the limit from the other side is -∞, so there is no limit that works for both sides.

If you like, you could also rewrite (x^2-5x+6)/(x-5) as x + 6/(x-5). The "x" part is obviously fine, so you just need to decide whether the "6/(x-5)" part has a limit. But the graph of 6/(x-5) is basically the same as the graph of 1/x, just shifted over by 5 and stretched out. You have probably seen that 1/x does not have a limit as x-->0; this is basically the same thing.

lim(as t approaches 5) (xˆ2-5x+6)/(x-5), here must be x instead of t

lim(as x approaches 5) (xˆ2-5x+6)/(x-5)=(*)

We will try to make simply right side x^2-5x+6=(x-x1)(x-x2), where x1 and x2 are results of quadratic equation x^2-5x+6. x1/2=(-b+-sqrt(b^2-4ac))/2a, and a=1, b=-5 and c=6
x1/2=(5+-sqrt(25-24))/2=(5+-1)/2, x1=(5+1)/2=3, x2=(5-1)/2=2
So, x^2-5x+6=(x-3)*(x-2)

(*)=lim(as x approaches 5) (x-3)*(x-2)/(x-5)= let x to be 5 = (5-3)*(5-2)/(5-5)=6/0=infinity

I think this is not good setting, maybe it is (x-3) or (x-2) in bottom part of fraction and x approaches 3 or x approaches 2 cause that would make a sense.

lim(as t approaches 5) (xˆ2-5x+6)/(x-5)

x cannot = 5, as x tends to infinity, small number
as x tends to -infinity, very small number

hence the limit doesn't exist. hope it helps.

You are probably trying to over-think it!
Take a step back from the problem and recall what operations are not allowed in math. For example, you know that you cannot divide by zero! Take your calculator and try to tell it to divide 1 by 0. It gives you an error because this operation is undefined.
Now, your upper limit in this problem is 5. Look at the denominator of your function. It is x-5.
So if you plugged in 5 for x what would you get? 5-5=0. This means you have 0 on the denominator, or, division by zero. Since this is not allowed, the limit does not exist because the upper bound of the limit renders an undefined answer.

In order for a limit to exist, there must be an indeterminant expression when making a substitution. These include 0/0, ∞/∞, ∞/0 and so on. If it ends up being an undefined expression, such as 1/0 or 9/0, it does not have a limit.
Plug the value of x as 5 for the expression. Check if it leads to an indeterminant expression.
(5^2 - 5*5 + 6)/(5 - 5)
(25 - 25 + 6)/0
6/0
This is an undefined expression. This leads to there being no 'defined' limit. The limit is actually infinity. This is why the limit does not exist.