Calculus math problem...help please and explain?

The first answer is missing the point. You don't need (x^2 + 4x - a)/(x^2 + 2x - 3) to be equal to 1-b. Rather, you need the limit as x-->1 to be 1-b; and that's a different thing altogether.

Write that as:

g(x) = (x^2 + 4x - a)/[(x + 3)(x - 1)]

...after factoring the denominator. You need the numerator to have a factor of (x-1) also. suppose:

x^2 + 4x - a = (x - 1)(x + c) . . . . for some number c. Then:
x^2 + 4x - a = ^2 + (c-1) - c . . . . multiplied out
4 = c-1 . . . . coefficients on x must be the same, so c=5
-a = -c . . . . constant coefficients must be the same so a=c=5

Now we have a value for a, use that in the original g(x) formula, and take the limit as x-->1:

g(x) = (x^2 + 4x - 5) / (x^2 + 2x - 3) ... original
g(x) = [(x + 5)(x - 1)] / [(x + 3)(x - 1)] .... factored
g(x) = (x + 5)/(x + 3) ... for all x near (but not at) x=1
lim x-->1 g(x) = (1+5)/(1+4) = 6/4 = 3/2

So, let 1-b = 3/2, b=-1/2 and you have a continuous solution. Remembering that a=5, that's answer C.

What this is wanting you to do is come up with values for a and b that makes both equations have the same value when x = 1.

Let's see what happens:

(x² + 4x - a) / (x² + 2x - 3) = x - b

This is true when x = 1, so:

(1² + 4 * 1 - a) / (1² + 2 * 1 - 3) = 1 - b
(1 + 4 - a) / (1 + 2 - 3) = 1 - b
(5 - a) / (3 - 3) = 1 - b
(5 - a) / 0 = 1 - b

Since when x = 1, we have a denominator of 0, no values of x or y will make this true as the first equation has no real value when x = 1. So the answer is option E.