Find an invertible matrix P and a diagonal matrix D such that A = PDP^-1?

First, find the eigenvalues of A by solving |A - λI| = 0.
|10-λ 12 0|
|-8 -10-λ 0| = 0
|4 4 -2-λ|

Expand on the last column:
(-2 - λ) [(λ^2 - 100) + 96] = 0
==> λ = -2, -2, 2.
--------------
Next, we find the corresponding eigenvectors:

For λ = -2, we solve (A - (-2)I)v = 0.
[12 12 0|0]
[-8 -8 0|0]
[4 4 0|0], which reduces to

[1 1 0|0]
[0 0 0|0]
[0 0 0|0], yielding v = (-a, a, b) as the general form for an eigenvector.

Letting a = 0, b = 1 (and vice versa) yields v = (-1, 1, 0) and (0, 0, 1).
---------
For λ = -2, we solve (A - 2I)v = 0.
[8 12 0|0]
[-8 -12 0|0]
[4 4 -4|0], which reduces to

[1 0 -3|0]
[0 1 2|0]
[0 0 0|0], yielding v = (3, -2, 1) as an eigenvector.
------------
Let D be the diagonal matrix of eigenvalues:
[-2 0 0]
[0 -2 0]
[0 0 2].

Then, P is the matrix whose columns are the corresponding eigenvectors.
[-1 0 3]
[1 0 -2]
[0 1 1].
--------------
I hope this helps!

P is based on the eigenvectors of A. Use an online calculator if you don't remember how to find them. Then, just take the inverse of P to get P^-1 (again, use an online calculator).

D is just the diagonal of A; also an basic calculation.