I do not know how to solve this. (Binomial Theorem) UPDATED?

All the terms that will cause powers higher than 3 are represented by "……".

(1 + 2x + 3x²)⁸
= [(1 + 2x) + 3x²]⁸
= (1 + 2x)⁸ + ₈C₁(1 + 2x)⁷(3x²) + ……
= [(1)⁸ + ₈C₁(1)⁷(2x) + ₈C₂(1)⁶(2x)² + ₈C₃(1)⁵(2x)³ + ….] + ₈C₁[(1)⁷ + ₇C₁(1)⁶(2x) + ……](3x²) + ….
= [1 + 8*(2x) + 28(4x²) + 56(8x³) + ….] + 8[1 + 7(2x) + ……](3x²) + ….
= [1 + 16x + 112x² + 448x³ + ….] + 8[1 + 14x + ……](3x²) + ….
= [1 + 16x + 112x² + 448x³ + ….] + [24x² + 336x³ + ……] + ….
= 1 + 16x + 136x² + 784x³ + …...

Tr+₁=nCr A^n-rB^r

= (1 + 2x + 3x²)⁸

= [(1 + 2x + 3x²)⁴]²

= [(1 + 2x + 3x²)².(1 + 2x + 3x²)²]²

= [(1 + 2x + 3x² + 2x + 4x² + 6x³ + 3x² + 6x³ + 9x⁴).(1 + 2x + 3x² + 2x + 4x² + 6x³ + 3x² + 6x³ + 9x⁴)]²

= [(1 + 4x + 10x² + 12x³ + 9x⁴).(1 + 4x + 10x² + 12x³ + 9x⁴)]²

= [1 + 4x + 10x² + 12x³ + 9x⁴ + 4x.(1 + 4x + 10x² + 12x³ + 9x⁴) + 10x².(1 + 4x + 10x² + 12x³ + 9x⁴) + 12x³.(1 + 4x + 10x² + 12x³ + 9x⁴) + 9x⁴.(1 + 4x + 10x² + 12x³ + 9x⁴)]²

…but you want only the term with x³ → so you can simplify

= [1 + 4x + 10x² + 12x³ + 9x⁴ + 4x.(1 + 4x + 10x² + 12x³ + 9x⁴) + 10x².(1 + 4x + 10x² + 12x³ + 9x⁴) + 12x³.(1 + 4x + 10x² + 12x³ + 9x⁴)]² → you can simplify once again

= [1 + 4x + 10x² + 12x³ + 4x.(1 + 4x + 10x²) + 10x².(1 + 4x) + 12x³]² → you expand

= [1 + 4x + 10x² + 12x³ + 4x + 16x² + 40x³ + 10x² + 40x³ + 12x³]² → you simplify

= [1 + 8x + 36x² + 104x³]²

= (1 + 8x + 36x² + 104x³).(1 + 8x + 36x² + 104x³)

= 1 + 8x + 36x² + 104x³ + 8x.(1 + 8x + 36x² + 104x³) + 36x².(1 + 8x + 36x² + 104x³) + 104x³.(1 + 8x + 36x² + 104x³)

…but you want only the term with x³ → so you can simplify

= 1 + 8x + 36x² + 104x³ + 8x.(1 + 8x + 36x²) + 36x².(1 + 8x) + 104x³ → you expand

= 1 + 8x + 36x² + 104x³ + 8x + 64x² + 288x³ + 36x² + 288x³ + 104x³ → you simplify

= 1 + 16x + 136x² + 784x³

6561x^16 + 34992x^15 + 99144x^14 + 190512x^13 + 274428x^12 + 311472x^11 + 286776x^10 + 217776x^9 + 137638x^8 + 72592x^7 + 31864x^6 + 11536x^5 + 3388x^4 + 784x^3 + 136x^2 + 16x + 1

Let a=1+2x
b=3x^2
(a+b)^8 = a^8 + 8C1 a^7 b + 8C2 a^6 b^2
a^8 = (1+2x)^8 = 1 + 8C1 1^7 (2x) + 8C2 1^6 (2x)^2+8C3 1^5 (2x)^3 + other term with expoent > 3
= 1 + (8)(2) x + (28)(4)x^2 + (56) (8) x^3
= 1 +16x +112x^2 + 448 x^3 ---------(1)

8 a^7 b = (8) (1+2x)^7 (3x)^2
= 72 (1+2x)^7 x^2

(1+2x)^7 = 1^7 + 7C1 1^6 (2x) + 7C2 1^5 (2x)^2 + other terms
8 (1+2x)^7 (3x^2) = 24x^2 (1^7 + 7C1 1^6 (2x) + 7C2 1^5 (2x)^2 + other terms)
=24x^2+(24)(7)(2)x^3
= 24x^2 + 336x^3 ------(2)

8 a^6 b^2 = (8) (1+2x)^6 (3x^2)^2
(3x^2)^2 = 9x^4 so there won't be any more x^3 terms

(1)+(2) = (448+336)x^3+(112+24)x^2+16x+1
= 784x^3 +136x^2 + 16x+1