To simplify the expression, we first expand the terms inside the brackets:

(x^2 - 3x)^2 = x^4 - 6x^3 + 9x^2
2x(x - 4x^2) = 2x^2 - 8x^3

Now we substitute these expanded terms back into the original expression:

(x^4 - 6x^3 + 9x^2) - (2x^2 - 8x^3)

Now we simplify the terms by combining like terms:

x^4 - 6x^3 + 9x^2 - 2x^2 + 8x^3

= x^4 - 6x^3 + 8x^3 + 9x^2 - 2x^2
= x^4 + 2x^3 + 7x^2

Therefore, the simplified expression is x^4 + 2x^3 + 7x^2.