Let's start by simplifying the expression:

(ctg^2a - cos^2a)(1/cos^2a - 1)

We can re-write cotangent as 1/tan:
(1/tan^2a - cos^2a)(1/cos^2a - 1)

Using the identity tan^2a + 1 = sec^2a, we can rewrite 1/tan^2a as sec^2a:
(sec^2a - cos^2a)(1/cos^2a - 1)

Now, we can expand the expression:
sec^2a/cos^2a - cos^2a/cos^2a - sec^2a + cos^2a

Simplifying further:
sec^2a/cos^2a - 1 - sec^2a + cos^2a

Now we can substitute sec^2a for 1 + tan^2a:
(1 + tan^2a) / cos^2a - 1 - (1 + tan^2a) + cos^2a

Expanding again:
tan^2a / cos^2a + 1 / cos^2a - 1 - 1 - tan^2a + cos^2a

Simplifying:
tan^2a / cos^2a + 1 / cos^2a - 2 - tan^2a + cos^2a

Now we can simplify further by using the trigonometric identities tan^2a = sec^2a - 1 and sec^2a = 1 + tan^2a:
(sec^2a - 1) / cos^2a + sec^2a / cos^2a - 2 - (sec^2a - 1) + cos^2a

Expanding:
sec^2a / cos^2a - 1 / cos^2a + sec^2a / cos^2a - 2 - sec^2a + 1 + cos^2a

Simplifying:
(sec^2a + sec^2a - 1) / cos^2a - 2 - sec^2a + cos^2a

Further simplification:
(2sec^2a - 1) / cos^2a - 2 - sec^2a + cos^2a

And that is the simplified expression for (ctg^2a - cos^2a)(1/cos^2a - 1).