Which sentence is correct?

I'd say none of them, because I don't think I've ever heard a native English speaker describe anything as "less small". It may be correct, technically, but it sounds awkward and odd. Most people would say "not as small" or "bigger". If forced to choose between those three, then the first is the only acceptable one.

You can write:

This house isn't as small as the other one.

This house is bigger than the other one.

1

2. This house is actually less smaller compared to the other one.

Only the first is even close to correct, Here is the thing, though: less small does not mean smaller. It means the opposite of smaller. More small=smaller. This makes me question whether your words accurately express the idea that you wish to express.

I would not use "compared to", I would use relative to, or in comparison with, or even the obvious and expected "than".

A thing could be less smaller (not as much more small) than another thing, but only relative to a third thing. This construct would not normally be a very good way to express the idea, though. It is better to say that both are smaller (than the third thing) but one of the two is less small than the other (or the one is even smaller than the other).

Ugh... None of them.

It would be easier on the ears to say: "This house is (larger/ slightly larger) than the other house."

the third one