Can't is the short form of cannot
when we are writing formal letter, we should use "cannot", otherwise using "can't" is acceptable.
SO, both of your sentence are correct, but you can use:
1) you cannot talk in the library
2) they can't talk in the library