is there any romantic place to stay for weekend in Tri-State Area? NY, NJ, CT, PA?

There's lots of Bed and Breakfasts in NY. You could go to Cold Spring, that's a cute little town right by the Hudson River. There's a B&B there called the Pig Hill Inn.
You could also try Rhinebeck, NY, which is a little further north, but also by the Hudson. There are also some nice B&Bs there.
In CT, you could try to go Mystic Seaport.
In NJ, you could try Cape May, which is by the ocean and has lots of Victorian style homes and also many Bed and Breakfasts.

Depending on the time of year, Montauk is beautiful and romantic. If you have more money, you can get a hotel room in Fire Island - there's nothing more romantic than the beach at night.