Obviously you will need a cage suitable for your mouse. It should have plenty of room and bar spacing small enough it can't escape.
You will need some bedding material, doesn't have to be expensive, the paper type is fine.
It will need a cage water bottle,with the sip tube and bottled water is suggested.
A small ceramic food bowl, I have even used clean jar lids. Various fresh foods and you can get rodent pellets at pets stores. They love sunflower seeds(unsalted) and a little crisp grain bread with a little peanut butter for treats.
Something in the cage for your mouse to hide in, there are different hidyhuts and soft hides.
Given your first pet and new to mice, you might do plenty of reading about keeping pet mice.
http://pet-mice.com/
http://www.drsfostersmith.com/pic/article.cfm?aid=797
After your mouse settles into its new home a couple days and is comfortable, then you can start to interact with your pet, train it, and bond with it.
http://exoticpets.about.com/od/careofmice/a/mousetame.htm
There are a number of videos on Youtube 'how to tame and bond with your pet mouse'
You will find pet mice and rats, to be rather easy to care for and not expensive.
They do take a little work like cleaning their cage, and seeing they have fresh food and water daily.
Spend a lot of time with your pet, they come to like their keepers attention and very social animals.
My first mouse I even trained to come to a bell for treats. Ring a little bird cage bell every time you feed or give them treats, In time they learn the bell means good. Handy if your pet should escape, ring the bell, they come running. Have patience, in time your pet will learn to trust you, and you will learn to trust it and know its behavior.
Good luck with your new pet.