How do Chinese dictionaries work if there aren't letters but characters that represent words?

There are two different ways of organizing a dictionary in Chinese.
The first is by radicals. A radical is a part of a character (or sometimes a whole character) that makes up the other characters. If it is just organized by this, there will be a radical index at the front that first lists (going from least amount of strokes to most) the page in the index the radical is on. Once turning to that page, it will show all the characters in the dictionary (again in ascending stroke amounts). That will have the page of the dictionary that the character is in.
The second way is to use the romanization (pinyin) of characters. Pinyin is the official way to spell the sounds of the characters by using the Latin based alphabetic. This will be arranged first by alphabetically, then by tone (first tone to fourth), - this means that for characters that have the same pinyin, first tone characters would be first in the listing, then for those characters that have the same pinyin and tone, by number of strokes - again, the first character will be the one with the least amount of strokes. This way of organization is very common, especially in conjunction with Chinese to other language dictionaries. However, even these dictionaries will have a radical index.

rbwtexan is right.
they are grouped by similar characters.
and within these characters, they are further grouped by how many strokes they have (chinese characters can be counted in strokes)

they also have chinese dictionaries that organize the words by Hanyu Pinyin. these can be arranged alphabetically.

Chinese characters are grouped into families of like characters (contain similar markings with the character). Chinese characters are grouped together in these families in the dictionary.

In mainland China, most of the modern Chinese dictionaries or Chinese-English dictionaries are organized in the alphabetically order of the pinyin spelling ( Pinyin is the romanized phonetic system, you can learn more about Pinyin at http://en.wikipedia.org/wiki/Pinyin. They also have the index of radicals in case you do not have to pronounce the words.

In old Chinese dictionaries or those published in Taiwan and Hong Kong, they use radical stroke number order (word with one stroke, two strokes, ...) to list the word entries.
Learn more about radical at
http://en.wikipedia.org/wiki/Radical_%28Chinese_character%29

they base it on 'root words(characters)' ans then is further subdivided into the number of strokes of the non-root word part of the character that is left. usually the root word/character is the left part of the word.