Does anyone know how the MBTI test works?

The test aren't all that accurate in determining one's mbti type. Also, nobody will posses 100% of any type's traits. A more efficient way of identifying your main type is to thoroughly study the eight cognitive functions. For a very simplified description of them, look for my post here (ignore the irrelevant info): https://answers.yahoo.com/question/index?qid=20141015142032AA2ra9O

ISFJ: dominant Si, auxiliary Fe, tertiary Ti, inferior Ne

Introverted sensing dominant ISFJ lives in world of nostalgia, sentiments, impressions, and memories.

Stereotypically speaking...

The ISFJ has an excellent ability to recall past sensory facts and experiences in accurate detail, giving him an anchor through which to robustly grasp the physical aspects of reality (dominant Si). He is prone to become nostalgic and sentimental. He feels at ease in a harmonious social atmosphere, and values etiquette such as politeness, courtesy, etc. (auxiliary Fe). He is attuned to the mood of others. Although he understands the unspoken expectations and nuances for cooperative behavior, he has relative difficulty in logically dissecting a situation in a purely rational, detached manner (tertiary Ti). He has an even harder time embracing change, thinking outside the box, and entertaining possibilities, preferring the security of his memories (inferior Ne).

ISFP, ISTP: https://answers.yahoo.com/question/index?qid=20140927183017AAu1dET

ISTJ: https://answers.yahoo.com/question/index?qid=20140926124944AAWRnEf

(look for my responses)

Tests for it aren't very accurate. It's better to use the cognitive functions, since that's what the types break down into. Contrary to popular belief, you can't be, say, ISxJ or ISxP because of the functions. ISTJs have Si (introverted sensing), Te (extroverted intuition), Fi (introverted feeling), and Ne (extroverted intuition), and while ISFJs have Si and Ne in the same spots, they have Fe instead of Te and Ti instead of Fi. SPs and SJs are also completely different, since SJs have Si while SPs have Se. Here's a list of the functions with brief descriptions: http://www.myersbriggs.org/my-mbti-personality-type/understanding-mbti-type-dynamics/the-eight-function-attitudes.asp To put it simply, your four functions go in order from strongest to weakest. Also, introverts always have an introverted function as their first and third functions (and extroverts extroverted functions in these spots), and an extroverted function as their second and fourth functions.
Also, I personally found the fence building thing at the bottom to be useful for determining my own functions. I don't know if you'll find it helpful, but here's the link anyway: http://web.archive.org/web/20130416184706/http://www.famoustype.com/functions.htm
Once you determine your functions, look up the type that has those functions and see if the personality profile that goes with it sounds like you.

What you're finding isn't that uncommon. When a person's score causes them to be either one or the other types, it just means you could be either. Look up the information for both ISFJ and ISTJ and know both are fitting for you.

I'm surprised you got a different result when you took the test the second time. This is rare. People usually are one type throughout their lives. Some show a slight change over the course of their lives.

Have you taken the official version of the test? It's longer than any of the free ones and it would likely give you a more firm answer.