First person singular "I" don't add s.?

it's not that "I" is treated as plural; it's just that, in Modern English, the inflectional s is the ending only for the third person singular present. Think of the forms of "to be." We, you, and they ARE; he, she, or it IS; but I AM, not is. Then consider the past tense forms of that verb: WAS in the singular, WERE in the plural.

As for "you," yes, it's always treated as plural in English even when one is addressing just one person. The true second person singular in English is "thou," which is almost obsolete. Its verbs usually end in st or est, except for "thou art." As in some other languages, such as French and German, the second person plural came to be used as a formal way of addressing an individual. In those languages, the singular form still exists as a means of addressing a family member or a close friend, but in English it has dropped out except in some religious contexts.

In short, "I" is not treated as plural--the first person singular just never, in every case but "was," had the same inflectional ending as the third.

add "s" only by he, she and it. Actually, it USED to be add "st" in Old English, but the "t" has fallen away.
Still, that answers your question, because the "st" ending is the same as the German "t" ending for he, she and it - which is where we get it from.