Is brightness just related to the number of photons per time and independent of the photons' frequency?

I think you need to distinguish between light "intensity" and "brightness". The two terms mean differently.

Intensity of light refers to the amount of energy falling onto a unit area in a unit time. The "unit"used is watts/metre^2 (or Joules per second per metre^2). In classical electromagnetic theory, the light intensity relates to the amplitude of the light wave. But in quantum theory, the intensity of light (I) is given by the relation: I = n.h.f where n is the no. of photons incident normally onto a unit area surface in a unit time, h is Plank's Constant, and f is the frequency of light. Hence, the intensity depends on both n and f.

Brightness, however, is more complicated, as it relates to the sensitivity of human eyes to different colours of light. Human eyes are most sensitive to light of wavelength around 550 nm (i.e. the yellow-green region of the visible spectrum), and less sensitive to wavelengths towards the two ends of the visible spectrum. In other words, given two colour-lights, say red and green, of the same intensity, it would appear to human eyes that the green light is brighter than the red light.