Although I am not sure what you call staph test is, from your description I assume it is based on antibody agglutination. So the protein is an antibody which will specifically bind with Staph. and it has a shape of a Y with the binding sites located at the tip of the 2 branches of the Y, so it can bind to site located on more than one Staph cell. And each staph cell can be attached by many antibodies, so in presence of a lot of antibodies, Staph gets attached in groups by the antibodies and you got visible clamps. But if you have other bacteria, nobody gets attached and the negative result is a simple homogeneous looking bacterial suspension.

I hope this helps.

Just to add something to previous post: the mechanism described by Canalon is often enhanced by linking IgG antibodies onto the surface of some kind of particles, e.g. latex beads. This allows one "antibody bead" to bind together much more antigen (and thus pathogens), and while doing that also other such particles – this in turn forms agglutination visible by naked eye. Sole IgG antibodies are relatively weakly agglutining (is this a proper term? :)), and thus this little tweak greatly improves the result. Furthermore, the particles are usually coloured, which makes it easier to detect even weak agglutination from the background.

I think one could think the antibody/bead combo as a sort of artificial IgM antibodies, which exist as pentamers in the body and thus have more binding sites and are more effective in forming agglutination than IgG. IIRC, IgG is the primary antibody when it comes to opsonising pathogens and thus activating complement or initiating phagocytosis etc. 🙂