have wasted two days wondering that had gone wrong with my plugin that is doing a small extra authentication because I didn’t feel like diving deep into code to figure it out, but once I did I got the answer really fast – the authentication filter has some unexpected weirdness that is unlike almost all other wordpress filters.
It is supposed to return a valid user but the initial value passed into it from the wp_authenticate function is NULL, and not as you might a valid user or error. The actual user validation is done by a core filter with a priority of 20. There is also another core filter with priority 99 that is denying login to users that were marked as spammers.
bottom line: if you want to implement a different authentication user/password scheme you need to hook your function on a priority which is less then 20. If you want to just enhance the core authentication use priority 21-98, and if you prefer to let wordpress reject network spammers before your function is called use priority of 100 and above.