Joel, that's a very interesting question you're asking and it is not a simple answer. The problem is there is no one answer.
First, the diluted field as you mention. Many more races spread out the field. Second, there seems to be a much greater focus on participation than competition. Even though there are more people in many of these events, many more are only interested in finishing, not concerned with time or place. Why is this? I have a theory I could get into but I won't unless someone really wants me to.
I think these are the two big issues but they are far from the only issues. The good news is things do seem to be changing. Fields are getting a bit more competitive than they were 5-10 years ago in at least some races. Anyone who cares about competitive running should be hoping that this very recent trend continues and even accelerates, overtaking the trend toward non-competitive “races”.