Broncos fan here...
I think the general consensus on our boards is that NE is our toughest challenge, by far. Whenever you have a hall of fame coach orchestrating a squad led by a hall of fame QB with 3-4 pro bowl level targets to throw at, they are going to scare opposing fans and put up points. Our defense will give yards and points. Still, Denver is a much different unit than what you seen last year, or even earlier this season... not just in terms of the main guys on the field, but how they have grouped together as a unit in a new scheme. We now have 4 CBs who could start on most teams, something very few clubs can boast about, and the safety play has picked up. But obviously guys like Welker and your TEs present matchup nightmares for anyone. I have a lot of concerns about Denvers ability to match up because of Brady's ability to hit intermediate and short throws near instantly (rendering our elite pass rush irrelevant), but if there's 2 or 3 teams that can challenge you in the secondary and hit Brady enough, I think we're one.
I wouldn't discount what Denver has done since the first few games- we played Tampa Bay, Baltimore (admittedly injury depleted, but even against the relatively healthy offense we dominated), San Diego (not great but those divisional rival games are always very tough, even KC in Arrowhead). We can't control our schedule, but in almost every win, we did it pretty convincingly. Early in the season when we lost a few games, the offense still did not have an identity and the chemistry Manning had with other guys was terrible.