BASIC認証が必要なサイトの収集がうまくいかない

Thanks for the details – I think the real culprit is this setting, for a non-intuitive reason:

[xml]3[/xml]

Indeed, if I lower my max-retries to 3 I can reproduce the problem.

So a quick workaround: increase your max-retries to 4. You’ll still be running a bit close to the edge – a momentary problem affecting DNS/robots/URI fetching might still push it over the limit – but in a usual situation you’ll succeed.

Another workaround: any other URI against the same site scheduled first would trigger the DNS and robots tries – so when the authentication-needing URI comes up, it would have all its tries left.