HtmlAgilityPack和身份validation

如果给定一个特定的url,我有一个获取id和xpath的方法。 如何通过请求传递用户名和密码,以便我可以抓取需要用户名和密码的url?

using HtmlAgilityPack; _web = new HtmlWeb(); internal Dictionary GetidsAndXPaths(string url) { var webidsAndXPaths = new Dictionary(); var doc = _web.Load(url); var nodes = doc.DocumentNode.SelectNodes("//*[@id]"); if (nodes == null) return webidsAndXPaths; // code to get all the xpaths and ids 

我应该使用Web请求获取页面源,然后将该文件传递给上面的方法吗?

 var wc = new WebClient(); wc.Credentials = new NetworkCredential("UserName", "Password"); wc.DownloadFile("http://somewebsite.com/page.aspx", @"C:\localfile.html"); 

HtmlWeb.Load有许多重载,它们接受NetworkCredential的实例,或者您可以直接传递用户名和密码。

 Name // Description Public method Load(String) //Gets an HTML document from an Internet resource. Public method Load(String, String) //Loads an HTML document from an Internet resource. Public method Load(String, String, WebProxy, NetworkCredential) //Loads an HTML document from an Internet resource. Public method Load(String, String, Int32, String, String) //Loads an HTML document from an Internet resource. 

您不需要传入WebProxy实例,也可以传入系统默认实例。

或者,您可以连接HtmlWeb.PreRequest并设置请求的凭据。

 htmlWeb.PreRequest += (request) => { request.Credentials = new NetworkCredential(...); return true; };