如何将cookie传递给HtmlAgilityPack或WebClient？

我使用此代码登录：

CookieCollection cookies = new CookieCollection(); HttpWebRequest request = (HttpWebRequest)WebRequest.Create("example.com"); request.CookieContainer = new CookieContainer(); request.CookieContainer.Add(cookies); HttpWebResponse response = (HttpWebResponse)request.GetResponse(); cookies = response.Cookies; string getUrl = "example.com"; string postData = String.Format("my parameters"); HttpWebRequest getRequest = (HttpWebRequest)WebRequest.Create(getUrl); getRequest.CookieContainer = new CookieContainer(); getRequest.CookieContainer.Add(cookies); getRequest.Method = WebRequestMethods.Http.Post; getRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0"; getRequest.AllowWriteStreamBuffering = true; getRequest.ProtocolVersion = HttpVersion.Version11; getRequest.AllowAutoRedirect = true; getRequest.ContentType = "application/x-www-form-urlencoded"; byte[] byteArray = Encoding.ASCII.GetBytes(postData); getRequest.ContentLength = byteArray.Length; Stream newStream = getRequest.GetRequestStream(); newStream.Write(byteArray, 0, byteArray.Length); newStream.Close(); HttpWebResponse getResponse = (HttpWebResponse)getRequest.GetResponse(); using (StreamReader sr = new StreamReader(getResponse.GetResponseStream(), Encoding.GetEncoding("windows-1251"))) { doc.LoadHtml(sr.ReadToEnd()); webBrowser1.DocumentText = doc.DocumentNode.OuterHtml; } 

然后我想使用HtmlWeb（HtmlAgilityPack）或Webclient将HTML解析为HtmlDocument（HtmlAgilityPack）。

我的问题是，当我使用时：

 WebClient wc = new WebClient(); webBrowser1.DocumentText = wc.DownloadString(site); 

要么

 doc = web.Load(site); webBrowser1.DocumentText = doc.DocumentNode.OuterHtml; 

登录消失所以我想我必须以某种方式通过cookie ..任何建议？

Answer 1

检查HtmlAgilityPack.HtmlDocument Cookies

以下是您正在寻找的示例（语法未经过100％测试，我只修改了一些我经常使用的类） ：

 public class MyWebClient { //The cookies will be here. private CookieContainer _cookies = new CookieContainer(); //In case you need to clear the cookies public void ClearCookies() { _cookies = new CookieContainer(); } public HtmlDocument GetPage(string url) { HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url); request.Method = "GET"; //Set more parameters here... //... //This is the important part. request.CookieContainer = _cookies; HttpWebResponse response = (HttpWebResponse)request.GetResponse(); var stream = response.GetResponseStream(); //When you get the response from the website, the cookies will be stored //automatically in "_cookies". using (var reader = new StreamReader(stream)) { string html = reader.ReadToEnd(); var doc = new HtmlDocument(); doc.LoadHtml(html); return doc; } } } 

以下是您使用它的方式：

 var client = new MyWebClient(); HtmlDocument doc = client.GetPage("http://somepage.com"); //This request will be sent with the cookies obtained from the page doc = client.GetPage("http://somepage.com/another-page"); 

注意：如果您还想使用POST方法，只需使用POST逻辑创建类似于GetPage的方法，重构类等。

Answer 2

这里有一些建议： 将CookieContainer与WebClient类一起使用

但是，使用HttpWebRequest并在CookieContainer设置cookie可能更容易：

• HTTPWebRequest和CookieContainer
• http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.cookiecontainer.aspx

代码看起来像这样：

  // Create a HttpWebRequest HttpWebRequest request = (HttpWebRequest)WebRequest.Create(getUrl); // Create the cookie container and add a cookie request.CookieContainer = new CookieContainer(); // Add all the cookies foreach (Cookie cookie in response.Cookies) { request.CookieContainer.Add(cookie); } 

第二件事是您不需要再次下载该站点，因为您已经从Web响应中获得了该站点，并且您将其保存在此处：

 HttpWebResponse getResponse = (HttpWebResponse)getRequest.GetResponse(); using (StreamReader sr = new StreamReader(getResponse.GetResponseStream(), Encoding.GetEncoding("windows-1251"))) { webBrowser1.DocumentText = doc.DocumentNode.OuterHtml; } 

您应该能够只使用HTML并使用HTML Agility Pack解析它：

 HtmlDocument doc = new HtmlDocument(); doc.LoadHtml(webBrowser1.DocumentText); 

应该这样做…… 🙂

Answer 3

尝试在本地缓存先前响应中的cookie，并按如下方式重新发送每个Web请求：

 private CookieCollection cookieCollection; ... parserObject = new HtmlWeb { AutoDetectEncoding = true, PreRequest = request => { if (cookieCollection != null) cookieCollection.Cast() .ForEach(cookie => request.CookieContainer.Add(cookie)); return true; }, PostResponse = (request, response) => { cookieCollection = response.Cookies; } };