ls /proc » spider http://www.lsproc.com/blog lsproc.com Fri, 18 Nov 2011 09:22:52 +0000 en hourly 1 http://wordpress.org/?v=3.3.1 python 抓取页面 http://www.lsproc.com/blog/python_spider/ http://www.lsproc.com/blog/python_spider/#comments Mon, 01 Feb 2010 06:34:31 +0000 lostsnow http://www.lsproc.com/blog/?p=363 Continue reading →]]> 转载时请标明文章原始出处和作者信息, 作者: lostsnow.
http://www.lsproc.com/blog/python_spider/

#coding=utf-8

import sys
import urllib2
import gzip
import StringIO

# 页面url
url = "http://china.toocle.com/company/show/pdetail--1000436--10532651.html"
# 页面编码
page_encode = "gbk"

request = urllib2.Request(url)
request.add_header("Accept-encoding", "gzip")
usock = urllib2.urlopen(request)
page = usock.read()
# 处理gzip过的页面
if usock.headers.get('content-encoding', None) == 'gzip':
    page = gzip.GzipFile(fileobj=StringIO.StringIO(page)).read()

# 转unicode(gbk/utf8)
if not isinstance(page, unicode):
    page = unicode(page, page_encode)

print(page)

-- EOF --

Related Posts

]]> http://www.lsproc.com/blog/python_spider/feed/ 0