UrlLib模块
urllib模块是适用于一个简单的网络数据获取和处理(不能处理有关验证和cookie等功能),官方文档的介绍主要是给了两个文档链接和三个标准:
- 基本网络名词的介绍和格式定义规则
文档地址:http://www.w3.org/pub/WWW/Addressing/Overview.html
- HTTP规范文档及其错误代码
文档地址:http://www.w3.org/pub/WWW/Protocols/
- 相关标准和规范
- RFC1808(相对URL规范)
- RFC1738(URL标准)
- RFC1630(URL规范)
urllib模块主要的功能就是提供了一个对www协议访问的接口urlopen(),让我们可以直接通过url对网页进行读取,操作比较简单,可以如同操作文件一样读取文件,但也仅限读取,操作相对还是比较简单.
如下,为一个简单的使用urllib模块对网页访问的操作:
import sys,urllib
def accessByUrllib():
print "access html by urllib"
url="https://www.enjoytoday.cn/posts/362"
page=urllib.urlopen(url)
#print "page html:%s" % page.read() #返回html文件内容
print "\npage info:%s " % page.info() #返回基本信息(头信息)
print "\npage code:%s"% page.getcode() #http请求响应码,若非http则返回None
print "\npage url:%s"%page.geturl() #请求url,真实访问url(重定问情况下和url不相同)
print "\npage headers: %s"%page.headers #返回头信息
#将html写入本地,两种方法
url_file=open("./test.html",'wb+')
url_file.write(page.read())
url_file.close()
#使用urllib模块直接写入
urllib.urlretrieve(url,"./test2.html")
if __name__ == "__main__":
accessByUrllib()
如下为返回数据(由于html文件内容较多,所以将其屏蔽):
access html by urllib
page info:Date: Thu, 09 Nov 2017 14:27:43 GMT
Server: Apache/2.4.7 (Ubuntu)
Set-Cookie: JSESSIONID=6266B422E20D2F229556E51B9C4BCF8B.ajp13_worker;path=/;HttpOnly
Content-Language: en-US
Vary: Accept-Encoding
Connection: close
Content-Type: text/html;charset=UTF-8
page code:200
page url:https://www.enjoytoday.cn/posts/362
page headers: Date: Thu, 09 Nov 2017 14:27:43 GMT
Server: Apache/2.4.7 (Ubuntu)
Set-Cookie: JSESSIONID=6266B422E20D2F229556E51B9C4BCF8B.ajp13_worker;path=/;HttpOnly
Content-Language: en-US
Vary: Accept-Encoding
Connection: close
Content-Type: text/html;charset=UTF-8
Process finished with exit code 0
如上,为urllib 中的基本对网页的处理,如下介绍一些简单的请求处理以及url的处理操作
默认的urllib采用GET方式进行请求操作,如下介绍如何指定请求方式:
def requestMethod(method="GET"):
'''
默认的urllib是以GET方式进行请求,可以通过使用urlencode()方法对其实现post请求,传入数据为类字典类型,key:vaule格式,但
默认参数类型为标准的application/x-www-form-urlencoded表单格式,不可更改
:param method: 指定请求方式:GET,POST
:return:
'''
url="https://www.enjoytoday.cn/posts/362"
if method:
if method=="GET":
page=urllib.urlopen(url)
print "get request return:%s"%page.read()
elif method=="POST":
reload(sys)
sys.setdefaultencoding('utf-8') #解决 'ascii' codec can't encode characters问题,注意需要在调用setdefaultencoding()之前先reload(sys)不然会抛出没有该方法.
dic={"name":u'飞云不在线',"sex":"man"}
params=urllib.urlencode(dic)
page=urllib.urlopen("%s?%s"%(url,params))
print "post request return:%s" %page.read()
else:
raise Exception("pass param format error.")
else:
raise Exception("param must not be Empty.")
如上为方法的基本使用.
def transferUrl():
'''
url 和本地路径转化
:return:
'''
path = "D://python/test/tt.txt"
url=urllib.pathname2url(pathname=path)
print "path transfer to url:%s" % url
print "url transfer to path:%s" %urllib.url2pathname(url)
def encodeStr():
'''
字符串的编码和解码
:return:
'''
s="飞云不在线%%_345&"
quote_s=urllib.quote(s)
quote_plus_s=urllib.quote_plus(s)
unquote_s=urllib.unquote(quote_s)
unquote_plus_s=urllib.unquote_plus(quote_plus_s)
#encode
print "quote encode:%s"%quote_s
print "quote plus encode:%s"%quote_plus_s
#decode
print "quote decode:%s" % unquote_s
print "quote plus decode:%s" % unquote_plus_s
#运行结果如下:
quote encode:%E9%A3%9E%E4%BA%91%E4%B8%8D%E5%9C%A8%E7%BA%BF%25%25_345%26
quote plus encode:%E9%A3%9E%E4%BA%91%E4%B8%8D%E5%9C%A8%E7%BA%BF%25%25_345%26
quote decode:飞云不在线%%_345&
quote plus decode:飞云不在线%%_345&