第一部分 创建爬虫

第一章 初见网络爬虫

1.1 网络连接

# py3 urllib
from urllib.request import urlopen

url = "http://www.baidu.com"
html = urlopen(url)
print(html.read())

官方文档：https://docs.python.org/3/library/urllib.html

1.2 BeautifulSoup

安装：

pip install beautifulsoup4

官方中文文档：
https://www.crummy.com/software/BeautifulSoup/bs4/doc.zh/index.html

推荐使用虚拟环境： Mac安装python环境以及虚拟环境

# py3 BeautifulSoup
from urllib.request import urlopen
from bs4 import BeautifulSoup

url = "http://www.baidu.com"
html = urlopen(url)

soup = BeautifulSoup(html.read()) # "html.parser"

print(soup.title)
# <title>百度一下，你就知道</title>

增加异常处理

# 异常处理
# 1、 服务器不存在
# 2、 网页不存在

# py3
from urllib.request import urlopen
from bs4 import BeautifulSoup

url = "http://www.baidu.com"

try:
    html = urlopen(url)

except Exception as e:
    print(e)

else:
    soup = BeautifulSoup(html.read(), "html.parser")
    print(soup.title)  # <title>百度一下，你就知道</title>
    tag = soup.xxxxx
    print(type(tag))  # <class 'NoneType'>

finally:
    pass

重新组织代码

# py3
from urllib.request import urlopen
from urllib.error import HTTPError, URLError
from bs4 import BeautifulSoup

def getTitle(url):
    """
    异常处理
    1、 服务器不存在
    2、 网页不存在
    3、 标签不存在
    """
    try:
        html = urlopen(url)
    except (HTTPError, URLError) as e:
        return None

    try:
        soup = BeautifulSoup(html.read(), "html.parser")
        title = soup.head.title
    except AttributeError as e:
        return None

    return title


url = "http://www.baidu.com"
title = getTitle(url)
if title == None:
    print("title is None")
else:
    print(title)
    # < title > 百度一下，你就知道 < / title >

第二章 复杂HTML解析

2.1 标签过滤

find, find_all 参数
name=None,      标签，或运算
attrs={},       属性，与运算
recursive=True, 递归 
text=None,      文本
limit=None,     范围限制  find <=> find_all(limit=1) 
**kwargs,       关键字 class_

url = "http://www.pythonscraping.com/pages/warandpeace.html"
html = urlopen(url)
soup = BeautifulSoup(html, "html.parser")
name_list = soup.find_all("span", {"class": "green"})
for name in name_list:
    print(name.get_text())
h1 = soup.find(text="Chapter 1")
print(h1)

2.2 BeautifulSoup对象

四个对象：
BeautifulSoup   文档
Tag             标签
NavigableString 标签文字
Comment         注释

2.3 导航树

# 孩子和后代
children 
descendants 

# 兄弟
next_siblings 
previous_siblings 
next_sibling
previous_sibling

# 父亲
parent
parents

url = "http://www.pythonscraping.com/pages/page3.html"
html = urlopen(url)
soup = BeautifulSoup(html, "html.parser")

table = soup.find("table", {"id": "giftList"})

from bs4.element import Tag

# for tr in table.children:
#     if isinstance(tr, Tag):
#         for td in tr.children:
#             print(td.get_text(), end="|")
#     print("\n")

img = table.find("img", {"src": "../img/gifts/img1.jpg"})
price = img.parent.previous_sibling.get_text()
print(price)

2.4 正则表达式

官方文档：https://docs.python.org/3/library/re.html
参考文章：Python编程：re正则库

正则邮箱：
[A-Za-z0-9\._+]+@[A-Za-z0-9]+\.(com|cn|org|edu|net)

import re

url = "http://www.pythonscraping.com/pages/page3.html"
html = urlopen(url)
soup = BeautifulSoup(html, "html.parser")
regex = re.compile(r"\.\.\/img\/gifts\/img.*\.jpg")
imgs = soup.find_all("img", {"src": regex})
for img in imgs:
    print(img.get("src")) # 获取属性 attrs

2.5 lambda表达式

# 返回True 或 False
tags = soup.find_all(lambda tag: len(tag.attrs) == 2)
print(tags)

2.6 html解析库

lxml： http://lxml.de/
html.parser：https://docs.python.org/3/library/html.parser.html