【Day 12】- 这页爬完了，爬下一页。PTT 爬好爬满！(实战 PTT 爬虫 2/3)

前情提要

前一篇文章带大家写了能爬取 PTT 当前页面文章的爬虫，且透过携带已满 18 岁的 cookies 避免验证 18 岁。

开始之前

本篇将继续带各位写 PTT 爬虫，今天会将持续爬取的部分做完，技术上来说就是每爬取到一个页面就也去爬取下一页的网址，爬取完文章後再发一个 requests 到下一页的网址重复此动作。

预期效果

当前页面全部文章爬取并跳过 18 岁检定(Day11 已实作)

爬取下一页网址

向下一页发送请求

重复此循环 n 次

实作

这是昨天的程序码。

import requests
from bs4 import BeautifulSoup
url = 'https://www.ptt.cc/bbs/Gossiping/index.html'
cookies = {
    'over18': '1'
}
resp = requests.get(url, cookies=cookies)
soup = BeautifulSoup(resp.text, 'html5lib')
arts = soup.find_all('div', class_='r-ent')
for art in arts:
    title = art.find('div', class_='title').getText().strip()
    link = 'https://www.ptt.cc' + \
        art.find('div', class_='title').a['href'].strip()
    author = art.find('div', class_='author').getText().strip()
    print(f'title: {title}\nlink: {link}\nauthor: {author}')

我们能先将取得 resp 以及爬取当前页面文章的功能写成一个 function。

import requests
from bs4 import BeautifulSoup
url = 'https://www.ptt.cc/bbs/Gossiping/index.html'

def get_resp():
    cookies = {
        'over18': '1'
    }
    resp = requests.get(url, cookies=cookies)
    if resp.status_code != 200:
        return 'error'
    else:
        return resp

def get_articles(resp):
    soup = BeautifulSoup(resp.text, 'html5lib')
    arts = soup.find_all('div', class_='r-ent')
    for art in arts:
        title = art.find('div', class_='title').getText().strip()
        link = 'https://www.ptt.cc' + \
            art.find('div', class_='title').a['href'].strip()
        author = art.find('div', class_='author').getText().strip()
        print(f'title: {title}\nlink: {link}\nauthor: {author}')

resp = get_resp()
get_articles(resp)

接下来，我们用开发工具找一下下一页(PTT 中会自动到最新，因此需找下一页)的网址在哪边。

可以发现到下一页网址出现在 class 为 btn-group btn-group-paging 的 div 下的第二个子元素 a 的 href 属性中。

知道哪在哪边後，我们可以加上一些 code 让程序能爬取到该网址，这边能直接使用 Css Selector 直接选取到该元素，若不知如何在开发工具上取得某元素的 Css Selector 可以去看这篇的後面部分【Day 08】- 有着资料清洗功能的 Requests-HTML。

next_url = 'https://www.ptt.cc' + soup.select('#action-bar-container > div > div.btn-group.btn-group-paging > a:nth-child(2)')[0]['href']

现在能取得下一页的文章网址了。因此，我们能写个回圈让它能够重复爬取吧。

import requests
from bs4 import BeautifulSoup

def get_resp(url):
    cookies = {
        'over18': '1'
    }
    resp = requests.get(url, cookies=cookies)
    if resp.status_code != 200:
        return 'error'
    else:
        return resp

def get_articles(resp):
    soup = BeautifulSoup(resp.text, 'html5lib')
    arts = soup.find_all('div', class_='r-ent')
    for art in arts:
        title = art.find('div', class_='title').getText().strip()
        link = 'https://www.ptt.cc' + \
            art.find('div', class_='title').a['href'].strip()
        author = art.find('div', class_='author').getText().strip()
        print(f'title: {title}\nlink: {link}\nauthor: {author}')
    # 利用 Css Selector 定位下一页网址
    next_url = 'https://www.ptt.cc' + \
        soup.select_one(
            '#action-bar-container > div > div.btn-group.btn-group-paging > a:nth-child(2)')['href']
    return next_url

# 当执行此程序时成立
if __name__ == '__main__':
    # 第一个页面网址
    url = 'https://www.ptt.cc/bbs/Gossiping/index.html'
    # 先让爬虫爬 10 页
    for now_page_number in range(10):
        resp = get_resp(url)
        if resp != 'error':
            url = get_articles(resp)
        print(f'======={now_page_number+1}/10=======')
''' 已将部分不必要内容删除
title: [问卦] 30岁的魔法该学冰系还是火系好？
link: https://www.ptt.cc/bbs/Gossiping/M.1632417857.A.562.html
author: ejo3and503
title: Re: [新闻] 清大设「後医系」 医师公会怒：医师浮滥
link: https://www.ptt.cc/bbs/Gossiping/M.1632417931.A.799.html
author: driftingjong
title: [问卦] 翁达瑞，是一群人吗？
link: https://www.ptt.cc/bbs/Gossiping/M.1632417965.A.716.html
author: LEDG
title: [问卦] 几岁开始刷Leetcode才有竞争力？
link: https://www.ptt.cc/bbs/Gossiping/M.1632418019.A.DE6.html
author: dixitdeus
title: Re: [爆卦] 美国教授踢爆高虹安大数据招牌造假
link: https://www.ptt.cc/bbs/Gossiping/M.1632418069.A.106.html
author: zombiechen
title: [新闻] 3+11没纪录？ 陈时中：再问100遍就是没有
link: https://www.ptt.cc/bbs/Gossiping/M.1632418088.A.8E8.html
author: shinmoner
title: [问卦] 为啥女生爱看耽美，男生没那麽爱看百合?
link: https://www.ptt.cc/bbs/Gossiping/M.1632418139.A.B6D.html
author: s9234032
title: [问卦] 在美国卖寿司是不是暴利？
link: https://www.ptt.cc/bbs/Gossiping/M.1632418248.A.F80.html
author: hwang1460
title: [问卦] 早九上课现在还没睡怎办？
link: https://www.ptt.cc/bbs/Gossiping/M.1632418281.A.9F1.html
author: WeiU
title: [问卦] 五倍券可以拿来吃鱼喝茶吗
link: https://www.ptt.cc/bbs/Gossiping/M.1632418285.A.C57.html
author: blessbless
title: Re: [问卦] 台湾为什麽不盛行餐酒文化?
link: https://www.ptt.cc/bbs/Gossiping/M.1632418312.A.A1F.html
author: noway
title: [问卦] 周杰伦是什麽时候开始走下坡的？
link: https://www.ptt.cc/bbs/Gossiping/M.1632418477.A.959.html
author: boboken
title: [公告] 八卦板板规(2021.05.11)
link: https://www.ptt.cc/bbs/Gossiping/M.1620716589.A.F0C.html
author: arsonlolita
title: [协寻] 求行车纪录画面(9/16上午内湖游戏橘子旁)
link: https://www.ptt.cc/bbs/Gossiping/M.1631948458.A.D73.html
author: umbrella0613
title: [公告] 中秋节我家兔兔辣麽口爱活动投票
link: https://www.ptt.cc/bbs/Gossiping/M.1632244429.A.388.html
author: ubcs
title: [协寻] 橘猫咪噜快点回家！（大安区）
link: https://www.ptt.cc/bbs/Gossiping/M.1632305989.A.5E0.html
author: k020231310
title: [协寻] 新北芦洲区环提大道行车记录器
link: https://www.ptt.cc/bbs/Gossiping/M.1632345107.A.8AF.html
author: anpep
=======1/10=======
title: Re: [新闻] 清大设「後医系」 医师公会怒：医师浮滥
link: https://www.ptt.cc/bbs/Gossiping/M.1632417296.A.2F5.html
author: organize222
'''

此时我们的程序会遇到一些问题，可以看出它没有文章连结，猜测应该是文章删除了。

Traceback (most recent call last):
  File "c:\Users\50205\OneDrive\桌面\a\test.py", line 41, in <module>
    url = get_articles(resp)
  File "c:\Users\50205\OneDrive\桌面\a\test.py", line 22, in get_articles
    art.find('div', class_='title').a['href'].strip()
TypeError: 'NoneType' object is not subscriptable

我们加个判断式，即可解决这个问题。

title = art.find('div', class_='title').getText().strip()
if not title.startswith('(本文已被删除)'):
    link = 'https://www.ptt.cc' + \
        art.find('div', class_='title').a['href'].strip()

整体程序码

import requests
from bs4 import BeautifulSoup

def get_resp(url):
    cookies = {
        'over18': '1'
    }
    resp = requests.get(url, cookies=cookies)
    if resp.status_code != 200:
        return 'error'
    else:
        return resp

def get_articles(resp):
    soup = BeautifulSoup(resp.text, 'html5lib')
    arts = soup.find_all('div', class_='r-ent')
    for art in arts:
        title = art.find('div', class_='title').getText().strip()
        if not title.startswith('(本文已被删除)'):
            link = 'https://www.ptt.cc' + \
                art.find('div', class_='title').a['href'].strip()
        author = art.find('div', class_='author').getText().strip()
        print(f'title: {title}\nlink: {link}\nauthor: {author}')
    # 利用 Css Selector 定位下一页网址
    next_url = 'https://www.ptt.cc' + \
        soup.select_one(
            '#action-bar-container > div > div.btn-group.btn-group-paging > a:nth-child(2)')['href']
    return next_url

# 当执行此程序时成立
if __name__ == '__main__':
    # 第一个页面网址
    url = 'https://www.ptt.cc/bbs/Gossiping/index.html'
    # 先让爬虫爬 10 页
    for now_page_number in range(10):
        print(f'crawing {url}')
        resp = get_resp(url)
        if resp != 'error':
            url = get_articles(resp)
        print(f'======={now_page_number+1}/10=======')