爬虫crawler -- 虾皮购物

许多厂商、卖家都会想知道自己的商品上架到平台贩售时，商品会排名在哪个位置？
大品牌厂商可能有经费每天派一名人力，定时去查找商品所在的排名；
要是小品牌或一般卖家不太会有经费或时间去做这样的事。
这时就可以透过爬虫程序，每天自动爬取想要查找的商品资讯，轻松、便宜又省事!

本篇会介绍当在虾皮购物官网，搜寻关键字时，如何将所有资料一次蒐集完成。

程序码相当简单，共分为三段：import 套件、爬取资料(主要)、输入关键字。

import 套件

# 计算与转换时间用
from datetime import datetime, timedelta
import time
from pytz import timezone,utc

# 爬虫使用
import re
import requests
from bs4 import BeautifulSoup
import json

爬取资料(主要)

1. 在 headers 的地方加入在 Network 中的'user-agent'及'referer'两个参数
   
2. 接下来的程序都会一层一层的往下执行 (从getShopeekey一路到parseItem)
3. getShopeekey：会接到下一段程序使用者自行输入的关键字
4. parseAllList：先从一组网址中找到关键字搜寻後，总共被搜寻到的笔数，再根据笔数组成所有的商品列表网址。
   e.g. 因虾皮的一个商品列表网址中，会包含50笔资料。假设今天搜寻後的结果是2100笔商品，则商品列表的页数便是(2000/50)+1=41(页)
5. parseList：将每一项商品会在清单中显示的资讯进行爬取(商品代号、名称、促销...)
   
6. parseItem：最後会进到商品内页，把更详细的资讯爬取下来(价格、免运、库存)
   

DOMAIN = 'https://shopee.tw/'

headers = {
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36',
    'referer': 'http://localhost:8888/'
}

def getShopeekey(keyword):
    data={}
    data['search_keyword'] = keyword
    parseAllList(data, keyword)
    
def parseAllList(data, keyword):
    ty_url = DOMAIN + 'api/v2/search_items/?by=relevancy&keyword={}&limit=50&newest=0&order=desc&page_type=search&version=2'.format(keyword)
    resp = requests.get(ty_url, headers=headers).text
    doc = json.loads(resp) 
    
    totalcount = doc['query_rewrite']['ori_totalCount']
    total_pg = (totalcount // 50) + 1
    doc['totalPage'] = total_pg

    for num in range(0, total_pg):
        pg_url = DOMAIN + 'api/v2/search_items/?by=relevancy&keyword={}&limit=50&newest={}&order=desc&page_type=search&version=2'.format(keyword, (num*50))
        data['Page'] = num + 1
        parseList(pg_url, data, num)
        
def parseList(pg_url, data, num):
    print(pg_url)
    resp = requests.get(pg_url, headers=headers).text
    doc = json.loads(resp)
    position = 0
    count = 0
    
    for i in doc['items']:
        productid = i['itemid']
        name = i['name']
        shopid = i['shopid']
        data['Brand'] =i['brand']

        data['PrdCode'] = productid
        article_url = DOMAIN + name + '-i.{}.{}'.format(shopid, productid)
        data['url'] = article_url    
        
        promotion = []
        if not i['ads_keyword'] == None:
            promotion.append('广告')
        if not i['add_on_deal_info'] == None:
            promotion.append(i['add_on_deal_info']['add_on_deal_label'])
        if int(i['show_discount']) > 0:
            promotion.append('折扣')
        if i['service_by_shopee_flag'] > 0:
            promotion.append('24h快速到货')
        
        if num == 0: position = position + 1
        else:
            count = count + 1
            position = (num * 50) + count
        data['position_section'] = position
        data['position_total'] = data['position_section']
        
        parseItem(shopid, productid, promotion, data) 
    
def parseItem(shopid, productid, promotion, data):
    pg_url = DOMAIN + 'api/v2/item/get?itemid={}&shopid={}'.format(productid, shopid)
    time.sleep(1)
    resp = requests.get(pg_url, headers=headers).text
    doc = json.loads(resp)
    
    flavor = 0
    if not len(doc['item']['models']) == 0:
        for i in doc['item']['models']: flavor = flavor + 1
    
    data['Description'] = doc['item']['name']
    data['like'] = doc['item']['liked_count']
    data['rating'] = doc['item']['item_rating']['rating_star']
    data['review_count'] = doc['item']['cmt_count']
    data['sold_unit'] = doc['item']['historical_sold']
    
    if not doc['item']['bundle_deal_info'] == None: promotion.append(doc['item']['bundle_deal_info']['bundle_deal_label'])
    if doc['item']['shopee_verified'] == True: promotion.append('虾皮优选')
    if doc['item']['show_official_shop_label'] == True: promotion.append('商城')
    data['promotion_tag'] = promotion
       
    if doc['item']['is_official_shop'] == True: 
        data['subchannel'] = '购物商城'
    if doc['item']['show_free_shipping'] == True: 
        data['delivery'] = '免运费'
    else: 
        data['delivery'] = '需运费'
        
    data['Selling_price'] = str(doc['item']['price_max'])[:-5]
    data['List_price'] = str(doc['item']['price_min_before_discount'])[:-5]
    if data['List_price'] == 0: data['List_price'] = str(doc['item']['price_max'])[:-5]
    
    crawler_tm = datetime.now(tz=timezone('Asia/Taipei'))
    data['rtime'] = datetime.strftime(crawler_tm, '%Y-%m-%d %H:%M:%S')
    
    if not flavor == 0:
        for i in range(0, flavor):
            data['option_flavor'] = doc['item']['models'][(i)]['name']
            data['List_price'] = str(doc['item']['models'][(i)]['price'])[:-5]
            print(data)
    else:
        data['option_flavor'] = ""
        print(data)

栏位说明

• Description、like、rating、sold_unit: 商品说明、星星评分、评价、售出单位
• promotion_tag: 各类型促销优惠、标签
• Selling_price、List_price: 促销价、原始售价
• flavor: 多种不同的商品型号、选项

输入关键字

最後只要在这边输入想要搜寻的关键字，就可以开始爬取资料啦！

keyword = input("请输入关键字:")
getShopeekey(keyword)

结果

爬取到的资料会呈现这项商品不同的型号、价格

贴心提醒： 在 print(data) 部分可以改成汇入 DB 或 CSV 档案中！