抓取豆瓣影評評分

正常的抓取

分析請求的url

https://movie.douban.com/subject/26322642/comments?start=20&limit=20&sort=new_score&status=P&percent_type=

里面有用的也就是start和limit參數，我嘗試過修改limit參數，但是沒有效果，可以認為是默認的
start參數是用來設置從第幾條數據開始查詢的

于是采用下面的代碼進行判斷是否還有下一個頁面

if next_url:    visit_URL('https://movie.douban.com/subject/24753477/comments'+next_url)

把數據寫入txt

import requestsfrom bs4 import BeautifulSoupfirst_url = 'https://movie.douban.com/subject/26322642/comments?status=P'# 請求頭部headers = {  'Host':'movie.douban.com',  'Referer':'https://movie.douban.com/subject/24753477/?tag=%E7%83%AD%E9%97%A8&from=gaia_video',  'Upgrade-Insecure-Requests':'1',  'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36',}def visit_URL(url):  res = requests.get(url=url,headers=headers)  soup = BeautifulSoup(res.content,'html5lib')  div_comment = soup.find_all('div',class_='comment-item') # 找到所有的評論模塊  for com in div_comment:    username = com.find('div',class_='avatar').a['title']    comment_time = com.find('span',class_='comment-time')['title']    votes = com.find('span',class_='votes').get_text()    comment = com.p.get_text()    with open('1.txt','a',encoding='utf8') as file:      file.write('評論人：'+username+'/n')      file.write('評論時間：'+comment_time+'/n')      file.write('支持人數：'+votes+'/n')      file.write('評論內容：'+comment+'/n')  # 檢查是否有下一頁  next_url = soup.find('a',class_='next')  if next_url:    temp = next_url['href'].strip().split('&') # 獲取下一個url    next_url = ''.join(temp)    print(next_url)  # print(next_url)  if next_url:    visit_URL('https://movie.douban.com/subject/24753477/comments'+next_url)if __name__ == '__main__':  visit_URL(first_url)

模仿移動端

很多時候模仿移動端獲得的頁面會比PC端的簡單，更加容易解析，這次模擬移動端，發現可以直接訪問api獲取json格式的數據，nice!

至于怎么模擬移動端只需要將user-agent修改為移動端的頭

useragents = [  "Mozilla/5.0 (iPhone; CPU iPhone OS 9_2 like Mac OS X) AppleWebKit/601.1 (KHTML, like Gecko) CriOS/47.0.2526.70 Mobile/13C71 Safari/601.1.46",  "Mozilla/5.0 (Linux; U; Android 4.4.4; Nexus 5 Build/KTU84P) AppleWebkit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30",  "Mozilla/5.0 (compatible; MSIE 9.0; Windows Phone OS 7.5; Trident/5.0; IEMobile/9.0)"