轻型微博爬虫

• hidadeng/weibo_crawler

• https://pypi.org/project/weibo-crawler/

weibo_crawler参考【nghuyong/WeiboSpider】 对代码用法进行了简化，可以做轻度的微博数据采集。

• 用户信息抓取

• 用户微博抓取(全量/指定时间段)

• 用户社交关系抓取(粉丝/关注)

• 微博评论抓取

• 基于关键词和时间段(粒度到小时)的微博抓取

• 微博转发抓取

使用简介：https://www.douban.com/group/topic/247718378/

安装

pip install weibo-crawler

Requirement already satisfied: weibo-crawler in /opt/anaconda3/lib/python3.7/site-packages (1.0)
Requirement already satisfied: pyquery in /opt/anaconda3/lib/python3.7/site-packages (from weibo-crawler) (1.4.3)
Requirement already satisfied: requests in /opt/anaconda3/lib/python3.7/site-packages (from weibo-crawler) (2.24.0)
Requirement already satisfied: cssselect>0.7.9 in /opt/anaconda3/lib/python3.7/site-packages (from pyquery->weibo-crawler) (1.1.0)
Requirement already satisfied: lxml>=2.1 in /opt/anaconda3/lib/python3.7/site-packages (from pyquery->weibo-crawler) (4.6.1)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/anaconda3/lib/python3.7/site-packages (from requests->weibo-crawler) (1.25.11)
Requirement already satisfied: idna<3,>=2.5 in /opt/anaconda3/lib/python3.7/site-packages (from requests->weibo-crawler) (2.8)
Requirement already satisfied: certifi>=2017.4.17 in /opt/anaconda3/lib/python3.7/site-packages (from requests->weibo-crawler) (2019.11.28)
Requirement already satisfied: chardet<4,>=3.0.2 in /opt/anaconda3/lib/python3.7/site-packages (from requests->weibo-crawler) (3.0.4)
Note: you may need to restart the kernel to use updated packages.

获取cookie

• 使用chrome浏览器打开手机微博 https://weibo.cn 登录

• 右键inspect（即打开开发者模式）

• 查看network内容

• 获取html文件header中的cookie信息

  • 其中可能需要SSOLoginState字段

from weibo_crawler import Profile

# 如果程序失败，需要传入你的微博cookies
cookies='_T_WM=9b80727fa0cc3b6b6c374b9262ff084d; SUB=_2A25MjZmZDeRhGeNI4lYX-S7FwjWIHXVscSfRrDV6PUJbktAKLXSmkW1NSAJ40ykLq1lxtFqpHJ4BRMiY1XKHNT6g; SUBP=0033WrSXqPxfM725Ws9jqgMF55529P9D9WhkPe1HSir85xF8hwHpTZa75NHD95QfSo.XSo.71K.4Ws4DqcjT9s8Xqgpyqoz7eK-t; SSOLoginState=1636428233'

# csv文件路径
prof=Profile(csvfile='./data/weibo-chenkun-intro.csv', delay=1, cookies=cookies)

prof.get_profile(userid='1087770692') # 陈坤微博的id

{'userid': '1087770692', 'nickname': '陈坤', 'gender': '男', 'province': '重庆', 'introduction': '莫失己道，莫扰他心。', 'birthday': '0001-00-00', 'vip_level': '7级送Ta会员', 'authentication': '演员，代表作《龙门飞甲》《画皮》等，行走的力量发起者', 'labels': '演员'}
采集完毕，请查看 ./data/weibo-chenkun-intro.csv 内的数据

更多操作见：

hidadeng/weibo_crawler