【学习心得】Python爬虫实战-Mooc个人排行榜

<p><strong>&nbsp;昨天和大家分享了用python获取团队排行榜的方法. 那么个人排行榜能不能用python爬虫抓取呢? </strong></p><p><strong>但个人总排行榜的页面似乎不存在. 这是个挑战. 我们来试试,如何? </strong></p><p><strong><br ></strong></p><p><strong>下一个目标:用python爬虫获取所有参赛课程列表.&nbsp; 敬请关注!</strong></p><p><strong><br ></strong></p><p><strong><br ></strong></p><p><strong><br ></strong></p><p><strong><img src="https://nos.netease.com/edu-image/31584ff2-8f47-4f66-9bda-e139ba4bc441.png?imageView&thumbnail=520x520&quality=100" /></strong></p><p><br /></p><p><code class="brush:python;toolbar:false" ></code>#!/usr/bin/python3<br >import requests,json<br >import numpy as np<br >import os<br >import pandas as pd<br >os.chdir(r'C:\Users\Administrator\Desktop')<br ><br ><br >from datetime import datetime as dt2<br >import datetime<br >now=dt2.now()<br ><br >def get_data():<br > &nbsp; &nbsp;url='https://www.icourse163.org/web/j/mocActivityScholarshipV3RpcBean.getTop100ByActivityId.rpc?csrfKey=c5b22864e441448894def7bb080f9e5c'<br > &nbsp; &nbsp;headers = {<br ><br > &nbsp; &nbsp; &nbsp; &nbsp;'Accept': '*/*',<br > &nbsp; &nbsp; &nbsp; &nbsp;'Accept-Encoding':'gzip, deflate',<br > &nbsp; &nbsp; &nbsp; &nbsp;'Accept-Language': 'zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2',<br > &nbsp; &nbsp; &nbsp; &nbsp;'Cache-Control':'max-age=0',<br > &nbsp; &nbsp; &nbsp; &nbsp;'Connection': 'keep-alive',<br > &nbsp; &nbsp; &nbsp; &nbsp;'Content-Length':'16',<br > &nbsp; &nbsp; &nbsp; &nbsp;'Content-Type': 'application/x-www-form-urlencoded',<br > &nbsp; &nbsp; &nbsp; &nbsp;'Cookie': '&nbsp;&nbsp; '&nbsp; #这里需填写自己电脑上的COOKIE信息。<br >&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'Host': 'www.icourse163.org',<br > &nbsp; &nbsp; &nbsp; &nbsp;'Referer':'https://www.icourse163.org/scholarship2017/rank.htm?id=17001',<br > &nbsp; &nbsp; &nbsp; &nbsp;'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; rv:57.0) Gecko/20100101 Firefox/57.0'<br ><br ><br > &nbsp; &nbsp;}<br ><br > &nbsp; &nbsp;data = {<br > &nbsp; &nbsp; &nbsp; &nbsp;'activityId':'17001'<br > &nbsp; &nbsp;}<br ><br > &nbsp; &nbsp;req = requests.post(url, data, headers=headers)<br > &nbsp; &nbsp;req.encoding = 'utf-8'<br > &nbsp; &nbsp;result = json.loads(req.text)<br > &nbsp; &nbsp;result = result['result']<br > &nbsp; &nbsp;id,name,value,ph=[],[],[],[]<br > &nbsp; &nbsp;for i in result:<br > &nbsp; &nbsp; &nbsp; &nbsp;id.append(i[&quot;memberId&quot;])<br > &nbsp; &nbsp; &nbsp; &nbsp;name.append(i[&quot;nickName&quot;])<br > &nbsp; &nbsp; &nbsp; &nbsp;value.append(i['value'])<br > &nbsp; &nbsp; &nbsp; &nbsp;ph.append(i['ranking'])<br > &nbsp; &nbsp;a={'排名':ph,'昵称':name,'成长值':value,'ID':id}<br > &nbsp; &nbsp;b=['排名','昵称','成长值','ID']<br > &nbsp; &nbsp;df=pd.DataFrame(a,columns=b,index=range(1,len(id)+1))<br > &nbsp; &nbsp;time_start = now.strftime('%m-%d')<br > &nbsp; &nbsp;df.to_excel(time_start+'-个人成长值排行榜.xls')<br > &nbsp; &nbsp;print(df)<br ><br >get_data()<br ></p><p><p><strong><br ></strong></p><p><strong><br ></strong></p><p></p><p><strong><br ></strong></p></p>