Zhe"ωan,稳,
点击:zhidao.baidu.com/dsrg/
http://23.34.67.21.99zgg.com?xcz-
网页爬取不一定要用Selenium,Selenium是为了注入浏览器获取点击行为的调试工具,如果网页无需人工交互就可以抓取,不建议你使用selenium。要使用它,你需要安装一个工具软件,使用Chrome浏览器需要下载chromedriver.exe到system32下,如使用firefox则要下载geckodriver.exe到system32下。下面以chromedriver驱动chrome为例:
# -*- coding:UTF-8 -*-from selenium import webdriverfrom bs4 import BeautifulSoupimport reimport timeif __name__ == '__main__': options = webdriver.ChromeOptions() options.add_argument('user-agent="Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.133 Mobile Safari/535.19"') driver = webdriver.Chrome() driver.get('url')#你要抓取百度文库的URL,随便找个几十页的替换掉 html = driver.page_source bf1 = BeautifulSoup(html, 'lxml') result = bf1.find_all(class_='rtcspage') bf2 = BeautifulSoup(str(result[0]), 'lxml') title = bf2.div.div.h1.string pagenum = bf2.find_all(class_='size') pagenum = BeautifulSoup(str(pagenum), 'lxml').span.string pagepattern = re.compile('页数:(\d+)页') num = int(pagepattern.findall(pagenum)[0]) print('文章标题:%s' % title) print('文章页数:%d' % num) while True: num = num / 5.0 html = driver.page_source bf1 = BeautifulSoup(html, 'lxml') result = bf1.find_all(class_='rtcspage') for each_result in result: bf2 = BeautifulSoup(str(each_result), 'lxml') texts = bf2.find_all('p') for each_text in texts: main_body = BeautifulSoup(str(each_text), 'lxml') for each in main_body.find_all(True): if each.name == 'span': print(each.string.replace('\xa0',''),end='') elif each.name == 'br': print('') print('\n') if num > 1: page = driver.find_elements_by_xpath("//div[@class='page']") driver.execute_script('arguments[0].scrollIntoView();', page[-1]) #拖动到可见的元素去 nextpage = driver.find_element_by_xpath("//a[@data-fun='next']") nextpage.click() time.sleep(3) else: break执行代码,chromedriver自动为你打开chrome浏览器,此时你翻页到最后,点击阅读更多,然后等一段时间后关闭浏览器,代码继续执行。