在settings.py文件中加入如下代码即可(USER_AGENT是固定写法),具体使用什么agent可以自己更改下面蓝色字体内容。
创新互联公司云计算的互联网服务提供商,拥有超过13年的服务器租用、移动服务器托管、云服务器、虚拟空间、网站系统开发经验,已先后获得国家工业和信息化部颁发的互联网数据中心业务许可证。专业提供云主机、虚拟空间、国际域名空间、VPS主机、云服务器、香港云服务器、免备案服务器等。
USER_AGENT = 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)
在settings.py文件中通过创建一个agent列表,然后通过从中随机取出1个赋值给USER_AGENT变量即可
代码如下:
import random
UserAgentList = [
"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)",
"Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 1.0.3705; .NET CLR 1.1.4322)",
]
USER_AGENT = random.choice(UserAgentList)
在settings.py文件中加入如下代码:
fromfake_useragentimportUserAgent
ua = UserAgent()
USER_AGENT = ua.random
在setting.py文件中写一个UserAgentList列表
UserAgentList = [
'Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1500.55 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1309.0 Safari/537.17'
]
在setting.py文件中启用DOWNLOADER_MIDDLEWARES,并且把默认的注释掉
DOWNLOADER_MIDDLEWARES = {
#'amazon.middlewares.MyCustomDownloaderMiddleware': 543,
'amazon.middlewares.MyUserAgentMiddleware':543,
}
在middlewares.py文件中增加以下代码:
romscrapy.downloadermiddlewares.useragentimportUserAgentMiddleware
fromamazon.settingsimportUserAgentList
importrandom
classMyUserAgentMiddleware(UserAgentMiddleware):
def__init__(self,user_agent=''):
self.user_agent = user_agent
defprocess_request(self,request,spider):
agent = random.choice(list(UserAgentList))
request.headers['User-Agent'] = agent
安装fake-useragent库(useragent的库)
pip install fake-useragent
在middlewares.py文件中导入fake-useragent库
from fake_useragent import UserAgent
在middlewares.py文件中编写一个middleware中间件,
fromfake_useragentimportUserAgent
fromscrapy.downloadermiddlewares.useragentimportUserAgentMiddleware
classMyUserAgentMiddleware(UserAgentMiddleware):
def__init__(self,user_agent=''):
self.user_agent = user_agent
defprocess_request(self,request,spider):
ua = UserAgent()
request.headers['User-Agent'] = ua.random
在setting.py文件中启用DOWNLOADER_MIDDLEWARES,并且把默认的注释掉
DOWNLOADER_MIDDLEWARES = {
#'amazon.middlewares.MyCustomDownloaderMiddleware': 543,
'amazon.middlewares.MyUserAgentMiddleware':543,
}