人脸爬取(人脸数据集的搜集)
在进行人脸相关处理中,人脸数据集是关键,这里描述一下怎样爬取人脸数据集
1、获取艺人名称
① 获取完整url路径
在百度中搜索“中国艺人”
得到以下界面
通过分析,url的完整路径为:
“https://sp0.baidu.com/8aQDcjqpAAV3otqbppnN2DJv/api.php?resource_id=28266&from_mid=500&format=json&ie=utf-8&oe=utf-8&query=%E4%B8%AD%E5%9B%BD%E8%89%BA%E4%BA%BA&sort_key=&sort_type=1&stat0=&stat1=&stat2=&stat3=&pn=”+pn+”&rn=100&_=1580457480665”
其中,pn为页码数。
② 解析全部艺人名称列表
解析上述url,通过requests获取网页内容,然后解析艺人姓名
def get_person_name(): person_list = [] pn_i=0 while(True): pn=str(pn_i) pn_i+=100 url=”https://sp0.baidu.com/8aQDcjqpAAV3otqbppnN2DJv/api.php?resource_id=28266&from_mid=500&format=json&ie=utf-8&oe=utf-8&query=%E4%B8%AD%E5%9B%BD%E8%89%BA%E4%BA%BA&sort_key=&sort_type=1&stat0=&stat1=&stat2=&stat3=&pn=”+pn+”&rn=100&_=1580457480665″
res = requests.get(url) try: json_str=json.loads(res.text) except: continue
figs=json_str[‘data’][0][‘result’]
for i in figs: name=i[‘ename’] print(name) person_list.append(name)
return person_list
2、爬取相应艺人的相应照片
① 获取图片的url
while pn < self.__amount:
url = “https://image.baidu.com/search/index?tn=baiduimage&ipn=r&ct=201326592&cl=2&lm=-1&st=-1&fm=result&fr=&sf=1&fmq=1639129009987_R&pv=&ic=&nc=1&z=&hd=&latest=©right=&se=1&showtab=0&fb=0&width=&height=&face=0&istype=2&dyTabStr=MCwzLDEsNiwyLDQsNSw3LDgsOQ%3D%3D&ie=utf-8&sid=&word=” + word_quote
try: time.sleep(self.time_sleep) req = urllib.request.Request(url=url, headers=self.headers) #print(req)
page = urllib.request.urlopen(req) rsp = page.read()
rsp = str(rsp) index_list = find_all_sub(“objURL”,rsp)
img_root = []
for i in range(0,len(index_list)): temp = [] if i == len(index_list) -1: temp = rsp[index_list[len(index_list) -1]:len(rsp)] else: temp = rsp[index_list[i]:index_list[i + 1]] img_root.append(temp)
for img_root_path in img_root: temp_url = img_root_path[9:] end = temp_url.find(‘”‘) image_temp_url = temp_url[:end] if not find_in_list(image_url_list,image_temp_url): image_url_list.append(image_temp_url)
except UnicodeDecodeError as e: print(e) print(‘—–UnicodeDecodeErrorurl:’, url) except urllib.error.URLError as e: print(e) print(“—–urlErrorurl:”, url) except socket.timeout as e: print(e) print(“—–socket timout:”, url) else:
# 读取下一页 print(“下载下一页”) pn += 60 finally: page.close()
② 下载对应的图片
image_root_path = “./” + word if not os.path.exists(image_root_path): os.mkdir(image_root_path) filepath = image_root_path + “/” + str(word) + “_” + str(number) + “.jpg”
for img_url in image_url_list: number += 1 filepath = image_root_path + “/” + str(word) + “_” + str(number) + “.jpg” print(filepath) count = 1 try: urllib.request.urlretrieve(img_url, filepath) except socket.timeout: while count <= 3: try: urllib.request.urlretrieve(img_url, filepath) break except socket.timeout: count += 1 finally: # display the raw url of images print(‘ %d %s’ % (number, img_url)) if count > 3: print(‘ %d %s failed’ % (number, img_url)) pass
print(“下载任务结束”)
相关文章
- 物联网设备接入层
- iOS键值观察KVO实例详解简介实例效果实现方式进阶用法
- iOS存储方式之归档
- URL访问网站的网络传输全过程
- Mysql防SQL注入
- LFW人脸数据集筛选有多张图的人
- Keras多GPU训练
- Zabbix自定义监控&Grafana图表配置
- 怎么样清除spyder的运行数据?
- LeetCode笔记:70. Climbing Stairs
- 处理筛选CelebA人脸数据集
- CNN训练与测试人脸图片分类(TensorFlow)
- 用FaceNet的模型计算人脸之间距离(TensorFlow)
- Realm、WCDB与SQLite移动数据库性能对比测试
- LeetCode笔记:171. Excel Sheet Column Number
- LeetCode笔记:202. Happy Number
- 清华团队夺SC超算大赛4连冠!华中科大斩获「图计算」2项世界第一
- 动漫界的ImageNet来了!人脸数据集AnimeCeleb,240万张图片生成「萌萌哒」动漫脸
- LeetCode笔记:219. Contains Duplicate II
- LeetCode笔记:223. Rectangle Area