baselines中环境包装器EpisodicLifeEnv的分析
环境 分析 包装
2023-09-11 14:19:19 时间
如题:
class EpisodicLifeEnv(gym.Wrapper): def __init__(self, env): """Make end-of-life == end-of-episode, but only reset on true game over. Done by DeepMind for the DQN and co. since it helps value estimation. """ gym.Wrapper.__init__(self, env) self.lives = 0 self.was_real_done = True def step(self, action): obs, reward, done, info = self.env.step(action) self.was_real_done = done # check current lives, make loss of life terminal, # then update lives to handle bonus lives lives = self.env.unwrapped.ale.lives() if lives < self.lives and lives > 0: # for Qbert sometimes we stay in lives == 0 condition for a few frames # so it's important to keep lives > 0, so that we only reset once # the environment advertises done. done = True self.lives = lives return obs, reward, done, info def reset(self, **kwargs): """Reset only when lives are exhausted. This way all states are still reachable even though lives are episodic, and the learner need not know about any of this behind-the-scenes. """ if self.was_real_done: obs = self.env.reset(**kwargs) else: # no-op step to advance from terminal/lost life state obs, _, _, _ = self.env.step(0) self.lives = self.env.unwrapped.ale.lives() return obs
EpisodicLifeEnv包装器是针对环境中有多条lives的,游戏中所剩的lives通过: lives = self.env.unwrapped.ale.lives()获得。
主要需要说明的代码为:
if lives < self.lives and lives > 0: # for Qbert sometimes we stay in lives == 0 condition for a few frames # so it's important to keep lives > 0, so that we only reset once # the environment advertises done. done = True
根据注释可以知道,对于游戏Qbert来说当所剩lives为0的时候这时返回的done为false,也就是说还需要几帧画面后才会获得done=True的反馈,如果我们将判断条件:
if lives < self.lives and lives > 0:
改为:
if lives < self.lives and lives >=0:
这样,step返回的 return obs, reward, done, info 将作为一个episode的最后一帧数据来处理,并调用reset函数中的:
else: # no-op step to advance from terminal/lost life state obs, _, _, _ = self.env.step(0)
这样,在随后的几帧数据中由于 self.was_real_done = False,而 lives = self.env.unwrapped.ale.lives()=0,会不断的循环调用reset操作。
当然针对Qbert游戏中的这种问题我们还可以使用其他的修改方式:
class EpisodicLifeEnv(gym.Wrapper): def __init__(self, env): """Make end-of-life == end-of-episode, but only reset on true game over. Done by DeepMind for the DQN and co. since it helps value estimation. """ gym.Wrapper.__init__(self, env) self.lives = 0 self.was_real_done = True def step(self, action): obs, reward, done, info = self.env.step(action) # self.was_real_done = done # check current lives, make loss of life terminal, # then update lives to handle bonus lives lives = self.env.unwrapped.ale.lives() if lives < self.lives: # for Qbert sometimes we stay in lives == 0 condition for a few frames # so it's important to keep lives > 0, so that we only reset once # the environment advertises done. done = True self.lives = lives return obs, reward, done, info def reset(self, **kwargs): """Reset only when lives are exhausted. This way all states are still reachable even though lives are episodic, and the learner need not know about any of this behind-the-scenes. """ # if self.was_real_done: if self.lives == 0: obs = self.env.reset(**kwargs) else: # no-op step to advance from terminal/lost life state obs, _, _, _ = self.env.step(0) self.lives = self.env.unwrapped.ale.lives() return obs
==================================================
相关文章
- 安卓逆向_17 --- IDA 动态调试【 环境搭建、so库调试【动态普通、动态debug模式】、JNI_OnLoad调试分析、java_ 开头函数分析】
- Simulink 环境基础知识(八)-- 使用注释描述模型
- Simulink 环境基础知识(五)--编程建模基础知识
- android开发环境重装系统之后的配置
- 深入理解Spark:核心思想与源码分析. 3.2 创建执行环境SparkEnv
- 深入理解Spark:核心思想与源码分析. 3.12 Spark环境更新
- 《Ansible权威指南》一1.8 Python多环境扩展管理
- ELK实时日志分析平台环境部署--完整记录(ElasticSearch+Logstash+Kibana )
- Kafka - kafka环境配置以及参数分析
- VS Code配置Matlab环境笔记-文本编辑、语法检查、运行脚本
- Android基础-系统架构分析,环境搭建,下载Android Studio,AndroidDevTools,Git使用教程,Github入门,界面设计介绍
- 《智能路由器开发指南》——第2章 开发环境及编译分析
- DIY Hadoop大数据环境的5大陷阱
- OpenWrt(一)——环境搭建与源码下载
- 《深入解析sas:数据处理、分析优化与商业应用》一1.3 SAS窗口环境
- 强化学习中仿真环境下大规模并行采样的可行性分析——以仿真环境为例
- Git 环境安装
- Android渗透测试虚拟机环境(VM)—AppUse
- 内网环境NTP服务及时间同步(CentOS6.x)配置和部署
- SSH框架总结(框架分析+环境搭建+实例源代码下载)
- Raspberry-Pi-PICO系列--第二篇 开发环境搭建-Mac系统
- ES 中文分词器ik及基于netcore的远程词库环境编写
- 2019玩转ELK(ElasticSearch + LogStash + Kibana = ELKStack),轻松几步搭建ELK环境,对数据进行实时监控和分析(上)
- 2019玩转TIG(Telegraf+Influxdb+Grafana),轻松几步搭建环境,通过各种表盘展示数据,对数据进行实时监控和分析(下)
- 2019玩转TIG(Telegraf+Influxdb+Grafana),轻松几步搭建环境,通过各种表盘展示数据,对数据进行实时监控和分析(上)