久久影院一区二区三区-久久影院午夜伦手机不四虎卡-久久影院毛片一区二区-久久影视一区-在线精品91青草国产在线观看-在线激情小视频

<返回

True Knowledge Comes from Practice: Aligning Large Language Models with Embodied Environments via Reinforcement Learning

Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, Bo An

ICLR 2024 Conference

May 2024

Keywords: Reinforcement Learning, Large Language Models, Parameter-Efficient Fine-Tuning

Abstract:

Despite the impressive performance across numerous tasks, large language models (LLMs) often fail in solving simple decision-making tasks due to the misalignment of the knowledge in LLMs with environments. On the contrary, reinforcement learning (RL) agents learn policies from scratch, which makes them always align with environments but difficult to incorporate prior knowledge for efficient explorations. To narrow the gap, we propose TWOSOME, a novel general online framework that deploys LLMs as decision-making agents to efficiently interact and align with embodied environments via RL without requiring any prepared datasets or prior knowledge of the environments. Firstly, we query the joint probabilities of each valid action with LLMs to form behavior policies. Then, to enhance the stability and robustness of the policies, we propose two normalization methods and summarize four prompt design principles. Finally, we design a novel parameter-efficient training architecture where the actor and critic share one frozen LLM equipped with low-rank adapters (LoRA) updated by PPO. We conduct extensive experiments to evaluate TWOSOME. i) TWOSOME exhibits significantly better sample efficiency and performance compared to the conventional RL method, PPO, and prompt tuning method, SayCan, in both classical decision-making environment, Overcooked, and simulated household environment, VirtualHome. ii) Benefiting from LLMs open-vocabulary feature, TWOSOME shows superior generalization ability to unseen tasks. iii) Under our framework, there is no significant loss of the LLMs original ability during online PPO finetuning.

View More PDF>>

主站蜘蛛池模板: 岑巩县| 宜黄县| 托里县| 庆元县| 红安县| 郴州市| 航空| 五寨县| 晴隆县| 绥滨县| 西畴县| 深泽县| 金坛市| 新建县| 天津市| 镇宁| 山西省| 平顺县| 马边| 孝昌县| 山东省| 梅河口市| 油尖旺区| 侯马市| 阳春市| 阳泉市| 广丰县| 临城县| 安义县| 师宗县| 沿河| 鄂尔多斯市| 邢台市| 永年县| 习水县| 天水市| 财经| 石景山区| 康保县| 云安县| 永丰县|