8000 学习率的问题?学习率最大6.25e-5 · Issue #97 · thu-coai/CDial-GPT · GitHub
[go: up one dir, main page]
Skip to content

学习率的问题?学习率最大6.25e-5 #97

New issue
New issue
Open
Open
学习率的问题?学习率最大6.25e-5#97

Description

@WuDiDaBinGe
WuDiDaBinGe
opened on Jul 18, 2022

文章中说学习率最大6.25e-5，noam schedule更小，要用这么小的学习率吗？

Activity

  1. silverriver commented on Jul 23, 2022

    @silverriver
    silverriver
    on Jul 23, 2022
    Collaborator

    您可以根据自己所用的bs调一下试试

  2. WuDiDaBinGe commented on Jul 24, 2022

    @WuDiDaBinGe
    WuDiDaBinGe
    on Jul 24, 2022
    Author

    您可以根据自己所用的bs调一下试试
    image
    文章中所有的 8(per_gpu)*64(gradient accumulation)*4(gpus) = 2048
    那我128 * 2 * 6 的话应该调整成两倍吧（bs增大多少倍数 学习率相应增大）

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions
      0