Repository navigation
adds Accelerate full-state (opt, lr_sched, params) - #235
Conversation
There was a problem hiding this comment.
I left a few comments but overall it looks good
|
|
||
| global_grad_norm = None | ||
| for epoch in range(args.num_epochs): | ||
| for epoch in range(args.current_epoch, args.num_epochs): |
There was a problem hiding this comment.
Needs to be
| for epoch in range(args.current_epoch, args.num_epochs): | |
| for epoch in range(args.current_epoch, args.num_epochs+1): |
Since range(start, stop) only goes from start, ..., stop - 1
| return | ||
|
|
||
| # picks checkpoint with the largest number of samples seen, by name. | ||
| checkpoint_list = sorted(list(output_dir.iterdir()), reverse=True) |
There was a problem hiding this comment.
Is this necessarily true? If the list is a list of strings then how does it handle a case like samples_1000 vs samples_99999
There was a problem hiding this comment.
Usually hash of the string or sum of the string characters, so 9999 > 1000.
There was a problem hiding this comment.
I would advise passing a custom comparison function to sorted here, see the following code snippet:
lst = ["hf_format_1", "hf_format_10", "hf_format_2"]
sorted_lst = sorted(lst)
print(sorted_lst)
> ['hf_format_1', 'hf_format_10', 'hf_format_2']You can pass a custom function that simply splits on the _ character and casts the last part of the string into a number, this should do what you want it to.
There was a problem hiding this comment.
This will get sorted to work how you want it to:
| checkpoint_list = sorted(list(output_dir.iterdir()), reverse=True) | |
| checkpoint_list = sorted(list(output_dir.iterdir()), reverse=True, key=lambda d: int(d.split('_')[-1])) |
| return | ||
|
|
||
|
|
||
| latest = checkpoint_list[0] |
There was a problem hiding this comment.
Are we guaranteed to have at least one element when we check this?
There was a problem hiding this comment.
Yeah because the check above checks for len 0
bf7a3a7 to
09314dd
Compare
09314dd to
cbe75b6
Compare
…ng for DeepSpeed and FSDP. Signed-off-by: James Kunstle <jkunstle@redhat.com>
cbe75b6 to
07aa037
Compare
Saving and reloading for deepspeed.
Todo: