Repository navigation
many files not find, "run_inference.sh" #7
Description
Model paths
export CONFIG_PATH="./configs/hrdt_config.yaml" #--------not find
export PRETRAINED_MODEL_PATH="./checkpoints/robotwin2/checkpoint-50000/pytorch_model.bin"
export LANG_EMBEDDINGS_PATH="./embeddings/grab_blue_cup.pt" #--------not find
export STAT_FILE_PATH="./utils/stats.json" #--------not find
Activity
HongzheBi commentedon Aug 25, 2025
Here is an example, because this is a real inference code example, everyone's data, task is different. What this means is that you need lang embeddings for t5_xxl, stats.json for min and max, and config yaml files
zhangyong333 commentedon Aug 25, 2025
but how to get the hrdt_config.yaml, what this file mean? and how to generate the .json and .pt file?
zhangyong333 commentedon Aug 25, 2025
can you give a real example and files, maybe can use robotwin2 data, just we can the infer to see what output
HongzheBi commentedon Aug 25, 2025
First of all, the fine-tuning process for any new dataset can refer to the robotwin2 configuration, refer to H_RDT/datasets/robotwin2. You'll need the lang embeddings pt file, the stats dataset stats.json（optional） and the yaml configuration file to modify.
Be specific to your problem：
1、hrdt_config.yaml is the training configuration file for fine-tuning with your own dataset, namely H_RDT/configs/hrdt_finetune.yaml. In the latest inference/real_example, I put an example hrdt.yaml, which is no different from hrdt_finetune.yaml. You need to change the yaml values for your joint DOF, single arm or dual arm, and number of cameras.
2、The.pt file is the embddings encoded by T5-XXL. You can reference H_RDT/datasets/robotwin2/encode_lang_batch.py modification, you first need to download the T5-XXL huggingface weights(https://huggingface.co/google/t5-v1_1-xxl/tree/main) （In the robotwin2 example I just gave the .pt files I used）
3、The stats.json file is mainly the min and max values of the statistics dataset to normalize the state and action (this is optional, I didn't use it for robotwin2 simulation, but normal real machine datasets need to be normalized).In particular, modify your dataset according to H_RDT/datasets/robotwin2/calc_stat.py. Note that P152-153 requires you to change "robotwin_dual_arm" to the name of your dataset, "stat_dict = {
"robotwin_dual_arm": {"
HongzheBi commentedon Aug 25, 2025
You need to modify your own fientune dataset based on the robotiwn2 example. In fact,.pt files(encode_lang_batch.py), stats.json(calc_stat.py), config.yaml(hrdt_finetune.py) are easy to modify. Of the most difficult is that you need to modify the training load code of personal dataset, that is you need according to H_RDT/datasets/robotwin2 /robotwin_agilex_dataset.py create your own dataset class, And load it in H_RDT/datasets/dataset.py when finetuning.
The advantage of this approach is that the personal dataset class can be read in a variety of storage formats, such as hdf5, zarr, lmdb, lerobot, mp4, etc., but you have to create your own "robotwin_agilex_dataset.py".
zhangyong333 commentedon Aug 25, 2025
export PRETRAINED_MODEL_PATH="./checkpoints/robotwin2/checkpoint-50000/pytorch_model.bin"
here, can i just use the pretrained model? like this:
export PRETRAINED_MODEL_PATH="./checkpoints/pretrain-0618/checkpoint-50000/pytorch_model.bin"
HongzheBi commentedon Aug 25, 2025
H-RDT pre-trained models have only been trained on human data and cannot zero-shot direct inference on arbitrary robots, requiring fine-tuning. Most state-of-the-art VLA models pre-trained on cross-embodied robot data are unable to perform zero-shot tasks unless this embodiment is heavily present in the pre-trained data.
zhangyong333 commentedon Aug 26, 2025
in inference， there are 2 folders, real_example and robotwin2_example, what is the difference?
xianglunkai commentedon Aug 27, 2025
could you release likely cobot-magic robot finetuned models for testing? Txs！
xianglunkai commentedon Aug 29, 2025
@zhangyong333 could you release likely cobot-magic robot finetuned models for testing? Txs！