Repository navigation
Could you share the training log of stage one? #21
Description
I am reimplementing the stage one of UniAD, but the result is very different from the metrics on the repo (AMOTA: 0.3491 vs 0.390).
Could you share your training log of stage one? Thanks!
2023-04-29 17:27:50,507 - mmdet - INFO - Epoch(val) [6][753] pts_bbox_NuScenes/car_AP_dist_0.5: 0.2318, pts_bbox_NuScenes/car_AP_dist_1.0: 0.5167, pts_bbox_NuScenes/car_AP_dist_2.0: 0.7178, pts_bbox_NuScenes/car_AP_dist_4.0: 0.8132, pts_bbox_NuScenes/car_trans_err: 0.4704, pts_bbox_NuScenes/car_scale_err: 0.1501, pts_bbox_NuScenes/car_orient_err: 0.0767, pts_bbox_NuScenes/car_vel_err: 0.3336, pts_bbox_NuScenes/car_attr_err: 0.2142, pts_bbox_NuScenes/mATE: 0.6905, pts_bbox_NuScenes/mASE: 0.2765, pts_bbox_NuScenes/mAOE: 0.3952, pts_bbox_NuScenes/mAVE: 0.3878, pts_bbox_NuScenes/mAAE: 0.2061, pts_bbox_NuScenes/truck_AP_dist_0.5: 0.0345, pts_bbox_NuScenes/truck_AP_dist_1.0: 0.1856, pts_bbox_NuScenes/truck_AP_dist_2.0: 0.4547, pts_bbox_NuScenes/truck_AP_dist_4.0: 0.5935, pts_bbox_NuScenes/truck_trans_err: 0.7482, pts_bbox_NuScenes/truck_scale_err: 0.2197, pts_bbox_NuScenes/truck_orient_err: 0.0908, pts_bbox_NuScenes/truck_vel_err: 0.3117, pts_bbox_NuScenes/truck_attr_err: 0.1936, pts_bbox_NuScenes/construction_vehicle_AP_dist_0.5: 0.0000, pts_bbox_NuScenes/construction_vehicle_AP_dist_1.0: 0.0197, pts_bbox_NuScenes/construction_vehicle_AP_dist_2.0: 0.1024, pts_bbox_NuScenes/construction_vehicle_AP_dist_4.0: 0.2860, pts_bbox_NuScenes/construction_vehicle_trans_err: 0.9201, pts_bbox_NuScenes/construction_vehicle_scale_err: 0.4812, pts_bbox_NuScenes/construction_vehicle_orient_err: 1.1397, pts_bbox_NuScenes/construction_vehicle_vel_err: 0.1425, pts_bbox_NuScenes/construction_vehicle_attr_err: 0.3970, pts_bbox_NuScenes/bus_AP_dist_0.5: 0.0418, pts_bbox_NuScenes/bus_AP_dist_1.0: 0.2882, pts_bbox_NuScenes/bus_AP_dist_2.0: 0.5570, pts_bbox_NuScenes/bus_AP_dist_4.0: 0.7124, pts_bbox_NuScenes/bus_trans_err: 0.7131, pts_bbox_NuScenes/bus_scale_err: 0.2101, pts_bbox_NuScenes/bus_orient_err: 0.0862, pts_bbox_NuScenes/bus_vel_err: 0.7570, pts_bbox_NuScenes/bus_attr_err: 0.2888, pts_bbox_NuScenes/trailer_AP_dist_0.5: 0.0000, pts_bbox_NuScenes/trailer_AP_dist_1.0: 0.0151, pts_bbox_NuScenes/trailer_AP_dist_2.0: 0.1401, pts_bbox_NuScenes/trailer_AP_dist_4.0: 0.4041, pts_bbox_NuScenes/trailer_trans_err: 1.0503, pts_bbox_NuScenes/trailer_scale_err: 0.2635, pts_bbox_NuScenes/trailer_orient_err: 0.4992, pts_bbox_NuScenes/trailer_vel_err: 0.3285, pts_bbox_NuScenes/trailer_attr_err: 0.0919, pts_bbox_NuScenes/barrier_AP_dist_0.5: 0.1771, pts_bbox_NuScenes/barrier_AP_dist_1.0: 0.4155, pts_bbox_NuScenes/barrier_AP_dist_2.0: 0.6018, pts_bbox_NuScenes/barrier_AP_dist_4.0: 0.7037, pts_bbox_NuScenes/barrier_trans_err: 0.5555, pts_bbox_NuScenes/barrier_scale_err: 0.2882, pts_bbox_NuScenes/barrier_orient_err: 0.1781, pts_bbox_NuScenes/barrier_vel_err: nan, pts_bbox_NuScenes/barrier_attr_err: nan, pts_bbox_NuScenes/motorcycle_AP_dist_0.5: 0.0846, pts_bbox_NuScenes/motorcycle_AP_dist_1.0: 0.2814, pts_bbox_NuScenes/motorcycle_AP_dist_2.0: 0.5137, pts_bbox_NuScenes/motorcycle_AP_dist_4.0: 0.6110, pts_bbox_NuScenes/motorcycle_trans_err: 0.6728, pts_bbox_NuScenes/motorcycle_scale_err: 0.2607, pts_bbox_NuScenes/motorcycle_orient_err: 0.4699, pts_bbox_NuScenes/motorcycle_vel_err: 0.5474, pts_bbox_NuScenes/motorcycle_attr_err: 0.2717, pts_bbox_NuScenes/bicycle_AP_dist_0.5: 0.1006, pts_bbox_NuScenes/bicycle_AP_dist_1.0: 0.3026, pts_bbox_NuScenes/bicycle_AP_dist_2.0: 0.4610, pts_bbox_NuScenes/bicycle_AP_dist_4.0: 0.5316, pts_bbox_NuScenes/bicycle_trans_err: 0.5727, pts_bbox_NuScenes/bicycle_scale_err: 0.2676, pts_bbox_NuScenes/bicycle_orient_err: 0.5571, pts_bbox_NuScenes/bicycle_vel_err: 0.2733, pts_bbox_NuScenes/bicycle_attr_err: 0.0177, pts_bbox_NuScenes/pedestrian_AP_dist_0.5: 0.0804, pts_bbox_NuScenes/pedestrian_AP_dist_1.0: 0.3098, pts_bbox_NuScenes/pedestrian_AP_dist_2.0: 0.5757, pts_bbox_NuScenes/pedestrian_AP_dist_4.0: 0.7174, pts_bbox_NuScenes/pedestrian_trans_err: 0.7118, pts_bbox_NuScenes/pedestrian_scale_err: 0.2931, pts_bbox_NuScenes/pedestrian_orient_err: 0.4587, pts_bbox_NuScenes/pedestrian_vel_err: 0.4087, pts_bbox_NuScenes/pedestrian_attr_err: 0.1734, pts_bbox_NuScenes/traffic_cone_AP_dist_0.5: 0.2482, pts_bbox_NuScenes/traffic_cone_AP_dist_1.0: 0.4972, pts_bbox_NuScenes/traffic_cone_AP_dist_2.0: 0.6581, pts_bbox_NuScenes/traffic_cone_AP_dist_4.0: 0.7315, pts_bbox_NuScenes/traffic_cone_trans_err: 0.4896, pts_bbox_NuScenes/traffic_cone_scale_err: 0.3307, pts_bbox_NuScenes/traffic_cone_orient_err: nan, pts_bbox_NuScenes/traffic_cone_vel_err: nan, pts_bbox_NuScenes/traffic_cone_attr_err: nan, pts_bbox_NuScenes/NDS: 0.4884, pts_bbox_NuScenes/mAP: 0.3679, pts_bbox_NuScenes/amota: 0.3491, pts_bbox_NuScenes/amotp: 1.3378, pts_bbox_NuScenes/recall: 0.4500, pts_bbox_NuScenes/motar: 0.6470, pts_bbox_NuScenes/gt: 14556.7143, pts_bbox_NuScenes/mota: 0.3015, pts_bbox_NuScenes/motp: 0.7156, pts_bbox_NuScenes/mt: 2043.0000, pts_bbox_NuScenes/ml: 2778.0000, pts_bbox_NuScenes/faf: 50.1770, pts_bbox_NuScenes/tp: 54523.0000, pts_bbox_NuScenes/fp: 14212.0000, pts_bbox_NuScenes/fn: 46560.0000, pts_bbox_NuScenes/ids: 814.0000, pts_bbox_NuScenes/frag: 774.0000, pts_bbox_NuScenes/tid: 1.6311, pts_bbox_NuScenes/lgd: 2.6311, drivable_iou: 0.6575, lanes_iou: 0.2736, divider_iou: 0.2135, crossing_iou: 0.0882, contour_iou: 0.2236, drivable_iou_mean: 0.6542, lanes_iou_mean: 0.2768, divider_iou_mean: 0.2124, crossing_iou_mean: 0.0472, contour_iou_mean: 0.2361
Activity
YTEP-ZHI commentedon May 8, 2023
Hi @VijayLuo, thanks for your feedback.
Did you load the pretrained bevformer weights correctly when training stage1?
UniAD/projects/configs/stage1_track_map/base_track_map.py
Line 567 in 059efe4
Could you please provide the full training log in this thread so I can track the problem down?
VijayLuo commentedon May 8, 2023
Thanks for your answer.
20230429_052541.log
YTEP-ZHI commentedon Jun 12, 2023
Hi, @VijayLuo. Thanks for reporting this issue on unable to reproduce the stage1 results. We have carefully investigated this problem and found that the performance disparity is led by two causes:
1) mistakenly adding the loss_past_traj
2) freezing the img_neck and BN in stage1 training
By removing loss_past_traj and unfreezing img_neck and BN in training, the reported results could be reproduced (AMOTA: 0.393).
Here is the full training log of the fixed stage1 model: https://github.com/OpenDriveLab/UniAD/releases/download/v1.0/uniad_reproduce_stage1_gpu16_train.log
sunbin1357 commentedon Jun 13, 2023
I find that loss_past_traj is removed by setting loss_past_traj_weight to 0.0 in config，track_instances.pred_past_trajs will still be obtained during training. It may be better to add a true/false in the code to determine whether to obtain track_instances.pred_past_trajs
YTEP-ZHI commentedon Jun 13, 2023
@sunbin1357 Thanks for your suggestion. You're right. Setting loss_past_traj_weight to 0 is an easy and correct implementation, but it could be better structured to completely remove pred_past_trajs in training. We will consider implementing it.
YTEP-ZHI commentedon Jun 15, 2023
I'm closing this issue as it's resolved, feel free to reopen it if needed @sunbin1357 @VijayLuo.
jagdishbhanushali commentedon May 7, 2025
Hi @VijayLuo , I recently just got started with UniAD. I am using latest branch v2.0. I can run eval single gpu, but on dual gpu doesn't work. Same things is for training. Launching training with multiple GPU throws exception without any error. Can you please share log file where you used v2.0 ? Also I noticed Dockerfile haven't changed for new version, it would be really helpful if you could update the dockerfile.
Thanks,
Jagdish Bhanushali