Problems in Reproducing the Results

[Ann997943]

I tried to reproduce the results of Figure 2, which show the impact of increasing test-time computation in reflection. I modified the maximum number of iterations in locomo_test.py and hotpotqa.py, and evaluated on locomo10.json and eval_400.json. However, the F1 score did not improve, and it seems that the deep reflection mechanism did not work as expected. Could you please advise on how to resolve this issue?

iteration1_batch_results_0_9.json
iteration1_batch_statistics_0_9.json

iteration3_batch_results_0_9.json
iteration3_batch_statistics_0_9.json

[Cooki-e]

Thanks for reporting this issue. Could you share a few more details about your experiment, such as the model you used, the code version, the exact changes you made to the reflection settings, and the final results? That would help us verify the issue and better understand why deeper reflection did not seem to improve performance in your setup.