You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Since we are editing the cutoff distance of the data generation tutorials in #528, when running the tutorials' training with the new settings we get nan losses. The reason is that the cutoff is now too low for the network to learn anything.
I got confused since this was not happening using my local environment from the main branch, and I think that the same happened to @DaniBodor.
Then I looked at the different HDF5 files generated and I noticed that the number of nodes and edges in each data point between the two "env" versions were different. So at first (not thinking at all about the cutoff distance edit, for some reason I missed it looking at the PR again), I thought that there was some bug going on that caused the processed data to be different in my local env case and the docker env case, either in how we released deeprank2 2.1.1 on PyPi or in some edits we did between 2.1.0 and 2.1.1 versions. I couldn't find anything though, so I double-checked again using both env on the same branch and it turned out to be just confusion between the different settings in the tutorials across the various branches.
The text was updated successfully, but these errors were encountered:
This turned out to be a False alarm.
Since we are editing the cutoff distance of the data generation tutorials in #528, when running the tutorials' training with the new settings we get nan losses. The reason is that the cutoff is now too low for the network to learn anything.
I got confused since this was not happening using my local environment from the main branch, and I think that the same happened to @DaniBodor.
Then I looked at the different HDF5 files generated and I noticed that the number of nodes and edges in each data point between the two "env" versions were different. So at first (not thinking at all about the cutoff distance edit, for some reason I missed it looking at the PR again), I thought that there was some bug going on that caused the processed data to be different in my local env case and the docker env case, either in how we released deeprank2 2.1.1 on PyPi or in some edits we did between 2.1.0 and 2.1.1 versions. I couldn't find anything though, so I double-checked again using both env on the same branch and it turned out to be just confusion between the different settings in the tutorials across the various branches.
The text was updated successfully, but these errors were encountered: