Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix allocation memory in Keras tests #840

Merged
merged 1 commit into from
Oct 23, 2023

Conversation

reuvenperetz
Copy link
Collaborator

@reuvenperetz reuvenperetz commented Oct 23, 2023

Move tests from Keras function tests to a new package of nonparallel tests. This package runs first in GitHub actions separately from other tests since the rest run parallel. This is a workaround to a memory issue because we use tf gradients on a multi-thread/process CPU environment (tensorflow/tensorflow#41718).

Several modifications:

  • Free gradients and gradients tape after using them for the score calculation.
  • Free hessian service in tests.
  • Remove printing code that was accidentally committed in the previous PR.
  • Speed-up tests in nonparallel (smaller models, disabling gradients computation where not needed, etc.)

@@ -139,7 +139,8 @@ def test_steps_by_order(self):
def rep_data():
yield [np.random.randn(1, 8, 8, 3)]

mp_qc = mct.core.MixedPrecisionQuantizationConfigV2(num_of_images=1)
mp_qc = mct.core.MixedPrecisionQuantizationConfigV2(num_of_images=1,
use_grad_based_weights=False)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why use_grad_based_weights=False if it's a non parallel test?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To speed it up. The tensorboard test does not test anything regarding the gradients weighting.

@reuvenperetz reuvenperetz merged commit 25570a0 into sony:main Oct 23, 2023
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants