Patchwork [1,of,3] transaction: delete callbacks after use

login
register
mail settings
Submitter Gregory Szorc
Date May 27, 2017, 12:42 a.m.
Message ID <922d86488388b186fa40.1495845754@ubuntu-vm-main>
Download mbox | patch
Permalink /patch/20954/
State Accepted
Headers show

Comments

Gregory Szorc - May 27, 2017, 12:42 a.m.
# HG changeset patch
# User Gregory Szorc <gregory.szorc@gmail.com>
# Date 1495830441 25200
#      Fri May 26 13:27:21 2017 -0700
# Node ID 922d86488388b186fa407eae78fbb811c57a3293
# Parent  ad37c569ec8121a99e4cd9da52365b114b82e744
transaction: delete callbacks after use

Before this change, localrepository instances that performed multiple
transactions would leak transaction objects. This could occur when
running `hg convert`. When running `hg convert`, the leak would be
~90 MB per 10,000 changesets as measured with the Mercurial repo itself.

The leak I tracked down involved the "validate" closure from
localrepository.transaction(). It appeared to be keeping a
reference to the original transaction via __closure__. __del__
semantics and a circular reference involving the repo object
may have also come into play.

Attempting to refactor the "validate" closure proved to be
difficult because the "tr" reference in that closure may
reference an object that isn't created until transaction.__init__
is called. And the "validate" closure is passed as an argument to
transaction.__init__. Plus there is a giant warning comment in
"validate" about how hacky it is. I did not want to venture into
the dragon den.

Anyway, we've had problems with transactions causing leaks before.
The solution then (14e683d6b273) is the same as the solution in this
patch: drop references to callbacks after they are called. This
not only breaks cycles in core Mercurial but can help break cycles
in extensions that accidentally introduce them.

While I only tracked down a leak due to self.validator, since this is
the 2nd time I've tracked down leaks due to transaction callbacks I
figure enough is enough and we should prevent the class of leak from
occurring regardless of the variable. That's why all callback variables
are now nuked.
Yuya Nishihara - May 31, 2017, 1:10 p.m.
On Fri, 26 May 2017 17:42:34 -0700, Gregory Szorc wrote:
> # HG changeset patch
> # User Gregory Szorc <gregory.szorc@gmail.com>
> # Date 1495830441 25200
> #      Fri May 26 13:27:21 2017 -0700
> # Node ID 922d86488388b186fa407eae78fbb811c57a3293
> # Parent  ad37c569ec8121a99e4cd9da52365b114b82e744
> transaction: delete callbacks after use

Queued this, thanks.

> --- a/mercurial/transaction.py
> +++ b/mercurial/transaction.py
> @@ -431,6 +431,7 @@ class transaction(object):
>          '''commit the transaction'''
>          if self.count == 1:
>              self.validator(self)  # will raise exception if needed
> +            self.validator = None # Help prevent cycles.
>              self._generatefiles(group=gengroupprefinalize)
>              categories = sorted(self._finalizecallback)
>              for cat in categories:
> @@ -464,6 +465,7 @@ class transaction(object):
>          self._writeundo()
>          if self.after:
>              self.after()
> +            self.after = None # Help prevent cycles.
>          if self.opener.isfile(self._backupjournal):
>              self.opener.unlink(self._backupjournal)
>          if self.opener.isfile(self.journal):
> @@ -487,6 +489,7 @@ class transaction(object):
>          self.journal = None
>  
>          self.releasefn(self, True) # notify success of closing transaction
> +        self.releasefn = None # Help prevent cycles.

I prefer setting a null function to validator and releasefn as we do in
__init__, but they aren't called in __del__ (because journal is None), so
that shouldn't matter.

Patch

diff --git a/mercurial/transaction.py b/mercurial/transaction.py
--- a/mercurial/transaction.py
+++ b/mercurial/transaction.py
@@ -431,6 +431,7 @@  class transaction(object):
         '''commit the transaction'''
         if self.count == 1:
             self.validator(self)  # will raise exception if needed
+            self.validator = None # Help prevent cycles.
             self._generatefiles(group=gengroupprefinalize)
             categories = sorted(self._finalizecallback)
             for cat in categories:
@@ -464,6 +465,7 @@  class transaction(object):
         self._writeundo()
         if self.after:
             self.after()
+            self.after = None # Help prevent cycles.
         if self.opener.isfile(self._backupjournal):
             self.opener.unlink(self._backupjournal)
         if self.opener.isfile(self.journal):
@@ -487,6 +489,7 @@  class transaction(object):
         self.journal = None
 
         self.releasefn(self, True) # notify success of closing transaction
+        self.releasefn = None # Help prevent cycles.
 
         # run post close action
         categories = sorted(self._postclosecallback)
@@ -557,6 +560,7 @@  class transaction(object):
         finally:
             self.journal = None
             self.releasefn(self, False) # notify failure of transaction
+            self.releasefn = None # Help prevent cycles.
 
 def rollback(opener, vfsmap, file, report):
     """Rolls back the transaction contained in the given file