From patchwork Wed Oct 21 20:30:55 2020
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Subject: D9236: utils: helper function to print top memory allocation site
From: phabricator <phabricator@mercurial-scm.org>
X-Patchwork-Id: 47502
Message-Id: <differential-rev-PHID-DREV-7swkvyzfmc62dy4sgzm7-req@mercurial-scm.org>
To: Phabricator <phabricator@mercurial-scm.org>
Cc: mercurial-devel@mercurial-scm.org
Date: Wed, 21 Oct 2020 20:30:55 +0000

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  The memorytop function uses Python's tracemalloc module to show the
  source lines / backtraces with the largest remaining allocations. This
  allows identifying the origins of active memory by placing calls in
  strategic locations. Allocations from C extensions will show up as long
  as they are using the Python allocators.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9236

AFFECTED FILES
  mercurial/utils/memorytop.py

CHANGE DETAILS




To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel

diff --git a/mercurial/utils/memorytop.py b/mercurial/utils/memorytop.py
new file mode 100644
--- /dev/null
+++ b/mercurial/utils/memorytop.py
@@ -0,0 +1,45 @@
+# memorytop requires Python 3.4
+#
+# Usage: set PYTHONTRACEMALLOC=n in the environment of the hg invocation,
+# where n>= is the number of frames to show in the backtrace. Put calls to
+# memorytop in strategic places to show the current memory use by allocation
+# site.
+
+import tracemalloc
+import gc
+
+
+def memorytop(limit=10):
+    gc.collect()
+    snapshot = tracemalloc.take_snapshot()
+
+    snapshot = snapshot.filter_traces(
+        (
+            tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
+            tracemalloc.Filter(False, "<frozen importlib._bootstrap_external>"),
+            tracemalloc.Filter(False, "<unknown>"),
+        )
+    )
+    stats = snapshot.statistics('traceback')
+
+    total = sum(stat.size for stat in stats)
+    print("\nTotal allocated size: %.1f KiB\n" % (total / 1024))
+    print("Lines with the biggest net allocations")
+    for index, stat in enumerate(stats[:limit], 1):
+        frame = stat.traceback[0]
+        print(
+            "#%d: %d objects using %.1f KiB"
+            % (index, stat.count, stat.size / 1024)
+        )
+        for line in stat.traceback.format(most_recent_first=True):
+            print('    ', line)
+
+    other = stats[limit:]
+    if other:
+        size = sum(stat.size for stat in other)
+        count = sum(stat.count for stat in other)
+        print(
+            "%s other: %d objects using %.1f KiB"
+            % (len(other), count, size / 1024)
+        )
+    print()