Patchwork [4,of,4] convert: parse perforce data on-demand

login
register
mail settings
Submitter David Soria Parra
Date Dec. 20, 2016, 6:07 p.m.
Message ID <a047a142f151709f8823.1482257242@devbig415.prn1.facebook.com>
Download mbox | patch
Permalink /patch/17983/
State Accepted
Headers show

Comments

David Soria Parra - Dec. 20, 2016, 6:07 p.m.
# HG changeset patch
# User David Soria Parra <davidsp@fb.com>
# Date 1482254630 28800
#      Tue Dec 20 09:23:50 2016 -0800
# Node ID a047a142f151709f882361916617e762816db619
# Parent  77e1912e2e0817a96633568c400450a9566e7e33
convert: parse perforce data on-demand

We are using read-only attributes that parse the perforce data on
demand. We are reading the data only once whenever an attribute is
requested and use it throughout the import process. This is equivalent
to the previous behavior, but we are avoiding reading from perforce when
we initialize the object, but instead run it during the actual import
process, when the first attribute is requested (usually getheads(), see
`convertcmd.convert`).
Augie Fackler - Dec. 20, 2016, 7:37 p.m.
On Tue, Dec 20, 2016 at 10:07:22AM -0800, David Soria Parra wrote:
> # HG changeset patch
> # User David Soria Parra <davidsp@fb.com>
> # Date 1482254630 28800
> #      Tue Dec 20 09:23:50 2016 -0800
> # Node ID a047a142f151709f882361916617e762816db619
> # Parent  77e1912e2e0817a96633568c400450a9566e7e33
> convert: parse perforce data on-demand

Queued these, thanks. How does it feel to be the convert maintainer?

>
> We are using read-only attributes that parse the perforce data on
> demand. We are reading the data only once whenever an attribute is
> requested and use it throughout the import process. This is equivalent
> to the previous behavior, but we are avoiding reading from perforce when
> we initialize the object, but instead run it during the actual import
> process, when the first attribute is requested (usually getheads(), see
> `convertcmd.convert`).
>
> diff --git a/hgext/convert/p4.py b/hgext/convert/p4.py
> --- a/hgext/convert/p4.py
> +++ b/hgext/convert/p4.py
> @@ -56,13 +56,8 @@
>          common.checktool('p4', abort=False)
>
>          self.revmap = {}
> -        self.heads = []
> -        self.changeset = {}
> -        self.files = {}
> -        self.copies = {}
>          self.encoding = self.ui.config('convert', 'p4.encoding',
>                                         default=convcmd.orig_encoding)
> -        self.depotname = {}           # mapping from local name to depot name
>          self.re_type = re.compile(
>              "([a-z]+)?(text|binary|symlink|apple|resource|unicode|utf\d+)"
>              "(\+\w+)?$")
> @@ -74,7 +69,6 @@
>          if revs and len(revs) > 1:
>              raise error.Abort(_("p4 source does not support specifying "
>                                 "multiple revisions"))
> -        self._parse_once(ui, path)
>
>      def setrevmap(self, revmap):
>          """Sets the parsed revmap dictionary.
> @@ -240,13 +234,29 @@
>              'depotname': depotname,
>          }
>
> -    def _parse_once(self, ui, path):
> -        d = self._parse(ui, path)
> -        self.changeset = d['changeset']
> -        self.heads = d['heads']
> -        self.files = d['files']
> -        self.copies = d['copies']
> -        self.depotname = d['depotname']
> +    @util.propertycache
> +    def _parse_once(self):
> +        return self._parse(self.ui, self.path)
> +
> +    @util.propertycache
> +    def copies(self):
> +        return self._parse_once['copies']
> +
> +    @util.propertycache
> +    def files(self):
> +        return self._parse_once['files']
> +
> +    @util.propertycache
> +    def changeset(self):
> +        return self._parse_once['changeset']
> +
> +    @util.propertycache
> +    def heads(self):
> +        return self._parse_once['heads']
> +
> +    @util.propertycache
> +    def depotname(self):
> +        return self._parse_once['depotname']
>
>      def getheads(self):
>          return self.heads
> diff --git a/tests/test-convert-p4-filetypes.t b/tests/test-convert-p4-filetypes.t
> --- a/tests/test-convert-p4-filetypes.t
> +++ b/tests/test-convert-p4-filetypes.t
> @@ -307,11 +307,11 @@
>  convert
>    $ hg convert -s p4 $DEPOTPATH dst
>    initializing destination dst repository
> +  scanning source...
>    reading p4 views
>    collecting p4 changelists
>    1 initial
>    2 keywords
> -  scanning source...
>    sorting...
>    converting...
>    1 initial
> diff --git a/tests/test-convert-p4.t b/tests/test-convert-p4.t
> --- a/tests/test-convert-p4.t
> +++ b/tests/test-convert-p4.t
> @@ -67,12 +67,12 @@
>  convert
>    $ hg convert -s p4 $DEPOTPATH dst
>    initializing destination dst repository
> +  scanning source...
>    reading p4 views
>    collecting p4 changelists
>    1 initial
>    2 change a
>    3 change b/c
> -  scanning source...
>    sorting...
>    converting...
>    2 initial
> @@ -98,13 +98,10 @@
>
>  convert again
>    $ hg convert -s p4 $DEPOTPATH dst
> +  scanning source...
>    reading p4 views
>    collecting p4 changelists
> -  1 initial
> -  2 change a
> -  3 change b/c
>    4 change a b/c
> -  scanning source...
>    sorting...
>    converting...
>    0 change a b/c
> @@ -130,14 +127,10 @@
>
>  convert again
>    $ hg convert -s p4 $DEPOTPATH dst
> +  scanning source...
>    reading p4 views
>    collecting p4 changelists
> -  1 initial
> -  2 change a
> -  3 change b/c
> -  4 change a b/c
>    5 add d e f
> -  scanning source...
>    sorting...
>    converting...
>    0 add d e f
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Patch

diff --git a/hgext/convert/p4.py b/hgext/convert/p4.py
--- a/hgext/convert/p4.py
+++ b/hgext/convert/p4.py
@@ -56,13 +56,8 @@ 
         common.checktool('p4', abort=False)
 
         self.revmap = {}
-        self.heads = []
-        self.changeset = {}
-        self.files = {}
-        self.copies = {}
         self.encoding = self.ui.config('convert', 'p4.encoding',
                                        default=convcmd.orig_encoding)
-        self.depotname = {}           # mapping from local name to depot name
         self.re_type = re.compile(
             "([a-z]+)?(text|binary|symlink|apple|resource|unicode|utf\d+)"
             "(\+\w+)?$")
@@ -74,7 +69,6 @@ 
         if revs and len(revs) > 1:
             raise error.Abort(_("p4 source does not support specifying "
                                "multiple revisions"))
-        self._parse_once(ui, path)
 
     def setrevmap(self, revmap):
         """Sets the parsed revmap dictionary.
@@ -240,13 +234,29 @@ 
             'depotname': depotname,
         }
 
-    def _parse_once(self, ui, path):
-        d = self._parse(ui, path)
-        self.changeset = d['changeset']
-        self.heads = d['heads']
-        self.files = d['files']
-        self.copies = d['copies']
-        self.depotname = d['depotname']
+    @util.propertycache
+    def _parse_once(self):
+        return self._parse(self.ui, self.path)
+
+    @util.propertycache
+    def copies(self):
+        return self._parse_once['copies']
+
+    @util.propertycache
+    def files(self):
+        return self._parse_once['files']
+
+    @util.propertycache
+    def changeset(self):
+        return self._parse_once['changeset']
+
+    @util.propertycache
+    def heads(self):
+        return self._parse_once['heads']
+
+    @util.propertycache
+    def depotname(self):
+        return self._parse_once['depotname']
 
     def getheads(self):
         return self.heads
diff --git a/tests/test-convert-p4-filetypes.t b/tests/test-convert-p4-filetypes.t
--- a/tests/test-convert-p4-filetypes.t
+++ b/tests/test-convert-p4-filetypes.t
@@ -307,11 +307,11 @@ 
 convert
   $ hg convert -s p4 $DEPOTPATH dst
   initializing destination dst repository
+  scanning source...
   reading p4 views
   collecting p4 changelists
   1 initial
   2 keywords
-  scanning source...
   sorting...
   converting...
   1 initial
diff --git a/tests/test-convert-p4.t b/tests/test-convert-p4.t
--- a/tests/test-convert-p4.t
+++ b/tests/test-convert-p4.t
@@ -67,12 +67,12 @@ 
 convert
   $ hg convert -s p4 $DEPOTPATH dst
   initializing destination dst repository
+  scanning source...
   reading p4 views
   collecting p4 changelists
   1 initial
   2 change a
   3 change b/c
-  scanning source...
   sorting...
   converting...
   2 initial
@@ -98,13 +98,10 @@ 
 
 convert again
   $ hg convert -s p4 $DEPOTPATH dst
+  scanning source...
   reading p4 views
   collecting p4 changelists
-  1 initial
-  2 change a
-  3 change b/c
   4 change a b/c
-  scanning source...
   sorting...
   converting...
   0 change a b/c
@@ -130,14 +127,10 @@ 
 
 convert again
   $ hg convert -s p4 $DEPOTPATH dst
+  scanning source...
   reading p4 views
   collecting p4 changelists
-  1 initial
-  2 change a
-  3 change b/c
-  4 change a b/c
   5 add d e f
-  scanning source...
   sorting...
   converting...
   0 add d e f