Patchwork [V3] fsmonitor: match watchman and filesystem encoding

login
register
mail settings
Submitter Olivier Trempe
Date April 7, 2017, 4:32 p.m.
Message ID <867e2a3b106825dbd637.1491582740@ronce.innov.local>
Download mbox | patch
Permalink /patch/20003/
State Accepted
Headers show

Comments

Olivier Trempe - April 7, 2017, 4:32 p.m.
# HG changeset patch
# User Olivier Trempe <oliviertrempe@gmail.com>
# Date 1488981822 18000
#      Wed Mar 08 09:03:42 2017 -0500
# Branch stable
# Node ID 867e2a3b106825dbd637aa404115301f9df70bfd
# Parent  68f263f52d2e3e2798b4f1e55cb665c6b043f93b
fsmonitor: match watchman and filesystem encoding

watchman's paths encoding can differ from filesystem encoding. For example,
on Windows, it's always utf-8.

Before this patch, on Windows, mismatch in path comparison between fsmonitor
state and osutil.statfiles would yield a clean status for added/modified files.

In addition to status reporting wrong results, this leads to files being
discarded from changesets while doing history editing operations such as rebase.

Benchmark:

There is a little overhead at module import:
python -m timeit "import hgext.fsmonitor"
Windows before patch: 1000000 loops, best of 3: 0.563 usec per loop
Windows after patch: 1000000 loops, best of 3: 0.583 usec per loop
Linx before patch: 1000000 loops, best of 3: 0.579 usec per loop
Linux after patch: 1000000 loops, best of 3: 0.588 usec per loop

10000 calls to _watchmantofsencoding:
python -m timeit -s "from hgext.fsmonitor import _watchmantofsencoding, _fixencoding" "fname = '/path/to/file'" "for i in range(10000):" "    if _fixencoding: fname = _watchmantofsencoding(fname)"
Windows (_fixencoding is True): 100 loops, best of 3: 19.5 msec per loop
Linux (_fixencoding is False): 100 loops, best of 3: 3.08 msec per loop
Yuya Nishihara - April 8, 2017, 7:53 a.m.
On Fri, 07 Apr 2017 12:32:20 -0400, Olivier Trempe wrote:
> # HG changeset patch
> # User Olivier Trempe <oliviertrempe@gmail.com>
> # Date 1488981822 18000
> #      Wed Mar 08 09:03:42 2017 -0500
> # Branch stable
> # Node ID 867e2a3b106825dbd637aa404115301f9df70bfd
> # Parent  68f263f52d2e3e2798b4f1e55cb665c6b043f93b
> fsmonitor: match watchman and filesystem encoding

Queued this. Thanks for including the benchmark result in commit message.

Patch

diff -r 68f263f52d2e -r 867e2a3b1068 hgext/fsmonitor/__init__.py
--- a/hgext/fsmonitor/__init__.py	Mon Apr 03 17:34:24 2017 -0400
+++ b/hgext/fsmonitor/__init__.py	Wed Mar 08 09:03:42 2017 -0500
@@ -91,14 +91,17 @@ 
 
 from __future__ import absolute_import
 
+import codecs
 import hashlib
 import os
 import stat
+import sys
 
 from mercurial.i18n import _
 from mercurial import (
     context,
     encoding,
+    error,
     extensions,
     localrepo,
     merge,
@@ -110,6 +113,7 @@ 
 from mercurial import match as matchmod
 
 from . import (
+    pywatchman,
     state,
     watchmanclient,
 )
@@ -159,6 +163,28 @@ 
     sha1.update('\0')
     return sha1.hexdigest()
 
+_watchmanencoding = pywatchman.encoding.get_local_encoding()
+_fsencoding = sys.getfilesystemencoding() or sys.getdefaultencoding()
+_fixencoding = codecs.lookup(_watchmanencoding) != codecs.lookup(_fsencoding)
+
+def _watchmantofsencoding(path):
+    """Fix path to match watchman and local filesystem encoding
+
+    watchman's paths encoding can differ from filesystem encoding. For example,
+    on Windows, it's always utf-8.
+    """
+    try:
+        decoded = path.decode(_watchmanencoding)
+    except UnicodeDecodeError as e:
+        raise error.Abort(str(e), hint='watchman encoding error')
+
+    try:
+        encoded = decoded.encode(_fsencoding, 'strict')
+    except UnicodeEncodeError as e:
+        raise error.Abort(str(e))
+
+    return encoded
+
 def overridewalk(orig, self, match, subrepos, unknown, ignored, full=True):
     '''Replacement for dirstate.walk, hooking into Watchman.
 
@@ -303,6 +329,8 @@ 
     # for name case changes.
     for entry in result['files']:
         fname = entry['name']
+        if _fixencoding:
+            fname = _watchmantofsencoding(fname)
         if switch_slashes:
             fname = fname.replace('\\', '/')
         if normalize: