Patchwork D7178: [RFC] rust-matchers: add `Matcher` trait and implement `AlwaysMatcher`

login
register
mail settings
Submitter phabricator
Date Oct. 29, 2019, 4:27 p.m.
Message ID <differential-rev-PHID-DREV-vqvamxsmoxe6c2qjl5vb-req@mercurial-scm.org>
Download mbox | patch
Permalink /patch/42613/
State Superseded
Headers show

Comments

phabricator - Oct. 29, 2019, 4:27 p.m.
Alphare created this revision.
Herald added subscribers: mercurial-devel, kevincox, durin42.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  In our quest of a faster Mercurial, we have arrived at the point where we need
  to implement the matchers in Rust.
  This RFC mainly for the `Matcher` trait to see if the changes proposed feel
  fine to people with more experience on the matter. While the `AlwaysMatcher`
  implementation is here as a trivial example, it should be the first step
  towards matchers use in Rust as it is currently the only supported one.
  
  Notable changes:
  
  - `exact` is renamed to `exact_match`
  - enums for `visit*` methods with `Recursive` instead of `'all'`, etc.
  - a new `roots`, separate from `file_set`
  - no `bad`, `explicitdir` or `traversedir` functions as they can be passed to the high functions instead of the matchers
  
  Thanks to Martin for suggesting the last two (most important) changes and for
  reaching out to help a few weeks ago.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D7178

AFFECTED FILES
  rust/hg-core/src/lib.rs
  rust/hg-core/src/matchers.rs

CHANGE DETAILS




To: Alphare, #hg-reviewers
Cc: durin42, kevincox, mercurial-devel
phabricator - Oct. 30, 2019, 7:12 a.m.
martinvonz added inline comments.
martinvonz added subscribers: spectral, martinvonz.

INLINE COMMENTS

> matchers.rs:19-22
> +    Recursive,
> +    Empty,
> +    Set(HashSet<HgPathBuf>),  // Should we implement a `NonEmptyHashSet`?
> +    This,

Could you add a comment explaining what each value means? (I *think* `Recursive` means to match all files recursively here, `This` means to visit this directory and all subdirectories, `Empty` means to not visit any subdirectories, and `Set` means to visit exactly that set.)

> matchers.rs:20-21
> +    Recursive,
> +    Empty,
> +    Set(HashSet<HgPathBuf>),  // Should we implement a `NonEmptyHashSet`?
> +    This,

Is `Empty` needed as an optimization? Or could we just use an empty set?

> matchers.rs:25
> +
> +pub trait Matcher {
> +    /// Returns whether `filename` is in `file_set`

It may be better to remove everything we're not sure we'll need to start with. That way it's easier for you to add new matchers and we won't have to clean up later if we never end up needing some of these things. Things I think we can remove to start with:

`roots()`
`prefix()`
`anypats()`

> matchers.rs:27
> +    /// Returns whether `filename` is in `file_set`
> +    fn exact_match(&self, _filename: impl AsRef<HgPath>) -> bool;
> +    fn file_set(&self) -> Option<HashSet<&HgPath>> {

Drop the `_` prefix from `_filename`?

> matchers.rs:28
> +    fn exact_match(&self, _filename: impl AsRef<HgPath>) -> bool;
> +    fn file_set(&self) -> Option<HashSet<&HgPath>> {
> +        None

`Option<CollectionType<T>>` is usually suspicious, IMO. Can we just return an empty set instead of `None`?

> matchers.rs:32
> +    fn roots(&self) -> Option<HashSet<&HgPath>> {
> +        None
> +    }

I think `{''}` (in Python syntax) makes more sense here.

> matchers.rs:35
> +    /// Returns whether `filename` is matched by this matcher
> +    fn match_fn(&self, _filename: impl AsRef<HgPath>) -> bool {
> +        true

Drop the `_` prefix here too. Does the compiler really warn about unused arguments even in trait definitions?

> matchers.rs:35
> +    /// Returns whether `filename` is matched by this matcher
> +    fn match_fn(&self, _filename: impl AsRef<HgPath>) -> bool {
> +        true

Call it `matches()` instead?

> matchers.rs:36
> +    fn match_fn(&self, _filename: impl AsRef<HgPath>) -> bool {
> +        true
> +    }

Default to `false` like the Python version does?

> matchers.rs:39-49
> +    /// Decides whether a directory should be visited based on whether it
> +    /// has potential matches in it or one of its subdirectories. This is
> +    /// based on the match's primary, included, and excluded patterns.
> +    ///
> +    /// Returns `VisitDir::Recursive` if the given directory and all
> +    /// subdirectories should be visited.
> +    /// Otherwise returns `VisitDir::This` or `VisitDir::No` indicating whether

Maybe we can try without this one? It doesn't seem like we need `visit_dir()` now that we have `visit_children_set()` and it seems to have a superset of the functionality. I asked @spectral about that and they thought we should only need the latter, except that it may be slower in some cases, since it needs to return a set of children to visit. However, it seems (to me) that that cost should be made up for by the speed gained by not visiting uninteresting subdirectories. So I would suggest not adding `visit_dir()` to start with and we can add it later if it seems useful in some case. We should also see if we can switch over existing uses of `visit_dir()` to use `visit_children_set()` in the Python code.

> matchers.rs:104
> +    /// optimization might be possible.
> +    fn always(&self) -> bool {
> +        false

Should this be `is_always()` per Rust naming conventions?

> matchers.rs:112-122
> +    /// Matcher will match the paths in `files_set()` recursively: optimization
> +    /// might be possible
> +    fn prefix(&self) -> bool {
> +        false
> +    }
> +    /// None of `.always()`, `.is_exact()`, and `.prefix()` is true:
> +    /// optimizations will be difficult.

I think `is_always()` and `is_exact()` are very often useful for optimizations, but I'm not sure about these two.

> matchers.rs:114
> +    /// might be possible
> +    fn prefix(&self) -> bool {
> +        false

Similarly, should this be `is_prefix()`?

> matchers.rs:119
> +    /// optimizations will be difficult.
> +    fn any_pats(&self) -> bool {
> +        // TODO rename? It's confusing

And here

> matchers.rs:130-132
> +    fn exact_match(&self, _filename: impl AsRef<HgPath>) -> bool {
> +        false
> +    }

Seems like a good default implementation. Put this in the trait instead?

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7178/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7178

To: Alphare, #hg-reviewers
Cc: martinvonz, spectral, durin42, kevincox, mercurial-devel
phabricator - Oct. 30, 2019, 3:24 p.m.
kevincox added a comment.


  I see a lot of the functions are here to give optimization hints. In order to make someone non-familiar with the code able to understand it each method should state the contracts that it is making. I am having a really hard time reconciling how the different functions interact and which methods have precedence over each other.

INLINE COMMENTS

> matchers.rs:28
> +    fn exact_match(&self, _filename: impl AsRef<HgPath>) -> bool;
> +    fn file_set(&self) -> Option<HashSet<&HgPath>> {
> +        None

Please document.

> matchers.rs:31
> +    }
> +    fn roots(&self) -> Option<HashSet<&HgPath>> {
> +        None

Please document.

> martinvonz wrote in matchers.rs:104
> Should this be `is_always()` per Rust naming conventions?

always isn't an adjective so I don't think that really makes sense. How about `matches_everything`.

> matchers.rs:104
> +    /// optimization might be possible.
> +    fn always(&self) -> bool {
> +        false

It is probably worth noting that false-negatives are fine but false-positives will lead to incorrect behaviour.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7178/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7178

To: Alphare, #hg-reviewers
Cc: martinvonz, spectral, durin42, kevincox, mercurial-devel
phabricator - Oct. 31, 2019, 1:40 p.m.
Alphare added inline comments.

INLINE COMMENTS

> martinvonz wrote in matchers.rs:20-21
> Is `Empty` needed as an optimization? Or could we just use an empty set?

I feel like Rust enums are really cheap to make and maintain, and since the original code hints at possible optimizations for this case I feel like it's worth it for readability. We can always back out of this if it turns out to be a pain

> martinvonz wrote in matchers.rs:35
> Drop the `_` prefix here too. Does the compiler really warn about unused arguments even in trait definitions?

It does when it's implemented. `matches()` sounds good

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7178/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7178

To: Alphare, #hg-reviewers
Cc: martinvonz, spectral, durin42, kevincox, mercurial-devel
phabricator - Nov. 6, 2019, 1:16 a.m.
martinvonz added a comment.


  From `test-check-commit.t`:
  
    @@ -25,3 +25,10 @@
       >     fi
       >   done
       > fi
    +  Revision e4f5ab3faa27 does not comply with rules
    +  ------------------------------------------------------
    +  6: summary line doesn't start with 'topic: '
    +   [RFC] rust-matchers: add `Matcher` trait and implement `AlwaysMatcher`
    +  6: summary keyword should be most user-relevant one-word command or topic
    +   [RFC] rust-matchers: add `Matcher` trait and implement `AlwaysMatcher`
    +

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7178/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7178

To: Alphare, #hg-reviewers
Cc: martinvonz, spectral, durin42, kevincox, mercurial-devel
phabricator - Nov. 6, 2019, 1:21 a.m.
martinvonz added a comment.


  In D7178#106175 <https://phab.mercurial-scm.org/D7178#106175>, @martinvonz wrote:
  
  > From `test-check-commit.t`:
  >
  >   @@ -25,3 +25,10 @@
  >      >     fi
  >      >   done
  >      > fi
  >   +  Revision e4f5ab3faa27 does not comply with rules
  >   +  ------------------------------------------------------
  >   +  6: summary line doesn't start with 'topic: '
  >   +   [RFC] rust-matchers: add `Matcher` trait and implement `AlwaysMatcher`
  >   +  6: summary keyword should be most user-relevant one-word command or topic
  >   +   [RFC] rust-matchers: add `Matcher` trait and implement `AlwaysMatcher`
  >   +
  
  Sorry, I didn't even see the `[RFC]` prefix, which is what the test was complaining about. I thought it didn't like the hyphen. I'll just drop the `[RFC]` in flight then.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D7178/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D7178

To: Alphare, #hg-reviewers
Cc: martinvonz, spectral, durin42, kevincox, mercurial-devel

Patch

diff --git a/rust/hg-core/src/matchers.rs b/rust/hg-core/src/matchers.rs
new file mode 100644
--- /dev/null
+++ b/rust/hg-core/src/matchers.rs
@@ -0,0 +1,142 @@ 
+// matchers.rs
+//
+// Copyright 2019 Raphaël Gomès <rgomes@octobus.net>
+//
+// This software may be used and distributed according to the terms of the
+// GNU General Public License version 2 or any later version.
+
+//! Structs and types for matching files and directories.
+
+use crate::utils::hg_path::{HgPath, HgPathBuf};
+use std::collections::HashSet;
+
+pub enum VisitDir {
+    Recursive,
+    This,
+    No,
+}
+pub enum VisitChildrenSet {
+    Recursive,
+    Empty,
+    Set(HashSet<HgPathBuf>),  // Should we implement a `NonEmptyHashSet`?
+    This,
+}
+
+pub trait Matcher {
+    /// Returns whether `filename` is in `file_set`
+    fn exact_match(&self, _filename: impl AsRef<HgPath>) -> bool;
+    fn file_set(&self) -> Option<HashSet<&HgPath>> {
+        None
+    }
+    fn roots(&self) -> Option<HashSet<&HgPath>> {
+        None
+    }
+    /// Returns whether `filename` is matched by this matcher
+    fn match_fn(&self, _filename: impl AsRef<HgPath>) -> bool {
+        true
+    }
+
+    /// Decides whether a directory should be visited based on whether it
+    /// has potential matches in it or one of its subdirectories. This is
+    /// based on the match's primary, included, and excluded patterns.
+    ///
+    /// Returns `VisitDir::Recursive` if the given directory and all
+    /// subdirectories should be visited.
+    /// Otherwise returns `VisitDir::This` or `VisitDir::No` indicating whether
+    /// the given directory should be visited.
+    fn visit_dir(&self, _directory: impl AsRef<HgPath>) -> VisitDir {
+        VisitDir::This
+    }
+
+    /// Decides whether a directory should be visited based on whether it
+    /// has potential matches in it or one of its subdirectories, and
+    /// potentially lists which subdirectories of that directory should be
+    /// visited. This is based on the match's primary, included, and excluded
+    /// patterns.
+    ///
+    /// This function is very similar to `visit_dir`, and the following mapping
+    /// can be applied:
+    ///
+    ///  `VisitDir`  | `VisitChildrenSet`
+    /// -------------+-------------------
+    ///  `No`        | `Empty`
+    ///  `Recursive` | `Recursive`
+    ///  `This`      | `This` OR non-empty `Set` of subdirs -or files- to visit
+    ///
+    /// # Example
+    ///
+    /// Assume matchers `['path:foo/bar', 'rootfilesin:qux']`, we would
+    /// return the following values (assuming the implementation of
+    /// visit_children_set is capable of recognizing this; some implementations
+    /// are not).
+    ///
+    /// ```ignore
+    /// '' -> {'foo', 'qux'}
+    /// 'baz' -> set()
+    /// 'foo' -> {'bar'}
+    /// // Ideally this would be `Recursive`, but since the prefix nature of
+    /// // matchers is applied to the entire matcher, we have to downgrade this
+    /// // to `This` due to the (yet to be implemented in Rust) non-prefix
+    /// // `RootFilesIn'-kind matcher being mixed in.
+    /// 'foo/bar' -> 'this'
+    /// 'qux' -> 'this'
+    /// ```
+    /// # Important
+    ///
+    /// Most matchers do not know if they're representing files or
+    /// directories. They see `['path:dir/f']` and don't know whether `f` is a
+    /// file or a directory, so `visit_children_set('dir')` for most matchers
+    /// will return `HashSet{ HgPath { "f" } }`, but if the matcher knows it's
+    /// a file (like the yet to be implemented in Rust `ExactMatcher` does),
+    /// it may return `VisitChildrenSet::This`.
+    /// Do not rely on the return being a `HashSet` indicating that there are
+    /// no files in this dir to investigate (or equivalently that if there are
+    /// files to investigate in 'dir' that it will always return
+    /// `VisitChildrenSet::This`).
+    fn visit_children_set(
+        &self,
+        _directory: impl AsRef<HgPath>,
+    ) -> VisitChildrenSet {
+        VisitChildrenSet::This
+    }
+    /// Matcher will match everything and `files_set()` will be empty:
+    /// optimization might be possible.
+    fn always(&self) -> bool {
+        false
+    }
+    /// Matcher will match exactly the files in `files_set()`: optimization
+    /// might be possible.
+    fn is_exact(&self) -> bool {
+        false
+    }
+    /// Matcher will match the paths in `files_set()` recursively: optimization
+    /// might be possible
+    fn prefix(&self) -> bool {
+        false
+    }
+    /// None of `.always()`, `.is_exact()`, and `.prefix()` is true:
+    /// optimizations will be difficult.
+    fn any_pats(&self) -> bool {
+        // TODO rename? It's confusing
+        !self.always() && !self.is_exact() && !self.prefix()
+    }
+}
+
+/// Matches everything.
+#[derive(Debug)]
+pub struct AlwaysMatcher;
+
+impl Matcher for AlwaysMatcher {
+    fn exact_match(&self, _filename: impl AsRef<HgPath>) -> bool {
+        false
+    }
+    fn visit_dir(&self, _directory: impl AsRef<HgPath>) -> VisitDir {
+        VisitDir::Recursive
+    }
+    fn visit_children_set(
+        &self,
+        _directory: impl AsRef<HgPath>,
+    ) -> VisitChildrenSet {
+        VisitChildrenSet::Recursive
+    }
+}
diff --git a/rust/hg-core/src/lib.rs b/rust/hg-core/src/lib.rs
--- a/rust/hg-core/src/lib.rs
+++ b/rust/hg-core/src/lib.rs
@@ -17,6 +17,7 @@ 
     StateMap, StateMapIter,
 };
 mod filepatterns;
+pub mod matchers;
 pub mod utils;
 
 use crate::utils::hg_path::HgPathBuf;