Patchwork [1,of,2] highlight: configurable file size limit (issue3005)

login
register
mail settings
Submitter Anton Shestakov
Date Aug. 3, 2015, 2:54 p.m.
Message ID <1426bfe350e2070e0fca.1438613644@neuro>
Download mbox | patch
Permalink /patch/10083/
State Changes Requested
Headers show

Comments

Anton Shestakov - Aug. 3, 2015, 2:54 p.m.
# HG changeset patch
# User Anton Shestakov <av6@dwimlabs.net>
# Date 1438508200 -28800
#      Sun Aug 02 17:36:40 2015 +0800
# Branch stable
# Node ID 1426bfe350e2070e0fcaf7c0a083e4e385f83805
# Parent  79f0cb97d7537a7c2948f8f9b0a89148825a3a1d
highlight: configurable file size limit (issue3005)

The issue is titled: "MemoryError when using highlight in hgweb", and one of
the proposed solutions to the problem was to make a configurable limit for
highlight extension so it doesn't pygmentize files above a certain size.

Let's introduce such an option, named highlightmax and set its default to 5 MB
(this value is, again, taken from the issue). If a file is above this limit,
highlight extension skips it completely.
Yuya Nishihara - Aug. 4, 2015, 3:03 p.m.
On Mon, 03 Aug 2015 22:54:04 +0800, Anton Shestakov wrote:
> # HG changeset patch
> # User Anton Shestakov <av6@dwimlabs.net>
> # Date 1438508200 -28800
> #      Sun Aug 02 17:36:40 2015 +0800
> # Branch stable
> # Node ID 1426bfe350e2070e0fcaf7c0a083e4e385f83805
> # Parent  79f0cb97d7537a7c2948f8f9b0a89148825a3a1d
> highlight: configurable file size limit (issue3005)
> 
> The issue is titled: "MemoryError when using highlight in hgweb", and one of
> the proposed solutions to the problem was to make a configurable limit for
> highlight extension so it doesn't pygmentize files above a certain size.
> 
> Let's introduce such an option, named highlightmax and set its default to 5 MB
> (this value is, again, taken from the issue). If a file is above this limit,
> highlight extension skips it completely.
> 
> diff --git a/hgext/highlight/__init__.py b/hgext/highlight/__init__.py
> --- a/hgext/highlight/__init__.py
> +++ b/hgext/highlight/__init__.py
> @@ -13,12 +13,11 @@
>  It depends on the Pygments syntax highlighting library:
>  http://pygments.org/
>  
> -There is a single configuration option::
> +There are two configuration options::
>  
>    [web]
> -  pygments_style = <style>
> -
> -The default is 'colorful'.
> +  pygments_style = <style> (default: 'colorful')
> +  highlightmax = <file size> (default: 5 MB)
>  """
>  
>  import highlight
> @@ -41,7 +40,9 @@ def filerevision_highlight(orig, web, re
>      # pygmentize a html file
>      if 'html' in mt:
>          style = web.config('web', 'pygments_style', 'colorful')
> -        highlight.pygmentize('fileline', fctx, style, tmpl)
> +        maxbytes = web.configbytes('web', 'highlightmax', '5 MB')

These patches look good to me.

Just an idea: it could be a fileset to mitigate a bug of pygments, for example,
to enable known-good parser for small .py files:

  set:**.py and size('<5M')
Anton Shestakov - Aug. 4, 2015, 3:24 p.m.
On Wed, 5 Aug 2015 00:03:25 +0900
Yuya Nishihara <yuya@tcha.org> wrote:
> 
> Just an idea: it could be a fileset to mitigate a bug of pygments,
> for example, to enable known-good parser for small .py files:
> 
>   set:**.py and size('<5M')

I like this idea. I was thinking of another possible config option for
highlight with allowed/denied extensions, but filesets are much more
flexible.

Since they can fully cover what highlightmax would do now, I'll try to
figure out how to do fileset matching and do that instead.

Please drop this patch. The second patch (exit early) doesn't depend on
this one.

Patch

diff --git a/hgext/highlight/__init__.py b/hgext/highlight/__init__.py
--- a/hgext/highlight/__init__.py
+++ b/hgext/highlight/__init__.py
@@ -13,12 +13,11 @@ 
 It depends on the Pygments syntax highlighting library:
 http://pygments.org/
 
-There is a single configuration option::
+There are two configuration options::
 
   [web]
-  pygments_style = <style>
-
-The default is 'colorful'.
+  pygments_style = <style> (default: 'colorful')
+  highlightmax = <file size> (default: 5 MB)
 """
 
 import highlight
@@ -41,7 +40,9 @@  def filerevision_highlight(orig, web, re
     # pygmentize a html file
     if 'html' in mt:
         style = web.config('web', 'pygments_style', 'colorful')
-        highlight.pygmentize('fileline', fctx, style, tmpl)
+        maxbytes = web.configbytes('web', 'highlightmax', '5 MB')
+        if fctx.size() <= maxbytes:
+            highlight.pygmentize('fileline', fctx, style, tmpl)
     return orig(web, req, tmpl, fctx)
 
 def annotate_highlight(orig, web, req, tmpl):
@@ -49,7 +50,9 @@  def annotate_highlight(orig, web, req, t
     if 'html' in mt:
         fctx = webutil.filectx(web.repo, req)
         style = web.config('web', 'pygments_style', 'colorful')
-        highlight.pygmentize('annotateline', fctx, style, tmpl)
+        maxbytes = web.configbytes('web', 'highlightmax', '5 MB')
+        if fctx.size() <= maxbytes:
+            highlight.pygmentize('annotateline', fctx, style, tmpl)
     return orig(web, req, tmpl)
 
 def generate_css(web, req, tmpl):
diff --git a/mercurial/hgweb/hgweb_mod.py b/mercurial/hgweb/hgweb_mod.py
--- a/mercurial/hgweb/hgweb_mod.py
+++ b/mercurial/hgweb/hgweb_mod.py
@@ -104,6 +104,10 @@  class hgweb(object):
         return self.repo.ui.configbool(section, name, default,
                                        untrusted=untrusted)
 
+    def configbytes(self, section, name, default=0, untrusted=True):
+        return self.repo.ui.configbytes(section, name, default,
+                                       untrusted=untrusted)
+
     def configlist(self, section, name, default=None, untrusted=True):
         return self.repo.ui.configlist(section, name, default,
                                        untrusted=untrusted)