Patchwork [9,of,9] hgweb: support multi-threaded zstandard compression for wire protocol

login
register
mail settings
Submitter Gregory Szorc
Date April 1, 2017, 10:32 p.m.
Message ID <197776401203a7627167.1491085920@ubuntu-vm-main>
Download mbox | patch
Permalink /patch/19899/
State Changes Requested
Headers show

Comments

Gregory Szorc - April 1, 2017, 10:32 p.m.
# HG changeset patch
# User Gregory Szorc <gregory.szorc@gmail.com>
# Date 1491085873 25200
#      Sat Apr 01 15:31:13 2017 -0700
# Node ID 197776401203a7627167f18481749860f9c69448
# Parent  c171ff452d36d4caeab1887df4da43e45a61022a
hgweb: support multi-threaded zstandard compression for wire protocol

Multi-threaded compression seems like the kind of thing that a server
operator may want to use - especially server operators that like
running their servers close to 100% CPU utilization (the cycles are
there, why not use them). Since the feature is easy enough to
implement, let's add it.

The default behavior is still to use a single thread for compression.
This is a reasonable default: server operators likely tune things so
the maximum concurrent requests don't oversaturate the machine. We
don't want to change the performance characterists of the server by
enabling this feature by default. Plus, as the help docs say,
multi-threaded compression doesn't help much for low compression
levels because Mercurial or the client can't produce or consume the
data fast enough.

Patch

diff --git a/mercurial/help/config.txt b/mercurial/help/config.txt
--- a/mercurial/help/config.txt
+++ b/mercurial/help/config.txt
@@ -1682,6 +1682,22 @@  Controls generic server settings.
 
     See also ``server.zliblevel``.
 
+``zstdthreads``
+    Integer number of threads to use for compressing data. The default value
+    (``0``) uses a single thread. Negative values attempt to detect the number
+    of available CPU cores and use that many threads.
+
+    For low compression levels, zstandard will be able to compress data faster
+    than either Mercurial can send it or the client can consume it, thus
+    undermining the benefits of multi-threaded compression. Therefore,
+    multi-threaded compression is most useful when combined with ``zstdlevel``
+    to increase the compression level while preserving reasonable throughput.
+
+    Each compression operation will use this many threads. So if the value is
+    ``4`` and there are ``3`` requests being served concurrently, ``12``
+    threads will perform compression. It is important to consider the effects
+    on server scaling and performance before setting this option.
+
 ``smtp``
 --------
 
diff --git a/mercurial/hgweb/protocol.py b/mercurial/hgweb/protocol.py
--- a/mercurial/hgweb/protocol.py
+++ b/mercurial/hgweb/protocol.py
@@ -127,6 +127,11 @@  class webproto(wireproto.abstractserverp
                     if level is not None:
                         opts['level'] = level
 
+                    if engine.name() == 'zstd':
+                        threads = self.ui.configint('server', 'zstdthreads')
+                        if threads:
+                            opts['threads'] = threads
+
                     return HGTYPE2, engine, opts
 
             # No mutually supported compression format. Fall back to the