Patchwork D2686: xdiff: add a preprocessing step that trims files

login
register
mail settings
Submitter phabricator
Date March 5, 2018, 12:50 a.m.
Message ID <differential-rev-PHID-DREV-bzfzspalrih6yaqvbxyn-req@phab.mercurial-scm.org>
Download mbox | patch
Permalink /patch/29036/
State Superseded
Headers show

Comments

phabricator - March 5, 2018, 12:50 a.m.
quark created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  xdiff has a `xdl_trim_ends` step that removes common lines, unmatchable
  lines. That is in theory good, but happens too late - after splitting,
  hashing, and adjusting the hash values so they are unique. Those splitting,
  hashing and adjusting hash values steps could have noticeable overhead.
  
  For not uncommon cases like diffing two large files with minor differences,
  the raw performance of those preparation steps seriously matter. Even
  allocating an O(N) array and storing line offsets to it is expensive.
  Therefore my previous attempts [1] [2] cannot be good enough since they do
  not remove the O(N) array assignment.
  
  This patch adds a preprocessing step - `xdl_trim_files` that runs before
  other preprocessing steps. It counts common prefix and suffix and lines in
  them (needed for displaying line number), without doing anything else.
  
  Testing with an emulated large (57MB) file, minor change case:
  
    open('a','w').write(''.join('%s\n' % (i % 100000) for i in xrange(10000000)))
    open('b','w').write(''.join('%s\n' % (i % 100000) for i in xrange(10000000) if i != 6000000))
  
  This patch improves the perf by more than 5x:
  
    # before this patch
    xdiff a b  0.65s user 0.37s system 98% cpu 1.033 total
    # after this patch, gcc -O2
    xdiff a b  0.10s user 0.08s system 97% cpu 0.179 total
    # after this patch, gcc -O3 -mavx2
    xdiff a b  0.06s user 0.08s system 98% cpu 0.142 total
    # gnu diffutils
    diff a b   0.04s user 0.05s system 98% cpu 0.088 total
    # (best of 20 runs)
  
  It's still slow than GNU diffutils in this case. But it's much closer.
  
  Testing with real repo data:
  
  For the whole repo, this patch makes xdiff 25% faster:
  
    # hg perfbdiff --count 100 --alldata -c d334afc585e2 --blocks [--xdiff]
    # xdiff, after
    ! wall 0.058861 comb 0.050000 user 0.050000 sys 0.000000 (best of 100)
    # xdiff, before
    ! wall 0.077816 comb 0.080000 user 0.080000 sys 0.000000 (best of 91)
    # bdiff
    ! wall 0.117473 comb 0.120000 user 0.120000 sys 0.000000 (best of 67)
  
  For files that are long (ex. commands.py), the speedup is more than 3x, very
  significant:
  
    # hg perfbdiff --count 3000 --blocks commands.py.i 1 [--xdiff]
    # xdiff, after
    ! wall 0.690583 comb 0.690000 user 0.690000 sys 0.000000 (best of 12)
    # xdiff, before
    ! wall 2.240361 comb 2.210000 user 2.210000 sys 0.000000 (best of 4)
    # bdiff
    ! wall 2.469852 comb 2.440000 user 2.440000 sys 0.000000 (best of 4)
  
  [1]: https://phab.mercurial-scm.org/D2631
  [1]: https://phab.mercurial-scm.org/D2634

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D2686

AFFECTED FILES
  mercurial/thirdparty/xdiff/xdiffi.c
  mercurial/thirdparty/xdiff/xemit.c
  mercurial/thirdparty/xdiff/xprepare.c
  mercurial/thirdparty/xdiff/xprepare.h
  mercurial/thirdparty/xdiff/xtypes.h

CHANGE DETAILS




To: quark, #hg-reviewers
Cc: mercurial-devel
phabricator - March 7, 2018, 3:55 a.m.
indygreg added a comment.


  @quark: Will you be refactoring this based on upstream feedback? Or rebasing due to other xdiff changes that have since landed? i.e. should I review this patch now?

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D2686

To: quark, #hg-reviewers
Cc: indygreg, mercurial-devel
phabricator - March 7, 2018, 4:04 a.m.
quark added a comment.


  I'll do the rebase (probably tomorrow). It will make the `xdl_do_diff2` change unnecessary, and maybe rename `prefix_lines` etc to be more like the original code. But the main feature (xdl_trim_files) will probably stay unchanged and is worth a look now.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D2686

To: quark, #hg-reviewers
Cc: indygreg, mercurial-devel
phabricator - March 7, 2018, 6:06 a.m.
quark added a comment.


  The patch sent to the git list was completely different from this one, because git has another layer `xdiff-interface.c` and trimming happens there. git also has some more complexity like context line handling, which is the hard part.
  
  Since the context lines logic was removed and Mercurial does context line in a higher layer, the git upstream's concerns about context lines are invalid here.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D2686

To: quark, #hg-reviewers
Cc: indygreg, mercurial-devel
phabricator - March 8, 2018, 6:04 a.m.
indygreg requested changes to this revision.
indygreg added a comment.
This revision now requires changes to proceed.


  I'm overall pretty happy with this. I'm requesting just a few minor fixups.
  
  Also, if you (or anyone else for that matter) wanted to spend time to use better variable names and add comments throughout this code, it would be greatly appreciated. I find this code challenging to read because of its almost non-existent documentation.

INLINE COMMENTS

> xprepare.c:169
> +	long plines = 0, pbytes = 0, slines = 0, sbytes = 0, i;
> +	const char *pp1, *pp2, *ps1, *ps2;
> +

Bonus points if you resubmit this with more expressive variable names. Just because xdiff's code is almost impossible to read doesn't mean we should follow suit :)

> xprepare.c:183-193
> +	pp1 = msmall.ptr, pp2 = mlarge.ptr;
> +	for (i = 0; i < msmall.size && *pp1 == *pp2; ++i) {
> +		plines += (*pp1 == '\n');
> +		pp1++, pp2++;
> +	}
> +
> +	ps1 = msmall.ptr + msmall.size - 1, ps2 = mlarge.ptr + mlarge.size - 1;

I'm still showing this as a hot point in the code when compiling with default settings used by Python packaging tools. I suspect we can get better results on typical compiler flags by tweaking things a bit. But we can do that after this lands.

> xprepare.c:199-202
> +		for (i = 0; i <= reserved;) {
> +			pp1--;
> +			i += (*pp1 == '\n');
> +		}

This is clever. But `memrchr()` will be easier to read. Plus I suspect it will be faster.

If you disagree, let's compromise at:

  i = 0;
  while (i <= reserved) {
     pp1--;
     i += (*pp1 == '\n');
  }

There's no sense using a `for` without the 3rd parameter IMO.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D2686

To: quark, #hg-reviewers, indygreg
Cc: indygreg, mercurial-devel
phabricator - March 9, 2018, 8:40 p.m.
quark added inline comments.

INLINE COMMENTS

> indygreg wrote in xprepare.c:169
> Bonus points if you resubmit this with more expressive variable names. Just because xdiff's code is almost impossible to read doesn't mean we should follow suit :)

The style guide in git community recommends using whatever style around the existing code base. I think we actually also do that, since new methods are not using `foo_bar` naming.

I'll add comments instead.

> indygreg wrote in xprepare.c:183-193
> I'm still showing this as a hot point in the code when compiling with default settings used by Python packaging tools. I suspect we can get better results on typical compiler flags by tweaking things a bit. But we can do that after this lands.

Yes. It's expected.

I did try various ways to optimize it before sending the patch, including:

- Like `memchr`, test 8 bytes at once. Difficulty: memory alignment is not guaranteed (ex. `msmall.ptr % 8 != mlarge.ptr % 8`).
- Use various SIMD related compiler flags.

The first makes things slower, even if I did tell the compiler "pretend the memory to be aligned". The second makes no difference.

> indygreg wrote in xprepare.c:199-202
> This is clever. But `memrchr()` will be easier to read. Plus I suspect it will be faster.
> 
> If you disagree, let's compromise at:
> 
>   i = 0;
>   while (i <= reserved) {
>      pp1--;
>      i += (*pp1 == '\n');
>   }
> 
> There's no sense using a `for` without the 3rd parameter IMO.

I think readability of the current code is better, since the memrchr version needs a "size" parameter, which is a burden to the existing logic.

I did some research before sending this patch. The glibc memchr is basically relying on `maybe_contain_zero_byte` that can test 8 bytes at once. But CPU SIMD instructions are faster than that trick.

The following code counts "\n"s in a file, using 3 ways: naive loop, testing 8 bytes at once, and actually using memchr. See the benchmark at the end.

  #include <fcntl.h>
  #include <stdint.h>
  #include <stdio.h>
  #include <string.h>
  #include <sys/stat.h>
  #include <sys/types.h>
  #include <unistd.h>
  
  char buf[64000000] __attribute__ ((aligned (16)));
  int size;
  
  static int count_naive() {
    int count = 0, i = 0;
    for (int i = 0; i < size; ++i) {
      count += buf[i] == '\n';
    }
    return count;
  }
  
  static int count_memchr() {
    int count = 0, i = 0;
    const char *p = buf;
    while (p) {
      p = memchr(p + 1, '\n', buf + size - p);
      count++;
    }
    return count;
  }
  
  static inline int maybe_contain_zero_byte(uint64_t x) {
    // See https://github.com/lattera/glibc/blob/master/string/memchr.c
    const uint64_t MAGIC_BITS = 0x7efefefefefefeff;
    return ((((x + MAGIC_BITS) ^ ~x) & ~MAGIC_BITS) != 0);
  }
  
  static int count_u64() {
    uint64_t *p = (uint64_t *)&buf;
    uint64_t x = '\n' + ('\n' << 8);
    int count = 0;
    x |= x << 16;
    x |= x << 32;
    for (int i = 0; i < size / 8; ++i, ++p) {
      uint64_t v = *p ^ x;
      if (maybe_contain_zero_byte(v)) {
        const char *c = (const char *) p;
        for (int j = 0; j < 8; ++j) {
          count += (((v >> (8 * j)) & 0xff) == 0);
        }
      }
    }
    return count;
  }
  
  int main(int argc, char const *argv[]) {
    int fd = open(argv[1], O_RDONLY);
    size = (int) read(fd, buf, sizeof buf);
    if (argv[2] && argv[2][0] == 'n') {
      printf("naive:  %d\n", count_naive());
    } else if (argv[2] && argv[2][0] == 'm') {
      printf("memchr: %d\n", count_memchr());
    } else {
      printf("u64:    %d\n", count_u64());
    }
    return 0;
  }
  
  /*
  # gcc 7.3.0
  gcc -O2 a.c -o ao2
  gcc -O3 -mavx2 a.c -o ao3
  
  # best of 50 runs, wall time
  # test case: random data
  # head -c 64000000 /dev/urandom > /tmp/r 
  ./ao2 naive  0.069
  ./ao2 u64    0.043
  ./ao2 memchr 0.039
  ./ao3 naive  0.038  # best
  ./ao3 u64    0.043
  ./ao3 memchr 0.039
  
  # test case: real code
  # v=read('/home/quark/hg-committed/mercurial/commands.py')
  # write('/tmp/c', v * (64000000/len(v)))
  ./ao2 naive  0.069
  ./ao2 u64    0.059
  ./ao2 memchr 0.055
  ./ao3 naive  0.038  # best
  ./ao3 u64    0.055
  ./ao3 memchr 0.055  # slower
  
  # ruby script to run the tests
  path = ARGV[0]
  %w[./ao2 ./ao3].product(%w[naive u64 memchr]).each do |exe, name|
    time = 50.times.map do
      t1 = Time.now
      system exe, path, name, 1=>'/dev/null'
      Time.now - t1
    end.min
    puts "#{exe} #{name.ljust(6)} #{time.round(3)}"
  end
  */

So I'd like to keep it simple and avoid over optimization. After all, this is O(100)-ish, assuming line length won't be ridiculously long. Even memchr is faster by 14%, it won't be noticeable. Not to say it's 31% slower in the -O3 case.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D2686

To: quark, #hg-reviewers, indygreg
Cc: indygreg, mercurial-devel
phabricator - March 9, 2018, 11:16 p.m.
indygreg accepted this revision.
indygreg added a comment.
This revision is now accepted and ready to land.


  I almost accepted the last version and this one is mostly cosmetic changes. So LGTM!
  
  Your work here is very much appreciated. Thank you for doing the thorough performance analysis.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D2686

To: quark, #hg-reviewers, indygreg
Cc: indygreg, mercurial-devel

Patch

diff --git a/mercurial/thirdparty/xdiff/xtypes.h b/mercurial/thirdparty/xdiff/xtypes.h
--- a/mercurial/thirdparty/xdiff/xtypes.h
+++ b/mercurial/thirdparty/xdiff/xtypes.h
@@ -94,6 +94,7 @@ 
 
 typedef struct s_xdfenv {
 	xdfile_t xdf1, xdf2;
+	long prefix_lines, suffix_lines;
 } xdfenv_t;
 
 
diff --git a/mercurial/thirdparty/xdiff/xprepare.h b/mercurial/thirdparty/xdiff/xprepare.h
--- a/mercurial/thirdparty/xdiff/xprepare.h
+++ b/mercurial/thirdparty/xdiff/xprepare.h
@@ -26,7 +26,7 @@ 
 
 
 int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
-		    xdfenv_t *xe);
+		    xdfenv_t *xe, xdemitconf_t const *xecfg);
 void xdl_free_env(xdfenv_t *xe);
 
 
diff --git a/mercurial/thirdparty/xdiff/xprepare.c b/mercurial/thirdparty/xdiff/xprepare.c
--- a/mercurial/thirdparty/xdiff/xprepare.c
+++ b/mercurial/thirdparty/xdiff/xprepare.c
@@ -61,6 +61,8 @@ 
 static void xdl_free_ctx(xdfile_t *xdf);
 static int xdl_clean_mmatch(char const *dis, long i, long s, long e);
 static int xdl_cleanup_records(xdlclassifier_t *cf, xdfile_t *xdf1, xdfile_t *xdf2);
+static void xdl_trim_files(mmfile_t *mf1, mmfile_t *mf2, long reserved,
+		xdfenv_t* xe, mmfile_t *out_mf1, mmfile_t *out_mf2);
 static int xdl_trim_ends(xdfile_t *xdf1, xdfile_t *xdf2);
 static int xdl_optimize_ctxs(xdlclassifier_t *cf, xdfile_t *xdf1, xdfile_t *xdf2);
 
@@ -156,6 +158,105 @@ 
 }
 
 
+/*
+ * Trim common prefix from files.
+ * Note: need to preserve some lines for shifting to work. For example,
+ *
+ *     a.py    | common | b.py    | common | diff
+ *             | p | s  |         | p | s  |
+ *     -------------------------------------------
+ *     try:    | Y      | try:    | Y      |
+ *       1     | Y      |   1     | Y      |
+ *     except: | Y  y   | except: | Y      |
+ *       pass  | Y  y   |   pass  | Y      |
+ *     try:    | Y  y   | try:    | Y      | +
+ *       3     |    Y   |   2     |        | +
+ *     except: |    Y   | except: |     y  | +
+ *       pass  |    Y   |   pass  |     y  | +
+ *             |        | try:    |     y  |
+ *             |        |   3     |     Y  |
+ *             |        | except: |     Y  |
+ *             |        |   pass  |     Y  |
+ *
+ * To make shifting work, common suffix is handled so it cannot overlap with
+ * common prefix (in the above case, lines marked with "y" will not count as
+ * common suffix). Then subtract some O(1) lines leaving room for shifting.
+ *
+ * Find common prefix first, since the diff algorithm moves hunks towards the
+ * end. For example,
+ *
+ *     #!python
+ *     open('a','w').write('x\n'* 1000)
+ *     open('b','w').write('x\n'* 1001)
+ *
+ * Diffing the two files in either direction, will print a hunk at the end.
+ */
+static void xdl_trim_files(mmfile_t *mf1, mmfile_t *mf2, long reserved,
+		xdfenv_t *xe, mmfile_t *out_mf1, mmfile_t *out_mf2) {
+	mmfile_t msmall, mlarge;
+	long plines = 0, pbytes = 0, slines = 0, sbytes = 0, i;
+	const char *pp1, *pp2, *ps1, *ps2;
+
+	/* reserved must >= 0 for the line boundary adjustment to work */
+	if (reserved < 0)
+		reserved = 0;
+
+	if (mf1->size < mf2->size) {
+		memcpy(&msmall, mf1, sizeof(mmfile_t));
+		memcpy(&mlarge, mf2, sizeof(mmfile_t));
+	} else {
+		memcpy(&msmall, mf2, sizeof(mmfile_t));
+		memcpy(&mlarge, mf1, sizeof(mmfile_t));
+	}
+
+	pp1 = msmall.ptr, pp2 = mlarge.ptr;
+	for (i = 0; i < msmall.size && *pp1 == *pp2; ++i) {
+		plines += (*pp1 == '\n');
+		pp1++, pp2++;
+	}
+
+	ps1 = msmall.ptr + msmall.size - 1, ps2 = mlarge.ptr + mlarge.size - 1;
+	for (; ps1 > pp1 && *ps1 == *ps2; ++i) {
+		slines += (*ps1 == '\n');
+		ps1--, ps2--;
+	}
+
+	/* Retract common prefix and suffix boundaries for reserved lines */
+	if (plines <= reserved + 1) {
+		plines = 0;
+	} else {
+		for (i = 0; i <= reserved;) {
+			pp1--;
+			i += (*pp1 == '\n');
+		}
+		/* The new mmfile starts at the next char just after '\n' */
+		pbytes = pp1 - msmall.ptr + 1;
+		plines -= reserved;
+	}
+
+	if (slines <= reserved + 1) {
+		slines = 0;
+	} else {
+		for (i = 0; i <= reserved;) {
+			ps1++;
+			i += (*ps1 == '\n');
+		}
+		/* The new mmfile includes this '\n' */
+		sbytes = msmall.ptr + msmall.size - ps1 - 1;
+		slines -= reserved;
+		if (msmall.ptr[msmall.size - 1] == '\n')
+			slines -= 1;
+	}
+
+	xe->prefix_lines = plines;
+	xe->suffix_lines = slines;
+	out_mf1->ptr = mf1->ptr + pbytes;
+	out_mf1->size = mf1->size - pbytes - sbytes;
+	out_mf2->ptr = mf2->ptr + pbytes;
+	out_mf2->size = mf2->size - pbytes - sbytes;
+}
+
+
 static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_t const *xpp,
 			   xdlclassifier_t *cf, xdfile_t *xdf) {
 	unsigned int hbits;
@@ -254,10 +355,14 @@ 
 	xdl_cha_free(&xdf->rcha);
 }
 
+/* Minimal reserved lines during file trimming. This is to leave room for
+ * shifting */
+#define MIN_RESERVED_LINES 100
 
 int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
-		    xdfenv_t *xe) {
-	long enl1, enl2, sample;
+		    xdfenv_t *xe, xdemitconf_t const *xecfg) {
+	mmfile_t tmf1, tmf2;
+	long enl1, enl2, sample, reserved_lines = MIN_RESERVED_LINES;
 	xdlclassifier_t cf;
 
 	memset(&cf, 0, sizeof(cf));
@@ -270,12 +375,16 @@ 
 	if (xdl_init_classifier(&cf, enl1 + enl2 + 1, xpp->flags) < 0)
 		return -1;
 
-	if (xdl_prepare_ctx(1, mf1, enl1, xpp, &cf, &xe->xdf1) < 0) {
+	if (xecfg)
+		reserved_lines += xecfg->ctxlen;
+	xdl_trim_files(mf1, mf2, reserved_lines, xe, &tmf1, &tmf2);
+
+	if (xdl_prepare_ctx(1, &tmf1, enl1, xpp, &cf, &xe->xdf1) < 0) {
 
 		xdl_free_classifier(&cf);
 		return -1;
 	}
-	if (xdl_prepare_ctx(2, mf2, enl2, xpp, &cf, &xe->xdf2) < 0) {
+	if (xdl_prepare_ctx(2, &tmf2, enl2, xpp, &cf, &xe->xdf2) < 0) {
 
 		xdl_free_ctx(&xe->xdf1);
 		xdl_free_classifier(&cf);
diff --git a/mercurial/thirdparty/xdiff/xemit.c b/mercurial/thirdparty/xdiff/xemit.c
--- a/mercurial/thirdparty/xdiff/xemit.c
+++ b/mercurial/thirdparty/xdiff/xemit.c
@@ -169,6 +169,7 @@ 
 	long s1, s2, e1, e2, lctx;
 	xdchange_t *xch, *xche;
 	long funclineprev = -1;
+	long p = xe->prefix_lines;
 	struct func_line func_line = { 0 };
 
 	for (xch = xscr; xch; xch = xche->next) {
@@ -261,7 +262,7 @@ 
 				      s1 - 1, funclineprev);
 			funclineprev = s1 - 1;
 		}
-		if (xdl_emit_hunk_hdr(s1 + 1, e1 - s1, s2 + 1, e2 - s2,
+		if (xdl_emit_hunk_hdr(s1 + 1 + p, e1 - s1, s2 + 1 + p, e2 - s2,
 				      func_line.buf, func_line.len, ecb) < 0)
 			return -1;
 
diff --git a/mercurial/thirdparty/xdiff/xdiffi.c b/mercurial/thirdparty/xdiff/xdiffi.c
--- a/mercurial/thirdparty/xdiff/xdiffi.c
+++ b/mercurial/thirdparty/xdiff/xdiffi.c
@@ -321,14 +321,14 @@ 
 }
 
 
-int xdl_do_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
-		xdfenv_t *xe) {
+int xdl_do_diff2(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
+		 xdfenv_t *xe, xdemitconf_t const *xecfg) {
 	long ndiags;
 	long *kvd, *kvdf, *kvdb;
 	xdalgoenv_t xenv;
 	diffdata_t dd1, dd2;
 
-	if (xdl_prepare_env(mf1, mf2, xpp, xe) < 0) {
+	if (xdl_prepare_env(mf1, mf2, xpp, xe, xecfg) < 0) {
 
 		return -1;
 	}
@@ -377,6 +377,12 @@ 
 }
 
 
+int xdl_do_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
+		xdfenv_t *xe) {
+	return xdl_do_diff2(mf1, mf2, xpp, xe, NULL);
+}
+
+
 static xdchange_t *xdl_add_change(xdchange_t *xscr, long i1, long i2, long chg1, long chg2) {
 	xdchange_t *xch;
 
@@ -1006,30 +1012,38 @@ 
 static int xdl_call_hunk_func(xdfenv_t *xe, xdchange_t *xscr, xdemitcb_t *ecb,
 			      xdemitconf_t const *xecfg)
 {
+	long p = xe->prefix_lines, s = xe->suffix_lines;
 	xdchange_t *xch, *xche;
 	if ((xecfg->flags & XDL_EMIT_BDIFFHUNK) != 0) {
 		long i1 = 0, i2 = 0, n1 = xe->xdf1.nrec, n2 = xe->xdf2.nrec;
 		for (xch = xscr; xch; xch = xche->next) {
 			xche = xdl_get_hunk(&xch, xecfg);
 			if (!xch)
 				break;
+			if (xch != xche)
+				xdl_bug("xch != xche");
+			xch->i1 += p;
+			xch->i2 += p;
 			if (xch->i1 > i1 || xch->i2 > i2) {
-				if (xecfg->hunk_func(i1, xch->i1, i2, xch->i2, ecb->priv) < 0)
+				if (xecfg->hunk_func(i1, xch->i1, i2, xch->i2,
+						     ecb->priv) < 0)
 					return -1;
 			}
-			i1 = xche->i1 + xche->chg1;
-			i2 = xche->i2 + xche->chg2;
+			i1 = xch->i1 + xch->chg1;
+			i2 = xch->i2 + xch->chg2;
 		}
-		if (xecfg->hunk_func(i1, n1, i2, n2, ecb->priv) < 0)
+		if (xecfg->hunk_func(i1, n1 + p + s, i2, n2 + p + s,
+				     ecb->priv) < 0)
 			return -1;
 	} else {
 		for (xch = xscr; xch; xch = xche->next) {
 			xche = xdl_get_hunk(&xch, xecfg);
 			if (!xch)
 				break;
-			if (xecfg->hunk_func(
-					xch->i1, xche->i1 + xche->chg1 - xch->i1,
-					xch->i2, xche->i2 + xche->chg2 - xch->i2,
+			if (xecfg->hunk_func(xch->i1 + p,
+					xche->i1 + xche->chg1 - xch->i1 + p,
+					xch->i2 + p,
+					xche->i2 + xche->chg2 - xch->i2 + p,
 					ecb->priv) < 0)
 				return -1;
 		}
@@ -1043,7 +1057,7 @@ 
 	xdfenv_t xe;
 	emit_func_t ef = xecfg->hunk_func ? xdl_call_hunk_func : xdl_emit_diff;
 
-	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0) {
+	if (xdl_do_diff2(mf1, mf2, xpp, &xe, xecfg) < 0) {
 
 		return -1;
 	}