Patchwork [V2] runtests: add 'diff' entry in json report

login
register
mail settings
Submitter Laurent Charignon
Date Jan. 5, 2016, 9:33 p.m.
Message ID <77ef17fd99a8ece6f2d5.1452029603@lcharignon-mbp.local>
Download mbox | patch
Permalink /patch/12544/
State Accepted
Headers show

Comments

Laurent Charignon - Jan. 5, 2016, 9:33 p.m.
# HG changeset patch
# User Laurent Charignon <lcharignon@fb.com>
# Date 1452029589 28800
#      Tue Jan 05 13:33:09 2016 -0800
# Node ID 77ef17fd99a8ece6f2d54648db2da0562136a5d7
# Parent  b8405d739149cdd6d8d9bd5e3dd2ad8487b1f09a
runtests: add 'diff' entry in json report

This patch adds to the json report the "diff" between expected and observed
result. This diff can be useful for automatically filing bug report on failing
tests.
Yuya Nishihara - Jan. 9, 2016, 3:07 a.m.
On Tue, 5 Jan 2016 13:33:23 -0800, Laurent Charignon wrote:
> # HG changeset patch
> # User Laurent Charignon <lcharignon@fb.com>
> # Date 1452029589 28800
> #      Tue Jan 05 13:33:09 2016 -0800
> # Node ID 77ef17fd99a8ece6f2d54648db2da0562136a5d7
> # Parent  b8405d739149cdd6d8d9bd5e3dd2ad8487b1f09a
> runtests: add 'diff' entry in json report
> 
> This patch adds to the json report the "diff" between expected and observed
> result. This diff can be useful for automatically filing bug report on failing
> tests.

Sounds good, pushed to the clowncopter, thanks.

>        "test-skip.t": {
>            "csys": "\s*[\d\.]{4,5}", ? (re)
>            "cuser": "\s*[\d\.]{4,5}", ? (re)
> +          "diff": "", 
>            "end": "\s*[\d\.]{4,5}", ? (re)
>            "result": "skip", ? (re)
>            "start": "\s*[\d\.]{4,5}", ? (re)

I've added " ? (re)" to all lines because it looks like necessary to work
around unstable JSON formatting.

Patch

diff --git a/tests/run-tests.py b/tests/run-tests.py
--- a/tests/run-tests.py
+++ b/tests/run-tests.py
@@ -1688,7 +1688,9 @@  class TextTestRunner(unittest.TextTestRu
                                     'cuser': ('%0.3f' % timesd[tc.name][0]),
                                     'csys': ('%0.3f' % timesd[tc.name][1]),
                                     'start': ('%0.3f' % timesd[tc.name][3]),
-                                    'end': ('%0.3f' % timesd[tc.name][4])}
+                                    'end': ('%0.3f' % timesd[tc.name][4]),
+                                    'diff': result.faildata.get(tc.name, ''),
+                                   }
                             outcome[tc.name] = tres
                     jsonout = json.dumps(outcome, sort_keys=True, indent=4)
                     fp.writelines(("testreport =", jsonout))
diff --git a/tests/test-run-tests.t b/tests/test-run-tests.t
--- a/tests/test-run-tests.t
+++ b/tests/test-run-tests.t
@@ -530,6 +530,7 @@  test for --json
       "test-failure.t": [\{] (re)
           "csys": "\s*[\d\.]{4,5}", ? (re)
           "cuser": "\s*[\d\.]{4,5}", ? (re)
+          "diff": "---.+\+\+\+.+", ? (re)
           "end": "\s*[\d\.]{4,5}", ? (re)
           "result": "failure", ? (re)
           "start": "\s*[\d\.]{4,5}", ? (re)
@@ -538,6 +539,7 @@  test for --json
       "test-skip.t": {
           "csys": "\s*[\d\.]{4,5}", ? (re)
           "cuser": "\s*[\d\.]{4,5}", ? (re)
+          "diff": "", 
           "end": "\s*[\d\.]{4,5}", ? (re)
           "result": "skip", ? (re)
           "start": "\s*[\d\.]{4,5}", ? (re)
@@ -546,6 +548,7 @@  test for --json
       "test-success.t": [\{] (re)
           "csys": "\s*[\d\.]{4,5}", ? (re)
           "cuser": "\s*[\d\.]{4,5}", ? (re)
+          "diff": "", 
           "end": "\s*[\d\.]{4,5}", ? (re)
           "result": "success", ? (re)
           "start": "\s*[\d\.]{4,5}", ? (re)
@@ -575,6 +578,7 @@  Test that failed test accepted through i
       "test-failure.t": [\{] (re)
           "csys": "\s*[\d\.]{4,5}", ? (re)
           "cuser": "\s*[\d\.]{4,5}", ? (re)
+          "diff": "", 
           "end": "\s*[\d\.]{4,5}", ? (re)
           "result": "success", ? (re)
           "start": "\s*[\d\.]{4,5}", ? (re)
@@ -583,6 +587,7 @@  Test that failed test accepted through i
       "test-skip.t": {
           "csys": "\s*[\d\.]{4,5}", ? (re)
           "cuser": "\s*[\d\.]{4,5}", ? (re)
+          "diff": "", 
           "end": "\s*[\d\.]{4,5}", ? (re)
           "result": "skip", ? (re)
           "start": "\s*[\d\.]{4,5}", ? (re)
@@ -591,6 +596,7 @@  Test that failed test accepted through i
       "test-success.t": [\{] (re)
           "csys": "\s*[\d\.]{4,5}", ? (re)
           "cuser": "\s*[\d\.]{4,5}", ? (re)
+          "diff": "", 
           "end": "\s*[\d\.]{4,5}", ? (re)
           "result": "success", ? (re)
           "start": "\s*[\d\.]{4,5}", ? (re)