Patchwork [V5] run-tests: added '--json' functionality to store test result data in json file

login
register
mail settings
Submitter Anurag Goel
Date Aug. 20, 2014, 5:11 p.m.
Message ID <bd5b8d1a4aebdbd7e249.1408554702@ubuntu.ubuntu-domain>
Download mbox | patch
Permalink /patch/5532/
State Changes Requested
Headers show

Comments

Anurag Goel - Aug. 20, 2014, 5:11 p.m.
# HG changeset patch
# User anuraggoel <anurag.dsps@gmail.com>
# Date 1408554532 -19800
#      Wed Aug 20 22:38:52 2014 +0530
# Node ID bd5b8d1a4aebdbd7e249a003fbc1b5b3c2cd6515
# Parent  de783f2403c498ef1e20121acf178b32ec27199c
run-tests: added '--json' functionality to store test result data in json file

This patch added a new functionality '--json'. While testing, if '--json'
is enabled then test result data gets stored in newly created "report.json"
file in the following format.

testreport ={
    "test-success.t": {
        "result": "success",
        "time": "2.041"
    }
    "test-failure.t": {
        "result": "failure",
        "time": "4.430"
    }
    "test-skip.t": {
        "result": "skip"
        "time": "3.754"
    }
}

This "report.json" file will further accessed by html/javascript file for
graph usage.
Pierre-Yves David - Aug. 21, 2014, 6:25 a.m.
On 08/20/2014 10:11 AM, Anurag Goel wrote:
> # HG changeset patch
> # User anuraggoel <anurag.dsps@gmail.com>
> # Date 1408554532 -19800
> #      Wed Aug 20 22:38:52 2014 +0530
> # Node ID bd5b8d1a4aebdbd7e249a003fbc1b5b3c2cd6515
> # Parent  de783f2403c498ef1e20121acf178b32ec27199c
> run-tests: added '--json' functionality to store test result data in json file
>
> This patch added a new functionality '--json'. While testing, if '--json'
> is enabled then test result data gets stored in newly created "report.json"
> file in the following format.
>
> testreport ={
>      "test-success.t": {
>          "result": "success",
>          "time": "2.041"
>      }
>      "test-failure.t": {
>          "result": "failure",
>          "time": "4.430"
>      }
>      "test-skip.t": {
>          "result": "skip"
>          "time": "3.754"
>      }
> }
>
> This "report.json" file will further accessed by html/javascript file for
> graph usage.

On my machine the patch produce:


 > python run-tests.py --json test-run-tests.t
Traceback (most recent call last):
   File "run-tests.py", line 61, in <module>
     import simplejson as json
ImportError: No module named simplejson
zsh: exit 1     python run-tests.py --json test-run-tests.t
 > python --version
Python 2.7.8

You need to handle this with exception catching. Also made sure 
run-tests.py keep working on machine with no json install at all (old 
python 2.4 for example)


> diff -r de783f2403c4 -r bd5b8d1a4aeb tests/run-tests.py
> --- a/tests/run-tests.py	Sat Aug 16 10:43:59 2014 +0900
> +++ b/tests/run-tests.py	Wed Aug 20 22:38:52 2014 +0530
> @@ -58,6 +58,7 @@
>   import killdaemons as killmod
>   import Queue as queue
>   from xml.dom import minidom
> +import simplejson as json
>   import unittest
>
>   processlock = threading.Lock()
> @@ -184,6 +185,8 @@
>       parser.add_option("-t", "--timeout", type="int",
>           help="kill errant tests after TIMEOUT seconds"
>                " (default: $%s or %d)" % defaults['timeout'])
> +    parser.add_option("--json", action="store_true",
> +        help="store test result data in 'report.json' file")
>       parser.add_option("--time", action="store_true",
>           help="time how long each test takes")
>       parser.add_option("--tmpdir", type="string",
> @@ -1419,6 +1422,30 @@
>               finally:
>                   xuf.close()
>
> +        if self._runner.options.json:
> +            fp = open(os.path.join(self._runner._testdir, 'report.json'), 'w')
> +            try:
> +                outcome = []
> +                timesd = dict(
> +                    (test, real) for test, cuser, csys, real in result.times)

This "on liner" is fairly hard to read (partly because it is not a one 
liner anymore)

You should use a simple for loop on mulitple lines.

> +                for tc in result.successes:
> +                    testresult = {'result': 'success',
> +                        'time': ('%0.3f' % timesd[tc.name])}

Bad indentation. The mercurial project use:

    testresult = {'result': 'success',
                  'time': ('%0.3f' % timesd[tc.name])}

(this apply to the other loop too



> +                    outcome.append((tc.name, testresult))
> +                for tc, err in sorted(result.faildata.iteritems()):
> +                    testresult = {'result': 'failure',
> +                        'time': ('%0.3f' % timesd[tc])}
> +                    outcome.append((tc, testresult))
> +                for tc, reason in result.skipped:
> +                    testresult = {'result': 'skip',
> +                        'time': ('%0.3f' % timesd[tc.name])}
> +                    outcome.append((tc.name, testresult))
> +                testdata = dict(outcome)
> +                fp.writelines(("testreport =", (json.dumps(testdata,
> +                                    sort_keys=True, indent=4))))

bad ident again

> +            finally:
> +                fp.close()
> +
>           self._runner._checkhglib('Tested')
>
>           self.stream.writeln('# Ran %d tests, %d skipped, %d warned, %d failed.'
> diff -r de783f2403c4 -r bd5b8d1a4aeb tests/test-run-tests.t
> --- a/tests/test-run-tests.t	Sat Aug 16 10:43:59 2014 +0900
> +++ b/tests/test-run-tests.t	Wed Aug 20 22:38:52 2014 +0530
> @@ -357,3 +357,40 @@
>     Skipped test-failure.t: blacklisted
>     # Ran 0 tests, 2 skipped, 0 warned, 0 failed.
>
> +test for --json
> +==================
> +
> +  $ $TESTDIR/run-tests.py --with-hg=`which hg` --json
> +
> +  --- $TESTTMP/test-failure.t
> +  +++ $TESTTMP/test-failure.t.err
> +  @@ -1,4 +1,4 @@
> +     $ echo babar
> +  -  rataxes
> +  +  babar
> +   This is a noop statement so that
> +   this test is still more bytes than success.
> +
> +  ERROR: test-failure.t output changed
> +  !.s
> +  Skipped test-skip.t: irrelevant
> +  Failed test-failure.t: output changed
> +  # Ran 2 tests, 1 skipped, 0 warned, 1 failed.
> +  python hash seed: * (glob)
> +  [1]
> +
> +  $ cat report.json
> +  testreport ={
> +      "test-failure.t": [\{] (re)
> +          "result": "failure",
> +          "time": "\s*[\d\.]{5}" (re)
> +      },
> +      "test-skip.t": {
> +          "result": "skip",
> +          "time": "\s*[\d\.]{5}" (re)
> +      },
> +      "test-success.t": [\{] (re)
> +          "result": "success",
> +          "time": "\s*[\d\.]{5}" (re)
> +      }
> +  } (no-eol)
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel@selenic.com
> http://selenic.com/mailman/listinfo/mercurial-devel
>

Patch

diff -r de783f2403c4 -r bd5b8d1a4aeb tests/run-tests.py
--- a/tests/run-tests.py	Sat Aug 16 10:43:59 2014 +0900
+++ b/tests/run-tests.py	Wed Aug 20 22:38:52 2014 +0530
@@ -58,6 +58,7 @@ 
 import killdaemons as killmod
 import Queue as queue
 from xml.dom import minidom
+import simplejson as json
 import unittest
 
 processlock = threading.Lock()
@@ -184,6 +185,8 @@ 
     parser.add_option("-t", "--timeout", type="int",
         help="kill errant tests after TIMEOUT seconds"
              " (default: $%s or %d)" % defaults['timeout'])
+    parser.add_option("--json", action="store_true",
+        help="store test result data in 'report.json' file")
     parser.add_option("--time", action="store_true",
         help="time how long each test takes")
     parser.add_option("--tmpdir", type="string",
@@ -1419,6 +1422,30 @@ 
             finally:
                 xuf.close()
 
+        if self._runner.options.json:
+            fp = open(os.path.join(self._runner._testdir, 'report.json'), 'w')
+            try:
+                outcome = []
+                timesd = dict(
+                    (test, real) for test, cuser, csys, real in result.times)
+                for tc in result.successes:
+                    testresult = {'result': 'success',
+                        'time': ('%0.3f' % timesd[tc.name])}
+                    outcome.append((tc.name, testresult))
+                for tc, err in sorted(result.faildata.iteritems()):
+                    testresult = {'result': 'failure',
+                        'time': ('%0.3f' % timesd[tc])}
+                    outcome.append((tc, testresult))
+                for tc, reason in result.skipped:
+                    testresult = {'result': 'skip',
+                        'time': ('%0.3f' % timesd[tc.name])}
+                    outcome.append((tc.name, testresult))
+                testdata = dict(outcome)
+                fp.writelines(("testreport =", (json.dumps(testdata,
+                                    sort_keys=True, indent=4))))
+            finally:
+                fp.close()
+
         self._runner._checkhglib('Tested')
 
         self.stream.writeln('# Ran %d tests, %d skipped, %d warned, %d failed.'
diff -r de783f2403c4 -r bd5b8d1a4aeb tests/test-run-tests.t
--- a/tests/test-run-tests.t	Sat Aug 16 10:43:59 2014 +0900
+++ b/tests/test-run-tests.t	Wed Aug 20 22:38:52 2014 +0530
@@ -357,3 +357,40 @@ 
   Skipped test-failure.t: blacklisted
   # Ran 0 tests, 2 skipped, 0 warned, 0 failed.
 
+test for --json
+==================
+
+  $ $TESTDIR/run-tests.py --with-hg=`which hg` --json
+  
+  --- $TESTTMP/test-failure.t
+  +++ $TESTTMP/test-failure.t.err
+  @@ -1,4 +1,4 @@
+     $ echo babar
+  -  rataxes
+  +  babar
+   This is a noop statement so that
+   this test is still more bytes than success.
+  
+  ERROR: test-failure.t output changed
+  !.s
+  Skipped test-skip.t: irrelevant
+  Failed test-failure.t: output changed
+  # Ran 2 tests, 1 skipped, 0 warned, 1 failed.
+  python hash seed: * (glob)
+  [1]
+
+  $ cat report.json
+  testreport ={
+      "test-failure.t": [\{] (re)
+          "result": "failure",
+          "time": "\s*[\d\.]{5}" (re)
+      },
+      "test-skip.t": {
+          "result": "skip",
+          "time": "\s*[\d\.]{5}" (re)
+      },
+      "test-success.t": [\{] (re)
+          "result": "success",
+          "time": "\s*[\d\.]{5}" (re)
+      }
+  } (no-eol)