Patchwork [V5] run-tests: added '--json' functionality to store test result data in json file

login
register
mail settings
Submitter Anurag Goel
Date Aug. 21, 2014, 10:06 p.m.
Message ID <CAMq1dJJ67Rfsu9j+uNOJSTvzqQ9a3O4NpgeM37L+RAms+hMuqA@mail.gmail.com>
Download mbox | patch
Permalink /patch/5548/
State Not Applicable
Headers show

Comments

Anurag Goel - Aug. 21, 2014, 10:06 p.m.
hi, i have made the changes as you suggested so. Please have a look if
there are any further changes require.



On Thu, Aug 21, 2014 at 11:55 AM, Pierre-Yves David <
pierre-yves.david@ens-lyon.org> wrote:

>
>
> On 08/20/2014 10:11 AM, Anurag Goel wrote:
>
>> # HG changeset patch
>> # User anuraggoel <anurag.dsps@gmail.com>
>> # Date 1408554532 -19800
>> #      Wed Aug 20 22:38:52 2014 +0530
>> # Node ID bd5b8d1a4aebdbd7e249a003fbc1b5b3c2cd6515
>> # Parent  de783f2403c498ef1e20121acf178b32ec27199c
>> run-tests: added '--json' functionality to store test result data in json
>> file
>>
>> This patch added a new functionality '--json'. While testing, if '--json'
>> is enabled then test result data gets stored in newly created
>> "report.json"
>> file in the following format.
>>
>> testreport ={
>>      "test-success.t": {
>>          "result": "success",
>>          "time": "2.041"
>>      }
>>      "test-failure.t": {
>>          "result": "failure",
>>          "time": "4.430"
>>      }
>>      "test-skip.t": {
>>          "result": "skip"
>>          "time": "3.754"
>>      }
>> }
>>
>> This "report.json" file will further accessed by html/javascript file for
>> graph usage.
>>
>
> On my machine the patch produce:
>
>
> > python run-tests.py --json test-run-tests.t
> Traceback (most recent call last):
>   File "run-tests.py", line 61, in <module>
>     import simplejson as json
> ImportError: No module named simplejson
> zsh: exit 1     python run-tests.py --json test-run-tests.t
> > python --version
> Python 2.7.8
>
> You need to handle this with exception catching. Also made sure
> run-tests.py keep working on machine with no json install at all (old
> python 2.4 for example)
>
>
>
>  diff -r de783f2403c4 -r bd5b8d1a4aeb tests/run-tests.py
>> --- a/tests/run-tests.py        Sat Aug 16 10:43:59 2014 +0900
>> +++ b/tests/run-tests.py        Wed Aug 20 22:38:52 2014 +0530
>> @@ -58,6 +58,7 @@
>>   import killdaemons as killmod
>>   import Queue as queue
>>   from xml.dom import minidom
>> +import simplejson as json
>>   import unittest
>>
>>   processlock = threading.Lock()
>> @@ -184,6 +185,8 @@
>>       parser.add_option("-t", "--timeout", type="int",
>>           help="kill errant tests after TIMEOUT seconds"
>>                " (default: $%s or %d)" % defaults['timeout'])
>> +    parser.add_option("--json", action="store_true",
>> +        help="store test result data in 'report.json' file")
>>       parser.add_option("--time", action="store_true",
>>           help="time how long each test takes")
>>       parser.add_option("--tmpdir", type="string",
>> @@ -1419,6 +1422,30 @@
>>               finally:
>>                   xuf.close()
>>
>> +        if self._runner.options.json:
>> +            fp = open(os.path.join(self._runner._testdir,
>> 'report.json'), 'w')
>> +            try:
>> +                outcome = []
>> +                timesd = dict(
>> +                    (test, real) for test, cuser, csys, real in
>> result.times)
>>
>
> This "on liner" is fairly hard to read (partly because it is not a one
> liner anymore)
>
> You should use a simple for loop on mulitple lines.
>
>
>  +                for tc in result.successes:
>> +                    testresult = {'result': 'success',
>> +                        'time': ('%0.3f' % timesd[tc.name])}
>>
>
> Bad indentation. The mercurial project use:
>
>
>    testresult = {'result': 'success',
>                  'time': ('%0.3f' % timesd[tc.name])}
>
> (this apply to the other loop too
>
>
>
>
>  +                    outcome.append((tc.name, testresult))
>> +                for tc, err in sorted(result.faildata.iteritems()):
>> +                    testresult = {'result': 'failure',
>> +                        'time': ('%0.3f' % timesd[tc])}
>> +                    outcome.append((tc, testresult))
>> +                for tc, reason in result.skipped:
>> +                    testresult = {'result': 'skip',
>> +                        'time': ('%0.3f' % timesd[tc.name])}
>> +                    outcome.append((tc.name, testresult))
>> +                testdata = dict(outcome)
>> +                fp.writelines(("testreport =", (json.dumps(testdata,
>> +                                    sort_keys=True, indent=4))))
>>
>
> bad ident again
>
>  +            finally:
>> +                fp.close()
>> +
>>           self._runner._checkhglib('Tested')
>>
>>           self.stream.writeln('# Ran %d tests, %d skipped, %d warned, %d
>> failed.'
>> diff -r de783f2403c4 -r bd5b8d1a4aeb tests/test-run-tests.t
>> --- a/tests/test-run-tests.t    Sat Aug 16 10:43:59 2014 +0900
>> +++ b/tests/test-run-tests.t    Wed Aug 20 22:38:52 2014 +0530
>> @@ -357,3 +357,40 @@
>>     Skipped test-failure.t: blacklisted
>>     # Ran 0 tests, 2 skipped, 0 warned, 0 failed.
>>
>> +test for --json
>> +==================
>> +
>> +  $ $TESTDIR/run-tests.py --with-hg=`which hg` --json
>> +
>> +  --- $TESTTMP/test-failure.t
>> +  +++ $TESTTMP/test-failure.t.err
>> +  @@ -1,4 +1,4 @@
>> +     $ echo babar
>> +  -  rataxes
>> +  +  babar
>> +   This is a noop statement so that
>> +   this test is still more bytes than success.
>> +
>> +  ERROR: test-failure.t output changed
>> +  !.s
>> +  Skipped test-skip.t: irrelevant
>> +  Failed test-failure.t: output changed
>> +  # Ran 2 tests, 1 skipped, 0 warned, 1 failed.
>> +  python hash seed: * (glob)
>> +  [1]
>> +
>> +  $ cat report.json
>> +  testreport ={
>> +      "test-failure.t": [\{] (re)
>> +          "result": "failure",
>> +          "time": "\s*[\d\.]{5}" (re)
>> +      },
>> +      "test-skip.t": {
>> +          "result": "skip",
>> +          "time": "\s*[\d\.]{5}" (re)
>> +      },
>> +      "test-success.t": [\{] (re)
>> +          "result": "success",
>> +          "time": "\s*[\d\.]{5}" (re)
>> +      }
>> +  } (no-eol)
>> _______________________________________________
>> Mercurial-devel mailing list
>> Mercurial-devel@selenic.com
>> http://selenic.com/mailman/listinfo/mercurial-devel
>>
>>
> --
> Pierre-Yves David
>
Pierre-Yves David - Aug. 22, 2014, 7:39 p.m.
On 08/21/2014 03:06 PM, Anurag Goel wrote:
> hi, i have made the changes as you suggested so. Please have a look if
> there are any further changes require.

You should patchbomb it again with a V6 flag

Patch

diff -r 91464337202e tests/run-tests.py
--- a/tests/run-tests.py    Thu Aug 21 16:23:24 2014 +0530
+++ b/tests/run-tests.py    Fri Aug 22 03:27:51 2014 +0530
@@ -184,6 +184,8 @@ 
     parser.add_option("-t", "--timeout", type="int",
         help="kill errant tests after TIMEOUT seconds"
              " (default: $%s or %d)" % defaults['timeout'])
+    parser.add_option("--json", action="store_true",
+        help="store test result data in 'report.json' file")
     parser.add_option("--time", action="store_true",
         help="time how long each test takes")
     parser.add_option("--tmpdir", type="string",
@@ -1419,6 +1421,40 @@ 
             finally:
                 xuf.close()

+        if self._runner.options.json:
+            try:
+                import simplejson as json
+
+                fp = open(os.path.join(self._runner._testdir,
+                          'report.json'), 'w')
+                try:
+                    timest = []
+                    for test, cuser, csys, real in result.times:
+                        timest.append((test, real))
+
+                    timesd = dict(timest)
+
+                    outcome = []
+                    for tc in result.successes:
+                        testresult = {'result': 'success',
+                                 'time': ('%0.3f' % timesd[tc.name])}
+                        outcome.append((tc.name, testresult))
+                    for tc, err in sorted(result.faildata.iteritems()):
+                        testresult = {'result': 'failure',
+                                 'time': ('%0.3f' % timesd[tc])}
+                        outcome.append((tc, testresult))
+                    for tc, reason in result.skipped:
+                        testresult = {'result': 'skip',
+                                 'time': ('%0.3f' % timesd[tc.name])}
+                        outcome.append((tc.name, testresult))
+                    testdata = dict(outcome)
+                    fp.writelines(("testreport =", (json.dumps(testdata,
+                                    sort_keys=True, indent=4))))
+                finally:
+                    fp.close()
+            except ImportError:
+                pass
+
         self._runner._checkhglib('Tested')