Experimental Physics and
Industrial Control System

Michael Davidsaver <[email protected]> · Mon, 17 Nov 2014 17:41:04 -0500

I've pushed a re-write of ringPointerTest which should address the
occasional test failures we've been seeing.

On 10/06/2014 01:09 PM, Andrew Johnson wrote:
> The instability that Jenkins has been reporting on the 3.15 branch is
> due to test failures in ringPointerTest, and I've just run the tests on
> Solaris (various unrelated commits to come that fix building here) and
> it's failing there too (with more failures than Jenkins is seeing).
> There were changes to this test program in the parallel-cbthreads-3
> branch, so I'm guessing that's what introduced the problem.
> 
> Note that our Jenkins server still has only one CPU, as does our Solaris
> box venus, so this might be related to that. Full Solaris results are
> given below.
> 
> BTW I'm not very happy about test programs that produce a variable
> number of results in the event of test failures, as this one does.
> Please don't write tests that call testPass() or testFail() within a
> loop that might finish early (or late) resulting in fewer (more) results
> than your plan said it should.
> 
> Giving a different number of test results than planned won't always flag
> a test failure (the Jenkins TAP parser isn't perfect either). Consider
> the possibility that one loop might finish early and another later than
> expected, with the result being the correct number of tests passing even
> though the software is not behaving as it should.
> 
> It's much better to introduce loop counters and check the value of those
> counters at the end of the loop (this also makes the test program output
> less verbose and easier to find the failures in, especially on
> VxWorks/RTEMS).
> 
> - Andrew
> 
> 
> venus% ./ringPointerTest
> 1..256
> # ******************************************************
> # ** Test 1: local ring pointer, check size and order **
> # ******************************************************
> ok  1 - Ring empty
> ok  2 - ring filled, 10 values
> ok  3 - Pop test: 0 == 0
> ok  4 - Pop test: 1 == 1
> ok  5 - Pop test: 2 == 2
> ok  6 - Pop test: 3 == 3
> ok  7 - Pop test: 4 == 4
> ok  8 - Pop test: 5 == 5
> ok  9 - Pop test: 6 == 6
> ok 10 - Pop test: 7 == 7
> ok 11 - Pop test: 8 == 8
> ok 12 - Pop test: 9 == 9
> ok 13 - Ring empty
> # **************************************************************
> # ** Test 2: unlocked ring pointer, one consumer, check order **
> # **************************************************************
> # consumer starting
> ok 14 - Pushing 0, ring not full
> ok 15 - Pushing 1, ring not full
> ok 16 - Pushing 2, ring not full
> ok 17 - Pushing 3, ring not full
> ok 18 - Pushing 4, ring not full
> ok 19 - Pushing 5, ring not full
> ok 20 - Pushing 6, ring not full
> ok 21 - Pushing 7, ring not full
> ok 22 - Pushing 8, ring not full
> ok 23 - Pushing 9, ring not full
> ok 24 - consumer: (got) 0 == 0 (expected)
> ok 25 - consumer: (got) 1 == 1 (expected)
> ok 26 - consumer: (got) 2 == 2 (expected)
> ok 27 - consumer: (got) 3 == 3 (expected)
> ok 28 - Pushing 10, ring not full
> ok 29 - Pushing 11, ring not full
> ok 30 - Pushing 12, ring not full
> ok 31 - Pushing 13, ring not full
> ok 32 - consumer: (got) 4 == 4 (expected)
> ok 33 - consumer: (got) 5 == 5 (expected)
> ok 34 - consumer: (got) 6 == 6 (expected)
> ok 35 - consumer: (got) 7 == 7 (expected)
> ok 36 - Pushing 14, ring not full
> ok 37 - Pushing 15, ring not full
> ok 38 - Pushing 16, ring not full
> ok 39 - Pushing 17, ring not full
> ok 40 - consumer: (got) 8 == 8 (expected)
> ok 41 - consumer: (got) 9 == 9 (expected)
> ok 42 - consumer: (got) 10 == 10 (expected)
> ok 43 - consumer: (got) 11 == 11 (expected)
> ok 44 - Pushing 18, ring not full
> ok 45 - Pushing 19, ring not full
> ok 46 - consumer: (got) 12 == 12 (expected)
> ok 47 - consumer: (got) 13 == 13 (expected)
> ok 48 - consumer: (got) 14 == 14 (expected)
> ok 49 - consumer: (got) 15 == 15 (expected)
> ok 50 - consumer: (got) 16 == 16 (expected)
> ok 51 - consumer: (got) 17 == 17 (expected)
> ok 52 - consumer: (got) 18 == 18 (expected)
> ok 53 - consumer: (got) 19 == 19 (expected)
> ok 54 - Ring empty
> ok 55 - Value test: 0 was processed
> ok 56 - Value test: 1 was processed
> ok 57 - Value test: 2 was processed
> ok 58 - Value test: 3 was processed
> ok 59 - Value test: 4 was processed
> ok 60 - Value test: 5 was processed
> ok 61 - Value test: 6 was processed
> ok 62 - Value test: 7 was processed
> ok 63 - Value test: 8 was processed
> ok 64 - Value test: 9 was processed
> ok 65 - Value test: 10 was processed
> ok 66 - Value test: 11 was processed
> ok 67 - Value test: 12 was processed
> ok 68 - Value test: 13 was processed
> ok 69 - Value test: 14 was processed
> ok 70 - Value test: 15 was processed
> ok 71 - Value test: 16 was processed
> ok 72 - Value test: 17 was processed
> ok 73 - Value test: 18 was processed
> ok 74 - Value test: 19 was processed
> #
> *************************************************************************************
> # ** Test 3: locked ring pointer, many consumers, many producers, check
> no of copies **
> #
> *************************************************************************************
> ok 75 - Ring empty
> # consumer0 starting
> # consumer1 starting
> # consumer2 starting
> # consumer3 starting
> # producer0 starting
> ok 76 - producer0: Pushing 0, ring not full
> ok 77 - producer0: Pushing 1, ring not full
> ok 78 - producer0: Pushing 2, ring not full
> ok 79 - producer0: Pushing 3, ring not full
> ok 80 - producer0: Pushing 4, ring not full
> ok 81 - producer0: Pushing 5, ring not full
> ok 82 - producer0: Pushing 6, ring not full
> ok 83 - producer0: Pushing 7, ring not full
> ok 84 - producer0: Pushing 8, ring not full
> ok 85 - producer0: Pushing 9, ring not full
> # producer1 starting
> # producer2 starting
> # producer3 starting
> ok 86 - consumer3: got a 0 (0 times seen before)
> ok 87 - consumer3: got a 1 (0 times seen before)
> ok 88 - consumer3: got a 2 (0 times seen before)
> ok 89 - consumer3: got a 3 (0 times seen before)
> ok 90 - producer0: Pushing 10, ring not full
> ok 91 - producer0: Pushing 11, ring not full
> ok 92 - producer0: Pushing 12, ring not full
> ok 93 - producer0: Pushing 13, ring not full
> ok 94 - consumer2: got a 4 (0 times seen before)
> ok 95 - consumer3: got a 5 (0 times seen before)
> ok 96 - consumer2: got a 6 (0 times seen before)
> ok 97 - consumer3: got a 7 (0 times seen before)
> ok 98 - consumer2: got a 8 (0 times seen before)
> ok 99 - consumer3: got a 9 (0 times seen before)
> ok 100 - consumer2: got a 10 (0 times seen before)
> ok 101 - consumer3: got a 11 (0 times seen before)
> ok 102 - producer0: Pushing 14, ring not full
> ok 103 - producer0: Pushing 15, ring not full
> ok 104 - producer0: Pushing 16, ring not full
> ok 105 - producer0: Pushing 17, ring not full
> ok 106 - producer0: Pushing 18, ring not full
> ok 107 - producer0: Pushing 19, ring not full
> ok 108 - producer1: Pushing 0, ring not full
> ok 109 - producer1: Pushing 1, ring not full
> ok 110 - consumer1: got a 12 (0 times seen before)
> ok 111 - consumer2: got a 13 (0 times seen before)
> ok 112 - consumer3: got a 14 (0 times seen before)
> ok 113 - consumer1: got a 15 (0 times seen before)
> ok 114 - consumer2: got a 16 (0 times seen before)
> ok 115 - consumer3: got a 17 (0 times seen before)
> ok 116 - consumer0: got a 18 (0 times seen before)
> ok 117 - consumer1: got a 19 (0 times seen before)
> ok 118 - consumer2: got a 0 (1 times seen before)
> ok 119 - consumer3: got a 1 (1 times seen before)
> ok 120 - producer1: Pushing 2, ring not full
> ok 121 - producer1: Pushing 3, ring not full
> ok 122 - producer1: Pushing 4, ring not full
> ok 123 - producer1: Pushing 5, ring not full
> ok 124 - producer1: Pushing 6, ring not full
> ok 125 - producer1: Pushing 7, ring not full
> ok 126 - producer1: Pushing 8, ring not full
> ok 127 - producer1: Pushing 9, ring not full
> ok 128 - producer1: Pushing 10, ring not full
> ok 129 - producer1: Pushing 11, ring not full
> ok 130 - consumer3: got a 2 (1 times seen before)
> ok 131 - consumer3: got a 3 (1 times seen before)
> ok 132 - consumer3: got a 4 (1 times seen before)
> ok 133 - consumer3: got a 5 (1 times seen before)
> ok 134 - producer1: Pushing 12, ring not full
> ok 135 - producer1: Pushing 13, ring not full
> ok 136 - producer1: Pushing 14, ring not full
> ok 137 - producer1: Pushing 15, ring not full
> ok 138 - consumer3: got a 6 (1 times seen before)
> ok 139 - consumer2: got a 7 (1 times seen before)
> ok 140 - consumer3: got a 8 (1 times seen before)
> ok 141 - consumer2: got a 9 (1 times seen before)
> ok 142 - consumer3: got a 10 (1 times seen before)
> ok 143 - consumer2: got a 11 (1 times seen before)
> ok 144 - consumer3: got a 12 (1 times seen before)
> ok 145 - consumer2: got a 13 (1 times seen before)
> ok 146 - producer1: Pushing 16, ring not full
> ok 147 - producer1: Pushing 17, ring not full
> ok 148 - producer1: Pushing 18, ring not full
> ok 149 - producer1: Pushing 19, ring not full
> ok 150 - producer2: Pushing 0, ring not full
> ok 151 - producer2: Pushing 1, ring not full
> ok 152 - producer2: Pushing 2, ring not full
> ok 153 - producer2: Pushing 3, ring not full
> ok 154 - consumer1: got a 14 (1 times seen before)
> ok 155 - consumer3: got a 15 (1 times seen before)
> ok 156 - consumer2: got a 16 (1 times seen before)
> ok 157 - consumer1: got a 17 (1 times seen before)
> ok 158 - consumer3: got a 18 (1 times seen before)
> ok 159 - consumer2: got a 19 (1 times seen before)
> ok 160 - consumer1: got a 0 (2 times seen before)
> ok 161 - consumer3: got a 1 (2 times seen before)
> ok 162 - consumer2: got a 2 (2 times seen before)
> ok 163 - consumer1: got a 3 (2 times seen before)
> ok 164 - producer2: Pushing 4, ring not full
> ok 165 - producer2: Pushing 5, ring not full
> ok 166 - producer2: Pushing 6, ring not full
> ok 167 - producer2: Pushing 7, ring not full
> ok 168 - producer2: Pushing 8, ring not full
> ok 169 - producer2: Pushing 9, ring not full
> ok 170 - producer2: Pushing 10, ring not full
> ok 171 - producer2: Pushing 11, ring not full
> ok 172 - producer2: Pushing 12, ring not full
> ok 173 - producer2: Pushing 13, ring not full
> ok 174 - consumer2: got a 4 (2 times seen before)
> ok 175 - consumer1: got a 5 (2 times seen before)
> ok 176 - consumer2: got a 6 (2 times seen before)
> ok 177 - consumer1: got a 7 (2 times seen before)
> ok 178 - consumer2: got a 8 (2 times seen before)
> ok 179 - consumer1: got a 9 (2 times seen before)
> ok 180 - consumer2: got a 10 (2 times seen before)
> ok 181 - consumer1: got a 11 (2 times seen before)
> ok 182 - producer2: Pushing 14, ring not full
> ok 183 - producer2: Pushing 15, ring not full
> ok 184 - producer2: Pushing 16, ring not full
> ok 185 - producer2: Pushing 17, ring not full
> ok 186 - producer2: Pushing 18, ring not full
> ok 187 - producer2: Pushing 19, ring not full
> ok 188 - producer3: Pushing 0, ring not full
> ok 189 - producer3: Pushing 1, ring not full
> ok 190 - consumer3: got a 12 (2 times seen before)
> ok 191 - consumer2: got a 13 (2 times seen before)
> ok 192 - consumer1: got a 14 (2 times seen before)
> ok 193 - consumer3: got a 15 (2 times seen before)
> ok 194 - consumer2: got a 16 (2 times seen before)
> ok 195 - consumer1: got a 17 (2 times seen before)
> not ok 196 - Ring empty
> not ok 197 - Value test: 0 was processed 4 times
> not ok 198 - Value test: 1 was processed 4 times
> not ok 199 - Value test: 2 was processed 4 times
> not ok 200 - Value test: 3 was processed 4 times
> not ok 201 - Value test: 4 was processed 4 times
> not ok 202 - Value test: 5 was processed 4 times
> not ok 203 - Value test: 6 was processed 4 times
> not ok 204 - Value test: 7 was processed 4 times
> not ok 205 - Value test: 8 was processed 4 times
> not ok 206 - Value test: 9 was processed 4 times
> not ok 207 - Value test: 10 was processed 4 times
> not ok 208 - Value test: 11 was processed 4 times
> not ok 209 - Value test: 12 was processed 4 times
> not ok 210 - Value test: 13 was processed 4 times
> not ok 211 - Value test: 14 was processed 4 times
> not ok 212 - Value test: 15 was processed 4 times
> not ok 213 - Value test: 16 was processed 4 times
> not ok 214 - Value test: 17 was processed 4 times
> not ok 215 - Value test: 18 was processed 4 times
> not ok 216 - Value test: 19 was processed 4 times
> ok 217 - consumer3: got a 18 (2 times seen before)
> ok 218 - consumer2: got a 19 (2 times seen before)
> ok 219 - consumer1: got a 0 (3 times seen before)
> ok 220 - consumer3: got a 1 (3 times seen before)
> 
> Planned 256 tests but only ran 220
> 
>     Results
>     =======
>        Tests: 220
>       Passed: 199 = 90.45%
>       Failed:  21 =  9.55%
> 

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System