Experimental Physics and
Industrial Control System

Mark Rivers <[email protected]> · Wed, 8 Jun 2016 02:36:43 +0000

Hi Stephanie,

Thanks for the reply.

>> I see that there are 2 cpuUsageTask tasks.  This is true on all of my VxWorks IOCs.
>> Is this normal?

> No, it is not normal.  It is possible that you loaded the devIocStats database twice with different
> names.  devIocStats is far from perfect - one of its imperfections is that it doesn't check if
> initialization is already done and goes ahead and does it again.   Need a bug report for this.

I found the problem.  My application was built with both devIocStats.dbd and the old vxStats.dbd.  It was not linked with the vxStats library, but it must have been getting one cpuUsageTask from devIocStats and the second from vxStats.  Once I removed vxStats.dbd from the build I only have a single cpuUsageTask.

Mark

________________________________________
From: Allison, Stephanie [[email protected]]
Sent: Tuesday, June 07, 2016 6:43 PM
To: Mark Rivers; 'Andrew Johnson'
Cc: [email protected]
Subject: RE: Questions about devIocStats on vxWorks

Hi Mark,

> I see that there are 2 cpuUsageTask tasks. This is true on all of my VxWorks IOCs. Is this
> normal?

No, it is not normal. It is possible that you loaded the devIocStats database twice with different names. devIocStats is far from perfect - one of its imperfections is that it doesn't check if initialization is already done and goes ahead and does it again. Need a bug report for this.

Stephanie

> -----Original Message-----
> From: Mark Rivers [mailto:[email protected]]
> Sent: Friday, June 03, 2016 10:17 AM
> To: 'Andrew Johnson'; Allison, Stephanie
> Cc: [email protected]
> Subject: Questions about devIocStats on vxWorks
>
> Hi,
>
>
>
> I am running devIocStats (current master branch from github) on VxWorks 6.9.4.1.
>
>
>
> I ran into some issues on this IOC yesterday where it would not accept any new channel access
> connections. That normally indicates either CPU saturation or memory low. The memory was fine
> (more than 300 MB free). I could not look at the devIocStats medm screens because it was not
> accepting any new CA connections and I did not happen to have that screen open.
>
>
>
> However, I was able to use spy and got the following output:
>
>
>
> NAME ENTRY TID PRI total % (ticks) delta % (ticks)
>
> ------------ ------------ ---------- --- --------------- ---------------
>
> tJobTask 0x1c6b00 0x23b8fa0 0 0% ( 0) 0% ( 0)
>
> tExcTask 0x1c61f8 0x33a5d0 0 0% ( 0) 0% ( 0)
>
> tLogTask logTask 0x23bc340 0 0% ( 0) 0% ( 0)
>
> tShell0 shellTask 0x7aae3e0 1 0% ( 0) 0% ( 0)
>
> tWdbTask 0x261f48 0x26e0a30 3 0% ( 0) 0% ( 0)
>
> tSpyTask spyComTask 0x7e94080 5 3% ( 147) 3% ( 72)
>
> ipcom_tickd 0x285b88 0x26c6ea0 20 0% ( 0) 0% ( 0)
>
> tVxdbgTask 0x13185c 0x26da910 25 0% ( 0) 0% ( 0)
>
> tNet0 ipcomNetTask 0x23c1060 50 2% ( 102) 1% ( 40)
>
> ipcom_syslog 0x16bf48 0x268d220 50 0% ( 0) 0% ( 0)
>
> tNetConf 0x1b74b8 0x26b9a50 50 0% ( 0) 0% ( 0)
>
> ipcom_telnet ipcom_telnet 0x26cdcc0 50 0% ( 0) 0% ( 0)
>
> ipsntps 0x1ba330 0x26d1450 50 0% ( 0) 0% ( 0)
>
> tPortmapd portmapd 0x26d5570 54 0% ( 0) 0% ( 0)
>
> NTPTimeSync 0x352d878 0x278c540 109 0% ( 0) 0% ( 0)
>
> ClockTimeSyn 0x352d878 0x278f770 109 0% ( 0) 0% ( 0)
>
> ipUnidig 0x352d878 0x3bc2b50 109 0% ( 0) 0% ( 0)
>
> Ip330intTask 0x352d878 0x3c05c20 109 0% ( 0) 0% ( 0)
>
> abDone00 0x321181c 0x4b203a0 122 0% ( 0) 0% ( 0)
>
> abScan00 0x3211d8c 0x4b1d940 123 0% ( 0) 0% ( 0)
>
> cbHigh 0x352d878 0x4b061e0 128 0% ( 0) 0% ( 0)
>
> timerQueue 0x352d878 0x4ae4bd0 129 0% ( 0) 0% ( 0)
>
> scanOnce 0x352d878 0x6a249b0 129 1% ( 56) 0% ( 20)
>
> IK320Logger 0x3247cdc 0x4b0e7b0 130 0% ( 0) 0% ( 0)
>
> scan0.1 0x352d878 0x6d08140 133 0% ( 7) 0% ( 1)
>
> scan0.2 0x352d878 0x6d01610 134 1% ( 46) 0% ( 10)
>
> cbMedium 0x352d878 0x4afd9a0 135 0% ( 21) 0% ( 18)
>
> scan0.5 0x352d878 0x6cfaae0 135 0% ( 7) 0% ( 2)
>
> drvQuadEMCal 0x352d878 0x3ed9630 136 0% ( 0) 0% ( 0)
>
> scan1 0x352d878 0x6cf3fb0 136 0% ( 0) 0% ( 0)
>
> CAS-client 0x352d878 0x7abaf40 136 0% ( 0) 0% ( 0)
>
> scan2 0x352d878 0x6ced480 137 0% ( 0) 0% ( 0)
>
> scan5 0x352d878 0x6ce6950 138 0% ( 5) 0% ( 2)
>
> timerQueue 0x352d878 0x39fa1e0 139 0% ( 0) 0% ( 0)
>
> timerQueue 0x352d878 0x3bb7240 139 0% ( 0) 0% ( 0)
>
> scan10 0x352d878 0x6cdfe20 139 0% ( 1) 0% ( 1)
>
> cbLow 0x352d878 0x4af5160 140 3% ( 179) 4% ( 89)
>
> timerQueue 0x352d878 0x4b24ee0 141 0% ( 0) 0% ( 0)
>
> CAC-UDP 0x352d878 0x6f8afe0 147 0% ( 1) 0% ( 0)
>
> CAC-UDP 0x352d878 0x7135d20 147 0% ( 1) 0% ( 0)
>
> timerQueue 0x352d878 0x6f76370 148 0% ( 8) 0% ( 2)
>
> CAC-TCP-send 0x352d878 0x6fcfcd0 148 0% ( 0) 0% ( 0)
>
> CAC-event 0x352d878 0x70844b0 148 2% ( 116) 2% ( 47)
>
> CAC-TCP-send 0x352d878 0x7093aa0 148 0% ( 0) 0% ( 0)
>
> CAC-TCP-send 0x352d878 0x70a3100 148 0% ( 0) 0% ( 0)
>
> CAC-TCP-send 0x352d878 0x70b2760 148 0% ( 0) 0% ( 0)
>
> CAC-TCP-send 0x352d878 0x70c1dc0 148 0% ( 0) 0% ( 0)
>
> CAC-event 0x352d878 0x7119160 148 0% ( 0) 0% ( 0)
>
> timerQueue 0x352d878 0x7121660 148 0% ( 16) 0% ( 6)
>
> CAC-event 0x352d878 0x26f7e20 148 0% ( 0) 0% ( 0)
>
> serial1 0x352d878 0x3c21b40 149 0% ( 0) 0% ( 0)
>
> serial2 0x352d878 0x3c2d260 149 0% ( 0) 0% ( 0)
>
> serial3 0x352d878 0x3c326a0 149 0% ( 0) 0% ( 0)
>
> serial4 0x352d878 0x3c3b940 149 0% ( 0) 0% ( 0)
>
> serial5 0x352d878 0x3c466e0 149 0% ( 0) 0% ( 0)
>
> serial6 0x352d878 0x3c4f060 149 0% ( 0) 0% ( 0)
>
> serial7 0x352d878 0x3c57e30 149 0% ( 4) 0% ( 0)
>
> serial8 0x352d878 0x3c62c20 149 0% ( 10) 0% ( 3)
>
> serial9 0x352d878 0x3c73690 149 0% ( 0) 0% ( 0)
>
> serial10 0x352d878 0x3c83ba0 149 0% ( 0) 0% ( 0)
>
> serial11 0x352d878 0x3cb2900 149 0% ( 0) 0% ( 0)
>
> serial12 0x352d878 0x3cbb0e0 149 0% ( 23) 0% ( 6)
>
> serial13 0x352d878 0x3ccd400 149 0% ( 0) 0% ( 0)
>
> serial14 0x352d878 0x3ce0370 149 0% ( 0) 0% ( 0)
>
> serial15 0x352d878 0x3d0ed90 149 0% ( 0) 0% ( 0)
>
> serial16 0x352d878 0x3d16fc0 149 0% ( 0) 0% ( 0)
>
> serial17 0x352d878 0x3d202f0 149 0% ( 0) 0% ( 0)
>
> serial18 0x352d878 0x3d49bd0 149 0% ( 0) 0% ( 0)
>
> serial19 0x352d878 0x3d73010 149 0% ( 0) 0% ( 0)
>
> serial20 0x352d878 0x3d796b0 149 0% ( 0) 0% ( 0)
>
> serial21 0x352d878 0x3d7fd50 149 0% ( 0) 0% ( 0)
>
> serial22 0x352d878 0x3d863f0 149 0% ( 0) 0% ( 0)
>
> serial23 0x352d878 0x3d8ca90 149 0% ( 0) 0% ( 0)
>
> serial24 0x352d878 0x3d93130 149 0% ( 0) 0% ( 0)
>
> IP_QE1 0x352d878 0x3da0750 149 0% ( 0) 0% ( 0)
>
> QE1 0x352d878 0x3daa930 149 0% ( 0) 0% ( 0)
>
> drvTetrAMMTa 0x352d878 0x3edffb0 149 13% ( 611) 13% ( 265)
>
> QE1_STATS1_P 0x352d878 0x3f00700 149 0% ( 0) 0% ( 0)
>
> QE1_STATS2_P 0x352d878 0x3f3b7c0 149 0% ( 0) 0% ( 0)
>
> QE1_STATS3_P 0x352d878 0x3f73510 149 0% ( 0) 0% ( 0)
>
> QE1_STATS4_P 0x352d878 0x3faa780 149 0% ( 0) 0% ( 0)
>
> QE1_STATS5_P 0x352d878 0x3fe1a20 149 0% ( 0) 0% ( 0)
>
> QE1_STATS6_P 0x352d878 0x40189a0 149 0% ( 0) 0% ( 0)
>
> QE1_STATS7_P 0x352d878 0x404fac0 149 0% ( 0) 0% ( 0)
>
> QE1_STATS8_P 0x352d878 0x4086bb0 149 0% ( 0) 0% ( 0)
>
> QE1_STATS9_P 0x352d878 0x40bdb90 149 0% ( 0) 0% ( 0)
>
> QE1_STATS10_ 0x352d878 0x40f4b70 149 0% ( 0) 0% ( 0)
>
> QE1_STATS11_ 0x352d878 0x412c190 149 0% ( 0) 0% ( 0)
>
> QE1_FileNetC 0x352d878 0x4162290 149 0% ( 0) 0% ( 0)
>
> QE1_FileNetC 0x352d878 0x416ad20 149 0% ( 0) 0% ( 0)
>
> QE1_Image1_P 0x352d878 0x418b300 149 0% ( 0) 0% ( 0)
>
> QE1_TS1_Plug 0x352d878 0x41bfb40 149 0% ( 0) 0% ( 0)
>
> QE1_FFT1_Plu 0x352d878 0x426b9e0 149 0% ( 0) 0% ( 0)
>
> QE1_FFT2_Plu 0x352d878 0x4289080 149 0% ( 0) 0% ( 0)
>
> QE1_FFT3_Plu 0x352d878 0x42a6320 149 0% ( 0) 0% ( 0)
>
> QE1_FFT4_Plu 0x352d878 0x42c3520 149 0% ( 0) 0% ( 0)
>
> QE1_FFT5_Plu 0x352d878 0x42e07f0 149 0% ( 0) 0% ( 0)
>
> QE1_FFT6_Plu 0x352d878 0x42fdf00 149 0% ( 0) 0% ( 0)
>
> QE1_FFT7_Plu 0x352d878 0x431b070 149 0% ( 0) 0% ( 0)
>
> QE1_FFT8_Plu 0x352d878 0x4338610 149 0% ( 0) 0% ( 0)
>
> QE1_FFT9_Plu 0x352d878 0x43557b0 149 0% ( 0) 0% ( 0)
>
> QE1_FFT10_Pl 0x352d878 0x4372950 149 0% ( 0) 0% ( 0)
>
> QE1_FFT11_Pl 0x352d878 0x438fbe0 149 0% ( 0) 0% ( 0)
>
> IP_QE2 0x352d878 0x4394a00 149 0% ( 0) 0% ( 0)
>
> QE2 0x352d878 0x43a9cb0 149 0% ( 0) 0% ( 0)
>
> drvQuadEMCal 0x352d878 0x44dbb20 149 0% ( 28) 0% ( 14)
>
> drvAHxxxTask 0x352d878 0x44e24a0 149 1% ( 73) 1% ( 33)
>
> QE2_STATS1_P 0x352d878 0x4502700 149 0% ( 7) 0% ( 3)
>
> QE2_STATS2_P 0x352d878 0x4539800 149 0% ( 10) 0% ( 4)
>
> QE2_STATS3_P 0x352d878 0x4570900 149 0% ( 8) 0% ( 1)
>
> QE2_STATS4_P 0x352d878 0x45a7a00 149 0% ( 4) 0% ( 2)
>
> QE2_STATS5_P 0x352d878 0x45df0b0 149 0% ( 7) 0% ( 3)
>
> QE2_STATS6_P 0x352d878 0x4616030 149 0% ( 9) 0% ( 4)
>
> QE2_STATS7_P 0x352d878 0x464cfb0 149 0% ( 12) 0% ( 7)
>
> QE2_STATS8_P 0x352d878 0x4684580 149 0% ( 11) 0% ( 5)
>
> QE2_STATS9_P 0x352d878 0x46bb560 149 0% ( 9) 0% ( 5)
>
> QE2_STATS10_ 0x352d878 0x46f2540 149 0% ( 7) 0% ( 3)
>
> QE2_STATS11_ 0x352d878 0x47295b0 149 0% ( 7) 0% ( 4)
>
> QE2_FileNetC 0x352d878 0x475f5e0 149 0% ( 0) 0% ( 0)
>
> QE2_FileNetC 0x352d878 0x4768070 149 0% ( 8) 0% ( 2)
>
> QE2_Image1_P 0x352d878 0x4788bf0 149 0% ( 9) 0% ( 5)
>
> QE2_TS1_Plug 0x352d878 0x47bd430 149 0% ( 11) 0% ( 4)
>
> QE2_FFT1_Plu 0x352d878 0x48692d0 149 0% ( 0) 0% ( 0)
>
> QE2_FFT2_Plu 0x352d878 0x4886970 149 0% ( 0) 0% ( 0)
>
> QE2_FFT3_Plu 0x352d878 0x48a3b70 149 0% ( 0) 0% ( 0)
>
> QE2_FFT4_Plu 0x352d878 0x48c0d70 149 0% ( 0) 0% ( 0)
>
> QE2_FFT5_Plu 0x352d878 0x48ddf70 149 0% ( 0) 0% ( 0)
>
> QE2_FFT6_Plu 0x352d878 0x48fb0e0 149 0% ( 0) 0% ( 0)
>
> QE2_FFT7_Plu 0x352d878 0x4918250 149 0% ( 0) 0% ( 0)
>
> QE2_FFT8_Plu 0x352d878 0x4935990 149 0% ( 0) 0% ( 0)
>
> QE2_FFT9_Plu 0x352d878 0x4952b30 149 0% ( 0) 0% ( 0)
>
> QE2_FFT10_Pl 0x352d878 0x496fcd0 149 0% ( 0) 0% ( 0)
>
> QE2_FFT11_Pl 0x352d878 0x498cf60 149 0% ( 0) 0% ( 0)
>
> dbCaLink 0x352d878 0x4b0cac0 149 0% ( 0) 0% ( 0)
>
> Oms58_motor 0x352d878 0x4b14cf0 149 0% ( 29) 1% ( 23)
>
> MAXv_motor 0x352d878 0x4b1aee0 149 0% ( 0) 0% ( 0)
>
> CAC-event 0x352d878 0x6d25330 149 0% ( 1) 0% ( 1)
>
> poolPoll 0x352d878 0x6f51280 149 0% ( 0) 0% ( 0)
>
> editSseq 0x352d878 0x70e1a50 149 0% ( 0) 0% ( 0)
>
> Keithley2kDM 0x352d878 0x74aa5c0 149 0% ( 0) 0% ( 0)
>
> Keithley2kDM 0x352d878 0x74b7720 149 0% ( 0) 0% ( 0)
>
> Keithley2kDM 0x352d878 0x74bc760 149 0% ( 0) 0% ( 0)
>
> Keithley2kDM 0x352d878 0x74c1930 149 0% ( 0) 0% ( 0)
>
> motorUtil 0x352d878 0x79ecf30 149 0% ( 0) 0% ( 0)
>
> soft_motor 0x352d878 0x4b36590 150 0% ( 0) 0% ( 0)
>
> CAC-TCP-recv 0x352d878 0x6fcb7d0 150 0% ( 1) 0% ( 0)
>
> CAC-TCP-recv 0x352d878 0x708f5a0 150 0% ( 0) 0% ( 0)
>
> CAC-TCP-recv 0x352d878 0x709ec20 150 0% ( 0) 0% ( 0)
>
> CAC-TCP-recv 0x352d878 0x70ae280 150 0% ( 0) 0% ( 0)
>
> CAC-TCP-recv 0x352d878 0x70bd8e0 150 0% ( 8) 0% ( 0)
>
> CAC-UDP 0x352d878 0x7722990 153 0% ( 0) 0% ( 0)
>
> CAC-UDP 0x352d878 0x3fde2d0 154 0% ( 1) 0% ( 0)
>
> CAC-event 0x352d878 0x76af970 154 0% ( 0) 0% ( 0)
>
> timerQueue 0x352d878 0x770dd40 154 0% ( 7) 0% ( 4)
>
> CAC-TCP-send 0x352d878 0x7a005b0 154 0% ( 0) 0% ( 0)
>
> timerQueue 0x352d878 0x45a42b0 155 0% ( 3) 0% ( 2)
>
> CAC-TCP-send 0x352d878 0x4015250 155 0% ( 0) 0% ( 0)
>
> CAC-TCP-send 0x352d878 0x42dd630 155 0% ( 0) 0% ( 0)
>
> saveDataTask 0x352d878 0x7692150 155 0% ( 0) 0% ( 0)
>
> recDynInp 0x352d878 0x68775d0 156 0% ( 1) 0% ( 0)
>
> recDynOut 0x352d878 0x6880170 156 0% ( 0) 0% ( 0)
>
> CAC-TCP-recv 0x352d878 0x79fc3e0 156 0% ( 0) 0% ( 0)
>
> CAC-TCP-recv 0x352d878 0x71cd3f0 157 0% ( 0) 0% ( 0)
>
> CAC-TCP-recv 0x352d878 0x7269fc0 157 0% ( 0) 0% ( 0)
>
> CAC-UDP 0x352d878 0x75a2d30 177 0% ( 0) 0% ( 0)
>
> CAC-event 0x352d878 0x48f7f20 178 0% ( 35) 0% ( 15)
>
> timerQueue 0x352d878 0x758e120 178 0% ( 2) 0% ( 1)
>
> save_restore 0x352d878 0x7278230 179 0% ( 0) 0% ( 0)
>
> CAS-client 0x352d878 0x7307fc0 179 0% ( 0) 0% ( 0)
>
> CAS-client 0x352d878 0x7472f90 179 0% ( 0) 0% ( 0)
>
> CAS-client 0x352d878 0x7e619b0 179 0% ( 0) 0% ( 0)
>
> CAS-client 0x352d878 0x7e6d470 179 0% ( 0) 0% ( 0)
>
> CAS-event 0x352d878 0x73016e0 180 0% ( 0) 0% ( 0)
>
> CAS-event 0x352d878 0x746c6b0 180 0% ( 0) 0% ( 0)
>
> CAS-event 0x352d878 0x230bcf0 180 1% ( 60) 1% ( 34)
>
> CAS-event 0x352d878 0x7e5ae80 180 0% ( 0) 0% ( 0)
>
> CAS-event 0x352d878 0x7e66940 180 0% ( 4) 0% ( 3)
>
> CAS-TCP 0x352d878 0x6f451f0 181 0% ( 0) 0% ( 0)
>
> CAS-beacon 0x352d878 0x6f491f0 182 0% ( 0) 0% ( 0)
>
> CAS-UDP 0x352d878 0x6f4da80 183 0% ( 0) 0% ( 0)
>
> errlog 0x352d878 0x27997a0 189 0% ( 0) 0% ( 0)
>
> taskwd 0x352d878 0x279f8e0 189 0% ( 0) 0% ( 0)
>
> timerQueue 0x352d878 0x3bf2050 189 0% ( 0) 0% ( 0)
>
> ipToAsciiPro 0x352d878 0x6f71a30 189 0% ( 0) 0% ( 0)
>
> CAC-repeater 0x352d878 0x6f901d0 189 0% ( 2) 0% ( 1)
>
> cpuUsageTask 0x352d878 0x4b2a400 199 29% ( 1326) 28% ( 582)
>
> cpuUsageTask 0x352d878 0x4b2f640 199 26% ( 1221) 28% ( 568)
>
> KERNEL 3% ( 170) 3% ( 74)
>
> INTERRUPT 1% ( 79) 1% ( 24)
>
> IDLE 0% ( 0) 0% ( 0)
>
> TOTAL 86% ( 4531) 86% ( 2015)
>
>
>
>
>
> I see that there are 2 cpuUsageTask tasks. This is true on all of my VxWorks IOCs. Is this
> normal?
>
>
>
> I also see that the IDLE time is 0%, and that the cpuUsageTasks are using all of the CPU time that
> other tasks are not using. Is this also normal?
>
>
>
> If it is normal, then I don't understand why the system was in a state that would not allow any new
> CA connnections. I used the VxWorks "i" command to look at all tasks, and there were no
> suspended tasks. checkStack showed no tasks with stack overflow.
>
>
>
> casr showed some things like this that look suspicious?
>
> TCP 164.54.160.93:51762(Ferrari): User="XAS_User", V4.13, 677 Channels, Priority=0
>
> Task Id=0x7a24820, Socket FD=75
>
> Secs since last send 593.58, Secs since last receive 593.58
>
> Unprocessed request bytes=16, Undelivered response bytes=5192
>
> State=up jumbo-send-buf
>
>
>
> Any ideas, and things to look at if I am able to reproduce the problem?
>
>
>
> Thanks,
>
> Mark
>
>

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System