Difference between revisions of "2012 run DAQ malfunction"
(Created page with "from: Oleksiy Kosinov <kosiolek@isu.edu> to: Tony Forest <foretony@isu.edu> date: Thu, Aug 23, 2012 at 11:09 AM subject: crash_report mailed-by: isu.edu Hello Dr. Forest, …") |
|||
| (14 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
| + | ==2n correlation experiment== | ||
| + | |||
| + | The experiment started on 08/02/2012 and ended on 08/24/2012. | ||
| + | |||
| + | 08/08/2012 - switch to "common stop" mode. No crashes detected so far. | ||
| + | |||
| + | 08/16/2012 - First crash observed on the first run on that day. Several minutes later after "beam down" failure another crash. 10:50 am another crash. 3 crashes in total. | ||
| + | |||
| + | 08/17/2012 - the crash happend 3 times. | ||
| + | |||
| + | 08/20/2012 - the crash happend 4 times. | ||
| + | |||
| + | 08/22/2012 - the crash happend 2 times. | ||
| + | |||
| + | 08/23/2012 - the crash happend 14 times. See the details below. | ||
| + | |||
| + | 08/24/2012 - the crash happend 4 times. | ||
| + | |||
| + | ==ROC crash reports (08/23/2012)== | ||
| + | |||
from: Oleksiy Kosinov <kosiolek@isu.edu> | from: Oleksiy Kosinov <kosiolek@isu.edu> | ||
to: Tony Forest <foretony@isu.edu> | to: Tony Forest <foretony@isu.edu> | ||
| Line 10: | Line 30: | ||
INFO : transition Prestart succeeded. | INFO : transition Prestart succeeded. | ||
| + | |||
INFO : LDS_ER go..... | INFO : LDS_ER go..... | ||
| + | |||
INFO : eb1 go..... | INFO : eb1 go..... | ||
| + | |||
INFO : roc1 go..... | INFO : roc1 go..... | ||
| + | |||
INFO : transition Go succeeded. | INFO : transition Go succeeded. | ||
| + | |||
WARN : roc1 has not reported status for 21 seconds | WARN : roc1 has not reported status for 21 seconds | ||
| + | |||
WARN : roc1 has not reported status for 31 seconds | WARN : roc1 has not reported status for 31 seconds | ||
| + | |||
WARN : roc1 has not reported status for 41 seconds | WARN : roc1 has not reported status for 41 seconds | ||
| + | |||
WARN : roc1 has not reported status for 51 seconds | WARN : roc1 has not reported status for 51 seconds | ||
| + | |||
WARN : roc1 has not reported status for 61 seconds | WARN : roc1 has not reported status for 61 seconds | ||
| + | |||
WARN : roc1 has not reported status for 71 seconds | WARN : roc1 has not reported status for 71 seconds | ||
| + | |||
WARN : roc1 has not reported status for 81 seconds | WARN : roc1 has not reported status for 81 seconds | ||
| + | |||
ERROR : End Run, roc1 no status for 141 seconds | ERROR : End Run, roc1 no status for 141 seconds | ||
| + | |||
ERROR : roc1 is in state disconnected should be active | ERROR : roc1 is in state disconnected should be active | ||
| + | |||
WARN : roc1 has not reported status for 91 seconds | WARN : roc1 has not reported status for 91 seconds | ||
| + | |||
ERROR : roc1 is in state disconnected should be active | ERROR : roc1 is in state disconnected should be active | ||
| + | |||
WARN : roc1 has not reported status for 101 seconds | WARN : roc1 has not reported status for 101 seconds | ||
| + | |||
ERROR : roc1 is in state disconnected should be active | ERROR : roc1 is in state disconnected should be active | ||
| + | |||
WARN : roc1 has not reported status for 111 seconds | WARN : roc1 has not reported status for 111 seconds | ||
| + | |||
ERROR : roc1 is in state disconnected should be active | ERROR : roc1 is in state disconnected should be active | ||
| + | |||
WARN : roc1 has not reported status for 121 seconds | WARN : roc1 has not reported status for 121 seconds | ||
| + | |||
ERROR : roc1 is in state disconnected should be active | ERROR : roc1 is in state disconnected should be active | ||
| + | |||
WARN : roc1 has not reported status for 131 seconds | WARN : roc1 has not reported status for 131 seconds | ||
| + | |||
ERROR : roc1 is in state disconnected should be active | ERROR : roc1 is in state disconnected should be active | ||
| + | |||
WARN : roc1 has not reported status for 141 seconds | WARN : roc1 has not reported status for 141 seconds | ||
| + | |||
ERROR : roc1 is in state disconnected should be active | ERROR : roc1 is in state disconnected should be active | ||
| + | |||
WARN : roc1 has not reported status for 151 seconds | WARN : roc1 has not reported status for 151 seconds | ||
| + | |||
ERROR : roc1 is in state disconnected should be active | ERROR : roc1 is in state disconnected should be active | ||
| + | |||
WARN : roc1 has not reported status for 161 seconds | WARN : roc1 has not reported status for 161 seconds | ||
| + | |||
ERROR : roc1 is in state disconnected should be active | ERROR : roc1 is in state disconnected should be active | ||
| + | |||
WARN : roc1 has not reported status for 171 seconds | WARN : roc1 has not reported status for 171 seconds | ||
| + | |||
ERROR : roc1 is in state disconnected should be active | ERROR : roc1 is in state disconnected should be active | ||
| Line 62: | Line 113: | ||
FO : roc1 go..... | FO : roc1 go..... | ||
| + | |||
INFO : transition Go succeeded. | INFO : transition Go succeeded. | ||
| + | |||
WARN : roc1 has not reported status for 16 seconds | WARN : roc1 has not reported status for 16 seconds | ||
| + | |||
WARN : roc1 has not reported status for 26 seconds | WARN : roc1 has not reported status for 26 seconds | ||
| + | |||
WARN : roc1 has not reported status for 36 seconds | WARN : roc1 has not reported status for 36 seconds | ||
| + | |||
WARN : roc1 has not reported status for 46 seconds | WARN : roc1 has not reported status for 46 seconds | ||
| + | |||
WARN : roc1 has not reported status for 56 seconds | WARN : roc1 has not reported status for 56 seconds | ||
| + | |||
WARN : roc1 has not reported status for 66 seconds | WARN : roc1 has not reported status for 66 seconds | ||
| + | |||
WARN : roc1 has not reported status for 76 seconds | WARN : roc1 has not reported status for 76 seconds | ||
| + | |||
ERROR : End Run, roc1 no status for 136 seconds | ERROR : End Run, roc1 no status for 136 seconds | ||
| + | |||
ERROR : roc1 is in state disconnected should be active | ERROR : roc1 is in state disconnected should be active | ||
| + | |||
WARN : roc1 has not reported status for 86 seconds | WARN : roc1 has not reported status for 86 seconds | ||
| + | |||
ERROR : roc1 is in state disconnected should be active | ERROR : roc1 is in state disconnected should be active | ||
| Line 90: | Line 153: | ||
INFO : LDS_ER go..... | INFO : LDS_ER go..... | ||
| + | |||
INFO : eb1 go..... | INFO : eb1 go..... | ||
| + | |||
INFO : roc1 go..... | INFO : roc1 go..... | ||
| + | |||
INFO : transition Go succeeded. | INFO : transition Go succeeded. | ||
| + | |||
WARN : roc1 has not reported status for 13 seconds | WARN : roc1 has not reported status for 13 seconds | ||
| + | |||
WARN : roc1 has not reported status for 23 seconds | WARN : roc1 has not reported status for 23 seconds | ||
| + | |||
WARN : roc1 has not reported status for 33 seconds | WARN : roc1 has not reported status for 33 seconds | ||
| + | |||
WARN : roc1 has not reported status for 43 seconds | WARN : roc1 has not reported status for 43 seconds | ||
| + | |||
WARN : roc1 has not reported status for 48 seconds | WARN : roc1 has not reported status for 48 seconds | ||
| + | |||
INFO : roc1 end...... | INFO : roc1 end...... | ||
| + | |||
WARN : roc1 has not reported status for 13 seconds | WARN : roc1 has not reported status for 13 seconds | ||
| + | |||
WARN : roc1 has not reported status for 14 seconds | WARN : roc1 has not reported status for 14 seconds | ||
| + | |||
WARN : roc1 has not reported status for 15 seconds | WARN : roc1 has not reported status for 15 seconds | ||
| + | |||
WARN : roc1 has not reported status for 15 seconds | WARN : roc1 has not reported status for 15 seconds | ||
| + | |||
WARN : roc1 has not reported status for 16 seconds | WARN : roc1 has not reported status for 16 seconds | ||
| + | |||
WARN : roc1 has not reported status for 17 secon | WARN : roc1 has not reported status for 17 secon | ||
| Line 109: | Line 187: | ||
daLogMsg: INFO: prestarted | daLogMsg: INFO: prestarted | ||
| + | |||
daLogMsg: INFO: activating | daLogMsg: INFO: activating | ||
| + | |||
informEB: msgQSend done... | informEB: msgQSend done... | ||
| + | |||
rolp->daproc = 5 | rolp->daproc = 5 | ||
| + | |||
daLogMsg: INFO: Entering User Go 2 | daLogMsg: INFO: Entering User Go 2 | ||
| + | |||
rolp->daproc = 5 | rolp->daproc = 5 | ||
| + | |||
daLogMsg: INFO: Entering User Go | daLogMsg: INFO: Entering User Go | ||
| + | |||
daLogMsg: INFO: active, events so far 0 token 0 | daLogMsg: INFO: active, events so far 0 token 0 | ||
| − | |||
| − | |||
| − | |||
| − | |||
-> | -> | ||
Latest revision as of 16:36, 12 August 2014
2n correlation experiment
The experiment started on 08/02/2012 and ended on 08/24/2012.
08/08/2012 - switch to "common stop" mode. No crashes detected so far.
08/16/2012 - First crash observed on the first run on that day. Several minutes later after "beam down" failure another crash. 10:50 am another crash. 3 crashes in total.
08/17/2012 - the crash happend 3 times.
08/20/2012 - the crash happend 4 times.
08/22/2012 - the crash happend 2 times.
08/23/2012 - the crash happend 14 times. See the details below.
08/24/2012 - the crash happend 4 times.
ROC crash reports (08/23/2012)
from: Oleksiy Kosinov <kosiolek@isu.edu> to: Tony Forest <foretony@isu.edu> date: Thu, Aug 23, 2012 at 11:09 AM subject: crash_report mailed-by: isu.edu
Hello Dr. Forest,
Here is daq crash report:
INFO : transition Prestart succeeded.
INFO : LDS_ER go.....
INFO : eb1 go.....
INFO : roc1 go.....
INFO : transition Go succeeded.
WARN : roc1 has not reported status for 21 seconds
WARN : roc1 has not reported status for 31 seconds
WARN : roc1 has not reported status for 41 seconds
WARN : roc1 has not reported status for 51 seconds
WARN : roc1 has not reported status for 61 seconds
WARN : roc1 has not reported status for 71 seconds
WARN : roc1 has not reported status for 81 seconds
ERROR : End Run, roc1 no status for 141 seconds
ERROR : roc1 is in state disconnected should be active
WARN : roc1 has not reported status for 91 seconds
ERROR : roc1 is in state disconnected should be active
WARN : roc1 has not reported status for 101 seconds
ERROR : roc1 is in state disconnected should be active
WARN : roc1 has not reported status for 111 seconds
ERROR : roc1 is in state disconnected should be active
WARN : roc1 has not reported status for 121 seconds
ERROR : roc1 is in state disconnected should be active
WARN : roc1 has not reported status for 131 seconds
ERROR : roc1 is in state disconnected should be active
WARN : roc1 has not reported status for 141 seconds
ERROR : roc1 is in state disconnected should be active
WARN : roc1 has not reported status for 151 seconds
ERROR : roc1 is in state disconnected should be active
WARN : roc1 has not reported status for 161 seconds
ERROR : roc1 is in state disconnected should be active
WARN : roc1 has not reported status for 171 seconds
ERROR : roc1 is in state disconnected should be active
from: Tony Forest <foretony@isu.edu>
to: Oleksiy Kosinov <kosiolek@isu.edu>
date: Thu, Aug 23, 2012 at 11:22 AM
subject: Re: crash_report
That is a ROC crash not a DAQ host computer (daq1) crash.
You should only need to reboot the ROC in the VME crate.
from: Oleksiy Kosinov <kosiolek@isu.edu>
to: Tony Forest <foretony@isu.edu>
date: Thu, Aug 23, 2012 at 11:27 AM
subject: Re: crash_report
There is another crash. 2 min ago.
FO : roc1 go.....
INFO : transition Go succeeded.
WARN : roc1 has not reported status for 16 seconds
WARN : roc1 has not reported status for 26 seconds
WARN : roc1 has not reported status for 36 seconds
WARN : roc1 has not reported status for 46 seconds
WARN : roc1 has not reported status for 56 seconds
WARN : roc1 has not reported status for 66 seconds
WARN : roc1 has not reported status for 76 seconds
ERROR : End Run, roc1 no status for 136 seconds
ERROR : roc1 is in state disconnected should be active
WARN : roc1 has not reported status for 86 seconds
ERROR : roc1 is in state disconnected should be active
Thanks,
Oleksiy
from: Oleksiy Kosinov <kosiolek@isu.edu>
to: Tony Forest <foretony@isu.edu>
date: Thu, Aug 23, 2012 at 11:58 AM
subject: Re: crash_report
mailed-by: isu.edu
Another crash:
INFO : LDS_ER go.....
INFO : eb1 go.....
INFO : roc1 go.....
INFO : transition Go succeeded.
WARN : roc1 has not reported status for 13 seconds
WARN : roc1 has not reported status for 23 seconds
WARN : roc1 has not reported status for 33 seconds
WARN : roc1 has not reported status for 43 seconds
WARN : roc1 has not reported status for 48 seconds
INFO : roc1 end......
WARN : roc1 has not reported status for 13 seconds
WARN : roc1 has not reported status for 14 seconds
WARN : roc1 has not reported status for 15 seconds
WARN : roc1 has not reported status for 15 seconds
WARN : roc1 has not reported status for 16 seconds
WARN : roc1 has not reported status for 17 secon
ROC window message:
daLogMsg: INFO: prestarted
daLogMsg: INFO: activating
informEB: msgQSend done...
rolp->daproc = 5
daLogMsg: INFO: Entering User Go 2
rolp->daproc = 5
daLogMsg: INFO: Entering User Go
daLogMsg: INFO: active, events so far 0 token 0
->