[ 1840.774771] LustreError: lustre-OST0000-osc-ffff9824100b0800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1840.810846] Lustre: 2355:0:(llite_lib.c:4233:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.130@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1846.893475] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 20:09:58 (1770685798)
[ 1848.566840] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1849.843425] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 20:10:01 (1770685801)
[ 1851.262950] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1852.575317] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 20:10:04 (1770685804)
[ 1855.344667] LustreError: lustre-MDT0000-mdc-ffff9824100b0800: operation ldlm_enqueue to node 192.168.203.130@tcp failed: rc = -19
[ 1855.350954] LustreError: Skipped 1 previous similar message
[ 1874.930519] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 192.168.203.130@tcp) was lost; in progress operations using this service will fail
[ 1875.010578] Lustre: Evicted from MGS (at 192.168.203.130@tcp) after server handle changed from 0x2dd273c4f65dc90a to 0x2dd273c4f65de4cb
[ 1973.879210] LustreError: lustre-MDT0000-mdc-ffff9824100b0800: operation ldlm_enqueue to node 192.168.203.130@tcp failed: rc = -19
[ 1973.887773] LustreError: Skipped 1 previous similar message
[ 1991.666808] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 192.168.203.130@tcp) was lost; in progress operations using this service will fail
[ 1991.707120] Lustre: Evicted from MGS (at 192.168.203.130@tcp) after server handle changed from 0x2dd273c4f65de4cb to 0x2dd273c4f660a8ab
[ 2001.717923] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 20:12:33 (1770685953)
[ 2002.037454] Lustre: *** cfs_fail_loc=305, val=0***
[ 2002.038952] Lustre: Skipped 2 previous similar messages
[ 2037.663174] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 192.168.203.130@tcp) was lost; in progress operations using this service will fail
[ 2037.682731] Lustre: Evicted from MGS (at 192.168.203.130@tcp) after server handle changed from 0x2dd273c4f660a8ab to 0x2dd273c4f660d296
[ 2047.499983] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2048.953699] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2056.051573] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 20:13:27 (1770686007)
[ 2068.471868] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 192.168.203.130@tcp) was lost; in progress operations using this service will fail
[ 2068.507261] Lustre: Evicted from MGS (at 192.168.203.130@tcp) after server handle changed from 0x2dd273c4f660d296 to 0x2dd273c4f660d790
[ 2068.572650] LustreError: lustre-MDT0000-mdc-ffff9824100b0800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2084.008092] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 20:13:55 (1770686035)
[ 2095.901857] LustreError: lustre-OST0000-osc-ffff9824100b0800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2110.220465] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 20:14:21 (1770686061)
[ 2122.948518] LustreError: lustre-MDT0000-mdc-ffff9824100b0800: operation mds_reint to node 192.168.203.130@tcp failed: rc = -19
[ 2122.955957] Lustre: lustre-MDT0000-mdc-ffff9824100b0800: Connection to lustre-MDT0000 (at 192.168.203.130@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2122.971665] Lustre: Skipped 14 previous similar messages
[ 2140.137021] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 192.168.203.130@tcp) was lost; in progress operations using this service will fail
[ 2140.180938] Lustre: Evicted from MGS (at 192.168.203.130@tcp) after server handle changed from 0x2dd273c4f660d790 to 0x2dd273c4f6614823
[ 2140.197035] Lustre: MGC192.168.203.130@tcp: Connection restored to 192.168.203.130@tcp (at 192.168.203.130@tcp)
[ 2140.207360] Lustre: Skipped 18 previous similar messages
[ 2148.995593] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2150.616288] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2232.323148] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 192.168.203.130@tcp) was lost; in progress operations using this service will fail
[ 2232.339317] Lustre: Evicted from MGS (at 192.168.203.130@tcp) after server handle changed from 0x2dd273c4f6614823 to 0x2dd273c4f663b586
[ 2244.435878] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2246.211696] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2309.974556] LustreError: lustre-MDT0000-mdc-ffff9824100b0800: operation ldlm_enqueue to node 192.168.203.130@tcp failed: rc = -19
[ 2309.987937] LustreError: Skipped 4 previous similar messages
[ 2328.556824] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 192.168.203.130@tcp) was lost; in progress operations using this service will fail
[ 2328.616201] Lustre: Evicted from MGS (at 192.168.203.130@tcp) after server handle changed from 0x2dd273c4f663b586 to 0x2dd273c4f6662e88
[ 2337.903273] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2340.072104] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2367.394459] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 20:18:38 (1770686318)
[ 2396.451424] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2420.703203] Lustre: 2358:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770686357/real 1770686357]  req@ffff98240346e300 x1856696758930688/t0(0) o400->MGC192.168.203.130@tcp@192.168.203.130@tcp:26/25 lens 224/224 e 0 to 1 dl 1770686373 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2420.725918] Lustre: 2358:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 2420.733251] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 192.168.203.130@tcp) was lost; in progress operations using this service will fail
[ 2420.748440] LustreError: Skipped 1 previous similar message
[ 2420.766182] Lustre: Evicted from MGS (at 192.168.203.130@tcp) after server handle changed from 0x2dd273c4f667525f to 0x2dd273c4f66794c3
[ 2420.781521] Lustre: Skipped 1 previous similar message
[ 2424.326315] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2456.457773] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2493.566368] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2540.336493] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2567.988146] LustreError: lustre-MDT0000-mdc-ffff9824100b0800: operation ldlm_enqueue to node 192.168.203.130@tcp failed: rc = -19
[ 2567.997739] LustreError: Skipped 9 previous similar messages
[ 2586.143187] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 192.168.203.130@tcp) was lost; in progress operations using this service will fail
[ 2586.151106] LustreError: Skipped 3 previous similar messages
[ 2586.163398] Lustre: Evicted from MGS (at 192.168.203.130@tcp) after server handle changed from 0x2dd273c4f66956ad to 0x2dd273c4f66a62c8
[ 2586.179066] Lustre: Skipped 3 previous similar messages
[ 2590.923093] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2673.440768] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 20:23:45 (1770686625)
[ 2717.841238] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2719.539380] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3015.904929] Lustre: lustre-OST0000-osc-ffff9824100b0800: Connection to lustre-OST0000 (at 192.168.203.130@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3015.914344] Lustre: Skipped 10 previous similar messages
[ 3042.593309] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3044.147993] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3347.442140] LustreError: lustre-OST0000-osc-ffff9824100b0800: operation ldlm_enqueue to node 192.168.203.130@tcp failed: rc = -19
[ 3347.455385] LustreError: Skipped 3 previous similar messages
[ 3374.403509] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3375.725294] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3641.255327] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 20:39:53 (1770687593)
[ 3657.695163] Lustre: 50221:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770687594/real 1770687594]  req@ffff98240624a300 x1856696792464000/t0(0) o101->lustre-MDT0000-mdc-ffff9824100b0800@192.168.203.130@tcp:12/10 lens 576/1152 e 0 to 1 dl 1770687610 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3657.727912] Lustre: 50221:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 3657.736559] Lustre: lustre-MDT0000-mdc-ffff9824100b0800: Connection to lustre-MDT0000 (at 192.168.203.130@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3657.748870] Lustre: Skipped 1 previous similar message
[ 3657.764535] Lustre: lustre-MDT0000-mdc-ffff9824100b0800: Connection restored to 192.168.203.130@tcp (at 192.168.203.130@tcp)
[ 3657.769584] Lustre: Skipped 20 previous similar messages
[ 3662.463320] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 20:40:14 (1770687614)
[ 3683.287884] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 20:40:35 (1770687635)
[ 3704.501077] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 20:40:56 (1770687656)
[ 3704.923963] Lustre: Mounted lustre-client
[ 3730.919551] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 192.168.203.130@tcp) was lost; in progress operations using this service will fail
[ 3730.931291] LustreError: Skipped 1 previous similar message
[ 3730.941656] Lustre: Evicted from MGS (at 192.168.203.130@tcp) after server handle changed from 0x2dd273c4f66b04bd to 0x2dd273c4f68ed0f1
[ 3730.946937] Lustre: Skipped 1 previous similar message
[ 3737.060215] Lustre: 2356:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770687673/real 1770687673]  req@ffff982407d96300 x1856696792480640/t0(0) o400->lustre-MDT0000-mdc-ffff9824100b0800@192.168.203.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1770687689 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3737.080116] Lustre: 2356:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3737.961695] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3738.807429] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3740.299023] LustreError: 52984:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff98240507f000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 3740.303822] LustreError: 52984:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 3740.309075] LustreError: 52984:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3740.311977] LustreError: 52984:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3740.327851] Lustre: Unmounted lustre-client
[ 3743.845986] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 20:41:35 (1770687695)
[ 3779.877445] Lustre: lustre-OST0000-osc-ffff9824100b0800: Connection restored to 192.168.203.130@tcp (at 192.168.203.130@tcp)
[ 3779.882134] Lustre: Skipped 5 previous similar messages
[ 3891.679147] Lustre: 2355:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770687828/real 1770687828]  req@ffff982407d94a80 x1856696792511232/t0(0) o4->lustre-OST0000-osc-ffff9824100b0800@192.168.203.130@tcp:6/4 lens 488/448 e 0 to 1 dl 1770687844 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 3891.690095] Lustre: 2355:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 64 previous similar messages
[ 3939.816501] Lustre: lustre-OST0000-osc-ffff9824100b0800: Connection restored to 192.168.203.130@tcp (at 192.168.203.130@tcp)
[ 3939.821084] Lustre: Skipped 9 previous similar messages
[ 4025.725449] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 20:46:17 (1770687977)
[ 4069.342688] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 20:47:01 (1770688021)
[ 4070.679522] LustreError: 55132:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9824100b0800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4070.683916] LustreError: 55132:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 4070.693982] LustreError: 55132:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 4070.696469] LustreError: 55132:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4070.714081] Lustre: Unmounted lustre-client
[ 4080.232225] Lustre: Mounted lustre-client
[ 4082.868977] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 20:47:14 (1770688034)
[ 4082.971842] LustreError: 56111:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4084.016066] LustreError: 56111:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4084.055492] Lustre: *** cfs_fail_loc=305, val=0***
[ 4102.541252] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 20:47:34 (1770688054)
[ 4102.714549] Lustre: Mounted lustre-client
[ 4103.281835] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4113.584602] LustreError: 56791:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4113.588186] LustreError: 56791:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 4113.609604] Lustre: Unmounted lustre-client
[ 4116.718031] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 20:47:48 (1770688068)
[ 4156.301489] LustreError: lustre-MDT0000-mdc-ffff982407fde800: operation mds_reint to node 192.168.203.130@tcp failed: rc = -19
[ 4170.209123] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 192.168.203.130@tcp) was lost; in progress operations using this service will fail
[ 4170.218367] Lustre: Evicted from MGS (at 192.168.203.130@tcp) after server handle changed from 0x2dd273c4f68ee8d0 to 0x2dd273c4f691bdc7
[ 4240.636192] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 20:49:52 (1770688192)
[ 4243.000726] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4251.777556] LustreError: lustre-MDT0000-mdc-ffff982407fde800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4251.790209] Lustre: lustre-MDT0000-mdc-ffff982407fde800: Connection restored to 192.168.203.130@tcp (at 192.168.203.130@tcp)
[ 4251.794295] Lustre: Skipped 9 previous similar messages
[ 4252.130666] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 192.168.203.130@tcp) was lost; in progress operations using this service will fail
[ 4252.137622] Lustre: Evicted from MGS (at 192.168.203.130@tcp) after server handle changed from 0x2dd273c4f691bdc7 to 0x2dd273c4f6964f29
[ 4255.791944] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 20:50:07 (1770688207)
[ 4255.956589] Lustre: Mounted lustre-client
[ 4261.351138] Lustre: lustre-OST0000-osc-ffff982407fde800: Connection to lustre-OST0000 (at 192.168.203.130@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4261.357874] Lustre: Skipped 25 previous similar messages
[ 4261.365498] LustreError: lustre-OST0000-osc-ffff982407fde800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4272.609520] Lustre: 2355:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770688209/real 1770688209]  req@ffff982405dc6a00 x1856696797118720/t0(0) o103->lustre-OST0000-osc-ffff982409858000@192.168.203.130@tcp:17/18 lens 328/224 e 0 to 1 dl 1770688225 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4272.621994] Lustre: 2355:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 71 previous similar messages
[ 4272.973170] LustreError: 59820:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff982409858000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4272.977126] LustreError: 59820:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 4272.984885] LustreError: 59820:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4272.988102] LustreError: 59820:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4273.008125] Lustre: Unmounted lustre-client
[ 4275.532489] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 20:50:27 (1770688227)
[ 4280.543298] LustreError: lustre-MDT0000-mdc-ffff982407fde800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4280.550075] LustreError: 60462:0:(file.c:6174:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4283.445439] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 20:50:35 (1770688235)
[ 4283.514450] LustreError: 61160:0:(recover.c:330:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4288.991172] LustreError: 61160:0:(recover.c:330:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4288.994556] LustreError: 61160:0:(import.c:710:ptlrpc_connect_import_locked()) already connected
[ 4289.334700] LustreError: lustre-MDT0000-mdc-ffff982407fde800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4289.340752] LustreError: 61182:0:(import.c:291:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4289.359342] LustreError: 61187:0:(file.c:6174:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4290.391054] LustreError: 61194:0:(file.c:6174:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4290.394676] LustreError: 61194:0:(file.c:6174:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4292.497227] LustreError: 61216:0:(file.c:6174:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4292.501398] LustreError: 61216:0:(file.c:6174:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4296.715454] LustreError: 61261:0:(file.c:6174:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4296.719129] LustreError: 61261:0:(file.c:6174:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4302.918508] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 20:50:54 (1770688254)
[ 4324.473184] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4325.070406] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4329.379658] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 20:51:21 (1770688281)
[ 4349.827510] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4350.454307] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4355.093334] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 20:51:47 (1770688307)
[ 4400.779283] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4401.395235] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4405.175341] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 20:52:37 (1770688357)
[ 4425.238065] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4425.917643] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4428.605498] LustreError: 66833:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff982407fde800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4428.610724] LustreError: 66833:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 4428.621808] LustreError: 66833:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 4428.624092] LustreError: 66833:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4428.644123] Lustre: Unmounted lustre-client
[ 4453.776175] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4455.328844] Lustre: Mounted lustre-client
[ 4457.970753] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 20:53:29 (1770688409)
[ 4459.021401] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4459.696836] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 20:53:31 (1770688411)
[ 4471.131259] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 20:53:43 (1770688423)
[ 4471.705485] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4472.336442] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 20:53:44 (1770688424)
[ 4472.873623] Lustre: *** cfs_fail_loc=805, val=0***
[ 4472.890880] Lustre: Mounted lustre-client
[ 4474.971894] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4491.213434] LustreError: 2354:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff98240980d000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff982406584200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4491.647094] LustreError: 70160:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff98240980d000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4491.649081] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 192.168.203.130@tcp) was lost; in progress operations using this service will fail
[ 4491.652276] LustreError: 70160:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 4491.659656] LustreError: 70160:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4491.660935] Lustre: Evicted from MGS (at 192.168.203.130@tcp) after server handle changed from 0x2dd273c4f6965939 to 0x2dd273c4f6965f1a
[ 4491.662518] LustreError: 70160:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 4491.681201] Lustre: Unmounted lustre-client
[ 4494.457817] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 20:54:06 (1770688446)
[ 4511.222585] LustreError: 2354:0:(client.c:3418:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff982403574e00 x1856696797185152/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff9824063fd000@192.168.203.130@tcp:12/10 lens 648/608 e 0 to 0 dl 1770688479 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 4514.265795] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4514.912545] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4518.746747] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 20:54:30 (1770688470)
[ 4519.298495] LustreError: lustre-OST0000-osc-ffff9824063fd000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4519.304216] Lustre: 2356:0:(llite_lib.c:4233:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.130@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4519.324207] LustreError: 72154:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-OST0000-osc-ffff9824063fd000: namespace resource [0x240000400:0x5e82:0x0].0x0 (ffff982406584500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4519.330368] LustreError: 72154:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4522.941913] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 20:54:35 (1770688475)
[ 4523.493766] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 4524.073220] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 20:54:36 (1770688476)
[ 4524.601212] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 4525.201839] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 20:54:37 (1770688477)
[ 4525.742538] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 4526.375179] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 20:54:38 (1770688478)
[ 4526.909119] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 4527.490389] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 20:54:39 (1770688479)
[ 4528.015572] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 4528.585813] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 20:54:40 (1770688480)
[ 4529.131813] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 4529.708269] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 20:54:41 (1770688481)
[ 4530.241641] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 4530.827593] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 20:54:42 (1770688482)
[ 4531.362390] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 4531.977245] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 20:54:44 (1770688484)
[ 4532.570244] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 4533.181123] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 20:54:45 (1770688485)
[ 4533.754775] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 4534.401561] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 20:54:46 (1770688486)
[ 4534.965579] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 4535.580437] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 20:54:47 (1770688487)
[ 4536.538791] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 4537.148731] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 20:54:49 (1770688489)
[ 4548.725367] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 20:55:00 (1770688500)
[ 4572.510805] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 20:55:24 (1770688524)
[ 4572.702173] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4579.117428] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 20:55:31 (1770688531)
[ 4579.224729] Lustre: *** cfs_fail_loc=51b, val=4***
[ 4585.661443] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 20:55:37 (1770688537)
[ 4585.778463] Lustre: *** cfs_fail_loc=50f, val=3***
[ 4589.658240] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 20:55:41 (1770688541)
[ 4589.757971] Lustre: *** cfs_fail_loc=50f, val=4***
[ 4593.653801] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 20:55:45 (1770688545)
[ 4593.754573] Lustre: *** cfs_fail_loc=510, val=3***
[ 4598.040499] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 20:55:50 (1770688550)
[ 4598.220413] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4604.701555] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 20:55:56 (1770688556)
[ 4666.081582] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 20:56:58 (1770688618)
[ 4666.189209] LustreError: 82332:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4668.534448] LustreError: lustre-MDT0000-mdc-ffff9824063fd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4668.541368] LustreError: 82346:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff9824063fd000: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 4668.552209] LustreError: 82346:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9824063fd000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff982407f15b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4668.560390] LustreError: 82351:0:(file.c:6174:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4668.564487] LustreError: 82351:0:(file.c:6174:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 4670.247110] LustreError: 82332:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4672.278356] LustreError: 82361:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4674.584510] LustreError: lustre-MDT0000-mdc-ffff9824063fd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4674.590520] LustreError: 82376:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9824063fd000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff982407f15400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4674.596111] LustreError: 82376:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4676.335113] LustreError: 82361:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4682.651704] LustreError: lustre-MDT0000-mdc-ffff9824063fd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4682.787776] LustreError: 82417:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4685.123405] LustreError: lustre-MDT0000-mdc-ffff9824063fd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4685.150728] LustreError: 82436:0:(file.c:6174:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4686.847117] LustreError: 82417:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4686.850448] LustreError: 82417:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff9824063fd000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4686.855949] LustreError: 82417:0:(file.c:249:ll_close_inode_openhandle()) Skipped 4 previous similar messages
[ 4691.451597] LustreError: 82491:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4693.757906] LustreError: lustre-MDT0000-mdc-ffff9824063fd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4693.765593] LustreError: 82505:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9824063fd000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff982405d45c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4693.773971] LustreError: 82505:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4695.511856] LustreError: 82491:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4701.773188] LustreError: lustre-MDT0000-mdc-ffff9824063fd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4701.883478] LustreError: 82547:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 4704.174286] LustreError: lustre-MDT0000-mdc-ffff9824063fd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4704.180730] LustreError: 82561:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9824063fd000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff98240615f400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4704.188131] LustreError: 82561:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4705.943114] LustreError: 82547:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 4708.296367] LustreError: lustre-MDT0000-mdc-ffff9824063fd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4710.031588] LustreError: 82574:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff9824063fd000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4719.158719] LustreError: lustre-MDT0000-mdc-ffff9824063fd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4719.163805] LustreError: 82662:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4719.167378] LustreError: 82662:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 2 previous similar messages
[ 4719.182407] LustreError: 82667:0:(file.c:6174:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4719.185704] LustreError: 82667:0:(file.c:6174:ll_inode_revalidate_fini()) Skipped 53 previous similar messages
[ 4720.687130] LustreError: 82645:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4720.690704] LustreError: 82645:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4720.694778] LustreError: 82645:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff9824063fd000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4725.697542] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 20:57:57 (1770688677)
[ 4746.271114] LustreError: 77796:0:(ldlm_lockd.c:2882:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 4746.274520] LustreError: 77796:0:(ldlm_lockd.c:2882:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 4748.642480] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 20:58:20 (1770688700)
[ 4792.987711] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 20:59:05 (1770688745)
[ 4851.873420] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 21:00:04 (1770688804)
[ 4877.804405] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 21:00:29 (1770688829)
[ 4877.919277] Lustre: Mounted lustre-client
[ 4998.676322] LustreError: 86363:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9824103cc800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4998.682324] LustreError: 86363:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 4998.686528] LustreError: 86363:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4998.689197] LustreError: 86363:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4998.708153] Lustre: Unmounted lustre-client
[ 5001.179227] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 21:02:33 (1770688953)
[ 5001.398916] LustreError: 86979:0:(osc_request.c:2944:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 4000ms
[ 5001.402275] LustreError: 86979:0:(osc_request.c:2944:osc_build_rpc()) Skipped 1 previous similar message
[ 5003.300539] LustreError: lustre-OST0000-osc-ffff9824063fd000: operation ost_statfs to node 192.168.203.130@tcp failed: rc = -107
[ 5003.303592] LustreError: Skipped 15 previous similar messages
[ 5003.305251] Lustre: lustre-OST0000-osc-ffff9824063fd000: Connection to lustre-OST0000 (at 192.168.203.130@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5003.310974] Lustre: Skipped 28 previous similar messages
[ 5003.316460] LustreError: lustre-OST0000-osc-ffff9824063fd000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5003.321356] LustreError: 87066:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-OST0000-osc-ffff9824063fd000: namespace resource [0x240000400:0x5ead:0x0].0x0 (ffff982407f15f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5003.327676] LustreError: 87066:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5005.463103] LustreError: 86979:0:(osc_request.c:2944:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 5005.466595] Lustre: 2358:0:(llite_lib.c:4233:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.130@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5005.474047] Lustre: lustre-OST0000-osc-ffff9824063fd000: Connection restored to 192.168.203.130@tcp (at 192.168.203.130@tcp)
[ 5005.478023] Lustre: Skipped 30 previous similar messages
[ 5007.881506] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 21:02:39 (1770688959)
[ 5065.695129] Lustre: 87645:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770688962/real 1770688962]  req@ffff982403575c00 x1856696797340288/t0(0) o101->lustre-MDT0000-mdc-ffff9824063fd000@192.168.203.130@tcp:12/10 lens 328/344 e 0 to 1 dl 1770689018 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5065.701950] Lustre: 87645:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 5067.886148] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 21:03:40 (1770689020)
[ 5068.395818] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5068.980221] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 21:03:41 (1770689021)
[ 5128.265969] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5128.821635] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 21:04:40 (1770689080)
[ 5185.075707] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 21:05:37 (1770689137)
[ 5185.905552] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5186.490882] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 21:05:38 (1770689138)
[ 5187.026411] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5187.721897] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 21:05:39 (1770689139)
[ 5197.818420] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 21:05:49 (1770689149)
[ 5201.210901] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5220.322590] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 192.168.203.130@tcp) was lost; in progress operations using this service will fail
[ 5220.326519] LustreError: Skipped 2 previous similar messages
[ 5220.332023] Lustre: Evicted from MGS (at 192.168.203.130@tcp) after server handle changed from 0x2dd273c4f6966732 to 0x2dd273c4f6968267
[ 5220.336380] Lustre: Skipped 2 previous similar messages
[ 5220.555310] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5221.057532] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5225.175895] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 21:06:17 (1770689177)
[ 5226.032715] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5226.612274] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 21:06:18 (1770689178)
[ 5234.693096] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 21:06:26 (1770689186)
[ 5252.765305] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 21:06:44 (1770689204)
[ 5273.975361] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5275.279397] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5358.527713] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5358.976120] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5377.489615] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5377.971651] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5430.124404] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 21:09:42 (1770689382)
[ 5455.361697] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5456.175766] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5637.753208] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 21:13:09 (1770689589)
[ 5664.737595] Lustre: lustre-MDT0000-mdc-ffff9824063fd000: Connection to lustre-MDT0000 (at 192.168.203.130@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5664.742339] Lustre: Skipped 10 previous similar messages
[ 5664.744684] Lustre: MGC192.168.203.130@tcp: Connection restored to 192.168.203.130@tcp (at 192.168.203.130@tcp)
[ 5664.747377] Lustre: Skipped 13 previous similar messages
[ 5685.278579] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 21:13:57 (1770689637)
[ 5685.752624] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 5686.299180] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 21:13:58 (1770689638)
[ 5686.975695] LustreError: lustre-MDT0000-mdc-ffff9824063fd000: operation ldlm_enqueue to node 192.168.203.130@tcp failed: rc = -107
[ 5686.980127] LustreError: Skipped 13 previous similar messages
[ 5686.985032] LustreError: lustre-MDT0000-mdc-ffff9824063fd000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5689.388451] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 21:14:01 (1770689641)
[ 5855.394432] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.203.130@tcp) after server handle changed from 0x2dd273c4f6a640b5 to 0x2dd273c4f6a641b1
[ 5855.398951] Lustre: Skipped 5 previous similar messages
[ 5855.400680] LustreError: lustre-OST0000-osc-ffff9824063fd000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5855.926140] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 21:16:48 (1770689808)
[ 5878.239237] Lustre: 2355:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770689810/real 1770689810]  req@ffff982415396680 x1856696815008896/t0(0) o4->lustre-OST0000-osc-ffff9824063fd000@192.168.203.130@tcp:6/4 lens 4584/448 e 0 to 1 dl 1770689830 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 5878.249819] Lustre: 2355:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 5889.206370] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 21:17:21 (1770689841)
[ 5889.657149] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 5890.143270] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 21:17:22 (1770689842)
[ 5890.567905] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 5891.036203] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 21:17:23 (1770689843)
[ 5915.156826] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 21:17:47 (1770689867)
[ 5942.753087] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 192.168.203.130@tcp) was lost; in progress operations using this service will fail
[ 5942.756706] LustreError: Skipped 5 previous similar messages
[ 5946.656526] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 21:18:18 (1770689898)
[ 5947.102532] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 5947.597584] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 21:18:19 (1770689899)
[ 5948.056917] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 5948.575848] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 21:18:20 (1770689900)
[ 5951.232089] LustreError: 105774:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9824063fd000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5951.236224] LustreError: 105774:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 5951.243089] LustreError: 105774:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 5951.246218] LustreError: 105774:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 5951.274313] Lustre: Unmounted lustre-client
[ 5951.652941] Lustre: Mounted lustre-client
[ 5952.547221] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5968.535585] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: total 1805