[ 1797.300303] LustreError: lustre-OST0000-osc-ffff8f8511b76800: operation ost_sync to node 192.168.206.155@tcp failed: rc = -107
[ 1797.321949] LustreError: lustre-OST0000-osc-ffff8f8511b76800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1797.348389] Lustre: 2355:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.155@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1805.222262] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 22:22:49 (1787365369)
[ 1807.191224] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1809.074561] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 22:22:54 (1787365374)
[ 1810.906698] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1812.975094] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 22:22:57 (1787365377)
[ 1816.291903] LustreError: lustre-MDT0000-mdc-ffff8f8511b76800: operation mds_reint to node 192.168.206.155@tcp failed: rc = -19
[ 1834.494492] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 192.168.206.155@tcp) was lost; in progress operations using this service will fail
[ 1834.524229] Lustre: Evicted from MGS (at 192.168.206.155@tcp) after server handle changed from 0xe765540838840703 to 0xe7655408388421b3
[ 1936.081135] LustreError: lustre-MDT0000-mdc-ffff8f8511b76800: operation ldlm_enqueue to node 192.168.206.155@tcp failed: rc = -19
[ 1936.093088] LustreError: Skipped 1 previous similar message
[ 1953.248230] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 192.168.206.155@tcp) was lost; in progress operations using this service will fail
[ 1953.266501] Lustre: Evicted from MGS (at 192.168.206.155@tcp) after server handle changed from 0xe7655408388421b3 to 0xe76554083886810f
[ 1964.522767] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 22:25:29 (1787365529)
[ 1965.004466] Lustre: *** cfs_fail_loc=305, val=0***
[ 1965.006883] Lustre: Skipped 2 previous similar messages
[ 2004.448667] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 192.168.206.155@tcp) was lost; in progress operations using this service will fail
[ 2014.953458] Lustre: Evicted from MGS (at 192.168.206.155@tcp) after server handle changed from 0xe76554083886810f to 0xe76554083886984d
[ 2019.014859] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2020.937312] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2029.263061] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 22:26:34 (1787365594)
[ 2040.854075] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 192.168.206.155@tcp) was lost; in progress operations using this service will fail
[ 2040.891604] Lustre: Evicted from MGS (at 192.168.206.155@tcp) after server handle changed from 0xe76554083886984d to 0xe7655408388699f8
[ 2040.937227] LustreError: lustre-MDT0000-mdc-ffff8f8511b76800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2061.171913] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 22:27:05 (1787365625)
[ 2066.411794] Lustre: lustre-OST0000-osc-ffff8f8511b76800: Connection to lustre-OST0000 (at 192.168.206.155@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2066.426842] Lustre: Skipped 13 previous similar messages
[ 2090.298717] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 22:27:35 (1787365655)
[ 2103.266331] LustreError: lustre-MDT0000-mdc-ffff8f8511b76800: operation ldlm_enqueue to node 192.168.206.155@tcp failed: rc = -19
[ 2103.276052] LustreError: Skipped 1 previous similar message
[ 2122.727609] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 192.168.206.155@tcp) was lost; in progress operations using this service will fail
[ 2122.749435] Lustre: Evicted from MGS (at 192.168.206.155@tcp) after server handle changed from 0xe7655408388699f8 to 0xe76554083886e9ec
[ 2122.774644] Lustre: MGC192.168.206.155@tcp: Connection restored to 192.168.206.155@tcp (at 192.168.206.155@tcp)
[ 2122.796126] Lustre: Skipped 17 previous similar messages
[ 2136.024830] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2137.814039] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2220.000418] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 192.168.206.155@tcp) was lost; in progress operations using this service will fail
[ 2230.207617] Lustre: Evicted from MGS (at 192.168.206.155@tcp) after server handle changed from 0xe76554083886e9ec to 0xe76554083888cf54
[ 2246.418561] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2248.431847] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2313.845506] LustreError: lustre-MDT0000-mdc-ffff8f8511b76800: operation mds_close to node 192.168.206.155@tcp failed: rc = -19
[ 2313.864643] LustreError: Skipped 2 previous similar messages
[ 2332.639179] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787365882/real 1787365882]  req@ffff8f8509cd5180 x1874186589593472/t0(0) o400->MGC192.168.206.155@tcp@192.168.206.155@tcp:26/25 lens 224/224 e 0 to 1 dl 1787365898 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2332.670923] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 2332.688304] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 192.168.206.155@tcp) was lost; in progress operations using this service will fail
[ 2341.868598] Lustre: Evicted from MGS (at 192.168.206.155@tcp) after server handle changed from 0xe76554083888cf54 to 0xe7655408388a8c83
[ 2359.102253] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2361.459257] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2390.977933] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 22:32:36 (1787365956)
[ 2413.535335] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 192.168.206.155@tcp) was lost; in progress operations using this service will fail
[ 2423.726962] Lustre: Evicted from MGS (at 192.168.206.155@tcp) after server handle changed from 0xe7655408388a8c83 to 0xe7655408388b5d64
[ 2433.544911] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2458.800213] Lustre: 15932:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.155@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2474.908793] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2509.446807] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2560.069480] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2584.219499] LustreError: lustre-MDT0000-mdc-ffff8f8511b76800: operation ldlm_enqueue to node 192.168.206.155@tcp failed: rc = -19
[ 2584.229079] LustreError: Skipped 5 previous similar messages
[ 2602.986829] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 192.168.206.155@tcp) was lost; in progress operations using this service will fail
[ 2603.003780] LustreError: Skipped 3 previous similar messages
[ 2603.024465] Lustre: Evicted from MGS (at 192.168.206.155@tcp) after server handle changed from 0xe7655408388bb8e2 to 0xe7655408388bf191
[ 2603.030959] Lustre: Skipped 3 previous similar messages
[ 2604.931410] Lustre: 15932:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.155@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2624.649375] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2681.282630] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2714.457673] Lustre: lustre-MDT0000-mdc-ffff8f8511b76800: Connection to lustre-MDT0000 (at 192.168.206.155@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2714.477539] Lustre: Skipped 8 previous similar messages
[ 2734.092931] Lustre: MGC192.168.206.155@tcp: Connection restored to 192.168.206.155@tcp (at 192.168.206.155@tcp)
[ 2734.104482] Lustre: Skipped 16 previous similar messages
[ 2764.845834] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 22:38:50 (1787366330)
[ 2812.617480] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2814.473863] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3110.744478] LustreError: lustre-OST0000-osc-ffff8f8511b76800: operation ldlm_enqueue to node 192.168.206.155@tcp failed: rc = -107
[ 3110.762617] LustreError: Skipped 4 previous similar messages
[ 3144.631684] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3146.273641] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3441.625794] Lustre: lustre-OST0000-osc-ffff8f8511b76800: Connection to lustre-OST0000 (at 192.168.206.155@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3441.637663] Lustre: Skipped 2 previous similar messages
[ 3481.922707] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3483.661272] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3737.140217] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 22:55:02 (1787367302)
[ 3753.951817] Lustre: 50593:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787367304/real 1787367304]  req@ffff8f8510d2c380 x1874186612068736/t0(0) o101->lustre-MDT0000-mdc-ffff8f8511b76800@192.168.206.155@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787367320 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3753.997242] Lustre: 50593:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 3754.038966] Lustre: lustre-MDT0000-mdc-ffff8f8511b76800: Connection restored to 192.168.206.155@tcp (at 192.168.206.155@tcp)
[ 3754.046677] Lustre: Skipped 1 previous similar message
[ 3761.188929] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 22:55:26 (1787367326)
[ 3787.207826] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 22:55:52 (1787367352)
[ 3812.795734] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 22:56:17 (1787367377)
[ 3813.270385] Lustre: Mounted lustre-client - version 2.17.57_47_gae0d133
[ 3843.999142] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787367394/real 1787367394]  req@ffff8f8510c7fb80 x1874186612087168/t0(0) o400->lustre-MDT0000-mdc-ffff8f8511b76800@192.168.206.155@tcp:12/10 lens 224/224 e 0 to 1 dl 1787367410 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3844.051298] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3844.061559] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 192.168.206.155@tcp) was lost; in progress operations using this service will fail
[ 3844.071818] LustreError: Skipped 2 previous similar messages
[ 3854.332642] Lustre: Evicted from MGS (at 192.168.206.155@tcp) after server handle changed from 0xe7655408388d2def to 0xe765540838a4f3dc
[ 3854.344254] Lustre: Skipped 2 previous similar messages
[ 3859.865908] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3861.492264] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3863.457314] Lustre: Unmounted lustre-client
[ 3869.802989] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 22:57:14 (1787367434)
[ 4004.832045] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787367555/real 1787367555]  req@ffff8f8510c7dc00 x1874186612118784/t0(0) o4->lustre-OST0000-osc-ffff8f8511b76800@192.168.206.155@tcp:6/4 lens 488/448 e 0 to 1 dl 1787367571 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4004.873560] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 71 previous similar messages
[ 4052.961201] Lustre: lustre-OST0000-osc-ffff8f8511b76800: Connection to lustre-OST0000 (at 192.168.206.155@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4052.980986] Lustre: Skipped 15 previous similar messages
[ 4159.206156] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 23:02:04 (1787367724)
[ 4206.434912] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 23:02:51 (1787367771)
[ 4209.667645] Lustre: Unmounted lustre-client
[ 4231.194444] Lustre: Mounted lustre-client - version 2.17.57_47_gae0d133
[ 4237.422692] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 23:03:22 (1787367802)
[ 4237.890052] LustreError: 56522:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4238.944566] LustreError: 56522:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4239.044369] Lustre: *** cfs_fail_loc=305, val=0***
[ 4261.992462] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 23:03:47 (1787367827)
[ 4262.426789] Lustre: Mounted lustre-client - version 2.17.57_47_gae0d133
[ 4264.308987] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4274.647765] Lustre: Unmounted lustre-client
[ 4280.181288] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 23:04:05 (1787367845)
[ 4358.293420] LustreError: lustre-MDT0000-mdc-ffff8f8502da5800: operation mds_reint to node 192.168.206.155@tcp failed: rc = -19
[ 4358.301630] LustreError: Skipped 1 previous similar message
[ 4381.162705] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 192.168.206.155@tcp) was lost; in progress operations using this service will fail
[ 4381.186550] Lustre: Evicted from MGS (at 192.168.206.155@tcp) after server handle changed from 0xe765540838a50f8f to 0xe765540838a7c55a
[ 4381.208693] Lustre: MGC192.168.206.155@tcp: Connection restored to 192.168.206.155@tcp (at 192.168.206.155@tcp)
[ 4381.218928] Lustre: Skipped 23 previous similar messages
[ 4517.052627] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 23:08:01 (1787368081)
[ 4522.099429] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4534.772889] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 192.168.206.155@tcp) was lost; in progress operations using this service will fail
[ 4534.800380] Lustre: Evicted from MGS (at 192.168.206.155@tcp) after server handle changed from 0xe765540838a7c55a to 0xe765540838ac8f41
[ 4534.817102] LustreError: lustre-MDT0000-mdc-ffff8f8502da5800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4547.542489] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 23:08:32 (1787368112)
[ 4548.197442] Lustre: Mounted lustre-client - version 2.17.57_47_gae0d133
[ 4555.558894] LustreError: lustre-OST0000-osc-ffff8f8502da5800: operation ldlm_enqueue to node 192.168.206.155@tcp failed: rc = -107
[ 4555.574593] LustreError: lustre-OST0000-osc-ffff8f8502da5800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4555.594802] LustreError: 60235:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8f8502da5800: namespace resource [0x240000400:0x4383:0x0].0x0 (ffff8f8505f5a300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4564.959212] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787368115/real 1787368115]  req@ffff8f8509a6ea00 x1874186616773376/t0(0) o103->lustre-OST0000-osc-ffff8f85079b5000@192.168.206.155@tcp:17/18 lens 328/224 e 0 to 1 dl 1787368131 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4564.982982] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 76 previous similar messages
[ 4566.949661] Lustre: Unmounted lustre-client
[ 4573.531912] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 23:08:58 (1787368138)
[ 4580.709625] LustreError: lustre-MDT0000-mdc-ffff8f8502da5800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4580.735624] LustreError: 60905:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4588.971771] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 23:09:13 (1787368153)
[ 4589.183408] LustreError: 61610:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4593.000254] LustreError: lustre-MDT0000-mdc-ffff8f8502da5800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4593.022520] LustreError: 61627:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4593.027500] LustreError: 61610:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1168
[ 4593.040508] LustreError: 61610:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4593.065780] LustreError: 61632:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4594.150503] LustreError: 61638:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4594.160704] LustreError: 61638:0:(file.c:6170:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4596.470965] LustreError: 61661:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4596.483421] LustreError: 61661:0:(file.c:6170:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4601.066986] LustreError: 61706:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4601.085732] LustreError: 61706:0:(file.c:6170:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4610.067851] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 23:09:35 (1787368175)
[ 4644.015175] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4645.219827] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4654.777470] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 23:10:19 (1787368219)
[ 4659.195520] Lustre: lustre-OST0000-osc-ffff8f8502da5800: Connection to lustre-OST0000 (at 192.168.206.155@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4659.223227] Lustre: Skipped 17 previous similar messages
[ 4696.711720] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4698.403672] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4707.928636] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 23:11:13 (1787368273)
[ 4770.238167] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4771.478334] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4779.351761] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 23:12:24 (1787368344)
[ 4814.405808] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4815.523429] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4821.065086] Lustre: Unmounted lustre-client
[ 4876.670931] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4879.356724] Lustre: Mounted lustre-client - version 2.17.57_47_gae0d133
[ 4885.978543] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 23:14:10 (1787368450)
[ 4888.746309] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4890.666729] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 23:14:15 (1787368455)
[ 4920.310925] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 23:14:45 (1787368485)
[ 4921.950743] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4923.780342] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 23:14:48 (1787368488)
[ 4925.155414] Lustre: *** cfs_fail_loc=805, val=0***
[ 4925.219592] Lustre: Mounted lustre-client - version 2.17.57_47_gae0d133
[ 4929.824996] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4951.007302] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 192.168.206.155@tcp) was lost; in progress operations using this service will fail
[ 4961.259660] Lustre: Evicted from MGS (at 192.168.206.155@tcp) after server handle changed from 0xe765540838ac9904 to 0xe765540838ac9cb5
[ 4961.281531] LustreError: 2352:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8f8520342800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8f8520f62600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4963.859375] Lustre: Unmounted lustre-client
[ 4970.610634] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 23:15:35 (1787368535)
[ 4991.994377] Lustre: MGC192.168.206.155@tcp: Connection restored to 192.168.206.155@tcp (at 192.168.206.155@tcp)
[ 4992.004508] Lustre: Skipped 15 previous similar messages
[ 4997.144931] LustreError: 2352:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8f8506c3fb80 x1874186616851584/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff8f8508ae4000@192.168.206.155@tcp:12/10 lens 664/608 e 0 to 0 dl 1787368579 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5004.442420] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5005.644935] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5013.575504] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 23:16:18 (1787368578)
[ 5014.849625] LustreError: lustre-OST0000-osc-ffff8f8508ae4000: operation ost_write to node 192.168.206.155@tcp failed: rc = -107
[ 5014.858760] LustreError: Skipped 2 previous similar messages
[ 5014.881248] LustreError: lustre-OST0000-osc-ffff8f8508ae4000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5014.894689] Lustre: 2356:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.155@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5014.918384] LustreError: 72809:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8f8508ae4000: namespace resource [0x240000400:0x43e2:0x0].0x0 (ffff8f8505f5a300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5014.941914] LustreError: 72809:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5023.303985] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 23:16:27 (1787368587)
[ 5025.262506] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5027.296959] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 23:16:31 (1787368591)
[ 5028.653717] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5030.451385] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 23:16:35 (1787368595)
[ 5031.973847] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5034.169685] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 23:16:38 (1787368598)
[ 5035.616540] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5037.889296] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 23:16:42 (1787368602)
[ 5039.390961] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5041.090563] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 23:16:45 (1787368605)
[ 5042.405953] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5044.115670] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 23:16:49 (1787368609)
[ 5045.363521] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5046.983674] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 23:16:51 (1787368611)
[ 5048.330875] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5049.948332] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 23:16:54 (1787368614)
[ 5051.146590] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5053.451913] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 23:16:57 (1787368617)
[ 5054.945297] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5056.618822] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 23:17:01 (1787368621)
[ 5057.974909] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5059.754058] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 23:17:04 (1787368624)
[ 5062.386095] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5063.798120] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 23:17:08 (1787368628)
[ 5094.883926] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 23:17:39 (1787368659)
[ 5125.416146] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 23:18:09 (1787368689)
[ 5125.905818] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5135.866475] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 23:18:20 (1787368700)
[ 5136.373070] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5145.948540] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 23:18:31 (1787368711)
[ 5146.337472] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5153.810178] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 23:18:38 (1787368718)
[ 5154.020537] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5161.158737] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 23:18:46 (1787368726)
[ 5161.424260] Lustre: *** cfs_fail_loc=510, val=3***
[ 5169.014314] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 23:18:54 (1787368734)
[ 5169.656715] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5179.551561] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 23:19:04 (1787368744)
[ 5179.816036] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5239.775671] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787368750/real 1787368750]  req@ffff8f8510c85f80 x1874186616896128/t0(0) o3->lustre-OST0001-osc-ffff8f8508ae4000@192.168.206.155@tcp:6/4 lens 488/4536 e 0 to 1 dl 1787368805 ref 2 fl Rpc:RXQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 5239.806020] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 16 previous similar messages
[ 5245.427427] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 23:20:10 (1787368810)
[ 5245.751605] LustreError: 83015:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5248.685040] LustreError: lustre-MDT0000-mdc-ffff8f8508ae4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5248.707607] LustreError: 83029:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8f8508ae4000: inode [0x20000a042:0xe:0x0] mdc close failed: rc = -108
[ 5248.725220] LustreError: 83029:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8f8508ae4000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8f8520f62800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5249.831276] LustreError: 83015:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5251.943267] LustreError: 83045:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5254.908861] LustreError: lustre-MDT0000-mdc-ffff8f8508ae4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5254.923495] LustreError: 83059:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8f8508ae4000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8f8520f62b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5254.936823] LustreError: 83059:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5256.015789] LustreError: 83045:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5262.966630] Lustre: lustre-MDT0000-mdc-ffff8f8508ae4000: Connection to lustre-MDT0000 (at 192.168.206.155@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5262.979310] Lustre: Skipped 11 previous similar messages
[ 5262.991797] LustreError: lustre-MDT0000-mdc-ffff8f8508ae4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5263.312758] LustreError: 83100:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5266.288896] LustreError: lustre-MDT0000-mdc-ffff8f8508ae4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5266.390484] LustreError: 83120:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5266.408054] LustreError: 83120:0:(file.c:6170:ll_inode_revalidate_fini()) Skipped 14 previous similar messages
[ 5267.383481] LustreError: 83100:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5267.393564] LustreError: 83100:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8f8508ae4000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5267.498925] LustreError: 83126:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5267.514616] LustreError: 83126:0:(file.c:6170:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5269.791972] LustreError: 83148:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5269.807288] LustreError: 83148:0:(file.c:6170:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5273.446516] LustreError: 83175:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5276.220193] LustreError: lustre-MDT0000-mdc-ffff8f8508ae4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5276.233174] LustreError: 83189:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8f8508ae4000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8f8505eef300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5276.279675] LustreError: 83189:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5277.527143] LustreError: 83175:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5284.454527] LustreError: lustre-MDT0000-mdc-ffff8f8508ae4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5284.919053] LustreError: 83230:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5287.876625] LustreError: lustre-MDT0000-mdc-ffff8f8508ae4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5288.919385] LustreError: 83230:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5292.128847] LustreError: lustre-MDT0000-mdc-ffff8f8508ae4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5292.223870] LustreError: 83277:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5292.247630] LustreError: 83277:0:(file.c:6170:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5293.144535] LustreError: 83258:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8f8508ae4000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5301.426488] LustreError: 83329:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5301.441806] LustreError: 83329:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5305.021067] LustreError: lustre-MDT0000-mdc-ffff8f8508ae4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5305.103185] LustreError: 83351:0:(file.c:6170:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5305.118190] LustreError: 83351:0:(file.c:6170:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5305.431782] LustreError: 83329:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5305.445491] LustreError: 83329:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5305.469818] LustreError: 83329:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8f8508ae4000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5315.441991] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 23:21:20 (1787368880)
[ 5333.599441] LustreError: 68269:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5333.616882] LustreError: 68269:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5337.695287] LustreError: 68269:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5337.705996] LustreError: 68269:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5343.371812] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 23:21:48 (1787368908)
[ 5393.039934] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 23:22:37 (1787368957)
[ 5456.381984] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 23:23:41 (1787369021)
[ 5488.821220] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 23:24:13 (1787369053)
[ 5489.159614] Lustre: Mounted lustre-client - version 2.17.57_47_gae0d133
[ 5613.132921] Lustre: Unmounted lustre-client
[ 5620.689464] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 23:26:25 (1787369185)
[ 5621.275105] LustreError: 87657:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5624.154520] LustreError: lustre-OST0000-osc-ffff8f8508ae4000: operation ldlm_enqueue to node 192.168.206.155@tcp failed: rc = -107
[ 5624.166369] LustreError: Skipped 9 previous similar messages
[ 5624.184915] LustreError: lustre-OST0000-osc-ffff8f8508ae4000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5624.217349] LustreError: 87739:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8f8508ae4000: namespace resource [0x240000400:0x440d:0x0].0x0 (ffff8f8505eefa00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5624.240280] LustreError: 87739:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 3 previous similar messages
[ 5624.527107] LustreError: 87657:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5624.533683] Lustre: 2356:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.155@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5631.093432] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 23:26:36 (1787369196)
[ 5688.299640] Lustre: lustre-MDT0000-mdc-ffff8f8508ae4000: Connection restored to 192.168.206.155@tcp (at 192.168.206.155@tcp)
[ 5688.314813] Lustre: Skipped 15 previous similar messages
[ 5695.143118] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 23:27:40 (1787369260)
[ 5697.349583] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5699.114382] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 23:27:43 (1787369263)
[ 5762.679874] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5765.233535] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 23:28:49 (1787369329)
[ 5824.473935] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 23:29:48 (1787369388)
[ 5826.994242] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5828.439323] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 23:29:53 (1787369393)
[ 5830.084100] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5831.799710] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 23:29:56 (1787369396)
[ 5866.759179] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 23:30:31 (1787369431)
[ 5878.295335] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5905.183142] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787369455/real 1787369455]  req@ffff8f8520fcb100 x1874186617058304/t0(0) o400->MGC192.168.206.155@tcp@192.168.206.155@tcp:26/25 lens 224/224 e 0 to 1 dl 1787369471 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5905.243409] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 5905.254433] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 192.168.206.155@tcp) was lost; in progress operations using this service will fail
[ 5905.276422] LustreError: Skipped 2 previous similar messages
[ 5911.459121] Lustre: lustre-MDT0000-mdc-ffff8f8508ae4000: Connection to lustre-MDT0000 (at 192.168.206.155@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5911.483474] Lustre: Skipped 13 previous similar messages
[ 5915.633442] Lustre: Evicted from MGS (at 192.168.206.155@tcp) after server handle changed from 0xe765540838aca766 to 0xe765540838acc113
[ 5915.642987] Lustre: Skipped 2 previous similar messages
[ 5928.641958] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5930.671535] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5941.260934] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 23:31:45 (1787369505)
[ 5944.034799] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5946.033200] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 23:31:50 (1787369510)
[ 5961.699127] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 192.168.206.155@tcp) was lost; in progress operations using this service will fail
[ 5961.728955] Lustre: Evicted from MGS (at 192.168.206.155@tcp) after server handle changed from 0xe765540838acc113 to 0xe765540838acc44d
[ 5971.875655] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 23:32:16 (1787369536)
[ 6021.746607] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 23:33:06 (1787369586)
[ 6079.323642] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6083.788711] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6171.622609] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 192.168.206.155@tcp) was lost; in progress operations using this service will fail
[ 6171.637656] LustreError: Skipped 1 previous similar message
[ 6171.652042] Lustre: Evicted from MGS (at 192.168.206.155@tcp) after server handle changed from 0xe765540838acc7cd to 0xe765540838acd4a0
[ 6171.669506] Lustre: Skipped 1 previous similar message
[ 6173.167557] Lustre: 68272:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.155@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6195.927633] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6197.203377] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6236.158549] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6237.773488] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6391.948826] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 23:39:15 (1787369955)
[ 6402.925178] LustreError: lustre-OST0000-osc-ffff8f8508ae4000: operation ost_setattr to node 192.168.206.155@tcp failed: rc = -19
[ 6402.938494] LustreError: Skipped 91 previous similar messages
[ 6455.633841] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6460.107267] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6723.689530] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 23:44:48 (1787370288)
[ 6846.958168] Lustre: lustre-MDT0000-mdc-ffff8f8508ae4000: Connection to lustre-MDT0000 (at 192.168.206.155@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6846.989177] Lustre: Skipped 6 previous similar messages
[ 6862.317164] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 192.168.206.155@tcp) was lost; in progress operations using this service will fail
[ 6862.333077] LustreError: Skipped 1 previous similar message
[ 6862.350453] Lustre: Evicted from MGS (at 192.168.206.155@tcp) after server handle changed from 0xe765540838acd8cf to 0xe765540838bc7bc5
[ 6862.357796] Lustre: Skipped 1 previous similar message
[ 6862.363989] Lustre: MGC192.168.206.155@tcp: Connection restored to 192.168.206.155@tcp (at 192.168.206.155@tcp)
[ 6862.369438] Lustre: Skipped 12 previous similar messages
[ 6895.964917] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 23:47:40 (1787370460)
[ 6897.443353] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6898.951633] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 23:47:43 (1787370463)
[ 6901.139973] LustreError: lustre-MDT0000-mdc-ffff8f8508ae4000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6908.656532] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 23:47:53 (1787370473)
[ 7075.821528] LustreError: lustre-OST0000-osc-ffff8f8508ae4000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7077.976942] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 23:50:42 (1787370642)
[ 7102.431544] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787370648/real 1787370649]  req@ffff8f8504487800 x1874186634767744/t0(0) o4->lustre-OST0000-osc-ffff8f8508ae4000@192.168.206.155@tcp:6/4 lens 4584/448 e 0 to 1 dl 1787370668 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7102.476029] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 7121.009414] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 23:51:25 (1787370685)
[ 7122.539501] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7124.441630] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 23:51:29 (1787370689)
[ 7125.879867] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7128.029363] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 23:51:32 (1787370692)
[ 7162.223542] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 23:52:06 (1787370726)
[ 7204.840329] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 192.168.206.155@tcp) was lost; in progress operations using this service will fail
[ 7204.864197] Lustre: Evicted from MGS (at 192.168.206.155@tcp) after server handle changed from 0xe765540838bc7bc5 to 0xe765540838bca41a
[ 7204.877973] Lustre: Skipped 1 previous similar message
[ 7218.254716] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 23:53:03 (1787370783)
[ 7219.852592] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7221.579363] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 23:53:06 (1787370786)
[ 7223.130529] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7225.171301] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 23:53:09 (1787370789)
[ 7234.903688] Lustre: Unmounted lustre-client
[ 7236.247082] Lustre: Mounted lustre-client - version 2.17.57_47_gae0d133
[ 7238.825038] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7269.190529] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2