[ 1857.517979] LustreError: lustre-OST0000-osc-ffff9666919fb800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1857.532446] Lustre: 2376:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.129@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1864.414120] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 02:33:27 (1788590007)
[ 1866.764872] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1868.198813] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 02:33:31 (1788590011)
[ 1869.495491] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1870.817958] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 02:33:34 (1788590014)
[ 1874.443736] LustreError: lustre-MDT0000-mdc-ffff9666919fb800: operation mds_close to node 192.168.206.129@tcp failed: rc = -19
[ 1874.470790] LustreError: Skipped 3 previous similar messages
[ 1895.383369] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 192.168.206.129@tcp) was lost; in progress operations using this service will fail
[ 1895.404128] Lustre: Evicted from MGS (at 192.168.206.129@tcp) after server handle changed from 0xa2b54e63d5335edf to 0xa2b54e63d5337766
[ 1997.618321] LustreError: lustre-MDT0000-mdc-ffff9666919fb800: operation ldlm_enqueue to node 192.168.206.129@tcp failed: rc = -19
[ 1997.646933] LustreError: Skipped 1 previous similar message
[ 2016.234786] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 192.168.206.129@tcp) was lost; in progress operations using this service will fail
[ 2016.262203] Lustre: Evicted from MGS (at 192.168.206.129@tcp) after server handle changed from 0xa2b54e63d5337766 to 0xa2b54e63d53528b7
[ 2031.956929] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 02:36:14 (1788590174)
[ 2032.777064] Lustre: *** cfs_fail_loc=305, val=0***
[ 2032.782833] Lustre: Skipped 2 previous similar messages
[ 2073.570259] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 192.168.206.129@tcp) was lost; in progress operations using this service will fail
[ 2073.636104] Lustre: Evicted from MGS (at 192.168.206.129@tcp) after server handle changed from 0xa2b54e63d53528b7 to 0xa2b54e63d5355374
[ 2088.921395] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2090.510469] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2099.823741] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 02:37:22 (1788590242)
[ 2114.533363] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 192.168.206.129@tcp) was lost; in progress operations using this service will fail
[ 2114.550072] Lustre: Evicted from MGS (at 192.168.206.129@tcp) after server handle changed from 0xa2b54e63d5355374 to 0xa2b54e63d535592b
[ 2114.609359] LustreError: lustre-MDT0000-mdc-ffff9666919fb800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2135.814314] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 02:37:58 (1788590278)
[ 2140.141093] Lustre: lustre-OST0000-osc-ffff9666919fb800: Connection to lustre-OST0000 (at 192.168.206.129@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2140.156907] Lustre: Skipped 13 previous similar messages
[ 2150.387325] LustreError: lustre-OST0000-osc-ffff9666919fb800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2166.364436] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 02:38:29 (1788590309)
[ 2179.204531] LustreError: lustre-MDT0000-mdc-ffff9666919fb800: operation mds_reint to node 192.168.206.129@tcp failed: rc = -19
[ 2197.471251] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 192.168.206.129@tcp) was lost; in progress operations using this service will fail
[ 2197.496497] Lustre: Evicted from MGS (at 192.168.206.129@tcp) after server handle changed from 0xa2b54e63d535592b to 0xa2b54e63d535b08f
[ 2197.519239] Lustre: MGC192.168.206.129@tcp: Connection restored to 192.168.206.129@tcp (at 192.168.206.129@tcp)
[ 2197.526395] Lustre: Skipped 17 previous similar messages
[ 2199.084773] Lustre: 15937:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.129@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2221.965166] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2224.010590] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2308.063659] Lustre: 2374:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788590436/real 1788590436]  req@ffff966686158e00 x1875470697818112/t0(0) o400->MGC192.168.206.129@tcp@192.168.206.129@tcp:26/25 lens 224/224 e 0 to 1 dl 1788590452 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2308.096905] Lustre: 2374:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 2308.106289] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 192.168.206.129@tcp) was lost; in progress operations using this service will fail
[ 2318.330554] Lustre: Evicted from MGS (at 192.168.206.129@tcp) after server handle changed from 0xa2b54e63d535b08f to 0xa2b54e63d5377b5c
[ 2334.901199] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2336.985378] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2402.325849] LustreError: lustre-MDT0000-mdc-ffff9666919fb800: operation mds_reint to node 192.168.206.129@tcp failed: rc = -19
[ 2402.340821] LustreError: Skipped 2 previous similar messages
[ 2420.704712] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 192.168.206.129@tcp) was lost; in progress operations using this service will fail
[ 2429.863479] Lustre: Evicted from MGS (at 192.168.206.129@tcp) after server handle changed from 0xa2b54e63d5377b5c to 0xa2b54e63d5393e5e
[ 2447.144871] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2449.653828] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2479.723347] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 02:43:42 (1788590622)
[ 2501.599199] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 192.168.206.129@tcp) was lost; in progress operations using this service will fail
[ 2510.782483] Lustre: Evicted from MGS (at 192.168.206.129@tcp) after server handle changed from 0xa2b54e63d5393e5e to 0xa2b54e63d53a0cc9
[ 2511.193484] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2543.865738] Lustre: 15937:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.129@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2556.688921] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2594.814869] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2646.103927] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2669.187618] LustreError: lustre-MDT0000-mdc-ffff9666919fb800: operation mds_close to node 192.168.206.129@tcp failed: rc = -19
[ 2669.202814] LustreError: Skipped 4 previous similar messages
[ 2687.008423] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 192.168.206.129@tcp) was lost; in progress operations using this service will fail
[ 2687.029040] LustreError: Skipped 3 previous similar messages
[ 2697.107983] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2697.261680] Lustre: Evicted from MGS (at 192.168.206.129@tcp) after server handle changed from 0xa2b54e63d53a97a3 to 0xa2b54e63d53b39fa
[ 2697.269438] Lustre: Skipped 3 previous similar messages
[ 2755.519116] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2789.164368] Lustre: lustre-MDT0000-mdc-ffff9666919fb800: Connection to lustre-MDT0000 (at 192.168.206.129@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2789.175682] Lustre: Skipped 8 previous similar messages
[ 2808.813386] Lustre: MGC192.168.206.129@tcp: Connection restored to 192.168.206.129@tcp (at 192.168.206.129@tcp)
[ 2808.819785] Lustre: Skipped 16 previous similar messages
[ 2841.918908] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 02:49:44 (1788590984)
[ 2896.047373] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2898.328173] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3187.195170] LustreError: lustre-OST0000-osc-ffff9666919fb800: operation ldlm_enqueue to node 192.168.206.129@tcp failed: rc = -19
[ 3187.203472] LustreError: Skipped 4 previous similar messages
[ 3232.117140] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3234.729403] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3520.395674] Lustre: lustre-OST0000-osc-ffff9666919fb800: Connection to lustre-OST0000 (at 192.168.206.129@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3520.416119] Lustre: Skipped 2 previous similar messages
[ 3543.779367] Lustre: lustre-OST0000-osc-ffff9666919fb800: Connection restored to 192.168.206.129@tcp (at 192.168.206.129@tcp)
[ 3543.788566] Lustre: Skipped 3 previous similar messages
[ 3563.311349] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3565.582388] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3815.227400] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 03:05:57 (1788591957)
[ 3832.287192] Lustre: 50561:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788591960/real 1788591960]  req@ffff966686b51c00 x1875470720330752/t0(0) o101->lustre-MDT0000-mdc-ffff9666919fb800@192.168.206.129@tcp:12/10 lens 576/1152 e 0 to 1 dl 1788591976 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3832.331543] Lustre: 50561:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 3840.461742] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 03:06:23 (1788591983)
[ 3867.481719] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 03:06:50 (1788592010)
[ 3894.017986] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 03:07:16 (1788592036)
[ 3894.648693] Lustre: Mounted lustre-client - version 2.17.58_2_g97ad22e
[ 3925.490645] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 192.168.206.129@tcp) was lost; in progress operations using this service will fail
[ 3925.505672] LustreError: Skipped 2 previous similar messages
[ 3925.544830] Lustre: Evicted from MGS (at 192.168.206.129@tcp) after server handle changed from 0xa2b54e63d53c52be to 0xa2b54e63d551ebf3
[ 3925.565146] Lustre: Skipped 2 previous similar messages
[ 3926.495284] Lustre: 2373:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788592054/real 1788592054]  req@ffff96668767dc00 x1875470720350336/t0(0) o400->lustre-MDT0000-mdc-ffff9666919fb800@192.168.206.129@tcp:12/10 lens 224/224 e 0 to 1 dl 1788592070 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3926.524852] Lustre: 2373:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3936.488093] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3937.758685] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3940.008190] Lustre: Unmounted lustre-client
[ 3945.837106] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 03:08:08 (1788592088)
[ 4083.167131] Lustre: 2375:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788592211/real 1788592211]  req@ffff966686b50a80 x1875470720383232/t0(0) o4->lustre-OST0000-osc-ffff9666919fb800@192.168.206.129@tcp:6/4 lens 488/448 e 0 to 1 dl 1788592227 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4083.227821] Lustre: 2375:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 65 previous similar messages
[ 4131.295637] Lustre: lustre-OST0000-osc-ffff9666919fb800: Connection to lustre-OST0000 (at 192.168.206.129@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4131.324822] Lustre: Skipped 15 previous similar messages
[ 4147.700475] Lustre: lustre-OST0000-osc-ffff9666919fb800: Connection restored to 192.168.206.129@tcp (at 192.168.206.129@tcp)
[ 4147.708283] Lustre: Skipped 17 previous similar messages
[ 4241.207731] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 03:13:04 (1788592384)
[ 4289.787498] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 03:13:52 (1788592432)
[ 4293.374968] Lustre: Unmounted lustre-client
[ 4314.900409] Lustre: Mounted lustre-client - version 2.17.58_2_g97ad22e
[ 4321.032724] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 03:14:23 (1788592463)
[ 4321.359290] LustreError: 56487:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4322.415115] LustreError: 56487:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4322.580733] Lustre: *** cfs_fail_loc=305, val=0***
[ 4345.559286] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 03:14:48 (1788592488)
[ 4346.034988] Lustre: Mounted lustre-client - version 2.17.58_2_g97ad22e
[ 4348.490616] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4358.833592] Lustre: Unmounted lustre-client
[ 4366.067363] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 03:15:08 (1788592508)
[ 4447.371141] LustreError: lustre-MDT0000-mdc-ffff966684752800: operation mds_reint to node 192.168.206.129@tcp failed: rc = -19
[ 4447.379370] LustreError: Skipped 1 previous similar message
[ 4465.120289] Lustre: 2376:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788592593/real 1788592593]  req@ffff9666946c0000 x1875470723062272/t0(0) o400->MGC192.168.206.129@tcp@192.168.206.129@tcp:26/25 lens 224/224 e 0 to 1 dl 1788592609 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4465.153150] Lustre: 2376:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 78 previous similar messages
[ 4465.169065] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 192.168.206.129@tcp) was lost; in progress operations using this service will fail
[ 4475.384416] Lustre: Evicted from MGS (at 192.168.206.129@tcp) after server handle changed from 0xa2b54e63d55204ff to 0xa2b54e63d554bd86
[ 4626.234443] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 03:19:29 (1788592769)
[ 4631.017384] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4643.317501] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 192.168.206.129@tcp) was lost; in progress operations using this service will fail
[ 4643.353982] Lustre: Evicted from MGS (at 192.168.206.129@tcp) after server handle changed from 0xa2b54e63d554bd86 to 0xa2b54e63d55981ee
[ 4643.428976] LustreError: lustre-MDT0000-mdc-ffff966684752800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4659.517187] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 03:20:02 (1788592802)
[ 4660.039777] Lustre: Mounted lustre-client - version 2.17.58_2_g97ad22e
[ 4667.519039] LustreError: lustre-OST0000-osc-ffff966684752800: operation ldlm_enqueue to node 192.168.206.129@tcp failed: rc = -107
[ 4667.554444] LustreError: lustre-OST0000-osc-ffff966684752800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4680.115884] Lustre: Unmounted lustre-client
[ 4686.753457] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 03:20:29 (1788592829)
[ 4693.137780] LustreError: lustre-MDT0000-mdc-ffff966684752800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4693.152407] LustreError: 60875:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4699.749910] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 03:20:42 (1788592842)
[ 4699.986315] LustreError: 61578:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4703.984755] LustreError: lustre-MDT0000-mdc-ffff966684752800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4703.991819] LustreError: 61595:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4703.997138] LustreError: 61578:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=994
[ 4704.003161] LustreError: 61578:0:(import.c:761:ptlrpc_connect_import_locked()) already connecting
[ 4704.044566] LustreError: 61600:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4705.118488] LustreError: 61606:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4705.134081] LustreError: 61606:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4707.505106] LustreError: 61628:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4707.531817] LustreError: 61628:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4712.245952] LustreError: 61673:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4712.282414] LustreError: 61673:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4721.441797] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 03:21:04 (1788592864)
[ 4759.844414] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4761.536325] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4772.419274] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 03:21:55 (1788592915)
[ 4780.023773] Lustre: lustre-OST0000-osc-ffff966684752800: Connection to lustre-OST0000 (at 192.168.206.129@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4780.042444] Lustre: Skipped 18 previous similar messages
[ 4798.031280] Lustre: lustre-OST0000-osc-ffff966684752800: Connection restored to 192.168.206.129@tcp (at 192.168.206.129@tcp)
[ 4798.049156] Lustre: Skipped 17 previous similar messages
[ 4812.750852] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4813.981369] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4824.567981] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 03:22:47 (1788592967)
[ 4889.307203] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4892.099570] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4901.026327] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 03:24:03 (1788593043)
[ 4938.049033] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4939.438194] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4944.689844] Lustre: Unmounted lustre-client
[ 4994.642631] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4997.729853] Lustre: Mounted lustre-client - version 2.17.58_2_g97ad22e
[ 5006.026422] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 03:25:48 (1788593148)
[ 5009.161846] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5011.314949] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 03:25:53 (1788593153)
[ 5043.757356] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 03:26:26 (1788593186)
[ 5045.615344] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5047.772219] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 03:26:30 (1788593190)
[ 5049.356421] Lustre: *** cfs_fail_loc=805, val=0***
[ 5049.405358] Lustre: Mounted lustre-client - version 2.17.58_2_g97ad22e
[ 5054.426488] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5074.783798] Lustre: 2376:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788593203/real 1788593203]  req@ffff966686b54380 x1875470725108608/t0(0) o400->lustre-MDT0000-mdc-ffff966685964800@192.168.206.129@tcp:12/10 lens 224/224 e 0 to 1 dl 1788593219 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5074.825900] Lustre: 2376:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5074.833488] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 192.168.206.129@tcp) was lost; in progress operations using this service will fail
[ 5085.170741] Lustre: Evicted from MGS (at 192.168.206.129@tcp) after server handle changed from 0xa2b54e63d5598c75 to 0xa2b54e63d559901f
[ 5085.205559] LustreError: 2372:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff966682bba000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff966683ea5600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5087.688677] Lustre: Unmounted lustre-client
[ 5095.129702] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 03:27:17 (1788593237)
[ 5122.066124] LustreError: 2372:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff9666857c1500 x1875470725119104/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff966685964800@192.168.206.129@tcp:12/10 lens 664/608 e 0 to 0 dl 1788593282 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5133.579262] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5135.288806] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5143.605071] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 03:28:06 (1788593286)
[ 5145.273970] LustreError: lustre-OST0000-osc-ffff966685964800: operation ost_write to node 192.168.206.129@tcp failed: rc = -107
[ 5145.284245] LustreError: Skipped 2 previous similar messages
[ 5145.292849] LustreError: lustre-OST0000-osc-ffff966685964800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5145.312802] Lustre: 2374:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.129@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5145.352540] LustreError: 72779:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff966685964800: namespace resource [0x240000400:0x3f62:0x0].0x0 (ffff9666947d7e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5145.383443] LustreError: 72779:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5153.649594] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 03:28:16 (1788593296)
[ 5155.142868] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5157.214179] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 03:28:19 (1788593299)
[ 5159.048698] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5160.837904] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 03:28:23 (1788593303)
[ 5162.219219] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5163.707339] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 03:28:26 (1788593306)
[ 5165.217525] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5166.772160] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 03:28:29 (1788593309)
[ 5168.082564] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5169.520067] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 03:28:32 (1788593312)
[ 5170.797126] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5172.376171] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 03:28:35 (1788593315)
[ 5173.625565] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5175.185439] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 03:28:38 (1788593318)
[ 5176.592540] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5178.117236] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 03:28:41 (1788593321)
[ 5179.582356] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5181.190910] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 03:28:44 (1788593324)
[ 5182.500248] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5184.001726] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 03:28:47 (1788593327)
[ 5185.510213] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5186.921538] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 03:28:50 (1788593330)
[ 5189.282472] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5190.999828] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 03:28:53 (1788593333)
[ 5218.946710] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 03:29:22 (1788593362)
[ 5248.227170] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 03:29:51 (1788593391)
[ 5248.727034] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5258.843326] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 03:30:01 (1788593401)
[ 5259.126620] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5268.233978] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 03:30:11 (1788593411)
[ 5268.474578] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5276.754264] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 03:30:19 (1788593419)
[ 5277.106227] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5285.807663] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 03:30:28 (1788593428)
[ 5286.227861] Lustre: *** cfs_fail_loc=510, val=3***
[ 5294.923265] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 03:30:37 (1788593437)
[ 5295.291549] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5305.248653] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 03:30:48 (1788593448)
[ 5305.643827] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5369.818831] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 03:31:53 (1788593513)
[ 5370.150343] LustreError: 82983:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5372.834913] LustreError: lustre-MDT0000-mdc-ffff966685964800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5372.852067] LustreError: 82997:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff966685964800: inode [0x20000a042:0x10:0x0] mdc close failed: rc = -108
[ 5372.871492] LustreError: 82997:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff966685964800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9666947bd900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5374.223686] LustreError: 82983:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5376.296625] LustreError: 83013:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5379.041942] LustreError: lustre-MDT0000-mdc-ffff966685964800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5379.064935] LustreError: 83027:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff966685964800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9666947d7f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5379.083135] LustreError: 83027:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5380.359484] LustreError: 83013:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5387.416931] Lustre: lustre-MDT0000-mdc-ffff966685964800: Connection to lustre-MDT0000 (at 192.168.206.129@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5387.427454] Lustre: Skipped 10 previous similar messages
[ 5387.442667] LustreError: lustre-MDT0000-mdc-ffff966685964800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5387.833580] LustreError: 83069:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5390.814506] LustreError: lustre-MDT0000-mdc-ffff966685964800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5390.869271] LustreError: 83088:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5390.875783] LustreError: 83088:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5391.903168] LustreError: 83069:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5391.910823] LustreError: 83069:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff966685964800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5391.947886] LustreError: 83094:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5391.968811] LustreError: 83094:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5394.272633] LustreError: 83117:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5394.291558] LustreError: 83117:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5398.037424] LustreError: 83144:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5400.950156] LustreError: lustre-MDT0000-mdc-ffff966685964800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5400.971886] LustreError: 83158:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff966685964800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9666858ece00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5401.006079] LustreError: 83158:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5401.036603] Lustre: lustre-MDT0000-mdc-ffff966685964800: Connection restored to 192.168.206.129@tcp (at 192.168.206.129@tcp)
[ 5401.044527] Lustre: Skipped 13 previous similar messages
[ 5402.138325] LustreError: 83144:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5408.809683] LustreError: lustre-MDT0000-mdc-ffff966685964800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5409.248256] LustreError: 83200:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5411.750383] LustreError: lustre-MDT0000-mdc-ffff966685964800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5411.770369] LustreError: 83214:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff966685964800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9666947d7c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5411.783869] LustreError: 83214:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5413.327280] LustreError: 83200:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5416.327836] LustreError: lustre-MDT0000-mdc-ffff966685964800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5416.396951] LustreError: 83247:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5416.402851] LustreError: 83247:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5417.519893] LustreError: 83228:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff966685964800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5425.346176] LustreError: 83299:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5425.368686] LustreError: 83299:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5428.672990] LustreError: lustre-MDT0000-mdc-ffff966685964800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5428.750786] LustreError: 83322:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5428.770846] LustreError: 83322:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5429.351146] LustreError: 83299:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5429.359226] LustreError: 83299:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5429.365213] LustreError: 83299:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff966685964800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5438.475701] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 03:33:01 (1788593581)
[ 5457.845178] LustreError: 68238:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5457.856377] LustreError: 68238:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5461.919104] LustreError: 68238:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5461.928226] LustreError: 68238:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5470.294610] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 03:33:32 (1788593612)
[ 5521.030995] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 03:34:23 (1788593663)
[ 5585.909673] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 03:35:28 (1788593728)
[ 5619.633341] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 03:36:02 (1788593762)
[ 5620.228096] Lustre: Mounted lustre-client - version 2.17.58_2_g97ad22e
[ 5744.200182] Lustre: Unmounted lustre-client
[ 5751.825397] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 03:38:14 (1788593894)
[ 5752.575520] LustreError: 87633:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5755.181050] LustreError: lustre-OST0000-osc-ffff966685964800: operation ldlm_enqueue to node 192.168.206.129@tcp failed: rc = -107
[ 5755.187827] LustreError: Skipped 9 previous similar messages
[ 5755.211785] LustreError: lustre-OST0000-osc-ffff966685964800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5755.233163] LustreError: 87715:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff966685964800: namespace resource [0x240000400:0x3f8d:0x0].0x0 (ffff9666947d7200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5755.282107] LustreError: 87715:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5755.543467] LustreError: 87633:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5755.549553] Lustre: 2376:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.129@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5762.686571] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 03:38:25 (1788593905)
[ 5821.407262] Lustre: 88301:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788593909/real 1788593909]  req@ffff966686b55880 x1875470725291776/t0(0) o101->lustre-MDT0000-mdc-ffff966685964800@192.168.206.129@tcp:12/10 lens 328/344 e 0 to 1 dl 1788593965 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5821.441638] Lustre: 88301:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 17 previous similar messages
[ 5829.051793] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 03:39:31 (1788593971)
[ 5830.861211] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5832.549885] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 03:39:35 (1788593975)
[ 5896.927196] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5898.789691] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 03:40:41 (1788594041)
[ 5958.248684] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 03:41:41 (1788594101)
[ 5960.406979] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5962.032556] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 03:41:45 (1788594105)
[ 5963.465832] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5965.178882] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 03:41:48 (1788594108)
[ 5996.977203] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 03:42:19 (1788594139)
[ 6006.961513] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6033.388484] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 192.168.206.129@tcp) was lost; in progress operations using this service will fail
[ 6033.398366] LustreError: Skipped 2 previous similar messages
[ 6033.402926] Lustre: lustre-MDT0000-mdc-ffff966685964800: Connection to lustre-MDT0000 (at 192.168.206.129@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6033.408810] Lustre: Evicted from MGS (at 192.168.206.129@tcp) after server handle changed from 0xa2b54e63d5599abb to 0xa2b54e63d559b47d
[ 6033.427403] Lustre: Skipped 13 previous similar messages
[ 6033.438362] Lustre: Skipped 2 previous similar messages
[ 6033.448122] Lustre: MGC192.168.206.129@tcp: Connection restored to 192.168.206.129@tcp (at 192.168.206.129@tcp)
[ 6033.461207] Lustre: Skipped 10 previous similar messages
[ 6044.172798] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6045.401915] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6054.740259] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 03:43:17 (1788594197)
[ 6057.293943] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6058.880583] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 03:43:21 (1788594201)
[ 6074.349141] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 192.168.206.129@tcp) was lost; in progress operations using this service will fail
[ 6074.387151] Lustre: Evicted from MGS (at 192.168.206.129@tcp) after server handle changed from 0xa2b54e63d559b47d to 0xa2b54e63d559b7b7
[ 6082.687935] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 03:43:45 (1788594225)
[ 6135.153746] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 03:44:38 (1788594278)
[ 6189.444353] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6192.038611] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6278.632595] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 192.168.206.129@tcp) was lost; in progress operations using this service will fail
[ 6278.638895] LustreError: Skipped 1 previous similar message
[ 6278.650748] Lustre: Evicted from MGS (at 192.168.206.129@tcp) after server handle changed from 0xa2b54e63d559bb3e to 0xa2b54e63d559c9a0
[ 6278.657193] Lustre: Skipped 1 previous similar message
[ 6289.535620] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6290.836941] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6329.294211] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6330.794452] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6454.498657] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 03:49:57 (1788594597)
[ 6464.532867] LustreError: lustre-OST0000-osc-ffff966685964800: operation ost_setattr to node 192.168.206.129@tcp failed: rc = -107
[ 6464.556429] LustreError: Skipped 13 previous similar messages
[ 6509.971251] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6511.994455] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6755.416689] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 03:54:58 (1788594898)
[ 6893.033932] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 192.168.206.129@tcp) was lost; in progress operations using this service will fail
[ 6893.035243] Lustre: lustre-MDT0000-mdc-ffff966685964800: Connection to lustre-MDT0000 (at 192.168.206.129@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6893.044951] LustreError: Skipped 1 previous similar message
[ 6893.066434] Lustre: Skipped 6 previous similar messages
[ 6893.069780] Lustre: Evicted from MGS (at 192.168.206.129@tcp) after server handle changed from 0xa2b54e63d559cc86 to 0xa2b54e63d569700f
[ 6893.086696] Lustre: Skipped 1 previous similar message
[ 6893.092357] Lustre: MGC192.168.206.129@tcp: Connection restored to 192.168.206.129@tcp (at 192.168.206.129@tcp)
[ 6893.100465] Lustre: Skipped 9 previous similar messages
[ 6922.585567] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 03:57:45 (1788595065)
[ 6923.809904] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6925.145422] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 03:57:48 (1788595068)
[ 6926.922826] LustreError: lustre-MDT0000-mdc-ffff966685964800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6932.914346] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 03:57:56 (1788595076)
[ 7099.762427] LustreError: lustre-OST0000-osc-ffff966685964800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7102.065879] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 04:00:44 (1788595244)
[ 7127.519153] Lustre: 2373:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788595251/real 1788595251]  req@ffff966686ae3800 x1875470742990720/t0(0) o4->lustre-OST0000-osc-ffff966685964800@192.168.206.129@tcp:6/4 lens 4584/448 e 0 to 1 dl 1788595271 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7127.564883] Lustre: 2373:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 7143.833516] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 04:01:26 (1788595286)
[ 7145.084552] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7146.527178] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 04:01:29 (1788595289)
[ 7147.867248] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7149.546219] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 04:01:32 (1788595292)
[ 7182.689540] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 04:02:05 (1788595325)
[ 7217.662058] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 192.168.206.129@tcp) was lost; in progress operations using this service will fail
[ 7217.688340] Lustre: Evicted from MGS (at 192.168.206.129@tcp) after server handle changed from 0xa2b54e63d569700f to 0xa2b54e63d56996f8
[ 7217.698023] Lustre: Skipped 1 previous similar message
[ 7234.133721] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 04:02:57 (1788595377)
[ 7235.414883] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7236.943670] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 04:02:59 (1788595379)
[ 7238.344732] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7240.062297] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 04:03:02 (1788595382)
[ 7248.760327] Lustre: Unmounted lustre-client
[ 7250.462576] Lustre: Mounted lustre-client - version 2.17.58_2_g97ad22e
[ 7253.080916] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7270.879402] Lustre: 2375:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788595399/real 1788595399]  req@ffff966692983b80 x1875470743177472/t0(0) o400->MGC192.168.206.129@tcp@192.168.206.129@tcp:26/25 lens 224/224 e 0 to 1 dl 1788595415 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7283.090617] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2