[ 1897.928815] LustreError: lustre-OST0000-osc-ffff95ce1336e800: operation ost_sync to node 192.168.201.144@tcp failed: rc = -107
[ 1897.975416] LustreError: lustre-OST0000-osc-ffff95ce1336e800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1898.005416] Lustre: 2347:0:(llite_lib.c:4202:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.144@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1900.057622] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ IGNORE (bz5494): multiop didn't fail fsync: 5 or close: 0
[ 1908.568382] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 04:13:21 (1787386401)
[ 1910.642526] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1912.707405] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 04:13:25 (1787386405)
[ 1914.432312] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1917.094420] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 04:13:29 (1787386409)
[ 1921.448175] LustreError: lustre-MDT0000-mdc-ffff95ce1336e800: operation ldlm_enqueue to node 192.168.201.144@tcp failed: rc = -19
[ 1921.457201] LustreError: Skipped 1 previous similar message
[ 1939.936628] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 192.168.201.144@tcp) was lost; in progress operations using this service will fail
[ 1950.205339] Lustre: Evicted from MGS (at 192.168.201.144@tcp) after server handle changed from 0x57f9108b680311ba to 0x57f9108b680328dc
[ 2046.968743] LustreError: lustre-MDT0000-mdc-ffff95ce1336e800: operation mds_reint to node 192.168.201.144@tcp failed: rc = -19
[ 2066.911613] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 192.168.201.144@tcp) was lost; in progress operations using this service will fail
[ 2077.111358] Lustre: Evicted from MGS (at 192.168.201.144@tcp) after server handle changed from 0x57f9108b680328dc to 0x57f9108b6804a3ca
[ 2094.640987] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 04:16:27 (1787386587)
[ 2095.112756] Lustre: *** cfs_fail_loc=305, val=0***
[ 2095.115842] Lustre: Skipped 2 previous similar messages
[ 2137.580705] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 192.168.201.144@tcp) was lost; in progress operations using this service will fail
[ 2137.621215] Lustre: Evicted from MGS (at 192.168.201.144@tcp) after server handle changed from 0x57f9108b6804a3ca to 0x57f9108b6804a7f9
[ 2152.739813] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2155.358590] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2166.819845] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 04:17:39 (1787386659)
[ 2183.678183] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 192.168.201.144@tcp) was lost; in progress operations using this service will fail
[ 2183.718060] Lustre: lustre-MDT0000-mdc-ffff95ce1336e800: Connection to lustre-MDT0000 (at 192.168.201.144@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2183.739061] Lustre: Skipped 12 previous similar messages
[ 2183.742811] Lustre: Evicted from MGS (at 192.168.201.144@tcp) after server handle changed from 0x57f9108b6804a7f9 to 0x57f9108b6804acf3
[ 2183.777111] Lustre: MGC192.168.201.144@tcp: Connection restored to 192.168.201.144@tcp (at 192.168.201.144@tcp)
[ 2183.796688] Lustre: Skipped 15 previous similar messages
[ 2183.887101] LustreError: lustre-MDT0000-mdc-ffff95ce1336e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2213.322678] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 04:18:25 (1787386705)
[ 2229.026771] LustreError: lustre-OST0000-osc-ffff95ce1336e800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2254.527829] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 04:19:07 (1787386747)
[ 2268.683088] LustreError: lustre-MDT0000-mdc-ffff95ce1336e800: operation mds_reint to node 192.168.201.144@tcp failed: rc = -19
[ 2268.702598] LustreError: Skipped 1 previous similar message
[ 2286.879289] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 192.168.201.144@tcp) was lost; in progress operations using this service will fail
[ 2297.075318] Lustre: Evicted from MGS (at 192.168.201.144@tcp) after server handle changed from 0x57f9108b6804acf3 to 0x57f9108b6804f3e1
[ 2315.260814] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2317.944917] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2403.811804] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787386882/real 1787386882]  req@ffff95ce05ad3100 x1874208533676160/t0(0) o400->MGC192.168.201.144@tcp@192.168.201.144@tcp:26/25 lens 224/224 e 0 to 1 dl 1787386898 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2403.839842] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 2403.851900] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 192.168.201.144@tcp) was lost; in progress operations using this service will fail
[ 2403.885453] Lustre: Evicted from MGS (at 192.168.201.144@tcp) after server handle changed from 0x57f9108b6804f3e1 to 0x57f9108b680682ef
[ 2405.248990] Lustre: 15904:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.144@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2433.830016] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2438.638421] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2506.454659] LustreError: lustre-MDT0000-mdc-ffff95ce1336e800: operation mds_reint to node 192.168.201.144@tcp failed: rc = -19
[ 2506.482715] LustreError: Skipped 1 previous similar message
[ 2526.501017] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 192.168.201.144@tcp) was lost; in progress operations using this service will fail
[ 2526.546347] Lustre: Evicted from MGS (at 192.168.201.144@tcp) after server handle changed from 0x57f9108b680682ef to 0x57f9108b6807f78c
[ 2528.517231] Lustre: 15904:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.144@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2553.007264] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2555.679072] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2588.847395] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 04:24:40 (1787387080)
[ 2611.680388] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 192.168.201.144@tcp) was lost; in progress operations using this service will fail
[ 2621.867658] Lustre: Evicted from MGS (at 192.168.201.144@tcp) after server handle changed from 0x57f9108b6807f78c to 0x57f9108b6808adaf
[ 2631.108991] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2672.462878] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2710.190711] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2742.751272] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 192.168.201.144@tcp) was lost; in progress operations using this service will fail
[ 2742.767072] LustreError: Skipped 2 previous similar messages
[ 2753.076631] Lustre: Evicted from MGS (at 192.168.201.144@tcp) after server handle changed from 0x57f9108b6808efa3 to 0x57f9108b68091ca5
[ 2753.096361] Lustre: Skipped 2 previous similar messages
[ 2761.694544] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2785.892883] LustreError: lustre-MDT0000-mdc-ffff95ce1336e800: operation mds_close to node 192.168.201.144@tcp failed: rc = -19
[ 2785.913723] LustreError: Skipped 4 previous similar messages
[ 2785.924067] Lustre: lustre-MDT0000-mdc-ffff95ce1336e800: Connection to lustre-MDT0000 (at 192.168.201.144@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2785.934943] Lustre: Skipped 8 previous similar messages
[ 2813.371580] Lustre: MGC192.168.201.144@tcp: Connection restored to 192.168.201.144@tcp (at 192.168.201.144@tcp)
[ 2813.379548] Lustre: Skipped 16 previous similar messages
[ 2823.431086] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2891.989772] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2956.397297] Lustre: 15904:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.144@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2992.165220] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 04:31:25 (1787387485)
[ 3054.944050] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3057.910583] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3338.832519] LustreError: lustre-OST0000-osc-ffff95ce1336e800: operation ldlm_enqueue to node 192.168.201.144@tcp failed: rc = -107
[ 3338.845170] LustreError: Skipped 8 previous similar messages
[ 3381.029540] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3383.203991] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3670.548499] Lustre: lustre-OST0000-osc-ffff95ce1336e800: Connection to lustre-OST0000 (at 192.168.201.144@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3670.556286] Lustre: Skipped 4 previous similar messages
[ 3711.654521] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3713.811934] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3966.205946] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 04:47:39 (1787388459)
[ 3983.839259] Lustre: 50650:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787388462/real 1787388462]  req@ffff95ce05ad3800 x1874208555216640/t0(0) o101->lustre-MDT0000-mdc-ffff95ce1336e800@192.168.201.144@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787388478 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3983.867045] Lustre: 50650:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 3983.920112] Lustre: lustre-MDT0000-mdc-ffff95ce1336e800: Connection restored to 192.168.201.144@tcp (at 192.168.201.144@tcp)
[ 3983.937150] Lustre: Skipped 6 previous similar messages
[ 3991.861601] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 04:48:04 (1787388484)
[ 4015.922478] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 04:48:29 (1787388509)
[ 4041.838457] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 04:48:54 (1787388534)
[ 4042.246315] Lustre: Mounted lustre-client
[ 4072.927280] Lustre: 2346:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787388551/real 1787388551]  req@ffff95ce10c8e680 x1874208555234816/t0(0) o400->MGC192.168.201.144@tcp@192.168.201.144@tcp:26/25 lens 224/224 e 0 to 1 dl 1787388567 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4072.947835] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 192.168.201.144@tcp) was lost; in progress operations using this service will fail
[ 4072.961528] Lustre: 2346:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 4072.987666] LustreError: Skipped 3 previous similar messages
[ 4073.024203] Lustre: Evicted from MGS (at 192.168.201.144@tcp) after server handle changed from 0x57f9108b680aa6c7 to 0x57f9108b681fb3e7
[ 4073.034369] Lustre: Skipped 3 previous similar messages
[ 4086.365350] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4087.942391] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4090.099778] Lustre: Unmounted lustre-client
[ 4096.285930] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 04:49:49 (1787388589)
[ 4233.695210] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787388712/real 1787388712]  req@ffff95ce080d7800 x1874208555268224/t0(0) o4->lustre-OST0000-osc-ffff95ce1336e800@192.168.201.144@tcp:6/4 lens 488/448 e 0 to 1 dl 1787388728 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4233.729102] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 55 previous similar messages
[ 4281.823322] Lustre: lustre-OST0000-osc-ffff95ce1336e800: Connection to lustre-OST0000 (at 192.168.201.144@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4281.861153] Lustre: Skipped 15 previous similar messages
[ 4392.703920] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 04:54:45 (1787388885)
[ 4440.918446] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 04:55:34 (1787388934)
[ 4444.257880] Lustre: Unmounted lustre-client
[ 4465.359556] Lustre: Mounted lustre-client
[ 4471.141296] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 04:56:04 (1787388964)
[ 4471.432511] LustreError: 56567:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4472.483075] LustreError: 56567:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4472.602626] Lustre: *** cfs_fail_loc=305, val=0***
[ 4496.020795] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 04:56:28 (1787388988)
[ 4496.479794] Lustre: Mounted lustre-client
[ 4498.628952] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4508.933376] Lustre: Unmounted lustre-client
[ 4515.317424] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 04:56:48 (1787389008)
[ 4595.342089] LustreError: lustre-MDT0000-mdc-ffff95ce0814a000: operation mds_reint to node 192.168.201.144@tcp failed: rc = -19
[ 4595.351424] LustreError: Skipped 1 previous similar message
[ 4615.149878] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 192.168.201.144@tcp) was lost; in progress operations using this service will fail
[ 4615.173235] Lustre: Evicted from MGS (at 192.168.201.144@tcp) after server handle changed from 0x57f9108b681fcfcb to 0x57f9108b6822878e
[ 4615.199665] Lustre: MGC192.168.201.144@tcp: Connection restored to 192.168.201.144@tcp (at 192.168.201.144@tcp)
[ 4615.211262] Lustre: Skipped 23 previous similar messages
[ 4757.140447] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 05:00:50 (1787389250)
[ 4762.062732] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4773.874251] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 192.168.201.144@tcp) was lost; in progress operations using this service will fail
[ 4773.899508] Lustre: Evicted from MGS (at 192.168.201.144@tcp) after server handle changed from 0x57f9108b6822878e to 0x57f9108b68274b8d
[ 4775.254752] LustreError: lustre-MDT0000-mdc-ffff95ce0814a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4791.709423] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 05:01:24 (1787389284)
[ 4792.089047] Lustre: Mounted lustre-client
[ 4799.372110] LustreError: lustre-OST0000-osc-ffff95ce0814a000: operation ldlm_enqueue to node 192.168.201.144@tcp failed: rc = -107
[ 4799.391233] LustreError: lustre-OST0000-osc-ffff95ce0814a000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4808.671688] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787389287/real 1787389287]  req@ffff95ce044ed880 x1874208559921408/t0(0) o103->lustre-OST0000-osc-ffff95ce10f4e800@192.168.201.144@tcp:17/18 lens 328/224 e 0 to 1 dl 1787389303 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4808.701691] Lustre: 2348:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 82 previous similar messages
[ 4811.122459] Lustre: Unmounted lustre-client
[ 4818.270766] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 05:01:51 (1787389311)
[ 4824.986337] LustreError: lustre-MDT0000-mdc-ffff95ce0814a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4825.006935] LustreError: 60949:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4831.849810] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 05:02:05 (1787389325)
[ 4832.184875] LustreError: 61654:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4837.345052] LustreError: 61654:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4837.350525] LustreError: 61654:0:(import.c:711:ptlrpc_connect_import_locked()) already connected
[ 4838.271952] LustreError: lustre-MDT0000-mdc-ffff95ce0814a000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4838.282026] LustreError: 61671:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4838.357517] LustreError: 61676:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4839.457824] LustreError: 61682:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4839.470540] LustreError: 61682:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4841.771950] LustreError: 61704:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4841.783411] LustreError: 61704:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4846.280888] LustreError: 61749:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4846.300232] LustreError: 61749:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4856.270272] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 05:02:29 (1787389349)
[ 4893.087834] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4894.583354] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4903.929952] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 05:03:17 (1787389397)
[ 4910.066303] Lustre: lustre-OST0000-osc-ffff95ce0814a000: Connection to lustre-OST0000 (at 192.168.201.144@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4910.085616] Lustre: Skipped 18 previous similar messages
[ 4940.609907] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4942.015300] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4952.844230] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 05:04:05 (1787389445)
[ 5017.506990] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5018.856297] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5025.578847] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 05:05:18 (1787389518)
[ 5061.279054] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5063.122646] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5069.546951] Lustre: Unmounted lustre-client
[ 5129.181227] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5131.662547] Lustre: Mounted lustre-client
[ 5138.324132] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 05:07:11 (1787389631)
[ 5140.462764] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5142.198274] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 05:07:15 (1787389635)
[ 5170.409822] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 05:07:43 (1787389663)
[ 5171.886450] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5173.445918] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 05:07:46 (1787389666)
[ 5174.715294] Lustre: *** cfs_fail_loc=805, val=0***
[ 5174.778230] Lustre: Mounted lustre-client
[ 5179.133262] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5200.362368] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 192.168.201.144@tcp) was lost; in progress operations using this service will fail
[ 5200.386081] Lustre: Evicted from MGS (at 192.168.201.144@tcp) after server handle changed from 0x57f9108b682757e9 to 0x57f9108b68275b85
[ 5207.855195] Lustre: Unmounted lustre-client
[ 5216.579400] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 05:08:29 (1787389709)
[ 5241.337091] Lustre: MGC192.168.201.144@tcp: Connection restored to 192.168.201.144@tcp (at 192.168.201.144@tcp)
[ 5241.346186] Lustre: Skipped 16 previous similar messages
[ 5241.387951] LustreError: 2345:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff95ce06865180 x1874208560004224/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff95ce10d02800@192.168.201.144@tcp:12/10 lens 664/608 e 0 to 0 dl 1787389751 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5254.617712] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5256.294282] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5264.334667] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 05:09:17 (1787389757)
[ 5266.038179] LustreError: lustre-OST0000-osc-ffff95ce10d02800: operation ost_write to node 192.168.201.144@tcp failed: rc = -107
[ 5266.051606] LustreError: Skipped 2 previous similar messages
[ 5266.077297] LustreError: lustre-OST0000-osc-ffff95ce10d02800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5266.093650] Lustre: 2349:0:(llite_lib.c:4202:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.144@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5266.129147] LustreError: 72774:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff95ce10d02800: namespace resource [0x240000400:0x3ca2:0x0].0x0 (ffff95ce07179e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5275.578046] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 05:09:28 (1787389768)
[ 5277.300459] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5279.397713] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 05:09:32 (1787389772)
[ 5281.337992] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5283.685864] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 05:09:36 (1787389776)
[ 5285.556816] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5287.004513] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 05:09:40 (1787389780)
[ 5288.651240] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5290.356892] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 05:09:43 (1787389783)
[ 5291.900478] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5293.803601] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 05:09:46 (1787389786)
[ 5295.061803] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5296.832838] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 05:09:49 (1787389789)
[ 5298.365888] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5299.878924] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 05:09:53 (1787389793)
[ 5301.155514] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5303.217711] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 05:09:55 (1787389795)
[ 5305.197645] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5306.897064] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 05:09:59 (1787389799)
[ 5308.284828] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5310.593885] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 05:10:03 (1787389803)
[ 5312.306884] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5314.218181] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 05:10:07 (1787389807)
[ 5317.304984] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5319.169502] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 05:10:12 (1787389812)
[ 5352.457651] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 05:10:45 (1787389845)
[ 5383.309712] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 05:11:16 (1787389876)
[ 5383.791561] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5395.929221] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 05:11:28 (1787389888)
[ 5396.274117] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5407.496375] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 05:11:40 (1787389900)
[ 5407.803611] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5417.508829] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 05:11:49 (1787389909)
[ 5417.881452] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5427.284276] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 05:12:00 (1787389920)
[ 5427.751909] Lustre: *** cfs_fail_loc=510, val=3***
[ 5436.241384] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 05:12:09 (1787389929)
[ 5436.757816] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5446.718392] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 05:12:19 (1787389939)
[ 5447.183636] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5507.551770] Lustre: 2347:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787389945/real 1787389945]  req@ffff95ce04679180 x1874208560050560/t0(0) o3->lustre-OST0001-osc-ffff95ce10d02800@192.168.201.144@tcp:6/4 lens 488/4536 e 0 to 1 dl 1787390001 ref 2 fl Rpc:RXQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 5507.575029] Lustre: 2347:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 5516.478000] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 05:13:28 (1787390008)
[ 5516.896947] LustreError: 82965:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5519.971574] Lustre: lustre-MDT0000-mdc-ffff95ce10d02800: Connection to lustre-MDT0000 (at 192.168.201.144@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5519.984087] Lustre: Skipped 9 previous similar messages
[ 5519.995257] LustreError: lustre-MDT0000-mdc-ffff95ce10d02800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5520.004745] LustreError: 82979:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff95ce10d02800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff95ce07003b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5520.983488] LustreError: 82965:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5523.174673] LustreError: 82995:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5526.235600] LustreError: lustre-MDT0000-mdc-ffff95ce10d02800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5526.243513] LustreError: 83009:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff95ce10d02800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff95ce10287c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5526.261429] LustreError: 83009:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5527.239153] LustreError: 82995:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5534.382311] LustreError: lustre-MDT0000-mdc-ffff95ce10d02800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5534.918562] LustreError: 83050:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5537.913937] LustreError: lustre-MDT0000-mdc-ffff95ce10d02800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5538.025296] LustreError: 83070:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5538.043865] LustreError: 83070:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5538.999449] LustreError: 83050:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5539.016698] LustreError: 83050:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff95ce10d02800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5539.196718] LustreError: 83076:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5539.212787] LustreError: 83076:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5541.579732] LustreError: 83098:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5541.601872] LustreError: 83098:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5545.533471] LustreError: 83125:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5549.615582] LustreError: 83125:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5549.819274] LustreError: lustre-MDT0000-mdc-ffff95ce10d02800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5556.595859] LustreError: lustre-MDT0000-mdc-ffff95ce10d02800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5556.944971] LustreError: 83181:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5559.858908] LustreError: lustre-MDT0000-mdc-ffff95ce10d02800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5559.890340] LustreError: 83195:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff95ce10d02800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff95ce05e71b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5559.937217] LustreError: 83195:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5560.945661] LustreError: 83181:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5564.323948] LustreError: lustre-MDT0000-mdc-ffff95ce10d02800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5564.403449] LustreError: 83228:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5564.424972] LustreError: 83228:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5565.156469] LustreError: 83208:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff95ce10d02800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5573.792090] LustreError: 83279:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5573.820968] LustreError: 83279:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5577.693758] LustreError: lustre-MDT0000-mdc-ffff95ce10d02800: operation ldlm_enqueue to node 192.168.201.144@tcp failed: rc = -107
[ 5577.701835] LustreError: Skipped 8 previous similar messages
[ 5577.730789] LustreError: lustre-MDT0000-mdc-ffff95ce10d02800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5577.807844] LustreError: 83302:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5577.816621] LustreError: 83302:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5577.831286] LustreError: 83279:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5577.844392] LustreError: 83279:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5577.862660] LustreError: 83279:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff95ce10d02800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5590.168189] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 05:14:42 (1787390082)
[ 5609.849872] LustreError: 81010:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5609.861688] LustreError: 81010:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5613.935183] LustreError: 81010:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5613.942074] LustreError: 81010:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5622.684624] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 05:15:15 (1787390115)
[ 5675.543258] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 05:16:08 (1787390168)
[ 5743.400887] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 05:17:16 (1787390236)
[ 5776.687435] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 05:17:49 (1787390269)
[ 5777.239141] Lustre: Mounted lustre-client
[ 5901.500680] Lustre: Unmounted lustre-client
[ 5909.319271] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 05:20:02 (1787390402)
[ 5910.016415] LustreError: 87611:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5912.473322] LustreError: lustre-OST0000-osc-ffff95ce10d02800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5912.487515] LustreError: 87693:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff95ce10d02800: namespace resource [0x240000400:0x3ccd:0x0].0x0 (ffff95ce068a7600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5912.515807] LustreError: 87693:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5912.755587] LustreError: 87611:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5912.763651] Lustre: 2348:0:(llite_lib.c:4202:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.144@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5919.385730] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 05:20:12 (1787390412)
[ 5978.090050] Lustre: lustre-MDT0000-mdc-ffff95ce10d02800: Connection restored to 192.168.201.144@tcp (at 192.168.201.144@tcp)
[ 5978.098376] Lustre: Skipped 15 previous similar messages
[ 5985.367661] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 05:21:18 (1787390478)
[ 5987.098183] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5988.802089] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 05:21:21 (1787390481)
[ 6054.060616] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6055.630950] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 05:22:28 (1787390548)
[ 6113.247891] Lustre: 90008:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787390551/real 1787390551]  req@ffff95ce06864700 x1874208560199680/t0(0) o36->lustre-MDT0000-mdc-ffff95ce10d02800@192.168.201.144@tcp:12/10 lens 496/440 e 0 to 1 dl 1787390607 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ln.0' uid:0 gid:0 projid:4294967295
[ 6113.319917] Lustre: 90008:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 6115.583222] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 05:23:28 (1787390608)
[ 6118.309933] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6121.152345] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 05:23:33 (1787390613)
[ 6122.617297] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6124.058823] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 05:23:37 (1787390617)
[ 6161.736264] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 05:24:14 (1787390654)
[ 6175.271541] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6201.119364] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 192.168.201.144@tcp) was lost; in progress operations using this service will fail
[ 6201.126150] LustreError: Skipped 2 previous similar messages
[ 6206.449747] Lustre: lustre-MDT0000-mdc-ffff95ce10d02800: Connection to lustre-MDT0000 (at 192.168.201.144@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6206.487793] Lustre: Skipped 15 previous similar messages
[ 6210.662961] Lustre: Evicted from MGS (at 192.168.201.144@tcp) after server handle changed from 0x57f9108b6827662f to 0x57f9108b6827814f
[ 6210.673233] Lustre: Skipped 2 previous similar messages
[ 6216.079749] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6217.616967] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6228.187344] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 05:25:21 (1787390721)
[ 6230.425708] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6231.971979] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 05:25:25 (1787390725)
[ 6241.777964] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 192.168.201.144@tcp) was lost; in progress operations using this service will fail
[ 6255.753917] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 05:25:48 (1787390748)
[ 6288.874612] Lustre: Evicted from MGS (at 192.168.201.144@tcp) after server handle changed from 0x57f9108b682782d7 to 0x57f9108b68278569
[ 6288.892232] Lustre: Skipped 1 previous similar message
[ 6309.775157] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 05:26:42 (1787390802)
[ 6318.658481] LustreError: lustre-OST0000-osc-ffff95ce10d02800: operation ost_setattr to node 192.168.201.144@tcp failed: rc = -19
[ 6318.674091] LustreError: Skipped 3 previous similar messages
[ 6375.089067] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6377.870478] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6460.897827] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 192.168.201.144@tcp) was lost; in progress operations using this service will fail
[ 6460.905138] LustreError: Skipped 1 previous similar message
[ 6471.209079] Lustre: Evicted from MGS (at 192.168.201.144@tcp) after server handle changed from 0x57f9108b68278569 to 0x57f9108b6827969c
[ 6479.589088] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6481.684203] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6519.118369] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6520.973153] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6649.584274] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 05:32:22 (1787391142)
[ 6713.858982] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6719.097527] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6993.058367] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 05:38:06 (1787391486)
[ 7172.587427] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 192.168.201.144@tcp) was lost; in progress operations using this service will fail
[ 7172.615857] LustreError: Skipped 1 previous similar message
[ 7172.636422] Lustre: lustre-MDT0000-mdc-ffff95ce10d02800: Connection to lustre-MDT0000 (at 192.168.201.144@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7172.675785] Lustre: Skipped 6 previous similar messages
[ 7172.688642] Lustre: Evicted from MGS (at 192.168.201.144@tcp) after server handle changed from 0x57f9108b68279a38 to 0x57f9108b683703a6
[ 7172.701026] Lustre: Skipped 1 previous similar message
[ 7172.717125] Lustre: MGC192.168.201.144@tcp: Connection restored to 192.168.201.144@tcp (at 192.168.201.144@tcp)
[ 7172.739828] Lustre: Skipped 12 previous similar messages
[ 7199.199686] Lustre: 2349:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787391656/real 1787391656]  req@ffff95ce0dbc6300 x1874208577657600/t0(0) o400->lustre-MDT0000-mdc-ffff95ce10d02800@192.168.201.144@tcp:12/10 lens 224/224 e 0 to 1 dl 1787391693 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7199.274730] Lustre: 2349:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
[ 7207.294351] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 05:41:40 (1787391700)
[ 7208.960000] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7210.804951] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 05:41:43 (1787391703)
[ 7213.293212] LustreError: lustre-MDT0000-mdc-ffff95ce10d02800: operation ldlm_enqueue to node 192.168.201.144@tcp failed: rc = -107
[ 7213.308515] LustreError: Skipped 436 previous similar messages
[ 7213.340699] LustreError: lustre-MDT0000-mdc-ffff95ce10d02800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7222.632208] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 05:41:55 (1787391715)
[ 7390.578637] LustreError: lustre-OST0000-osc-ffff95ce10d02800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7392.374796] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 05:44:45 (1787391885)
[ 7436.080050] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 05:45:28 (1787391928)
[ 7438.006144] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7439.626544] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 05:45:32 (1787391932)
[ 7441.005925] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7442.786640] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 05:45:35 (1787391935)
[ 7479.379757] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 05:46:12 (1787391972)
[ 7518.712328] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 192.168.201.144@tcp) was lost; in progress operations using this service will fail
[ 7518.730821] Lustre: Evicted from MGS (at 192.168.201.144@tcp) after server handle changed from 0x57f9108b683703a6 to 0x57f9108b68372ce9
[ 7518.741245] Lustre: Skipped 1 previous similar message
[ 7538.914812] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 05:47:11 (1787392031)
[ 7540.600787] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7542.329715] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 05:47:15 (1787392035)
[ 7544.137464] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7546.158603] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 05:47:18 (1787392038)
[ 7562.679863] Lustre: Unmounted lustre-client
[ 7564.735109] Lustre: Mounted lustre-client
[ 7567.537260] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7601.177903] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2