[ 1898.790853] LustreError: lustre-OST0000-osc-ffff96328383f800: operation ost_sync to node 192.168.203.144@tcp failed: rc = -107
[ 1898.806309] LustreError: lustre-OST0000-osc-ffff96328383f800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1898.842857] Lustre: 2379:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1906.713819] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 13:52:37 (1789062757)
[ 1909.428971] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1911.485625] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 13:52:41 (1789062761)
[ 1913.255707] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1915.141874] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 13:52:45 (1789062765)
[ 1919.663446] LustreError: lustre-MDT0000-mdc-ffff96328383f800: operation mds_reint to node 192.168.203.144@tcp failed: rc = -19
[ 1941.471325] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 1941.513199] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xc784f7068e69993f to 0xc784f7068e69b500
[ 1943.100421] Lustre: 15955:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2054.524803] LustreError: lustre-MDT0000-mdc-ffff96328383f800: operation ldlm_enqueue to node 192.168.203.144@tcp failed: rc = -19
[ 2073.569431] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 2083.763966] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xc784f7068e69b500 to 0xc784f7068e6b7267
[ 2100.322788] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 13:55:51 (1789062951)
[ 2101.065287] Lustre: *** cfs_fail_loc=305, val=0***
[ 2101.071165] Lustre: Skipped 2 previous similar messages
[ 2144.223306] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 2154.484179] Lustre: lustre-MDT0000-mdc-ffff96328383f800: Connection to lustre-MDT0000 (at 192.168.203.144@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2154.486514] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xc784f7068e6b7267 to 0xc784f7068e6b7d50
[ 2154.504836] Lustre: Skipped 11 previous similar messages
[ 2154.544205] Lustre: MGC192.168.203.144@tcp: Connection restored to 192.168.203.144@tcp (at 192.168.203.144@tcp)
[ 2154.558940] Lustre: Skipped 13 previous similar messages
[ 2170.211324] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2172.371807] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2182.878277] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 13:57:13 (1789063033)
[ 2200.576599] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 2200.624839] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xc784f7068e6b7d50 to 0xc784f7068e6b8178
[ 2200.657675] LustreError: lustre-MDT0000-mdc-ffff96328383f800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2229.130169] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 13:57:59 (1789063079)
[ 2245.337602] LustreError: lustre-OST0000-osc-ffff96328383f800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2268.148164] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 13:58:38 (1789063118)
[ 2282.357274] LustreError: lustre-MDT0000-mdc-ffff96328383f800: operation mds_reint to node 192.168.203.144@tcp failed: rc = -19
[ 2282.369913] LustreError: Skipped 1 previous similar message
[ 2302.943238] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 2313.141615] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xc784f7068e6b8178 to 0xc784f7068e6bc4c3
[ 2330.602248] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2332.954314] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2419.711496] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 2419.750685] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xc784f7068e6bc4c3 to 0xc784f7068e6d1587
[ 2439.275529] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2442.601480] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2508.714927] LustreError: lustre-MDT0000-mdc-ffff96328383f800: operation mds_close to node 192.168.203.144@tcp failed: rc = -19
[ 2508.728927] LustreError: Skipped 3 previous similar messages
[ 2527.200070] Lustre: 2379:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789063363/real 1789063363]  req@ffff9632898ac000 x1875966334433280/t0(0) o400->MGC192.168.203.144@tcp@192.168.203.144@tcp:26/25 lens 224/224 e 0 to 1 dl 1789063379 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2527.243598] Lustre: 2379:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 19 previous similar messages
[ 2527.254818] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 2537.454757] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xc784f7068e6d1587 to 0xc784f7068e6ed143
[ 2558.295293] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2560.763744] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2591.291574] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 14:04:01 (1789063441)
[ 2614.258766] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 2624.499430] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xc784f7068e6ed143 to 0xc784f7068e6f9b32
[ 2634.929812] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2661.236210] Lustre: 15955:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2676.299040] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2723.518139] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2756.063276] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 2756.074133] LustreError: Skipped 2 previous similar messages
[ 2766.256765] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xc784f7068e6fce8c to 0xc784f7068e700ba2
[ 2766.265471] Lustre: Skipped 2 previous similar messages
[ 2766.292071] Lustre: MGC192.168.203.144@tcp: Connection restored to 192.168.203.144@tcp (at 192.168.203.144@tcp)
[ 2766.302469] Lustre: Skipped 16 previous similar messages
[ 2777.516296] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2802.296274] LustreError: lustre-MDT0000-mdc-ffff96328383f800: operation mds_close to node 192.168.203.144@tcp failed: rc = -19
[ 2802.306769] LustreError: Skipped 8 previous similar messages
[ 2802.319333] Lustre: lustre-MDT0000-mdc-ffff96328383f800: Connection to lustre-MDT0000 (at 192.168.203.144@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2802.342446] Lustre: Skipped 9 previous similar messages
[ 2824.495415] Lustre: 15955:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2841.527287] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2909.827469] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2998.003384] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 14:10:48 (1789063848)
[ 3061.259880] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3063.973342] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3345.452996] LustreError: lustre-OST0000-osc-ffff96328383f800: operation ldlm_enqueue to node 192.168.203.144@tcp failed: rc = -19
[ 3345.476670] LustreError: Skipped 4 previous similar messages
[ 3370.303745] Lustre: lustre-OST0000-osc-ffff96328383f800: Connection restored to 192.168.203.144@tcp (at 192.168.203.144@tcp)
[ 3370.315269] Lustre: Skipped 8 previous similar messages
[ 3389.746412] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3391.781243] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3675.091638] Lustre: lustre-OST0000-osc-ffff96328383f800: Connection to lustre-OST0000 (at 192.168.203.144@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3675.117663] Lustre: Skipped 4 previous similar messages
[ 3714.192016] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3716.684196] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3973.472555] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 14:27:03 (1789064823)
[ 3991.007931] Lustre: 50598:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789064827/real 1789064827]  req@ffff963291239880 x1875966355324416/t0(0) o101->lustre-MDT0000-mdc-ffff96328383f800@192.168.203.144@tcp:12/10 lens 576/1152 e 0 to 1 dl 1789064843 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3991.032154] Lustre: 50598:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 3991.076368] Lustre: lustre-MDT0000-mdc-ffff96328383f800: Connection restored to 192.168.203.144@tcp (at 192.168.203.144@tcp)
[ 3991.094811] Lustre: Skipped 1 previous similar message
[ 4001.144251] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 14:27:31 (1789064851)
[ 4029.814458] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 14:28:00 (1789064880)
[ 4054.928924] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 14:28:25 (1789064905)
[ 4055.409533] Lustre: Mounted lustre-client - version 2.17.58_39_ga794951
[ 4087.272873] Lustre: 2378:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789064923/real 1789064923]  req@ffff963291a80380 x1875966355344640/t0(0) o400->lustre-MDT0000-mdc-ffff963284f5b000@192.168.203.144@tcp:12/10 lens 224/224 e 0 to 1 dl 1789064939 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4087.308402] Lustre: 2378:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4087.343509] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 4087.359780] LustreError: Skipped 3 previous similar messages
[ 4097.517239] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xc784f7068e71f157 to 0xc784f7068e876268
[ 4097.524724] Lustre: Skipped 3 previous similar messages
[ 4103.374152] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4104.868770] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4106.922511] Lustre: Unmounted lustre-client
[ 4112.836111] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 14:29:23 (1789064963)
[ 4251.103149] Lustre: 2379:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789065087/real 1789065087]  req@ffff963291a80e00 x1875966355378560/t0(0) o4->lustre-OST0000-osc-ffff96328383f800@192.168.203.144@tcp:6/4 lens 488/448 e 0 to 1 dl 1789065103 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4251.121221] Lustre: 2379:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 76 previous similar messages
[ 4282.847307] Lustre: lustre-OST0000-osc-ffff96328383f800: Connection to lustre-OST0000 (at 192.168.203.144@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4282.870875] Lustre: Skipped 14 previous similar messages
[ 4410.111592] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 14:34:20 (1789065260)
[ 4459.199454] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 14:35:09 (1789065309)
[ 4462.747209] Lustre: Unmounted lustre-client
[ 4483.409817] Lustre: Mounted lustre-client - version 2.17.58_39_ga794951
[ 4489.431479] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 14:35:40 (1789065340)
[ 4489.675703] LustreError: 56525:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4490.733149] LustreError: 56525:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4490.930722] Lustre: *** cfs_fail_loc=305, val=0***
[ 4514.696411] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 14:36:05 (1789065365)
[ 4515.187691] Lustre: Mounted lustre-client - version 2.17.58_39_ga794951
[ 4517.312889] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4527.712899] Lustre: Unmounted lustre-client
[ 4535.915670] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 14:36:26 (1789065386)
[ 4628.399920] LustreError: lustre-MDT0000-mdc-ffff963290c89000: operation mds_reint to node 192.168.203.144@tcp failed: rc = -19
[ 4628.409946] LustreError: Skipped 1 previous similar message
[ 4649.439434] Lustre: 2379:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789065485/real 1789065485]  req@ffff9632904a4380 x1875966358062336/t0(0) o400->MGC192.168.203.144@tcp@192.168.203.144@tcp:26/25 lens 224/224 e 0 to 1 dl 1789065501 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4649.495273] Lustre: 2379:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 53 previous similar messages
[ 4649.502375] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 4659.703065] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xc784f7068e877de3 to 0xc784f7068e8a36cc
[ 4659.734193] Lustre: MGC192.168.203.144@tcp: Connection restored to 192.168.203.144@tcp (at 192.168.203.144@tcp)
[ 4659.747858] Lustre: Skipped 24 previous similar messages
[ 4802.810473] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 14:40:53 (1789065653)
[ 4807.866859] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4821.475515] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 4821.492132] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xc784f7068e8a36cc to 0xc784f7068e8efab6
[ 4821.567512] LustreError: lustre-MDT0000-mdc-ffff963290c89000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4840.846140] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 14:41:31 (1789065691)
[ 4841.446846] Lustre: Mounted lustre-client - version 2.17.58_39_ga794951
[ 4849.248838] LustreError: lustre-OST0000-osc-ffff963290c89000: operation ldlm_enqueue to node 192.168.203.144@tcp failed: rc = -107
[ 4849.277401] LustreError: lustre-OST0000-osc-ffff963290c89000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4862.160768] Lustre: Unmounted lustre-client
[ 4869.392681] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 14:42:00 (1789065720)
[ 4876.175512] LustreError: lustre-MDT0000-mdc-ffff963290c89000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4876.201386] LustreError: 60917:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4884.765618] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 14:42:15 (1789065735)
[ 4885.038783] Lustre: lustre-MDT0000-mdc-ffff963290c89000: Connection to lustre-MDT0000 (at 192.168.203.144@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4885.053636] Lustre: Skipped 16 previous similar messages
[ 4885.056727] LustreError: 61622:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4889.041551] LustreError: lustre-MDT0000-mdc-ffff963290c89000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4889.058893] LustreError: 61639:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4889.063335] LustreError: 61622:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=998
[ 4889.076587] LustreError: 61622:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting
[ 4889.153692] LustreError: 61644:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4890.308203] LustreError: 61650:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4890.334096] LustreError: 61650:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4892.786774] LustreError: 61672:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4892.801233] LustreError: 61672:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4897.547043] LustreError: 61717:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4897.566822] LustreError: 61717:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4906.683707] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 14:42:37 (1789065757)
[ 4945.645762] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4947.220307] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4958.055214] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 14:43:28 (1789065808)
[ 4996.309379] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4998.115329] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5010.234256] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 14:44:20 (1789065860)
[ 5080.286600] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5082.819947] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5091.373940] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 14:45:41 (1789065941)
[ 5130.746684] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5132.433717] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5138.906191] Lustre: Unmounted lustre-client
[ 5188.447396] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5190.706044] Lustre: Mounted lustre-client - version 2.17.58_39_ga794951
[ 5196.703803] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 14:47:27 (1789066047)
[ 5199.461473] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5201.407556] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 14:47:32 (1789066052)
[ 5236.753054] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 14:48:07 (1789066087)
[ 5238.299516] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5240.320750] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 14:48:10 (1789066090)
[ 5241.840808] Lustre: *** cfs_fail_loc=805, val=0***
[ 5241.920637] Lustre: Mounted lustre-client - version 2.17.58_39_ga794951
[ 5247.691669] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5268.447422] Lustre: 2379:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789066104/real 1789066104]  req@ffff963291bfed80 x1875966360107776/t0(0) o400->lustre-MDT0000-mdc-ffff963283f94000@192.168.203.144@tcp:12/10 lens 224/224 e 0 to 1 dl 1789066120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5268.450355] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 5268.473360] Lustre: 2379:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 5268.527085] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xc784f7068e8f04b1 to 0xc784f7068e8f06cc
[ 5268.549264] Lustre: MGC192.168.203.144@tcp: Connection restored to 192.168.203.144@tcp (at 192.168.203.144@tcp)
[ 5268.561022] Lustre: Skipped 13 previous similar messages
[ 5270.510302] Lustre: 68282:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5285.804061] Lustre: Unmounted lustre-client
[ 5292.737681] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 14:49:03 (1789066143)
[ 5334.554587] LustreError: 2377:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff963285fc4000 x1875966360119680/t94489280517(94489280517) o101->lustre-MDT0000-mdc-ffff963290088000@192.168.203.144@tcp:12/10 lens 648/608 e 0 to 0 dl 1789066202 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5339.627172] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5341.017081] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5349.928350] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 14:50:00 (1789066200)
[ 5351.662909] LustreError: lustre-OST0000-osc-ffff963290088000: operation ost_write to node 192.168.203.144@tcp failed: rc = -107
[ 5351.678588] LustreError: Skipped 2 previous similar messages
[ 5351.719127] LustreError: lustre-OST0000-osc-ffff963290088000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5351.736914] Lustre: 2381:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5351.779808] LustreError: 72822:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff963290088000: namespace resource [0x240000400:0x3ee2:0x0].0x0 (ffff9632904b2e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5360.315775] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 14:50:10 (1789066210)
[ 5361.861351] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5364.096335] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 14:50:14 (1789066214)
[ 5365.586913] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5367.190961] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 14:50:18 (1789066218)
[ 5368.578955] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5370.300595] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 14:50:21 (1789066221)
[ 5371.656876] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5373.615371] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 14:50:24 (1789066224)
[ 5375.374095] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5377.060751] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 14:50:27 (1789066227)
[ 5378.466754] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5380.051583] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 14:50:30 (1789066230)
[ 5381.642169] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5383.327836] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 14:50:34 (1789066234)
[ 5384.921838] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5387.028149] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 14:50:37 (1789066237)
[ 5388.645575] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5390.948796] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 14:50:41 (1789066241)
[ 5392.701874] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5394.890220] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 14:50:45 (1789066245)
[ 5396.690308] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5398.888138] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 14:50:49 (1789066249)
[ 5402.398353] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5404.624892] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 14:50:54 (1789066254)
[ 5437.851954] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 14:51:28 (1789066288)
[ 5467.445211] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 14:51:58 (1789066318)
[ 5467.891899] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5478.008238] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 14:52:08 (1789066328)
[ 5478.493291] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5488.681662] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 14:52:19 (1789066339)
[ 5489.081097] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5498.059716] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 14:52:28 (1789066348)
[ 5498.621978] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5509.195989] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 14:52:39 (1789066359)
[ 5509.503259] Lustre: *** cfs_fail_loc=510, val=3***
[ 5520.120340] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 14:52:50 (1789066370)
[ 5520.514918] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5531.732941] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 14:53:02 (1789066382)
[ 5531.983464] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5592.544812] Lustre: lustre-OST0001-osc-ffff963290088000: Connection to lustre-OST0001 (at 192.168.203.144@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5592.577101] Lustre: Skipped 11 previous similar messages
[ 5600.283902] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 14:54:11 (1789066451)
[ 5600.726678] LustreError: 83033:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5603.638519] LustreError: lustre-MDT0000-mdc-ffff963290088000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5603.664585] LustreError: 83048:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff963290088000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff96328306dc00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5604.815137] LustreError: 83033:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5606.952093] LustreError: 83064:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5609.986561] LustreError: lustre-MDT0000-mdc-ffff963290088000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5610.000866] LustreError: 83078:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff963290088000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9632904b2e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5610.030798] LustreError: 83078:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5611.031142] LustreError: 83064:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5618.159397] LustreError: lustre-MDT0000-mdc-ffff963290088000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5618.520593] LustreError: 83119:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5621.438845] LustreError: lustre-MDT0000-mdc-ffff963290088000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5621.526710] LustreError: 83139:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5621.539813] LustreError: 83139:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5622.607293] LustreError: 83119:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5622.622870] LustreError: 83119:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff963290088000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5622.652703] LustreError: 83145:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5622.671158] LustreError: 83145:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5624.985853] LustreError: 83167:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5624.998901] LustreError: 83167:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5628.628609] LustreError: 83194:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5631.715603] LustreError: lustre-MDT0000-mdc-ffff963290088000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5631.738342] LustreError: 83209:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff963290088000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9632904f2200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5631.776103] LustreError: 83209:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5632.711436] LustreError: 83194:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5639.608591] LustreError: lustre-MDT0000-mdc-ffff963290088000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5640.137587] LustreError: 83250:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5643.149368] LustreError: lustre-MDT0000-mdc-ffff963290088000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5643.187214] LustreError: 83265:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff963290088000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9632904b2000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5643.223709] LustreError: 83265:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5644.239243] LustreError: 83250:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5647.342118] LustreError: lustre-MDT0000-mdc-ffff963290088000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5647.436316] LustreError: 83298:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5647.450245] LustreError: 83298:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5648.432320] LustreError: 83278:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff963290088000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5656.507592] LustreError: 83350:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5656.520188] LustreError: 83350:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5660.599111] LustreError: 83350:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5660.612765] LustreError: 83350:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5660.741694] LustreError: lustre-MDT0000-mdc-ffff963290088000: operation ldlm_enqueue to node 192.168.203.144@tcp failed: rc = -107
[ 5660.755524] LustreError: Skipped 8 previous similar messages
[ 5660.771588] LustreError: lustre-MDT0000-mdc-ffff963290088000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5668.897325] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 14:55:19 (1789066519)
[ 5688.644049] LustreError: 68280:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5692.727126] LustreError: 68280:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5699.379898] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 14:55:49 (1789066549)
[ 5747.891381] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 14:56:39 (1789066599)
[ 5810.915669] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 14:57:42 (1789066662)
[ 5844.538637] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 14:58:15 (1789066695)
[ 5845.138316] Lustre: Mounted lustre-client - version 2.17.58_39_ga794951
[ 5970.024659] Lustre: Unmounted lustre-client
[ 5977.946718] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 15:00:28 (1789066828)
[ 5978.583822] LustreError: 87634:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5981.031977] LustreError: lustre-OST0000-osc-ffff963290088000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5981.047252] LustreError: 87716:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff963290088000: namespace resource [0x240000400:0x3f0d:0x0].0x0 (ffff963285771100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5981.077170] LustreError: 87716:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5981.431131] LustreError: 87634:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5981.445485] Lustre: 2379:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5988.301326] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 15:00:38 (1789066838)
[ 6046.687197] Lustre: 88300:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789066843/real 1789066843]  req@ffff9632913a3b80 x1875966360288512/t0(0) o101->lustre-MDT0000-mdc-ffff963290088000@192.168.203.144@tcp:12/10 lens 328/344 e 0 to 1 dl 1789066898 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6046.710359] Lustre: 88300:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 19 previous similar messages
[ 6046.734766] Lustre: lustre-MDT0000-mdc-ffff963290088000: Connection restored to 192.168.203.144@tcp (at 192.168.203.144@tcp)
[ 6046.745662] Lustre: Skipped 18 previous similar messages
[ 6052.786404] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 15:01:43 (1789066903)
[ 6054.172214] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6055.706886] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 15:01:46 (1789066906)
[ 6118.476940] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6119.906617] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 15:02:50 (1789066970)
[ 6178.429975] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 15:03:49 (1789067029)
[ 6181.399282] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6183.027689] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 15:03:53 (1789067033)
[ 6184.300378] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6185.895561] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 15:03:56 (1789067036)
[ 6218.822586] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 15:04:29 (1789067069)
[ 6229.619287] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6254.561144] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 6254.572470] LustreError: Skipped 2 previous similar messages
[ 6260.703716] Lustre: lustre-MDT0000-mdc-ffff963290088000: Connection to lustre-MDT0000 (at 192.168.203.144@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6260.724088] Lustre: Skipped 16 previous similar messages
[ 6263.851509] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xc784f7068e8f1305 to 0xc784f7068e8f2af2
[ 6263.859201] Lustre: Skipped 2 previous similar messages
[ 6276.633261] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6278.012488] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6287.484266] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 15:05:38 (1789067138)
[ 6289.399459] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6291.191311] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 15:05:42 (1789067142)
[ 6305.774755] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 6305.804918] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xc784f7068e8f2af2 to 0xc784f7068e8f2fad
[ 6314.553531] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 15:06:05 (1789067165)
[ 6337.582735] Lustre: 68282:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6361.111677] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 15:06:52 (1789067212)
[ 6367.442866] LustreError: lustre-OST0000-osc-ffff963290088000: operation ost_setattr to node 192.168.203.144@tcp failed: rc = -19
[ 6367.458727] LustreError: Skipped 1 previous similar message
[ 6422.869572] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6427.325592] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6514.668137] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 6514.680327] LustreError: Skipped 1 previous similar message
[ 6514.690698] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xc784f7068e8f31ba to 0xc784f7068e8f4103
[ 6514.702335] Lustre: Skipped 1 previous similar message
[ 6527.975350] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6529.474993] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6563.127846] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6564.203542] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6683.207473] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 15:12:14 (1789067534)
[ 6729.094940] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6731.572385] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6963.284881] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 15:16:54 (1789067814)
[ 7108.587503] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 7108.598112] Lustre: lustre-MDT0000-mdc-ffff963290088000: Connection to lustre-MDT0000 (at 192.168.203.144@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7108.610356] LustreError: Skipped 1 previous similar message
[ 7108.626321] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xc784f7068e8f442f to 0xc784f7068e9ee72c
[ 7108.631581] Lustre: Skipped 6 previous similar messages
[ 7108.655930] Lustre: Skipped 1 previous similar message
[ 7108.669152] Lustre: MGC192.168.203.144@tcp: Connection restored to 192.168.203.144@tcp (at 192.168.203.144@tcp)
[ 7108.684905] Lustre: Skipped 12 previous similar messages
[ 7139.120214] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 15:19:50 (1789067990)
[ 7140.613929] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7142.236637] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 15:19:53 (1789067993)
[ 7144.427993] LustreError: lustre-MDT0000-mdc-ffff963290088000: operation ldlm_enqueue to node 192.168.203.144@tcp failed: rc = -107
[ 7144.438881] LustreError: Skipped 203 previous similar messages
[ 7144.453060] LustreError: lustre-MDT0000-mdc-ffff963290088000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7151.562970] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 15:20:02 (1789068002)
[ 7318.507415] LustreError: lustre-OST0000-osc-ffff963290088000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7320.107280] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 15:22:51 (1789068171)
[ 7345.119517] Lustre: 2378:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789068176/real 1789068176]  req@ffff9632908c1500 x1875966378009216/t0(0) o4->lustre-OST0000-osc-ffff963290088000@192.168.203.144@tcp:6/4 lens 4584/448 e 0 to 1 dl 1789068196 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7345.149532] Lustre: 2378:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 18 previous similar messages
[ 7363.027679] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 15:23:33 (1789068213)
[ 7364.675228] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7366.812476] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 15:23:37 (1789068217)
[ 7368.219657] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7370.067060] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 15:23:40 (1789068220)
[ 7403.839525] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 15:24:14 (1789068254)
[ 7442.419543] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 192.168.203.144@tcp) was lost; in progress operations using this service will fail
[ 7442.447670] Lustre: Evicted from MGS (at 192.168.203.144@tcp) after server handle changed from 0xc784f7068e9ee72c to 0xc784f7068e9f0efc
[ 7442.459820] Lustre: Skipped 1 previous similar message
[ 7457.890668] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 15:25:08 (1789068308)
[ 7459.561323] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7461.156788] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 15:25:12 (1789068312)
[ 7462.452228] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7463.872595] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 15:25:14 (1789068314)
[ 7472.064920] Lustre: Unmounted lustre-client
[ 7473.062860] Lustre: Mounted lustre-client - version 2.17.58_39_ga794951
[ 7475.077606] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7494.623285] Lustre: 2378:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789068330/real 1789068330]  req@ffff9632908c1180 x1875966378195456/t0(0) o400->MGC192.168.203.144@tcp@192.168.203.144@tcp:26/25 lens 224/224 e 0 to 1 dl 1789068346 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7513.938679] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2