[ 1934.333783] LustreError: lustre-OST0000-osc-ffff8e2e46387800: operation ost_sync to node 192.168.202.151@tcp failed: rc = -107
[ 1934.362087] LustreError: lustre-OST0000-osc-ffff8e2e46387800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1934.384707] Lustre: 2353:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.151@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1943.751826] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 00:15:28 (1788668128)
[ 1946.435356] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1948.472218] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 00:15:33 (1788668133)
[ 1950.865571] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1952.329235] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 00:15:37 (1788668137)
[ 1956.783819] LustreError: lustre-MDT0000-mdc-ffff8e2e46387800: operation mds_reint to node 192.168.202.151@tcp failed: rc = -19
[ 1977.853892] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 1977.871545] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x5fcef3c379c73dcf to 0x5fcef3c379c7594a
[ 1979.506630] Lustre: 15919:0:(mgc_request.c:1898:mgc_process_log()) MGC192.168.202.151@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2092.599135] LustreError: lustre-MDT0000-mdc-ffff8e2e46387800: operation mds_reint to node 192.168.202.151@tcp failed: rc = -19
[ 2110.944552] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 2121.138530] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x5fcef3c379c7594a to 0x5fcef3c379c8dfec
[ 2138.333313] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 00:18:43 (1788668323)
[ 2139.048349] Lustre: *** cfs_fail_loc=305, val=0***
[ 2139.053307] Lustre: Skipped 2 previous similar messages
[ 2159.816344] LustreError: lustre-OST0001-osc-ffff8e2e46387800: operation ost_connect to node 192.168.202.151@tcp failed: rc = -75
[ 2186.720424] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 2196.971418] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x5fcef3c379c8dfec to 0x5fcef3c379c8e3c0
[ 2196.986601] Lustre: MGC192.168.202.151@tcp: Connection restored to 192.168.202.151@tcp (at 192.168.202.151@tcp)
[ 2197.001142] Lustre: Skipped 13 previous similar messages
[ 2198.189375] Lustre: lustre-MDT0000-mdc-ffff8e2e46387800: Connection to lustre-MDT0000 (at 192.168.202.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2198.203566] Lustre: Skipped 11 previous similar messages
[ 2214.202616] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2216.093667] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2225.506957] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 00:20:10 (1788668410)
[ 2243.056378] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 2243.083288] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x5fcef3c379c8e3c0 to 0x5fcef3c379c8e9b6
[ 2243.203156] LustreError: lustre-MDT0000-mdc-ffff8e2e46387800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2266.191239] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 00:20:51 (1788668451)
[ 2279.384798] LustreError: lustre-OST0000-osc-ffff8e2e46387800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2298.382393] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 00:21:23 (1788668483)
[ 2312.152880] LustreError: lustre-MDT0000-mdc-ffff8e2e46387800: operation mds_reint to node 192.168.202.151@tcp failed: rc = -19
[ 2331.104557] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 2340.270357] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x5fcef3c379c8e9b6 to 0x5fcef3c379c93c35
[ 2357.930717] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2360.327202] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2446.848911] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 2446.900191] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x5fcef3c379c93c35 to 0x5fcef3c379cb056c
[ 2465.865398] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2468.188465] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2554.336677] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788668725/real 1788668725]  req@ffff8e2e45ba5500 x1875552514299264/t0(0) o400->MGC192.168.202.151@tcp@192.168.202.151@tcp:26/25 lens 224/224 e 0 to 1 dl 1788668741 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2554.373669] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 16 previous similar messages
[ 2554.377286] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 2564.585640] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x5fcef3c379cb056c to 0x5fcef3c379cce8ce
[ 2582.158704] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2584.444432] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2613.713468] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 00:26:39 (1788668799)
[ 2617.750127] LustreError: lustre-MDT0000-mdc-ffff8e2e46387800: operation mds_close to node 192.168.202.151@tcp failed: rc = -19
[ 2617.766825] LustreError: Skipped 5 previous similar messages
[ 2635.809912] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 2635.851403] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x5fcef3c379cce8ce to 0x5fcef3c379cdd59a
[ 2646.035932] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2686.711768] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2720.761253] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2763.128759] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2803.234463] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 2803.256052] LustreError: Skipped 3 previous similar messages
[ 2813.185965] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2813.479629] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x5fcef3c379ce3803 to 0x5fcef3c379ce95cd
[ 2813.494163] Lustre: Skipped 3 previous similar messages
[ 2813.518358] Lustre: MGC192.168.202.151@tcp: Connection restored to 192.168.202.151@tcp (at 192.168.202.151@tcp)
[ 2813.535212] Lustre: Skipped 17 previous similar messages
[ 2841.841928] Lustre: lustre-MDT0000-mdc-ffff8e2e46387800: Connection to lustre-MDT0000 (at 192.168.202.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2841.854757] Lustre: Skipped 9 previous similar messages
[ 2870.484399] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2958.783355] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 00:32:24 (1788669144)
[ 3013.882674] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3016.443850] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3304.248435] LustreError: lustre-OST0000-osc-ffff8e2e46387800: operation ldlm_enqueue to node 192.168.202.151@tcp failed: rc = -19
[ 3304.260756] LustreError: Skipped 8 previous similar messages
[ 3345.973760] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3348.420119] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3636.464790] Lustre: lustre-OST0000-osc-ffff8e2e46387800: Connection to lustre-OST0000 (at 192.168.202.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3636.487925] Lustre: Skipped 3 previous similar messages
[ 3661.302363] Lustre: lustre-OST0000-osc-ffff8e2e46387800: Connection restored to 192.168.202.151@tcp (at 192.168.202.151@tcp)
[ 3661.322956] Lustre: Skipped 7 previous similar messages
[ 3683.746147] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3686.812808] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3934.043394] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 00:48:39 (1788670119)
[ 3951.584328] Lustre: 50548:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788670122/real 1788670122]  req@ffff8e2e47d0b800 x1875552534732544/t0(0) o101->lustre-MDT0000-mdc-ffff8e2e46387800@192.168.202.151@tcp:12/10 lens 576/1152 e 0 to 1 dl 1788670138 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3951.642344] Lustre: 50548:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 3960.187775] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 00:49:05 (1788670145)
[ 3985.891871] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 00:49:31 (1788670171)
[ 4012.821462] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 00:49:57 (1788670197)
[ 4013.213335] Lustre: Mounted lustre-client - version 2.17.58_3_gb7bcc87
[ 4044.256653] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788670215/real 1788670215]  req@ffff8e2e47e01f80 x1875552534750720/t0(0) o400->MGC192.168.202.151@tcp@192.168.202.151@tcp:26/25 lens 224/224 e 0 to 1 dl 1788670231 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4044.282853] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4044.292696] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 4044.303085] LustreError: Skipped 2 previous similar messages
[ 4054.507063] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x5fcef3c379cf9474 to 0x5fcef3c379e4ede5
[ 4054.515799] Lustre: Skipped 2 previous similar messages
[ 4059.997421] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4061.606494] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4063.525185] Lustre: Unmounted lustre-client
[ 4069.452877] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 00:50:55 (1788670255)
[ 4206.561352] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788670377/real 1788670378]  req@ffff8e2e48043100 x1875552534783744/t0(0) o4->lustre-OST0000-osc-ffff8e2e46387800@192.168.202.151@tcp:6/4 lens 488/448 e 0 to 1 dl 1788670393 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4206.588061] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 81 previous similar messages
[ 4238.308102] Lustre: lustre-OST0000-osc-ffff8e2e46387800: Connection to lustre-OST0000 (at 192.168.202.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4238.320944] Lustre: Skipped 14 previous similar messages
[ 4271.086519] Lustre: lustre-OST0000-osc-ffff8e2e46387800: Connection restored to 192.168.202.151@tcp (at 192.168.202.151@tcp)
[ 4271.099506] Lustre: Skipped 17 previous similar messages
[ 4364.519541] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 00:55:49 (1788670549)
[ 4413.942654] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 00:56:39 (1788670599)
[ 4417.251264] Lustre: Unmounted lustre-client
[ 4440.807772] Lustre: Mounted lustre-client - version 2.17.58_3_gb7bcc87
[ 4447.524319] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 00:57:12 (1788670632)
[ 4447.930524] LustreError: 56471:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4448.987607] LustreError: 56471:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4449.183064] Lustre: *** cfs_fail_loc=305, val=0***
[ 4472.925768] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 00:57:38 (1788670658)
[ 4473.219769] Lustre: Mounted lustre-client - version 2.17.58_3_gb7bcc87
[ 4474.611370] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4484.960707] Lustre: Unmounted lustre-client
[ 4492.005405] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 00:57:57 (1788670677)
[ 4568.295190] LustreError: lustre-MDT0000-mdc-ffff8e2e580a0800: operation mds_reint to node 192.168.202.151@tcp failed: rc = -19
[ 4568.302359] LustreError: Skipped 1 previous similar message
[ 4586.808818] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788670757/real 1788670757]  req@ffff8e2e47393b80 x1875552537452672/t0(0) o400->MGC192.168.202.151@tcp@192.168.202.151@tcp:26/25 lens 224/224 e 0 to 1 dl 1788670773 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4586.847876] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 85 previous similar messages
[ 4586.864076] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 4597.108513] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x5fcef3c379e50569 to 0x5fcef3c379e7bc92
[ 4748.600264] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 01:02:13 (1788670933)
[ 4753.637147] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4769.298020] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 4769.324201] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x5fcef3c379e7bc92 to 0x5fcef3c379ec8403
[ 4769.399782] LustreError: lustre-MDT0000-mdc-ffff8e2e580a0800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4785.247978] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 01:02:50 (1788670970)
[ 4785.938382] Lustre: Mounted lustre-client - version 2.17.58_3_gb7bcc87
[ 4793.311656] LustreError: lustre-OST0000-osc-ffff8e2e580a0800: operation ldlm_enqueue to node 192.168.202.151@tcp failed: rc = -107
[ 4793.317869] LustreError: lustre-OST0000-osc-ffff8e2e580a0800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4793.323128] LustreError: 60203:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8e2e580a0800: namespace resource [0x240000400:0x3d63:0x0].0x0 (ffff8e2e473b4500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4806.428207] Lustre: Unmounted lustre-client
[ 4814.109986] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 01:03:19 (1788670999)
[ 4821.206273] LustreError: lustre-MDT0000-mdc-ffff8e2e580a0800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4821.230795] LustreError: 60874:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4831.699831] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 01:03:36 (1788671016)
[ 4831.888785] LustreError: 61573:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4836.274070] LustreError: lustre-MDT0000-mdc-ffff8e2e580a0800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4836.290144] LustreError: 61596:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4836.303473] LustreError: 61573:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=597
[ 4836.317162] LustreError: 61573:0:(import.c:761:ptlrpc_connect_import_locked()) already connecting
[ 4836.379754] LustreError: 61602:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4837.519177] LustreError: 61608:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4837.541606] LustreError: 61608:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4840.080361] LustreError: 61630:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4840.103479] LustreError: 61630:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4844.094862] LustreError: 61672:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4844.101867] LustreError: 61672:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 24 previous similar messages
[ 4856.247751] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 01:04:01 (1788671041)
[ 4861.940749] Lustre: lustre-OST0000-osc-ffff8e2e580a0800: Connection to lustre-OST0000 (at 192.168.202.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4861.961877] Lustre: Skipped 18 previous similar messages
[ 4897.227372] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4898.810237] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4911.903533] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 01:04:56 (1788671096)
[ 4938.202897] Lustre: lustre-OST0000-osc-ffff8e2e580a0800: Connection restored to 192.168.202.151@tcp (at 192.168.202.151@tcp)
[ 4938.221601] Lustre: Skipped 17 previous similar messages
[ 4955.022660] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4957.105775] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4970.303298] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 01:05:55 (1788671155)
[ 5036.945373] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5039.092469] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5048.529787] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 01:07:13 (1788671233)
[ 5087.997956] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5089.632557] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5096.658496] Lustre: Unmounted lustre-client
[ 5154.951220] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5157.706946] Lustre: Mounted lustre-client - version 2.17.58_3_gb7bcc87
[ 5165.374856] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 01:09:10 (1788671350)
[ 5167.957645] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5169.796464] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 01:09:15 (1788671355)
[ 5200.062966] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 01:09:45 (1788671385)
[ 5201.745382] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5203.850526] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 01:09:48 (1788671388)
[ 5205.128086] Lustre: *** cfs_fail_loc=805, val=0***
[ 5205.176942] Lustre: Mounted lustre-client - version 2.17.58_3_gb7bcc87
[ 5210.735543] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5232.096423] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788671402/real 1788671402]  req@ffff8e2e446a9500 x1875552539500800/t0(0) o400->lustre-MDT0000-mdc-ffff8e2e5a663800@192.168.202.151@tcp:12/10 lens 224/224 e 0 to 1 dl 1788671418 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5232.148626] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5232.162740] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 5232.195218] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x5fcef3c379ec8e36 to 0x5fcef3c379ec9051
[ 5234.096471] Lustre: 68248:0:(mgc_request.c:1898:mgc_process_log()) MGC192.168.202.151@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5253.230701] Lustre: Unmounted lustre-client
[ 5261.504560] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 01:10:46 (1788671446)
[ 5288.519617] LustreError: 2351:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8e2e47df4000 x1875552539514368/t90194313221(90194313221) o101->lustre-MDT0000-mdc-ffff8e2e5a663800@192.168.202.151@tcp:12/10 lens 664/608 e 0 to 0 dl 1788671491 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5302.602889] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5304.256542] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5314.074418] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 01:11:39 (1788671499)
[ 5315.540299] LustreError: lustre-OST0000-osc-ffff8e2e5a663800: operation ost_write to node 192.168.202.151@tcp failed: rc = -107
[ 5315.552955] LustreError: Skipped 2 previous similar messages
[ 5315.577778] LustreError: lustre-OST0000-osc-ffff8e2e5a663800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5315.590584] Lustre: 2353:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.151@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5315.621511] LustreError: 72786:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8e2e5a663800: namespace resource [0x240000400:0x3dc2:0x0].0x0 (ffff8e2e478bc900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5315.637495] LustreError: 72786:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5325.082653] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 01:11:50 (1788671510)
[ 5326.635641] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5328.596462] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 01:11:53 (1788671513)
[ 5330.157622] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5332.333728] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 01:11:57 (1788671517)
[ 5333.458551] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5335.235242] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 01:12:00 (1788671520)
[ 5336.992893] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5339.291619] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 01:12:04 (1788671524)
[ 5341.582837] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5344.075158] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 01:12:08 (1788671528)
[ 5345.943478] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5347.565575] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 01:12:13 (1788671533)
[ 5349.398230] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5351.326263] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 01:12:16 (1788671536)
[ 5352.665858] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5354.552264] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 01:12:20 (1788671540)
[ 5356.588350] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5358.574810] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 01:12:23 (1788671543)
[ 5360.287455] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5362.768648] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 01:12:27 (1788671547)
[ 5364.513297] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5366.718274] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 01:12:31 (1788671551)
[ 5369.503831] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5371.603812] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 01:12:36 (1788671556)
[ 5410.793778] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 01:13:15 (1788671595)
[ 5442.103561] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 01:13:47 (1788671627)
[ 5442.713625] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5453.762234] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 01:13:59 (1788671639)
[ 5454.093472] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5466.350879] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 01:14:11 (1788671651)
[ 5466.709316] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5475.715359] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 01:14:21 (1788671661)
[ 5476.019172] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5484.812131] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 01:14:30 (1788671670)
[ 5485.243670] Lustre: *** cfs_fail_loc=510, val=3***
[ 5494.480861] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 01:14:39 (1788671679)
[ 5494.973544] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5555.168413] Lustre: lustre-OST0001-osc-ffff8e2e5a663800: Connection to lustre-OST0001 (at 192.168.202.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5555.208535] Lustre: Skipped 7 previous similar messages
[ 5563.229353] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 01:15:48 (1788671748)
[ 5563.718275] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5634.048623] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 01:16:58 (1788671818)
[ 5634.603691] LustreError: 83008:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5638.056167] LustreError: lustre-MDT0000-mdc-ffff8e2e5a663800: operation ldlm_enqueue to node 192.168.202.151@tcp failed: rc = -107
[ 5638.073742] LustreError: lustre-MDT0000-mdc-ffff8e2e5a663800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5638.096756] LustreError: 83022:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8e2e5a663800: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 5638.122181] LustreError: 83022:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e2e5a663800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8e2e4780c800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5638.144455] Lustre: lustre-MDT0000-mdc-ffff8e2e5a663800: Connection restored to 192.168.202.151@tcp (at 192.168.202.151@tcp)
[ 5638.150885] Lustre: Skipped 8 previous similar messages
[ 5638.680803] LustreError: 83008:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5640.827366] LustreError: 83038:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5643.815048] LustreError: lustre-MDT0000-mdc-ffff8e2e5a663800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5643.822922] LustreError: 83052:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e2e5a663800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8e2e473b4f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5643.836676] LustreError: 83052:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5644.904131] LustreError: 83038:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5651.876214] LustreError: lustre-MDT0000-mdc-ffff8e2e5a663800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5652.245218] LustreError: 83094:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5655.259229] LustreError: lustre-MDT0000-mdc-ffff8e2e5a663800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5655.301312] LustreError: 83113:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5655.306064] LustreError: 83113:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5656.328159] LustreError: 83094:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5656.340697] LustreError: 83094:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8e2e5a663800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5656.355930] LustreError: 83094:0:(file.c:251:ll_close_inode_openhandle()) Skipped 2 previous similar messages
[ 5656.374183] LustreError: 83119:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5656.382386] LustreError: 83119:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5658.801837] LustreError: 83142:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5658.813394] LustreError: 83142:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5662.375958] LustreError: 83169:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5665.147861] LustreError: lustre-MDT0000-mdc-ffff8e2e5a663800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5665.165531] LustreError: 83183:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e2e5a663800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8e2e473b4d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5665.199557] LustreError: 83183:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5666.384642] LustreError: 83169:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5674.059480] LustreError: lustre-MDT0000-mdc-ffff8e2e5a663800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5674.581566] LustreError: 83225:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5677.944571] LustreError: lustre-MDT0000-mdc-ffff8e2e5a663800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5677.964669] LustreError: 83240:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8e2e5a663800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8e2e47e98600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5678.025723] LustreError: 83240:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5678.656146] LustreError: 83225:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5682.119424] LustreError: lustre-MDT0000-mdc-ffff8e2e5a663800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5682.246483] LustreError: 83273:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5682.274851] LustreError: 83273:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5682.841117] LustreError: 83253:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8e2e5a663800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5691.609165] LustreError: 83325:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5691.631716] LustreError: 83325:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5695.080300] LustreError: lustre-MDT0000-mdc-ffff8e2e5a663800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5695.226091] LustreError: 83347:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5695.256733] LustreError: 83347:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5695.626151] LustreError: 83325:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5695.630784] LustreError: 83325:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5695.636836] LustreError: 83325:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8e2e5a663800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5708.602564] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 01:18:13 (1788671893)
[ 5728.830214] LustreError: 68246:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5728.847584] LustreError: 68246:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5732.833525] LustreError: 68246:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5732.850195] LustreError: 68246:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5740.849388] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 01:18:46 (1788671926)
[ 5793.784272] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 01:19:38 (1788671978)
[ 5852.128160] Lustre: 85504:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788671982/real 1788671982]  req@ffff8e2e48214700 x1875552539644928/t0(0) o101->lustre-MDT0000-mdc-ffff8e2e5a663800@192.168.202.151@tcp:12/10 lens 576/1152 e 0 to 1 dl 1788672038 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0 projid:0
[ 5852.156549] Lustre: 85504:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 18 previous similar messages
[ 5859.765464] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 01:20:45 (1788672045)
[ 5894.929935] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 01:21:20 (1788672080)
[ 5895.647757] Lustre: Mounted lustre-client - version 2.17.58_3_gb7bcc87
[ 6019.447912] Lustre: Unmounted lustre-client
[ 6027.250432] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 01:23:32 (1788672212)
[ 6028.062962] LustreError: 87650:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6031.390245] LustreError: lustre-OST0000-osc-ffff8e2e5a663800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6031.403529] LustreError: 87737:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8e2e5a663800: namespace resource [0x240000400:0x3ded:0x0].0x0 (ffff8e2e473b4b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6031.429203] LustreError: 87737:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6031.641281] LustreError: 87650:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6031.646888] Lustre: 2355:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.151@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6039.540911] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 01:23:44 (1788672224)
[ 6105.717558] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 01:24:51 (1788672291)
[ 6107.546342] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6109.880611] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 01:24:54 (1788672294)
[ 6167.525299] Lustre: lustre-MDT0000-mdc-ffff8e2e5a663800: Connection to lustre-MDT0000 (at 192.168.202.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6167.546805] Lustre: Skipped 15 previous similar messages
[ 6174.172644] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6175.594324] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 01:26:01 (1788672361)
[ 6234.602287] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 01:27:00 (1788672420)
[ 6236.506576] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6238.182252] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 01:27:03 (1788672423)
[ 6239.841840] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6241.393877] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 01:27:07 (1788672427)
[ 6277.287774] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 01:27:42 (1788672462)
[ 6290.679793] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6320.116332] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 6320.132777] LustreError: Skipped 2 previous similar messages
[ 6320.165257] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x5fcef3c379ec9ca6 to 0x5fcef3c379ecb511
[ 6320.171386] Lustre: Skipped 2 previous similar messages
[ 6320.180114] Lustre: MGC192.168.202.151@tcp: Connection restored to 192.168.202.151@tcp (at 192.168.202.151@tcp)
[ 6320.188810] Lustre: Skipped 15 previous similar messages
[ 6321.544603] Lustre: 68248:0:(mgc_request.c:1898:mgc_process_log()) MGC192.168.202.151@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6344.797634] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6346.760476] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6358.954393] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 01:29:04 (1788672544)
[ 6361.703279] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6363.854125] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 01:29:09 (1788672549)
[ 6377.452571] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 6377.515947] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x5fcef3c379ecb511 to 0x5fcef3c379ecb867
[ 6392.364815] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 01:29:37 (1788672577)
[ 6424.816464] Lustre: 68248:0:(mgc_request.c:1898:mgc_process_log()) MGC192.168.202.151@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6444.310227] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 01:30:29 (1788672629)
[ 6450.195458] LustreError: lustre-OST0000-osc-ffff8e2e5a663800: operation ost_setattr to node 192.168.202.151@tcp failed: rc = -19
[ 6450.202579] LustreError: Skipped 9 previous similar messages
[ 6509.312225] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6515.198834] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6602.720180] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788672773/real 1788672773]  req@ffff8e2e71682300 x1875552546266880/t0(0) o400->MGC192.168.202.151@tcp@192.168.202.151@tcp:26/25 lens 224/224 e 0 to 1 dl 1788672789 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6602.747499] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 6602.762301] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 6602.776612] LustreError: Skipped 1 previous similar message
[ 6612.969456] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x5fcef3c379ecbbd9 to 0x5fcef3c379ecca2d
[ 6612.980278] Lustre: Skipped 1 previous similar message
[ 6614.314950] Lustre: 68248:0:(mgc_request.c:1898:mgc_process_log()) MGC192.168.202.151@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6632.466820] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6634.288957] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6673.953972] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6675.558618] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6826.009978] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 01:36:51 (1788673011)
[ 6834.707074] Lustre: lustre-OST0000-osc-ffff8e2e5a663800: Connection to lustre-OST0000 (at 192.168.202.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6834.731529] Lustre: Skipped 7 previous similar messages
[ 6892.436209] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6896.076821] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7176.264478] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 01:42:41 (1788673361)
[ 7329.776491] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 7329.789203] LustreError: Skipped 1 previous similar message
[ 7329.832574] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x5fcef3c379ecce5c to 0x5fcef3c379fc7159
[ 7329.839566] Lustre: Skipped 1 previous similar message
[ 7329.845202] Lustre: MGC192.168.202.151@tcp: Connection restored to 192.168.202.151@tcp (at 192.168.202.151@tcp)
[ 7329.851229] Lustre: Skipped 9 previous similar messages
[ 7366.707623] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 01:45:52 (1788673552)
[ 7368.953853] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7370.683700] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 01:45:56 (1788673556)
[ 7372.752385] LustreError: lustre-MDT0000-mdc-ffff8e2e5a663800: operation ldlm_enqueue to node 192.168.202.151@tcp failed: rc = -107
[ 7372.771012] LustreError: Skipped 455 previous similar messages
[ 7372.787139] LustreError: lustre-MDT0000-mdc-ffff8e2e5a663800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7379.646969] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 01:46:05 (1788673565)
[ 7546.868832] LustreError: lustre-OST0000-osc-ffff8e2e5a663800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7549.263266] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 01:48:54 (1788673734)
[ 7575.521080] Lustre: 2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788673742/real 1788673742]  req@ffff8e2e460dc000 x1875552557454208/t0(0) o4->lustre-OST0000-osc-ffff8e2e5a663800@192.168.202.151@tcp:6/4 lens 4584/448 e 0 to 1 dl 1788673762 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7575.558204] Lustre: 2352:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 7575.570386] Lustre: lustre-OST0000-osc-ffff8e2e5a663800: Connection to lustre-OST0000 (at 192.168.202.151@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7575.579884] Lustre: Skipped 3 previous similar messages
[ 7594.719081] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 01:49:39 (1788673779)
[ 7596.842704] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7598.324882] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 01:49:44 (1788673784)
[ 7600.092665] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7601.915217] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 01:49:47 (1788673787)
[ 7638.665141] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 01:50:23 (1788673823)
[ 7677.947146] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 192.168.202.151@tcp) was lost; in progress operations using this service will fail
[ 7677.970363] Lustre: Evicted from MGS (at 192.168.202.151@tcp) after server handle changed from 0x5fcef3c379fc7159 to 0x5fcef3c379fc9850
[ 7677.983557] Lustre: Skipped 1 previous similar message
[ 7698.770830] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 01:51:24 (1788673884)
[ 7700.508772] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7702.853988] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 01:51:27 (1788673887)
[ 7704.588218] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7706.464601] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 01:51:31 (1788673891)
[ 7718.991539] Lustre: Unmounted lustre-client
[ 7721.287365] Lustre: Mounted lustre-client - version 2.17.58_3_gb7bcc87
[ 7724.928476] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7748.337764] Lustre: 106787:0:(mgc_request.c:1898:mgc_process_log()) MGC192.168.202.151@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 7766.833688] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2