[ 1872.632074] LustreError: lustre-OST0000-osc-ffff9924918b9800: operation ost_sync to node 192.168.206.119@tcp failed: rc = -107
[ 1872.690789] LustreError: lustre-OST0000-osc-ffff9924918b9800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1872.729515] Lustre: 2345:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.119@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1878.833674] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 19:53:07 (1789602787)
[ 1881.015105] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1882.490338] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 19:53:11 (1789602791)
[ 1884.359276] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1886.545867] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 19:53:15 (1789602795)
[ 1890.365714] LustreError: lustre-MDT0000-mdc-ffff9924918b9800: operation mds_reint to node 192.168.206.119@tcp failed: rc = -19
[ 1910.687217] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 1910.720838] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xcd0520285e6f9ba5 to 0xcd0520285e6fb671
[ 2010.486138] LustreError: lustre-MDT0000-mdc-ffff9924918b9800: operation mds_close to node 192.168.206.119@tcp failed: rc = -19
[ 2030.569335] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 2030.601570] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xcd0520285e6fb671 to 0xcd0520285e717a99
[ 2032.247460] Lustre: 15937:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.119@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2056.860462] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 19:56:05 (1789602965)
[ 2057.327476] Lustre: *** cfs_fail_loc=305, val=0***
[ 2057.329376] Lustre: Skipped 2 previous similar messages
[ 2096.102308] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 2106.410062] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xcd0520285e717a99 to 0xcd0520285e71834b
[ 2113.088435] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2114.669475] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2123.483121] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 19:57:12 (1789603032)
[ 2142.182035] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 2142.201740] Lustre: lustre-MDT0000-mdc-ffff9924918b9800: Connection to lustre-MDT0000 (at 192.168.206.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2142.212216] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xcd0520285e71834b to 0xcd0520285e7185f2
[ 2142.215902] Lustre: Skipped 12 previous similar messages
[ 2142.290501] LustreError: lustre-MDT0000-mdc-ffff9924918b9800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2163.522953] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 19:57:52 (1789603072)
[ 2178.028325] LustreError: lustre-OST0000-osc-ffff9924918b9800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2178.047831] Lustre: lustre-OST0000-osc-ffff9924918b9800: Connection restored to 192.168.206.119@tcp (at 192.168.206.119@tcp)
[ 2178.054955] Lustre: Skipped 16 previous similar messages
[ 2195.942494] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 19:58:24 (1789603104)
[ 2209.853594] LustreError: lustre-MDT0000-mdc-ffff9924918b9800: operation mds_reint to node 192.168.206.119@tcp failed: rc = -19
[ 2209.865898] LustreError: Skipped 1 previous similar message
[ 2229.228684] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 2229.277741] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xcd0520285e7185f2 to 0xcd0520285e71d2d6
[ 2245.192728] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2247.437632] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2333.535789] Lustre: 2345:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789603226/real 1789603226]  req@ffff992486620380 x1876532639086976/t0(0) o400->MGC192.168.206.119@tcp@192.168.206.119@tcp:26/25 lens 224/224 e 0 to 1 dl 1789603242 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2333.591213] Lustre: 2345:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 2333.597103] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 2333.619778] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xcd0520285e71d2d6 to 0xcd0520285e73deb7
[ 2358.978789] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2361.193285] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2426.765912] LustreError: lustre-MDT0000-mdc-ffff9924918b9800: operation mds_reint to node 192.168.206.119@tcp failed: rc = -19
[ 2426.780475] LustreError: Skipped 1 previous similar message
[ 2444.258554] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 2454.506592] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xcd0520285e73deb7 to 0xcd0520285e759716
[ 2455.786186] Lustre: 15937:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.119@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2472.091072] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2474.353509] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2502.666324] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 20:03:31 (1789603411)
[ 2525.151710] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 2534.530270] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2535.334137] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xcd0520285e759716 to 0xcd0520285e76783c
[ 2575.334890] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2611.435458] Lustre: 15937:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.119@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2622.327147] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2656.225043] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 2656.243522] LustreError: Skipped 2 previous similar messages
[ 2666.489038] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xcd0520285e76bdbe to 0xcd0520285e771ead
[ 2666.497568] Lustre: Skipped 2 previous similar messages
[ 2667.572588] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2691.006641] LustreError: lustre-MDT0000-mdc-ffff9924918b9800: operation ldlm_enqueue to node 192.168.206.119@tcp failed: rc = -19
[ 2691.018972] LustreError: Skipped 3 previous similar messages
[ 2714.610825] Lustre: 15937:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.119@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2731.583703] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2760.470286] Lustre: lustre-MDT0000-mdc-ffff9924918b9800: Connection to lustre-MDT0000 (at 192.168.206.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2760.496802] Lustre: Skipped 8 previous similar messages
[ 2787.198038] Lustre: MGC192.168.206.119@tcp: Connection restored to 192.168.206.119@tcp (at 192.168.206.119@tcp)
[ 2787.215764] Lustre: Skipped 15 previous similar messages
[ 2788.452213] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2844.154449] Lustre: 15937:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.119@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2883.699809] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 20:09:52 (1789603792)
[ 2937.841276] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2940.117889] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3229.526152] LustreError: lustre-OST0000-osc-ffff9924918b9800: operation ldlm_enqueue to node 192.168.206.119@tcp failed: rc = -19
[ 3229.547232] LustreError: Skipped 3 previous similar messages
[ 3277.065684] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3280.690327] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3561.165952] Lustre: lustre-OST0000-osc-ffff9924918b9800: Connection to lustre-OST0000 (at 192.168.206.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3561.193680] Lustre: Skipped 3 previous similar messages
[ 3584.519364] Lustre: lustre-OST0000-osc-ffff9924918b9800: Connection restored to 192.168.206.119@tcp (at 192.168.206.119@tcp)
[ 3584.532289] Lustre: Skipped 5 previous similar messages
[ 3606.290335] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3609.223298] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3856.361712] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 20:26:04 (1789604764)
[ 3874.783576] Lustre: 50592:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789604768/real 1789604768]  req@ffff9924843ee300 x1876532659189888/t0(0) o101->lustre-MDT0000-mdc-ffff9924918b9800@192.168.206.119@tcp:12/10 lens 576/1152 e 0 to 1 dl 1789604784 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3874.820059] Lustre: 50592:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 3883.209684] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 20:26:31 (1789604791)
[ 3909.198544] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 20:26:57 (1789604817)
[ 3935.256611] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 20:27:23 (1789604843)
[ 3935.818863] Lustre: Mounted lustre-client - version 2.17.58_39_g6aa67ea
[ 3967.455528] Lustre: 2346:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789604861/real 1789604861]  req@ffff99248628b800 x1876532659208832/t0(0) o400->lustre-MDT0000-mdc-ffff9924918b9800@192.168.206.119@tcp:12/10 lens 224/224 e 0 to 1 dl 1789604877 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3967.475647] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 3967.490420] Lustre: 2346:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 3967.541973] LustreError: Skipped 3 previous similar messages
[ 3967.582495] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xcd0520285e7873d8 to 0xcd0520285e8b7e1d
[ 3967.594872] Lustre: Skipped 3 previous similar messages
[ 3981.875188] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3983.504728] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3985.752423] Lustre: Unmounted lustre-client
[ 3993.112683] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 20:28:21 (1789604901)
[ 4132.319186] Lustre: 2347:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789605026/real 1789605026]  req@ffff992484f2aa00 x1876532659243520/t0(0) o4->lustre-OST0000-osc-ffff9924918b9800@192.168.206.119@tcp:6/4 lens 488/448 e 0 to 1 dl 1789605042 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4132.365559] Lustre: 2347:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 75 previous similar messages
[ 4163.999326] Lustre: lustre-OST0000-osc-ffff9924918b9800: Connection to lustre-OST0000 (at 192.168.206.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4164.028760] Lustre: Skipped 14 previous similar messages
[ 4196.846719] Lustre: lustre-OST0000-osc-ffff9924918b9800: Connection restored to 192.168.206.119@tcp (at 192.168.206.119@tcp)
[ 4196.855351] Lustre: Skipped 17 previous similar messages
[ 4292.124214] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 20:33:20 (1789605200)
[ 4341.616271] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 20:34:10 (1789605250)
[ 4344.620576] Lustre: Unmounted lustre-client
[ 4367.160224] Lustre: Mounted lustre-client - version 2.17.58_39_g6aa67ea
[ 4374.483443] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 20:34:43 (1789605283)
[ 4374.749507] LustreError: 56518:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4375.806083] LustreError: 56518:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4376.080561] Lustre: *** cfs_fail_loc=305, val=0***
[ 4400.404504] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 20:35:08 (1789605308)
[ 4400.846888] Lustre: Mounted lustre-client - version 2.17.58_39_g6aa67ea
[ 4403.273767] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4413.597701] Lustre: Unmounted lustre-client
[ 4420.210564] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 20:35:29 (1789605329)
[ 4503.501744] LustreError: lustre-MDT0000-mdc-ffff992485e32000: operation mds_reint to node 192.168.206.119@tcp failed: rc = -19
[ 4503.511866] LustreError: Skipped 1 previous similar message
[ 4522.990344] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 4523.020361] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xcd0520285e8b9569 to 0xcd0520285e8e4d5d
[ 4663.561615] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 20:39:32 (1789605572)
[ 4669.252130] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4681.700070] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 4681.721107] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xcd0520285e8e4d5d to 0xcd0520285e931266
[ 4681.769588] LustreError: lustre-MDT0000-mdc-ffff992485e32000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4698.298447] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 20:40:07 (1789605607)
[ 4698.649743] Lustre: Mounted lustre-client - version 2.17.58_39_g6aa67ea
[ 4705.768114] LustreError: lustre-OST0000-osc-ffff992485e32000: operation ldlm_enqueue to node 192.168.206.119@tcp failed: rc = -107
[ 4705.782399] LustreError: lustre-OST0000-osc-ffff992485e32000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4714.975206] Lustre: 2348:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789605609/real 1789605609]  req@ffff992490a76680 x1876532663899648/t0(0) o103->lustre-OST0000-osc-ffff992483eba000@192.168.206.119@tcp:17/18 lens 328/224 e 0 to 1 dl 1789605625 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4715.009066] Lustre: 2348:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 77 previous similar messages
[ 4716.639630] Lustre: Unmounted lustre-client
[ 4722.569970] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 20:40:31 (1789605631)
[ 4728.952680] LustreError: lustre-MDT0000-mdc-ffff992485e32000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4728.964358] LustreError: 60903:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4735.803205] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 20:40:44 (1789605644)
[ 4736.100800] LustreError: 61607:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4740.552725] LustreError: lustre-MDT0000-mdc-ffff992485e32000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4740.571038] LustreError: 61624:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4740.591372] LustreError: 61607:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=524
[ 4740.615832] LustreError: 61607:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting
[ 4740.662327] LustreError: 61629:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4741.772585] LustreError: 61635:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4741.786450] LustreError: 61635:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4744.247590] LustreError: 61657:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4744.261142] LustreError: 61657:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4748.816623] LustreError: 61702:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4748.828573] LustreError: 61702:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4758.221763] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 20:41:07 (1789605667)
[ 4766.180908] Lustre: lustre-OST0000-osc-ffff992485e32000: Connection to lustre-OST0000 (at 192.168.206.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4766.193402] Lustre: Skipped 17 previous similar messages
[ 4795.844824] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4797.320381] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4807.476557] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 20:41:56 (1789605716)
[ 4831.142091] Lustre: lustre-OST0000-osc-ffff992485e32000: Connection restored to 192.168.206.119@tcp (at 192.168.206.119@tcp)
[ 4831.156123] Lustre: Skipped 16 previous similar messages
[ 4845.829714] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4847.286257] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4857.004825] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 20:42:46 (1789605766)
[ 4919.408250] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4921.147615] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4930.537571] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 20:43:59 (1789605839)
[ 4972.196577] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4973.716528] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4980.180559] Lustre: Unmounted lustre-client
[ 5029.828294] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5032.152729] Lustre: Mounted lustre-client - version 2.17.58_39_g6aa67ea
[ 5037.705102] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 20:45:46 (1789605946)
[ 5039.869275] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5041.643620] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 20:45:50 (1789605950)
[ 5068.050902] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 20:46:16 (1789605976)
[ 5069.266594] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5070.771087] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 20:46:19 (1789605979)
[ 5071.929656] Lustre: *** cfs_fail_loc=805, val=0***
[ 5071.972095] Lustre: Mounted lustre-client - version 2.17.58_39_g6aa67ea
[ 5076.752704] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5097.951245] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 5098.010156] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xcd0520285e931d1e to 0xcd0520285e9320b3
[ 5104.694126] Lustre: Unmounted lustre-client
[ 5111.007630] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 20:47:00 (1789606020)
[ 5135.761516] Lustre: 68271:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.119@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5153.307565] LustreError: 2344:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff992487ff1180 x1876532663978112/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff9924900af800@192.168.206.119@tcp:12/10 lens 664/608 e 0 to 0 dl 1789606079 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5158.637097] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5160.625591] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5168.531899] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 20:47:57 (1789606077)
[ 5169.732379] LustreError: lustre-OST0000-osc-ffff9924900af800: operation ost_write to node 192.168.206.119@tcp failed: rc = -107
[ 5169.743134] LustreError: Skipped 2 previous similar messages
[ 5169.766181] LustreError: lustre-OST0000-osc-ffff9924900af800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5169.794189] Lustre: 2345:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.119@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5169.831569] LustreError: 72818:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9924900af800: namespace resource [0x240000400:0x3b02:0x0].0x0 (ffff992490aa6800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5177.601424] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 20:48:06 (1789606086)
[ 5178.972909] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5180.432503] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 20:48:09 (1789606089)
[ 5181.886407] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5183.451244] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 20:48:12 (1789606092)
[ 5185.041170] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5186.955503] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 20:48:15 (1789606095)
[ 5188.425909] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5190.154401] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 20:48:19 (1789606099)
[ 5191.593456] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5193.104813] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 20:48:22 (1789606102)
[ 5194.534831] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5196.130672] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 20:48:24 (1789606104)
[ 5197.669344] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5199.278589] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 20:48:28 (1789606108)
[ 5200.906824] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5202.341588] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 20:48:31 (1789606111)
[ 5204.110379] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5205.974211] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 20:48:34 (1789606114)
[ 5207.361471] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5209.139466] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 20:48:37 (1789606117)
[ 5210.840131] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5212.771816] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 20:48:41 (1789606121)
[ 5215.104497] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5216.537667] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 20:48:45 (1789606125)
[ 5247.485280] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 20:49:16 (1789606156)
[ 5277.036924] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 20:49:45 (1789606185)
[ 5277.546276] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5287.712905] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 20:49:56 (1789606196)
[ 5287.962116] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5297.563684] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 20:50:06 (1789606206)
[ 5297.811897] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5304.934975] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 20:50:14 (1789606214)
[ 5305.123771] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5312.320514] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 20:50:21 (1789606221)
[ 5312.662201] Lustre: *** cfs_fail_loc=510, val=3***
[ 5320.274965] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 20:50:29 (1789606229)
[ 5320.806961] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5380.575196] Lustre: 2345:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789606235/real 1789606235]  req@ffff99248780fb80 x1876532664020352/t0(0) o3->lustre-OST0001-osc-ffff9924900af800@192.168.206.119@tcp:6/4 lens 488/4536 e 0 to 1 dl 1789606290 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 5380.615722] Lustre: 2345:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 5380.626769] Lustre: lustre-OST0001-osc-ffff9924900af800: Connection to lustre-OST0001 (at 192.168.206.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5380.639183] Lustre: Skipped 9 previous similar messages
[ 5387.887795] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 20:51:36 (1789606296)
[ 5388.314496] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5457.275487] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 20:52:45 (1789606365)
[ 5457.743705] LustreError: 83028:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5460.623874] LustreError: lustre-MDT0000-mdc-ffff9924900af800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5460.678842] LustreError: 83042:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9924900af800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff992485981500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5460.734332] Lustre: lustre-MDT0000-mdc-ffff9924900af800: Connection restored to 192.168.206.119@tcp (at 192.168.206.119@tcp)
[ 5460.748392] Lustre: Skipped 10 previous similar messages
[ 5461.783140] LustreError: 83028:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5463.898519] LustreError: 83058:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5466.846469] LustreError: lustre-MDT0000-mdc-ffff9924900af800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5466.859973] LustreError: 83072:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9924900af800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff992490aa6f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5466.896087] LustreError: 83072:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5467.975406] LustreError: 83058:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5475.203044] LustreError: lustre-MDT0000-mdc-ffff9924900af800: operation ldlm_enqueue to node 192.168.206.119@tcp failed: rc = -107
[ 5475.208490] LustreError: Skipped 2 previous similar messages
[ 5475.219415] LustreError: lustre-MDT0000-mdc-ffff9924900af800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5475.529365] LustreError: 83113:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5478.789981] LustreError: lustre-MDT0000-mdc-ffff9924900af800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5478.860743] LustreError: 83133:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5478.875262] LustreError: 83133:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5479.623178] LustreError: 83113:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5479.632339] LustreError: 83113:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9924900af800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5480.013570] LustreError: 83139:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5480.022028] LustreError: 83139:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5482.313347] LustreError: 83161:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5482.324485] LustreError: 83161:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5485.961878] LustreError: 83188:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5488.745978] LustreError: lustre-MDT0000-mdc-ffff9924900af800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5488.763816] LustreError: 83202:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9924900af800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff99248661e700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5488.798610] LustreError: 83202:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5490.047215] LustreError: 83188:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5497.074208] LustreError: lustre-MDT0000-mdc-ffff9924900af800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5497.697469] LustreError: 83244:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5500.803759] LustreError: lustre-MDT0000-mdc-ffff9924900af800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5500.845510] LustreError: 83258:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9924900af800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff992490aa6200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5500.903627] LustreError: 83258:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5501.783169] LustreError: 83244:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5505.363617] LustreError: lustre-MDT0000-mdc-ffff9924900af800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5505.464904] LustreError: 83291:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5505.478326] LustreError: 83291:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5506.047707] LustreError: 83271:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9924900af800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5514.897770] LustreError: 83342:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5514.910334] LustreError: 83342:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5518.393143] LustreError: lustre-MDT0000-mdc-ffff9924900af800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5518.497221] LustreError: 83365:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5518.510894] LustreError: 83365:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5518.911446] LustreError: 83342:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5518.924240] LustreError: 83342:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5518.948985] LustreError: 83342:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9924900af800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5529.668889] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 20:53:58 (1789606438)
[ 5549.481280] LustreError: 70785:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5549.491523] LustreError: 70785:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5553.559160] LustreError: 70785:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5553.568444] LustreError: 70785:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5561.536513] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 20:54:30 (1789606470)
[ 5612.365075] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 20:55:21 (1789606521)
[ 5677.079934] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 20:56:25 (1789606585)
[ 5711.081437] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 20:56:59 (1789606619)
[ 5711.515909] Lustre: Mounted lustre-client - version 2.17.58_39_g6aa67ea
[ 5835.027717] Lustre: Unmounted lustre-client
[ 5841.718916] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 20:59:10 (1789606750)
[ 5842.243418] LustreError: 87669:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5844.989273] LustreError: lustre-OST0000-osc-ffff9924900af800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5845.005982] LustreError: 87756:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9924900af800: namespace resource [0x240000400:0x3b2d:0x0].0x0 (ffff992485981400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5845.020322] LustreError: 87756:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5845.191139] LustreError: 87669:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5845.196490] Lustre: 2345:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.119@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5852.310324] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 20:59:21 (1789606761)
[ 5919.427561] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 21:00:27 (1789606827)
[ 5921.081293] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5922.519237] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 21:00:31 (1789606831)
[ 5986.273895] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5988.396727] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 21:01:36 (1789606896)
[ 6045.663782] Lustre: 90079:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789606900/real 1789606900]  req@ffff99248780fb80 x1876532664170240/t0(0) o36->lustre-MDT0000-mdc-ffff9924900af800@192.168.206.119@tcp:12/10 lens 496/440 e 0 to 1 dl 1789606955 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ln.0' uid:0 gid:0 projid:4294967295
[ 6045.689515] Lustre: 90079:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 6045.697823] Lustre: lustre-MDT0000-mdc-ffff9924900af800: Connection to lustre-MDT0000 (at 192.168.206.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6045.706782] Lustre: Skipped 16 previous similar messages
[ 6047.260130] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 21:02:36 (1789606956)
[ 6049.994373] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6051.801324] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 21:02:40 (1789606960)
[ 6053.022847] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6054.426786] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 21:02:43 (1789606963)
[ 6084.995796] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 21:03:13 (1789606993)
[ 6096.451164] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6123.487634] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 6123.523903] LustreError: Skipped 2 previous similar messages
[ 6128.794955] Lustre: lustre-MDT0000-mdc-ffff9924900af800: Connection restored to 192.168.206.119@tcp (at 192.168.206.119@tcp)
[ 6128.812411] Lustre: Skipped 15 previous similar messages
[ 6133.812347] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xcd0520285e932b5d to 0xcd0520285e934653
[ 6133.822825] Lustre: Skipped 2 previous similar messages
[ 6140.323393] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6142.027376] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6152.965714] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 21:04:21 (1789607061)
[ 6155.264555] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6156.870361] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 21:04:25 (1789607065)
[ 6174.702923] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 6174.747298] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xcd0520285e934653 to 0xcd0520285e93486e
[ 6184.625198] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 21:04:53 (1789607093)
[ 6227.512413] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 21:05:36 (1789607136)
[ 6234.219616] LustreError: lustre-OST0000-osc-ffff9924900af800: operation ost_setattr to node 192.168.206.119@tcp failed: rc = -107
[ 6234.234975] LustreError: Skipped 6 previous similar messages
[ 6282.511732] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6286.645277] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6371.744562] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 6371.753163] LustreError: Skipped 1 previous similar message
[ 6381.031088] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xcd0520285e934dca to 0xcd0520285e9358c8
[ 6381.037706] Lustre: Skipped 1 previous similar message
[ 6382.386381] Lustre: 68271:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.119@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6395.590969] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6397.205329] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6430.537400] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6431.921710] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6559.711844] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 21:11:08 (1789607468)
[ 6618.815652] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6621.391395] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6865.864717] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 21:16:14 (1789607774)
[ 6995.445708] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 6995.446586] Lustre: lustre-MDT0000-mdc-ffff9924900af800: Connection to lustre-MDT0000 (at 192.168.206.119@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6995.446598] Lustre: Skipped 7 previous similar messages
[ 6995.484918] LustreError: Skipped 1 previous similar message
[ 6995.503117] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xcd0520285e936038 to 0xcd0520285ea300cd
[ 6995.519992] Lustre: Skipped 1 previous similar message
[ 6995.529624] Lustre: MGC192.168.206.119@tcp: Connection restored to 192.168.206.119@tcp (at 192.168.206.119@tcp)
[ 6995.538915] Lustre: Skipped 9 previous similar messages
[ 7027.081271] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 21:18:55 (1789607935)
[ 7028.619841] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7031.072472] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 21:18:59 (1789607939)
[ 7033.038612] LustreError: lustre-MDT0000-mdc-ffff9924900af800: operation ldlm_enqueue to node 192.168.206.119@tcp failed: rc = -107
[ 7033.051309] LustreError: Skipped 349 previous similar messages
[ 7033.074064] LustreError: lustre-MDT0000-mdc-ffff9924900af800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7040.482340] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 21:19:09 (1789607949)
[ 7207.406748] LustreError: lustre-OST0000-osc-ffff9924900af800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7209.529616] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 21:21:58 (1789608118)
[ 7235.039513] Lustre: 2345:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789608125/real 1789608125]  req@ffff99249dcb4e00 x1876532681891584/t0(0) o4->lustre-OST0000-osc-ffff9924900af800@192.168.206.119@tcp:6/4 lens 4584/448 e 0 to 1 dl 1789608145 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7235.074610] Lustre: 2345:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 27 previous similar messages
[ 7253.141977] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 21:22:41 (1789608161)
[ 7254.726784] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7257.108508] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 21:22:45 (1789608165)
[ 7258.944287] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7261.033939] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 21:22:49 (1789608169)
[ 7296.570509] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 21:23:25 (1789608205)
[ 7337.446452] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 192.168.206.119@tcp) was lost; in progress operations using this service will fail
[ 7337.490055] Lustre: Evicted from MGS (at 192.168.206.119@tcp) after server handle changed from 0xcd0520285ea300cd to 0xcd0520285ea32922
[ 7337.496831] Lustre: Skipped 1 previous similar message
[ 7348.541402] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 21:24:17 (1789608257)
[ 7349.864767] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7351.261835] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 21:24:20 (1789608260)
[ 7352.684060] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7354.528901] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 21:24:23 (1789608263)
[ 7363.605298] Lustre: Unmounted lustre-client
[ 7364.723384] Lustre: Mounted lustre-client - version 2.17.58_39_g6aa67ea
[ 7367.410285] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7394.155513] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2