[ 1631.208506] LustreError: lustre-OST0000-osc-ffff91fcd284f800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1631.214607] Lustre: 2364:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.142@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1633.743924] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 04:22:45 (1789546965)
[ 1634.555689] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1635.200344] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 04:22:46 (1789546966)
[ 1635.868992] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1636.553121] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 04:22:47 (1789546967)
[ 1656.803855] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 192.168.203.142@tcp) was lost; in progress operations using this service will fail
[ 1656.819506] Lustre: Evicted from MGS (at 192.168.203.142@tcp) after server handle changed from 0xb908dd7c30a0c5dc to 0xb908dd7c30a0e5b0
[ 1746.419430] LustreError: lustre-MDT0000-mdc-ffff91fcd284f800: operation mds_close to node 192.168.203.142@tcp failed: rc = -19
[ 1746.423819] LustreError: Skipped 1 previous similar message
[ 1763.300984] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 192.168.203.142@tcp) was lost; in progress operations using this service will fail
[ 1763.317908] Lustre: Evicted from MGS (at 192.168.203.142@tcp) after server handle changed from 0xb908dd7c30a0e5b0 to 0xb908dd7c30a56aaf
[ 1766.019411] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 04:24:57 (1789547097)
[ 1766.200054] Lustre: *** cfs_fail_loc=305, val=0***
[ 1766.202917] Lustre: Skipped 4 previous similar messages
[ 1799.143589] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 192.168.203.142@tcp) was lost; in progress operations using this service will fail
[ 1799.152555] Lustre: Evicted from MGS (at 192.168.203.142@tcp) after server handle changed from 0xb908dd7c30a56aaf to 0xb908dd7c30a56dc6
[ 1804.633717] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1805.263277] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1808.848176] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 04:25:40 (1789547140)
[ 1814.502108] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 192.168.203.142@tcp) was lost; in progress operations using this service will fail
[ 1814.511261] Lustre: Evicted from MGS (at 192.168.203.142@tcp) after server handle changed from 0xb908dd7c30a56dc6 to 0xb908dd7c30a572c7
[ 1814.520363] LustreError: lustre-MDT0000-mdc-ffff91fcd284f800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 1822.559571] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 04:25:53 (1789547153)
[ 1827.814646] LustreError: lustre-OST0000-osc-ffff91fcd284f800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1834.900050] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 04:26:06 (1789547166)
[ 1846.030992] LustreError: lustre-MDT0000-mdc-ffff91fcd284f800: operation mds_reint to node 192.168.203.142@tcp failed: rc = -19
[ 1846.034070] LustreError: Skipped 1 previous similar message
[ 1860.580256] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 192.168.203.142@tcp) was lost; in progress operations using this service will fail
[ 1860.587730] Lustre: Evicted from MGS (at 192.168.203.142@tcp) after server handle changed from 0xb908dd7c30a572c7 to 0xb908dd7c30a6098d
[ 1868.601174] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1869.407404] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1931.922501] LustreError: lustre-MDT0000-mdc-ffff91fcd284f800: operation mds_reint to node 192.168.203.142@tcp failed: rc = -19
[ 1947.618197] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 192.168.203.142@tcp) was lost; in progress operations using this service will fail
[ 1947.645629] Lustre: Evicted from MGS (at 192.168.203.142@tcp) after server handle changed from 0xb908dd7c30a6098d to 0xb908dd7c30a95fda
[ 1951.245664] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 1952.121672] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2029.538530] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 192.168.203.142@tcp) was lost; in progress operations using this service will fail
[ 2029.546559] Lustre: Evicted from MGS (at 192.168.203.142@tcp) after server handle changed from 0xb908dd7c30a95fda to 0xb908dd7c30ad2961
[ 2037.231159] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2038.037748] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2061.952508] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 04:29:53 (1789547393)
[ 2064.037958] LustreError: lustre-MDT0000-mdc-ffff91fcd284f800: operation mds_reint to node 192.168.203.142@tcp failed: rc = -19
[ 2064.041949] LustreError: Skipped 1 previous similar message
[ 2079.812505] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2096.098685] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 192.168.203.142@tcp) was lost; in progress operations using this service will fail
[ 2096.102244] LustreError: Skipped 1 previous similar message
[ 2096.104534] Lustre: 15925:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.142@tcp: IR log lustre-cliir failed, not fatal: rc = -5
[ 2096.108639] Lustre: Evicted from MGS (at 192.168.203.142@tcp) after server handle changed from 0xb908dd7c30aeb9c6 to 0xb908dd7c30aec82f
[ 2096.112044] Lustre: Skipped 1 previous similar message
[ 2097.213873] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2118.832671] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2120.696563] Lustre: MGC192.168.203.142@tcp: Connection restored to 192.168.203.142@tcp (at 192.168.203.142@tcp)
[ 2120.698821] Lustre: Skipped 26 previous similar messages
[ 2129.780862] Lustre: lustre-MDT0000-mdc-ffff91fcd284f800: Connection to lustre-MDT0000 (at 192.168.203.142@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2129.787681] Lustre: Skipped 18 previous similar messages
[ 2145.529293] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2182.386749] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2223.946233] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2225.123701] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 192.168.203.142@tcp) was lost; in progress operations using this service will fail
[ 2225.129097] LustreError: Skipped 3 previous similar messages
[ 2225.137420] Lustre: Evicted from MGS (at 192.168.203.142@tcp) after server handle changed from 0xb908dd7c30b11aa3 to 0xb908dd7c30b266c1
[ 2225.141753] Lustre: Skipped 3 previous similar messages
[ 2292.111520] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 04:33:43 (1789547623)
[ 2325.298487] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2325.914880] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2634.530082] LustreError: lustre-OST0000-osc-ffff91fcd284f800: operation ldlm_enqueue to node 192.168.203.142@tcp failed: rc = -19
[ 2634.534032] LustreError: Skipped 7 previous similar messages
[ 2657.000661] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2657.900182] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2967.418298] Lustre: lustre-OST0000-osc-ffff91fcd284f800: Connection to lustre-OST0000 (at 192.168.203.142@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2967.431676] Lustre: Skipped 5 previous similar messages
[ 2989.590922] Lustre: lustre-OST0000-osc-ffff91fcd284f800: Connection restored to 192.168.203.142@tcp (at 192.168.203.142@tcp)
[ 2989.598600] Lustre: Skipped 11 previous similar messages
[ 3003.309739] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3005.379781] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3261.385942] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 04:49:52 (1789548592)
[ 3278.816429] Lustre: 50660:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789548594/real 1789548594]  req@ffff91fcc6166680 x1876474464244096/t0(0) o101->lustre-MDT0000-mdc-ffff91fcd284f800@192.168.203.142@tcp:12/10 lens 576/1152 e 0 to 1 dl 1789548610 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3278.840603] Lustre: 50660:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 3283.036052] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 04:50:14 (1789548614)
[ 3303.106231] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 04:50:34 (1789548634)
[ 3323.248973] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 04:50:54 (1789548654)
[ 3323.509796] Lustre: Mounted lustre-client - version 2.17.58_39_g897064a
[ 3354.600495] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 192.168.203.142@tcp) was lost; in progress operations using this service will fail
[ 3354.606844] LustreError: Skipped 1 previous similar message
[ 3354.615783] Lustre: Evicted from MGS (at 192.168.203.142@tcp) after server handle changed from 0xb908dd7c30b42054 to 0xb908dd7c30dc31df
[ 3354.621403] Lustre: Skipped 1 previous similar message
[ 3355.616419] Lustre: 2364:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789548671/real 1789548671]  req@ffff91fcc5eb5180 x1876474464261632/t0(0) o400->lustre-MDT0000-mdc-ffff91fcc60f6800@192.168.203.142@tcp:12/10 lens 224/224 e 0 to 1 dl 1789548687 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3355.632410] Lustre: 2364:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 3357.818302] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3358.588475] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3360.154215] Lustre: Unmounted lustre-client
[ 3363.505562] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 04:51:34 (1789548694)
[ 3513.312136] Lustre: 2364:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789548829/real 1789548829]  req@ffff91fcc5eb7800 x1876474464292480/t0(0) o4->lustre-OST0000-osc-ffff91fcd284f800@192.168.203.142@tcp:6/4 lens 488/448 e 0 to 1 dl 1789548845 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 3513.330233] Lustre: 2364:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 78 previous similar messages
[ 3578.656543] Lustre: lustre-OST0000-osc-ffff91fcd284f800: Connection to lustre-OST0000 (at 192.168.203.142@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3578.667779] Lustre: Skipped 17 previous similar messages
[ 3594.223516] Lustre: lustre-OST0000-osc-ffff91fcd284f800: Connection restored to 192.168.203.142@tcp (at 192.168.203.142@tcp)
[ 3594.231801] Lustre: Skipped 19 previous similar messages
[ 3650.004236] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 04:56:21 (1789548981)
[ 3694.679624] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 04:57:05 (1789549025)
[ 3696.662483] Lustre: Unmounted lustre-client
[ 3710.271788] Lustre: Mounted lustre-client - version 2.17.58_39_g897064a
[ 3713.724627] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 04:57:24 (1789549044)
[ 3713.855635] LustreError: 56576:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 3714.897101] LustreError: 56576:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 3714.950268] Lustre: *** cfs_fail_loc=305, val=0***
[ 3733.806156] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 04:57:45 (1789549065)
[ 3734.010844] Lustre: Mounted lustre-client - version 2.17.58_39_g897064a
[ 3734.900185] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 3745.215188] Lustre: Unmounted lustre-client
[ 3748.758413] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 04:57:59 (1789549079)
[ 3795.578494] LustreError: lustre-MDT0000-mdc-ffff91fcd28a5000: operation mds_reint to node 192.168.203.142@tcp failed: rc = -19
[ 3795.583609] LustreError: Skipped 1 previous similar message
[ 3810.791760] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 192.168.203.142@tcp) was lost; in progress operations using this service will fail
[ 3810.802454] Lustre: Evicted from MGS (at 192.168.203.142@tcp) after server handle changed from 0xb908dd7c30dc4b31 to 0xb908dd7c30df166c
[ 3879.583853] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 05:00:10 (1789549210)
[ 3882.014752] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 3892.045775] LustreError: lustre-MDT0000-mdc-ffff91fcd28a5000: operation mds_statfs to node 192.168.203.142@tcp failed: rc = -107
[ 3892.069623] LustreError: lustre-MDT0000-mdc-ffff91fcd28a5000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3892.707276] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 192.168.203.142@tcp) was lost; in progress operations using this service will fail
[ 3892.715522] Lustre: Evicted from MGS (at 192.168.203.142@tcp) after server handle changed from 0xb908dd7c30df166c to 0xb908dd7c30e3b947
[ 3896.520775] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 05:00:27 (1789549227)
[ 3896.736891] Lustre: Mounted lustre-client - version 2.17.58_39_g897064a
[ 3901.930526] LustreError: lustre-OST0000-osc-ffff91fcd28a5000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 3913.184184] Lustre: 2365:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789549229/real 1789549229]  req@ffff91fcd0cdea00 x1876474468917888/t0(0) o103->lustre-OST0000-osc-ffff91fcd02db000@192.168.203.142@tcp:17/18 lens 328/224 e 0 to 1 dl 1789549245 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 3913.196325] Lustre: 2365:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 67 previous similar messages
[ 3914.191083] Lustre: Unmounted lustre-client
[ 3917.450796] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 05:00:48 (1789549248)
[ 3922.763178] LustreError: lustre-MDT0000-mdc-ffff91fcd28a5000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3922.770944] LustreError: 60934:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 3926.316227] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 05:00:57 (1789549257)
[ 3926.408386] LustreError: 61635:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 3930.054648] LustreError: lustre-MDT0000-mdc-ffff91fcd28a5000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3930.060115] LustreError: 61664:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 3930.068172] LustreError: 61635:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1345
[ 3930.075661] LustreError: 61635:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting
[ 3930.094184] LustreError: 61669:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3931.141555] LustreError: 61675:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3931.148193] LustreError: 61675:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 3933.282455] LustreError: 61698:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3933.286890] LustreError: 61698:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 3937.513805] LustreError: 61742:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 3937.518978] LustreError: 61742:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 3943.877559] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 05:01:15 (1789549275)
[ 3967.146393] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3967.851943] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3972.851712] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 05:01:44 (1789549304)
[ 3994.942236] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3995.625981] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4000.575787] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 05:02:11 (1789549331)
[ 4047.449994] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4048.068717] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4051.951635] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 05:03:03 (1789549383)
[ 4073.028533] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4073.627139] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4076.085287] Lustre: Unmounted lustre-client
[ 4103.580427] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4105.203564] Lustre: Mounted lustre-client - version 2.17.58_39_g897064a
[ 4108.029297] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 05:03:59 (1789549439)
[ 4109.045287] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4109.704383] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 05:04:01 (1789549441)
[ 4123.051482] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 05:04:14 (1789549454)
[ 4123.720426] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4124.419983] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 05:04:15 (1789549455)
[ 4124.962313] Lustre: *** cfs_fail_loc=805, val=0***
[ 4124.983804] Lustre: Mounted lustre-client - version 2.17.58_39_g897064a
[ 4127.141526] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4144.013242] LustreError: lustre-MDT0000-mdc-ffff91fcd00a2000: operation ldlm_enqueue to node 192.168.203.142@tcp failed: rc = -107
[ 4144.018536] LustreError: Skipped 2 previous similar messages
[ 4144.026626] LustreError: 2361:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff91fcd00a2000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff91fcc6891300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4144.503663] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 192.168.203.142@tcp) was lost; in progress operations using this service will fail
[ 4144.512678] Lustre: Evicted from MGS (at 192.168.203.142@tcp) after server handle changed from 0xb908dd7c30e3c33b to 0xb908dd7c30e3c92a
[ 4144.528094] Lustre: Unmounted lustre-client
[ 4147.775866] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 05:04:39 (1789549479)
[ 4164.602842] LustreError: 2361:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff91fcc785b800 x1876474468984192/t94489280515(94489280515) o101->lustre-MDT0000-mdc-ffff91fcd28ce800@192.168.203.142@tcp:12/10 lens 664/608 e 0 to 0 dl 1789549512 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 4169.539903] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4170.222747] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4174.561452] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 05:05:05 (1789549505)
[ 4175.239338] LustreError: lustre-OST0000-osc-ffff91fcd28ce800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4175.247760] Lustre: 2364:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.142@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4175.283961] LustreError: 72825:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff91fcd28ce800: namespace resource [0x240000400:0x6f02:0x0].0x0 (ffff91fcc6a06200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4175.296543] LustreError: 72825:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4179.398529] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 05:05:10 (1789549510)
[ 4180.094380] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 4180.789082] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 05:05:12 (1789549512)
[ 4181.395621] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 4182.094106] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 05:05:13 (1789549513)
[ 4182.771656] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 4183.513136] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 05:05:14 (1789549514)
[ 4184.182795] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 4184.904156] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 05:05:16 (1789549516)
[ 4185.512437] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 4186.194042] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 05:05:17 (1789549517)
[ 4186.785907] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 4187.467970] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 05:05:18 (1789549518)
[ 4188.093281] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 4188.798435] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 05:05:20 (1789549520)
[ 4189.433991] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 4190.154531] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 05:05:21 (1789549521)
[ 4190.767135] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 4191.465695] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 05:05:22 (1789549522)
[ 4192.046149] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 4192.703186] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 05:05:24 (1789549524)
[ 4193.271350] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 4193.910926] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 05:05:25 (1789549525)
[ 4194.980693] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 4195.650327] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 05:05:26 (1789549526)
[ 4200.420056] Lustre: lustre-MDT0000-mdc-ffff91fcd28ce800: Connection to lustre-MDT0000 (at 192.168.203.142@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4200.426151] Lustre: Skipped 23 previous similar messages
[ 4200.427639] Lustre: MGC192.168.203.142@tcp: Connection restored to 192.168.203.142@tcp (at 192.168.203.142@tcp)
[ 4200.431279] Lustre: Skipped 25 previous similar messages
[ 4205.540396] Lustre: 68300:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.142@tcp: IR log lustre-cliir failed, not fatal: rc = -5
[ 4207.703198] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 05:05:38 (1789549538)
[ 4231.367799] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 05:06:02 (1789549562)
[ 4231.490812] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4237.556705] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 05:06:08 (1789549568)
[ 4237.656760] Lustre: *** cfs_fail_loc=51b, val=4***
[ 4243.963837] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 05:06:15 (1789549575)
[ 4244.088554] Lustre: *** cfs_fail_loc=50f, val=3***
[ 4247.813351] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 05:06:19 (1789549579)
[ 4247.887978] Lustre: *** cfs_fail_loc=50f, val=4***
[ 4251.507901] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 05:06:22 (1789549582)
[ 4251.597604] Lustre: *** cfs_fail_loc=510, val=3***
[ 4255.280653] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 05:06:26 (1789549586)
[ 4255.409322] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4261.621365] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 05:06:33 (1789549593)
[ 4323.611705] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 05:07:34 (1789549654)
[ 4323.757061] LustreError: 83017:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4326.334604] LustreError: lustre-MDT0000-mdc-ffff91fcd28ce800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4326.343402] LustreError: 83031:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff91fcd28ce800: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 4326.354059] LustreError: 83031:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff91fcd28ce800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff91fcc6945600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4326.360443] LustreError: 83036:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4326.362129] LustreError: 83031:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4326.377300] LustreError: 83036:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 4327.816106] LustreError: 83017:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4329.858775] LustreError: 83046:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4332.500263] LustreError: lustre-MDT0000-mdc-ffff91fcd28ce800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4332.529731] LustreError: 83061:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff91fcd28ce800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff91fcc6a06200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4332.562776] LustreError: 83061:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4333.936122] LustreError: 83046:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4340.644926] LustreError: lustre-MDT0000-mdc-ffff91fcd28ce800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4340.655951] LustreError: 83089:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff91fcd28ce800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff91fcc6891800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4340.663352] LustreError: 83089:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4341.055653] LustreError: 83102:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4343.808650] LustreError: lustre-MDT0000-mdc-ffff91fcd28ce800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4343.855453] LustreError: 83122:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4343.864250] LustreError: 83122:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 1 previous similar message
[ 4345.120136] LustreError: 83102:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4345.129488] LustreError: 83102:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff91fcd28ce800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4345.149202] LustreError: 83102:0:(file.c:251:ll_close_inode_openhandle()) Skipped 4 previous similar messages
[ 4350.954191] LustreError: 83177:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4354.038085] LustreError: lustre-MDT0000-mdc-ffff91fcd28ce800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4354.068066] LustreError: 83192:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff91fcd28ce800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff91fcc6891100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4355.024206] LustreError: 83177:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4362.985336] LustreError: lustre-MDT0000-mdc-ffff91fcd28ce800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4363.597423] LustreError: 83233:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 4367.437661] LustreError: lustre-MDT0000-mdc-ffff91fcd28ce800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4367.608121] LustreError: 83233:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 4374.698782] LustreError: lustre-MDT0000-mdc-ffff91fcd28ce800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4374.714399] LustreError: 83262:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff91fcd28ce800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -5
[ 4380.646398] LustreError: 83289:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4380.657686] LustreError: 83289:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4384.344607] LustreError: lustre-MDT0000-mdc-ffff91fcd28ce800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4384.466775] LustreError: 83311:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4384.487411] LustreError: 83311:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 4384.656293] LustreError: 83289:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4384.674912] LustreError: 83289:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4384.712409] LustreError: 83289:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff91fcd28ce800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4395.113738] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 05:08:45 (1789549725)
[ 4415.640461] LustreError: 68297:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 4415.661662] LustreError: 68297:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 4419.666491] LustreError: 68297:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 4419.677874] LustreError: 68297:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 4430.656553] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 05:09:20 (1789549760)
[ 4483.763114] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 05:10:14 (1789549814)
[ 4540.386134] Lustre: 85469:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789549817/real 1789549817]  req@ffff91fcd00f5880 x1876474469095296/t0(0) o101->lustre-MDT0000-mdc-ffff91fcd28ce800@192.168.203.142@tcp:12/10 lens 576/1152 e 0 to 1 dl 1789549872 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'stat.0' uid:0 gid:0 projid:0
[ 4540.414967] Lustre: 85469:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 4545.961104] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 05:11:16 (1789549876)
[ 4577.118663] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 05:11:48 (1789549908)
[ 4577.637192] Lustre: Mounted lustre-client - version 2.17.58_39_g897064a
[ 4700.732143] Lustre: Unmounted lustre-client
[ 4706.410506] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 05:13:57 (1789550037)
[ 4707.024150] LustreError: 87616:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 4709.417925] LustreError: lustre-OST0000-osc-ffff91fcd28ce800: operation ldlm_enqueue to node 192.168.203.142@tcp failed: rc = -107
[ 4709.430785] LustreError: Skipped 11 previous similar messages
[ 4709.446064] LustreError: lustre-OST0000-osc-ffff91fcd28ce800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4709.455332] LustreError: 87699:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff91fcd28ce800: namespace resource [0x240000400:0x6f2d:0x0].0x0 (ffff91fcc6945100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4709.470238] LustreError: 87699:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 3 previous similar messages
[ 4709.752128] LustreError: 87616:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 4709.762678] Lustre: 2362:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.142@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 4715.211205] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 05:14:05 (1789550045)
[ 4778.658048] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 05:15:09 (1789550109)
[ 4779.872622] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 4781.095708] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 05:15:11 (1789550111)
[ 4837.344236] Lustre: lustre-MDT0000-mdc-ffff91fcd28ce800: Connection to lustre-MDT0000 (at 192.168.203.142@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4837.358454] Lustre: Skipped 15 previous similar messages
[ 4837.370445] Lustre: lustre-MDT0000-mdc-ffff91fcd28ce800: Connection restored to 192.168.203.142@tcp (at 192.168.203.142@tcp)
[ 4837.377962] Lustre: Skipped 15 previous similar messages
[ 4843.205390] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 4844.756951] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 05:16:15 (1789550175)
[ 4902.724330] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 05:17:12 (1789550232)
[ 4905.035679] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 4907.003607] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 05:17:17 (1789550237)
[ 4908.888666] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 4910.665316] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 05:17:21 (1789550241)
[ 4937.977773] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 05:17:48 (1789550268)
[ 4947.687423] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4973.280231] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 192.168.203.142@tcp) was lost; in progress operations using this service will fail
[ 4973.298869] LustreError: Skipped 3 previous similar messages
[ 4983.798868] Lustre: Evicted from MGS (at 192.168.203.142@tcp) after server handle changed from 0xb908dd7c30e3d173 to 0xb908dd7c30e3e9ad
[ 4983.815018] Lustre: Skipped 3 previous similar messages
[ 4996.930468] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4998.179201] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5006.579216] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 05:18:57 (1789550337)
[ 5008.487164] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5009.718022] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 05:19:00 (1789550340)
[ 5029.335425] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 05:19:20 (1789550360)
[ 5061.136674] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 05:19:52 (1789550392)
[ 5112.907862] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5115.946226] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5211.357330] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5212.790376] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5242.847962] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5244.043597] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5274.592206] Lustre: 2362:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789550551/real 1789550551]  req@ffff91fd0091dc00 x1876474475688576/t0(0) o400->lustre-MDT0000-mdc-ffff91fcd28ce800@192.168.203.142@tcp:12/10 lens 224/224 e 0 to 1 dl 1789550606 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5274.619947] Lustre: 2362:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
[ 5342.696817] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 05:24:33 (1789550673)
[ 5350.991625] LustreError: lustre-OST0000-osc-ffff91fcd28ce800: operation ost_setattr to node 192.168.203.142@tcp failed: rc = -107
[ 5350.996996] LustreError: Skipped 208 previous similar messages
[ 5396.162932] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5398.635823] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5614.154907] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 05:29:05 (1789550945)
[ 5726.694596] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 192.168.203.142@tcp) was lost; in progress operations using this service will fail
[ 5726.703621] LustreError: Skipped 4 previous similar messages
[ 5726.710256] Lustre: lustre-MDT0000-mdc-ffff91fcd28ce800: Connection to lustre-MDT0000 (at 192.168.203.142@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5726.714491] Lustre: Evicted from MGS (at 192.168.203.142@tcp) after server handle changed from 0xb908dd7c30e40433 to 0xb908dd7c30f3a6ab
[ 5726.722136] Lustre: Skipped 8 previous similar messages
[ 5726.737575] Lustre: Skipped 4 previous similar messages
[ 5726.746089] Lustre: MGC192.168.203.142@tcp: Connection restored to 192.168.203.142@tcp (at 192.168.203.142@tcp)
[ 5726.750630] Lustre: Skipped 11 previous similar messages
[ 5748.908139] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 05:31:19 (1789551079)
[ 5749.922659] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 5751.045534] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 05:31:22 (1789551082)
[ 5752.352616] LustreError: lustre-MDT0000-mdc-ffff91fcd28ce800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5757.794736] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 05:31:28 (1789551088)
[ 5924.339645] LustreError: lustre-OST0000-osc-ffff91fcd28ce800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5925.631695] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 05:34:16 (1789551256)
[ 5948.832771] Lustre: 2363:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789551260/real 1789551260]  req@ffff91fcd0c89880 x1876474486865024/t0(0) o4->lustre-OST0000-osc-ffff91fcd28ce800@192.168.203.142@tcp:6/4 lens 4584/448 e 0 to 1 dl 1789551280 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 5948.865913] Lustre: 2363:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5964.886486] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 05:34:55 (1789551295)
[ 5966.293681] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 5967.568190] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 05:34:58 (1789551298)
[ 5968.486379] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 5969.739656] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 05:35:00 (1789551300)
[ 5999.731728] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 05:35:30 (1789551330)
[ 6044.980611] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 05:36:15 (1789551375)
[ 6045.987874] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 6047.262669] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 05:36:18 (1789551378)
[ 6048.306420] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 6049.521228] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 05:36:20 (1789551380)
[ 6056.836509] Lustre: Unmounted lustre-client
[ 6057.794418] Lustre: Mounted lustre-client - version 2.17.58_39_g897064a
[ 6059.937386] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6084.451643] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2