[ 1954.426946] LustreError: lustre-OST0000-osc-ffff8b608b472000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1954.440185] Lustre: lustre-OST0000-osc-ffff8b608b472000: Connection restored to 192.168.204.155@tcp (at 192.168.204.155@tcp)
[ 1954.460115] Lustre: Skipped 14 previous similar messages
[ 1956.268060] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ IGNORE (bz5494): multiop didn't fail fsync: 5 or close: 0
[ 1959.291181] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ FAIL: no discarded dirty page found!
[ 1976.763291] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 12:15:23 (1784045723)
[ 1978.991737] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1980.908677] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 12:15:26 (1784045726)
[ 1983.309275] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1985.362758] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 12:15:31 (1784045731)
[ 1989.012874] LustreError: lustre-MDT0000-mdc-ffff8b608b472000: operation mds_reint to node 192.168.204.155@tcp failed: rc = -19
[ 1989.028959] LustreError: Skipped 2 previous similar messages
[ 2005.472516] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 192.168.204.155@tcp) was lost; in progress operations using this service will fail
[ 2015.666306] Lustre: Evicted from MGS (at 192.168.204.155@tcp) after server handle changed from 0x113dfa895f153bc to 0x113dfa895f16bda
[ 2120.289901] LustreError: lustre-MDT0000-mdc-ffff8b608b472000: operation mds_close to node 192.168.204.155@tcp failed: rc = -19
[ 2120.309403] LustreError: Skipped 2 previous similar messages
[ 2137.568386] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 192.168.204.155@tcp) was lost; in progress operations using this service will fail
[ 2147.761512] Lustre: Evicted from MGS (at 192.168.204.155@tcp) after server handle changed from 0x113dfa895f16bda to 0x113dfa895f32f06
[ 2164.928268] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 12:18:30 (1784045910)
[ 2165.418782] Lustre: *** cfs_fail_loc=305, val=0***
[ 2165.420660] Lustre: Skipped 4 previous similar messages
[ 2204.128450] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 192.168.204.155@tcp) was lost; in progress operations using this service will fail
[ 2214.447092] Lustre: Evicted from MGS (at 192.168.204.155@tcp) after server handle changed from 0x113dfa895f32f06 to 0x113dfa895f33549
[ 2220.830921] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2222.416619] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2230.839297] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 12:19:36 (1784045976)
[ 2245.112739] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 192.168.204.155@tcp) was lost; in progress operations using this service will fail
[ 2245.147152] Lustre: Evicted from MGS (at 192.168.204.155@tcp) after server handle changed from 0x113dfa895f33549 to 0x113dfa895f338a6
[ 2245.214402] LustreError: lustre-MDT0000-mdc-ffff8b608b472000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2271.093652] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 12:20:16 (1784046016)
[ 2281.338323] LustreError: lustre-OST0000-osc-ffff8b608b472000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2300.057677] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 12:20:45 (1784046045)
[ 2313.419133] LustreError: lustre-MDT0000-mdc-ffff8b608b472000: operation mds_close to node 192.168.204.155@tcp failed: rc = -19
[ 2313.429978] LustreError: Skipped 1 previous similar message
[ 2332.147861] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 192.168.204.155@tcp) was lost; in progress operations using this service will fail
[ 2332.192839] Lustre: Evicted from MGS (at 192.168.204.155@tcp) after server handle changed from 0x113dfa895f338a6 to 0x113dfa895f38aae
[ 2334.211610] Lustre: 15913:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.155@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2359.485196] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2362.105390] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2443.745052] Lustre: 2363:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784046175/real 1784046175]  req@ffff8b6088525f80 x1870705588623744/t0(0) o400->MGC192.168.204.155@tcp@192.168.204.155@tcp:26/25 lens 224/224 e 0 to 1 dl 1784046191 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2443.806345] Lustre: 2363:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 2443.815987] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 192.168.204.155@tcp) was lost; in progress operations using this service will fail
[ 2454.013586] Lustre: Evicted from MGS (at 192.168.204.155@tcp) after server handle changed from 0x113dfa895f38aae to 0x113dfa895f51cbe
[ 2472.556071] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2474.879797] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2560.480241] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 192.168.204.155@tcp) was lost; in progress operations using this service will fail
[ 2560.520157] Lustre: Evicted from MGS (at 192.168.204.155@tcp) after server handle changed from 0x113dfa895f51cbe to 0x113dfa895f6be64
[ 2560.542433] Lustre: MGC192.168.204.155@tcp: Connection restored to 192.168.204.155@tcp (at 192.168.204.155@tcp)
[ 2560.549478] Lustre: Skipped 12 previous similar messages
[ 2589.914258] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2592.792585] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2624.420828] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 12:26:09 (1784046369)
[ 2629.639517] LustreError: lustre-MDT0000-mdc-ffff8b608b472000: operation ldlm_enqueue to node 192.168.204.155@tcp failed: rc = -19
[ 2629.653149] LustreError: Skipped 3 previous similar messages
[ 2629.657940] Lustre: lustre-MDT0000-mdc-ffff8b608b472000: Connection to lustre-MDT0000 (at 192.168.204.155@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2629.685367] Lustre: Skipped 9 previous similar messages
[ 2646.496861] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 192.168.204.155@tcp) was lost; in progress operations using this service will fail
[ 2656.748924] Lustre: Evicted from MGS (at 192.168.204.155@tcp) after server handle changed from 0x113dfa895f6be64 to 0x113dfa895f77b41
[ 2658.243772] Lustre: 15913:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.155@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2670.575187] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2702.962393] Lustre: 15913:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.155@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2717.677446] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2755.273274] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2786.784202] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 192.168.204.155@tcp) was lost; in progress operations using this service will fail
[ 2786.792781] LustreError: Skipped 2 previous similar messages
[ 2796.971286] Lustre: Evicted from MGS (at 192.168.204.155@tcp) after server handle changed from 0x113dfa895f7e727 to 0x113dfa895f81820
[ 2796.980190] Lustre: Skipped 2 previous similar messages
[ 2798.011674] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2849.371343] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2906.971663] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2962.289857] Lustre: 15913:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.155@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 3001.437735] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 12:32:27 (1784046747)
[ 3057.794497] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3060.473906] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3346.213973] LustreError: lustre-OST0000-osc-ffff8b608b472000: operation ldlm_enqueue to node 192.168.204.155@tcp failed: rc = -107
[ 3346.223068] LustreError: Skipped 9 previous similar messages
[ 3346.234144] Lustre: lustre-OST0000-osc-ffff8b608b472000: Connection to lustre-OST0000 (at 192.168.204.155@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3346.250087] Lustre: Skipped 7 previous similar messages
[ 3366.619325] Lustre: lustre-OST0000-osc-ffff8b608b472000: Connection restored to 192.168.204.155@tcp (at 192.168.204.155@tcp)
[ 3366.629190] Lustre: Skipped 16 previous similar messages
[ 3380.817600] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3382.983848] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3723.935091] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3726.621721] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3974.315429] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 12:48:40 (1784047720)
[ 3992.032724] Lustre: 50808:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784047723/real 1784047723]  req@ffff8b6083f85f80 x1870705610666112/t0(0) o101->lustre-MDT0000-mdc-ffff8b608b472000@192.168.204.155@tcp:12/10 lens 576/1152 e 0 to 1 dl 1784047739 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3992.068370] Lustre: 50808:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 3992.078225] Lustre: lustre-MDT0000-mdc-ffff8b608b472000: Connection to lustre-MDT0000 (at 192.168.204.155@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3992.096022] Lustre: Skipped 1 previous similar message
[ 3992.136964] Lustre: lustre-MDT0000-mdc-ffff8b608b472000: Connection restored to 192.168.204.155@tcp (at 192.168.204.155@tcp)
[ 3992.145147] Lustre: Skipped 1 previous similar message
[ 3999.249375] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 12:49:05 (1784047745)
[ 4024.107894] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 12:49:30 (1784047770)
[ 4047.445921] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 12:49:53 (1784047793)
[ 4047.763099] Lustre: Mounted lustre-client
[ 4078.560176] Lustre: 2362:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784047810/real 1784047810]  req@ffff8b6086a68000 x1870705610684672/t0(0) o400->lustre-MDT0000-mdc-ffff8b608b472000@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 1 dl 1784047826 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4078.560635] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 192.168.204.155@tcp) was lost; in progress operations using this service will fail
[ 4078.584828] Lustre: 2362:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 4078.615089] LustreError: Skipped 3 previous similar messages
[ 4078.655751] Lustre: Evicted from MGS (at 192.168.204.155@tcp) after server handle changed from 0x113dfa895fa179e to 0x113dfa8960ed2fc
[ 4078.665229] Lustre: Skipped 3 previous similar messages
[ 4091.329700] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4092.547044] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4094.841266] Lustre: Unmounted lustre-client
[ 4100.572292] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 12:50:46 (1784047846)
[ 4237.088414] Lustre: 2365:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784047968/real 1784047968]  req@ffff8b6083f85880 x1870705610717056/t0(0) o4->lustre-OST0000-osc-ffff8b608b472000@192.168.204.155@tcp:6/4 lens 488/448 e 0 to 1 dl 1784047984 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4237.124694] Lustre: 2365:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 67 previous similar messages
[ 4392.218121] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 12:55:38 (1784048138)
[ 4439.855423] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 12:56:26 (1784048186)
[ 4443.007158] Lustre: Unmounted lustre-client
[ 4463.865360] Lustre: Mounted lustre-client
[ 4469.225557] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 12:56:55 (1784048215)
[ 4469.462543] LustreError: 56710:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4470.515218] LustreError: 56710:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4470.628945] Lustre: *** cfs_fail_loc=305, val=0***
[ 4494.521273] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 12:57:20 (1784048240)
[ 4495.043191] Lustre: Mounted lustre-client
[ 4497.471566] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4507.795634] Lustre: Unmounted lustre-client
[ 4513.996868] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 12:57:40 (1784048260)
[ 4603.417504] LustreError: lustre-MDT0000-mdc-ffff8b6086b49800: operation mds_reint to node 192.168.204.155@tcp failed: rc = -19
[ 4603.426787] LustreError: Skipped 6 previous similar messages
[ 4603.442452] Lustre: lustre-MDT0000-mdc-ffff8b6086b49800: Connection to lustre-MDT0000 (at 192.168.204.155@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4603.458096] Lustre: Skipped 23 previous similar messages
[ 4624.354635] Lustre: 2364:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784048355/real 1784048355]  req@ffff8b60910c7800 x1870705613390208/t0(0) o400->MGC192.168.204.155@tcp@192.168.204.155@tcp:26/25 lens 224/224 e 0 to 1 dl 1784048371 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4624.400648] Lustre: 2364:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 79 previous similar messages
[ 4624.413863] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 192.168.204.155@tcp) was lost; in progress operations using this service will fail
[ 4624.438313] Lustre: Evicted from MGS (at 192.168.204.155@tcp) after server handle changed from 0x113dfa8960eedeb to 0x113dfa89611a403
[ 4624.464214] Lustre: MGC192.168.204.155@tcp: Connection restored to 192.168.204.155@tcp (at 192.168.204.155@tcp)
[ 4624.479260] Lustre: Skipped 24 previous similar messages
[ 4805.387295] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 13:02:31 (1784048551)
[ 4811.932276] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4827.122921] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 192.168.204.155@tcp) was lost; in progress operations using this service will fail
[ 4827.142563] Lustre: Evicted from MGS (at 192.168.204.155@tcp) after server handle changed from 0x113dfa89611a403 to 0x113dfa89616689c
[ 4828.402834] LustreError: lustre-MDT0000-mdc-ffff8b6086b49800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4846.137661] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 13:03:12 (1784048592)
[ 4846.624275] Lustre: Mounted lustre-client
[ 4854.391789] LustreError: lustre-OST0000-osc-ffff8b6086b49800: operation ldlm_enqueue to node 192.168.204.155@tcp failed: rc = -107
[ 4854.421293] LustreError: lustre-OST0000-osc-ffff8b6086b49800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4854.442925] LustreError: 60429:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8b6086b49800: namespace resource [0x240000400:0x3de3:0x0].0x0 (ffff8b6099a70100) refcount nonzero (2) after lock cleanup; forcing cleanup.
[ 4866.852598] Lustre: Unmounted lustre-client
[ 4875.659671] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 13:03:41 (1784048621)
[ 4882.695696] LustreError: lustre-MDT0000-mdc-ffff8b6086b49800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4882.724163] LustreError: 61101:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4890.827827] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 13:03:57 (1784048637)
[ 4891.069134] LustreError: 61802:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4895.223299] LustreError: lustre-MDT0000-mdc-ffff8b6086b49800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4895.245877] LustreError: 61824:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4895.264748] LustreError: 61802:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=815
[ 4895.295097] LustreError: 61802:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4895.312509] LustreError: 61829:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4896.365241] LustreError: 61835:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4896.383126] LustreError: 61835:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4898.696619] LustreError: 61857:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4898.716579] LustreError: 61857:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4903.394909] LustreError: 61902:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4903.409839] LustreError: 61902:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4915.136404] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 13:04:20 (1784048660)
[ 4955.401996] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4956.987296] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4967.190587] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 13:05:13 (1784048713)
[ 5004.712453] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5006.604908] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5016.262838] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 13:06:02 (1784048762)
[ 5083.556852] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5085.141154] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5093.677591] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 13:07:19 (1784048839)
[ 5129.165946] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5130.915823] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5137.032550] Lustre: Unmounted lustre-client
[ 5190.696315] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5193.265289] Lustre: Mounted lustre-client
[ 5200.007698] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 13:09:06 (1784048946)
[ 5202.745509] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5204.588662] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 13:09:10 (1784048950)
[ 5235.104679] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 13:09:41 (1784048981)
[ 5236.373185] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5238.125603] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 13:09:44 (1784048984)
[ 5239.722308] Lustre: *** cfs_fail_loc=805, val=0***
[ 5239.766827] Lustre: Mounted lustre-client
[ 5244.914160] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5265.387574] Lustre: lustre-MDT0000-mdc-ffff8b609022e800: Connection to lustre-MDT0000 (at 192.168.204.155@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5265.406645] Lustre: Skipped 12 previous similar messages
[ 5265.413025] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 192.168.204.155@tcp) was lost; in progress operations using this service will fail
[ 5265.433485] Lustre: Evicted from MGS (at 192.168.204.155@tcp) after server handle changed from 0x113dfa8961674ce to 0x113dfa89616786a
[ 5265.451095] Lustre: MGC192.168.204.155@tcp: Connection restored to 192.168.204.155@tcp (at 192.168.204.155@tcp)
[ 5265.459169] Lustre: Skipped 13 previous similar messages
[ 5266.400708] Lustre: 2363:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784048997/real 1784048997]  req@ffff8b6088b7df80 x1870705615436032/t0(0) o400->lustre-MDT0000-mdc-ffff8b609022e800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 1 dl 1784049013 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5266.439559] Lustre: 2363:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5271.933706] Lustre: Unmounted lustre-client
[ 5278.731185] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 13:10:25 (1784049025)
[ 5306.442275] LustreError: 2361:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8b608925e300 x1870705615446784/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff8b609022e800@192.168.204.155@tcp:12/10 lens 648/608 e 0 to 0 dl 1784049069 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5313.427325] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5315.127970] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5323.357570] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 13:11:09 (1784049069)
[ 5324.671934] LustreError: lustre-OST0000-osc-ffff8b609022e800: operation ost_write to node 192.168.204.155@tcp failed: rc = -107
[ 5324.683345] LustreError: Skipped 2 previous similar messages
[ 5324.704769] LustreError: lustre-OST0000-osc-ffff8b609022e800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5324.721489] Lustre: 2365:0:(llite_lib.c:4327:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.155@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5324.765151] LustreError: 72935:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8b609022e800: namespace resource [0x240000400:0x3e42:0x0].0x0 (ffff8b6091125d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5332.337965] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 13:11:18 (1784049078)
[ 5334.121502] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5335.768571] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 13:11:21 (1784049081)
[ 5337.379446] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5339.102377] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 13:11:25 (1784049085)
[ 5340.492051] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5341.949361] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 13:11:28 (1784049088)
[ 5343.843642] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5345.571898] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 13:11:31 (1784049091)
[ 5347.061234] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5348.504513] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 13:11:34 (1784049094)
[ 5349.946646] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5351.628155] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 13:11:37 (1784049097)
[ 5353.081736] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5354.603919] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 13:11:40 (1784049100)
[ 5356.169684] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5358.145282] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 13:11:44 (1784049104)
[ 5359.654598] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5361.250697] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 13:11:47 (1784049107)
[ 5362.762425] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5364.606546] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 13:11:50 (1784049110)
[ 5366.336768] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5368.363837] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 13:11:54 (1784049114)
[ 5370.768513] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5372.620745] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 13:11:58 (1784049118)
[ 5401.072564] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 13:12:27 (1784049147)
[ 5430.763880] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 13:12:57 (1784049177)
[ 5431.336679] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5442.179477] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 13:13:07 (1784049187)
[ 5442.910209] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5453.811980] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 13:13:19 (1784049199)
[ 5454.282509] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5463.367835] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 13:13:29 (1784049209)
[ 5463.695093] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5472.642377] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 13:13:38 (1784049218)
[ 5473.089909] Lustre: *** cfs_fail_loc=510, val=3***
[ 5481.658352] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 13:13:48 (1784049228)
[ 5482.045392] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5493.806250] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 13:13:59 (1784049239)
[ 5494.386618] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5560.180724] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 13:15:06 (1784049306)
[ 5560.707820] LustreError: 83125:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5563.539935] LustreError: lustre-MDT0000-mdc-ffff8b609022e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5563.573318] LustreError: 83140:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8b609022e800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8b6089234800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5564.784840] LustreError: 83125:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5566.913319] LustreError: 83155:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5570.254088] LustreError: lustre-MDT0000-mdc-ffff8b609022e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5570.288215] LustreError: 83169:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8b609022e800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8b6091074f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5570.335921] LustreError: 83169:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5571.000164] LustreError: 83155:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5578.229898] LustreError: lustre-MDT0000-mdc-ffff8b609022e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5578.766489] LustreError: 83211:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5581.896795] LustreError: lustre-MDT0000-mdc-ffff8b609022e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5582.048328] LustreError: 83231:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5582.067581] LustreError: 83231:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5582.832244] LustreError: 83211:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5582.847407] LustreError: 83211:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8b609022e800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5583.204191] LustreError: 83237:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5583.215412] LustreError: 83237:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5585.538239] LustreError: 83259:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5585.552807] LustreError: 83259:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5589.291443] LustreError: 83286:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5592.270732] LustreError: lustre-MDT0000-mdc-ffff8b609022e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5592.289736] LustreError: 83301:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8b609022e800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8b6091074c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5592.308470] LustreError: 83301:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5593.392545] LustreError: 83286:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5600.652858] LustreError: lustre-MDT0000-mdc-ffff8b609022e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5601.321288] LustreError: 83343:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5604.385515] LustreError: lustre-MDT0000-mdc-ffff8b609022e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5604.400767] LustreError: 83357:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8b609022e800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8b6091125600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5604.429372] LustreError: 83357:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5605.400188] LustreError: 83343:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5608.526242] LustreError: lustre-MDT0000-mdc-ffff8b609022e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5608.607407] LustreError: 83390:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5608.616922] LustreError: 83390:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5609.701134] LustreError: 83370:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8b609022e800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5618.000804] LustreError: 83442:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5618.019060] LustreError: 83442:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5622.106928] LustreError: 83442:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5622.119024] LustreError: 83442:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5622.387878] LustreError: lustre-MDT0000-mdc-ffff8b609022e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5630.071601] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 13:16:16 (1784049376)
[ 5649.147142] LustreError: 68420:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5653.208394] LustreError: 68420:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5666.708434] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 13:16:52 (1784049412)
[ 5721.497552] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 13:17:47 (1784049467)
[ 5788.528797] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 13:18:54 (1784049534)
[ 5822.941086] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 13:19:28 (1784049568)
[ 5823.492059] Lustre: Mounted lustre-client
[ 5947.891574] Lustre: Unmounted lustre-client
[ 5955.882894] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 13:21:41 (1784049701)
[ 5956.766439] LustreError: 87724:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5959.748729] LustreError: lustre-OST0000-osc-ffff8b609022e800: operation ost_statfs to node 192.168.204.155@tcp failed: rc = -107
[ 5959.762268] LustreError: Skipped 9 previous similar messages
[ 5959.774520] Lustre: lustre-OST0000-osc-ffff8b609022e800: Connection to lustre-OST0000 (at 192.168.204.155@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5959.792579] Lustre: Skipped 17 previous similar messages
[ 5959.819567] LustreError: lustre-OST0000-osc-ffff8b609022e800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5959.851576] LustreError: 87811:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8b609022e800: namespace resource [0x240000400:0x3e6d:0x0].0x0 (ffff8b6091125b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5959.873680] LustreError: 87811:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5960.149120] LustreError: 87724:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5960.162956] Lustre: 2364:0:(llite_lib.c:4327:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.155@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5960.186430] Lustre: lustre-OST0000-osc-ffff8b609022e800: Connection restored to 192.168.204.155@tcp (at 192.168.204.155@tcp)
[ 5960.208165] Lustre: Skipped 17 previous similar messages
[ 5967.435390] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 13:21:53 (1784049713)
[ 6026.208164] Lustre: 88392:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784049717/real 1784049717]  req@ffff8b608925d500 x1870705615612160/t0(0) o101->lustre-MDT0000-mdc-ffff8b609022e800@192.168.204.155@tcp:12/10 lens 328/344 e 0 to 1 dl 1784049773 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6026.285926] Lustre: 88392:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 6034.550392] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 13:23:00 (1784049780)
[ 6036.339389] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6038.707363] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 13:23:04 (1784049784)
[ 6109.177515] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6111.190981] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 13:24:17 (1784049857)
[ 6171.593070] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 13:25:17 (1784049917)
[ 6174.042415] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6176.299651] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 13:25:22 (1784049922)
[ 6178.173878] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6179.927459] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 13:25:26 (1784049926)
[ 6216.886089] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 13:26:03 (1784049963)
[ 6229.546793] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6256.608249] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 192.168.204.155@tcp) was lost; in progress operations using this service will fail
[ 6256.622269] LustreError: Skipped 2 previous similar messages
[ 6256.656827] Lustre: Evicted from MGS (at 192.168.204.155@tcp) after server handle changed from 0x113dfa89616831b to 0x113dfa896169bfd
[ 6256.671349] Lustre: Skipped 2 previous similar messages
[ 6268.974647] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6270.780596] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6281.583080] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 13:27:07 (1784050027)
[ 6284.689742] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6286.582223] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 13:27:12 (1784050032)
[ 6302.721935] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 192.168.204.155@tcp) was lost; in progress operations using this service will fail
[ 6302.751242] Lustre: Evicted from MGS (at 192.168.204.155@tcp) after server handle changed from 0x113dfa896169bfd to 0x113dfa896169fb5
[ 6313.433859] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 13:27:39 (1784050059)
[ 6360.621401] Lustre: 68423:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.155@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6378.799027] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 13:28:44 (1784050124)
[ 6434.109256] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6437.483638] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6521.843985] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 192.168.204.155@tcp) was lost; in progress operations using this service will fail
[ 6521.856150] LustreError: Skipped 1 previous similar message
[ 6521.894541] Lustre: Evicted from MGS (at 192.168.204.155@tcp) after server handle changed from 0x113dfa89616a1c2 to 0x113dfa89616b112
[ 6521.900407] Lustre: Skipped 1 previous similar message
[ 6536.612633] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6538.239512] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6562.799537] Lustre: lustre-MDT0000-mdc-ffff8b609022e800: Connection to lustre-MDT0000 (at 192.168.204.155@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6562.813173] Lustre: Skipped 8 previous similar messages
[ 6562.824474] Lustre: MGC192.168.204.155@tcp: Connection restored to 192.168.204.155@tcp (at 192.168.204.155@tcp)
[ 6562.835409] Lustre: Skipped 11 previous similar messages
[ 6575.916152] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6577.315914] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6718.834286] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 13:34:25 (1784050465)
[ 6728.186322] LustreError: lustre-OST0000-osc-ffff8b609022e800: operation ost_setattr to node 192.168.204.155@tcp failed: rc = -19
[ 6728.199407] LustreError: Skipped 32 previous similar messages
[ 6776.411235] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6779.126736] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7039.637460] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 13:39:45 (1784050785)
[ 7197.688852] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 192.168.204.155@tcp) was lost; in progress operations using this service will fail
[ 7197.720884] LustreError: Skipped 1 previous similar message
[ 7197.728742] Lustre: lustre-MDT0000-mdc-ffff8b609022e800: Connection to lustre-MDT0000 (at 192.168.204.155@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7197.742094] Lustre: Skipped 1 previous similar message
[ 7197.742591] Lustre: Evicted from MGS (at 192.168.204.155@tcp) after server handle changed from 0x113dfa89616b4bc to 0x113dfa896265734
[ 7197.764721] Lustre: Skipped 1 previous similar message
[ 7197.777455] Lustre: MGC192.168.204.155@tcp: Connection restored to 192.168.204.155@tcp (at 192.168.204.155@tcp)
[ 7197.792304] Lustre: Skipped 1 previous similar message
[ 7223.265042] Lustre: 2365:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784050934/real 1784050934]  req@ffff8b60b1763b80 x1870705633296768/t0(0) o400->lustre-MDT0000-mdc-ffff8b609022e800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 1 dl 1784050970 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7223.303508] Lustre: 2365:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 21 previous similar messages
[ 7232.112205] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 13:42:58 (1784050978)
[ 7233.789204] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7235.509945] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 13:43:01 (1784050981)
[ 7237.454729] LustreError: lustre-MDT0000-mdc-ffff8b609022e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7246.150371] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 13:43:12 (1784050992)
[ 7413.735281] LustreError: lustre-OST0000-osc-ffff8b609022e800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7416.163700] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 13:46:01 (1784051161)
[ 7460.019571] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 13:46:45 (1784051205)
[ 7461.947793] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7463.876302] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 13:46:49 (1784051209)
[ 7465.880921] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7468.097864] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 13:46:53 (1784051213)
[ 7504.481129] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 13:47:30 (1784051250)
[ 7543.796702] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 192.168.204.155@tcp) was lost; in progress operations using this service will fail
[ 7543.834255] Lustre: Evicted from MGS (at 192.168.204.155@tcp) after server handle changed from 0x113dfa896265734 to 0x113dfa896267f27
[ 7543.843731] Lustre: Skipped 1 previous similar message
[ 7561.660589] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 13:48:27 (1784051307)
[ 7563.231149] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7565.027573] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 13:48:31 (1784051311)
[ 7566.690137] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7568.530928] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 13:48:34 (1784051314)
[ 7585.595474] Lustre: Unmounted lustre-client
[ 7587.752230] Lustre: Mounted lustre-client
[ 7590.859208] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7615.209605] Lustre: 106767:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.155@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 7630.968317] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2