[ 2023.202913] LustreError: lustre-OST0000-osc-ffff95e6894b6000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2023.230171] Lustre: 2357:0:(llite_lib.c:4339:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.115@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 2030.841333] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 17:16:58 (1785791818)
[ 2033.128529] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 2034.876728] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 17:17:02 (1785791822)
[ 2037.190170] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 2039.261839] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 17:17:06 (1785791826)
[ 2042.951079] LustreError: lustre-MDT0000-mdc-ffff95e6894b6000: operation ldlm_enqueue to node 192.168.201.115@tcp failed: rc = -19
[ 2042.961225] LustreError: Skipped 1 previous similar message
[ 2059.744965] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 192.168.201.115@tcp) was lost; in progress operations using this service will fail
[ 2070.012265] Lustre: Evicted from MGS (at 192.168.201.115@tcp) after server handle changed from 0xe91c1e98b040b8ca to 0xe91c1e98b040d2e0
[ 2177.966502] LustreError: lustre-MDT0000-mdc-ffff95e6894b6000: operation ldlm_enqueue to node 192.168.201.115@tcp failed: rc = -19
[ 2197.984271] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 192.168.201.115@tcp) was lost; in progress operations using this service will fail
[ 2208.182813] Lustre: Evicted from MGS (at 192.168.201.115@tcp) after server handle changed from 0xe91c1e98b040d2e0 to 0xe91c1e98b0424a32
[ 2226.657770] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 17:20:14 (1785792014)
[ 2227.319785] Lustre: *** cfs_fail_loc=305, val=0***
[ 2227.323512] Lustre: Skipped 4 previous similar messages
[ 2269.664159] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785792042/real 1785792042]  req@ffff95e6986f7b80 x1872536367939584/t0(0) o400->MGC192.168.201.115@tcp@192.168.201.115@tcp:26/25 lens 224/224 e 0 to 1 dl 1785792058 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2269.722926] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 2269.742664] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 192.168.201.115@tcp) was lost; in progress operations using this service will fail
[ 2279.949832] Lustre: Evicted from MGS (at 192.168.201.115@tcp) after server handle changed from 0xe91c1e98b0424a32 to 0xe91c1e98b0425186
[ 2315.506642] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2318.945958] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2332.176038] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 17:21:58 (1785792118)
[ 2355.758938] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 192.168.201.115@tcp) was lost; in progress operations using this service will fail
[ 2355.803330] Lustre: Evicted from MGS (at 192.168.201.115@tcp) after server handle changed from 0xe91c1e98b0425186 to 0xe91c1e98b042574b
[ 2355.854095] LustreError: lustre-MDT0000-mdc-ffff95e6894b6000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2389.427726] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 17:22:57 (1785792177)
[ 2432.457875] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 17:23:39 (1785792219)
[ 2446.919200] LustreError: lustre-MDT0000-mdc-ffff95e6894b6000: operation mds_close to node 192.168.201.115@tcp failed: rc = -19
[ 2468.320476] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 192.168.201.115@tcp) was lost; in progress operations using this service will fail
[ 2478.514911] Lustre: Evicted from MGS (at 192.168.201.115@tcp) after server handle changed from 0xe91c1e98b042574b to 0xe91c1e98b0429a34
[ 2497.336286] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2499.668921] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2566.124057] Lustre: lustre-MDT0000-mdc-ffff95e6894b6000: Connection to lustre-MDT0000 (at 192.168.201.115@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2566.147621] Lustre: Skipped 10 previous similar messages
[ 2586.593745] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 192.168.201.115@tcp) was lost; in progress operations using this service will fail
[ 2595.752413] Lustre: Evicted from MGS (at 192.168.201.115@tcp) after server handle changed from 0xe91c1e98b0429a34 to 0xe91c1e98b0441a38
[ 2595.783452] Lustre: MGC192.168.201.115@tcp: Connection restored to 192.168.201.115@tcp (at 192.168.201.115@tcp)
[ 2595.794072] Lustre: Skipped 14 previous similar messages
[ 2620.761195] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2623.488243] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2689.795463] LustreError: lustre-MDT0000-mdc-ffff95e6894b6000: operation ldlm_enqueue to node 192.168.201.115@tcp failed: rc = -19
[ 2689.818974] LustreError: Skipped 3 previous similar messages
[ 2708.448191] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 192.168.201.115@tcp) was lost; in progress operations using this service will fail
[ 2718.696449] Lustre: Evicted from MGS (at 192.168.201.115@tcp) after server handle changed from 0xe91c1e98b0441a38 to 0xe91c1e98b045d1b7
[ 2743.245378] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2746.555221] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2782.247983] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 17:29:28 (1785792568)
[ 2807.265712] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 192.168.201.115@tcp) was lost; in progress operations using this service will fail
[ 2817.555802] Lustre: Evicted from MGS (at 192.168.201.115@tcp) after server handle changed from 0xe91c1e98b045d1b7 to 0xe91c1e98b04688d6
[ 2830.221942] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2863.095291] Lustre: 15907:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.115@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2876.939736] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2901.984473] Lustre: 2357:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785792675/real 1785792675]  req@ffff95e690e1d500 x1872536371913088/t0(0) o400->MGC192.168.201.115@tcp@192.168.201.115@tcp:26/25 lens 224/224 e 0 to 1 dl 1785792691 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2902.069266] Lustre: 2357:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 2913.632363] Lustre: 15907:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.115@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2938.359828] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2953.849558] LustreError: lustre-MDT0000-mdc-ffff95e6894b6000: operation mds_close to node 192.168.201.115@tcp failed: rc = -19
[ 2953.869201] LustreError: Skipped 5 previous similar messages
[ 2973.729729] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 192.168.201.115@tcp) was lost; in progress operations using this service will fail
[ 2973.757137] LustreError: Skipped 2 previous similar messages
[ 2983.982034] Lustre: Evicted from MGS (at 192.168.201.115@tcp) after server handle changed from 0xe91c1e98b046efc9 to 0xe91c1e98b0474f5a
[ 2983.994613] Lustre: Skipped 2 previous similar messages
[ 2985.211704] Lustre: 15907:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.115@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 3004.075908] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 3068.306433] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 3129.519101] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3200.428044] Lustre: lustre-MDT0000-mdc-ffff95e6894b6000: Connection restored to 192.168.201.115@tcp (at 192.168.201.115@tcp)
[ 3200.434477] Lustre: Skipped 16 previous similar messages
[ 3235.390615] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 17:37:02 (1785793022)
[ 3250.027612] Lustre: lustre-OST0000-osc-ffff95e6894b6000: Connection to lustre-OST0000 (at 192.168.201.115@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3250.038745] Lustre: Skipped 8 previous similar messages
[ 3292.498693] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3294.959105] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3582.310532] LustreError: lustre-OST0000-osc-ffff95e6894b6000: operation ldlm_enqueue to node 192.168.201.115@tcp failed: rc = -107
[ 3582.335264] LustreError: Skipped 5 previous similar messages
[ 3634.948453] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3638.169118] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3913.367180] Lustre: lustre-OST0000-osc-ffff95e6894b6000: Connection to lustre-OST0000 (at 192.168.201.115@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3913.385707] Lustre: Skipped 1 previous similar message
[ 3936.222453] Lustre: lustre-OST0000-osc-ffff95e6894b6000: Connection restored to 192.168.201.115@tcp (at 192.168.201.115@tcp)
[ 3936.235527] Lustre: Skipped 1 previous similar message
[ 3958.145025] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3960.860832] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4213.994423] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 17:53:21 (1785794001)
[ 4232.672148] Lustre: 50392:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785794005/real 1785794005]  req@ffff95e6833aaa00 x1872536388347520/t0(0) o101->lustre-MDT0000-mdc-ffff95e6894b6000@192.168.201.115@tcp:12/10 lens 576/1152 e 0 to 1 dl 1785794021 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4232.698701] Lustre: 50392:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 4244.841599] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 17:53:51 (1785794031)
[ 4273.758317] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 17:54:21 (1785794061)
[ 4301.612909] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 17:54:49 (1785794089)
[ 4302.313506] Lustre: Mounted lustre-client
[ 4332.832758] Lustre: 2355:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785794106/real 1785794106]  req@ffff95e6833a8700 x1872536388367360/t0(0) o400->lustre-MDT0000-mdc-ffff95e6894b6000@192.168.201.115@tcp:12/10 lens 224/224 e 0 to 1 dl 1785794122 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4332.888488] Lustre: 2355:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4332.907927] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 192.168.201.115@tcp) was lost; in progress operations using this service will fail
[ 4332.930052] LustreError: Skipped 3 previous similar messages
[ 4343.317824] Lustre: Evicted from MGS (at 192.168.201.115@tcp) after server handle changed from 0xe91c1e98b049443c to 0xe91c1e98b05a2dc3
[ 4343.338655] Lustre: Skipped 3 previous similar messages
[ 4344.540335] Lustre: 15907:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.115@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4367.467909] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4369.675222] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4372.231042] Lustre: Unmounted lustre-client
[ 4380.394509] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 17:56:07 (1785794167)
[ 4489.697071] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785794262/real 1785794262]  req@ffff95e6983dc000 x1872536388404096/t0(0) o4->lustre-OST0000-osc-ffff95e6894b6000@192.168.201.115@tcp:6/4 lens 488/448 e 0 to 1 dl 1785794278 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4489.779947] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 52 previous similar messages
[ 4522.464726] Lustre: lustre-OST0000-osc-ffff95e6894b6000: Connection to lustre-OST0000 (at 192.168.201.115@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4522.499574] Lustre: Skipped 12 previous similar messages
[ 4537.776936] Lustre: lustre-OST0000-osc-ffff95e6894b6000: Connection restored to 192.168.201.115@tcp (at 192.168.201.115@tcp)
[ 4537.796497] Lustre: Skipped 14 previous similar messages
[ 4684.991564] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 18:01:12 (1785794472)
[ 4738.149386] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 18:02:05 (1785794525)
[ 4742.483459] Lustre: Unmounted lustre-client
[ 4771.744531] Lustre: Mounted lustre-client
[ 4785.573341] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 18:02:51 (1785794571)
[ 4786.216870] LustreError: 56324:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4787.281552] LustreError: 56324:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4787.560122] Lustre: *** cfs_fail_loc=305, val=0***
[ 4815.158243] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 18:03:22 (1785794602)
[ 4815.961215] Lustre: Mounted lustre-client
[ 4819.308789] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4829.642787] Lustre: Unmounted lustre-client
[ 4838.942147] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 18:03:46 (1785794626)
[ 4950.252601] LustreError: lustre-MDT0000-mdc-ffff95e698c39800: operation mds_reint to node 192.168.201.115@tcp failed: rc = -19
[ 4950.265113] LustreError: Skipped 1 previous similar message
[ 4970.468101] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 192.168.201.115@tcp) was lost; in progress operations using this service will fail
[ 4970.492083] Lustre: Evicted from MGS (at 192.168.201.115@tcp) after server handle changed from 0xe91c1e98b05a4a1e to 0xe91c1e98b05cfe29
[ 4972.456055] Lustre: 55721:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.115@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5182.397462] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 18:09:29 (1785794969)
[ 5190.600398] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5210.084522] Lustre: lustre-MDT0000-mdc-ffff95e698c39800: Connection to lustre-MDT0000 (at 192.168.201.115@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5210.097459] Lustre: Skipped 12 previous similar messages
[ 5210.112730] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 192.168.201.115@tcp) was lost; in progress operations using this service will fail
[ 5210.149033] Lustre: Evicted from MGS (at 192.168.201.115@tcp) after server handle changed from 0xe91c1e98b05cfe29 to 0xe91c1e98b061c8db
[ 5210.171464] Lustre: MGC192.168.201.115@tcp: Connection restored to 192.168.201.115@tcp (at 192.168.201.115@tcp)
[ 5210.183373] Lustre: Skipped 12 previous similar messages
[ 5210.196545] LustreError: lustre-MDT0000-mdc-ffff95e698c39800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5236.164608] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 18:10:23 (1785795023)
[ 5237.192172] Lustre: Mounted lustre-client
[ 5245.268091] LustreError: lustre-OST0000-osc-ffff95e698c39800: operation ldlm_enqueue to node 192.168.201.115@tcp failed: rc = -107
[ 5245.292158] LustreError: lustre-OST0000-osc-ffff95e698c39800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5254.737804] Lustre: 2354:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785795028/real 1785795028]  req@ffff95e685865f80 x1872536393068672/t0(0) o103->lustre-OST0000-osc-ffff95e688b71800@192.168.201.115@tcp:17/18 lens 328/224 e 0 to 1 dl 1785795044 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 5254.865484] Lustre: 2354:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 96 previous similar messages
[ 5258.729561] Lustre: Unmounted lustre-client
[ 5269.988948] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 18:10:57 (1785795057)
[ 5279.515369] LustreError: lustre-MDT0000-mdc-ffff95e698c39800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5279.545732] LustreError: 60732:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 5291.526619] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 18:11:18 (1785795078)
[ 5291.937376] LustreError: 61421:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 5297.120370] LustreError: 61421:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 5297.128792] LustreError: 61421:0:(import.c:711:ptlrpc_connect_import_locked()) already connected
[ 5298.371958] LustreError: lustre-MDT0000-mdc-ffff95e698c39800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5298.395440] LustreError: 61438:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 5298.502178] LustreError: 61443:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5299.656161] LustreError: 61449:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5299.675240] LustreError: 61449:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5302.097687] LustreError: 61472:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5302.117730] LustreError: 61472:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5306.124593] LustreError: 61513:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5306.145984] LustreError: 61513:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 23 previous similar messages
[ 5320.508681] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 18:11:47 (1785795107)
[ 5372.696169] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5375.361546] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5390.565461] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 18:12:57 (1785795177)
[ 5438.824909] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5441.955701] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5466.264650] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 18:14:12 (1785795252)
[ 5540.256613] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5542.465859] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5553.963221] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 18:15:41 (1785795341)
[ 5600.819276] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5602.825787] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5610.276602] Lustre: Unmounted lustre-client
[ 5680.179911] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5684.100682] Lustre: Mounted lustre-client
[ 5692.779632] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 18:18:00 (1785795480)
[ 5697.327527] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5700.395627] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 18:18:07 (1785795487)
[ 5741.667430] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 18:18:48 (1785795528)
[ 5743.624653] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5745.758663] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 18:18:53 (1785795533)
[ 5748.013551] Lustre: *** cfs_fail_loc=805, val=0***
[ 5748.124935] Lustre: Mounted lustre-client
[ 5755.878890] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5778.914600] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 192.168.201.115@tcp) was lost; in progress operations using this service will fail
[ 5789.170276] Lustre: Evicted from MGS (at 192.168.201.115@tcp) after server handle changed from 0xe91c1e98b061d2cf to 0xe91c1e98b061d4ea
[ 5804.537827] LustreError: 2353:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff95e698e3f000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff95e686be8b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5808.783414] Lustre: Unmounted lustre-client
[ 5819.636063] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 18:20:07 (1785795607)
[ 5825.022977] Lustre: lustre-MDT0000-mdc-ffff95e682f03000: Connection to lustre-MDT0000 (at 192.168.201.115@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5825.042686] Lustre: Skipped 11 previous similar messages
[ 5846.328454] Lustre: MGC192.168.201.115@tcp: Connection restored to 192.168.201.115@tcp (at 192.168.201.115@tcp)
[ 5846.339567] Lustre: Skipped 10 previous similar messages
[ 5848.195950] Lustre: 68050:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.115@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5865.741580] LustreError: 2353:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff95e685909c00 x1872536393160576/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff95e682f03000@192.168.201.115@tcp:12/10 lens 664/608 e 0 to 0 dl 1785795670 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5871.851359] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5873.412826] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5884.669349] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 18:21:12 (1785795672)
[ 5886.542356] LustreError: lustre-OST0000-osc-ffff95e682f03000: operation ost_write to node 192.168.201.115@tcp failed: rc = -107
[ 5886.561217] LustreError: Skipped 2 previous similar messages
[ 5886.580077] LustreError: lustre-OST0000-osc-ffff95e682f03000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5886.611987] Lustre: 2355:0:(llite_lib.c:4339:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.115@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5900.283823] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 18:21:27 (1785795687)
[ 5902.981843] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5905.390845] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 18:21:32 (1785795692)
[ 5908.176390] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5910.992362] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 18:21:38 (1785795698)
[ 5914.117503] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5917.594841] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 18:21:44 (1785795704)
[ 5920.729898] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5923.856518] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 18:21:50 (1785795710)
[ 5925.431707] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5926.674405] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 18:21:54 (1785795714)
[ 5928.451951] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5930.964281] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 18:21:58 (1785795718)
[ 5933.128328] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5936.923182] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 18:22:02 (1785795722)
[ 5939.200529] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5942.706355] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 18:22:08 (1785795728)
[ 5944.478717] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5946.907888] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 18:22:14 (1785795734)
[ 5949.194489] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5951.308759] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 18:22:18 (1785795738)
[ 5952.720471] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5954.787996] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 18:22:22 (1785795742)
[ 5958.110352] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5959.993960] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 18:22:27 (1785795747)
[ 5983.713817] Lustre: 2354:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785795757/real 1785795757]  req@ffff95e685867480 x1872536393178752/t0(0) o400->lustre-MDT0000-mdc-ffff95e682f03000@192.168.201.115@tcp:12/10 lens 224/224 e 0 to 1 dl 1785795773 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5983.742460] Lustre: 2354:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 5998.418228] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 18:23:06 (1785795786)
[ 6029.102701] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 18:23:36 (1785795816)
[ 6029.754935] Lustre: *** cfs_fail_loc=51b, val=3***
[ 6042.632338] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 18:23:50 (1785795830)
[ 6043.410204] Lustre: *** cfs_fail_loc=51b, val=4***
[ 6056.571697] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 18:24:04 (1785795844)
[ 6057.145778] Lustre: *** cfs_fail_loc=50f, val=3***
[ 6068.824539] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 18:24:16 (1785795856)
[ 6069.226241] Lustre: *** cfs_fail_loc=50f, val=4***
[ 6079.427705] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 18:24:26 (1785795866)
[ 6079.818843] Lustre: *** cfs_fail_loc=510, val=3***
[ 6091.367590] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 18:24:39 (1785795879)
[ 6091.943850] Lustre: *** cfs_fail_loc=51b, val=3***
[ 6104.964993] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 18:24:52 (1785795892)
[ 6105.448444] Lustre: *** cfs_fail_loc=51c, val=3***
[ 6176.663676] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 18:26:03 (1785795963)
[ 6177.588526] LustreError: 82769:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 6181.438450] LustreError: lustre-MDT0000-mdc-ffff95e682f03000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6181.481839] LustreError: 82784:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff95e682f03000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff95e685946800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6181.531035] LustreError: 82784:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6181.672589] LustreError: 82769:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 6184.308844] LustreError: 82799:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 6187.761777] LustreError: lustre-MDT0000-mdc-ffff95e682f03000: operation ldlm_enqueue to node 192.168.201.115@tcp failed: rc = -107
[ 6187.772912] LustreError: Skipped 1 previous similar message
[ 6187.802549] LustreError: lustre-MDT0000-mdc-ffff95e682f03000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6187.826302] LustreError: 82814:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff95e682f03000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff95e686be8e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6187.857018] LustreError: 82814:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6188.329082] LustreError: 82799:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 6195.628422] LustreError: lustre-MDT0000-mdc-ffff95e682f03000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6195.663547] LustreError: 82842:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff95e682f03000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff95e68700ad00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6195.695168] LustreError: 82842:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6196.317323] LustreError: 82855:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 6200.319522] LustreError: lustre-MDT0000-mdc-ffff95e682f03000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6200.338041] LustreError: 82855:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 6200.377588] LustreError: 82855:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff95e682f03000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 6200.454521] LustreError: 82875:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 6200.469813] LustreError: 82875:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 17 previous similar messages
[ 6201.572499] LustreError: 82881:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 6201.593517] LustreError: 82881:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 6204.061518] LustreError: 82903:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 6204.084589] LustreError: 82903:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 6208.011554] LustreError: 82930:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 6211.262661] LustreError: lustre-MDT0000-mdc-ffff95e682f03000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6211.302091] LustreError: 82944:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff95e682f03000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff95e698637200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6212.024690] LustreError: 82930:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 6219.766630] LustreError: lustre-MDT0000-mdc-ffff95e682f03000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6220.384654] LustreError: 82986:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 6223.242993] LustreError: lustre-MDT0000-mdc-ffff95e682f03000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6223.268949] LustreError: 83000:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff95e682f03000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff95e698637e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6223.307986] LustreError: 83000:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6224.481267] LustreError: 82986:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 6228.183980] LustreError: lustre-MDT0000-mdc-ffff95e682f03000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6228.295835] LustreError: 83033:0:(file.c:6154:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 6228.312190] LustreError: 83033:0:(file.c:6154:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 6228.738351] LustreError: 83013:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff95e682f03000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 6237.770799] LustreError: 83085:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 6237.791461] LustreError: 83085:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 6241.875657] LustreError: 83085:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 6241.896393] LustreError: 83085:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 6241.997704] LustreError: lustre-MDT0000-mdc-ffff95e682f03000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6250.917611] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 18:27:18 (1785796038)
[ 6270.320873] LustreError: 68047:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 6274.400129] LustreError: 68047:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 6282.477850] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 18:27:50 (1785796070)
[ 6335.158308] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 18:28:42 (1785796122)
[ 6401.967583] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 18:29:49 (1785796189)
[ 6437.325416] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 18:30:25 (1785796225)
[ 6438.207706] Lustre: Mounted lustre-client
[ 6565.546762] Lustre: Unmounted lustre-client
[ 6576.100467] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 18:32:43 (1785796363)
[ 6577.479053] LustreError: 87370:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6581.000896] Lustre: lustre-OST0000-osc-ffff95e682f03000: Connection to lustre-OST0000 (at 192.168.201.115@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6581.018837] Lustre: Skipped 15 previous similar messages
[ 6581.049809] LustreError: lustre-OST0000-osc-ffff95e682f03000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6581.060319] LustreError: 87458:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff95e682f03000: namespace resource [0x240000400:0x37ad:0x0].0x0 (ffff95e698637000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6581.099304] LustreError: 87458:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6581.472186] LustreError: 87370:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6581.487330] Lustre: 2357:0:(llite_lib.c:4339:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.115@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6581.524870] Lustre: lustre-OST0000-osc-ffff95e682f03000: Connection restored to 192.168.201.115@tcp (at 192.168.201.115@tcp)
[ 6581.540645] Lustre: Skipped 15 previous similar messages
[ 6590.797676] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 18:32:58 (1785796378)
[ 6648.800344] Lustre: 88037:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785796383/real 1785796383]  req@ffff95e68590a680 x1872536393332096/t0(0) o101->lustre-MDT0000-mdc-ffff95e682f03000@192.168.201.115@tcp:12/10 lens 328/344 e 0 to 1 dl 1785796438 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6648.855929] Lustre: 88037:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 6658.467829] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 18:34:06 (1785796446)
[ 6661.063551] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6663.786662] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 18:34:10 (1785796450)
[ 6729.337911] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6731.606762] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 18:35:19 (1785796519)
[ 6790.596841] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 18:36:17 (1785796577)
[ 6793.735301] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6796.853123] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 18:36:23 (1785796583)
[ 6799.318705] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6803.301787] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 18:36:29 (1785796589)
[ 6852.553544] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 18:37:19 (1785796639)
[ 6866.448989] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6896.608651] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 192.168.201.115@tcp) was lost; in progress operations using this service will fail
[ 6896.630993] LustreError: Skipped 2 previous similar messages
[ 6905.967727] Lustre: Evicted from MGS (at 192.168.201.115@tcp) after server handle changed from 0xe91c1e98b061e1a8 to 0xe91c1e98b061fc2e
[ 6905.979220] Lustre: Skipped 2 previous similar messages
[ 6916.247707] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6918.127782] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6930.905292] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 18:38:38 (1785796718)
[ 6934.806943] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6936.743896] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 18:38:44 (1785796724)
[ 6957.582615] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 192.168.201.115@tcp) was lost; in progress operations using this service will fail
[ 6957.629741] Lustre: Evicted from MGS (at 192.168.201.115@tcp) after server handle changed from 0xe91c1e98b061fc2e to 0xe91c1e98b061fe49
[ 6975.273690] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 18:39:22 (1785796762)
[ 7043.033818] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 18:40:30 (1785796830)
[ 7053.732138] LustreError: lustre-OST0000-osc-ffff95e682f03000: operation ost_setattr to node 192.168.201.115@tcp failed: rc = -19
[ 7053.749652] LustreError: Skipped 9 previous similar messages
[ 7114.958430] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 7118.878344] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7205.344240] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 192.168.201.115@tcp) was lost; in progress operations using this service will fail
[ 7205.373366] LustreError: Skipped 1 previous similar message
[ 7215.598995] Lustre: Evicted from MGS (at 192.168.201.115@tcp) after server handle changed from 0xe91c1e98b0620056 to 0xe91c1e98b0620ea3
[ 7215.619066] Lustre: Skipped 1 previous similar message
[ 7215.649580] Lustre: MGC192.168.201.115@tcp: Connection restored to 192.168.201.115@tcp (at 192.168.201.115@tcp)
[ 7215.663725] Lustre: Skipped 9 previous similar messages
[ 7217.653754] Lustre: lustre-MDT0000-mdc-ffff95e682f03000: Connection to lustre-MDT0000 (at 192.168.201.115@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7217.673018] Lustre: Skipped 7 previous similar messages
[ 7234.761121] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 7236.880430] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7249.271919] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785796983/real 1785796983]  req@ffff95e69821c700 x1872536399897728/t0(0) o400->lustre-MDT0000-mdc-ffff95e682f03000@192.168.201.115@tcp:12/10 lens 224/224 e 0 to 1 dl 1785797038 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7249.324406] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 21 previous similar messages
[ 7292.044036] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 7294.320160] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7455.746127] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 18:47:23 (1785797243)
[ 7538.637876] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 7544.296866] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7825.431431] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 18:53:32 (1785797612)
[ 7993.824178] Lustre: 2354:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785797767/real 1785797767]  req@ffff95e6858fb480 x1872536411064832/t0(0) o400->MGC192.168.201.115@tcp@192.168.201.115@tcp:26/25 lens 224/224 e 0 to 1 dl 1785797783 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7993.848400] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 192.168.201.115@tcp) was lost; in progress operations using this service will fail
[ 7993.896158] Lustre: 2354:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 7993.938608] LustreError: Skipped 1 previous similar message
[ 7993.978688] Lustre: lustre-MDT0000-mdc-ffff95e682f03000: Connection to lustre-MDT0000 (at 192.168.201.115@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7993.993039] Lustre: Evicted from MGS (at 192.168.201.115@tcp) after server handle changed from 0xe91c1e98b06212af to 0xe91c1e98b071b6d2
[ 7994.000120] Lustre: Skipped 2 previous similar messages
[ 7994.018824] Lustre: Skipped 1 previous similar message
[ 7994.024332] Lustre: MGC192.168.201.115@tcp: Connection restored to 192.168.201.115@tcp (at 192.168.201.115@tcp)
[ 7994.031965] Lustre: Skipped 3 previous similar messages
[ 8035.234878] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 18:57:02 (1785797822)
[ 8037.574583] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 8039.801650] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 18:57:07 (1785797827)
[ 8042.147653] LustreError: lustre-MDT0000-mdc-ffff95e682f03000: operation ldlm_enqueue to node 192.168.201.115@tcp failed: rc = -107
[ 8042.155267] LustreError: Skipped 335 previous similar messages
[ 8042.169358] LustreError: lustre-MDT0000-mdc-ffff95e682f03000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 8051.584201] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 18:57:19 (1785797839)
[ 8219.118123] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.201.115@tcp) after server handle changed from 0xe91c1e98b071b9f7 to 0xe91c1e98b071bb01
[ 8219.129113] LustreError: lustre-OST0000-osc-ffff95e682f03000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 8221.890601] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 19:00:09 (1785798009)
[ 8268.439922] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 19:00:55 (1785798055)
[ 8269.550982] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 8271.317401] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 19:00:59 (1785798059)
[ 8273.577757] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 8275.483811] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 19:01:03 (1785798063)
[ 8313.698721] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 19:01:41 (1785798101)
[ 8357.355560] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 192.168.201.115@tcp) was lost; in progress operations using this service will fail
[ 8357.397282] Lustre: Evicted from MGS (at 192.168.201.115@tcp) after server handle changed from 0xe91c1e98b071b6d2 to 0xe91c1e98b071de55
[ 8380.270896] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 19:02:47 (1785798167)
[ 8382.072527] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 8385.072768] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 19:02:52 (1785798172)
[ 8386.869510] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 8389.323362] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 19:02:56 (1785798176)
[ 8408.665933] Lustre: Unmounted lustre-client
[ 8411.633795] Lustre: Mounted lustre-client
[ 8415.545108] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 8438.241232] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 192.168.201.115@tcp) was lost; in progress operations using this service will fail
[ 8448.959256] Lustre: Evicted from MGS (at 192.168.201.115@tcp) after server handle changed from 0xe91c1e98b071e75b to 0xe91c1e98b071ef6c
[ 8451.462953] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2