[ 1855.739258] LustreError: lustre-OST0000-osc-ffff9f7f85a5e000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1855.759282] Lustre: 2356:0:(llite_lib.c:4327:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.104@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1855.796850] Lustre: lustre-OST0000-osc-ffff9f7f85a5e000: Connection restored to 192.168.203.104@tcp (at 192.168.203.104@tcp)
[ 1855.808698] Lustre: Skipped 14 previous similar messages
[ 1861.968982] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 10:16:33 (1783952193)
[ 1864.166884] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1865.854474] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 10:16:36 (1783952196)
[ 1867.783737] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1869.809131] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 10:16:40 (1783952200)
[ 1892.831250] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 1892.866699] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xad71ec7478cb9fe8 to 0xad71ec7478cbba98
[ 1996.610555] LustreError: lustre-MDT0000-mdc-ffff9f7f85a5e000: operation ldlm_enqueue to node 192.168.203.104@tcp failed: rc = -19
[ 1996.616091] LustreError: Skipped 2 previous similar messages
[ 2014.692900] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 2024.879642] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xad71ec7478cbba98 to 0xad71ec7478cd78fb
[ 2042.162356] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 10:19:33 (1783952373)
[ 2042.817302] Lustre: *** cfs_fail_loc=305, val=0***
[ 2042.823058] Lustre: Skipped 4 previous similar messages
[ 2061.986172] LustreError: lustre-OST0001-osc-ffff9f7f85a5e000: operation ost_connect to node 192.168.203.104@tcp failed: rc = -75
[ 2085.354079] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 2085.381307] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xad71ec7478cd78fb to 0xad71ec7478cd7ca5
[ 2086.522158] Lustre: 15897:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.104@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2107.651765] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2109.271310] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2118.697700] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 10:20:49 (1783952449)
[ 2131.442255] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 2131.469780] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xad71ec7478cd7ca5 to 0xad71ec7478cd8144
[ 2132.622674] LustreError: lustre-MDT0000-mdc-ffff9f7f85a5e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2209.936197] Lustre: DEBUG MARKER: recovery-small test_29a: @@@@@@ FAIL: mds1: import is not in FULL state after 50
[ 2224.981220] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 10:22:36 (1783952556)
[ 2238.957491] LustreError: lustre-OST0000-osc-ffff9f7f85a5e000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2257.067683] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 10:23:08 (1783952588)
[ 2271.570719] LustreError: lustre-MDT0000-mdc-ffff9f7f85a5e000: operation ldlm_enqueue to node 192.168.203.104@tcp failed: rc = -19
[ 2271.586299] LustreError: Skipped 1 previous similar message
[ 2290.179617] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 2290.225512] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xad71ec7478cd8144 to 0xad71ec7478cdcf0f
[ 2307.481215] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2309.917539] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2392.552918] Lustre: 2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783952709/real 1783952709]  req@ffff9f7f90cf0380 x1870607592129792/t0(0) o400->MGC192.168.203.104@tcp@192.168.203.104@tcp:26/25 lens 224/224 e 0 to 1 dl 1783952725 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2392.657581] Lustre: 2358:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 2392.686902] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 2403.303397] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xad71ec7478cdcf0f to 0xad71ec7478cfbd06
[ 2422.647440] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2425.334609] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2490.064180] Lustre: lustre-MDT0000-mdc-ffff9f7f85a5e000: Connection to lustre-MDT0000 (at 192.168.203.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2490.081608] Lustre: Skipped 8 previous similar messages
[ 2509.808880] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 2509.863882] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xad71ec7478cfbd06 to 0xad71ec7478d18812
[ 2509.878389] Lustre: MGC192.168.203.104@tcp: Connection restored to 192.168.203.104@tcp (at 192.168.203.104@tcp)
[ 2509.893208] Lustre: Skipped 13 previous similar messages
[ 2527.520449] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2530.162634] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2558.665365] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 10:28:10 (1783952890)
[ 2562.304910] LustreError: lustre-MDT0000-mdc-ffff9f7f85a5e000: operation ldlm_enqueue to node 192.168.203.104@tcp failed: rc = -19
[ 2562.313265] LustreError: Skipped 2 previous similar messages
[ 2581.485967] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 2581.532561] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xad71ec7478d18812 to 0xad71ec7478d29101
[ 2591.684061] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2634.849439] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2671.953633] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2711.810180] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2755.039605] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 2755.056224] LustreError: Skipped 3 previous similar messages
[ 2765.290756] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xad71ec7478d395cf to 0xad71ec7478d4576e
[ 2765.302869] Lustre: Skipped 3 previous similar messages
[ 2774.268337] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2830.822254] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2915.467898] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 10:34:06 (1783953246)
[ 2961.544226] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2963.018539] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3260.562486] LustreError: lustre-OST0000-osc-ffff9f7f85a5e000: operation ldlm_enqueue to node 192.168.203.104@tcp failed: rc = -19
[ 3260.578342] LustreError: Skipped 12 previous similar messages
[ 3260.582450] Lustre: lustre-OST0000-osc-ffff9f7f85a5e000: Connection to lustre-OST0000 (at 192.168.203.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3260.606556] Lustre: Skipped 8 previous similar messages
[ 3303.127397] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3304.921842] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3630.229758] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3632.116501] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3886.764697] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 10:50:17 (1783954217)
[ 3904.481034] Lustre: 50541:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783954220/real 1783954220]  req@ffff9f7f859b8000 x1870607616769792/t0(0) o101->lustre-MDT0000-mdc-ffff9f7f85a5e000@192.168.203.104@tcp:12/10 lens 576/1152 e 0 to 1 dl 1783954236 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3904.506805] Lustre: 50541:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 3904.514881] Lustre: lustre-MDT0000-mdc-ffff9f7f85a5e000: Connection to lustre-MDT0000 (at 192.168.203.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3904.538492] Lustre: Skipped 1 previous similar message
[ 3904.578204] Lustre: lustre-MDT0000-mdc-ffff9f7f85a5e000: Connection restored to 192.168.203.104@tcp (at 192.168.203.104@tcp)
[ 3904.584060] Lustre: Skipped 16 previous similar messages
[ 3912.105368] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 10:50:43 (1783954243)
[ 3937.342491] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 10:51:08 (1783954268)
[ 3962.070620] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 10:51:33 (1783954293)
[ 3962.490830] Lustre: Mounted lustre-client
[ 3993.567168] Lustre: 2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783954310/real 1783954310]  req@ffff9f7f86233480 x1870607616786944/t0(0) o400->lustre-MDT0000-mdc-ffff9f7f85a5e000@192.168.203.104@tcp:12/10 lens 224/224 e 0 to 1 dl 1783954326 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3993.592739] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 3993.595332] Lustre: 2358:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3993.608064] LustreError: Skipped 2 previous similar messages
[ 3993.626847] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xad71ec7478d5a514 to 0xad71ec7478ec8726
[ 3993.630311] Lustre: Skipped 2 previous similar messages
[ 3993.637195] Lustre: MGC192.168.203.104@tcp: Connection restored to 192.168.203.104@tcp (at 192.168.203.104@tcp)
[ 3993.642571] Lustre: Skipped 2 previous similar messages
[ 4008.306814] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4009.893996] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4011.787196] Lustre: Unmounted lustre-client
[ 4017.521821] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 10:52:28 (1783954348)
[ 4155.361992] Lustre: 2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783954471/real 1783954471]  req@ffff9f7f85c06a00 x1870607616819712/t0(0) o4->lustre-OST0000-osc-ffff9f7f85a5e000@192.168.203.104@tcp:6/4 lens 488/448 e 0 to 1 dl 1783954487 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4155.399367] Lustre: 2358:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 69 previous similar messages
[ 4155.402373] Lustre: lustre-OST0000-osc-ffff9f7f85a5e000: Connection restored to 192.168.203.104@tcp (at 192.168.203.104@tcp)
[ 4155.458375] Lustre: Skipped 8 previous similar messages
[ 4314.053937] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 10:57:25 (1783954645)
[ 4362.854098] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 10:58:13 (1783954693)
[ 4367.089567] Lustre: Unmounted lustre-client
[ 4388.536892] Lustre: Mounted lustre-client
[ 4394.480249] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 10:58:45 (1783954725)
[ 4394.863075] LustreError: 56448:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4395.912945] LustreError: 56448:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4396.089374] Lustre: *** cfs_fail_loc=305, val=0***
[ 4419.876723] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 10:59:11 (1783954751)
[ 4420.333151] Lustre: Mounted lustre-client
[ 4422.796557] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4433.108667] Lustre: Unmounted lustre-client
[ 4439.843931] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 10:59:31 (1783954771)
[ 4530.646417] LustreError: lustre-MDT0000-mdc-ffff9f7f85bca000: operation mds_reint to node 192.168.203.104@tcp failed: rc = -19
[ 4530.657239] LustreError: Skipped 1 previous similar message
[ 4530.662420] Lustre: lustre-MDT0000-mdc-ffff9f7f85bca000: Connection to lustre-MDT0000 (at 192.168.203.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4530.679075] Lustre: Skipped 23 previous similar messages
[ 4548.575147] Lustre: 2357:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783954865/real 1783954865]  req@ffff9f7f879f2680 x1870607619492096/t0(0) o400->MGC192.168.203.104@tcp@192.168.203.104@tcp:26/25 lens 224/224 e 0 to 1 dl 1783954881 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4548.609461] Lustre: 2357:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 78 previous similar messages
[ 4548.625885] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 4558.824472] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xad71ec7478ec9fd7 to 0xad71ec7478ef53f0
[ 4558.853098] Lustre: MGC192.168.203.104@tcp: Connection restored to 192.168.203.104@tcp (at 192.168.203.104@tcp)
[ 4558.862774] Lustre: Skipped 11 previous similar messages
[ 4718.246342] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 11:04:09 (1783955049)
[ 4720.766389] Lustre: DEBUG MARKER: SKIP: recovery-small test_61 don't have skip orphan feature
[ 4722.788900] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 11:04:13 (1783955053)
[ 4723.293509] Lustre: Mounted lustre-client
[ 4730.530094] LustreError: lustre-OST0000-osc-ffff9f7f85bca000: operation ldlm_enqueue to node 192.168.203.104@tcp failed: rc = -107
[ 4742.087948] Lustre: Unmounted lustre-client
[ 4748.584513] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 11:04:39 (1783955079)
[ 4754.725238] LustreError: lustre-MDT0000-mdc-ffff9f7f85bca000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4754.746049] LustreError: 60128:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4762.361975] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 11:04:53 (1783955093)
[ 4762.688907] LustreError: 60832:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4767.711394] LustreError: 60832:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4767.724936] LustreError: 60832:0:(import.c:711:ptlrpc_connect_import_locked()) already connected
[ 4769.651080] LustreError: lustre-MDT0000-mdc-ffff9f7f85bca000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4769.669109] LustreError: 60849:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4769.723358] LustreError: 60854:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4770.793231] LustreError: 60860:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4770.802619] LustreError: 60860:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4773.060951] LustreError: 60883:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4773.082223] LustreError: 60883:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4777.671618] LustreError: 60927:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4777.689530] LustreError: 60927:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4787.429582] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 11:05:18 (1783955118)
[ 4826.256770] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4828.284384] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4843.775728] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 11:06:14 (1783955174)
[ 4882.083298] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4883.803865] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4896.684859] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 11:07:07 (1783955227)
[ 4959.966107] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4961.565667] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4969.011063] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 11:08:20 (1783955300)
[ 5004.263481] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5005.776677] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5012.720916] Lustre: Unmounted lustre-client
[ 5062.494373] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5064.997922] Lustre: Mounted lustre-client
[ 5071.337729] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 11:10:02 (1783955402)
[ 5074.209489] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5076.054700] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 11:10:06 (1783955406)
[ 5106.514468] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 11:10:37 (1783955437)
[ 5107.967563] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5109.738839] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 11:10:40 (1783955440)
[ 5111.493651] Lustre: *** cfs_fail_loc=805, val=0***
[ 5111.543058] Lustre: Mounted lustre-client
[ 5117.134657] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5137.394026] Lustre: lustre-MDT0000-mdc-ffff9f7f8610d000: Connection to lustre-MDT0000 (at 192.168.203.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5137.417972] Lustre: Skipped 11 previous similar messages
[ 5137.425455] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 5137.457295] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xad71ec7478f42372 to 0xad71ec7478f4259b
[ 5147.040112] Lustre: Unmounted lustre-client
[ 5154.801125] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 11:11:25 (1783955485)
[ 5172.192260] Lustre: 70605:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783955488/real 1783955488]  req@ffff9f7f8789df80 x1870607621546112/t0(0) o36->lustre-MDT0000-mdc-ffff9f7f85658800@192.168.203.104@tcp:12/10 lens 504/576 e 0 to 1 dl 1783955504 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'mkdir.0' uid:0 gid:0 projid:4294967295
[ 5172.227394] Lustre: 70605:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 5177.338397] Lustre: MGC192.168.203.104@tcp: Connection restored to 192.168.203.104@tcp (at 192.168.203.104@tcp)
[ 5177.343542] Lustre: Skipped 13 previous similar messages
[ 5182.523278] LustreError: 2354:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff9f7f86848000 x1870607621545600/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff9f7f85658800@192.168.203.104@tcp:12/10 lens 664/608 e 0 to 0 dl 1783955530 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5192.360729] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5194.379930] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5203.823233] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 11:12:15 (1783955535)
[ 5204.859252] LustreError: lustre-OST0000-osc-ffff9f7f85658800: operation ost_write to node 192.168.203.104@tcp failed: rc = -107
[ 5204.864903] LustreError: Skipped 2 previous similar messages
[ 5204.878248] LustreError: lustre-OST0000-osc-ffff9f7f85658800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5204.892230] Lustre: 2355:0:(llite_lib.c:4327:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.104@tcp:/lustre/fid: [0x200009872:0x6:0x0]// may get corrupted (rc -5)
[ 5204.944400] LustreError: 71977:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9f7f85658800: namespace resource [0x240000400:0x43a2:0x0].0x0 (ffff9f7f85a6f200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5212.908062] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 11:12:23 (1783955543)
[ 5214.149093] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5215.808518] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 11:12:26 (1783955546)
[ 5217.146444] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5218.624573] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 11:12:29 (1783955549)
[ 5219.985160] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5221.526824] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 11:12:32 (1783955552)
[ 5222.954298] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5224.460963] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 11:12:35 (1783955555)
[ 5225.824163] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5227.331528] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 11:12:38 (1783955558)
[ 5228.851889] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5230.511320] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 11:12:41 (1783955561)
[ 5232.150933] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5233.743136] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 11:12:44 (1783955564)
[ 5235.282118] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5236.895342] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 11:12:48 (1783955568)
[ 5238.468579] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5240.061665] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 11:12:51 (1783955571)
[ 5241.566684] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5243.251464] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 11:12:54 (1783955574)
[ 5244.878481] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5246.682801] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 11:12:57 (1783955577)
[ 5249.332975] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5251.165442] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 11:13:02 (1783955582)
[ 5269.476551] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 5269.493547] LustreError: Skipped 1 previous similar message
[ 5269.519481] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xad71ec7478f42b67 to 0xad71ec7478f431d4
[ 5269.541488] Lustre: Skipped 1 previous similar message
[ 5282.202352] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 11:13:33 (1783955613)
[ 5311.260384] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 11:14:02 (1783955642)
[ 5312.029203] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5321.862428] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 11:14:13 (1783955653)
[ 5322.084292] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5331.943470] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 11:14:23 (1783955663)
[ 5332.223379] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5340.326789] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 11:14:31 (1783955671)
[ 5340.588819] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5349.531026] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 11:14:40 (1783955680)
[ 5350.083526] Lustre: *** cfs_fail_loc=510, val=3***
[ 5359.188824] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 11:14:50 (1783955690)
[ 5359.588789] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5425.905780] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 11:15:57 (1783955757)
[ 5426.315591] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5492.699595] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 11:17:03 (1783955823)
[ 5493.018978] LustreError: 82173:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5495.873706] LustreError: lustre-MDT0000-mdc-ffff9f7f85658800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5495.908122] LustreError: 82188:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9f7f85658800: namespace resource [0x200009872:0x11:0x0].0xc (ffff9f7f86161d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5497.104633] LustreError: 82173:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5499.220586] LustreError: 82203:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5502.269092] LustreError: lustre-MDT0000-mdc-ffff9f7f85658800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5502.288697] LustreError: 82217:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9f7f85658800: namespace resource [0x200009872:0x11:0x0].0xc (ffff9f7f86161000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5502.317327] LustreError: 82217:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5503.287408] LustreError: 82203:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5510.142406] LustreError: lustre-MDT0000-mdc-ffff9f7f85658800: operation ldlm_enqueue to node 192.168.203.104@tcp failed: rc = -107
[ 5510.149991] LustreError: Skipped 2 previous similar messages
[ 5510.164334] LustreError: lustre-MDT0000-mdc-ffff9f7f85658800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5510.484437] LustreError: 82259:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5513.368963] LustreError: lustre-MDT0000-mdc-ffff9f7f85658800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5513.457889] LustreError: 82278:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5513.464299] LustreError: 82278:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 16 previous similar messages
[ 5514.554422] LustreError: 82285:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5514.572433] LustreError: 82285:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5514.572637] LustreError: 82259:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5514.597991] LustreError: 82259:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9f7f85658800: inode [0x200009872:0x11:0x0] mdc close failed: rc = -108
[ 5516.833343] LustreError: 82307:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5516.844455] LustreError: 82307:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5520.466705] LustreError: 82334:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5523.315358] LustreError: lustre-MDT0000-mdc-ffff9f7f85658800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5523.346718] LustreError: 82348:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9f7f85658800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9f7f87ea8700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5523.395254] LustreError: 82348:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5524.541615] LustreError: 82334:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5531.629731] LustreError: lustre-MDT0000-mdc-ffff9f7f85658800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5531.660916] LustreError: 82377:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9f7f85658800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9f7f86161f00) refcount nonzero (2) after lock cleanup; forcing cleanup.
[ 5531.695098] LustreError: 82377:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5532.093435] LustreError: 82390:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5534.964067] LustreError: lustre-MDT0000-mdc-ffff9f7f85658800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5536.167172] LustreError: 82390:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5539.231959] LustreError: lustre-MDT0000-mdc-ffff9f7f85658800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5539.313852] LustreError: 82437:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5539.322532] LustreError: 82437:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5540.367696] LustreError: 82418:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9f7f85658800: inode [0x200009872:0x11:0x0] mdc close failed: rc = -108
[ 5548.241609] LustreError: 82489:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5548.258988] LustreError: 82489:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5551.825809] LustreError: lustre-MDT0000-mdc-ffff9f7f85658800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5551.919279] LustreError: 82512:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5551.926474] LustreError: 82512:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5552.327185] LustreError: 82489:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5552.344706] LustreError: 82489:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5552.358835] LustreError: 82489:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9f7f85658800: inode [0x200009872:0x11:0x0] mdc close failed: rc = -108
[ 5563.084650] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 11:18:14 (1783955894)
[ 5582.939885] LustreError: 67454:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5582.948181] LustreError: 67454:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5587.016381] LustreError: 67454:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5587.029481] LustreError: 67454:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5593.565435] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 11:18:44 (1783955924)
[ 5642.243392] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 11:19:33 (1783955973)
[ 5706.204036] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 11:20:37 (1783956037)
[ 5738.904085] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 11:21:10 (1783956070)
[ 5739.362277] Lustre: Mounted lustre-client
[ 5862.945423] Lustre: Unmounted lustre-client
[ 5869.954233] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 11:23:21 (1783956201)
[ 5870.514858] LustreError: 86816:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5872.981533] Lustre: lustre-OST0000-osc-ffff9f7f85658800: Connection to lustre-OST0000 (at 192.168.203.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5872.996879] Lustre: Skipped 18 previous similar messages
[ 5873.013606] LustreError: lustre-OST0000-osc-ffff9f7f85658800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5873.039155] LustreError: 86899:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9f7f85658800: namespace resource [0x240000400:0x43cd:0x0].0x0 (ffff9f7f86161100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5873.056227] LustreError: 86899:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5873.447191] LustreError: 86816:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5873.456785] Lustre: 2355:0:(llite_lib.c:4327:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.104@tcp:/lustre/fid: [0x20000b7b9:0x9:0x0]// may get corrupted (rc -108)
[ 5879.508753] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 11:23:30 (1783956210)
[ 5937.119203] Lustre: 87484:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783956214/real 1783956214]  req@ffff9f7f8789ea00 x1870607621721472/t0(0) o101->lustre-MDT0000-mdc-ffff9f7f85658800@192.168.203.104@tcp:12/10 lens 328/344 e 0 to 1 dl 1783956269 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5937.145010] Lustre: 87484:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 5937.163717] Lustre: lustre-MDT0000-mdc-ffff9f7f85658800: Connection restored to 192.168.203.104@tcp (at 192.168.203.104@tcp)
[ 5937.178522] Lustre: Skipped 15 previous similar messages
[ 5943.452375] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 11:24:34 (1783956274)
[ 5944.675418] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5946.208584] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 11:24:37 (1783956277)
[ 6008.633164] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6010.071643] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 11:25:41 (1783956341)
[ 6067.340896] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 11:26:38 (1783956398)
[ 6069.151543] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6070.579678] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 11:26:41 (1783956401)
[ 6071.918872] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6073.145944] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 11:26:44 (1783956404)
[ 6098.826690] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 11:27:10 (1783956430)
[ 6106.727263] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6127.598989] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 6127.622880] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xad71ec7478f431d4 to 0xad71ec7478f44a1c
[ 6137.879517] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6139.010725] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6147.978833] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 11:27:59 (1783956479)
[ 6150.001564] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6151.433446] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 11:28:02 (1783956482)
[ 6173.281501] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 11:28:24 (1783956504)
[ 6209.454237] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 11:29:00 (1783956540)
[ 6215.714355] LustreError: lustre-OST0000-osc-ffff9f7f85658800: operation ost_setattr to node 192.168.203.104@tcp failed: rc = -107
[ 6215.723848] LustreError: Skipped 7 previous similar messages
[ 6260.839353] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6264.190898] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6349.865092] Lustre: 67457:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.104@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6374.320524] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6375.810733] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6407.700316] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6408.854912] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6533.979217] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 11:34:25 (1783956865)
[ 6544.312652] Lustre: lustre-OST0000-osc-ffff9f7f85658800: Connection to lustre-OST0000 (at 192.168.203.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6544.323316] Lustre: Skipped 9 previous similar messages
[ 6595.638176] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6598.323357] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6844.154253] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 11:39:35 (1783957175)
[ 6999.018563] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 192.168.203.104@tcp) was lost; in progress operations using this service will fail
[ 6999.030258] LustreError: Skipped 4 previous similar messages
[ 6999.054806] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0xad71ec7478f464be to 0xad71ec7479037d4a
[ 6999.060794] Lustre: Skipped 4 previous similar messages
[ 6999.066756] Lustre: MGC192.168.203.104@tcp: Connection restored to 192.168.203.104@tcp (at 192.168.203.104@tcp)
[ 6999.070963] Lustre: Skipped 12 previous similar messages
[ 7028.829177] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 11:42:40 (1783957360)
[ 7030.242782] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7032.439336] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 11:42:42 (1783957362)
[ 7034.054119] LustreError: lustre-MDT0000-mdc-ffff9f7f85658800: operation ldlm_enqueue to node 192.168.203.104@tcp failed: rc = -107
[ 7034.072869] LustreError: Skipped 173 previous similar messages
[ 7034.092613] LustreError: lustre-MDT0000-mdc-ffff9f7f85658800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7040.394101] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 11:42:51 (1783957371)
[ 7206.889154] LustreError: lustre-OST0000-osc-ffff9f7f85658800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7209.066758] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 11:45:39 (1783957539)
[ 7234.530017] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783957546/real 1783957546]  req@ffff9f7faba53b80 x1870607638789760/t0(0) o4->lustre-OST0000-osc-ffff9f7f85658800@192.168.203.104@tcp:6/4 lens 4584/448 e 0 to 1 dl 1783957566 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7234.551262] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 7234.556556] Lustre: lustre-OST0000-osc-ffff9f7f85658800: Connection to lustre-OST0000 (at 192.168.203.104@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7234.565407] Lustre: Skipped 3 previous similar messages
[ 7251.397910] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 11:46:22 (1783957582)
[ 7252.910678] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7254.591600] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 11:46:25 (1783957585)
[ 7256.365565] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7258.123616] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 11:46:29 (1783957589)
[ 7293.314750] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 11:47:04 (1783957624)
[ 7346.186855] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 11:47:57 (1783957677)
[ 7347.429279] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7349.125877] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 11:48:00 (1783957680)
[ 7350.561415] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7352.095531] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 11:48:03 (1783957683)
[ 7366.153434] Lustre: Unmounted lustre-client
[ 7367.725655] Lustre: Mounted lustre-client
[ 7370.189885] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7397.745899] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2