[ 1915.520453] LustreError: lustre-OST0000-osc-ffff9e365053c800: operation ost_sync to node 192.168.203.105@tcp failed: rc = -107
[ 1915.551783] LustreError: lustre-OST0000-osc-ffff9e365053c800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1915.563445] Lustre: 2355:0:(llite_lib.c:4202:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.105@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1915.571912] Lustre: lustre-OST0000-osc-ffff9e365053c800: Connection restored to 192.168.203.105@tcp (at 192.168.203.105@tcp)
[ 1915.578965] Lustre: Skipped 14 previous similar messages
[ 1922.001496] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 18:54:48 (1786402488)
[ 1924.134503] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1925.562431] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 18:54:51 (1786402491)
[ 1927.456973] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1929.477954] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 18:54:55 (1786402495)
[ 1935.042935] LustreError: lustre-MDT0000-mdc-ffff9e365053c800: operation mds_reint to node 192.168.203.105@tcp failed: rc = -19
[ 1952.735675] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 192.168.203.105@tcp) was lost; in progress operations using this service will fail
[ 1962.985743] Lustre: Evicted from MGS (at 192.168.203.105@tcp) after server handle changed from 0x496bfb278a3719f2 to 0x496bfb278a372bdb
[ 2068.369686] LustreError: lustre-MDT0000-mdc-ffff9e365053c800: operation mds_close to node 192.168.203.105@tcp failed: rc = -107
[ 2068.422515] LustreError: Skipped 2 previous similar messages
[ 2086.368067] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 192.168.203.105@tcp) was lost; in progress operations using this service will fail
[ 2095.660719] Lustre: Evicted from MGS (at 192.168.203.105@tcp) after server handle changed from 0x496bfb278a372bdb to 0x496bfb278a38438e
[ 2113.893257] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 18:57:59 (1786402679)
[ 2114.576503] Lustre: *** cfs_fail_loc=305, val=0***
[ 2114.579487] Lustre: Skipped 4 previous similar messages
[ 2156.579069] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 192.168.203.105@tcp) was lost; in progress operations using this service will fail
[ 2156.605430] Lustre: Evicted from MGS (at 192.168.203.105@tcp) after server handle changed from 0x496bfb278a38438e to 0x496bfb278a384a41
[ 2169.349297] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2171.550795] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2182.217359] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 18:59:07 (1786402747)
[ 2192.381601] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 192.168.203.105@tcp) was lost; in progress operations using this service will fail
[ 2192.495708] Lustre: Evicted from MGS (at 192.168.203.105@tcp) after server handle changed from 0x496bfb278a384a41 to 0x496bfb278a384d51
[ 2193.867198] LustreError: lustre-MDT0000-mdc-ffff9e365053c800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2218.104080] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 18:59:44 (1786402784)
[ 2233.319788] LustreError: lustre-OST0000-osc-ffff9e365053c800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2248.222656] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 19:00:14 (1786402814)
[ 2261.635516] LustreError: lustre-MDT0000-mdc-ffff9e365053c800: operation ldlm_enqueue to node 192.168.203.105@tcp failed: rc = -19
[ 2279.668079] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 192.168.203.105@tcp) was lost; in progress operations using this service will fail
[ 2290.202987] Lustre: Evicted from MGS (at 192.168.203.105@tcp) after server handle changed from 0x496bfb278a384d51 to 0x496bfb278a387957
[ 2314.154231] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2319.854467] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2407.417542] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 192.168.203.105@tcp) was lost; in progress operations using this service will fail
[ 2407.577552] Lustre: Evicted from MGS (at 192.168.203.105@tcp) after server handle changed from 0x496bfb278a387957 to 0x496bfb278a3985bf
[ 2429.769755] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2433.209446] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2502.139539] LustreError: lustre-MDT0000-mdc-ffff9e365053c800: operation mds_reint to node 192.168.203.105@tcp failed: rc = -19
[ 2502.182917] LustreError: Skipped 2 previous similar messages
[ 2521.055316] Lustre: 2355:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786403072/real 1786403072]  req@ffff9e364638aa00 x1873176880739968/t0(0) o400->MGC192.168.203.105@tcp@192.168.203.105@tcp:26/25 lens 224/224 e 0 to 1 dl 1786403088 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2521.070073] Lustre: 2355:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 2521.074421] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 192.168.203.105@tcp) was lost; in progress operations using this service will fail
[ 2531.257253] Lustre: Evicted from MGS (at 192.168.203.105@tcp) after server handle changed from 0x496bfb278a3985bf to 0x496bfb278a3aacc9
[ 2531.301407] Lustre: MGC192.168.203.105@tcp: Connection restored to 192.168.203.105@tcp (at 192.168.203.105@tcp)
[ 2531.325218] Lustre: Skipped 13 previous similar messages
[ 2550.639039] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2553.007720] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2584.038457] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 19:05:50 (1786403150)
[ 2589.357254] Lustre: lustre-MDT0000-mdc-ffff9e365053c800: Connection to lustre-MDT0000 (at 192.168.203.105@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2589.365119] Lustre: Skipped 9 previous similar messages
[ 2608.095234] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 192.168.203.105@tcp) was lost; in progress operations using this service will fail
[ 2618.321422] Lustre: Evicted from MGS (at 192.168.203.105@tcp) after server handle changed from 0x496bfb278a3aacc9 to 0x496bfb278a3b3152
[ 2625.412900] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2654.725854] Lustre: 15907:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.105@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2667.909976] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2703.218173] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2739.231464] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 192.168.203.105@tcp) was lost; in progress operations using this service will fail
[ 2739.239483] LustreError: Skipped 2 previous similar messages
[ 2747.141126] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2772.217627] LustreError: lustre-MDT0000-mdc-ffff9e365053c800: operation mds_close to node 192.168.203.105@tcp failed: rc = -19
[ 2772.238361] LustreError: Skipped 4 previous similar messages
[ 2792.430550] Lustre: Evicted from MGS (at 192.168.203.105@tcp) after server handle changed from 0x496bfb278a3b65a1 to 0x496bfb278a3b90dc
[ 2792.435409] Lustre: Skipped 3 previous similar messages
[ 2794.358472] Lustre: 15907:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.105@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2809.589363] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2873.414814] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2962.663350] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 19:12:08 (1786403528)
[ 3031.709945] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3036.905589] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3308.775730] LustreError: lustre-OST0000-osc-ffff9e365053c800: operation ldlm_enqueue to node 192.168.203.105@tcp failed: rc = -19
[ 3308.795513] LustreError: Skipped 4 previous similar messages
[ 3308.830228] Lustre: lustre-OST0000-osc-ffff9e365053c800: Connection to lustre-OST0000 (at 192.168.203.105@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3308.841561] Lustre: Skipped 6 previous similar messages
[ 3359.657302] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3363.784032] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3694.546470] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3699.999290] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3934.815681] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 19:28:20 (1786404500)
[ 3951.583314] Lustre: 50351:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786404503/real 1786404503]  req@ffff9e3648960700 x1873176892097280/t0(0) o101->lustre-MDT0000-mdc-ffff9e365053c800@192.168.203.105@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786404519 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3951.595313] Lustre: 50351:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 3951.599089] Lustre: lustre-MDT0000-mdc-ffff9e365053c800: Connection to lustre-MDT0000 (at 192.168.203.105@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3951.605761] Lustre: Skipped 1 previous similar message
[ 3951.718178] Lustre: lustre-MDT0000-mdc-ffff9e365053c800: Connection restored to 192.168.203.105@tcp (at 192.168.203.105@tcp)
[ 3951.723565] Lustre: Skipped 15 previous similar messages
[ 3962.297821] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 19:28:48 (1786404528)
[ 3989.212318] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 19:29:15 (1786404555)
[ 4014.884425] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 19:29:41 (1786404581)
[ 4015.537864] Lustre: Mounted lustre-client
[ 4046.346524] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 192.168.203.105@tcp) was lost; in progress operations using this service will fail
[ 4046.353091] LustreError: Skipped 3 previous similar messages
[ 4046.441797] Lustre: Evicted from MGS (at 192.168.203.105@tcp) after server handle changed from 0x496bfb278a3c794f to 0x496bfb278a47f517
[ 4046.447302] Lustre: Skipped 2 previous similar messages
[ 4046.452838] Lustre: MGC192.168.203.105@tcp: Connection restored to 192.168.203.105@tcp (at 192.168.203.105@tcp)
[ 4046.457967] Lustre: Skipped 2 previous similar messages
[ 4047.391189] Lustre: 2355:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786404598/real 1786404598]  req@ffff9e3648391880 x1873176892115200/t0(0) o400->lustre-MDT0000-mdc-ffff9e365053c800@192.168.203.105@tcp:12/10 lens 224/224 e 0 to 1 dl 1786404614 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4047.427478] Lustre: 2355:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4060.105657] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4061.777988] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4063.931688] Lustre: Unmounted lustre-client
[ 4071.672144] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 19:30:37 (1786404637)
[ 4208.095140] Lustre: 2354:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786404759/real 1786404759]  req@ffff9e3648f51880 x1873176892148352/t0(0) o4->lustre-OST0000-osc-ffff9e365053c800@192.168.203.105@tcp:6/4 lens 488/448 e 0 to 1 dl 1786404775 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4208.113366] Lustre: lustre-OST0000-osc-ffff9e365053c800: Connection restored to 192.168.203.105@tcp (at 192.168.203.105@tcp)
[ 4208.119147] Lustre: 2354:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 64 previous similar messages
[ 4208.148978] Lustre: Skipped 8 previous similar messages
[ 4368.764773] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 19:35:34 (1786404934)
[ 4417.741588] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 19:36:23 (1786404983)
[ 4420.416283] Lustre: Unmounted lustre-client
[ 4441.315180] Lustre: Mounted lustre-client
[ 4448.406606] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 19:36:53 (1786405013)
[ 4448.729765] LustreError: 56254:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4449.792176] LustreError: 56254:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4449.859859] Lustre: *** cfs_fail_loc=305, val=0***
[ 4475.328155] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 19:37:21 (1786405041)
[ 4476.182285] Lustre: Mounted lustre-client
[ 4477.980040] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4488.254737] Lustre: Unmounted lustre-client
[ 4496.947349] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 19:37:42 (1786405062)
[ 4592.843966] LustreError: lustre-MDT0000-mdc-ffff9e3649946800: operation mds_reint to node 192.168.203.105@tcp failed: rc = -107
[ 4592.850716] LustreError: Skipped 1 previous similar message
[ 4592.855186] Lustre: lustre-MDT0000-mdc-ffff9e3649946800: Connection to lustre-MDT0000 (at 192.168.203.105@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4592.860623] Lustre: Skipped 23 previous similar messages
[ 4610.527127] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786405161/real 1786405161]  req@ffff9e3647047b80 x1873176894810368/t0(0) o400->MGC192.168.203.105@tcp@192.168.203.105@tcp:26/25 lens 224/224 e 0 to 1 dl 1786405177 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4610.549698] Lustre: 2356:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 80 previous similar messages
[ 4610.556115] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 192.168.203.105@tcp) was lost; in progress operations using this service will fail
[ 4619.749502] Lustre: Evicted from MGS (at 192.168.203.105@tcp) after server handle changed from 0x496bfb278a481117 to 0x496bfb278a4abea0
[ 4619.773975] Lustre: MGC192.168.203.105@tcp: Connection restored to 192.168.203.105@tcp (at 192.168.203.105@tcp)
[ 4619.790522] Lustre: Skipped 11 previous similar messages
[ 4803.729224] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 19:42:49 (1786405369)
[ 4810.430963] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4824.163421] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 192.168.203.105@tcp) was lost; in progress operations using this service will fail
[ 4824.215777] Lustre: Evicted from MGS (at 192.168.203.105@tcp) after server handle changed from 0x496bfb278a4abea0 to 0x496bfb278a4f96b1
[ 4824.264756] LustreError: lustre-MDT0000-mdc-ffff9e3649946800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4842.476438] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 19:43:28 (1786405408)
[ 4843.390367] Lustre: Mounted lustre-client
[ 4850.942172] LustreError: lustre-OST0000-osc-ffff9e3649946800: operation ldlm_enqueue to node 192.168.203.105@tcp failed: rc = -107
[ 4850.976804] LustreError: lustre-OST0000-osc-ffff9e3649946800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4850.995908] LustreError: 59972:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9e3649946800: namespace resource [0x240000400:0x28a3:0x0].0x0 (ffff9e36443fa100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4862.434252] Lustre: Unmounted lustre-client
[ 4871.503913] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 19:43:57 (1786405437)
[ 4879.245727] LustreError: lustre-MDT0000-mdc-ffff9e3649946800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4879.254608] LustreError: 60643:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4887.434879] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 19:44:13 (1786405453)
[ 4887.572291] LustreError: 61340:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4892.639465] LustreError: 61340:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4892.652929] LustreError: 61340:0:(import.c:711:ptlrpc_connect_import_locked()) already connected
[ 4893.838817] LustreError: lustre-MDT0000-mdc-ffff9e3649946800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4893.844955] LustreError: 61365:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4893.888190] LustreError: 61370:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4894.977625] LustreError: 61376:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4894.988162] LustreError: 61376:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4897.165127] LustreError: 61398:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4897.171609] LustreError: 61398:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4901.758249] LustreError: 61443:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4901.763612] LustreError: 61443:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4914.014262] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 19:44:39 (1786405479)
[ 4957.509582] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4959.406058] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4971.671796] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 19:45:37 (1786405537)
[ 5012.227301] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5014.379796] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5025.776472] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 19:46:32 (1786405592)
[ 5090.160451] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5092.134851] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5100.710386] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 19:47:46 (1786405666)
[ 5136.898516] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5138.353134] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5144.394346] Lustre: Unmounted lustre-client
[ 5197.185509] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5200.448242] Lustre: Mounted lustre-client
[ 5208.760942] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 19:49:33 (1786405773)
[ 5211.967761] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5214.336925] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 19:49:39 (1786405779)
[ 5243.609418] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 19:50:09 (1786405809)
[ 5245.045507] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5247.017729] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 19:50:13 (1786405813)
[ 5249.214239] Lustre: *** cfs_fail_loc=805, val=0***
[ 5249.288379] Lustre: Mounted lustre-client
[ 5254.768808] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5275.106359] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 192.168.203.105@tcp) was lost; in progress operations using this service will fail
[ 5275.118934] Lustre: Evicted from MGS (at 192.168.203.105@tcp) after server handle changed from 0x496bfb278a4fa0f9 to 0x496bfb278a4fa314
[ 5275.124791] Lustre: MGC192.168.203.105@tcp: Connection restored to 192.168.203.105@tcp (at 192.168.203.105@tcp)
[ 5275.129090] Lustre: Skipped 10 previous similar messages
[ 5276.127156] Lustre: 2355:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786405827/real 1786405827]  req@ffff9e364419f100 x1873176896886016/t0(0) o400->lustre-MDT0000-mdc-ffff9e3647fa8800@192.168.203.105@tcp:12/10 lens 224/224 e 0 to 1 dl 1786405843 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5276.127297] Lustre: lustre-MDT0000-mdc-ffff9e365053e800: Connection to lustre-MDT0000 (at 192.168.203.105@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5276.142849] Lustre: 2355:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5276.200993] Lustre: Skipped 11 previous similar messages
[ 5276.521502] Lustre: 67956:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.105@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5292.836716] Lustre: Unmounted lustre-client
[ 5301.539291] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 19:51:07 (1786405867)
[ 5329.497358] LustreError: 2352:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff9e36442f0380 x1873176896898432/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff9e3647fa8800@192.168.203.105@tcp:12/10 lens 664/608 e 0 to 0 dl 1786405912 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5337.398978] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5339.476651] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5347.975799] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 19:51:54 (1786405914)
[ 5350.089731] LustreError: lustre-OST0000-osc-ffff9e3647fa8800: operation ost_write to node 192.168.203.105@tcp failed: rc = -107
[ 5350.123079] LustreError: Skipped 2 previous similar messages
[ 5350.174092] LustreError: 2352:0:(osc_request.c:1024:osc_init_grant()) lustre-OST0000-osc-ffff9e3647fa8800: granted 3407872 but already consumed 5111808
[ 5350.208726] LustreError: lustre-OST0000-osc-ffff9e3647fa8800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5350.232969] Lustre: 2353:0:(llite_lib.c:4202:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.105@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5359.782173] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 19:52:05 (1786405925)
[ 5362.022600] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5363.212991] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 19:52:09 (1786405929)
[ 5365.145925] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5367.137536] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 19:52:12 (1786405932)
[ 5368.962563] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5371.170461] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 19:52:16 (1786405936)
[ 5373.202292] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5374.966631] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 19:52:21 (1786405941)
[ 5376.777861] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5378.599437] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 19:52:24 (1786405944)
[ 5380.234502] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5382.177898] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 19:52:28 (1786405948)
[ 5383.993682] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5385.422601] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 19:52:31 (1786405951)
[ 5386.991438] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5389.097409] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 19:52:34 (1786405954)
[ 5390.952354] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5392.965906] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 19:52:39 (1786405959)
[ 5394.516662] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5396.366572] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 19:52:42 (1786405962)
[ 5398.305840] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5400.465807] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 19:52:46 (1786405966)
[ 5403.903345] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5405.853545] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 19:52:51 (1786405971)
[ 5438.045897] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 19:53:23 (1786406003)
[ 5469.085994] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 19:53:54 (1786406034)
[ 5469.619315] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5480.758881] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 19:54:06 (1786406046)
[ 5481.593434] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5492.569652] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 19:54:18 (1786406058)
[ 5492.973751] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5502.820455] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 19:54:28 (1786406068)
[ 5503.143832] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5512.946181] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 19:54:38 (1786406078)
[ 5513.325994] Lustre: *** cfs_fail_loc=510, val=3***
[ 5523.567394] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 19:54:49 (1786406089)
[ 5524.620849] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5537.081714] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 19:55:03 (1786406103)
[ 5538.619429] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5606.556984] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 19:56:12 (1786406172)
[ 5607.170561] LustreError: 82667:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5610.391552] LustreError: lustre-MDT0000-mdc-ffff9e3647fa8800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5610.404476] LustreError: 82682:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9e3647fa8800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9e36443fa900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5611.247164] LustreError: 82667:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5613.415766] LustreError: 82698:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5616.849447] LustreError: lustre-MDT0000-mdc-ffff9e3647fa8800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5616.860829] LustreError: 82712:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9e3647fa8800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9e3648e7ec00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5616.907795] LustreError: 82712:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5617.447133] LustreError: 82698:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5624.255393] LustreError: lustre-MDT0000-mdc-ffff9e3647fa8800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5624.282644] LustreError: 82740:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9e3647fa8800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9e36443fa000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5624.315868] LustreError: 82740:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5624.922872] LustreError: 82753:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5627.523410] LustreError: lustre-MDT0000-mdc-ffff9e3647fa8800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5627.617464] LustreError: 82773:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5627.628571] LustreError: 82773:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5628.735402] LustreError: 82779:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5628.752644] LustreError: 82779:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5628.959431] LustreError: 82753:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5628.968820] LustreError: 82753:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e3647fa8800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5630.989112] LustreError: 82802:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5631.003719] LustreError: 82802:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5634.657165] LustreError: 82828:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5637.650836] LustreError: lustre-MDT0000-mdc-ffff9e3647fa8800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5637.670482] LustreError: 82843:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9e3647fa8800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9e36443fad00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5638.711117] LustreError: 82828:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5645.305661] LustreError: lustre-MDT0000-mdc-ffff9e3647fa8800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5645.712675] LustreError: 82884:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5648.943709] LustreError: lustre-MDT0000-mdc-ffff9e3647fa8800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5648.957833] LustreError: 82899:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9e3647fa8800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9e364702fc00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5649.012991] LustreError: 82899:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5649.727126] LustreError: 82884:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5653.057940] LustreError: lustre-MDT0000-mdc-ffff9e3647fa8800: operation ldlm_enqueue to node 192.168.203.105@tcp failed: rc = -107
[ 5653.070333] LustreError: Skipped 8 previous similar messages
[ 5653.140602] LustreError: lustre-MDT0000-mdc-ffff9e3647fa8800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5653.281550] LustreError: 82932:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5653.289172] LustreError: 82932:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5653.794295] LustreError: 82912:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e3647fa8800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5662.541224] LustreError: 82984:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5662.556671] LustreError: 82984:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5665.838992] LustreError: lustre-MDT0000-mdc-ffff9e3647fa8800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5665.917314] LustreError: 83006:0:(file.c:6088:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5665.924100] LustreError: 83006:0:(file.c:6088:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5666.623323] LustreError: 82984:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5666.632721] LustreError: 82984:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5666.641718] LustreError: 82984:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e3647fa8800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5677.544260] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 19:57:23 (1786406243)
[ 5697.245035] LustreError: 79417:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5697.265322] LustreError: 79417:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5701.255137] LustreError: 79417:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5701.262206] LustreError: 79417:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5709.225797] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 19:57:55 (1786406275)
[ 5761.161822] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 19:58:46 (1786406326)
[ 5826.936531] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 19:59:52 (1786406392)
[ 5862.649780] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 20:00:28 (1786406428)
[ 5863.470122] Lustre: Mounted lustre-client
[ 5987.933100] Lustre: Unmounted lustre-client
[ 5996.162912] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 20:02:41 (1786406561)
[ 5997.354978] LustreError: 87324:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6000.757666] Lustre: lustre-OST0000-osc-ffff9e3647fa8800: Connection to lustre-OST0000 (at 192.168.203.105@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6000.765438] Lustre: Skipped 16 previous similar messages
[ 6000.781804] LustreError: lustre-OST0000-osc-ffff9e3647fa8800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6000.791289] LustreError: 87406:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9e3647fa8800: namespace resource [0x240000400:0x292d:0x0].0x0 (ffff9e36443fa500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6000.802637] LustreError: 87406:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6001.151144] LustreError: 87324:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6001.161973] Lustre: 2356:0:(llite_lib.c:4202:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.105@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6008.821553] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 20:02:54 (1786406574)
[ 6066.143487] Lustre: 87993:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786406578/real 1786406578]  req@ffff9e36442f3800 x1873176897072128/t0(0) o101->lustre-MDT0000-mdc-ffff9e3647fa8800@192.168.203.105@tcp:12/10 lens 328/344 e 0 to 1 dl 1786406633 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6066.192104] Lustre: 87993:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 21 previous similar messages
[ 6066.252279] Lustre: lustre-MDT0000-mdc-ffff9e3647fa8800: Connection restored to 192.168.203.105@tcp (at 192.168.203.105@tcp)
[ 6066.259917] Lustre: Skipped 18 previous similar messages
[ 6074.148312] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 20:04:00 (1786406640)
[ 6076.223622] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6078.770332] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 20:04:03 (1786406643)
[ 6145.055216] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6147.174577] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 20:05:13 (1786406713)
[ 6206.854702] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 20:06:12 (1786406772)
[ 6209.122309] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6211.075536] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 20:06:17 (1786406777)
[ 6212.757416] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6214.415671] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 20:06:20 (1786406780)
[ 6253.618276] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 20:06:59 (1786406819)
[ 6266.960473] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6295.598151] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 192.168.203.105@tcp) was lost; in progress operations using this service will fail
[ 6295.602549] LustreError: Skipped 2 previous similar messages
[ 6295.616914] Lustre: Evicted from MGS (at 192.168.203.105@tcp) after server handle changed from 0x496bfb278a4faf69 to 0x496bfb278a4fc7b1
[ 6295.629287] Lustre: Skipped 2 previous similar messages
[ 6296.955527] Lustre: 67956:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.105@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6324.354353] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6326.695645] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6336.968472] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 20:08:23 (1786406903)
[ 6339.389306] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6341.235495] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 20:08:27 (1786406907)
[ 6356.983927] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 192.168.203.105@tcp) was lost; in progress operations using this service will fail
[ 6357.014742] Lustre: Evicted from MGS (at 192.168.203.105@tcp) after server handle changed from 0x496bfb278a4fc7b1 to 0x496bfb278a4fcc6c
[ 6375.256864] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 20:09:00 (1786406940)
[ 6430.685653] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 20:09:56 (1786406996)
[ 6438.691401] LustreError: lustre-OST0000-osc-ffff9e3647fa8800: operation ost_setattr to node 192.168.203.105@tcp failed: rc = -107
[ 6438.698625] LustreError: Skipped 2 previous similar messages
[ 6517.822287] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6525.195096] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6600.917993] Lustre: lustre-MDT0000-mdc-ffff9e3647fa8800: Connection to lustre-MDT0000 (at 192.168.203.105@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6600.981373] Lustre: Skipped 7 previous similar messages
[ 6621.225800] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 192.168.203.105@tcp) was lost; in progress operations using this service will fail
[ 6621.236800] LustreError: Skipped 1 previous similar message
[ 6621.272523] Lustre: Evicted from MGS (at 192.168.203.105@tcp) after server handle changed from 0x496bfb278a4fcf9f to 0x496bfb278a4fdcb8
[ 6621.280256] Lustre: Skipped 1 previous similar message
[ 6622.893267] Lustre: 67956:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.105@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6647.830664] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6649.670697] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6671.839319] Lustre: 2354:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786407223/real 1786407223]  req@ffff9e3670439880 x1873176903474688/t0(0) o400->MGC192.168.203.105@tcp@192.168.203.105@tcp:26/25 lens 224/224 e 0 to 1 dl 1786407239 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6671.866070] Lustre: 2354:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
[ 6682.140259] Lustre: MGC192.168.203.105@tcp: Connection restored to 192.168.203.105@tcp (at 192.168.203.105@tcp)
[ 6682.144607] Lustre: Skipped 10 previous similar messages
[ 6696.871915] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6698.964934] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6852.678832] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 20:16:58 (1786407418)
[ 6942.498913] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6950.193809] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7217.722806] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 20:23:03 (1786407783)
[ 7377.430040] Lustre: lustre-MDT0000-mdc-ffff9e3647fa8800: Connection to lustre-MDT0000 (at 192.168.203.105@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7377.449120] Lustre: Skipped 2 previous similar messages
[ 7387.622495] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 192.168.203.105@tcp) was lost; in progress operations using this service will fail
[ 7387.633541] LustreError: Skipped 1 previous similar message
[ 7387.651981] Lustre: Evicted from MGS (at 192.168.203.105@tcp) after server handle changed from 0x496bfb278a4fe0cb to 0x496bfb278a5f844d
[ 7387.664443] Lustre: Skipped 1 previous similar message
[ 7387.672777] Lustre: MGC192.168.203.105@tcp: Connection restored to 192.168.203.105@tcp (at 192.168.203.105@tcp)
[ 7387.684744] Lustre: Skipped 1 previous similar message
[ 7431.377335] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 20:26:37 (1786407997)
[ 7433.607769] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7435.494980] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 20:26:41 (1786408001)
[ 7438.025320] LustreError: lustre-MDT0000-mdc-ffff9e3647fa8800: operation ldlm_enqueue to node 192.168.203.105@tcp failed: rc = -107
[ 7438.029711] LustreError: Skipped 10 previous similar messages
[ 7438.045521] LustreError: lustre-MDT0000-mdc-ffff9e3647fa8800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7446.987415] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 20:26:52 (1786408012)
[ 7613.763265] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.203.105@tcp) after server handle changed from 0x496bfb278a5f8710 to 0x496bfb278a5f881a
[ 7613.832111] LustreError: lustre-OST0000-osc-ffff9e3647fa8800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7615.907585] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 20:29:41 (1786408181)
[ 7641.567140] Lustre: 2353:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786408188/real 1786408188]  req@ffff9e367178a680 x1873176914654336/t0(0) o4->lustre-OST0000-osc-ffff9e3647fa8800@192.168.203.105@tcp:6/4 lens 4584/448 e 0 to 1 dl 1786408208 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7641.601497] Lustre: 2353:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 7659.935614] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 20:30:25 (1786408225)
[ 7661.593670] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7663.547391] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 20:30:29 (1786408229)
[ 7665.724614] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7667.832446] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 20:30:33 (1786408233)
[ 7705.179535] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 20:31:11 (1786408271)
[ 7747.070575] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 192.168.203.105@tcp) was lost; in progress operations using this service will fail
[ 7747.084468] Lustre: Evicted from MGS (at 192.168.203.105@tcp) after server handle changed from 0x496bfb278a5f844d to 0x496bfb278a5faa9c
[ 7764.915974] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 20:32:11 (1786408331)
[ 7766.658568] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7769.881741] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 20:32:14 (1786408334)
[ 7771.622074] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7773.353408] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 20:32:19 (1786408339)
[ 7784.749083] Lustre: Unmounted lustre-client
[ 7786.900835] Lustre: Mounted lustre-client
[ 7790.115730] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7808.479302] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 192.168.203.105@tcp) was lost; in progress operations using this service will fail
[ 7817.767545] Lustre: Evicted from MGS (at 192.168.203.105@tcp) after server handle changed from 0x496bfb278a5fb53f to 0x496bfb278a5fb8db
[ 7820.686724] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2