[ 1820.323428] LustreError: lustre-OST0000-osc-ffff957de029e800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1820.346901] Lustre: 2365:0:(llite_lib.c:4238:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.120@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1827.010704] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 15:12:18 (1761246738)
[ 1829.018636] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1830.336844] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 15:12:22 (1761246742)
[ 1831.881438] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1833.279735] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 15:12:24 (1761246744)
[ 1836.784918] Lustre: lustre-MDT0000-mdc-ffff957de029e800: Connection to lustre-MDT0000 (at 192.168.204.120@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 1836.795069] Lustre: Skipped 14 previous similar messages
[ 1853.924992] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 1853.934979] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x22b2fe1779f6e31a to 0x22b2fe1779f6fbb6
[ 1853.940471] Lustre: MGC192.168.204.120@tcp: Connection restored to 192.168.204.120@tcp (at 192.168.204.120@tcp)
[ 1853.943396] Lustre: Skipped 14 previous similar messages
[ 1953.161401] LustreError: lustre-MDT0000-mdc-ffff957de029e800: operation mds_reint to node 192.168.204.120@tcp failed: rc = -19
[ 1953.170645] LustreError: Skipped 2 previous similar messages
[ 1971.694989] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 1971.734241] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x22b2fe1779f6fbb6 to 0x22b2fe1779f9d481
[ 1984.154779] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 15:14:55 (1761246895)
[ 1984.714571] Lustre: *** cfs_fail_loc=305, val=0***
[ 1984.716595] Lustre: Skipped 4 previous similar messages
[ 2022.886674] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 2022.902897] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x22b2fe1779f9d481 to 0x22b2fe1779f9f5b3
[ 2032.225139] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2033.333828] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2040.221651] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 15:15:51 (1761246951)
[ 2053.615650] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 2053.632489] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x22b2fe1779f9f5b3 to 0x22b2fe1779f9faa6
[ 2053.659437] LustreError: lustre-MDT0000-mdc-ffff957de029e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2065.629044] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 15:16:17 (1761246977)
[ 2074.844483] LustreError: lustre-OST0000-osc-ffff957de029e800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2088.559360] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 15:16:40 (1761247000)
[ 2101.590756] LustreError: lustre-MDT0000-mdc-ffff957de029e800: operation mds_close to node 192.168.204.120@tcp failed: rc = -19
[ 2101.595434] LustreError: Skipped 1 previous similar message
[ 2120.175427] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 2120.194921] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x22b2fe1779f9faa6 to 0x22b2fe1779fa53b5
[ 2127.032260] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2128.542904] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2211.302943] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 2211.317556] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x22b2fe1779fa53b5 to 0x22b2fe1779fc5f03
[ 2216.403610] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2218.012290] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2282.620735] LustreError: lustre-MDT0000-mdc-ffff957de029e800: operation mds_close to node 192.168.204.120@tcp failed: rc = -19
[ 2282.628029] LustreError: Skipped 2 previous similar messages
[ 2298.344114] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 2298.371577] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x22b2fe1779fc5f03 to 0x22b2fe1779fe6f75
[ 2308.812483] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2310.822346] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2337.439281] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 15:20:49 (1761247249)
[ 2365.955263] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2384.851800] Lustre: 2363:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761247281/real 1761247281]  req@ffff957dc3528700 x1846799238609664/t0(0) o400->MGC192.168.204.120@tcp@192.168.204.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1761247297 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2384.864670] Lustre: 2363:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
[ 2384.871113] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 2384.883717] LustreError: Skipped 1 previous similar message
[ 2384.899338] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x22b2fe1779ff5032 to 0x22b2fe1779ff8ede
[ 2384.919844] Lustre: Skipped 1 previous similar message
[ 2395.242733] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2427.963627] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2441.234476] Lustre: lustre-MDT0000-mdc-ffff957de029e800: Connection to lustre-MDT0000 (at 192.168.204.120@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2441.250192] Lustre: Skipped 10 previous similar messages
[ 2460.686822] Lustre: MGC192.168.204.120@tcp: Connection restored to 192.168.204.120@tcp (at 192.168.204.120@tcp)
[ 2460.700271] Lustre: Skipped 20 previous similar messages
[ 2465.606980] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2511.196064] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2557.944098] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 2557.951580] LustreError: Skipped 3 previous similar messages
[ 2557.962639] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x22b2fe177a0127e0 to 0x22b2fe177a022767
[ 2557.973038] Lustre: Skipped 3 previous similar messages
[ 2563.166724] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2595.285895] LustreError: lustre-MDT0000-mdc-ffff957de029e800: operation mds_reint to node 192.168.204.120@tcp failed: rc = -19
[ 2595.292320] LustreError: Skipped 8 previous similar messages
[ 2645.164157] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 15:25:56 (1761247556)
[ 2697.281297] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2699.603949] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3029.849213] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3032.229282] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3321.106832] LustreError: lustre-OST0000-osc-ffff957de029e800: operation ldlm_enqueue to node 192.168.204.120@tcp failed: rc = -107
[ 3321.115603] LustreError: Skipped 2 previous similar messages
[ 3321.120386] Lustre: lustre-OST0000-osc-ffff957de029e800: Connection to lustre-OST0000 (at 192.168.204.120@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3321.131889] Lustre: Skipped 5 previous similar messages
[ 3352.148967] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3353.761341] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3615.906295] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 15:42:07 (1761248527)
[ 3633.632263] Lustre: 50268:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761248530/real 1761248530]  req@ffff957de16e4a80 x1846799262151168/t0(0) o101->lustre-MDT0000-mdc-ffff957de029e800@192.168.204.120@tcp:12/10 lens 576/1152 e 0 to 1 dl 1761248546 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3633.676382] Lustre: lustre-MDT0000-mdc-ffff957de029e800: Connection restored to 192.168.204.120@tcp (at 192.168.204.120@tcp)
[ 3633.682289] Lustre: Skipped 9 previous similar messages
[ 3639.426975] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 15:42:31 (1761248551)
[ 3663.215680] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 15:42:54 (1761248574)
[ 3687.434579] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 15:43:18 (1761248598)
[ 3687.812526] Lustre: Mounted lustre-client
[ 3718.656629] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 3718.668156] LustreError: Skipped 1 previous similar message
[ 3718.690112] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x22b2fe177a03492a to 0x22b2fe177a1b0b12
[ 3718.709424] Lustre: Skipped 1 previous similar message
[ 3719.713071] Lustre: 2365:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761248616/real 1761248616]  req@ffff957de16e4700 x1846799262167808/t0(0) o400->lustre-MDT0000-mdc-ffff957de029e800@192.168.204.120@tcp:12/10 lens 224/224 e 0 to 1 dl 1761248632 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3719.730072] Lustre: 2365:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3727.705683] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3729.107488] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3730.941414] LustreError: 53029:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff957dd1989000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 3730.948548] LustreError: 53029:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 3730.960481] LustreError: 53029:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3730.963033] LustreError: 53029:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3731.011578] Lustre: Unmounted lustre-client
[ 3736.165575] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 15:44:07 (1761248647)
[ 3871.136219] Lustre: 2362:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761248767/real 1761248767]  req@ffff957dd02b7b80 x1846799262199424/t0(0) o4->lustre-OST0000-osc-ffff957de029e800@192.168.204.120@tcp:6/4 lens 488/448 e 0 to 1 dl 1761248783 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 3871.161048] Lustre: 2362:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 64 previous similar messages
[ 3934.694616] Lustre: lustre-OST0000-osc-ffff957de029e800: Connection to lustre-OST0000 (at 192.168.204.120@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3934.704629] Lustre: Skipped 16 previous similar messages
[ 4026.403268] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 15:48:58 (1761248938)
[ 4073.096381] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 15:49:44 (1761248984)
[ 4075.931295] LustreError: 55183:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff957de029e800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4075.942588] LustreError: 55183:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 4075.969416] LustreError: 55183:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 4075.971712] LustreError: 55183:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4076.015235] Lustre: Unmounted lustre-client
[ 4098.702848] Lustre: Mounted lustre-client
[ 4104.283694] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 15:50:15 (1761249015)
[ 4104.725464] LustreError: 56165:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4105.790155] LustreError: 56165:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4105.932728] Lustre: *** cfs_fail_loc=305, val=0***
[ 4127.436293] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 15:50:39 (1761249039)
[ 4127.890671] Lustre: Mounted lustre-client
[ 4128.976337] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4139.247261] LustreError: 56845:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4139.250126] LustreError: 56845:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 4139.295231] Lustre: Unmounted lustre-client
[ 4144.738956] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 15:50:56 (1761249056)
[ 4225.854956] LustreError: lustre-MDT0000-mdc-ffff957dd1537000: operation mds_reint to node 192.168.204.120@tcp failed: rc = -19
[ 4245.986216] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 4246.005771] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x22b2fe177a1b2281 to 0x22b2fe177a1ddad7
[ 4246.020216] Lustre: MGC192.168.204.120@tcp: Connection restored to 192.168.204.120@tcp (at 192.168.204.120@tcp)
[ 4246.024960] Lustre: Skipped 23 previous similar messages
[ 4368.207962] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 15:54:40 (1761249280)
[ 4371.757824] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4384.245509] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 4384.263680] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x22b2fe177a1ddad7 to 0x22b2fe177a22a003
[ 4384.303091] LustreError: lustre-MDT0000-mdc-ffff957dd1537000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4395.670172] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 15:55:07 (1761249307)
[ 4396.251519] Lustre: Mounted lustre-client
[ 4403.532074] LustreError: lustre-OST0000-osc-ffff957dd1537000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4403.543287] LustreError: 59869:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-OST0000-osc-ffff957dd1537000: namespace resource [0x240000400:0x4943:0x0].0x0 (ffff957dc5ad0c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4413.536572] Lustre: 2364:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761249310/real 1761249310]  req@ffff957dc4405880 x1846799266840960/t0(0) o103->lustre-OST0000-osc-ffff957dc8c6c800@192.168.204.120@tcp:17/18 lens 328/224 e 0 to 1 dl 1761249326 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4413.558290] Lustre: 2364:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 82 previous similar messages
[ 4414.657604] LustreError: 59898:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff957dc8c6c800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4414.666368] LustreError: 59898:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 4414.687361] LustreError: 59898:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4414.690589] LustreError: 59898:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4414.741805] Lustre: Unmounted lustre-client
[ 4420.256357] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 15:55:31 (1761249331)
[ 4426.123866] LustreError: lustre-MDT0000-mdc-ffff957dd1537000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4426.140759] LustreError: 60540:0:(file.c:6108:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4431.262883] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 15:55:43 (1761249343)
[ 4431.455655] LustreError: 61243:0:(recover.c:330:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4436.188269] LustreError: lustre-MDT0000-mdc-ffff957dd1537000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4436.198530] LustreError: 61260:0:(import.c:291:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4436.206048] LustreError: 61243:0:(recover.c:330:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=254
[ 4436.211246] LustreError: 61243:0:(import.c:716:ptlrpc_connect_import_locked()) already connecting
[ 4436.232576] LustreError: 61265:0:(file.c:6108:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4437.295505] LustreError: 61271:0:(file.c:6108:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4437.312413] LustreError: 61271:0:(file.c:6108:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4439.505847] LustreError: 61294:0:(file.c:6108:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4439.514540] LustreError: 61294:0:(file.c:6108:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4444.060891] LustreError: 61338:0:(file.c:6108:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4444.090363] LustreError: 61338:0:(file.c:6108:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4452.550471] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 15:56:04 (1761249364)
[ 4485.241506] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4486.516975] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4495.930269] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 15:56:47 (1761249407)
[ 4528.132699] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4529.563572] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4537.836866] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 15:57:29 (1761249449)
[ 4543.989911] Lustre: lustre-OST0000-osc-ffff957dd1537000: Connection to lustre-OST0000 (at 192.168.204.120@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4544.005616] Lustre: Skipped 17 previous similar messages
[ 4595.140247] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4596.974554] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4605.194434] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 15:58:36 (1761249516)
[ 4636.844733] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4637.913433] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4642.400635] LustreError: 66920:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff957dd1537000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4642.413617] LustreError: 66920:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 4642.433937] LustreError: 66920:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 4642.438888] LustreError: 66920:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4642.497492] Lustre: Unmounted lustre-client
[ 4682.992726] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4685.291185] Lustre: Mounted lustre-client
[ 4690.965923] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 16:00:02 (1761249602)
[ 4693.517643] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4695.176535] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 16:00:06 (1761249606)
[ 4724.216583] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 16:00:35 (1761249635)
[ 4725.685755] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4727.230306] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 16:00:38 (1761249638)
[ 4728.734301] Lustre: *** cfs_fail_loc=805, val=0***
[ 4728.776729] Lustre: Mounted lustre-client
[ 4733.849374] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4754.208330] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 4754.238471] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x22b2fe177a22a9f7 to 0x22b2fe177a22ac12
[ 4770.415913] LustreError: 70279:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff957dc8f76800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4770.423244] LustreError: 70279:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 4770.447918] LustreError: 70279:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4770.451220] LustreError: 70279:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 4770.475956] Lustre: Unmounted lustre-client
[ 4777.552440] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 16:01:29 (1761249689)
[ 4805.086864] LustreError: 2361:0:(client.c:3408:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff957dc4404700 x1846799266920192/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff957dc36ed800@192.168.204.120@tcp:12/10 lens 664/608 e 0 to 0 dl 1761249733 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 4811.295599] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4812.792731] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4821.462523] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 16:02:13 (1761249733)
[ 4823.025284] LustreError: lustre-OST0000-osc-ffff957dc36ed800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4823.043069] Lustre: 2365:0:(llite_lib.c:4238:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.120@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4830.913883] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 16:02:22 (1761249742)
[ 4832.779658] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 4834.413368] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 16:02:25 (1761249745)
[ 4835.720205] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 4837.308931] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 16:02:29 (1761249749)
[ 4838.644906] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 4840.092229] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 16:02:31 (1761249751)
[ 4841.262733] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 4842.762562] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 16:02:34 (1761249754)
[ 4844.194918] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 4845.741803] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 16:02:37 (1761249757)
[ 4847.172427] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 4849.118465] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 16:02:40 (1761249760)
[ 4850.519579] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 4852.146927] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 16:02:43 (1761249763)
[ 4853.711235] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 4855.182156] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 16:02:46 (1761249766)
[ 4856.747208] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 4858.255763] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 16:02:49 (1761249769)
[ 4860.036695] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 4861.788938] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 16:02:53 (1761249773)
[ 4863.139327] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 4864.853562] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 16:02:56 (1761249776)
[ 4867.433449] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 4869.044290] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 16:03:00 (1761249780)
[ 4887.023286] Lustre: MGC192.168.204.120@tcp: Connection restored to 192.168.204.120@tcp (at 192.168.204.120@tcp)
[ 4887.027044] Lustre: Skipped 17 previous similar messages
[ 4897.613864] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 16:03:28 (1761249808)
[ 4927.201122] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 16:03:58 (1761249838)
[ 4927.784784] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4937.510230] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 16:04:09 (1761249849)
[ 4938.112466] Lustre: *** cfs_fail_loc=51b, val=4***
[ 4948.212960] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 16:04:19 (1761249859)
[ 4948.486519] Lustre: *** cfs_fail_loc=50f, val=3***
[ 4956.341231] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 16:04:27 (1761249867)
[ 4956.559439] Lustre: *** cfs_fail_loc=50f, val=4***
[ 4964.717812] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 16:04:36 (1761249876)
[ 4965.058819] Lustre: *** cfs_fail_loc=510, val=3***
[ 4973.653331] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 16:04:45 (1761249885)
[ 4973.921127] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4982.903488] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 16:04:54 (1761249894)
[ 4983.087219] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5042.656170] Lustre: 2362:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761249899/real 1761249899]  req@ffff957dc4763100 x1846799266963968/t0(0) o3->lustre-OST0001-osc-ffff957dc36ed800@192.168.204.120@tcp:6/4 lens 488/4536 e 0 to 1 dl 1761249955 ref 2 fl Rpc:RXQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 5042.673577] Lustre: 2362:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 5047.413920] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 16:05:59 (1761249959)
[ 5047.786188] LustreError: 82463:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5050.272897] LustreError: lustre-MDT0000-mdc-ffff957dc36ed800: operation ldlm_enqueue to node 192.168.204.120@tcp failed: rc = -107
[ 5050.280896] LustreError: Skipped 4 previous similar messages
[ 5050.289352] LustreError: lustre-MDT0000-mdc-ffff957dc36ed800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5050.305145] LustreError: 82477:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff957dc36ed800: inode [0x20000a042:0xe:0x0] mdc close failed: rc = -108
[ 5050.312667] LustreError: 82477:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff957dc36ed800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff957dc4489400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5050.322721] LustreError: 82477:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5051.864378] LustreError: 82463:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5053.942360] LustreError: 82493:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5056.519568] LustreError: lustre-MDT0000-mdc-ffff957dc36ed800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5056.535683] LustreError: 82507:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff957dc36ed800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff957dc4489f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5056.544507] LustreError: 82507:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5058.016129] LustreError: 82493:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5064.673773] LustreError: lustre-MDT0000-mdc-ffff957dc36ed800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5064.684103] LustreError: 82535:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff957dc36ed800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff957dc5a75700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5064.700781] LustreError: 82535:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5065.026926] LustreError: 82549:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5067.661327] LustreError: lustre-MDT0000-mdc-ffff957dc36ed800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5067.719426] LustreError: 82568:0:(file.c:6108:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5067.727683] LustreError: 82568:0:(file.c:6108:ll_inode_revalidate_fini()) Skipped 20 previous similar messages
[ 5068.800692] LustreError: 82575:0:(file.c:6108:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5068.805906] LustreError: 82575:0:(file.c:6108:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5069.098675] LustreError: 82549:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5069.107672] LustreError: 82549:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff957dc36ed800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5069.112897] LustreError: 82549:0:(file.c:249:ll_close_inode_openhandle()) Skipped 1 previous similar message
[ 5071.124989] LustreError: 82597:0:(file.c:6108:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5071.130807] LustreError: 82597:0:(file.c:6108:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5074.769654] LustreError: 82623:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5077.494065] LustreError: lustre-MDT0000-mdc-ffff957dc36ed800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5077.515249] LustreError: 82638:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff957dc36ed800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff957dc58ce300) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5078.840176] LustreError: 82623:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5085.825390] LustreError: lustre-MDT0000-mdc-ffff957dc36ed800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5086.214870] LustreError: 82679:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5088.874738] LustreError: lustre-MDT0000-mdc-ffff957dc36ed800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5088.882639] LustreError: 82694:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff957dc36ed800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff957dc5a75c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5088.890788] LustreError: 82694:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5090.280355] LustreError: 82679:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5093.164510] LustreError: lustre-MDT0000-mdc-ffff957dc36ed800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5093.229543] LustreError: 82726:0:(file.c:6108:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5093.244060] LustreError: 82726:0:(file.c:6108:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5094.450063] LustreError: 82707:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff957dc36ed800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5102.156807] LustreError: 82778:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5102.162454] LustreError: 82778:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5105.275167] LustreError: lustre-MDT0000-mdc-ffff957dc36ed800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5105.325887] LustreError: 82800:0:(file.c:6108:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5105.332128] LustreError: 82800:0:(file.c:6108:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5106.248160] LustreError: 82778:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5106.251903] LustreError: 82778:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5106.256428] LustreError: 82778:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff957dc36ed800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5113.960848] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 16:07:05 (1761250025)
[ 5131.388055] LustreError: 70280:0:(ldlm_lockd.c:2915:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5131.393525] LustreError: 70280:0:(ldlm_lockd.c:2915:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5135.472179] LustreError: 70280:0:(ldlm_lockd.c:2915:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5135.475922] LustreError: 70280:0:(ldlm_lockd.c:2915:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5140.449888] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 16:07:32 (1761250052)
[ 5182.436111] Lustre: lustre-MDT0000-mdc-ffff957dc36ed800: Connection to lustre-MDT0000 (at 192.168.204.120@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5182.444601] Lustre: Skipped 18 previous similar messages
[ 5188.611963] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 16:08:20 (1761250100)
[ 5251.715131] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 16:09:23 (1761250163)
[ 5285.247170] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 16:09:56 (1761250196)
[ 5285.679289] Lustre: Mounted lustre-client
[ 5408.629038] LustreError: 86489:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff957dd1536800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5408.643178] LustreError: 86489:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 5408.648124] LustreError: 86489:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 5408.651712] LustreError: 86489:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 5408.689322] Lustre: Unmounted lustre-client
[ 5416.121228] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 16:12:07 (1761250327)
[ 5416.753158] LustreError: 87105:0:(osc_request.c:2978:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 4000ms
[ 5420.038152] LustreError: lustre-OST0000-osc-ffff957dc36ed800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5420.059662] LustreError: 87187:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-OST0000-osc-ffff957dc36ed800: namespace resource [0x240000400:0x49cd:0x0].0x0 (ffff957dc3749f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5420.078764] LustreError: 87187:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5420.762538] LustreError: 87105:0:(osc_request.c:2978:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 5420.781562] Lustre: 2363:0:(llite_lib.c:4238:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.204.120@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5426.782506] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 16:12:18 (1761250338)
[ 5490.974601] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 16:13:22 (1761250402)
[ 5492.422475] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5494.342340] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 16:13:25 (1761250405)
[ 5551.097395] Lustre: lustre-MDT0000-mdc-ffff957dc36ed800: Connection restored to 192.168.204.120@tcp (at 192.168.204.120@tcp)
[ 5551.119359] Lustre: Skipped 14 previous similar messages
[ 5558.280897] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5560.586558] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 16:14:31 (1761250471)
[ 5619.069763] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 16:15:29 (1761250529)
[ 5621.416608] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5623.220379] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 16:15:34 (1761250534)
[ 5625.055494] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5627.171398] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 16:15:38 (1761250538)
[ 5664.708784] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 16:16:16 (1761250576)
[ 5675.585565] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5698.528249] Lustre: 2365:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761250595/real 1761250595]  req@ffff957dc4405880 x1846799267123328/t0(0) o400->MGC192.168.204.120@tcp@192.168.204.120@tcp:26/25 lens 224/224 e 0 to 1 dl 1761250611 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5698.552270] Lustre: 2365:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 5698.558887] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 5698.565022] LustreError: Skipped 2 previous similar messages
[ 5708.721676] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x22b2fe177a22b828 to 0x22b2fe177a22d077
[ 5708.729536] Lustre: Skipped 2 previous similar messages
[ 5718.800400] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5720.275752] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5729.906911] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 16:17:21 (1761250641)
[ 5732.022833] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5734.188090] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 16:17:25 (1761250645)
[ 5745.647989] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 5745.693085] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x22b2fe177a22d077 to 0x22b2fe177a22d405
[ 5758.574686] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 16:17:50 (1761250670)
[ 5804.205792] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 16:18:35 (1761250715)
[ 5810.695342] LustreError: lustre-OST0000-osc-ffff957dc36ed800: operation ost_setattr to node 192.168.204.120@tcp failed: rc = -19
[ 5810.700034] Lustre: lustre-OST0000-osc-ffff957dc36ed800: Connection to lustre-OST0000 (at 192.168.204.120@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5810.700043] Lustre: Skipped 8 previous similar messages
[ 5810.736670] LustreError: Skipped 9 previous similar messages
[ 5854.999639] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5857.753543] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5944.352303] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 5944.359775] LustreError: Skipped 1 previous similar message
[ 5944.405170] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x22b2fe177a22d723 to 0x22b2fe177a22e55b
[ 5944.414102] Lustre: Skipped 1 previous similar message
[ 5968.251748] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5970.222588] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6004.088769] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6005.727201] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6113.992190] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 16:23:45 (1761251025)
[ 6174.133089] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6178.072778] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6413.939094] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 16:28:45 (1761251325)
[ 6562.807314] Lustre: lustre-MDT0000-mdc-ffff957dc36ed800: Connection to lustre-MDT0000 (at 192.168.204.120@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6562.836519] Lustre: Skipped 3 previous similar messages
[ 6573.041688] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 6573.049777] LustreError: Skipped 1 previous similar message
[ 6573.075299] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x22b2fe177a22ead3 to 0x22b2fe177a328cb8
[ 6573.090120] Lustre: Skipped 1 previous similar message
[ 6573.097661] Lustre: MGC192.168.204.120@tcp: Connection restored to 192.168.204.120@tcp (at 192.168.204.120@tcp)
[ 6573.104026] Lustre: Skipped 11 previous similar messages
[ 6613.804893] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 16:32:05 (1761251525)
[ 6615.356641] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6617.364977] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 16:32:08 (1761251528)
[ 6619.253912] LustreError: lustre-MDT0000-mdc-ffff957dc36ed800: operation ldlm_enqueue to node 192.168.204.120@tcp failed: rc = -107
[ 6619.259878] LustreError: Skipped 286 previous similar messages
[ 6619.274340] LustreError: lustre-MDT0000-mdc-ffff957dc36ed800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6627.295395] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 16:32:18 (1761251538)
[ 6794.218068] LustreError: lustre-OST0000-osc-ffff957dc36ed800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6796.072995] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 16:35:07 (1761251707)
[ 6820.832211] Lustre: 2364:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761251713/real 1761251713]  req@ffff957dc717f800 x1846799284821120/t0(0) o4->lustre-OST0000-osc-ffff957dc36ed800@192.168.204.120@tcp:6/4 lens 4584/448 e 0 to 1 dl 1761251733 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 6820.850911] Lustre: 2364:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 16 previous similar messages
[ 6837.898416] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 16:35:49 (1761251749)
[ 6839.158484] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 6840.996567] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 16:35:52 (1761251752)
[ 6842.366155] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 6844.306169] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 16:35:55 (1761251755)
[ 6876.624463] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 16:36:28 (1761251788)
[ 6913.016119] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 192.168.204.120@tcp) was lost; in progress operations using this service will fail
[ 6913.047648] Lustre: Evicted from MGS (at 192.168.204.120@tcp) after server handle changed from 0x22b2fe177a328cb8 to 0x22b2fe177a32b4a4
[ 6913.059977] Lustre: Skipped 1 previous similar message
[ 6925.432362] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 16:37:16 (1761251836)
[ 6926.837935] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 6928.578246] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 16:37:20 (1761251840)
[ 6929.837504] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 6931.609219] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 16:37:23 (1761251843)
[ 6940.468564] LustreError: 105992:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff957dc36ed800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 6940.473527] LustreError: 105992:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 6940.512223] LustreError: 105992:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 6940.515073] LustreError: 105992:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 6940.620733] Lustre: Unmounted lustre-client
[ 6942.102574] Lustre: Mounted lustre-client
[ 6945.063231] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6975.893669] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: total 1812