[ 1511.974638] LustreError: lustre-OST0000-osc-ffff999b460ef800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1513.136483] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ IGNORE (bz5494): multiop didn't fail fsync: 5 or close: 0
[ 1515.172689] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ FAIL: no discarded dirty page found!
[ 1524.011782] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 01:44:45 (1761284685)
[ 1525.378634] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1526.356865] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 01:44:48 (1761284688)
[ 1527.332583] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1528.229313] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 01:44:50 (1761284690)
[ 1530.614680] LustreError: lustre-MDT0000-mdc-ffff999b460ef800: operation mds_close to node 192.168.202.143@tcp failed: rc = -19
[ 1530.620267] LustreError: Skipped 1 previous similar message
[ 1547.754212] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 1547.800654] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x268f34dc746ceb4 to 0x268f34dc746ee88
[ 1646.832365] LustreError: lustre-MDT0000-mdc-ffff999b460ef800: operation ldlm_enqueue to node 192.168.202.143@tcp failed: rc = -19
[ 1646.848200] LustreError: Skipped 1 previous similar message
[ 1665.487400] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 1665.534186] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x268f34dc746ee88 to 0x268f34dc74919f0
[ 1691.538918] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 01:47:32 (1761284852)
[ 1692.094291] Lustre: *** cfs_fail_loc=305, val=0***
[ 1692.097072] Lustre: Skipped 4 previous similar messages
[ 1710.941614] LustreError: lustre-OST0000-osc-ffff999b460ef800: operation ost_connect to node 192.168.202.143@tcp failed: rc = -75
[ 1736.671276] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 1736.703797] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x268f34dc74919f0 to 0x268f34dc749203a
[ 1747.777097] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1749.701033] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1758.185813] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 01:48:39 (1761284919)
[ 1772.531694] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 1772.549552] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x268f34dc749203a to 0x268f34dc7492518
[ 1772.617119] LustreError: lustre-MDT0000-mdc-ffff999b460ef800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 1792.069877] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 01:49:13 (1761284953)
[ 1818.566178] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 01:49:39 (1761284979)
[ 1831.392418] LustreError: lustre-MDT0000-mdc-ffff999b460ef800: operation ldlm_enqueue to node 192.168.202.143@tcp failed: rc = -19
[ 1849.327357] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 1849.362629] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x268f34dc7492518 to 0x268f34dc7498589
[ 1863.521472] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1865.645727] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1946.591573] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 1956.778322] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x268f34dc7498589 to 0x268f34dc74bb1d8
[ 1966.120720] Lustre: lustre-MDT0000-mdc-ffff999b460ef800: Connection restored to 192.168.202.143@tcp (at 192.168.202.143@tcp)
[ 1966.134754] Lustre: Skipped 18 previous similar messages
[ 1970.921535] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1972.571699] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2037.481265] LustreError: lustre-MDT0000-mdc-ffff999b460ef800: operation mds_reint to node 192.168.202.143@tcp failed: rc = -19
[ 2037.487220] LustreError: Skipped 1 previous similar message
[ 2037.494727] Lustre: lustre-MDT0000-mdc-ffff999b460ef800: Connection to lustre-MDT0000 (at 192.168.202.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2037.507637] Lustre: Skipped 14 previous similar messages
[ 2058.222290] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 2058.265690] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x268f34dc74bb1d8 to 0x268f34dc74d938f
[ 2069.033251] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2071.592042] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2100.271621] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 01:54:21 (1761285261)
[ 2124.781917] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 2124.817138] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x268f34dc74d938f to 0x268f34dc74e9c5b
[ 2140.950938] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2160.548223] Lustre: 2368:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761285307/real 1761285307]  req@ffff999b47d6d880 x1846839324922496/t0(0) o400->MGC192.168.202.143@tcp@192.168.202.143@tcp:26/25 lens 224/224 e 0 to 1 dl 1761285323 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2160.587194] Lustre: 2368:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
[ 2169.793528] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2204.103583] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2245.785501] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2286.559731] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 2286.564241] LustreError: Skipped 3 previous similar messages
[ 2296.758776] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x268f34dc74ee7b0 to 0x268f34dc74f4278
[ 2296.782602] Lustre: Skipped 3 previous similar messages
[ 2305.466489] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2333.893359] LustreError: lustre-MDT0000-mdc-ffff999b460ef800: operation mds_reint to node 192.168.202.143@tcp failed: rc = -19
[ 2333.898694] LustreError: Skipped 7 previous similar messages
[ 2360.054638] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2446.166996] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 02:00:07 (1761285607)
[ 2496.989886] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2499.423964] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2791.119290] Lustre: lustre-OST0000-osc-ffff999b460ef800: Connection to lustre-OST0000 (at 192.168.202.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2791.127223] Lustre: Skipped 7 previous similar messages
[ 2827.452786] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2829.876386] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3122.250405] LustreError: lustre-OST0000-osc-ffff999b460ef800: operation ldlm_enqueue to node 192.168.202.143@tcp failed: rc = -19
[ 3122.260920] LustreError: Skipped 3 previous similar messages
[ 3140.127116] Lustre: lustre-OST0000-osc-ffff999b460ef800: Connection restored to 192.168.202.143@tcp (at 192.168.202.143@tcp)
[ 3140.134808] Lustre: Skipped 15 previous similar messages
[ 3149.783300] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3151.737426] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3417.437867] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 02:16:18 (1761286578)
[ 3434.464989] Lustre: 50573:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761286581/real 1761286581]  req@ffff999b5018b100 x1846839349926272/t0(0) o101->lustre-MDT0000-mdc-ffff999b460ef800@192.168.202.143@tcp:12/10 lens 576/1152 e 0 to 1 dl 1761286597 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3434.479193] Lustre: 50573:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 3434.486064] Lustre: lustre-MDT0000-mdc-ffff999b460ef800: Connection to lustre-MDT0000 (at 192.168.202.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3434.495807] Lustre: Skipped 1 previous similar message
[ 3440.288880] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 02:16:41 (1761286601)
[ 3464.208643] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 02:17:05 (1761286625)
[ 3487.369828] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 02:17:29 (1761286649)
[ 3487.722880] Lustre: Mounted lustre-client
[ 3518.432418] Lustre: 2366:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761286665/real 1761286665]  req@ffff999b4774c700 x1846839349943040/t0(0) o400->MGC192.168.202.143@tcp@192.168.202.143@tcp:26/25 lens 224/224 e 0 to 1 dl 1761286681 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3518.450218] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 3518.492643] Lustre: 2366:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3518.500442] LustreError: Skipped 2 previous similar messages
[ 3518.521785] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x268f34dc7509056 to 0x268f34dc76be1e2
[ 3518.526123] Lustre: Skipped 2 previous similar messages
[ 3527.041027] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3528.336112] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3530.186720] LustreError: 53326:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff999b467ce000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 3530.199534] LustreError: 53326:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 3530.221089] LustreError: 53326:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3530.224802] LustreError: 53326:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3530.270922] Lustre: Unmounted lustre-client
[ 3536.173532] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 02:18:17 (1761286697)
[ 3672.031161] Lustre: 2368:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761286818/real 1761286818]  req@ffff999b46730380 x1846839349974656/t0(0) o4->lustre-OST0000-osc-ffff999b460ef800@192.168.202.143@tcp:6/4 lens 488/448 e 0 to 1 dl 1761286834 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 3672.077259] Lustre: 2368:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 57 previous similar messages
[ 3751.917461] Lustre: lustre-OST0000-osc-ffff999b460ef800: Connection restored to 192.168.202.143@tcp (at 192.168.202.143@tcp)
[ 3751.928015] Lustre: Skipped 17 previous similar messages
[ 3827.213088] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 02:23:08 (1761286988)
[ 3875.692883] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 02:23:57 (1761287037)
[ 3878.174342] LustreError: 55481:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff999b460ef800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 3878.179406] LustreError: 55481:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 3878.191848] LustreError: 55481:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 3878.194660] LustreError: 55481:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 3878.233143] Lustre: Unmounted lustre-client
[ 3895.969914] Lustre: Mounted lustre-client
[ 3901.785055] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 02:24:23 (1761287063)
[ 3902.102078] LustreError: 56465:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 3903.160053] LustreError: 56465:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 3903.290700] Lustre: *** cfs_fail_loc=305, val=0***
[ 3923.271637] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 02:24:45 (1761287085)
[ 3923.502833] Lustre: Mounted lustre-client
[ 3924.389788] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 3934.638909] LustreError: 57144:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 3934.646626] LustreError: 57144:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 3934.667102] Lustre: Unmounted lustre-client
[ 3939.246743] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 02:25:00 (1761287100)
[ 4010.898631] LustreError: lustre-MDT0000-mdc-ffff999b434e5000: operation mds_reint to node 192.168.202.143@tcp failed: rc = -19
[ 4030.949323] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 4030.963372] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x268f34dc76bfea6 to 0x268f34dc76eb95d
[ 4156.842059] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 02:28:38 (1761287318)
[ 4161.896947] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4174.305567] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 4174.320339] Lustre: lustre-MDT0000-mdc-ffff999b434e5000: Connection to lustre-MDT0000 (at 192.168.202.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4174.322824] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x268f34dc76eb95d to 0x268f34dc7737a29
[ 4174.337834] Lustre: Skipped 24 previous similar messages
[ 4174.391716] LustreError: lustre-MDT0000-mdc-ffff999b434e5000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4184.877719] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 02:29:06 (1761287346)
[ 4185.261766] Lustre: Mounted lustre-client
[ 4192.314501] LustreError: lustre-OST0000-osc-ffff999b434e5000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4192.327995] LustreError: 60164:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-OST0000-osc-ffff999b434e5000: namespace resource [0x240000400:0x4ac3:0x0].0x0 (ffff999b6048c000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4201.953086] Lustre: 2365:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761287348/real 1761287348]  req@ffff999b48035880 x1846839354612480/t0(0) o103->lustre-OST0000-osc-ffff999b460ed800@192.168.202.143@tcp:17/18 lens 328/224 e 0 to 1 dl 1761287364 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4201.985343] Lustre: 2365:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 74 previous similar messages
[ 4203.108381] LustreError: 60193:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff999b460ed800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4203.116116] LustreError: 60193:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 4203.128233] LustreError: 60193:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4203.132502] LustreError: 60193:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4203.172357] Lustre: Unmounted lustre-client
[ 4208.921381] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 02:29:30 (1761287370)
[ 4215.093596] LustreError: lustre-MDT0000-mdc-ffff999b434e5000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4215.104833] LustreError: 60920:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff999b434e5000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff999b480f6d00) refcount nonzero (2) after lock cleanup; forcing cleanup.
[ 4215.121857] LustreError: 60836:0:(file.c:6108:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4220.612509] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 02:29:42 (1761287382)
[ 4220.803871] LustreError: 61539:0:(recover.c:330:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4224.574540] LustreError: lustre-MDT0000-mdc-ffff999b434e5000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4224.590927] LustreError: 61559:0:(import.c:291:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4224.595450] LustreError: 61539:0:(recover.c:330:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=1172
[ 4224.601571] LustreError: 61539:0:(import.c:716:ptlrpc_connect_import_locked()) already connecting
[ 4224.626512] LustreError: 61564:0:(file.c:6108:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4225.696815] LustreError: 61570:0:(file.c:6108:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4225.707500] LustreError: 61570:0:(file.c:6108:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4227.940570] LustreError: 61593:0:(file.c:6108:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4227.955745] LustreError: 61593:0:(file.c:6108:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4232.501155] LustreError: 61638:0:(file.c:6108:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4232.513637] LustreError: 61638:0:(file.c:6108:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4241.246883] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 02:30:02 (1761287402)
[ 4272.164806] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4273.793378] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4282.665678] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 02:30:44 (1761287444)
[ 4314.501270] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4316.098151] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4324.959495] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 02:31:26 (1761287486)
[ 4373.047797] Lustre: lustre-OST0000-osc-ffff999b434e5000: Connection restored to 192.168.202.143@tcp (at 192.168.202.143@tcp)
[ 4373.058678] Lustre: Skipped 16 previous similar messages
[ 4381.861743] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4383.740860] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4391.825544] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 02:32:33 (1761287553)
[ 4422.383386] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4423.795933] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4428.204346] LustreError: 67206:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff999b434e5000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4428.221735] LustreError: 67206:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 4428.246317] LustreError: 67206:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 4428.248896] LustreError: 67206:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 4428.299220] Lustre: Unmounted lustre-client
[ 4470.308880] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4472.459413] Lustre: Mounted lustre-client
[ 4477.705286] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 02:33:59 (1761287639)
[ 4479.627699] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4480.949822] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 02:34:02 (1761287642)
[ 4503.355355] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 02:34:24 (1761287664)
[ 4504.888772] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 4506.120850] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 02:34:27 (1761287667)
[ 4507.263671] Lustre: *** cfs_fail_loc=805, val=0***
[ 4507.306535] Lustre: Mounted lustre-client
[ 4511.475227] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4533.224913] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 4533.267464] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x268f34dc7738424 to 0x268f34dc77387b2
[ 4537.505272] LustreError: 70549:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff999b483ab800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 4537.510918] LustreError: 70549:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 4537.533243] LustreError: 70549:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 4537.535237] LustreError: 70549:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages
[ 4537.552441] Lustre: Unmounted lustre-client
[ 4544.891385] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 02:35:06 (1761287706)
[ 4571.638699] LustreError: 2364:0:(client.c:3408:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff999b45c22680 x1846839354689280/t90194313221(90194313221) o101->lustre-MDT0000-mdc-ffff999b450de000@192.168.202.143@tcp:12/10 lens 648/608 e 0 to 0 dl 1761287750 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 4576.526888] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4577.890438] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4585.129638] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 02:35:46 (1761287746)
[ 4586.381315] LustreError: lustre-OST0000-osc-ffff999b450de000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4586.388202] Lustre: 2366:0:(llite_lib.c:4238:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.143@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 4593.654310] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 02:35:55 (1761287755)
[ 4594.902661] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 4596.351899] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 02:35:57 (1761287757)
[ 4597.773323] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 4599.444473] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 02:36:00 (1761287760)
[ 4601.095207] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 4602.861055] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 02:36:04 (1761287764)
[ 4604.287952] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 4605.772802] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 02:36:07 (1761287767)
[ 4607.142924] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 4608.940109] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 02:36:10 (1761287770)
[ 4610.324892] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 4611.941277] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 02:36:13 (1761287773)
[ 4613.398433] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 4614.800390] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 02:36:16 (1761287776)
[ 4616.016037] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 4617.495341] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 02:36:18 (1761287778)
[ 4618.876911] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 4620.550953] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 02:36:21 (1761287781)
[ 4621.753441] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 4623.511089] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 02:36:24 (1761287784)
[ 4624.839479] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 4626.674144] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 02:36:27 (1761287787)
[ 4628.947604] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 4630.562427] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 02:36:31 (1761287791)
[ 4659.907586] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 02:37:01 (1761287821)
[ 4689.778977] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 02:37:30 (1761287850)
[ 4690.463601] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4703.180413] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 02:37:43 (1761287863)
[ 4703.727077] Lustre: *** cfs_fail_loc=51b, val=4***
[ 4715.931561] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 02:37:56 (1761287876)
[ 4716.529889] Lustre: *** cfs_fail_loc=50f, val=3***
[ 4724.876711] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 02:38:06 (1761287886)
[ 4725.199670] Lustre: *** cfs_fail_loc=50f, val=4***
[ 4733.399391] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 02:38:14 (1761287894)
[ 4733.865112] Lustre: *** cfs_fail_loc=510, val=3***
[ 4742.647572] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 02:38:23 (1761287903)
[ 4743.120671] Lustre: *** cfs_fail_loc=51b, val=3***
[ 4753.735839] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 02:38:34 (1761287914)
[ 4754.021854] Lustre: *** cfs_fail_loc=51c, val=3***
[ 4813.279153] Lustre: 2365:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761287920/real 1761287920]  req@ffff999b45c20700 x1846839354733952/t0(0) o3->lustre-OST0001-osc-ffff999b450de000@192.168.202.143@tcp:6/4 lens 488/4536 e 0 to 1 dl 1761287975 ref 2 fl Rpc:RXQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4813.304986] Lustre: 2365:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 4813.313453] Lustre: lustre-OST0001-osc-ffff999b450de000: Connection to lustre-OST0001 (at 192.168.202.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4813.325796] Lustre: Skipped 16 previous similar messages
[ 4819.249693] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 02:39:40 (1761287980)
[ 4819.800702] LustreError: 82744:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4822.667927] LustreError: lustre-MDT0000-mdc-ffff999b450de000: operation ldlm_enqueue to node 192.168.202.143@tcp failed: rc = -107
[ 4822.676024] LustreError: Skipped 4 previous similar messages
[ 4822.685849] LustreError: lustre-MDT0000-mdc-ffff999b450de000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4822.721278] LustreError: 82759:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff999b450de000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff999b48186b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4823.896277] LustreError: 82744:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4826.024966] LustreError: 82775:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4828.780960] LustreError: lustre-MDT0000-mdc-ffff999b450de000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4828.802348] LustreError: 82789:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff999b450de000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff999b6048c200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4828.815147] LustreError: 82789:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4830.119512] LustreError: 82775:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4836.921725] LustreError: lustre-MDT0000-mdc-ffff999b450de000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4837.316099] LustreError: 82830:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4840.193882] LustreError: lustre-MDT0000-mdc-ffff999b450de000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4840.241635] LustreError: 82850:0:(file.c:6108:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4840.248797] LustreError: 82850:0:(file.c:6108:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4841.383281] LustreError: 82830:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4841.404616] LustreError: 82830:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff999b450de000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4847.213688] LustreError: 82905:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 4850.057123] LustreError: lustre-MDT0000-mdc-ffff999b450de000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4850.087138] LustreError: 82919:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff999b450de000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff999b6048c800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4850.126764] LustreError: 82919:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4851.303173] LustreError: 82905:0:(ldlm_flock.c:807:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 4858.207664] LustreError: lustre-MDT0000-mdc-ffff999b450de000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4858.712921] LustreError: 82960:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 4861.703876] LustreError: lustre-MDT0000-mdc-ffff999b450de000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4861.729747] LustreError: 82975:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff999b450de000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff999b48186400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4861.763337] LustreError: 82975:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 4862.791267] LustreError: 82960:0:(ldlm_flock.c:856:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 4865.876621] LustreError: lustre-MDT0000-mdc-ffff999b450de000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4865.936937] LustreError: 83007:0:(file.c:6108:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4865.943448] LustreError: 83007:0:(file.c:6108:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 4867.055792] LustreError: 82988:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff999b450de000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4875.068880] LustreError: 83059:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 4875.073613] LustreError: 83059:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4878.659905] LustreError: lustre-MDT0000-mdc-ffff999b450de000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4879.135151] LustreError: 83059:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 4879.151292] LustreError: 83059:0:(ldlm_flock.c:802:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 4879.162621] LustreError: 83059:0:(file.c:249:ll_close_inode_openhandle()) lustre-clilmv-ffff999b450de000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 4889.444000] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 02:40:50 (1761288050)
[ 4908.457557] LustreError: 68039:0:(ldlm_lockd.c:2915:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 4908.473143] LustreError: 68039:0:(ldlm_lockd.c:2915:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 4912.535176] LustreError: 68039:0:(ldlm_lockd.c:2915:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 4912.549786] LustreError: 68039:0:(ldlm_lockd.c:2915:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 4918.178602] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 02:41:19 (1761288079)
[ 4966.730889] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 02:42:08 (1761288128)
[ 5022.705653] Lustre: lustre-MDT0000-mdc-ffff999b450de000: Connection restored to 192.168.202.143@tcp (at 192.168.202.143@tcp)
[ 5022.709011] Lustre: Skipped 20 previous similar messages
[ 5028.566580] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 02:43:10 (1761288190)
[ 5057.408075] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 02:43:39 (1761288219)
[ 5057.638103] Lustre: Mounted lustre-client
[ 5179.162401] LustreError: 86769:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff999b509d3000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 5179.167071] LustreError: 86769:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 5179.171628] LustreError: 86769:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup
[ 5179.175299] LustreError: 86769:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 5179.204050] Lustre: Unmounted lustre-client
[ 5182.334504] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 02:45:44 (1761288344)
[ 5182.592605] LustreError: 87385:0:(osc_request.c:2978:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 4000ms
[ 5184.605817] LustreError: lustre-OST0000-osc-ffff999b450de000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5184.613082] LustreError: 87468:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-OST0000-osc-ffff999b450de000: namespace resource [0x240000400:0x4b4d:0x0].0x0 (ffff999b412e4700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5184.620371] LustreError: 87468:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5186.655223] LustreError: 87385:0:(osc_request.c:2978:osc_build_rpc()) cfs_fail_timeout id 414 awake
[ 5186.661616] Lustre: 2367:0:(llite_lib.c:4238:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.143@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5189.517488] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 02:45:51 (1761288351)
[ 5256.330922] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 02:46:57 (1761288417)
[ 5258.424916] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5260.232789] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 02:47:01 (1761288421)
[ 5322.081832] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5323.418664] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 02:48:04 (1761288484)
[ 5381.514895] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 02:49:02 (1761288542)
[ 5383.588195] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5385.424889] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 02:49:06 (1761288546)
[ 5386.749074] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5388.287725] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 02:49:09 (1761288549)
[ 5414.956960] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 02:49:36 (1761288576)
[ 5423.514705] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5446.623186] Lustre: 2368:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761288593/real 1761288593]  req@ffff999b48135f80 x1846839354893952/t0(0) o400->MGC192.168.202.143@tcp@192.168.202.143@tcp:26/25 lens 224/224 e 0 to 1 dl 1761288609 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5446.638695] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 5446.643595] Lustre: 2368:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 5446.654882] LustreError: Skipped 2 previous similar messages
[ 5446.660799] Lustre: lustre-MDT0000-mdc-ffff999b450de000: Connection to lustre-MDT0000 (at 192.168.202.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5446.670689] Lustre: Skipped 16 previous similar messages
[ 5446.671939] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x268f34dc77391d0 to 0x268f34dc773ac2c
[ 5446.681211] Lustre: Skipped 2 previous similar messages
[ 5453.915261] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5455.196811] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5462.918777] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 02:50:24 (1761288624)
[ 5464.665118] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 5466.300128] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 02:50:27 (1761288627)
[ 5488.687240] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 02:50:50 (1761288650)
[ 5508.078305] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 5508.086872] LustreError: Skipped 1 previous similar message
[ 5508.095978] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x268f34dc773af58 to 0x268f34dc773b165
[ 5508.099396] Lustre: Skipped 1 previous similar message
[ 5530.929101] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 02:51:32 (1761288692)
[ 5536.322953] LustreError: lustre-OST0000-osc-ffff999b450de000: operation ost_setattr to node 192.168.202.143@tcp failed: rc = -107
[ 5536.326540] LustreError: Skipped 9 previous similar messages
[ 5569.708103] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5572.132929] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5650.404242] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 5650.416130] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x268f34dc773b165 to 0x268f34dc773c092
[ 5650.423853] Lustre: MGC192.168.202.143@tcp: Connection restored to 192.168.202.143@tcp (at 192.168.202.143@tcp)
[ 5650.428271] Lustre: Skipped 9 previous similar messages
[ 5655.915390] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5656.642624] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5678.736334] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5679.397305] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5751.454454] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 02:55:13 (1761288913)
[ 5778.717892] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5779.913026] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5970.408589] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 02:58:52 (1761289132)
[ 6060.004764] Lustre: lustre-MDT0000-mdc-ffff999b450de000: Connection to lustre-MDT0000 (at 192.168.202.143@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6060.011936] Lustre: Skipped 6 previous similar messages
[ 6060.012373] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 6060.020298] LustreError: Skipped 1 previous similar message
[ 6060.156043] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x268f34dc773c5f5 to 0x268f34dc7836691
[ 6060.160823] Lustre: Skipped 1 previous similar message
[ 6079.726429] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 03:00:41 (1761289241)
[ 6080.365364] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6081.099930] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 03:00:43 (1761289243)
[ 6081.946504] LustreError: lustre-MDT0000-mdc-ffff999b450de000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6085.194476] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 03:00:47 (1761289247)
[ 6250.982803] LustreError: lustre-OST0000-osc-ffff999b450de000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6251.810409] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 03:03:33 (1761289413)
[ 6274.527106] Lustre: 2366:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761289416/real 1761289416]  req@ffff999b7983e300 x1846839372543872/t0(0) o4->lustre-OST0000-osc-ffff999b450de000@192.168.202.143@tcp:6/4 lens 4584/448 e 0 to 1 dl 1761289436 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 6274.541046] Lustre: 2366:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[ 6274.550669] Lustre: lustre-OST0000-osc-ffff999b450de000: Connection restored to 192.168.202.143@tcp (at 192.168.202.143@tcp)
[ 6274.553880] Lustre: Skipped 6 previous similar messages
[ 6287.017347] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 03:04:09 (1761289449)
[ 6287.692381] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 6288.458866] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 03:04:10 (1761289450)
[ 6289.104338] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 6289.915169] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 03:04:11 (1761289451)
[ 6315.975323] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 03:04:38 (1761289478)
[ 6348.259568] LustreError: MGC192.168.202.143@tcp: Connection to MGS (at 192.168.202.143@tcp) was lost; in progress operations using this service will fail
[ 6348.267563] Lustre: Evicted from MGS (at 192.168.202.143@tcp) after server handle changed from 0x268f34dc7836691 to 0x268f34dc7838e14
[ 6348.271768] Lustre: Skipped 1 previous similar message
[ 6355.958162] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 03:05:17 (1761289517)
[ 6356.625349] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 6357.335374] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 03:05:19 (1761289519)
[ 6357.983192] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 6358.714286] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 03:05:20 (1761289520)
[ 6362.788387] LustreError: 106239:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff999b450de000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
[ 6362.793030] LustreError: 106239:0:(lov_obd.c:783:lov_cleanup()) Skipped 1 previous similar message
[ 6362.802787] LustreError: 106239:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup
[ 6362.805319] LustreError: 106239:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages
[ 6362.840518] Lustre: Unmounted lustre-client
[ 6363.441452] Lustre: Mounted lustre-client
[ 6364.677636] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6382.414335] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: total 1809