[ 1871.971058] LustreError: lustre-OST0000-osc-ffff9ba7ca371000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1871.990819] Lustre: 2349:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.137@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1878.386332] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 21:12:04 (1788311524)
[ 1880.450115] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1882.149181] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 21:12:08 (1788311528)
[ 1883.542181] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1885.283258] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 21:12:11 (1788311531)
[ 1888.595545] LustreError: lustre-MDT0000-mdc-ffff9ba7ca371000: operation mds_reint to node 192.168.203.137@tcp failed: rc = -19
[ 1888.610903] LustreError: Skipped 1 previous similar message
[ 1907.936327] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail
[ 1907.969142] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0xef14793d4f1c87a2 to 0xef14793d4f1ca1b1
[ 2008.318161] LustreError: lustre-MDT0000-mdc-ffff9ba7ca371000: operation mds_close to node 192.168.203.137@tcp failed: rc = -19
[ 2008.329118] LustreError: Skipped 1 previous similar message
[ 2029.035636] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail
[ 2029.078300] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0xef14793d4f1ca1b1 to 0xef14793d4f1e4ee8
[ 2045.353077] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 21:14:51 (1788311691)
[ 2045.815250] Lustre: *** cfs_fail_loc=305, val=0***
[ 2045.821382] Lustre: Skipped 2 previous similar messages
[ 2086.367312] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail
[ 2086.399914] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0xef14793d4f1e4ee8 to 0xef14793d4f1e7b1f
[ 2102.387652] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2104.153485] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2113.600265] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 21:15:59 (1788311759)
[ 2127.364725] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail
[ 2127.390259] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0xef14793d4f1e7b1f to 0xef14793d4f1e7ed7
[ 2127.523846] LustreError: lustre-MDT0000-mdc-ffff9ba7ca371000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2150.262531] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 21:16:36 (1788311796)
[ 2152.936605] Lustre: lustre-OST0000-osc-ffff9ba7ca371000: Connection to lustre-OST0000 (at 192.168.203.137@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2152.943300] Lustre: Skipped 13 previous similar messages
[ 2164.066397] LustreError: lustre-OST0000-osc-ffff9ba7ca371000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2184.556806] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 21:17:10 (1788311830)
[ 2197.871032] LustreError: lustre-MDT0000-mdc-ffff9ba7ca371000: operation ldlm_enqueue to node 192.168.203.137@tcp failed: rc = -19
[ 2214.943232] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail
[ 2225.137756] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0xef14793d4f1e7ed7 to 0xef14793d4f1ec341
[ 2225.150696] Lustre: MGC192.168.203.137@tcp: Connection restored to 192.168.203.137@tcp (at 192.168.203.137@tcp)
[ 2225.156287] Lustre: Skipped 17 previous similar messages
[ 2242.907721] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2245.576479] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2331.615179] Lustre: 2351:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788311963/real 1788311963]  req@ffff9ba7d0358380 x1875178644901760/t0(0) o400->MGC192.168.203.137@tcp@192.168.203.137@tcp:26/25 lens 224/224 e 0 to 1 dl 1788311979 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2331.656942] Lustre: 2351:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 2331.664750] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail
[ 2341.862277] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0xef14793d4f1ec341 to 0xef14793d4f203362
[ 2358.339484] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2360.519211] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2425.353111] LustreError: lustre-MDT0000-mdc-ffff9ba7ca371000: operation mds_close to node 192.168.203.137@tcp failed: rc = -19
[ 2425.372283] LustreError: Skipped 3 previous similar messages
[ 2446.304276] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail
[ 2446.332649] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0xef14793d4f203362 to 0xef14793d4f21cc72
[ 2471.986910] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2474.672712] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2504.370922] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 21:22:30 (1788312150)
[ 2526.245704] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail
[ 2536.463329] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2536.504519] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0xef14793d4f21cc72 to 0xef14793d4f2299e8
[ 2565.684716] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2600.420713] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2640.236289] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2681.824867] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail
[ 2681.848363] LustreError: Skipped 3 previous similar messages
[ 2692.019250] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0xef14793d4f22ca08 to 0xef14793d4f231207
[ 2692.034449] Lustre: Skipped 3 previous similar messages
[ 2692.140912] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2720.460882] LustreError: lustre-MDT0000-mdc-ffff9ba7ca371000: operation ldlm_enqueue to node 192.168.203.137@tcp failed: rc = -19
[ 2720.476015] LustreError: Skipped 6 previous similar messages
[ 2748.888393] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2782.204958] Lustre: lustre-MDT0000-mdc-ffff9ba7ca371000: Connection to lustre-MDT0000 (at 192.168.203.137@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2782.224759] Lustre: Skipped 8 previous similar messages
[ 2834.839919] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 21:28:01 (1788312481)
[ 2871.640933] Lustre: lustre-OST0000-osc-ffff9ba7ca371000: Connection restored to 192.168.203.137@tcp (at 192.168.203.137@tcp)
[ 2871.672595] Lustre: Skipped 18 previous similar messages
[ 2890.226134] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2893.190746] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3216.975280] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3219.303816] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3511.823730] LustreError: lustre-OST0000-osc-ffff9ba7ca371000: operation ost_write to node 192.168.203.137@tcp failed: rc = -19
[ 3511.839035] LustreError: Skipped 4 previous similar messages
[ 3511.850402] Lustre: lustre-OST0000-osc-ffff9ba7ca371000: Connection to lustre-OST0000 (at 192.168.203.137@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3511.876387] Lustre: Skipped 2 previous similar messages
[ 3555.165975] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3558.574172] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3806.346408] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 21:44:12 (1788313452)
[ 3824.096677] Lustre: 50587:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788313455/real 1788313455]  req@ffff9ba7c8cdc700 x1875178665593856/t0(0) o101->lustre-MDT0000-mdc-ffff9ba7ca371000@192.168.203.137@tcp:12/10 lens 576/1152 e 0 to 1 dl 1788313471 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3824.139247] Lustre: 50587:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 3824.196751] Lustre: lustre-MDT0000-mdc-ffff9ba7ca371000: Connection restored to 192.168.203.137@tcp (at 192.168.203.137@tcp)
[ 3832.085784] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 21:44:37 (1788313477)
[ 3858.067770] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 21:45:04 (1788313504)
[ 3884.832940] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 21:45:30 (1788313530)
[ 3885.355781] Lustre: Mounted lustre-client - version 2.17.57_45_g07d9be3
[ 3917.279171] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788313548/real 1788313548]  req@ffff9ba7c8cdca80 x1875178665612160/t0(0) o400->lustre-MDT0000-mdc-ffff9ba7ca371000@192.168.203.137@tcp:12/10 lens 224/224 e 0 to 1 dl 1788313564 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3917.315073] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3917.335806] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail
[ 3917.347088] LustreError: Skipped 2 previous similar messages
[ 3917.384052] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0xef14793d4f243a45 to 0xef14793d4f3879cc
[ 3917.398336] Lustre: Skipped 2 previous similar messages
[ 3933.455841] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3935.179296] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3937.277559] Lustre: Unmounted lustre-client
[ 3944.640808] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 21:46:30 (1788313590)
[ 4083.167174] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788313714/real 1788313714]  req@ffff9ba7c8cf2a00 x1875178665645056/t0(0) o4->lustre-OST0000-osc-ffff9ba7ca371000@192.168.203.137@tcp:6/4 lens 488/448 e 0 to 1 dl 1788313730 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4083.233879] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 62 previous similar messages
[ 4114.911268] Lustre: lustre-OST0000-osc-ffff9ba7ca371000: Connection to lustre-OST0000 (at 192.168.203.137@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4114.925364] Lustre: Skipped 14 previous similar messages
[ 4240.427191] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 21:51:26 (1788313886)
[ 4290.152722] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 21:52:16 (1788313936)
[ 4293.574974] Lustre: Unmounted lustre-client
[ 4316.535499] Lustre: Mounted lustre-client - version 2.17.57_45_g07d9be3
[ 4324.093039] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 21:52:49 (1788313969)
[ 4324.570979] LustreError: 56516:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4325.617539] LustreError: 56516:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4325.896208] Lustre: *** cfs_fail_loc=305, val=0***
[ 4351.210569] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 21:53:16 (1788313996)
[ 4352.006905] Lustre: Mounted lustre-client - version 2.17.57_45_g07d9be3
[ 4354.453973] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4364.766837] Lustre: Unmounted lustre-client
[ 4373.005898] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 21:53:38 (1788314018)
[ 4462.156686] LustreError: lustre-MDT0000-mdc-ffff9ba7c8dca000: operation mds_reint to node 192.168.203.137@tcp failed: rc = -19
[ 4479.968765] Lustre: 2351:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788314111/real 1788314112]  req@ffff9ba7c583c380 x1875178668321280/t0(0) o400->MGC192.168.203.137@tcp@192.168.203.137@tcp:26/25 lens 224/224 e 0 to 1 dl 1788314127 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4479.989072] Lustre: 2351:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 75 previous similar messages
[ 4479.995737] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail
[ 4490.216502] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0xef14793d4f3893e2 to 0xef14793d4f3b47b5
[ 4490.239089] Lustre: MGC192.168.203.137@tcp: Connection restored to 192.168.203.137@tcp (at 192.168.203.137@tcp)
[ 4490.251059] Lustre: Skipped 23 previous similar messages
[ 4660.463404] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 21:58:25 (1788314305)
[ 4666.790633] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4688.864415] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail
[ 4688.908552] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0xef14793d4f3b47b5 to 0xef14793d4f4012ad
[ 4688.979387] LustreError: lustre-MDT0000-mdc-ffff9ba7c8dca000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4705.297828] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 21:59:11 (1788314351)
[ 4706.044417] Lustre: Mounted lustre-client - version 2.17.57_45_g07d9be3
[ 4713.322379] LustreError: lustre-OST0000-osc-ffff9ba7c8dca000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4722.655705] Lustre: lustre-OST0000-osc-ffff9ba7c4dec000: Connection to lustre-OST0000 (at 192.168.203.137@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4722.677486] Lustre: Skipped 12 previous similar messages
[ 4725.017137] Lustre: Unmounted lustre-client
[ 4732.275191] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 21:59:38 (1788314378)
[ 4739.590293] LustreError: lustre-MDT0000-mdc-ffff9ba7c8dca000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4739.616712] LustreError: 60995:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9ba7c8dca000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9ba7c588f900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 4739.663986] LustreError: 60911:0:(file.c:6166:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4747.873877] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 21:59:53 (1788314393)
[ 4748.246288] LustreError: 61615:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4753.377515] LustreError: 61615:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4753.384545] LustreError: 61615:0:(import.c:711:ptlrpc_connect_import_locked()) already connected
[ 4754.397619] LustreError: lustre-MDT0000-mdc-ffff9ba7c8dca000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4754.423180] LustreError: 61633:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4754.492919] LustreError: 61638:0:(file.c:6166:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4755.595719] LustreError: 61644:0:(file.c:6166:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4755.612808] LustreError: 61644:0:(file.c:6166:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4757.964151] LustreError: 61666:0:(file.c:6166:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4757.976201] LustreError: 61666:0:(file.c:6166:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4762.791235] LustreError: 61711:0:(file.c:6166:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4762.818470] LustreError: 61711:0:(file.c:6166:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4774.892061] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 22:00:20 (1788314420)
[ 4819.113702] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4820.851577] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4833.362707] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 22:01:18 (1788314478)
[ 4873.548512] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4874.887218] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4886.356861] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 22:02:12 (1788314532)
[ 4949.640746] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4951.201334] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4959.946475] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 22:03:25 (1788314605)
[ 4999.720585] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5001.473455] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5007.184657] Lustre: Unmounted lustre-client
[ 5060.793909] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5063.622167] Lustre: Mounted lustre-client - version 2.17.57_45_g07d9be3
[ 5070.916081] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 22:05:16 (1788314716)
[ 5073.899357] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5075.542641] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 22:05:21 (1788314721)
[ 5108.657991] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 22:05:53 (1788314753)
[ 5110.917369] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5113.049992] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 22:05:58 (1788314758)
[ 5115.098429] Lustre: *** cfs_fail_loc=805, val=0***
[ 5115.165433] Lustre: Mounted lustre-client - version 2.17.57_45_g07d9be3
[ 5120.747860] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5140.896517] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788314772/real 1788314772]  req@ffff9ba7d0e6dc00 x1875178670374528/t0(0) o400->MGC192.168.203.137@tcp@192.168.203.137@tcp:26/25 lens 224/224 e 0 to 1 dl 1788314788 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5140.939644] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5140.945731] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail
[ 5151.207502] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0xef14793d4f401cd2 to 0xef14793d4f401eed
[ 5151.223574] Lustre: MGC192.168.203.137@tcp: Connection restored to 192.168.203.137@tcp (at 192.168.203.137@tcp)
[ 5151.238235] Lustre: Skipped 12 previous similar messages
[ 5166.585741] LustreError: 2348:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9ba7c50fd800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9ba7cbbe9f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5169.245820] Lustre: Unmounted lustre-client
[ 5178.896111] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 22:07:04 (1788314824)
[ 5206.103701] LustreError: 2348:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff9ba7d0780700 x1875178670385920/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff9ba7d18b1800@192.168.203.137@tcp:12/10 lens 648/608 e 0 to 0 dl 1788314869 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5218.946812] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5220.546434] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5230.157862] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 22:07:56 (1788314876)
[ 5231.417395] LustreError: lustre-OST0000-osc-ffff9ba7d18b1800: operation ost_write to node 192.168.203.137@tcp failed: rc = -107
[ 5231.427208] LustreError: Skipped 3 previous similar messages
[ 5231.436783] LustreError: lustre-OST0000-osc-ffff9ba7d18b1800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5231.447025] Lustre: 2351:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.137@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5231.471830] LustreError: 72827:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9ba7d18b1800: namespace resource [0x240000400:0x3bc2:0x0].0x0 (ffff9ba7c588f100) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5231.505583] LustreError: 72827:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5239.972709] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 22:08:05 (1788314885)
[ 5241.689488] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5243.482603] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 22:08:09 (1788314889)
[ 5245.459178] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5247.299396] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 22:08:13 (1788314893)
[ 5249.180390] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5251.452233] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 22:08:17 (1788314897)
[ 5253.011264] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5255.024956] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 22:08:20 (1788314900)
[ 5256.768710] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5259.439338] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 22:08:24 (1788314904)
[ 5261.376799] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5263.364709] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 22:08:29 (1788314909)
[ 5265.291129] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5267.478632] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 22:08:33 (1788314913)
[ 5269.654210] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5271.967641] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 22:08:37 (1788314917)
[ 5273.754512] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5276.181368] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 22:08:41 (1788314921)
[ 5278.031049] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5279.745157] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 22:08:45 (1788314925)
[ 5281.230335] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5283.397689] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 22:08:49 (1788314929)
[ 5285.938734] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5287.728701] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 22:08:53 (1788314933)
[ 5311.985594] Lustre: 68288:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.137@tcp: IR log lustre-cliir failed, not fatal: rc = -5
[ 5324.721385] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 22:09:30 (1788314970)
[ 5357.630230] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 22:10:03 (1788315003)
[ 5358.127087] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5368.002773] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 22:10:14 (1788315014)
[ 5368.314364] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5378.043382] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 22:10:24 (1788315024)
[ 5378.534513] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5387.607832] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 22:10:33 (1788315033)
[ 5387.923917] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5396.816577] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 22:10:42 (1788315042)
[ 5397.189554] Lustre: *** cfs_fail_loc=510, val=3***
[ 5404.906278] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 22:10:51 (1788315051)
[ 5405.235249] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5415.307975] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 22:11:01 (1788315061)
[ 5415.781784] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5474.783250] Lustre: lustre-OST0000-osc-ffff9ba7d18b1800: Connection to lustre-OST0000 (at 192.168.203.137@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5474.813405] Lustre: Skipped 14 previous similar messages
[ 5482.485229] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 22:12:08 (1788315128)
[ 5483.044432] LustreError: 83039:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5485.917703] LustreError: lustre-MDT0000-mdc-ffff9ba7d18b1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5485.945289] LustreError: 83053:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9ba7d18b1800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9ba7c8d65000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5487.128127] LustreError: 83039:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5489.283073] LustreError: 83069:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5492.214504] LustreError: lustre-MDT0000-mdc-ffff9ba7d18b1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5492.231704] LustreError: 83084:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9ba7d18b1800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9ba7c8d65900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5492.250970] LustreError: 83084:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5493.375196] LustreError: 83069:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5500.365096] LustreError: lustre-MDT0000-mdc-ffff9ba7d18b1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5500.709947] LustreError: 83125:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5503.893605] LustreError: lustre-MDT0000-mdc-ffff9ba7d18b1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5503.991436] LustreError: 83145:0:(file.c:6166:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5504.008358] LustreError: 83145:0:(file.c:6166:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5504.787696] LustreError: 83125:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5504.809096] LustreError: 83125:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9ba7d18b1800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5505.140457] LustreError: 83151:0:(file.c:6166:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5505.165240] LustreError: 83151:0:(file.c:6166:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5507.622179] LustreError: 83174:0:(file.c:6166:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5507.644088] LustreError: 83174:0:(file.c:6166:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5511.335087] LustreError: 83200:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5514.554531] LustreError: lustre-MDT0000-mdc-ffff9ba7d18b1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5514.583570] LustreError: 83215:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9ba7d18b1800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9ba7cbbe9e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5514.630654] LustreError: 83215:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5515.407130] LustreError: 83200:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5522.560036] LustreError: lustre-MDT0000-mdc-ffff9ba7d18b1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5522.983630] LustreError: 83257:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5525.982787] LustreError: lustre-MDT0000-mdc-ffff9ba7d18b1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5527.071141] LustreError: 83257:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5530.167984] LustreError: lustre-MDT0000-mdc-ffff9ba7d18b1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5530.250143] LustreError: 83304:0:(file.c:6166:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5530.269129] LustreError: 83304:0:(file.c:6166:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5531.280691] LustreError: 83284:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9ba7d18b1800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5539.419556] LustreError: 83356:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5539.429439] LustreError: 83356:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5543.011119] LustreError: lustre-MDT0000-mdc-ffff9ba7d18b1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5543.087921] LustreError: 83378:0:(file.c:6166:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5543.108485] LustreError: 83378:0:(file.c:6166:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5543.423128] LustreError: 83356:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5543.432261] LustreError: 83356:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5543.439377] LustreError: 83356:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9ba7d18b1800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5555.424323] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 22:13:21 (1788315201)
[ 5575.587884] LustreError: 68285:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5575.610567] LustreError: 68285:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5579.673600] LustreError: 68285:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5579.681356] LustreError: 68285:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5587.555607] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 22:13:53 (1788315233)
[ 5640.668531] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 22:14:46 (1788315286)
[ 5707.131068] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 22:15:52 (1788315352)
[ 5741.584939] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 22:16:27 (1788315387)
[ 5742.093397] Lustre: Mounted lustre-client - version 2.17.57_45_g07d9be3
[ 5866.520684] Lustre: Unmounted lustre-client
[ 5874.188963] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 22:18:40 (1788315520)
[ 5875.101070] LustreError: 87683:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5878.137254] LustreError: lustre-OST0000-osc-ffff9ba7d18b1800: operation ost_statfs to node 192.168.203.137@tcp failed: rc = -107
[ 5878.142252] LustreError: Skipped 9 previous similar messages
[ 5878.161315] LustreError: lustre-OST0000-osc-ffff9ba7d18b1800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5878.209268] LustreError: 87770:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9ba7d18b1800: namespace resource [0x240000400:0x3bed:0x0].0x0 (ffff9ba7c588fd00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5878.237902] LustreError: 87770:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 4 previous similar messages
[ 5878.487108] LustreError: 87683:0:(osc_request.c:2990:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5878.501240] Lustre: 2352:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.137@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5878.523298] Lustre: lustre-OST0000-osc-ffff9ba7d18b1800: Connection restored to 192.168.203.137@tcp (at 192.168.203.137@tcp)
[ 5878.544707] Lustre: Skipped 19 previous similar messages
[ 5885.546859] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 22:18:51 (1788315531)
[ 5943.264085] Lustre: 88350:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788315535/real 1788315535]  req@ffff9ba7d132d180 x1875178670556544/t0(0) o101->lustre-MDT0000-mdc-ffff9ba7d18b1800@192.168.203.137@tcp:12/10 lens 328/344 e 0 to 1 dl 1788315590 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5943.297974] Lustre: 88350:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 17 previous similar messages
[ 5950.499160] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 22:19:56 (1788315596)
[ 5952.322235] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5954.463375] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 22:20:00 (1788315600)
[ 6018.224232] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6020.544600] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 22:21:06 (1788315666)
[ 6077.408435] Lustre: lustre-MDT0000-mdc-ffff9ba7d18b1800: Connection to lustre-MDT0000 (at 192.168.203.137@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6077.421515] Lustre: Skipped 15 previous similar messages
[ 6079.197252] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 22:22:05 (1788315725)
[ 6081.977616] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6083.748441] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 22:22:09 (1788315729)
[ 6085.717207] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6087.698783] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 22:22:13 (1788315733)
[ 6125.081044] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 22:22:50 (1788315770)
[ 6139.921332] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6170.591943] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail
[ 6170.598347] LustreError: Skipped 3 previous similar messages
[ 6170.621618] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0xef14793d4f402b3b to 0xef14793d4f404352
[ 6170.626957] Lustre: Skipped 3 previous similar messages
[ 6185.395700] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6187.944920] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6201.741538] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 22:24:07 (1788315847)
[ 6204.418956] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6205.979963] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 22:24:12 (1788315852)
[ 6221.812078] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail
[ 6221.859929] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0xef14793d4f404352 to 0xef14793d4f4047f8
[ 6232.020341] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 22:24:38 (1788315878)
[ 6266.835597] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 22:25:13 (1788315913)
[ 6315.718150] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6318.831991] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6405.102944] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail
[ 6405.119242] LustreError: Skipped 1 previous similar message
[ 6405.138575] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0xef14793d4f404b01 to 0xef14793d4f405852
[ 6405.157758] Lustre: Skipped 1 previous similar message
[ 6406.960261] Lustre: 68288:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.137@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6432.819908] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6434.406678] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6458.545221] Lustre: 68288:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.137@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6479.912211] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6481.116263] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6598.830904] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 22:30:45 (1788316245)
[ 6609.180481] LustreError: lustre-OST0000-osc-ffff9ba7d18b1800: operation ost_setattr to node 192.168.203.137@tcp failed: rc = -19
[ 6609.186307] LustreError: Skipped 21 previous similar messages
[ 6654.826269] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6657.363374] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6920.095723] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 22:36:06 (1788316566)
[ 7050.236481] Lustre: lustre-MDT0000-mdc-ffff9ba7d18b1800: Connection to lustre-MDT0000 (at 192.168.203.137@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7050.258430] Lustre: Skipped 7 previous similar messages
[ 7060.456701] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail
[ 7060.476092] LustreError: Skipped 1 previous similar message
[ 7060.498238] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0xef14793d4f405c65 to 0xef14793d4f4ffffc
[ 7060.505730] Lustre: Skipped 1 previous similar message
[ 7060.510981] Lustre: MGC192.168.203.137@tcp: Connection restored to 192.168.203.137@tcp (at 192.168.203.137@tcp)
[ 7060.521085] Lustre: Skipped 13 previous similar messages
[ 7099.162655] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 22:39:05 (1788316745)
[ 7101.104473] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7103.266387] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 22:39:09 (1788316749)
[ 7105.200636] LustreError: lustre-MDT0000-mdc-ffff9ba7d18b1800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7113.124397] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 22:39:19 (1788316759)
[ 7280.564414] LustreError: lustre-OST0000-osc-ffff9ba7d18b1800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7282.303806] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 22:42:08 (1788316928)
[ 7307.231160] Lustre: 2349:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788316934/real 1788316934]  req@ffff9ba7e65b2680 x1875178688279552/t0(0) o4->lustre-OST0000-osc-ffff9ba7d18b1800@192.168.203.137@tcp:6/4 lens 4584/448 e 0 to 1 dl 1788316954 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7307.260650] Lustre: 2349:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 27 previous similar messages
[ 7325.854890] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 22:42:51 (1788316971)
[ 7327.559599] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7329.203720] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 22:42:55 (1788316975)
[ 7330.672560] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7332.712624] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 22:42:58 (1788316978)
[ 7366.628751] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 22:43:32 (1788317012)
[ 7408.616543] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail
[ 7408.640869] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0xef14793d4f4ffffc to 0xef14793d4f5027ef
[ 7408.654954] Lustre: Skipped 1 previous similar message
[ 7427.356767] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 22:44:33 (1788317073)
[ 7429.225421] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7431.065573] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 22:44:36 (1788317076)
[ 7432.847484] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7434.872687] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 22:44:40 (1788317080)
[ 7447.540672] Lustre: Unmounted lustre-client
[ 7449.531497] Lustre: Mounted lustre-client - version 2.17.57_45_g07d9be3
[ 7452.802283] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7476.192627] Lustre: 2349:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788317107/real 1788317107]  req@ffff9ba7e7c3f480 x1875178688467200/t0(0) o400->MGC192.168.203.137@tcp@192.168.203.137@tcp:26/25 lens 224/224 e 0 to 1 dl 1788317123 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7495.333037] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2