[ 1857.303148] LustreError: lustre-OST0000-osc-ffff8fc350b56000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1857.328665] Lustre: 2350:0:(llite_lib.c:4352:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.139@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1864.278977] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 09:22:42 (1787836962)
[ 1866.416038] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1868.379395] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 09:22:46 (1787836966)
[ 1870.077029] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1871.992579] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 09:22:49 (1787836969)
[ 1893.345278] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 192.168.206.139@tcp) was lost; in progress operations using this service will fail
[ 1903.530334] Lustre: Evicted from MGS (at 192.168.206.139@tcp) after server handle changed from 0xab8c53c924f86894 to 0xab8c53c924f87def
[ 2003.057358] LustreError: lustre-MDT0000-mdc-ffff8fc350b56000: operation ldlm_enqueue to node 192.168.206.139@tcp failed: rc = -19
[ 2003.071125] LustreError: Skipped 2 previous similar messages
[ 2021.343392] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 192.168.206.139@tcp) was lost; in progress operations using this service will fail
[ 2021.414467] Lustre: Evicted from MGS (at 192.168.206.139@tcp) after server handle changed from 0xab8c53c924f87def to 0xab8c53c924fa54e7
[ 2047.997643] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 09:25:45 (1787837145)
[ 2048.429475] Lustre: *** cfs_fail_loc=305, val=0***
[ 2048.432125] Lustre: Skipped 2 previous similar messages
[ 2068.549731] LustreError: lustre-OST0000-osc-ffff8fc350b56000: operation ost_connect to node 192.168.206.139@tcp failed: rc = -75
[ 2091.871241] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 192.168.206.139@tcp) was lost; in progress operations using this service will fail
[ 2102.501455] Lustre: Evicted from MGS (at 192.168.206.139@tcp) after server handle changed from 0xab8c53c924fa54e7 to 0xab8c53c924fa5d0d
[ 2106.685519] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2108.044157] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2115.460134] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 09:26:53 (1787837213)
[ 2118.129636] Lustre: lustre-MDT0000-mdc-ffff8fc350b56000: Connection to lustre-MDT0000 (at 192.168.206.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2118.148029] Lustre: Skipped 12 previous similar messages
[ 2128.362324] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 192.168.206.139@tcp) was lost; in progress operations using this service will fail
[ 2128.387966] Lustre: Evicted from MGS (at 192.168.206.139@tcp) after server handle changed from 0xab8c53c924fa5d0d to 0xab8c53c924fa5fa6
[ 2128.398619] Lustre: MGC192.168.206.139@tcp: Connection restored to 192.168.206.139@tcp (at 192.168.206.139@tcp)
[ 2128.404120] Lustre: Skipped 15 previous similar messages
[ 2138.659419] LustreError: lustre-MDT0000-mdc-ffff8fc350b56000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2154.615244] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 09:27:32 (1787837252)
[ 2181.683294] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 09:27:59 (1787837279)
[ 2211.295174] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 192.168.206.139@tcp) was lost; in progress operations using this service will fail
[ 2221.609710] Lustre: Evicted from MGS (at 192.168.206.139@tcp) after server handle changed from 0xab8c53c924fa5fa6 to 0xab8c53c924fab79d
[ 2241.360050] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2243.262524] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2308.083607] LustreError: lustre-MDT0000-mdc-ffff8fc350b56000: operation ldlm_enqueue to node 192.168.206.139@tcp failed: rc = -19
[ 2308.090077] LustreError: Skipped 1 previous similar message
[ 2327.528750] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 192.168.206.139@tcp) was lost; in progress operations using this service will fail
[ 2327.566653] Lustre: Evicted from MGS (at 192.168.206.139@tcp) after server handle changed from 0xab8c53c924fab79d to 0xab8c53c924fcc545
[ 2329.269447] Lustre: 15925:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.139@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2353.422441] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2355.322901] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2439.135231] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787837522/real 1787837523]  req@ffff8fc344f21e00 x1874681042271104/t0(0) o400->MGC192.168.206.139@tcp@192.168.206.139@tcp:26/25 lens 224/224 e 0 to 1 dl 1787837538 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2439.167444] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
[ 2439.174283] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 192.168.206.139@tcp) was lost; in progress operations using this service will fail
[ 2439.197247] Lustre: Evicted from MGS (at 192.168.206.139@tcp) after server handle changed from 0xab8c53c924fcc545 to 0xab8c53c924feb0fe
[ 2440.555768] Lustre: 15925:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.139@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2464.373395] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2466.029874] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2494.187954] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 09:33:11 (1787837591)
[ 2514.912773] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 192.168.206.139@tcp) was lost; in progress operations using this service will fail
[ 2523.509765] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2525.117107] Lustre: Evicted from MGS (at 192.168.206.139@tcp) after server handle changed from 0xab8c53c924feb0fe to 0xab8c53c924ffa2a8
[ 2554.267333] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2587.954670] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2600.729546] LustreError: lustre-MDT0000-mdc-ffff8fc350b56000: operation mds_reint to node 192.168.206.139@tcp failed: rc = -19
[ 2600.747404] LustreError: Skipped 5 previous similar messages
[ 2630.261855] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2672.607389] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 192.168.206.139@tcp) was lost; in progress operations using this service will fail
[ 2672.629350] LustreError: Skipped 3 previous similar messages
[ 2672.645242] Lustre: Evicted from MGS (at 192.168.206.139@tcp) after server handle changed from 0xab8c53c9250038aa to 0xab8c53c92500fefd
[ 2672.659046] Lustre: Skipped 3 previous similar messages
[ 2674.227477] Lustre: 15925:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.139@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2689.839551] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2736.140305] Lustre: MGC192.168.206.139@tcp: Connection restored to 192.168.206.139@tcp (at 192.168.206.139@tcp)
[ 2736.151784] Lustre: Skipped 16 previous similar messages
[ 2744.251692] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2777.250654] Lustre: lustre-MDT0000-mdc-ffff8fc350b56000: Connection to lustre-MDT0000 (at 192.168.206.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2777.267369] Lustre: Skipped 9 previous similar messages
[ 2828.472800] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 09:38:46 (1787837926)
[ 2881.222996] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2882.964585] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3174.312201] LustreError: lustre-OST0000-osc-ffff8fc350b56000: operation ldlm_enqueue to node 192.168.206.139@tcp failed: rc = -107
[ 3174.321465] LustreError: Skipped 4 previous similar messages
[ 3216.336337] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3218.691354] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3506.456356] Lustre: lustre-OST0000-osc-ffff8fc350b56000: Connection to lustre-OST0000 (at 192.168.206.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3506.472023] Lustre: Skipped 2 previous similar messages
[ 3528.156323] Lustre: lustre-OST0000-osc-ffff8fc350b56000: Connection restored to 192.168.206.139@tcp (at 192.168.206.139@tcp)
[ 3528.166837] Lustre: Skipped 4 previous similar messages
[ 3543.614973] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3545.454789] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3800.870165] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 09:54:58 (1787838898)
[ 3817.953680] Lustre: 50541:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787838901/real 1787838901]  req@ffff8fc3463d4b40 x1874681066140928/t0(0) o101->lustre-MDT0000-mdc-ffff8fc350b56000@192.168.206.139@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787838917 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3818.006802] Lustre: 50541:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 3826.275800] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 09:55:24 (1787838924)
[ 3851.680659] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 09:55:48 (1787838948)
[ 3878.928679] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 09:56:16 (1787838976)
[ 3879.748893] Lustre: Mounted lustre-client - version 2.17.57_97_gdd6e0cb
[ 3911.647177] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787838994/real 1787838994]  req@ffff8fc3462e6d00 x1874681066160384/t0(0) o400->lustre-MDT0000-mdc-ffff8fc350b56000@192.168.206.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1787839010 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3911.675774] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3911.681874] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 192.168.206.139@tcp) was lost; in progress operations using this service will fail
[ 3911.692753] LustreError: Skipped 2 previous similar messages
[ 3920.882171] Lustre: Evicted from MGS (at 192.168.206.139@tcp) after server handle changed from 0xab8c53c92502dea7 to 0xab8c53c9251abb8c
[ 3920.889547] Lustre: Skipped 2 previous similar messages
[ 3941.763180] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3943.206521] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3945.276828] Lustre: Unmounted lustre-client
[ 3951.423934] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 09:57:29 (1787839049)
[ 4071.391331] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787839154/real 1787839154]  req@ffff8fc34622de00 x1874681066194816/t0(0) o4->lustre-OST0000-osc-ffff8fc350b56000@192.168.206.139@tcp:6/4 lens 488/448 e 0 to 1 dl 1787839170 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4071.409706] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 54 previous similar messages
[ 4120.543255] Lustre: lustre-OST0000-osc-ffff8fc350b56000: Connection to lustre-OST0000 (at 192.168.206.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4120.560545] Lustre: Skipped 14 previous similar messages
[ 4135.936089] Lustre: lustre-OST0000-osc-ffff8fc350b56000: Connection restored to 192.168.206.139@tcp (at 192.168.206.139@tcp)
[ 4135.944357] Lustre: Skipped 16 previous similar messages
[ 4246.640295] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 10:02:24 (1787839344)
[ 4295.874345] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 10:03:13 (1787839393)
[ 4298.659635] Lustre: Unmounted lustre-client
[ 4318.551468] Lustre: Mounted lustre-client - version 2.17.57_97_gdd6e0cb
[ 4324.109511] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 10:03:42 (1787839422)
[ 4324.482606] LustreError: 56480:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4325.536673] LustreError: 56480:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4325.667426] Lustre: *** cfs_fail_loc=305, val=0***
[ 4348.828413] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 10:04:06 (1787839446)
[ 4349.182030] Lustre: Mounted lustre-client - version 2.17.57_97_gdd6e0cb
[ 4350.464248] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4360.789468] Lustre: Unmounted lustre-client
[ 4366.758126] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 10:04:24 (1787839464)
[ 4454.904339] LustreError: lustre-MDT0000-mdc-ffff8fc347e9a800: operation mds_reint to node 192.168.206.139@tcp failed: rc = -19
[ 4454.911788] LustreError: Skipped 1 previous similar message
[ 4473.311412] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787839556/real 1787839556]  req@ffff8fc34648e580 x1874681068875008/t0(0) o400->MGC192.168.206.139@tcp@192.168.206.139@tcp:26/25 lens 224/224 e 0 to 1 dl 1787839572 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4473.343329] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 91 previous similar messages
[ 4473.349391] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 192.168.206.139@tcp) was lost; in progress operations using this service will fail
[ 4483.562081] Lustre: Evicted from MGS (at 192.168.206.139@tcp) after server handle changed from 0xab8c53c9251ad2df to 0xab8c53c9251d8ada
[ 4623.584689] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 10:08:41 (1787839721)
[ 4628.346620] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4641.268509] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 192.168.206.139@tcp) was lost; in progress operations using this service will fail
[ 4641.309319] Lustre: Evicted from MGS (at 192.168.206.139@tcp) after server handle changed from 0xab8c53c9251d8ada to 0xab8c53c9252250ca
[ 4651.553162] LustreError: lustre-MDT0000-mdc-ffff8fc347e9a800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4658.834359] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 10:09:16 (1787839756)
[ 4659.115593] Lustre: Mounted lustre-client - version 2.17.57_97_gdd6e0cb
[ 4666.040708] LustreError: lustre-OST0000-osc-ffff8fc347e9a800: operation ldlm_enqueue to node 192.168.206.139@tcp failed: rc = -107
[ 4666.055588] LustreError: lustre-OST0000-osc-ffff8fc347e9a800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4678.558440] Lustre: Unmounted lustre-client
[ 4684.635089] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 10:09:42 (1787839782)
[ 4691.047520] LustreError: lustre-MDT0000-mdc-ffff8fc347e9a800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4691.073141] LustreError: 60864:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4697.541606] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 10:09:55 (1787839795)
[ 4697.754663] LustreError: 61569:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4703.200704] LustreError: 61569:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4703.213339] LustreError: 61569:0:(import.c:711:ptlrpc_connect_import_locked()) already connected
[ 4703.952567] LustreError: lustre-MDT0000-mdc-ffff8fc347e9a800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4703.971605] LustreError: 61585:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4704.028654] LustreError: 61590:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4705.144565] LustreError: 61597:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4705.155456] LustreError: 61597:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4707.432412] LustreError: 61619:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4707.443182] LustreError: 61619:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4712.060950] LustreError: 61664:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4712.071109] LustreError: 61664:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4721.447553] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 10:10:19 (1787839819)
[ 4728.814464] Lustre: lustre-OST0000-osc-ffff8fc347e9a800: Connection to lustre-OST0000 (at 192.168.206.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4728.833387] Lustre: Skipped 17 previous similar messages
[ 4758.781615] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4760.802892] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4769.798221] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 10:11:07 (1787839867)
[ 4794.478126] Lustre: lustre-OST0000-osc-ffff8fc347e9a800: Connection restored to 192.168.206.139@tcp (at 192.168.206.139@tcp)
[ 4794.492680] Lustre: Skipped 17 previous similar messages
[ 4808.386929] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4810.022195] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4819.140622] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 10:11:57 (1787839917)
[ 4877.885647] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4878.944270] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4884.743606] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 10:13:03 (1787839983)
[ 4913.190286] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4914.342948] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4918.996535] Lustre: Unmounted lustre-client
[ 4966.092190] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4968.396942] Lustre: Mounted lustre-client - version 2.17.57_97_gdd6e0cb
[ 4974.492851] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 10:14:32 (1787840072)
[ 4976.767865] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 4978.042420] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 10:14:36 (1787840076)
[ 5002.303609] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 10:15:00 (1787840100)
[ 5003.571656] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5004.862378] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 10:15:03 (1787840103)
[ 5005.728462] Lustre: *** cfs_fail_loc=805, val=0***
[ 5005.773282] Lustre: Mounted lustre-client - version 2.17.57_97_gdd6e0cb
[ 5009.711431] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5031.398288] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 192.168.206.139@tcp) was lost; in progress operations using this service will fail
[ 5031.437484] Lustre: Evicted from MGS (at 192.168.206.139@tcp) after server handle changed from 0xab8c53c925225af6 to 0xab8c53c925225e92
[ 5035.445725] Lustre: Unmounted lustre-client
[ 5041.816860] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 10:15:39 (1787840139)
[ 5069.277908] LustreError: 2348:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8fc350891e00 x1874681070929536/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff8fc344184800@192.168.206.139@tcp:12/10 lens 664/608 e 0 to 0 dl 1787840184 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5074.426627] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5075.815490] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5083.630597] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 10:16:21 (1787840181)
[ 5084.962121] LustreError: lustre-OST0000-osc-ffff8fc344184800: operation ost_write to node 192.168.206.139@tcp failed: rc = -107
[ 5084.976065] LustreError: Skipped 2 previous similar messages
[ 5084.995370] LustreError: lustre-OST0000-osc-ffff8fc344184800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5085.023028] Lustre: 2349:0:(llite_lib.c:4352:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.139@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5085.053620] LustreError: 72763:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8fc344184800: namespace resource [0x240000400:0x45c2:0x0].0x0 (ffff8fc34863c400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5092.817828] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 10:16:30 (1787840190)
[ 5094.446470] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5096.383297] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 10:16:34 (1787840194)
[ 5098.589054] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5100.543405] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 10:16:38 (1787840198)
[ 5102.065529] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5104.208439] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 10:16:41 (1787840201)
[ 5106.083152] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5108.445450] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 10:16:45 (1787840205)
[ 5110.337318] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5112.257967] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 10:16:49 (1787840209)
[ 5114.161227] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5115.920137] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 10:16:53 (1787840213)
[ 5117.371924] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5118.831735] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 10:16:56 (1787840216)
[ 5120.322885] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5121.843636] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 10:16:59 (1787840219)
[ 5123.349931] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5125.207827] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 10:17:02 (1787840222)
[ 5126.786345] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5128.694771] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 10:17:06 (1787840226)
[ 5129.896549] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5131.335036] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 10:17:09 (1787840229)
[ 5133.591563] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5135.315443] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 10:17:13 (1787840233)
[ 5155.295180] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787840238/real 1787840238]  req@ffff8fc350843c00 x1874681070945152/t0(0) o400->lustre-MDT0000-mdc-ffff8fc344184800@192.168.206.139@tcp:12/10 lens 224/224 e 0 to 1 dl 1787840254 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5155.325815] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 5162.263966] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 10:17:40 (1787840260)
[ 5190.144163] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 10:18:08 (1787840288)
[ 5190.616067] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5200.728600] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 10:18:18 (1787840298)
[ 5201.070584] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5211.783969] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 10:18:29 (1787840309)
[ 5212.149556] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5221.227337] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 10:18:39 (1787840319)
[ 5221.415981] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5229.161817] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 10:18:47 (1787840327)
[ 5229.494587] Lustre: *** cfs_fail_loc=510, val=3***
[ 5236.890457] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 10:18:54 (1787840334)
[ 5237.278099] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5304.423492] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 10:20:02 (1787840402)
[ 5305.128969] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5364.191246] Lustre: lustre-OST0001-osc-ffff8fc344184800: Connection to lustre-OST0001 (at 192.168.206.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5364.204329] Lustre: Skipped 10 previous similar messages
[ 5369.724989] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 10:21:07 (1787840467)
[ 5370.283678] LustreError: 82980:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5373.045489] LustreError: lustre-MDT0000-mdc-ffff8fc344184800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5373.063707] LustreError: 82994:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8fc344184800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8fc35096f800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5374.351363] LustreError: 82980:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5376.455489] LustreError: 83010:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5379.517845] LustreError: lustre-MDT0000-mdc-ffff8fc344184800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5379.549537] LustreError: 83024:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8fc344184800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8fc350983e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5379.585497] LustreError: 83024:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5380.535144] LustreError: 83010:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5387.456472] LustreError: lustre-MDT0000-mdc-ffff8fc344184800: operation ldlm_enqueue to node 192.168.206.139@tcp failed: rc = -107
[ 5387.460978] LustreError: Skipped 2 previous similar messages
[ 5387.485490] LustreError: lustre-MDT0000-mdc-ffff8fc344184800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5387.895524] LustreError: 83066:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5390.869640] LustreError: lustre-MDT0000-mdc-ffff8fc344184800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5390.973718] LustreError: 83085:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5390.984708] LustreError: 83085:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 16 previous similar messages
[ 5391.984981] LustreError: 83066:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5392.010945] LustreError: 83066:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8fc344184800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5392.066228] LustreError: 83092:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5392.083699] LustreError: 83092:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5394.424765] LustreError: 83114:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5394.438364] LustreError: 83114:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5394.915831] Lustre: lustre-MDT0000-mdc-ffff8fc344184800: Connection restored to 192.168.206.139@tcp (at 192.168.206.139@tcp)
[ 5394.937388] Lustre: Skipped 13 previous similar messages
[ 5398.199141] LustreError: 83141:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5401.199810] LustreError: lustre-MDT0000-mdc-ffff8fc344184800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5401.234128] LustreError: 83155:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8fc344184800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8fc34863c200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5401.275519] LustreError: 83155:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5401.293652] LustreError: 83160:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5401.293661] LustreError: 83160:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5402.263433] LustreError: 83141:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5409.595926] LustreError: lustre-MDT0000-mdc-ffff8fc344184800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5410.086713] LustreError: 83208:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5412.957744] LustreError: lustre-MDT0000-mdc-ffff8fc344184800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5412.975564] LustreError: 83223:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8fc344184800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff8fc35096fb00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5413.007157] LustreError: 83223:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5413.008251] LustreError: 83228:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5413.021123] LustreError: 83228:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 1 previous similar message
[ 5414.176743] LustreError: 83208:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5417.458103] LustreError: lustre-MDT0000-mdc-ffff8fc344184800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5418.447748] LustreError: 83236:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8fc344184800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5426.493645] LustreError: 83307:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5426.504266] LustreError: 83307:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5430.179485] LustreError: lustre-MDT0000-mdc-ffff8fc344184800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5430.281280] LustreError: 83329:0:(file.c:6184:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5430.289318] LustreError: 83329:0:(file.c:6184:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 5430.592723] LustreError: 83307:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5430.600190] LustreError: 83307:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5430.613460] LustreError: 83307:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8fc344184800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5441.325362] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 10:22:19 (1787840539)
[ 5461.565915] LustreError: 79722:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5461.579144] LustreError: 79722:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5465.647117] LustreError: 79722:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5465.655988] LustreError: 79722:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5472.220431] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 10:22:50 (1787840570)
[ 5521.955477] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 10:23:39 (1787840619)
[ 5584.125252] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 10:24:42 (1787840682)
[ 5614.784711] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 10:25:13 (1787840713)
[ 5615.017118] Lustre: Mounted lustre-client - version 2.17.57_97_gdd6e0cb
[ 5737.485383] Lustre: Unmounted lustre-client
[ 5742.287117] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 10:27:20 (1787840840)
[ 5742.762959] LustreError: 87632:0:(osc_request.c:3192:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5744.784460] LustreError: lustre-OST0000-osc-ffff8fc344184800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5744.792504] LustreError: 87714:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff8fc344184800: namespace resource [0x240000400:0x45ed:0x0].0x0 (ffff8fc34863c200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5744.800281] LustreError: 87714:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5745.063568] LustreError: 87632:0:(osc_request.c:3192:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5745.071029] Lustre: 2352:0:(llite_lib.c:4352:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.139@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5751.001312] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 10:27:28 (1787840848)
[ 5808.096332] Lustre: 88298:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787840852/real 1787840852]  req@ffff8fc347cee1c0 x1874681071106688/t0(0) o101->lustre-MDT0000-mdc-ffff8fc344184800@192.168.206.139@tcp:12/10 lens 328/344 e 0 to 1 dl 1787840907 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 5808.125288] Lustre: 88298:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 5814.652172] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 10:28:32 (1787840912)
[ 5816.146184] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5817.987592] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 10:28:35 (1787840915)
[ 5883.530454] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5885.024265] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 10:29:43 (1787840983)
[ 5942.768914] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 10:30:40 (1787841040)
[ 5944.485983] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 5945.885979] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 10:30:44 (1787841044)
[ 5947.312587] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 5949.025455] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 10:30:46 (1787841046)
[ 5980.209303] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 10:31:17 (1787841077)
[ 5992.452773] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6017.951813] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 192.168.206.139@tcp) was lost; in progress operations using this service will fail
[ 6017.971433] LustreError: Skipped 2 previous similar messages
[ 6028.269562] Lustre: Evicted from MGS (at 192.168.206.139@tcp) after server handle changed from 0xab8c53c92522695f to 0xab8c53c9252281e6
[ 6028.286693] Lustre: Skipped 2 previous similar messages
[ 6028.300094] Lustre: MGC192.168.206.139@tcp: Connection restored to 192.168.206.139@tcp (at 192.168.206.139@tcp)
[ 6028.313269] Lustre: Skipped 11 previous similar messages
[ 6030.253190] Lustre: 68231:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.139@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6031.790037] Lustre: lustre-MDT0000-mdc-ffff8fc344184800: Connection to lustre-MDT0000 (at 192.168.206.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6031.806980] Lustre: Skipped 16 previous similar messages
[ 6044.762735] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6046.375285] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6055.370441] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 10:32:33 (1787841153)
[ 6057.361182] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6058.784414] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 10:32:36 (1787841156)
[ 6074.346388] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 192.168.206.139@tcp) was lost; in progress operations using this service will fail
[ 6074.377315] Lustre: Evicted from MGS (at 192.168.206.139@tcp) after server handle changed from 0xab8c53c9252281e6 to 0xab8c53c9252286a1
[ 6083.628713] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 10:33:01 (1787841181)
[ 6124.660467] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 10:33:42 (1787841222)
[ 6132.832975] LustreError: lustre-OST0000-osc-ffff8fc344184800: operation ost_setattr to node 192.168.206.139@tcp failed: rc = -107
[ 6132.857867] LustreError: Skipped 7 previous similar messages
[ 6182.871052] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6185.900058] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6268.895183] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 192.168.206.139@tcp) was lost; in progress operations using this service will fail
[ 6268.904784] LustreError: Skipped 1 previous similar message
[ 6279.149962] Lustre: Evicted from MGS (at 192.168.206.139@tcp) after server handle changed from 0xab8c53c925228bfd to 0xab8c53c9252296fb
[ 6279.175535] Lustre: Skipped 1 previous similar message
[ 6292.825972] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6294.278938] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6329.812765] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6331.568774] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6460.705504] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 10:39:18 (1787841558)
[ 6519.902679] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6523.182272] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6778.750615] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 10:44:36 (1787841876)
[ 6900.201601] Lustre: lustre-MDT0000-mdc-ffff8fc344184800: Connection to lustre-MDT0000 (at 192.168.206.139@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6900.221348] Lustre: Skipped 6 previous similar messages
[ 6915.580536] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 192.168.206.139@tcp) was lost; in progress operations using this service will fail
[ 6915.589408] LustreError: Skipped 1 previous similar message
[ 6915.615675] Lustre: Evicted from MGS (at 192.168.206.139@tcp) after server handle changed from 0xab8c53c925229d84 to 0xab8c53c925323e19
[ 6915.639135] Lustre: Skipped 1 previous similar message
[ 6915.647226] Lustre: MGC192.168.206.139@tcp: Connection restored to 192.168.206.139@tcp (at 192.168.206.139@tcp)
[ 6915.654657] Lustre: Skipped 9 previous similar messages
[ 6952.227172] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 10:47:29 (1787842049)
[ 6953.810995] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 6955.667905] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 10:47:33 (1787842053)
[ 6957.977587] LustreError: lustre-MDT0000-mdc-ffff8fc344184800: operation ldlm_enqueue to node 192.168.206.139@tcp failed: rc = -107
[ 6957.985726] LustreError: Skipped 110 previous similar messages
[ 6958.005643] LustreError: lustre-MDT0000-mdc-ffff8fc344184800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 6965.231432] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 10:47:43 (1787842063)
[ 7132.143832] LustreError: lustre-OST0000-osc-ffff8fc344184800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7133.998285] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 10:50:31 (1787842231)
[ 7159.775192] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787842238/real 1787842238]  req@ffff8fc367dde580 x1874681088817024/t0(0) o4->lustre-OST0000-osc-ffff8fc344184800@192.168.206.139@tcp:6/4 lens 4584/448 e 0 to 1 dl 1787842258 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7159.812603] Lustre: 2350:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 28 previous similar messages
[ 7175.328488] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 10:51:13 (1787842273)
[ 7176.555305] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7177.851602] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 10:51:16 (1787842276)
[ 7179.185752] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7180.746040] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 10:51:18 (1787842278)
[ 7212.200356] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 10:51:50 (1787842310)
[ 7251.949524] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 192.168.206.139@tcp) was lost; in progress operations using this service will fail
[ 7251.970435] Lustre: Evicted from MGS (at 192.168.206.139@tcp) after server handle changed from 0xab8c53c925323e19 to 0xab8c53c925326667
[ 7251.980632] Lustre: Skipped 1 previous similar message
[ 7265.423488] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 10:52:43 (1787842363)
[ 7267.119446] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7268.391796] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 10:52:46 (1787842366)
[ 7270.374404] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7272.134591] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 10:52:49 (1787842369)
[ 7282.093640] Lustre: Unmounted lustre-client
[ 7284.142701] Lustre: Mounted lustre-client - version 2.17.57_97_gdd6e0cb
[ 7287.160133] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7305.695638] Lustre: 2352:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787842388/real 1787842388]  req@ffff8fc36797e580 x1874681089002368/t0(0) o400->MGC192.168.206.139@tcp@192.168.206.139@tcp:26/25 lens 224/224 e 0 to 1 dl 1787842404 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7316.088088] Lustre: 106736:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.139@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 7326.112920] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2