[ 1913.285697] LustreError: lustre-OST0000-osc-ffff995244eab000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1913.296749] Lustre: 2374:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.122@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1918.712848] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 21:19:06 (1786756746)
[ 1920.484612] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1921.839070] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 21:19:10 (1786756750)
[ 1923.451326] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1925.085312] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 21:19:13 (1786756753)
[ 1927.850337] LustreError: lustre-MDT0000-mdc-ffff995244eab000: operation mds_reint to node 192.168.206.122@tcp failed: rc = -19
[ 1927.862975] LustreError: Skipped 2 previous similar messages
[ 1947.626139] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 1947.691403] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0x8fe0c8425c6e1575 to 0x8fe0c8425c6e3286
[ 2046.325490] LustreError: lustre-MDT0000-mdc-ffff995244eab000: operation mds_reint to node 192.168.206.122@tcp failed: rc = -19
[ 2046.334806] LustreError: Skipped 1 previous similar message
[ 2065.392818] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 2065.429510] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0x8fe0c8425c6e3286 to 0x8fe0c8425c706c2d
[ 2076.592446] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 21:21:44 (1786756904)
[ 2077.064544] Lustre: *** cfs_fail_loc=305, val=0***
[ 2077.066973] Lustre: Skipped 2 previous similar messages
[ 2117.087429] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 2127.350182] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0x8fe0c8425c706c2d to 0x8fe0c8425c708fe3
[ 2141.890189] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2143.616202] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2152.019486] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 21:22:59 (1786756979)
[ 2163.191180] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 2163.228845] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0x8fe0c8425c708fe3 to 0x8fe0c8425c709371
[ 2164.681733] LustreError: lustre-MDT0000-mdc-ffff995244eab000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2188.175557] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 21:23:36 (1786757016)
[ 2193.901069] Lustre: lustre-OST0000-osc-ffff995244eab000: Connection to lustre-OST0000 (at 192.168.206.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2193.915508] Lustre: Skipped 12 previous similar messages
[ 2199.745855] LustreError: lustre-OST0000-osc-ffff995244eab000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2199.761558] Lustre: lustre-OST0000-osc-ffff995244eab000: Connection restored to 192.168.206.122@tcp (at 192.168.206.122@tcp)
[ 2199.767473] Lustre: Skipped 16 previous similar messages
[ 2218.151737] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 21:24:05 (1786757045)
[ 2231.626419] LustreError: lustre-MDT0000-mdc-ffff995244eab000: operation ldlm_enqueue to node 192.168.206.122@tcp failed: rc = -19
[ 2251.235306] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 2251.278205] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0x8fe0c8425c709371 to 0x8fe0c8425c70e07f
[ 2252.361215] Lustre: 15949:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.122@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2277.257937] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2279.047098] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2362.847165] Lustre: 2373:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786757176/real 1786757176]  req@ffff99524a8e1880 x1873548334476288/t0(0) o400->MGC192.168.206.122@tcp@192.168.206.122@tcp:26/25 lens 224/224 e 0 to 1 dl 1786757192 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2362.900473] Lustre: 2373:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
[ 2362.919461] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 2373.176981] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0x8fe0c8425c70e07f to 0x8fe0c8425c72bd27
[ 2392.083322] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2395.122649] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2459.575520] LustreError: lustre-MDT0000-mdc-ffff995244eab000: operation mds_reint to node 192.168.206.122@tcp failed: rc = -19
[ 2459.588813] LustreError: Skipped 2 previous similar messages
[ 2480.031162] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 2480.100032] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0x8fe0c8425c72bd27 to 0x8fe0c8425c746fdd
[ 2493.846353] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2495.979630] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2524.159344] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 21:29:12 (1786757352)
[ 2546.667756] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 2546.697862] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0x8fe0c8425c746fdd to 0x8fe0c8425c756da4
[ 2547.949760] Lustre: 15949:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.122@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2563.842500] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2591.316984] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2623.870597] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2659.055941] Lustre: 15949:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.122@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2672.972928] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2713.592538] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 2713.604312] LustreError: Skipped 3 previous similar messages
[ 2713.646182] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0x8fe0c8425c75890a to 0x8fe0c8425c75fae6
[ 2713.658632] Lustre: Skipped 3 previous similar messages
[ 2722.371786] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2749.549954] LustreError: lustre-MDT0000-mdc-ffff995244eab000: operation ldlm_enqueue to node 192.168.206.122@tcp failed: rc = -19
[ 2749.559262] LustreError: Skipped 6 previous similar messages
[ 2781.039764] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2814.286668] Lustre: lustre-MDT0000-mdc-ffff995244eab000: Connection to lustre-MDT0000 (at 192.168.206.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2814.307196] Lustre: Skipped 8 previous similar messages
[ 2841.611287] Lustre: MGC192.168.206.122@tcp: Connection restored to 192.168.206.122@tcp (at 192.168.206.122@tcp)
[ 2841.616777] Lustre: Skipped 17 previous similar messages
[ 2877.991422] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 21:35:05 (1786757705)
[ 2925.074934] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 2926.703210] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3258.193835] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3260.043177] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3555.999600] LustreError: lustre-OST0000-osc-ffff995244eab000: operation ldlm_enqueue to node 192.168.206.122@tcp failed: rc = -19
[ 3556.020876] LustreError: Skipped 4 previous similar messages
[ 3556.026632] Lustre: lustre-OST0000-osc-ffff995244eab000: Connection to lustre-OST0000 (at 192.168.206.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3556.049164] Lustre: Skipped 2 previous similar messages
[ 3581.610819] Lustre: lustre-OST0000-osc-ffff995244eab000: Connection restored to 192.168.206.122@tcp (at 192.168.206.122@tcp)
[ 3581.621789] Lustre: Skipped 3 previous similar messages
[ 3604.395479] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3607.020110] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3848.599368] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 21:51:16 (1786758676)
[ 3866.592150] Lustre: 50618:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786758679/real 1786758679]  req@ffff99524a83c380 x1873548358149760/t0(0) o101->lustre-MDT0000-mdc-ffff995244eab000@192.168.206.122@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786758695 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3866.624418] Lustre: 50618:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 3873.626961] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 21:51:41 (1786758701)
[ 3898.735418] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 21:52:06 (1786758726)
[ 3924.599558] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 21:52:32 (1786758752)
[ 3925.055635] Lustre: Mounted lustre-client - version 2.17.57_1_g2c756e6
[ 3956.703382] Lustre: 2374:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786758769/real 1786758769]  req@ffff99524a83fb80 x1873548358169472/t0(0) o400->lustre-MDT0000-mdc-ffff99524aae2000@192.168.206.122@tcp:12/10 lens 224/224 e 0 to 1 dl 1786758785 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 3956.704468] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 3956.735650] Lustre: 2374:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 3956.751300] LustreError: Skipped 2 previous similar messages
[ 3966.970058] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0x8fe0c8425c77ec79 to 0x8fe0c8425c8e5bb3
[ 3966.989790] Lustre: Skipped 2 previous similar messages
[ 3973.430734] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 3975.082353] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3977.092229] Lustre: Unmounted lustre-client
[ 3983.428544] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 21:53:31 (1786758811)
[ 4121.568950] Lustre: 2373:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786758934/real 1786758934]  req@ffff995250353100 x1873548358202624/t0(0) o4->lustre-OST0000-osc-ffff995244eab000@192.168.206.122@tcp:6/4 lens 488/448 e 0 to 1 dl 1786758950 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4121.611805] Lustre: 2373:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 65 previous similar messages
[ 4169.695610] Lustre: lustre-OST0000-osc-ffff995244eab000: Connection to lustre-OST0000 (at 192.168.206.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4169.723527] Lustre: Skipped 15 previous similar messages
[ 4186.111284] Lustre: lustre-OST0000-osc-ffff995244eab000: Connection restored to 192.168.206.122@tcp (at 192.168.206.122@tcp)
[ 4186.123627] Lustre: Skipped 17 previous similar messages
[ 4280.558678] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 21:58:28 (1786759108)
[ 4329.962464] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 21:59:17 (1786759157)
[ 4333.000555] Lustre: Unmounted lustre-client
[ 4354.258875] Lustre: Mounted lustre-client - version 2.17.57_1_g2c756e6
[ 4360.992930] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 21:59:49 (1786759189)
[ 4361.193780] LustreError: 56544:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4362.239254] LustreError: 56544:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4362.448765] Lustre: *** cfs_fail_loc=305, val=0***
[ 4385.350916] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 22:00:13 (1786759213)
[ 4385.820029] Lustre: Mounted lustre-client - version 2.17.57_1_g2c756e6
[ 4388.428955] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4398.754566] Lustre: Unmounted lustre-client
[ 4405.549464] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 22:00:33 (1786759233)
[ 4483.814501] LustreError: lustre-MDT0000-mdc-ffff995245698800: operation mds_reint to node 192.168.206.122@tcp failed: rc = -19
[ 4504.549868] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 4504.564581] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0x8fe0c8425c8e742c to 0x8fe0c8425c912f14
[ 4664.649551] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 22:04:52 (1786759492)
[ 4669.589296] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4683.751803] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 4683.779329] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0x8fe0c8425c912f14 to 0x8fe0c8425c95efcb
[ 4683.864878] LustreError: lustre-MDT0000-mdc-ffff995245698800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4697.625885] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 22:05:25 (1786759525)
[ 4698.139290] Lustre: Mounted lustre-client - version 2.17.57_1_g2c756e6
[ 4705.445046] LustreError: lustre-OST0000-osc-ffff995245698800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4715.488577] Lustre: 2373:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786759528/real 1786759528]  req@ffff99525054ce00 x1873548362855040/t0(0) o103->lustre-OST0000-osc-ffff995242d9b000@192.168.206.122@tcp:17/18 lens 328/224 e 0 to 1 dl 1786759544 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4715.514865] Lustre: 2373:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 70 previous similar messages
[ 4718.118157] Lustre: Unmounted lustre-client
[ 4725.704901] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 22:05:53 (1786759553)
[ 4732.874164] LustreError: lustre-MDT0000-mdc-ffff995245698800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4732.885459] LustreError: 60945:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4741.922210] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 22:06:09 (1786759569)
[ 4742.091261] LustreError: 61645:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4746.199973] LustreError: lustre-MDT0000-mdc-ffff995245698800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4746.224531] LustreError: 61666:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4746.243282] LustreError: 61645:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=867
[ 4746.258978] LustreError: 61645:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting
[ 4746.278416] LustreError: 61671:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4747.392831] LustreError: 61677:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4747.402188] LustreError: 61677:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4749.843521] LustreError: 61699:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4749.864886] LustreError: 61699:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4754.767288] LustreError: 61744:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4754.791502] LustreError: 61744:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4767.698963] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 22:06:34 (1786759594)
[ 4776.936110] Lustre: lustre-OST0000-osc-ffff995245698800: Connection to lustre-OST0000 (at 192.168.206.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4776.972719] Lustre: Skipped 17 previous similar messages
[ 4814.173650] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4815.991120] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4827.998742] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 22:07:35 (1786759655)
[ 4852.846237] Lustre: lustre-OST0000-osc-ffff995245698800: Connection restored to 192.168.206.122@tcp (at 192.168.206.122@tcp)
[ 4852.855803] Lustre: Skipped 17 previous similar messages
[ 4870.502044] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4872.447422] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4883.103630] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 22:08:31 (1786759711)
[ 4946.701404] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4947.972283] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4955.813332] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 22:09:43 (1786759783)
[ 5003.354518] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5005.100802] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5014.361662] Lustre: Unmounted lustre-client
[ 5074.522795] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5077.142915] Lustre: Mounted lustre-client - version 2.17.57_1_g2c756e6
[ 5083.921691] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 22:11:51 (1786759911)
[ 5086.761105] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5088.676606] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 22:11:56 (1786759916)
[ 5115.177848] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 22:12:23 (1786759943)
[ 5116.533306] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5118.280749] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 22:12:26 (1786759946)
[ 5119.451260] Lustre: *** cfs_fail_loc=805, val=0***
[ 5119.493261] Lustre: Mounted lustre-client - version 2.17.57_1_g2c756e6
[ 5124.658498] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5145.055255] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 5145.073725] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0x8fe0c8425c95f9bf to 0x8fe0c8425c95fccf
[ 5154.357322] Lustre: Unmounted lustre-client
[ 5161.573142] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 22:13:09 (1786759989)
[ 5204.436175] LustreError: 2371:0:(client.c:3435:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff99525371b800 x1873548362939776/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff9952492c9000@192.168.206.122@tcp:12/10 lens 664/608 e 0 to 0 dl 1786760049 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5214.192376] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5216.003348] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5224.034837] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 22:14:12 (1786760052)
[ 5225.597647] LustreError: lustre-OST0000-osc-ffff9952492c9000: operation ost_write to node 192.168.206.122@tcp failed: rc = -107
[ 5225.610204] LustreError: Skipped 3 previous similar messages
[ 5225.656768] LustreError: lustre-OST0000-osc-ffff9952492c9000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5225.671347] Lustre: 2373:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.122@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5225.733966] LustreError: 72850:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9952492c9000: namespace resource [0x240000400:0x42a2:0x0].0x0 (ffff99524703af00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5233.761710] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 22:14:21 (1786760061)
[ 5235.084961] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5236.676349] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 22:14:24 (1786760064)
[ 5238.254513] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5239.673499] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 22:14:27 (1786760067)
[ 5241.486054] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5243.089971] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 22:14:31 (1786760071)
[ 5244.362151] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5245.966486] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 22:14:33 (1786760073)
[ 5247.507111] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5249.032741] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 22:14:37 (1786760077)
[ 5250.862986] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5252.470919] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 22:14:40 (1786760080)
[ 5254.071575] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5255.845345] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 22:14:43 (1786760083)
[ 5257.572655] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5259.559334] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 22:14:47 (1786760087)
[ 5261.169355] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5262.965748] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 22:14:50 (1786760090)
[ 5264.450578] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5265.766505] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 22:14:53 (1786760093)
[ 5267.458806] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5269.270497] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 22:14:56 (1786760096)
[ 5271.631992] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5273.300609] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 22:15:01 (1786760101)
[ 5306.211112] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 22:15:34 (1786760134)
[ 5336.837629] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 22:16:04 (1786760164)
[ 5337.755814] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5347.701198] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 22:16:15 (1786760175)
[ 5348.071635] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5359.001949] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 22:16:26 (1786760186)
[ 5359.481938] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5368.358427] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 22:16:36 (1786760196)
[ 5368.545322] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5376.604837] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 22:16:44 (1786760204)
[ 5376.913690] Lustre: *** cfs_fail_loc=510, val=3***
[ 5384.620952] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 22:16:52 (1786760212)
[ 5384.902793] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5394.497973] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 22:17:02 (1786760222)
[ 5394.822829] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5454.239741] Lustre: 2374:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786760228/real 1786760228]  req@ffff995244089f80 x1873548362985600/t0(0) o3->lustre-OST0001-osc-ffff9952492c9000@192.168.206.122@tcp:6/4 lens 488/4536 e 0 to 1 dl 1786760283 ref 2 fl Rpc:RXQ/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 5454.276644] Lustre: 2374:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[ 5454.283731] Lustre: lustre-OST0001-osc-ffff9952492c9000: Connection to lustre-OST0001 (at 192.168.206.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5454.306696] Lustre: Skipped 9 previous similar messages
[ 5461.316681] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 22:18:09 (1786760289)
[ 5461.648226] LustreError: 83055:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5464.512608] LustreError: lustre-MDT0000-mdc-ffff9952492c9000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5464.536180] LustreError: 83070:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9952492c9000: inode [0x20000a042:0xf:0x0] mdc close failed: rc = -108
[ 5464.555269] LustreError: 83070:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9952492c9000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff995244179800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5464.588475] Lustre: lustre-MDT0000-mdc-ffff9952492c9000: Connection restored to 192.168.206.122@tcp (at 192.168.206.122@tcp)
[ 5464.598409] Lustre: Skipped 10 previous similar messages
[ 5465.727137] LustreError: 83055:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5467.842353] LustreError: 83085:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5470.628293] LustreError: lustre-MDT0000-mdc-ffff9952492c9000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5470.651464] LustreError: 83100:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9952492c9000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff995244283e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5470.695375] LustreError: 83100:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5470.708322] LustreError: 83105:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5470.720145] LustreError: 83105:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5471.911126] LustreError: 83085:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5478.644282] LustreError: lustre-MDT0000-mdc-ffff9952492c9000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5478.658073] LustreError: 83128:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9952492c9000: namespace resource [0x200000007:0x1:0x0].0x0 (ffff995244179200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5478.680943] LustreError: 83128:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5479.060879] LustreError: 83141:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5481.859692] LustreError: lustre-MDT0000-mdc-ffff9952492c9000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5481.944890] LustreError: 83160:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5483.127402] LustreError: 83141:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5483.137140] LustreError: 83141:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9952492c9000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5484.194047] LustreError: 83178:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5484.204824] LustreError: 83178:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 12 previous similar messages
[ 5488.907839] LustreError: 83215:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5491.749987] LustreError: lustre-MDT0000-mdc-ffff9952492c9000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5491.783125] LustreError: 83230:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9952492c9000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff99524703a700) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5492.975307] LustreError: 83215:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5499.893962] LustreError: lustre-MDT0000-mdc-ffff9952492c9000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5500.259955] LustreError: 83271:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5503.294546] LustreError: lustre-MDT0000-mdc-ffff9952492c9000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5503.330225] LustreError: 83286:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9952492c9000: namespace resource [0x20000a042:0x11:0x0].0xc (ffff995244283b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5503.364234] LustreError: 83286:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5504.343191] LustreError: 83271:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5507.388207] LustreError: lustre-MDT0000-mdc-ffff9952492c9000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5507.481122] LustreError: 83318:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5507.495925] LustreError: 83318:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5508.539494] LustreError: 83299:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9952492c9000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5516.630227] LustreError: 83370:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5516.638847] LustreError: 83370:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5520.198166] LustreError: lustre-MDT0000-mdc-ffff9952492c9000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5520.319588] LustreError: 83392:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5520.325200] LustreError: 83392:0:(file.c:6156:ll_inode_revalidate_fini()) Skipped 26 previous similar messages
[ 5520.720038] LustreError: 83370:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5520.731218] LustreError: 83370:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5520.740700] LustreError: 83370:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9952492c9000: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5531.121711] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 22:19:18 (1786760358)
[ 5552.291289] LustreError: 68310:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5552.309813] LustreError: 68310:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5556.391123] LustreError: 68310:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5556.407169] LustreError: 68310:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5565.715275] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 22:19:52 (1786760392)
[ 5615.949988] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 22:20:44 (1786760444)
[ 5681.359761] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 22:21:48 (1786760508)
[ 5714.723800] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 22:22:22 (1786760542)
[ 5715.140666] Lustre: Mounted lustre-client - version 2.17.57_1_g2c756e6
[ 5839.324842] Lustre: Unmounted lustre-client
[ 5846.040799] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 22:24:33 (1786760673)
[ 5846.851446] LustreError: 87699:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5849.564371] LustreError: lustre-OST0000-osc-ffff9952492c9000: operation ldlm_enqueue to node 192.168.206.122@tcp failed: rc = -107
[ 5849.573714] LustreError: Skipped 9 previous similar messages
[ 5849.590503] LustreError: lustre-OST0000-osc-ffff9952492c9000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5849.618372] LustreError: 87781:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9952492c9000: namespace resource [0x240000400:0x42cd:0x0].0x0 (ffff99524703ad00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5849.645029] LustreError: 87781:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5850.007122] LustreError: 87699:0:(osc_request.c:2989:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5850.016368] Lustre: 2372:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.206.122@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5857.203479] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 22:24:45 (1786760685)
[ 5923.942021] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 22:25:51 (1786760751)
[ 5925.867657] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 5928.177903] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 22:25:55 (1786760755)
[ 5993.468949] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 5995.582797] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 22:27:03 (1786760823)
[ 6054.760986] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 22:28:02 (1786760882)
[ 6057.890706] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6059.563768] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 22:28:07 (1786760887)
[ 6061.303798] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6063.091904] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 22:28:10 (1786760890)
[ 6099.686787] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 22:28:47 (1786760927)
[ 6111.788511] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6139.872193] Lustre: 2372:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786760953/real 1786760953]  req@ffff99525054c000 x1873548363148160/t0(0) o400->MGC192.168.206.122@tcp@192.168.206.122@tcp:26/25 lens 224/224 e 0 to 1 dl 1786760969 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6139.928864] Lustre: 2372:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 6139.939611] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 6139.973873] LustreError: Skipped 2 previous similar messages
[ 6146.015422] Lustre: lustre-MDT0000-mdc-ffff9952492c9000: Connection to lustre-MDT0000 (at 192.168.206.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6146.041738] Lustre: Skipped 16 previous similar messages
[ 6150.198292] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0x8fe0c8425c960828 to 0x8fe0c8425c962023
[ 6150.210558] Lustre: Skipped 2 previous similar messages
[ 6150.220352] Lustre: MGC192.168.206.122@tcp: Connection restored to 192.168.206.122@tcp (at 192.168.206.122@tcp)
[ 6150.236659] Lustre: Skipped 14 previous similar messages
[ 6166.159186] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6167.591016] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6177.535548] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 22:30:05 (1786761005)
[ 6179.875271] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6181.379749] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 22:30:09 (1786761009)
[ 6197.765455] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 6197.816850] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0x8fe0c8425c962023 to 0x8fe0c8425c9624c9
[ 6211.698634] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 22:30:38 (1786761038)
[ 6261.165793] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 22:31:29 (1786761089)
[ 6319.846804] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6323.400486] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6407.647310] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 6407.654016] LustreError: Skipped 1 previous similar message
[ 6416.950641] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0x8fe0c8425c962850 to 0x8fe0c8425c963809
[ 6416.963559] Lustre: Skipped 1 previous similar message
[ 6422.699238] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6424.265537] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6461.177360] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6462.806826] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6594.789360] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 22:37:02 (1786761422)
[ 6604.163960] LustreError: lustre-OST0000-osc-ffff9952492c9000: operation ost_setattr to node 192.168.206.122@tcp failed: rc = -19
[ 6604.172897] LustreError: Skipped 19 previous similar messages
[ 6647.222109] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6649.965724] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6874.949847] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 22:41:43 (1786761703)
[ 7036.396858] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 7036.408071] Lustre: lustre-MDT0000-mdc-ffff9952492c9000: Connection to lustre-MDT0000 (at 192.168.206.122@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7036.408978] LustreError: Skipped 1 previous similar message
[ 7036.414444] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0x8fe0c8425c96394b to 0x8fe0c8425ca5524e
[ 7036.433527] Lustre: Skipped 6 previous similar messages
[ 7036.452789] Lustre: Skipped 1 previous similar message
[ 7036.476414] Lustre: MGC192.168.206.122@tcp: Connection restored to 192.168.206.122@tcp (at 192.168.206.122@tcp)
[ 7036.487707] Lustre: Skipped 9 previous similar messages
[ 7067.115729] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 22:44:54 (1786761894)
[ 7069.332258] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7071.635956] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 22:44:59 (1786761899)
[ 7074.559190] LustreError: lustre-MDT0000-mdc-ffff9952492c9000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7083.868873] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 22:45:11 (1786761911)
[ 7250.921511] LustreError: lustre-OST0000-osc-ffff9952492c9000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7252.614169] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 22:48:00 (1786762080)
[ 7277.471147] Lustre: 2375:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786762086/real 1786762086]  req@ffff995267808700 x1873548380174848/t0(0) o4->lustre-OST0000-osc-ffff9952492c9000@192.168.206.122@tcp:6/4 lens 4584/448 e 0 to 1 dl 1786762106 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7277.519546] Lustre: 2375:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 22 previous similar messages
[ 7293.765502] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 22:48:41 (1786762121)
[ 7295.020480] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7297.082488] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 22:48:44 (1786762124)
[ 7298.765494] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7300.578935] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 22:48:48 (1786762128)
[ 7334.345881] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 22:49:22 (1786762162)
[ 7374.836217] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 192.168.206.122@tcp) was lost; in progress operations using this service will fail
[ 7374.853707] Lustre: Evicted from MGS (at 192.168.206.122@tcp) after server handle changed from 0x8fe0c8425ca5524e to 0x8fe0c8425ca57e5b
[ 7374.861896] Lustre: Skipped 1 previous similar message
[ 7386.547394] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 22:50:14 (1786762214)
[ 7387.915302] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7389.647333] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 22:50:17 (1786762217)
[ 7391.334360] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7393.118626] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 22:50:20 (1786762220)
[ 7406.539393] Lustre: Unmounted lustre-client
[ 7408.789037] Lustre: Mounted lustre-client - version 2.17.57_1_g2c756e6
[ 7411.435468] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7438.906833] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2