[ 1971.240163] LustreError: lustre-OST0000-osc-ffff9e5b52866000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1971.258936] Lustre: 2356:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.125@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108)
[ 1979.864878] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 18:41:47 (1787784107)
[ 1982.746919] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1985.488062] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 18:41:52 (1787784112)
[ 1987.267811] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1988.937126] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 18:41:56 (1787784116)
[ 1993.166094] LustreError: lustre-MDT0000-mdc-ffff9e5b52866000: operation mds_reint to node 192.168.201.125@tcp failed: rc = -19
[ 1993.181221] LustreError: Skipped 2 previous similar messages
[ 2012.639500] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 192.168.201.125@tcp) was lost; in progress operations using this service will fail
[ 2022.822195] Lustre: Evicted from MGS (at 192.168.201.125@tcp) after server handle changed from 0xf66ec02754a42616 to 0xf66ec02754a43cc8
[ 2124.613322] LustreError: lustre-MDT0000-mdc-ffff9e5b52866000: operation mds_reint to node 192.168.201.125@tcp failed: rc = -19
[ 2144.735317] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 192.168.201.125@tcp) was lost; in progress operations using this service will fail
[ 2144.789704] Lustre: Evicted from MGS (at 192.168.201.125@tcp) after server handle changed from 0xf66ec02754a43cc8 to 0xf66ec02754a65eba
[ 2171.532906] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 18:44:59 (1787784299)
[ 2172.250267] Lustre: *** cfs_fail_loc=305, val=0***
[ 2172.254076] Lustre: Skipped 4 previous similar messages
[ 2191.653706] LustreError: lustre-OST0001-osc-ffff9e5b52866000: operation ost_connect to node 192.168.201.125@tcp failed: rc = -75
[ 2218.463247] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 192.168.201.125@tcp) was lost; in progress operations using this service will fail
[ 2218.507613] Lustre: Evicted from MGS (at 192.168.201.125@tcp) after server handle changed from 0xf66ec02754a65eba to 0xf66ec02754a6636e
[ 2234.863933] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2236.155674] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2244.736911] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 18:46:12 (1787784372)
[ 2258.415654] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 192.168.201.125@tcp) was lost; in progress operations using this service will fail
[ 2258.440795] Lustre: Evicted from MGS (at 192.168.201.125@tcp) after server handle changed from 0xf66ec02754a6636e to 0xf66ec02754a66a3d
[ 2258.472333] LustreError: lustre-MDT0000-mdc-ffff9e5b52866000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2281.502052] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 18:46:49 (1787784409)
[ 2294.268848] LustreError: lustre-OST0000-osc-ffff9e5b52866000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2312.299690] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 18:47:20 (1787784440)
[ 2326.492979] LustreError: lustre-MDT0000-mdc-ffff9e5b52866000: operation mds_close to node 192.168.201.125@tcp failed: rc = -19
[ 2346.463253] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 192.168.201.125@tcp) was lost; in progress operations using this service will fail
[ 2346.495808] Lustre: Evicted from MGS (at 192.168.201.125@tcp) after server handle changed from 0xf66ec02754a66a3d to 0xf66ec02754a6bf63
[ 2348.092403] Lustre: 15936:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.125@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2376.097586] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2378.534148] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2463.263298] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787784576/real 1787784576]  req@ffff9e5b58379e00 x1874625538228608/t0(0) o400->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1787784592 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2463.305911] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 2463.311598] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 192.168.201.125@tcp) was lost; in progress operations using this service will fail
[ 2473.527761] Lustre: Evicted from MGS (at 192.168.201.125@tcp) after server handle changed from 0xf66ec02754a6bf63 to 0xf66ec02754a8406a
[ 2491.986745] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2494.995772] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2579.935519] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 192.168.201.125@tcp) was lost; in progress operations using this service will fail
[ 2590.214285] Lustre: Evicted from MGS (at 192.168.201.125@tcp) after server handle changed from 0xf66ec02754a8406a to 0xf66ec02754a9c990
[ 2590.248444] Lustre: MGC192.168.201.125@tcp: Connection restored to 192.168.201.125@tcp (at 192.168.201.125@tcp)
[ 2590.263681] Lustre: Skipped 15 previous similar messages
[ 2613.962734] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 2616.854178] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2645.878769] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 18:52:53 (1787784773)
[ 2650.296156] LustreError: lustre-MDT0000-mdc-ffff9e5b52866000: operation mds_reint to node 192.168.201.125@tcp failed: rc = -19
[ 2650.312041] LustreError: Skipped 4 previous similar messages
[ 2650.313751] Lustre: lustre-MDT0000-mdc-ffff9e5b52866000: Connection to lustre-MDT0000 (at 192.168.201.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2650.337944] Lustre: Skipped 9 previous similar messages
[ 2667.873821] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 192.168.201.125@tcp) was lost; in progress operations using this service will fail
[ 2677.047439] Lustre: Evicted from MGS (at 192.168.201.125@tcp) after server handle changed from 0xf66ec02754a9c990 to 0xf66ec02754aa950e
[ 2678.276555] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2709.941481] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2757.711486] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2798.765372] Lustre: 15936:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.125@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2811.163390] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2853.343860] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 192.168.201.125@tcp) was lost; in progress operations using this service will fail
[ 2853.353436] LustreError: Skipped 3 previous similar messages
[ 2863.531890] Lustre: Evicted from MGS (at 192.168.201.125@tcp) after server handle changed from 0xf66ec02754ab0e14 to 0xf66ec02754abbd22
[ 2863.539389] Lustre: Skipped 3 previous similar messages
[ 2864.186371] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2919.931513] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 3011.160710] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 18:58:58 (1787785138)
[ 3068.476886] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3070.822353] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3357.412623] LustreError: lustre-OST0000-osc-ffff9e5b52866000: operation ldlm_enqueue to node 192.168.201.125@tcp failed: rc = -19
[ 3357.424685] LustreError: Skipped 8 previous similar messages
[ 3357.430698] Lustre: lustre-OST0000-osc-ffff9e5b52866000: Connection to lustre-OST0000 (at 192.168.201.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3357.448138] Lustre: Skipped 6 previous similar messages
[ 3381.597755] Lustre: lustre-OST0000-osc-ffff9e5b52866000: Connection restored to 192.168.201.125@tcp (at 192.168.201.125@tcp)
[ 3381.610361] Lustre: Skipped 15 previous similar messages
[ 3402.946891] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3405.359612] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3733.207596] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 3735.353338] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3985.537571] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 19:15:13 (1787786113)
[ 4002.271190] Lustre: 50591:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787786115/real 1787786115]  req@ffff9e5b4344de00 x1874625560203008/t0(0) o101->lustre-MDT0000-mdc-ffff9e5b52866000@192.168.201.125@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787786131 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 4002.301808] Lustre: 50591:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[ 4002.307809] Lustre: lustre-MDT0000-mdc-ffff9e5b52866000: Connection to lustre-MDT0000 (at 192.168.201.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4002.318167] Lustre: Skipped 1 previous similar message
[ 4002.366459] Lustre: lustre-MDT0000-mdc-ffff9e5b52866000: Connection restored to 192.168.201.125@tcp (at 192.168.201.125@tcp)
[ 4002.372151] Lustre: Skipped 1 previous similar message
[ 4011.445334] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 19:15:39 (1787786139)
[ 4040.218820] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 19:16:06 (1787786166)
[ 4070.387838] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 19:16:37 (1787786197)
[ 4071.098603] Lustre: Mounted lustre-client - version 2.17.57_55_ge1b14e7
[ 4102.114296] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787786215/real 1787786215]  req@ffff9e5b4344e1c0 x1874625560222208/t0(0) o400->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1787786231 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4102.165626] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 4102.183487] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 192.168.201.125@tcp) was lost; in progress operations using this service will fail
[ 4102.211046] LustreError: Skipped 2 previous similar messages
[ 4112.412909] Lustre: Evicted from MGS (at 192.168.201.125@tcp) after server handle changed from 0xf66ec02754ad06ae to 0xf66ec02754c20f60
[ 4112.441592] Lustre: Skipped 2 previous similar messages
[ 4128.002347] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 4129.965925] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4131.982721] Lustre: Unmounted lustre-client
[ 4138.721846] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 19:17:46 (1787786266)
[ 4260.833857] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787786373/real 1787786373]  req@ffff9e5b42d8c3c0 x1874625560256128/t0(0) o4->lustre-OST0000-osc-ffff9e5b52866000@192.168.201.125@tcp:6/4 lens 488/448 e 0 to 1 dl 1787786389 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4260.903658] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 68 previous similar messages
[ 4434.031147] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 19:22:40 (1787786560)
[ 4485.020081] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 19:23:32 (1787786612)
[ 4487.939755] Lustre: Unmounted lustre-client
[ 4509.667214] Lustre: Mounted lustre-client - version 2.17.57_55_ge1b14e7
[ 4517.022234] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 19:24:04 (1787786644)
[ 4517.563457] LustreError: 56527:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4518.614906] LustreError: 56527:0:(mdc_locks.c:1337:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4518.851842] Lustre: *** cfs_fail_loc=305, val=0***
[ 4542.487650] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 19:24:30 (1787786670)
[ 4543.096253] Lustre: Mounted lustre-client - version 2.17.57_55_ge1b14e7
[ 4546.805940] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4557.203127] Lustre: Unmounted lustre-client
[ 4564.488709] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 19:24:52 (1787786692)
[ 4646.203115] LustreError: lustre-MDT0000-mdc-ffff9e5b51b64800: operation mds_reint to node 192.168.201.125@tcp failed: rc = -19
[ 4646.207500] LustreError: Skipped 1 previous similar message
[ 4646.211745] Lustre: lustre-MDT0000-mdc-ffff9e5b51b64800: Connection to lustre-MDT0000 (at 192.168.201.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4646.229879] Lustre: Skipped 23 previous similar messages
[ 4666.343781] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 192.168.201.125@tcp) was lost; in progress operations using this service will fail
[ 4666.361451] Lustre: Evicted from MGS (at 192.168.201.125@tcp) after server handle changed from 0xf66ec02754c22a17 to 0xf66ec02754c4dc62
[ 4666.378131] Lustre: MGC192.168.201.125@tcp: Connection restored to 192.168.201.125@tcp (at 192.168.201.125@tcp)
[ 4666.385027] Lustre: Skipped 24 previous similar messages
[ 4667.427860] Lustre: 55925:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.125@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4835.622614] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 19:29:23 (1787786963)
[ 4840.728645] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4855.795363] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 192.168.201.125@tcp) was lost; in progress operations using this service will fail
[ 4855.814364] Lustre: Evicted from MGS (at 192.168.201.125@tcp) after server handle changed from 0xf66ec02754c4dc62 to 0xf66ec02754c9aa71
[ 4855.901667] LustreError: lustre-MDT0000-mdc-ffff9e5b51b64800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4874.080143] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 19:30:01 (1787787001)
[ 4874.736736] Lustre: Mounted lustre-client - version 2.17.57_55_ge1b14e7
[ 4882.306736] LustreError: lustre-OST0000-osc-ffff9e5b51b64800: operation ldlm_enqueue to node 192.168.201.125@tcp failed: rc = -107
[ 4882.347206] LustreError: lustre-OST0000-osc-ffff9e5b51b64800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4892.639284] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787787005/real 1787787005]  req@ffff9e5b6109c3c0 x1874625564906240/t0(0) o103->lustre-OST0000-osc-ffff9e5b45938000@192.168.201.125@tcp:17/18 lens 328/224 e 0 to 1 dl 1787787021 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4892.673320] Lustre: 2356:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 82 previous similar messages
[ 4895.696543] Lustre: Unmounted lustre-client
[ 4903.917980] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 19:30:31 (1787787031)
[ 4910.777351] LustreError: lustre-MDT0000-mdc-ffff9e5b51b64800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4910.831418] LustreError: 60921:0:(file.c:6176:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4918.398533] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 19:30:46 (1787787046)
[ 4918.734438] LustreError: 61626:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4923.871153] LustreError: 61626:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4923.873781] LustreError: 61626:0:(import.c:711:ptlrpc_connect_import_locked()) already connected
[ 4925.413971] LustreError: lustre-MDT0000-mdc-ffff9e5b51b64800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4925.464414] LustreError: 61643:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4925.566386] LustreError: 61649:0:(file.c:6176:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4926.688103] LustreError: 61655:0:(file.c:6176:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4926.697582] LustreError: 61655:0:(file.c:6176:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4929.108337] LustreError: 61678:0:(file.c:6176:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4929.132632] LustreError: 61678:0:(file.c:6176:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4933.861585] LustreError: 61723:0:(file.c:6176:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4933.866477] LustreError: 61723:0:(file.c:6176:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4944.628357] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 19:31:12 (1787787072)
[ 4987.941359] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 4989.505209] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4999.887108] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 19:32:07 (1787787127)
[ 5041.968578] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5044.206368] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5055.395837] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 19:33:03 (1787787183)
[ 5127.045955] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5129.570180] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5138.998579] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 19:34:26 (1787787266)
[ 5180.608627] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5181.958785] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5188.549978] Lustre: Unmounted lustre-client
[ 5245.414659] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 5247.868664] Lustre: Mounted lustre-client - version 2.17.57_55_ge1b14e7
[ 5255.549365] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 19:36:22 (1787787382)
[ 5258.605582] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5260.934370] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 19:36:28 (1787787388)
[ 5297.104238] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 19:37:04 (1787787424)
[ 5298.689887] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5301.522814] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 19:37:08 (1787787428)
[ 5303.394333] Lustre: *** cfs_fail_loc=805, val=0***
[ 5303.458380] Lustre: Mounted lustre-client - version 2.17.57_55_ge1b14e7
[ 5310.908865] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5330.399935] Lustre: lustre-MDT0000-mdc-ffff9e5b4593d800: Connection to lustre-MDT0000 (at 192.168.201.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5330.417035] Lustre: Skipped 12 previous similar messages
[ 5330.424388] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 192.168.201.125@tcp) was lost; in progress operations using this service will fail
[ 5340.655648] Lustre: Evicted from MGS (at 192.168.201.125@tcp) after server handle changed from 0xf66ec02754c9b46c to 0xf66ec02754c9b687
[ 5340.672820] Lustre: MGC192.168.201.125@tcp: Connection restored to 192.168.201.125@tcp (at 192.168.201.125@tcp)
[ 5340.677142] Lustre: Skipped 12 previous similar messages
[ 5342.206117] Lustre: 68292:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.125@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 5354.997899] LustreError: 2352:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9e5b50d21800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9e5b4789bb00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5358.226737] Lustre: Unmounted lustre-client
[ 5367.708832] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 19:38:15 (1787787495)
[ 5410.156207] LustreError: 2352:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff9e5b6109c3c0 x1874625564993408/t90194313220(90194313220) o101->lustre-MDT0000-mdc-ffff9e5b4593d800@192.168.201.125@tcp:12/10 lens 664/608 e 0 to 0 dl 1787787555 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5416.477780] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5417.880397] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5428.461328] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 19:39:15 (1787787555)
[ 5430.342350] LustreError: lustre-OST0000-osc-ffff9e5b4593d800: operation ost_write to node 192.168.201.125@tcp failed: rc = -107
[ 5430.354647] LustreError: Skipped 2 previous similar messages
[ 5430.379767] LustreError: lustre-OST0000-osc-ffff9e5b4593d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5430.402876] Lustre: 2353:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.125@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5430.452780] LustreError: 72835:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9e5b4593d800: namespace resource [0x240000400:0x3e82:0x0].0x0 (ffff9e5b46bbff00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5430.467922] LustreError: 72835:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5440.954269] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 19:39:28 (1787787568)
[ 5443.410794] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5445.436530] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 19:39:32 (1787787572)
[ 5446.985338] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5448.891132] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 19:39:36 (1787787576)
[ 5450.418801] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5452.457821] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 19:39:39 (1787787579)
[ 5454.246295] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5455.955254] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 19:39:43 (1787787583)
[ 5457.867694] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5459.679901] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 19:39:47 (1787787587)
[ 5461.942963] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5464.204239] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 19:39:51 (1787787591)
[ 5465.921807] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5468.664920] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 19:39:55 (1787787595)
[ 5470.461597] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5472.503938] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 19:40:00 (1787787600)
[ 5474.410739] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5476.524994] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 19:40:03 (1787787603)
[ 5478.562842] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5480.823841] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 19:40:08 (1787787608)
[ 5483.425351] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5485.767514] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 19:40:12 (1787787612)
[ 5488.738126] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5490.616811] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 19:40:18 (1787787618)
[ 5513.695169] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787787626/real 1787787626]  req@ffff9e5b448f9a40 x1874625565009792/t0(0) o400->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1787787642 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5513.735553] Lustre: 2354:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
[ 5530.530751] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 19:40:58 (1787787658)
[ 5561.428611] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 19:41:28 (1787787688)
[ 5562.028481] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5573.888262] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 19:41:41 (1787787701)
[ 5574.205975] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5586.587614] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 19:41:53 (1787787713)
[ 5587.116060] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5597.372926] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 19:42:04 (1787787724)
[ 5597.685254] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5607.194751] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 19:42:14 (1787787734)
[ 5607.562304] Lustre: *** cfs_fail_loc=510, val=3***
[ 5615.548339] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 19:42:23 (1787787743)
[ 5616.347683] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5627.392734] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 19:42:34 (1787787754)
[ 5628.053945] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5695.784502] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 19:43:43 (1787787823)
[ 5696.584994] LustreError: 83045:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5699.860802] LustreError: lustre-MDT0000-mdc-ffff9e5b4593d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5699.877912] LustreError: 83059:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e5b4593d800: inode [0x20000a042:0x8:0x0] mdc close failed: rc = -108
[ 5699.919724] LustreError: 83059:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9e5b4593d800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9e5b46b2b400) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5700.663225] LustreError: 83045:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5702.839205] LustreError: 83075:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5705.908154] LustreError: lustre-MDT0000-mdc-ffff9e5b4593d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5705.942180] LustreError: 83089:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9e5b4593d800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9e5b43752b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5705.985080] LustreError: 83089:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5706.927272] LustreError: 83075:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5714.358092] LustreError: lustre-MDT0000-mdc-ffff9e5b4593d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5714.379368] LustreError: 83118:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9e5b4593d800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9e5b46bbf200) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5714.409977] LustreError: 83118:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5714.850251] LustreError: 83132:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5717.888340] LustreError: lustre-MDT0000-mdc-ffff9e5b4593d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5717.995299] LustreError: 83151:0:(file.c:6176:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5718.011233] LustreError: 83151:0:(file.c:6176:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5718.927142] LustreError: 83132:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5718.942351] LustreError: 83132:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e5b4593d800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5719.111534] LustreError: 83157:0:(file.c:6176:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5719.141983] LustreError: 83157:0:(file.c:6176:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5721.642702] LustreError: 83180:0:(file.c:6176:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5721.657169] LustreError: 83180:0:(file.c:6176:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5725.639438] LustreError: 83206:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5729.655463] LustreError: 83206:0:(ldlm_flock.c:808:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5730.163785] LustreError: lustre-MDT0000-mdc-ffff9e5b4593d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5736.920598] LustreError: lustre-MDT0000-mdc-ffff9e5b4593d800: operation ldlm_enqueue to node 192.168.201.125@tcp failed: rc = -107
[ 5736.936385] LustreError: Skipped 5 previous similar messages
[ 5736.958407] LustreError: lustre-MDT0000-mdc-ffff9e5b4593d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5737.547761] LustreError: 83262:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5741.074736] LustreError: lustre-MDT0000-mdc-ffff9e5b4593d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5741.098661] LustreError: 83277:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9e5b4593d800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9e5b46b2b900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5741.633767] LustreError: 83262:0:(ldlm_flock.c:857:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5744.652327] LustreError: lustre-MDT0000-mdc-ffff9e5b4593d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5744.744338] LustreError: 83309:0:(file.c:6176:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5744.750409] LustreError: 83309:0:(file.c:6176:ll_inode_revalidate_fini()) Skipped 4 previous similar messages
[ 5745.879497] LustreError: 83290:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e5b4593d800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5754.160041] LustreError: 83361:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5754.171496] LustreError: 83361:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5758.248398] LustreError: 83361:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5758.264944] LustreError: 83361:0:(ldlm_flock.c:803:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5758.605529] LustreError: lustre-MDT0000-mdc-ffff9e5b4593d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5767.777317] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 19:44:55 (1787787895)
[ 5787.754718] LustreError: 79798:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5791.815416] LustreError: 79798:0:(ldlm_lockd.c:2890:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5799.931872] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 19:45:27 (1787787927)
[ 5852.645375] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 19:46:20 (1787787980)
[ 5918.288733] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 19:47:25 (1787788045)
[ 5954.173690] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 19:48:01 (1787788081)
[ 5954.789249] Lustre: Mounted lustre-client - version 2.17.57_55_ge1b14e7
[ 6078.936764] Lustre: Unmounted lustre-client
[ 6086.381620] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 19:50:13 (1787788213)
[ 6087.140720] LustreError: 87643:0:(osc_request.c:3129:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 6089.724782] Lustre: lustre-OST0000-osc-ffff9e5b4593d800: Connection to lustre-OST0000 (at 192.168.201.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6089.730295] Lustre: Skipped 17 previous similar messages
[ 6089.751577] LustreError: lustre-OST0000-osc-ffff9e5b4593d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 6089.768227] LustreError: 87730:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0000-osc-ffff9e5b4593d800: namespace resource [0x240000400:0x3ead:0x0].0x0 (ffff9e5b43752500) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 6089.776720] LustreError: 87730:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 6090.007124] LustreError: 87643:0:(osc_request.c:3129:osc_build_rpc()) cfs_fail_timeout interrupted
[ 6090.012639] Lustre: 2353:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.125@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 6090.020350] Lustre: lustre-OST0000-osc-ffff9e5b4593d800: Connection restored to 192.168.201.125@tcp (at 192.168.201.125@tcp)
[ 6090.023979] Lustre: Skipped 18 previous similar messages
[ 6099.238696] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 19:50:26 (1787788226)
[ 6157.279336] Lustre: 88310:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787788231/real 1787788231]  req@ffff9e5b44be4000 x1874625565160448/t0(0) o101->lustre-MDT0000-mdc-ffff9e5b4593d800@192.168.201.125@tcp:12/10 lens 328/344 e 0 to 1 dl 1787788286 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0
[ 6157.326188] Lustre: 88310:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
[ 6166.338373] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 19:51:33 (1787788293)
[ 6168.231636] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6170.278479] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 19:51:37 (1787788297)
[ 6234.069410] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6235.565552] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 19:52:43 (1787788363)
[ 6294.186704] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 19:53:41 (1787788421)
[ 6296.806215] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6298.427435] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 19:53:46 (1787788426)
[ 6299.690784] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6301.206733] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 19:53:49 (1787788429)
[ 6340.073262] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 19:54:27 (1787788467)
[ 6352.486542] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6379.508917] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 192.168.201.125@tcp) was lost; in progress operations using this service will fail
[ 6379.517161] LustreError: Skipped 2 previous similar messages
[ 6379.530506] Lustre: Evicted from MGS (at 192.168.201.125@tcp) after server handle changed from 0xf66ec02754c9c2f1 to 0xf66ec02754c9dafa
[ 6379.541932] Lustre: Skipped 2 previous similar messages
[ 6381.621113] Lustre: 68292:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.125@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6405.067415] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6406.866514] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6419.001925] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 19:55:46 (1787788546)
[ 6421.974662] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6424.253894] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 19:55:51 (1787788551)
[ 6439.918963] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 192.168.201.125@tcp) was lost; in progress operations using this service will fail
[ 6439.951683] Lustre: Evicted from MGS (at 192.168.201.125@tcp) after server handle changed from 0xf66ec02754c9dafa to 0xf66ec02754c9dfb5
[ 6451.027694] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 19:56:19 (1787788579)
[ 6501.601395] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 19:57:09 (1787788629)
[ 6508.012618] LustreError: lustre-OST0000-osc-ffff9e5b4593d800: operation ost_setattr to node 192.168.201.125@tcp failed: rc = -19
[ 6508.023112] LustreError: Skipped 4 previous similar messages
[ 6558.733722] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6561.769170] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6648.811941] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 192.168.201.125@tcp) was lost; in progress operations using this service will fail
[ 6648.833240] LustreError: Skipped 1 previous similar message
[ 6648.849307] Lustre: Evicted from MGS (at 192.168.201.125@tcp) after server handle changed from 0xf66ec02754c9e1c2 to 0xf66ec02754c9f00f
[ 6648.856281] Lustre: Skipped 1 previous similar message
[ 6650.098057] Lustre: 68292:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.125@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6673.757549] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6675.390913] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6698.986851] Lustre: lustre-MDT0000-mdc-ffff9e5b4593d800: Connection to lustre-MDT0000 (at 192.168.201.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6699.020474] Lustre: Skipped 8 previous similar messages
[ 6699.052528] Lustre: MGC192.168.201.125@tcp: Connection restored to 192.168.201.125@tcp (at 192.168.201.125@tcp)
[ 6699.065220] Lustre: Skipped 11 previous similar messages
[ 6716.610995] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 6718.237453] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6853.802814] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 20:03:00 (1787788980)
[ 6920.072734] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
[ 6924.196661] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7178.932171] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 20:08:26 (1787789306)
[ 7318.019051] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 192.168.201.125@tcp) was lost; in progress operations using this service will fail
[ 7318.032491] LustreError: Skipped 1 previous similar message
[ 7318.223191] Lustre: Evicted from MGS (at 192.168.201.125@tcp) after server handle changed from 0xf66ec02754c9f4d1 to 0xf66ec02754d99814
[ 7318.240985] Lustre: Skipped 1 previous similar message
[ 7318.259279] Lustre: MGC192.168.201.125@tcp: Connection restored to 192.168.201.125@tcp (at 192.168.201.125@tcp)
[ 7318.274101] Lustre: Skipped 1 previous similar message
[ 7354.940719] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 20:11:21 (1787789481)
[ 7357.257236] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7359.715889] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 20:11:26 (1787789486)
[ 7362.270232] LustreError: lustre-MDT0000-mdc-ffff9e5b4593d800: operation ldlm_enqueue to node 192.168.201.125@tcp failed: rc = -107
[ 7362.287207] LustreError: Skipped 120 previous similar messages
[ 7362.301367] Lustre: lustre-MDT0000-mdc-ffff9e5b4593d800: Connection to lustre-MDT0000 (at 192.168.201.125@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7362.331605] Lustre: Skipped 2 previous similar messages
[ 7362.357143] LustreError: lustre-MDT0000-mdc-ffff9e5b4593d800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7362.375709] LustreError: 101999:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9e5b4593d800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9e5b6027d600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 7362.406598] LustreError: 101999:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message
[ 7371.833291] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 20:11:39 (1787789499)
[ 7539.686503] LustreError: lustre-OST0000-osc-ffff9e5b4593d800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7541.149757] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 20:14:29 (1787789669)
[ 7565.792400] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787789674/real 1787789674]  req@ffff9e5b4418c000 x1874625582876032/t0(0) o4->lustre-OST0000-osc-ffff9e5b4593d800@192.168.201.125@tcp:6/4 lens 4584/448 e 0 to 1 dl 1787789694 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 7565.819902] Lustre: 2355:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
[ 7582.771200] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 20:15:10 (1787789710)
[ 7584.252718] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7586.111099] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 20:15:13 (1787789713)
[ 7587.523261] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7589.018785] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 20:15:16 (1787789716)
[ 7622.170316] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 20:15:49 (1787789749)
[ 7663.084714] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 192.168.201.125@tcp) was lost; in progress operations using this service will fail
[ 7663.109264] Lustre: Evicted from MGS (at 192.168.201.125@tcp) after server handle changed from 0xf66ec02754d99814 to 0xf66ec02754d9c10a
[ 7663.125793] Lustre: Skipped 1 previous similar message
[ 7675.461114] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 20:16:43 (1787789803)
[ 7677.036664] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7678.885593] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 20:16:46 (1787789806)
[ 7680.360587] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7681.809503] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 20:16:49 (1787789809)
[ 7691.658514] Lustre: Unmounted lustre-client
[ 7692.793061] Lustre: Mounted lustre-client - version 2.17.57_55_ge1b14e7
[ 7695.443180] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7713.247396] Lustre: 2353:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787789826/real 1787789826]  req@ffff9e5b6e1070c0 x1874625583062400/t0(0) o400->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 224/224 e 0 to 1 dl 1787789842 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7733.280173] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2