[ 1937.163748] LustreError: lustre-OST0000-osc-ffff9fe44554c000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1939.017346] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ IGNORE (bz5494): multiop didn't fail fsync: 5 or close: 0
[ 1942.754297] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ FAIL: no discarded dirty page found!
[ 1959.384744] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 22:04:12 (1781229852)
[ 1961.667806] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node
[ 1963.338595] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 22:04:16 (1781229856)
[ 1965.215110] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
[ 1967.240408] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 22:04:20 (1781229860)
[ 1971.259796] LustreError: lustre-MDT0000-mdc-ffff9fe44554c000: operation mds_reint to node 192.168.203.114@tcp failed: rc = -19
[ 1971.270934] LustreError: Skipped 1 previous similar message
[ 1988.575769] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 1997.820102] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0xa371501195b1adf2 to 0xa371501195b1c71a
[ 1998.959159] Lustre: 15891:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.203.114@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2102.063846] LustreError: lustre-MDT0000-mdc-ffff9fe44554c000: operation ldlm_enqueue to node 192.168.203.114@tcp failed: rc = -19
[ 2119.669534] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 2119.691842] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0xa371501195b1c71a to 0xa371501195b3c716
[ 2135.190404] Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ========================================================== 22:07:08 (1781230028)
[ 2135.708631] Lustre: *** cfs_fail_loc=305, val=0***
[ 2135.712555] Lustre: Skipped 4 previous similar messages
[ 2175.967565] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 2186.220404] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0xa371501195b3c716 to 0xa371501195b3f1cc
[ 2196.506662] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2197.817901] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2205.663861] Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 22:08:18 (1781230098)
[ 2216.939449] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 2216.956043] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0xa371501195b3f1cc to 0xa371501195b3f4f8
[ 2219.031917] LustreError: lustre-MDT0000-mdc-ffff9fe44554c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2240.590707] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ========================================================== 22:08:53 (1781230133)
[ 2257.905730] LustreError: lustre-OST0000-osc-ffff9fe44554c000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 2274.579138] Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load ======= 22:09:27 (1781230167)
[ 2288.782735] LustreError: lustre-MDT0000-mdc-ffff9fe44554c000: operation mds_close to node 192.168.203.114@tcp failed: rc = -19
[ 2309.094127] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 2309.118952] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0xa371501195b3f4f8 to 0xa371501195b4408c
[ 2311.108214] Lustre: 15891:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.203.114@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2334.346658] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2336.580340] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2420.713868] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 2420.735049] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0xa371501195b4408c to 0xa371501195b5f7be
[ 2422.198452] Lustre: 15891:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.203.114@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 2447.663962] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2450.372039] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2516.661843] Lustre: lustre-MDT0000-mdc-ffff9fe44554c000: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 2516.679187] Lustre: Skipped 9 previous similar messages
[ 2533.919473] Lustre: 2383:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781230412/real 1781230412]  req@ffff9fe44701d2c0 x1867752905368320/t0(0) o400->MGC192.168.203.114@tcp@192.168.203.114@tcp:26/25 lens 224/224 e 0 to 1 dl 1781230428 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 2533.960518] Lustre: 2383:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 2533.972041] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 2544.174550] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0xa371501195b5f7be to 0xa371501195b7af52
[ 2544.215473] Lustre: MGC192.168.203.114@tcp: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp)
[ 2544.223207] Lustre: Skipped 14 previous similar messages
[ 2559.140423] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2561.723100] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2592.642620] Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 22:14:45 (1781230485)
[ 2597.128783] LustreError: lustre-MDT0000-mdc-ffff9fe44554c000: operation mds_close to node 192.168.203.114@tcp failed: rc = -19
[ 2597.143324] LustreError: Skipped 5 previous similar messages
[ 2614.751332] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 2624.947136] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0xa371501195b7af52 to 0xa371501195b86eba
[ 2633.009148] Lustre: DEBUG MARKER: test_51: failover in 1 sec
[ 2664.185087] Lustre: DEBUG MARKER: test_51: failover in 5 sec
[ 2702.342625] Lustre: DEBUG MARKER: test_51: failover in 10 sec
[ 2746.919637] Lustre: DEBUG MARKER: test_51: failover in 20 sec
[ 2787.807482] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 2787.826461] LustreError: Skipped 3 previous similar messages
[ 2798.014772] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0xa371501195b94567 to 0xa371501195ba0363
[ 2798.035120] Lustre: Skipped 3 previous similar messages
[ 2806.211053] Lustre: DEBUG MARKER: test_51: failover in 25 sec
[ 2862.780507] Lustre: DEBUG MARKER: test_51: failover in 30 sec
[ 2949.446286] Lustre: DEBUG MARKER: == recovery-small test 52: failover OST under load ======= 22:20:42 (1781230842)
[ 2998.876759] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3001.109934] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3296.860136] LustreError: lustre-OST0000-osc-ffff9fe44554c000: operation ldlm_enqueue to node 192.168.203.114@tcp failed: rc = -19
[ 3296.881138] LustreError: Skipped 9 previous similar messages
[ 3296.885869] Lustre: lustre-OST0000-osc-ffff9fe44554c000: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3296.911627] Lustre: Skipped 8 previous similar messages
[ 3320.578934] Lustre: lustre-OST0000-osc-ffff9fe44554c000: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp)
[ 3320.591461] Lustre: Skipped 16 previous similar messages
[ 3340.276691] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3343.600131] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3663.826592] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3666.118499] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3922.517205] Lustre: DEBUG MARKER: == recovery-small test 53a: touch: drop rep ============== 22:36:55 (1781231815)
[ 3940.321807] Lustre: 50653:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781231818/real 1781231818]  req@ffff9fe45810b480 x1867752925755008/t0(0) o101->lustre-MDT0000-mdc-ffff9fe44554c000@192.168.203.114@tcp:12/10 lens 576/1152 e 0 to 1 dl 1781231834 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0
[ 3940.336670] Lustre: 50653:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 3940.342297] Lustre: lustre-MDT0000-mdc-ffff9fe44554c000: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 3940.352803] Lustre: Skipped 1 previous similar message
[ 3940.391548] Lustre: lustre-MDT0000-mdc-ffff9fe44554c000: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp)
[ 3940.399581] Lustre: Skipped 1 previous similar message
[ 3947.443316] Lustre: DEBUG MARKER: == recovery-small test 53b: touch: drop rep ============== 22:37:20 (1781231840)
[ 3974.622645] Lustre: DEBUG MARKER: == recovery-small test 53c: touch: drop rep ============== 22:37:47 (1781231867)
[ 4001.922802] Lustre: DEBUG MARKER: == recovery-small test 54: back in time ================== 22:38:14 (1781231894)
[ 4002.338963] Lustre: Mounted lustre-client
[ 4032.992679] Lustre: 2385:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781231911/real 1781231911]  req@ffff9fe448dee940 x1867752925773824/t0(0) o400->lustre-MDT0000-mdc-ffff9fe44554c000@192.168.203.114@tcp:12/10 lens 224/224 e 0 to 1 dl 1781231927 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 4032.997118] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 4033.018322] Lustre: 2385:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 4033.022416] LustreError: Skipped 2 previous similar messages
[ 4043.263118] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0xa371501195bbaa88 to 0xa371501195cf81d3
[ 4043.277216] Lustre: Skipped 2 previous similar messages
[ 4047.061695] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4048.736242] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4051.040861] Lustre: Unmounted lustre-client
[ 4056.979129] Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ========================================================== 22:39:10 (1781231950)
[ 4192.735621] Lustre: 2384:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781232071/real 1781232071]  req@ffff9fe45810a1c0 x1867752925806720/t0(0) o4->lustre-OST0000-osc-ffff9fe44554c000@192.168.203.114@tcp:6/4 lens 488/448 e 0 to 1 dl 1781232087 ref 2 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 4192.775736] Lustre: 2384:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 58 previous similar messages
[ 4351.003654] Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend ========================================================== 22:44:03 (1781232243)
[ 4401.290234] Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash ========================================================== 22:44:53 (1781232293)
[ 4405.063189] Lustre: Unmounted lustre-client
[ 4426.325915] Lustre: Mounted lustre-client
[ 4433.998766] Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ========================================================== 22:45:26 (1781232326)
[ 4434.373322] LustreError: 56535:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout id 801 sleeping for 20000ms
[ 4435.429849] LustreError: 56535:0:(mdc_locks.c:1335:mdc_finish_intent_lock()) cfs_fail_timeout interrupted
[ 4435.736055] Lustre: *** cfs_fail_loc=305, val=0***
[ 4459.160990] Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ========================================================== 22:45:52 (1781232352)
[ 4459.613249] Lustre: Mounted lustre-client
[ 4461.943439] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
[ 4472.268892] Lustre: Unmounted lustre-client
[ 4481.103255] Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ========================================================== 22:46:13 (1781232373)
[ 4568.285453] LustreError: lustre-MDT0000-mdc-ffff9fe446b20000: operation mds_reint to node 192.168.203.114@tcp failed: rc = -19
[ 4568.291604] LustreError: Skipped 1 previous similar message
[ 4568.297406] Lustre: lustre-MDT0000-mdc-ffff9fe446b20000: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 4568.308593] Lustre: Skipped 23 previous similar messages
[ 4588.515376] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 4588.539498] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0xa371501195cf9a0d to 0xa371501195d24d8c
[ 4588.586132] Lustre: MGC192.168.203.114@tcp: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp)
[ 4588.606705] Lustre: Skipped 24 previous similar messages
[ 4589.742383] Lustre: 55932:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.203.114@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 4755.981162] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ========================================================== 22:50:49 (1781232649)
[ 4760.718346] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 4776.931491] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 4776.954162] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0xa371501195d24d8c to 0xa371501195d71933
[ 4777.065145] LustreError: lustre-MDT0000-mdc-ffff9fe446b20000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4792.298822] Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export ========================================================== 22:51:25 (1781232685)
[ 4792.801223] Lustre: Mounted lustre-client
[ 4800.214940] LustreError: lustre-OST0000-osc-ffff9fe446b20000: operation ldlm_enqueue to node 192.168.203.114@tcp failed: rc = -107
[ 4800.231131] LustreError: lustre-OST0000-osc-ffff9fe446b20000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 4810.208761] Lustre: 2383:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781232688/real 1781232688]  req@ffff9fe450802940 x1867752930455424/t0(0) o103->lustre-OST0000-osc-ffff9fe442eef000@192.168.203.114@tcp:17/18 lens 328/224 e 0 to 1 dl 1781232704 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295
[ 4810.266369] Lustre: 2383:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 54 previous similar messages
[ 4812.985249] Lustre: Unmounted lustre-client
[ 4820.877627] Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction ========================================================== 22:51:53 (1781232713)
[ 4827.477234] LustreError: lustre-MDT0000-mdc-ffff9fe446b20000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4827.493098] LustreError: 60911:0:(file.c:6109:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
[ 4834.305891] Lustre: DEBUG MARKER: == recovery-small test 67: connect vs import invalidate race ========================================================== 22:52:07 (1781232727)
[ 4834.464714] LustreError: 61609:0:(recover.c:329:ptlrpc_recover_import()) cfs_race id 531 sleeping
[ 4839.904179] LustreError: 61609:0:(recover.c:329:ptlrpc_recover_import()) cfs_fail_race id 531 awake: rc=0
[ 4839.925459] LustreError: 61609:0:(import.c:713:ptlrpc_connect_import_locked()) already connected
[ 4841.288195] LustreError: lustre-MDT0000-mdc-ffff9fe446b20000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 4841.299657] LustreError: 61632:0:(import.c:293:ptlrpc_invalidate_import()) cfs_fail_race id 531 waking
[ 4841.373536] LustreError: 61637:0:(file.c:6109:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4842.512049] LustreError: 61644:0:(file.c:6109:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4842.528164] LustreError: 61644:0:(file.c:6109:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 4844.948205] LustreError: 61666:0:(file.c:6109:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4844.958034] LustreError: 61666:0:(file.c:6109:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 4849.673289] LustreError: 61711:0:(file.c:6109:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 4849.685561] LustreError: 61711:0:(file.c:6109:ll_inode_revalidate_fini()) Skipped 27 previous similar messages
[ 4860.490907] Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR ========================================================== 22:52:33 (1781232753)
[ 4897.181618] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4899.004135] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4909.935287] Lustre: DEBUG MARKER: == recovery-small test 101a: IR: Make sure IR works w/o normal recovery ========================================================== 22:53:23 (1781232803)
[ 4944.409279] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4946.184602] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4956.822698] Lustre: DEBUG MARKER: == recovery-small test 101b: IR: Make sure IR works w/o normal recovery and proceed EAGAIN ========================================================== 22:54:09 (1781232849)
[ 5020.510487] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5022.676823] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5033.130564] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ========================================================== 22:55:25 (1781232925)
[ 5072.256298] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5074.196424] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5081.288795] Lustre: Unmounted lustre-client
[ 5134.572376] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5137.231125] Lustre: Mounted lustre-client
[ 5144.645238] Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later ========================================================== 22:57:17 (1781233037)
[ 5147.882605] Lustre: DEBUG MARKER: SKIP: recovery-small test_103 needs separate mgs and mds
[ 5150.072590] Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily ========================================================== 22:57:22 (1781233042)
[ 5182.856227] Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support === 22:57:55 (1781233075)
[ 5184.683341] Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
[ 5186.589422] Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support ========================================================== 22:57:59 (1781233079)
[ 5188.486169] Lustre: *** cfs_fail_loc=805, val=0***
[ 5188.530420] Lustre: Mounted lustre-client
[ 5193.963817] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 5215.201519] Lustre: lustre-MDT0000-mdc-ffff9fe443728800: Connection to lustre-MDT0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5215.214510] Lustre: Skipped 12 previous similar messages
[ 5215.224708] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 5225.446933] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0xa371501195d72358 to 0xa371501195d7266f
[ 5225.473209] Lustre: MGC192.168.203.114@tcp: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp)
[ 5225.493524] Lustre: Skipped 13 previous similar messages
[ 5234.770644] Lustre: Unmounted lustre-client
[ 5244.137332] Lustre: DEBUG MARKER: == recovery-small test 107: drop reint reply, then restart MDT ========================================================== 22:58:56 (1781233136)
[ 5271.577981] LustreError: 2382:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff9fe451cce940 x1867752930536192/t94489280516(94489280516) o101->lustre-MDT0000-mdc-ffff9fe443728800@192.168.203.114@tcp:12/10 lens 648/608 e 0 to 0 dl 1781233181 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0
[ 5280.819094] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5282.459887] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5291.077358] Lustre: DEBUG MARKER: == recovery-small test 108: client eviction don't crash == 22:59:44 (1781233184)
[ 5292.644880] LustreError: lustre-OST0000-osc-ffff9fe443728800: operation ost_write to node 192.168.203.114@tcp failed: rc = -107
[ 5292.662330] LustreError: Skipped 2 previous similar messages
[ 5292.674628] LustreError: lustre-OST0000-osc-ffff9fe443728800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5292.700645] Lustre: 2386:0:(llite_lib.c:4198:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.114@tcp:/lustre/fid: [0x20000a042:0x6:0x0]// may get corrupted (rc -5)
[ 5292.732857] LustreError: 72611:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-OST0000-osc-ffff9fe443728800: namespace resource [0x240000400:0x3ea2:0x0].0x0 (ffff9fe451dc8800) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5301.252034] Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req ========================================================== 22:59:54 (1781233194)
[ 5302.733597] Lustre: DEBUG MARKER: SKIP: recovery-small test_110a needs >= 2 MDTs
[ 5304.446680] Lustre: DEBUG MARKER: == recovery-small test 110b: create remote directory: drop Master rep ========================================================== 22:59:57 (1781233197)
[ 5306.367378] Lustre: DEBUG MARKER: SKIP: recovery-small test_110b needs >= 2 MDTs
[ 5308.300639] Lustre: DEBUG MARKER: == recovery-small test 110c: create remote directory: drop update rep on slave MDT ========================================================== 23:00:01 (1781233201)
[ 5309.779875] Lustre: DEBUG MARKER: SKIP: recovery-small test_110c needs >= 2 MDTs
[ 5311.254976] Lustre: DEBUG MARKER: == recovery-small test 110d: remove remote directory: drop client req ========================================================== 23:00:04 (1781233204)
[ 5312.748415] Lustre: DEBUG MARKER: SKIP: recovery-small test_110d needs >= 2 MDTs
[ 5314.372903] Lustre: DEBUG MARKER: == recovery-small test 110e: remove remote directory: drop master rep ========================================================== 23:00:07 (1781233207)
[ 5316.125106] Lustre: DEBUG MARKER: SKIP: recovery-small test_110e needs >= 2 MDTs
[ 5318.208968] Lustre: DEBUG MARKER: == recovery-small test 110f: remove remote directory: drop slave rep ========================================================== 23:00:10 (1781233210)
[ 5319.573496] Lustre: DEBUG MARKER: SKIP: recovery-small test_110f needs >= 2 MDTs
[ 5321.424479] Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration ========================================================== 23:00:14 (1781233214)
[ 5323.224460] Lustre: DEBUG MARKER: SKIP: recovery-small test_110g needs >= 2 MDTs
[ 5325.269193] Lustre: DEBUG MARKER: == recovery-small test 110h: drop update reply during cross-MDT file rename ========================================================== 23:00:17 (1781233217)
[ 5327.014214] Lustre: DEBUG MARKER: SKIP: recovery-small test_110h needs >= 2 MDTs
[ 5328.412909] Lustre: DEBUG MARKER: == recovery-small test 110i: drop update reply during cross-MDT dir rename ========================================================== 23:00:21 (1781233221)
[ 5329.950625] Lustre: DEBUG MARKER: SKIP: recovery-small test_110i needs >= 2 MDTs
[ 5331.795616] Lustre: DEBUG MARKER: == recovery-small test 110j: drop update reply during cross-MDT ln ========================================================== 23:00:24 (1781233224)
[ 5333.092441] Lustre: DEBUG MARKER: SKIP: recovery-small test_110j needs >= 2 MDTs
[ 5334.749853] Lustre: DEBUG MARKER: == recovery-small test 110k: FID_QUERY failed during recovery ========================================================== 23:00:27 (1781233227)
[ 5336.367706] Lustre: DEBUG MARKER: SKIP: recovery-small test_110k needs >= 2 MDTS
[ 5338.239849] Lustre: DEBUG MARKER: == recovery-small test 110m: update resent vs original RPC race ========================================================== 23:00:30 (1781233230)
[ 5340.574776] Lustre: DEBUG MARKER: SKIP: recovery-small test_110m needs at least 2 MDTs
[ 5342.317056] Lustre: DEBUG MARKER: == recovery-small test 111: mdd setup fail should not cause umount oops ========================================================== 23:00:35 (1781233235)
[ 5372.007846] Lustre: DEBUG MARKER: == recovery-small test 112a: bulk resend while orignal request is in progress ========================================================== 23:01:05 (1781233265)
[ 5402.011333] Lustre: DEBUG MARKER: == recovery-small test 115a: read: late REQ MDunlink and no bulk ========================================================== 23:01:34 (1781233294)
[ 5402.689613] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5413.716437] Lustre: DEBUG MARKER: == recovery-small test 115b: write: late REQ MDunlink and no bulk ========================================================== 23:01:46 (1781233306)
[ 5414.169697] Lustre: *** cfs_fail_loc=51b, val=4***
[ 5424.614506] Lustre: DEBUG MARKER: == recovery-small test 115c: read: late Reply MDunlink and no bulk ========================================================== 23:01:57 (1781233317)
[ 5425.014048] Lustre: *** cfs_fail_loc=50f, val=3***
[ 5435.589319] Lustre: DEBUG MARKER: == recovery-small test 115d: write: late Reply MDunlink and no bulk ========================================================== 23:02:08 (1781233328)
[ 5435.857335] Lustre: *** cfs_fail_loc=50f, val=4***
[ 5444.889494] Lustre: DEBUG MARKER: == recovery-small test 115e: read: late Bulk MDunlink and no reply ========================================================== 23:02:17 (1781233337)
[ 5445.173627] Lustre: *** cfs_fail_loc=510, val=3***
[ 5454.796768] Lustre: DEBUG MARKER: == recovery-small test 115f: read: late REQ MDunlink and no reply ========================================================== 23:02:27 (1781233347)
[ 5455.149521] Lustre: *** cfs_fail_loc=51b, val=3***
[ 5514.719217] Lustre: 2385:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781233353/real 1781233353]  req@ffff9fe446cade00 x1867752930578304/t0(0) o3->lustre-OST0001-osc-ffff9fe443728800@192.168.203.114@tcp:6/4 lens 488/4536 e 0 to 1 dl 1781233408 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
[ 5514.759374] Lustre: 2385:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 17 previous similar messages
[ 5522.966580] Lustre: DEBUG MARKER: == recovery-small test 115g: read: late REQ MDunlink and Reply MDunlink ========================================================== 23:03:35 (1781233415)
[ 5523.439818] Lustre: *** cfs_fail_loc=51c, val=3***
[ 5589.070708] Lustre: DEBUG MARKER: == recovery-small test 120: flock race: completion vs. evict ========================================================== 23:04:42 (1781233482)
[ 5589.583047] LustreError: 82821:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5592.501934] LustreError: lustre-MDT0000-mdc-ffff9fe443728800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5592.535130] LustreError: 82835:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9fe443728800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9fe446de2c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5593.663135] LustreError: 82821:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5595.754857] LustreError: 82851:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5598.925705] LustreError: lustre-MDT0000-mdc-ffff9fe443728800: operation ldlm_enqueue to node 192.168.203.114@tcp failed: rc = -107
[ 5598.936485] LustreError: Skipped 1 previous similar message
[ 5598.959783] LustreError: lustre-MDT0000-mdc-ffff9fe443728800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5598.984378] LustreError: 82865:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9fe443728800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9fe446de2900) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5599.027555] LustreError: 82865:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5599.823135] LustreError: 82851:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5606.780156] LustreError: lustre-MDT0000-mdc-ffff9fe443728800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5606.795304] LustreError: 82894:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9fe443728800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9fe446de2600) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5606.822966] LustreError: 82894:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5607.122769] LustreError: 82907:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5610.087728] LustreError: lustre-MDT0000-mdc-ffff9fe443728800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5610.165732] LustreError: 82927:0:(file.c:6109:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5610.189556] LustreError: 82927:0:(file.c:6109:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5611.224741] LustreError: 82907:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5611.237859] LustreError: 82907:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff9fe443728800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5611.339880] LustreError: 82933:0:(file.c:6109:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5611.349746] LustreError: 82933:0:(file.c:6109:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
[ 5613.586355] LustreError: 82955:0:(file.c:6109:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5613.601867] LustreError: 82955:0:(file.c:6109:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
[ 5617.281123] LustreError: 82982:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 sleeping for 4000ms
[ 5620.230359] LustreError: lustre-MDT0000-mdc-ffff9fe443728800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5620.252690] LustreError: 82997:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9fe443728800: namespace resource [0x20000a042:0x11:0x0].0xc (ffff9fe446de2000) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5620.258569] LustreError: 82996:0:(file.c:6109:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5620.283666] LustreError: 82996:0:(file.c:6109:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
[ 5621.379533] LustreError: 82982:0:(ldlm_flock.c:809:ldlm_flock_completion_ast()) cfs_fail_timeout id 321 awake
[ 5628.412808] LustreError: lustre-MDT0000-mdc-ffff9fe443728800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5628.891692] LustreError: 83038:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 sleeping for 4000ms
[ 5632.129688] LustreError: lustre-MDT0000-mdc-ffff9fe443728800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5632.147926] LustreError: 83053:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff9fe443728800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff9fe458831e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5632.177210] LustreError: 83053:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 1 previous similar message
[ 5632.978723] LustreError: 83038:0:(ldlm_flock.c:858:ldlm_flock_completion_ast()) cfs_fail_timeout id 322 awake
[ 5636.003975] LustreError: lustre-MDT0000-mdc-ffff9fe443728800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5636.110427] LustreError: 83086:0:(file.c:6109:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5637.248925] LustreError: 83066:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff9fe443728800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5644.985377] LustreError: 83138:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 sleeping for 4000ms
[ 5644.995226] LustreError: 83138:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5648.870263] LustreError: lustre-MDT0000-mdc-ffff9fe443728800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 5649.072067] LustreError: 83138:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) cfs_fail_timeout id 320 awake
[ 5649.082478] LustreError: 83138:0:(ldlm_flock.c:804:ldlm_flock_completion_ast()) Skipped 1 previous similar message
[ 5649.093846] LustreError: 83138:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff9fe443728800: inode [0x20000a042:0x11:0x0] mdc close failed: rc = -108
[ 5652.387473] LustreError: 83189:0:(file.c:6109:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
[ 5652.400715] LustreError: 83189:0:(file.c:6109:ll_inode_revalidate_fini()) Skipped 46 previous similar messages
[ 5660.623326] Lustre: DEBUG MARKER: == recovery-small test 113: ldlm enqueue dropped reply should not cause deadlocks ========================================================== 23:05:53 (1781233553)
[ 5680.470754] LustreError: 68113:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms
[ 5680.478094] LustreError: 68113:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5684.535605] LustreError: 68113:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f awake
[ 5684.548305] LustreError: 68113:0:(ldlm_lockd.c:2883:ldlm_bl_thread_blwi()) Skipped 1 previous similar message
[ 5692.819477] Lustre: DEBUG MARKER: == recovery-small test 130a: enqueue resend on not existing file ========================================================== 23:06:25 (1781233585)
[ 5744.482634] Lustre: DEBUG MARKER: == recovery-small test 130b: enqueue resend on a stale inode ========================================================== 23:07:17 (1781233637)
[ 5807.417706] Lustre: DEBUG MARKER: == recovery-small test 130c: layout intent resend on a stale inode ========================================================== 23:08:20 (1781233700)
[ 5842.680234] Lustre: DEBUG MARKER: == recovery-small test 132: long punch =================== 23:08:55 (1781233735)
[ 5843.170888] Lustre: Mounted lustre-client
[ 5967.939315] Lustre: Unmounted lustre-client
[ 5976.659924] Lustre: DEBUG MARKER: == recovery-small test 131: IO vs evict results to IO under staled lock ========================================================== 23:11:09 (1781233869)
[ 5977.342096] LustreError: 87466:0:(osc_request.c:3128:osc_build_rpc()) cfs_fail_timeout id 414 sleeping for 10000ms
[ 5980.051492] Lustre: lustre-OST0000-osc-ffff9fe443728800: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 5980.074898] Lustre: Skipped 18 previous similar messages
[ 5980.085279] LustreError: lustre-OST0000-osc-ffff9fe443728800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 5980.106624] LustreError: 87553:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-OST0000-osc-ffff9fe443728800: namespace resource [0x240000400:0x3ecd:0x0].0x0 (ffff9fe446de2a00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5980.124952] LustreError: 87553:0:(ldlm_resource.c:1180:ldlm_resource_complain()) Skipped 2 previous similar messages
[ 5980.393573] LustreError: 87466:0:(osc_request.c:3128:osc_build_rpc()) cfs_fail_timeout interrupted
[ 5980.400211] Lustre: 2384:0:(llite_lib.c:4198:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.114@tcp:/lustre/fid: [0x20000bf89:0x9:0x0]// may get corrupted (rc -108)
[ 5980.427470] Lustre: lustre-OST0000-osc-ffff9fe443728800: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp)
[ 5980.439625] Lustre: Skipped 17 previous similar messages
[ 5987.122510] Lustre: DEBUG MARKER: == recovery-small test 133: don't fail on flock resend === 23:11:20 (1781233880)
[ 6051.190456] Lustre: DEBUG MARKER: == recovery-small test 134: race between failover and search for reply data free slot ========================================================== 23:12:24 (1781233944)
[ 6052.639219] Lustre: DEBUG MARKER: SKIP: recovery-small test_134 Need 2+ clients, have 1
[ 6054.719669] Lustre: DEBUG MARKER: == recovery-small test 135: DOM: open/create resend to return size ========================================================== 23:12:27 (1781233947)
[ 6117.649342] Lustre: DEBUG MARKER: SKIP: recovery-small test_136 skipping excluded test 136
[ 6119.334661] Lustre: DEBUG MARKER: == recovery-small test 137: late resend must be skipped if already applied ========================================================== 23:13:32 (1781234012)
[ 6175.712242] Lustre: 89866:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781234015/real 1781234015]  req@ffff9fe45146a580 x1867752930728320/t0(0) o36->lustre-MDT0000-mdc-ffff9fe443728800@192.168.203.114@tcp:12/10 lens 496/440 e 0 to 1 dl 1781234070 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ln.0' uid:0 gid:0 projid:4294967295
[ 6175.800040] Lustre: 89866:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 6178.038783] Lustre: DEBUG MARKER: == recovery-small test 138: Umount MDT during recovery === 23:14:30 (1781234070)
[ 6180.482305] Lustre: DEBUG MARKER: SKIP: recovery-small test_138 needs >= 2 MDTs
[ 6182.903877] Lustre: DEBUG MARKER: == recovery-small test 139: corrupted catid won't cause crash ========================================================== 23:14:35 (1781234075)
[ 6185.355180] Lustre: DEBUG MARKER: SKIP: recovery-small test_139 needs >= 2 MDTs
[ 6188.591575] Lustre: DEBUG MARKER: == recovery-small test 140a: local mount is flagged properly ========================================================== 23:14:40 (1781234080)
[ 6226.120975] Lustre: DEBUG MARKER: == recovery-small test 140b: local mount is excluded from recovery ========================================================== 23:15:19 (1781234119)
[ 6237.307106] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 6262.751569] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 6262.768982] LustreError: Skipped 2 previous similar messages
[ 6273.070432] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0xa371501195d73189 to 0xa371501195d74c86
[ 6273.089935] Lustre: Skipped 2 previous similar messages
[ 6276.143928] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6278.085582] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6289.467665] Lustre: DEBUG MARKER: == recovery-small test 141: do not lose locks on MGS restart ========================================================== 23:16:22 (1781234182)
[ 6291.643424] Lustre: DEBUG MARKER: SKIP: recovery-small test_141 cannot run in local mode or from build tree
[ 6293.434429] Lustre: DEBUG MARKER: == recovery-small test 142: orphan name stub can be cleaned up in startup ========================================================== 23:16:26 (1781234186)
[ 6308.837619] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 6308.870848] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0xa371501195d74c86 to 0xa371501195d74e93
[ 6319.272627] Lustre: DEBUG MARKER: == recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed ========================================================== 23:16:52 (1781234212)
[ 6351.599673] Lustre: 68115:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.203.114@tcp: IR log lustre-cliir failed, not fatal: rc = -2
[ 6371.930399] Lustre: DEBUG MARKER: == recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 23:17:44 (1781234264)
[ 6379.308877] LustreError: lustre-OST0000-osc-ffff9fe443728800: operation ost_setattr to node 192.168.203.114@tcp failed: rc = -19
[ 6379.317980] LustreError: Skipped 7 previous similar messages
[ 6432.597157] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6436.993380] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6522.847536] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 6522.864760] LustreError: Skipped 1 previous similar message
[ 6533.096930] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0xa371501195d750a0 to 0xa371501195d75ef4
[ 6533.103910] Lustre: Skipped 1 previous similar message
[ 6546.664689] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6548.420634] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6583.122264] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6584.848842] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6730.349797] Lustre: DEBUG MARKER: == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 23:23:43 (1781234623)
[ 6741.523562] Lustre: lustre-OST0000-osc-ffff9fe443728800: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 6741.545846] Lustre: Skipped 9 previous similar messages
[ 6789.071018] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6792.351658] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 7050.543595] Lustre: DEBUG MARKER: == recovery-small test 144c: reconnection during orphan cleanup shouldn't lose LAST_ID synchronization ========================================================== 23:29:03 (1781234943)
[ 7244.269328] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 7244.277286] LustreError: Skipped 1 previous similar message
[ 7244.328927] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0xa371501195d76576 to 0xa371501195e6b354
[ 7244.348793] Lustre: Skipped 1 previous similar message
[ 7244.376887] Lustre: MGC192.168.203.114@tcp: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp)
[ 7244.389279] Lustre: Skipped 13 previous similar messages
[ 7270.881355] Lustre: 2384:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781235128/real 1781235128]  req@ffff9fe4474ed680 x1867752948009856/t0(0) o400->lustre-MDT0000-mdc-ffff9fe443728800@192.168.203.114@tcp:12/10 lens 224/224 e 0 to 1 dl 1781235165 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 7270.916075] Lustre: 2384:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 28 previous similar messages
[ 7274.457301] Lustre: DEBUG MARKER: == recovery-small test 145: connect mdtlovs and process update logs after recovery expire ========================================================== 23:32:47 (1781235167)
[ 7275.938766] Lustre: DEBUG MARKER: SKIP: recovery-small test_145 needs >= 3 MDTs
[ 7277.787121] Lustre: DEBUG MARKER: == recovery-small test 146: test eviction is counted properly ========================================================== 23:32:50 (1781235170)
[ 7279.621362] LustreError: lustre-MDT0000-mdc-ffff9fe443728800: operation ldlm_enqueue to node 192.168.203.114@tcp failed: rc = -107
[ 7279.636321] LustreError: Skipped 54 previous similar messages
[ 7279.653102] LustreError: lustre-MDT0000-mdc-ffff9fe443728800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 7286.686310] Lustre: DEBUG MARKER: == recovery-small test 147: Check client reconnect ======= 23:32:59 (1781235179)
[ 7453.746316] LustreError: lustre-OST0000-osc-ffff9fe443728800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 7455.547602] Lustre: DEBUG MARKER: == recovery-small test 148: data corruption through resend ========================================================== 23:35:48 (1781235348)
[ 7481.311207] Lustre: lustre-OST0000-osc-ffff9fe443728800: Connection to lustre-OST0000 (at 192.168.203.114@tcp) was lost; in progress operations using this service will wait for recovery to complete
[ 7481.332224] Lustre: Skipped 3 previous similar messages
[ 7481.345781] Lustre: lustre-OST0000-osc-ffff9fe443728800: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp)
[ 7481.356829] Lustre: Skipped 2 previous similar messages
[ 7498.625081] Lustre: DEBUG MARKER: == recovery-small test 149: skip orphan removal at umount ========================================================== 23:36:31 (1781235391)
[ 7499.992553] Lustre: DEBUG MARKER: SKIP: recovery-small test_149 needs >= 2 MDTs
[ 7501.619742] Lustre: DEBUG MARKER: == recovery-small test 150: statfs when MDT0 offline with lazystatfs option ========================================================== 23:36:34 (1781235394)
[ 7503.133560] Lustre: DEBUG MARKER: SKIP: recovery-small test_150 needs >= 2 MDTs
[ 7504.620617] Lustre: DEBUG MARKER: == recovery-small test 152: QoS object allocation could be awakened in case of OST failover ========================================================== 23:36:37 (1781235397)
[ 7536.285222] Lustre: DEBUG MARKER: == recovery-small test 153: evict vs reconnect race ====== 23:37:09 (1781235429)
[ 7571.443894] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 192.168.203.114@tcp) was lost; in progress operations using this service will fail
[ 7571.463000] Lustre: Evicted from MGS (at 192.168.203.114@tcp) after server handle changed from 0xa371501195e6b354 to 0xa371501195e6dbe8
[ 7571.476639] Lustre: Skipped 1 previous similar message
[ 7586.984720] Lustre: DEBUG MARKER: == recovery-small test 154a: corruption update llog can be skipped ========================================================== 23:38:00 (1781235480)
[ 7588.556177] Lustre: DEBUG MARKER: SKIP: recovery-small test_154a needs >= 2 MDTs
[ 7590.197539] Lustre: DEBUG MARKER: == recovery-small test 154b: restore update llog after failed recovery ========================================================== 23:38:03 (1781235483)
[ 7591.827235] Lustre: DEBUG MARKER: SKIP: recovery-small test_154b needs >= 2 MDTs
[ 7593.477577] Lustre: DEBUG MARKER: == recovery-small test 155: failover after client remount ========================================================== 23:38:06 (1781235486)
[ 7608.798163] Lustre: Unmounted lustre-client
[ 7610.317919] Lustre: Mounted lustre-client
[ 7612.797728] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
[ 7636.095436] Lustre: lustre-MDT0000-mdc-ffff9fe447e88800: Connection restored to 192.168.203.114@tcp (at 192.168.203.114@tcp)
[ 7636.119054] Lustre: Skipped 2 previous similar messages
[ 7640.333755] Lustre: DEBUG MARKER: recovery-small test_155: @@@@@@ FAIL: ls1 != ls2