== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:01:17 (1788440477) start mds service on oleg351-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 3 07:51 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: kmemleak disabled oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: detected 4 online CPUs by sysfs oleg351-server: MODOPTS_LIBCFS= oleg351-server: Force libcfs to create 2 CPU partitions oleg351-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg351-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg351-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg351-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg351-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 238 (time 1788440568.14 total 10.02 last 23.74) - open/close 472 (time 1788440579.21 total 21.09 last 21.15) - open/close 752 (time 1788440589.83 total 31.71 last 26.37) - open/close 996 (time 1788440600.84 total 42.72 last 22.16) - open/close 1159 (time 1788440611.32 total 53.20 last 15.54) - open/close 1371 (time 1788440621.76 total 63.64 last 20.31) - open/close 1437 (time 1788440632.65 total 74.53 last 6.06) - open/close 1648 (time 1788440642.65 total 84.53 last 21.10) - open/close 1798 (time 1788440653.63 total 95.51 last 13.66) - open/close 2000 (time 1788440664.47 total 106.35 last 18.64) - open/close 2139 (time 1788440674.50 total 116.38 last 13.86) - open/close 2305 (time 1788440685.20 total 127.08 last 15.51) - open/close 2447 (time 1788440695.62 total 137.50 last 13.62) - open/close 2676 (time 1788440706.30 total 148.18 last 21.45) - open/close 2918 (time 1788440716.34 total 158.22 last 24.10) - open/close 3152 (time 1788440726.35 total 168.23 last 23.37) - open/close 3362 (time 1788440736.59 total 178.47 last 20.52) - open/close 3470 (time 1788440747.40 total 189.28 last 9.99) - open/close 3614 (time 1788440757.78 total 199.66 last 13.87) - open/close 3827 (time 1788440768.05 total 209.93 last 20.73) - open/close 3958 (time 1788440778.07 total 219.95 last 13.08) - open/close 4184 (time 1788440788.08 total 229.96 last 22.58) - open/close 4415 (time 1788440798.10 total 239.98 last 23.06) - open/close 4544 (time 1788440808.10 total 249.98 last 12.89) - open/close 4734 (time 1788440818.10 total 259.98 last 18.99) - open/close 4944 (time 1788440829.08 total 270.96 last 19.13) - open/close 5143 (time 1788440839.81 total 281.69 last 18.55) - open/close 5324 (time 1788440851.76 total 293.64 last 15.15) - open/close 5513 (time 1788440861.76 total 303.64 last 18.89) - open/close 5710 (time 1788440872.64 total 314.52 last 18.11) - open/close 5945 (time 1788440883.21 total 325.09 last 22.23) - open/close 6155 (time 1788440893.31 total 335.19 last 20.80) - open/close 6396 (time 1788440903.95 total 345.83 last 22.65) - open/close 6564 (time 1788440914.88 total 356.76 last 15.36) - open/close 6807 (time 1788440924.89 total 366.77 last 24.29) - open/close 6949 (time 1788440934.92 total 376.80 last 14.15) - open/close 7066 (time 1788440945.69 total 387.57 last 10.86) - open/close 7258 (time 1788440955.69 total 397.57 last 19.20) - open/close 7440 (time 1788440967.02 total 408.90 last 16.07) - open/close 7621 (time 1788440977.03 total 418.91 last 18.08) - open/close 7852 (time 1788440987.04 total 428.92 last 23.08) - open/close 8105 (time 1788440998.02 total 439.89 last 23.06) - open/close 8350 (time 1788441008.77 total 450.65 last 22.78) - open/close 8599 (time 1788441018.78 total 460.66 last 24.88) - open/close 8807 (time 1788441028.93 total 470.81 last 20.50) - open/close 9033 (time 1788441038.93 total 480.81 last 22.60) - open/close 9266 (time 1788441049.92 total 491.80 last 21.19) - open/close 9519 (time 1788441060.82 total 502.70 last 23.22) - open/close 9735 (time 1788441070.83 total 512.71 last 21.58) - open/close 9994 (time 1788441080.83 total 522.71 last 25.89) - open/close 10184 (time 1788441091.00 total 532.88 last 18.68) - open/close 10401 (time 1788441101.57 total 543.45 last 20.53) - open/close 10623 (time 1788441111.59 total 553.47 last 22.17) - open/close 10782 (time 1788441121.63 total 563.51 last 15.83) - open/close 10869 (time 1788441131.82 total 573.70 last 8.54) - open/close 10970 (time 1788441141.90 total 583.78 last 10.03) - open/close 11109 (time 1788441151.90 total 593.78 last 13.90) - open/close 11340 (time 1788441161.91 total 603.79 last 23.08) - open/close 11605 (time 1788441172.87 total 614.75 last 24.17) - open/close 11848 (time 1788441182.94 total 624.82 last 24.14) - open/close 11931 (time 1788441192.98 total 634.86 last 8.26) - open/close 12044 (time 1788441203.90 total 645.78 last 10.35) - open/close 12163 (time 1788441213.90 total 655.78 last 11.90) - open/close 12383 (time 1788441226.77 total 668.65 last 17.10) - open/close 12504 (time 1788441236.78 total 678.66 last 12.09) - open/close 12594 (time 1788441246.78 total 688.66 last 9.00) - open/close 12772 (time 1788441259.02 total 700.90 last 14.54) - open/close 12985 (time 1788441269.76 total 711.64 last 19.84) - open/close 13252 (time 1788441280.00 total 721.88 last 26.08) - open/close 13512 (time 1788441290.01 total 731.89 last 25.98) - open/close 13724 (time 1788441300.75 total 742.63 last 19.74) - open/close 13963 (time 1788441311.38 total 753.26 last 22.48) - open/close 14135 (time 1788441322.18 total 764.06 last 15.93) - open/close 14384 (time 1788441333.20 total 775.08 last 22.59) - open/close 14650 (time 1788441346.02 total 787.90 last 20.74) - open/close 14879 (time 1788441356.03 total 797.91 last 22.89) - open/close 15069 (time 1788441366.85 total 808.73 last 17.56) - open/close 15324 (time 1788441376.87 total 818.75 last 25.44) - open/close 15558 (time 1788441387.20 total 829.08 last 22.67) - open/close 15763 (time 1788441397.21 total 839.09 last 20.48) - open/close 16016 (time 1788441407.94 total 849.82 last 23.58) - open/close 16198 (time 1788441418.49 total 860.37 last 17.24) - open/close 16297 (time 1788441429.30 total 871.18 last 9.16) - open/close 16470 (time 1788441440.68 total 882.56 last 15.21) - open/close 16740 (time 1788441450.69 total 892.57 last 26.96) - open/close 16942 (time 1788441461.67 total 903.55 last 18.39) - open/close 17182 (time 1788441471.68 total 913.56 last 23.99) - open/close 17412 (time 1788441484.42 total 926.30 last 18.04) - open/close 17641 (time 1788441495.18 total 937.06 last 21.29) - open/close 17823 (time 1788441505.20 total 947.08 last 18.16) - open/close 17951 (time 1788441515.26 total 957.14 last 12.73) - open/close 18130 (time 1788441526.22 total 968.10 last 16.33) - open/close 18295 (time 1788441537.87 total 979.75 last 14.17) - open/close 18450 (time 1788441548.34 total 990.22 last 14.80) - open/close 18810 (time 1788441558.34 total 1000.22 last 36.00) - open/close 19475 (time 1788441568.34 total 1010.22 last 66.50) - open/close 20000 (time 1788441576.33 total 1018.21 last 65.76) - open/close 20676 (time 1788441586.33 total 1028.21 last 67.57) - open/close 21374 (time 1788441596.34 total 1038.22 last 69.76) - open/close 22037 (time 1788441606.34 total 1048.22 last 66.30) - open/close 22635 (time 1788441616.35 total 1058.23 last 59.73) - open/close 23284 (time 1788441626.35 total 1068.23 last 64.87) - open/close 23925 (time 1788441636.36 total 1078.24 last 64.04) - open/close 24457 (time 1788441646.37 total 1088.25 last 53.17) - open/close 24938 (time 1788441656.37 total 1098.25 last 48.09) - open/close 25517 (time 1788441666.37 total 1108.25 last 57.88) - open/close 26172 (time 1788441676.38 total 1118.26 last 65.49) - open/close 26876 (time 1788441686.39 total 1128.27 last 70.32) - open/close 27601 (time 1788441696.39 total 1138.27 last 72.47) - open/close 28272 (time 1788441706.41 total 1148.29 last 66.96) - open/close 28856 (time 1788441716.42 total 1158.30 last 58.36) - open/close 29357 (time 1788441726.43 total 1168.31 last 50.02) - open/close 30000 (time 1788441736.34 total 1178.22 last 64.91) - open/close 30577 (time 1788441746.34 total 1188.22 last 57.69) - open/close 31196 (time 1788441756.35 total 1198.23 last 61.87) - open/close 31801 (time 1788441766.37 total 1208.25 last 60.37) - open/close 32415 (time 1788441776.38 total 1218.26 last 61.33) - open/close 33058 (time 1788441786.39 total 1228.27 last 64.20) - open/close 33703 (time 1788441796.42 total 1238.30 last 64.31) - open/close 34278 (time 1788441806.44 total 1248.32 last 57.39) - open/close 34834 (time 1788441816.44 total 1258.32 last 55.59) - open/close 35320 (time 1788441826.45 total 1268.33 last 48.56) - open/close 35894 (time 1788441836.46 total 1278.34 last 57.35) - open/close 36468 (time 1788441846.48 total 1288.36 last 57.32) - open/close 36980 (time 1788441856.50 total 1298.38 last 51.07) - open/close 37512 (time 1788441866.50 total 1308.38 last 53.19) - open/close 38145 (time 1788441876.51 total 1318.39 last 63.23) - open/close 38748 (time 1788441886.52 total 1328.40 last 60.27) - open/close 39373 (time 1788441896.52 total 1338.40 last 62.49) - open/close 39989 (time 1788441906.52 total 1348.40 last 61.56) - open/close 40610 (time 1788441916.52 total 1358.40 last 62.10) - open/close 41271 (time 1788441926.53 total 1368.41 last 66.05) - open/close 41907 (time 1788441936.54 total 1378.42 last 63.54) - open/close 42479 (time 1788441946.55 total 1388.43 last 57.13) - open/close 43153 (time 1788441956.58 total 1398.46 last 67.26) - open/close 43767 (time 1788441966.59 total 1408.47 last 61.34) - open/close 44415 (time 1788441976.60 total 1418.48 last 64.68) - open/close 45038 (time 1788441986.62 total 1428.50 last 62.22) - open/close 45677 (time 1788441996.63 total 1438.51 last 63.85) - open/close 46312 (time 1788442006.65 total 1448.53 last 63.38) - open/close 46925 (time 1788442016.65 total 1458.53 last 61.27) - open/close 47556 (time 1788442026.65 total 1468.53 last 63.07) - open/close 48152 (time 1788442036.66 total 1478.54 last 59.56) - open/close 48778 (time 1788442046.67 total 1488.55 last 62.56) - open/close 49407 (time 1788442056.67 total 1498.55 last 62.88) - open/close 49985 (time 1788442066.67 total 1508.55 last 57.80) total: 50000 open/close in 1509.01 seconds: 33.13 ops/second - unlinked 0 (time 1788442072 ; total 1 ; last 1) - unlinked 10000 (time 1788442236 ; total 165 ; last 164) - unlinked 20000 (time 1788442406 ; total 335 ; last 170) - unlinked 30000 (time 1788442574 ; total 503 ; last 168) - unlinked 40000 (time 1788442744 ; total 673 ; last 170) total: 50000 unlinks in 840 seconds: 59.523811 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg351-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server unloading modules via unload_modules_local on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing unload_modules_local oleg351-server: kmemleak disabled modules unloaded. pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2