== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:42:44 (1788442964) start mds service on oleg132-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 3 07:51 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing load_modules_local oleg132-server: kmemleak disabled oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: detected 4 online CPUs by sysfs oleg132-server: MODOPTS_LIBCFS= oleg132-server: Force libcfs to create 2 CPU partitions oleg132-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg132-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg132-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg132-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg132-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0000 oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 145 (time 1788443062.00 total 10.88 last 13.32) - open/close 288 (time 1788443072.07 total 20.95 last 14.20) - open/close 404 (time 1788443082.08 total 30.96 last 11.58) - open/close 611 (time 1788443092.55 total 41.43 last 19.78) - open/close 789 (time 1788443102.55 total 51.43 last 17.79) - open/close 1004 (time 1788443113.28 total 62.16 last 20.04) - open/close 1169 (time 1788443123.29 total 72.17 last 16.49) - open/close 1256 (time 1788443133.62 total 82.50 last 8.42) - open/close 1288 (time 1788443143.80 total 92.69 last 3.14) - open/close 1371 (time 1788443154.56 total 103.44 last 7.71) - open/close 1467 (time 1788443166.13 total 115.02 last 8.30) - open/close 1520 (time 1788443177.04 total 125.92 last 4.86) - open/close 1646 (time 1788443187.05 total 135.93 last 12.58) - open/close 1755 (time 1788443197.06 total 145.94 last 10.89) - open/close 1843 (time 1788443209.73 total 158.61 last 6.95) - open/close 1919 (time 1788443219.90 total 168.78 last 7.47) - open/close 2014 (time 1788443232.79 total 181.67 last 7.37) - open/close 2183 (time 1788443243.44 total 192.32 last 15.87) - open/close 2287 (time 1788443253.46 total 202.34 last 10.38) - open/close 2532 (time 1788443263.87 total 212.76 last 23.52) - open/close 2767 (time 1788443273.90 total 222.78 last 23.44) - open/close 2906 (time 1788443284.09 total 232.98 last 13.63) - open/close 3125 (time 1788443294.12 total 243.00 last 21.85) - open/close 3372 (time 1788443304.12 total 253.00 last 24.69) - open/close 3547 (time 1788443314.12 total 263.01 last 17.50) - open/close 3781 (time 1788443324.92 total 273.81 last 21.66) - open/close 3904 (time 1788443334.95 total 283.83 last 12.27) - open/close 4121 (time 1788443344.95 total 293.84 last 21.69) - open/close 4209 (time 1788443354.97 total 303.86 last 8.78) - open/close 4414 (time 1788443365.36 total 314.25 last 19.73) - open/close 4555 (time 1788443375.49 total 324.38 last 13.92) - open/close 4690 (time 1788443386.17 total 335.05 last 12.64) - open/close 4790 (time 1788443398.18 total 347.07 last 8.32) - open/close 4903 (time 1788443408.67 total 357.56 last 10.78) - open/close 5015 (time 1788443419.19 total 368.08 last 10.64) - open/close 5129 (time 1788443429.20 total 378.09 last 11.39) - open/close 5243 (time 1788443439.54 total 388.42 last 11.03) - open/close 5362 (time 1788443449.91 total 398.80 last 11.47) - open/close 5515 (time 1788443460.68 total 409.57 last 14.21) - open/close 5648 (time 1788443470.71 total 419.59 last 13.27) - open/close 5790 (time 1788443480.72 total 429.61 last 14.18) - open/close 5932 (time 1788443491.57 total 440.45 last 13.09) - open/close 6032 (time 1788443501.61 total 450.49 last 9.96) - open/close 6251 (time 1788443512.13 total 461.02 last 20.81) - open/close 6355 (time 1788443522.33 total 471.22 last 10.20) - open/close 6478 (time 1788443533.36 total 482.25 last 11.15) - open/close 6608 (time 1788443543.76 total 492.64 last 12.50) - open/close 6783 (time 1788443554.06 total 502.94 last 16.99) - open/close 6908 (time 1788443564.42 total 513.30 last 12.07) - open/close 7086 (time 1788443575.48 total 524.37 last 16.09) - open/close 7261 (time 1788443585.72 total 534.61 last 17.10) - open/close 7396 (time 1788443596.52 total 545.40 last 12.50) - open/close 7571 (time 1788443607.30 total 556.18 last 16.23) - open/close 7782 (time 1788443618.34 total 567.23 last 19.10) - open/close 7872 (time 1788443628.86 total 577.75 last 8.55) - open/close 7961 (time 1788443639.43 total 588.31 last 8.43) - open/close 8092 (time 1788443651.82 total 600.71 last 10.57) - open/close 8224 (time 1788443662.52 total 611.40 last 12.34) - open/close 8407 (time 1788443672.94 total 621.82 last 17.56) - open/close 8559 (time 1788443683.72 total 632.61 last 14.10) - open/close 8712 (time 1788443695.66 total 644.54 last 12.82) - open/close 8843 (time 1788443705.75 total 654.63 last 12.99) - open/close 8955 (time 1788443715.83 total 664.71 last 11.11) - open/close 9074 (time 1788443725.85 total 674.74 last 11.87) - open/close 9242 (time 1788443735.87 total 684.75 last 16.77) - open/close 9424 (time 1788443745.87 total 694.76 last 18.19) - open/close 9564 (time 1788443756.29 total 705.18 last 13.43) - open/close 9707 (time 1788443766.30 total 715.19 last 14.29) - open/close 9811 (time 1788443777.30 total 726.18 last 9.46) - open/close 9981 (time 1788443787.92 total 736.80 last 16.01) - open/close 10210 (time 1788443797.92 total 746.80 last 22.90) - open/close 10352 (time 1788443808.31 total 757.20 last 13.67) - open/close 10585 (time 1788443818.32 total 767.20 last 23.29) - open/close 10777 (time 1788443828.35 total 777.23 last 19.14) - open/close 10989 (time 1788443838.72 total 787.61 last 20.43) - open/close 11216 (time 1788443848.73 total 797.62 last 22.68) - open/close 11458 (time 1788443858.74 total 807.63 last 24.18) - open/close 11639 (time 1788443869.75 total 818.64 last 16.44) - open/close 11862 (time 1788443879.77 total 828.65 last 22.26) - open/close 12083 (time 1788443895.63 total 844.52 last 13.93) - open/close 12328 (time 1788443906.17 total 855.05 last 23.26) - open/close 12596 (time 1788443917.04 total 865.92 last 24.65) - open/close 12832 (time 1788443927.54 total 876.42 last 22.48) - open/close 12982 (time 1788443937.89 total 886.77 last 14.49) - open/close 13212 (time 1788443948.27 total 897.16 last 22.15) - open/close 13325 (time 1788443959.31 total 908.19 last 10.24) - open/close 13440 (time 1788443969.33 total 918.22 last 11.47) - open/close 13637 (time 1788443979.38 total 928.27 last 19.61) - open/close 13765 (time 1788443989.90 total 938.78 last 12.17) - open/close 13851 (time 1788443999.92 total 948.80 last 8.59) - open/close 13982 (time 1788444009.93 total 958.82 last 13.08) - open/close 14195 (time 1788444020.46 total 969.34 last 20.23) - open/close 14420 (time 1788444031.07 total 979.96 last 21.20) - open/close 14588 (time 1788444041.62 total 990.51 last 15.92) - open/close 14651 (time 1788444051.66 total 1000.54 last 6.28) - open/close 14783 (time 1788444062.07 total 1010.95 last 12.68) - open/close 14940 (time 1788444072.07 total 1020.96 last 15.70) - open/close 15061 (time 1788444082.63 total 1031.51 last 11.46) - open/close 15199 (time 1788444092.63 total 1041.52 last 13.80) - open/close 15268 (time 1788444104.34 total 1053.23 last 5.89) - open/close 15399 (time 1788444114.74 total 1063.62 last 12.60) - open/close 15510 (time 1788444124.85 total 1073.74 last 10.97) - open/close 15689 (time 1788444134.86 total 1083.74 last 17.89) - open/close 16432 (time 1788444144.90 total 1093.79 last 74.00) - open/close 17135 (time 1788444154.91 total 1103.80 last 70.21) - open/close 17774 (time 1788444164.93 total 1113.81 last 63.82) - open/close 18478 (time 1788444174.93 total 1123.81 last 70.39) - open/close 19050 (time 1788444184.93 total 1133.82 last 57.18) - open/close 19848 (time 1788444194.94 total 1143.83 last 79.72) - open/close 20589 (time 1788444204.94 total 1153.83 last 74.09) - open/close 21272 (time 1788444214.96 total 1163.84 last 68.21) - open/close 22109 (time 1788444224.96 total 1173.84 last 83.67) - open/close 22669 (time 1788444234.97 total 1183.85 last 55.96) - open/close 23245 (time 1788444244.97 total 1193.85 last 57.58) - open/close 23871 (time 1788444254.99 total 1203.87 last 62.49) - open/close 24568 (time 1788444265.02 total 1213.91 last 69.46) - open/close 25228 (time 1788444275.03 total 1223.92 last 65.94) - open/close 25927 (time 1788444285.06 total 1233.95 last 69.69) - open/close 26454 (time 1788444295.07 total 1243.95 last 52.67) - open/close 26941 (time 1788444305.09 total 1253.97 last 48.59) - open/close 27345 (time 1788444315.09 total 1263.98 last 40.38) - open/close 27923 (time 1788444325.10 total 1273.99 last 57.75) - open/close 28463 (time 1788444335.13 total 1284.01 last 53.86) - open/close 29054 (time 1788444345.14 total 1294.02 last 59.06) - open/close 29649 (time 1788444355.15 total 1304.03 last 59.43) - open/close 30000 (time 1788444360.89 total 1309.77 last 61.18) - open/close 30682 (time 1788444370.91 total 1319.79 last 68.06) - open/close 31402 (time 1788444380.92 total 1329.80 last 71.92) - open/close 32046 (time 1788444390.93 total 1339.81 last 64.33) - open/close 32778 (time 1788444400.93 total 1349.82 last 73.16) - open/close 33457 (time 1788444410.98 total 1359.86 last 67.60) - open/close 34105 (time 1788444420.99 total 1369.88 last 64.70) - open/close 34674 (time 1788444431.01 total 1379.90 last 56.80) - open/close 35229 (time 1788444441.02 total 1389.90 last 55.45) - open/close 35799 (time 1788444451.04 total 1399.92 last 56.92) - open/close 36558 (time 1788444461.04 total 1409.92 last 75.87) - open/close 37167 (time 1788444471.05 total 1419.94 last 60.82) - open/close 37901 (time 1788444481.05 total 1429.94 last 73.38) - open/close 38552 (time 1788444491.06 total 1439.95 last 65.03) - open/close 39247 (time 1788444501.07 total 1449.95 last 69.46) - open/close 39927 (time 1788444511.07 total 1459.95 last 68.00) - open/close 40615 (time 1788444521.09 total 1469.97 last 68.67) - open/close 41275 (time 1788444531.10 total 1479.99 last 65.91) - open/close 41904 (time 1788444541.10 total 1489.99 last 62.89) - open/close 42454 (time 1788444551.11 total 1499.99 last 54.98) - open/close 43152 (time 1788444561.12 total 1510.00 last 69.73) - open/close 43962 (time 1788444571.14 total 1520.02 last 80.82) - open/close 44681 (time 1788444581.15 total 1530.03 last 71.86) - open/close 45375 (time 1788444591.15 total 1540.04 last 69.35) - open/close 46003 (time 1788444601.16 total 1550.04 last 62.76) - open/close 46599 (time 1788444611.16 total 1560.05 last 59.57) - open/close 47068 (time 1788444621.17 total 1570.05 last 46.87) - open/close 47672 (time 1788444631.20 total 1580.08 last 60.23) - open/close 48270 (time 1788444641.20 total 1590.08 last 59.79) - open/close 48945 (time 1788444651.22 total 1600.11 last 67.34) - open/close 49610 (time 1788444661.23 total 1610.11 last 66.48) total: 50000 open/close in 1615.07 seconds: 30.96 ops/second - unlinked 0 (time 1788444671 ; total 1 ; last 1) - unlinked 10000 (time 1788444844 ; total 174 ; last 173) - unlinked 20000 (time 1788445010 ; total 340 ; last 166) - unlinked 30000 (time 1788445169 ; total 499 ; last 159) - unlinked 40000 (time 1788445340 ; total 670 ; last 171) total: 50000 unlinks in 837 seconds: 59.737156 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg132-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server unloading modules via unload_modules_local on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing unload_modules_local oleg132-server: kmemleak disabled modules unloaded. pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2 pdsh@oleg132-client: oleg132-client: ssh exited with exit code 2 pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2 pdsh@oleg132-client: oleg132-client: ssh exited with exit code 2