== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:19:06 (1789053546) start mds service on oleg209-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 10 01:48 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing load_modules_local oleg209-server: kmemleak disabled oleg209-server: Loading modules from /home/green/git/lustre-release/lustre oleg209-server: detected 4 online CPUs by sysfs oleg209-server: MODOPTS_LIBCFS= oleg209-server: Force libcfs to create 2 CPU partitions oleg209-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg209-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg209-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg209-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg209-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 149 (time 1789053652.39 total 10.01 last 14.88) - open/close 239 (time 1789053662.82 total 20.44 last 8.63) - open/close 361 (time 1789053672.85 total 30.47 last 12.17) - open/close 531 (time 1789053683.86 total 41.48 last 15.44) - open/close 716 (time 1789053693.87 total 51.49 last 18.49) - open/close 907 (time 1789053704.64 total 62.26 last 17.73) - open/close 1143 (time 1789053715.36 total 72.98 last 22.02) - open/close 1298 (time 1789053726.84 total 84.46 last 13.50) - open/close 1474 (time 1789053736.86 total 94.48 last 17.56) - open/close 1634 (time 1789053746.86 total 104.48 last 16.00) - open/close 1819 (time 1789053756.88 total 114.50 last 18.47) - open/close 2059 (time 1789053767.55 total 125.17 last 22.49) - open/close 2272 (time 1789053777.93 total 135.55 last 20.53) - open/close 2510 (time 1789053788.32 total 145.94 last 22.91) - open/close 2655 (time 1789053800.47 total 158.09 last 11.94) - open/close 2870 (time 1789053810.48 total 168.10 last 21.48) - open/close 3110 (time 1789053820.49 total 178.11 last 23.97) - open/close 3302 (time 1789053831.13 total 188.75 last 18.04) - open/close 3467 (time 1789053842.07 total 199.69 last 15.09) - open/close 3601 (time 1789053852.96 total 210.58 last 12.30) - open/close 3807 (time 1789053862.97 total 220.59 last 20.58) - open/close 3921 (time 1789053875.41 total 233.03 last 9.17) - open/close 4091 (time 1789053885.45 total 243.07 last 16.93) - open/close 4300 (time 1789053895.88 total 253.50 last 20.04) - open/close 4545 (time 1789053905.94 total 263.56 last 24.35) - open/close 4771 (time 1789053915.95 total 273.57 last 22.59) - open/close 4938 (time 1789053926.04 total 283.66 last 16.55) - open/close 5173 (time 1789053936.40 total 294.02 last 22.67) - open/close 5342 (time 1789053946.46 total 304.08 last 16.80) - open/close 5557 (time 1789053956.73 total 314.35 last 20.95) - open/close 5628 (time 1789053967.06 total 324.68 last 6.87) - open/close 5723 (time 1789053977.72 total 335.34 last 8.91) - open/close 5829 (time 1789053987.73 total 345.35 last 10.59) - open/close 6055 (time 1789053998.71 total 356.33 last 20.58) - open/close 6180 (time 1789054008.80 total 366.42 last 12.39) - open/close 6373 (time 1789054019.46 total 377.08 last 18.10) - open/close 6590 (time 1789054031.77 total 389.39 last 17.64) - open/close 6660 (time 1789054045.10 total 402.72 last 5.25) - open/close 6793 (time 1789054055.19 total 412.81 last 13.19) - open/close 6877 (time 1789054065.88 total 423.50 last 7.86) - open/close 6958 (time 1789054076.65 total 434.27 last 7.52) - open/close 7031 (time 1789054086.66 total 444.28 last 7.29) - open/close 7173 (time 1789054097.16 total 454.78 last 13.52) - open/close 7370 (time 1789054107.53 total 465.15 last 19.00) - open/close 7519 (time 1789054118.04 total 475.66 last 14.18) - open/close 7742 (time 1789054128.04 total 485.66 last 22.29) - open/close 7922 (time 1789054138.38 total 496.00 last 17.41) - open/close 7999 (time 1789054149.11 total 506.73 last 7.18) - open/close 8196 (time 1789054159.76 total 517.38 last 18.49) - open/close 8408 (time 1789054170.38 total 528.00 last 19.97) - open/close 8597 (time 1789054180.38 total 538.00 last 18.90) - open/close 8774 (time 1789054190.55 total 548.17 last 17.41) - open/close 9005 (time 1789054200.58 total 558.20 last 23.03) - open/close 9248 (time 1789054210.62 total 568.24 last 24.21) - open/close 9387 (time 1789054220.62 total 578.24 last 13.90) - open/close 9482 (time 1789054230.73 total 588.35 last 9.39) - open/close 9600 (time 1789054240.90 total 598.52 last 11.60) - open/close 9698 (time 1789054250.92 total 608.54 last 9.78) - open/close 9875 (time 1789054260.94 total 618.56 last 17.66) - open/close 10000 (time 1789054269.16 total 626.78 last 15.21) - open/close 10101 (time 1789054279.37 total 636.99 last 9.89) - open/close 10290 (time 1789054292.12 total 649.74 last 14.82) - open/close 10365 (time 1789054302.68 total 660.30 last 7.10) - open/close 10609 (time 1789054313.33 total 670.95 last 22.92) - open/close 10769 (time 1789054323.48 total 681.10 last 15.76) - open/close 10972 (time 1789054333.83 total 691.45 last 19.63) - open/close 11138 (time 1789054343.84 total 701.46 last 16.58) - open/close 11213 (time 1789054354.02 total 711.64 last 7.37) - open/close 11321 (time 1789054364.02 total 721.64 last 10.80) - open/close 11493 (time 1789054374.10 total 731.72 last 17.06) - open/close 11661 (time 1789054384.98 total 742.60 last 15.44) - open/close 11793 (time 1789054394.99 total 752.61 last 13.19) - open/close 11970 (time 1789054405.02 total 762.64 last 17.66) - open/close 12209 (time 1789054415.03 total 772.65 last 23.88) - open/close 12327 (time 1789054425.23 total 782.85 last 11.56) - open/close 12503 (time 1789054437.48 total 795.10 last 14.37) - open/close 12621 (time 1789054448.04 total 805.66 last 11.17) - open/close 12775 (time 1789054458.04 total 815.66 last 15.39) - open/close 12984 (time 1789054468.04 total 825.66 last 20.90) - open/close 13228 (time 1789054478.51 total 836.13 last 23.32) - open/close 13377 (time 1789054489.47 total 847.09 last 13.60) - open/close 13536 (time 1789054499.79 total 857.41 last 15.40) - open/close 13615 (time 1789054511.11 total 868.73 last 6.98) - open/close 13800 (time 1789054521.93 total 879.55 last 17.09) - open/close 13922 (time 1789054532.19 total 889.81 last 11.90) - open/close 14073 (time 1789054543.00 total 900.62 last 13.97) - open/close 14190 (time 1789054553.01 total 910.63 last 11.69) - open/close 14330 (time 1789054563.17 total 920.79 last 13.78) - open/close 14484 (time 1789054574.19 total 931.81 last 13.97) - open/close 14565 (time 1789054585.00 total 942.62 last 7.50) - open/close 14748 (time 1789054595.94 total 953.56 last 16.73) - open/close 14935 (time 1789054607.01 total 964.63 last 16.90) - open/close 15129 (time 1789054619.56 total 977.18 last 15.46) - open/close 15283 (time 1789054631.92 total 989.54 last 12.46) - open/close 15527 (time 1789054641.92 total 999.54 last 24.39) - open/close 15705 (time 1789054651.92 total 1009.54 last 17.80) - open/close 15879 (time 1789054661.99 total 1019.61 last 17.28) - open/close 16071 (time 1789054672.14 total 1029.76 last 18.92) - open/close 16180 (time 1789054682.26 total 1039.88 last 10.78) - open/close 16222 (time 1789054692.70 total 1050.32 last 4.02) - open/close 16370 (time 1789054702.92 total 1060.54 last 14.48) - open/close 16573 (time 1789054712.93 total 1070.55 last 20.28) - open/close 17062 (time 1789054722.96 total 1080.58 last 48.77) - open/close 17560 (time 1789054732.97 total 1090.59 last 49.76) - open/close 18200 (time 1789054742.97 total 1100.59 last 63.99) - open/close 18908 (time 1789054752.98 total 1110.60 last 70.73) - open/close 19596 (time 1789054762.98 total 1120.60 last 68.79) - open/close 20000 (time 1789054769.57 total 1127.19 last 61.32) - open/close 20643 (time 1789054779.57 total 1137.19 last 64.28) - open/close 21281 (time 1789054789.58 total 1147.20 last 63.72) - open/close 21875 (time 1789054799.59 total 1157.21 last 59.38) - open/close 22359 (time 1789054809.59 total 1167.21 last 48.36) - open/close 22854 (time 1789054819.61 total 1177.23 last 49.41) - open/close 23584 (time 1789054829.62 total 1187.24 last 72.94) - open/close 24225 (time 1789054839.63 total 1197.25 last 64.04) - open/close 24904 (time 1789054849.64 total 1207.26 last 67.82) - open/close 25582 (time 1789054859.65 total 1217.27 last 67.75) - open/close 26301 (time 1789054869.66 total 1227.28 last 71.84) - open/close 27086 (time 1789054879.66 total 1237.28 last 78.43) - open/close 27832 (time 1789054889.67 total 1247.29 last 74.53) - open/close 28456 (time 1789054899.68 total 1257.30 last 62.35) - open/close 29044 (time 1789054909.71 total 1267.33 last 58.62) - open/close 29585 (time 1789054919.73 total 1277.35 last 53.99) - open/close 30000 (time 1789054927.68 total 1285.30 last 52.20) - open/close 30602 (time 1789054937.69 total 1295.31 last 60.19) - open/close 31381 (time 1789054947.70 total 1305.32 last 77.82) - open/close 32056 (time 1789054957.70 total 1315.32 last 67.44) - open/close 32696 (time 1789054967.71 total 1325.33 last 63.97) - open/close 33319 (time 1789054977.71 total 1335.33 last 62.28) - open/close 33955 (time 1789054987.72 total 1345.34 last 63.56) - open/close 34486 (time 1789054997.73 total 1355.35 last 53.04) - open/close 35195 (time 1789055007.75 total 1365.37 last 70.76) - open/close 35862 (time 1789055017.76 total 1375.38 last 66.63) - open/close 36530 (time 1789055027.77 total 1385.39 last 66.73) - open/close 37078 (time 1789055037.78 total 1395.40 last 54.75) - open/close 37736 (time 1789055047.78 total 1405.40 last 65.79) - open/close 38434 (time 1789055057.79 total 1415.41 last 69.74) - open/close 39018 (time 1789055067.82 total 1425.44 last 58.26) - open/close 39673 (time 1789055077.85 total 1435.47 last 65.31) - open/close 40000 (time 1789055082.96 total 1440.58 last 63.96) - open/close 40683 (time 1789055092.97 total 1450.59 last 68.21) - open/close 41277 (time 1789055102.99 total 1460.61 last 59.32) - open/close 41734 (time 1789055112.99 total 1470.61 last 45.70) - open/close 42236 (time 1789055123.00 total 1480.62 last 50.15) - open/close 42834 (time 1789055133.00 total 1490.62 last 59.77) - open/close 43421 (time 1789055143.06 total 1500.68 last 58.37) - open/close 44018 (time 1789055153.06 total 1510.68 last 59.67) - open/close 44576 (time 1789055163.06 total 1520.69 last 55.80) - open/close 45174 (time 1789055173.08 total 1530.70 last 59.72) - open/close 45770 (time 1789055183.09 total 1540.71 last 59.54) - open/close 46486 (time 1789055193.10 total 1550.72 last 71.54) - open/close 47217 (time 1789055203.10 total 1560.72 last 73.04) - open/close 48087 (time 1789055213.11 total 1570.73 last 86.94) - open/close 48808 (time 1789055223.12 total 1580.74 last 72.03) - open/close 49493 (time 1789055233.13 total 1590.75 last 68.44) total: 50000 open/close in 1597.74 seconds: 31.29 ops/second - unlinked 0 (time 1789055245 ; total 0 ; last 0) - unlinked 10000 (time 1789055400 ; total 155 ; last 155) - unlinked 20000 (time 1789055569 ; total 324 ; last 169) - unlinked 30000 (time 1789055725 ; total 480 ; last 156) - unlinked 40000 (time 1789055904 ; total 659 ; last 179) total: 50000 unlinks in 828 seconds: 60.386475 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg209-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg209-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server stop mds service on oleg209-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg209-server unloading modules via unload_modules_local on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing unload_modules_local oleg209-server: kmemleak disabled modules unloaded. pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2