== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:45:03 (1788835503) start mds service on oleg656-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 7 17:47 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg656-server' oleg656-server: oleg656-server.virtnet: executing load_modules_local oleg656-server: kmemleak disabled oleg656-server: Loading modules from /home/green/git/lustre-release/lustre oleg656-server: detected 4 online CPUs by sysfs oleg656-server: MODOPTS_LIBCFS= oleg656-server: Force libcfs to create 2 CPU partitions oleg656-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg656-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg656-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg656-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg656-server: oleg656-server.virtnet: executing set_default_debug -1 all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg656-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg656-server: oleg656-server.virtnet: executing set_default_debug -1 all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-OST0000 oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg656-client.virtnet: -o user_xattr,flock 192.168.206.156@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 252 (time 1788835600.19 total 10.01 last 25.18) - open/close 489 (time 1788835610.65 total 20.47 last 22.66) - open/close 714 (time 1788835620.65 total 30.47 last 22.48) - open/close 921 (time 1788835631.32 total 41.14 last 19.41) - open/close 1074 (time 1788835641.75 total 51.57 last 14.67) - open/close 1164 (time 1788835652.06 total 61.88 last 8.73) - open/close 1273 (time 1788835662.76 total 72.58 last 10.18) - open/close 1504 (time 1788835672.76 total 82.58 last 23.09) - open/close 1636 (time 1788835683.22 total 93.04 last 12.63) - open/close 1676 (time 1788835694.58 total 104.40 last 3.52) - open/close 1835 (time 1788835704.58 total 114.40 last 15.89) - open/close 1982 (time 1788835715.09 total 124.91 last 13.99) - open/close 2123 (time 1788835726.11 total 135.93 last 12.80) - open/close 2353 (time 1788835736.14 total 145.96 last 22.93) - open/close 2477 (time 1788835746.16 total 155.98 last 12.37) - open/close 2675 (time 1788835756.18 total 166.00 last 19.76) - open/close 2851 (time 1788835766.50 total 176.32 last 17.06) - open/close 2989 (time 1788835776.50 total 186.32 last 13.79) - open/close 3221 (time 1788835786.51 total 196.33 last 23.19) - open/close 3382 (time 1788835797.40 total 207.22 last 14.78) - open/close 3520 (time 1788835808.10 total 217.92 last 12.90) - open/close 3644 (time 1788835818.21 total 228.03 last 12.27) - open/close 3826 (time 1788835829.27 total 239.09 last 16.44) - open/close 4065 (time 1788835839.34 total 249.16 last 23.74) - open/close 4182 (time 1788835849.36 total 259.18 last 11.68) - open/close 4330 (time 1788835859.92 total 269.74 last 14.01) - open/close 4550 (time 1788835870.89 total 280.71 last 20.05) - open/close 4815 (time 1788835881.67 total 291.49 last 24.59) - open/close 4953 (time 1788835891.86 total 301.68 last 13.54) - open/close 5092 (time 1788835902.48 total 312.30 last 13.09) - open/close 5157 (time 1788835912.51 total 322.33 last 6.48) - open/close 5276 (time 1788835923.02 total 332.84 last 11.32) - open/close 5510 (time 1788835933.13 total 342.95 last 23.13) - open/close 5695 (time 1788835943.15 total 352.97 last 18.47) - open/close 5951 (time 1788835953.56 total 363.38 last 24.60) - open/close 6078 (time 1788835963.58 total 373.39 last 12.68) - open/close 6296 (time 1788835974.30 total 384.12 last 20.33) - open/close 6496 (time 1788835984.30 total 394.12 last 19.99) - open/close 6692 (time 1788835994.46 total 404.28 last 19.30) - open/close 6945 (time 1788836004.47 total 414.29 last 25.28) - open/close 7138 (time 1788836014.69 total 424.51 last 18.89) - open/close 7324 (time 1788836024.69 total 434.51 last 18.59) - open/close 7534 (time 1788836035.04 total 444.86 last 20.31) - open/close 7648 (time 1788836045.73 total 455.54 last 10.66) - open/close 7867 (time 1788836055.73 total 465.55 last 21.89) - open/close 8070 (time 1788836065.74 total 475.56 last 20.28) - open/close 8266 (time 1788836075.75 total 485.57 last 19.59) - open/close 8515 (time 1788836086.25 total 496.07 last 23.72) - open/close 8728 (time 1788836096.49 total 506.31 last 20.80) - open/close 8938 (time 1788836106.83 total 516.65 last 20.30) - open/close 9036 (time 1788836117.39 total 527.21 last 9.28) - open/close 9197 (time 1788836128.59 total 538.40 last 14.38) - open/close 9334 (time 1788836138.61 total 548.43 last 13.67) - open/close 9533 (time 1788836148.63 total 558.45 last 19.85) - open/close 9781 (time 1788836159.65 total 569.47 last 22.51) - open/close 9989 (time 1788836169.76 total 579.58 last 20.58) - open/close 10182 (time 1788836179.81 total 589.63 last 19.21) - open/close 10431 (time 1788836190.76 total 600.58 last 22.74) - open/close 10681 (time 1788836200.76 total 610.58 last 24.99) - open/close 10955 (time 1788836210.77 total 620.59 last 27.38) - open/close 11159 (time 1788836221.21 total 631.03 last 19.53) - open/close 11376 (time 1788836232.08 total 641.90 last 19.97) - open/close 11623 (time 1788836242.09 total 651.91 last 24.68) - open/close 11887 (time 1788836252.89 total 662.71 last 24.44) - open/close 12069 (time 1788836262.89 total 672.71 last 18.19) - open/close 12306 (time 1788836273.90 total 683.72 last 21.54) - open/close 12406 (time 1788836284.71 total 694.53 last 9.25) - open/close 12457 (time 1788836295.45 total 705.27 last 4.75) - open/close 12579 (time 1788836306.53 total 716.35 last 11.01) - open/close 12710 (time 1788836316.54 total 726.36 last 13.10) - open/close 12878 (time 1788836327.13 total 736.95 last 15.87) - open/close 13171 (time 1788836337.13 total 746.95 last 29.28) - open/close 13402 (time 1788836347.18 total 757.00 last 23.01) - open/close 13611 (time 1788836357.45 total 767.27 last 20.33) - open/close 13868 (time 1788836367.46 total 777.28 last 25.69) - open/close 14040 (time 1788836378.00 total 787.82 last 16.31) - open/close 14331 (time 1788836389.01 total 798.83 last 26.43) - open/close 14541 (time 1788836399.24 total 809.06 last 20.52) - open/close 14708 (time 1788836410.07 total 819.89 last 15.42) - open/close 14964 (time 1788836420.50 total 830.32 last 24.55) - open/close 15218 (time 1788836431.32 total 841.14 last 23.47) - open/close 15383 (time 1788836441.57 total 851.39 last 16.09) - open/close 15611 (time 1788836451.62 total 861.44 last 22.70) - open/close 15758 (time 1788836462.71 total 872.53 last 13.25) - open/close 15905 (time 1788836473.04 total 882.86 last 14.23) - open/close 16160 (time 1788836483.98 total 893.80 last 23.29) - open/close 16427 (time 1788836495.08 total 904.90 last 24.06) - open/close 16546 (time 1788836505.57 total 915.39 last 11.35) - open/close 16677 (time 1788836515.58 total 925.40 last 13.09) - open/close 16810 (time 1788836525.58 total 935.40 last 13.29) - open/close 17046 (time 1788836535.69 total 945.51 last 23.35) - open/close 17265 (time 1788836545.70 total 955.52 last 21.88) - open/close 17525 (time 1788836556.50 total 966.32 last 24.07) - open/close 17735 (time 1788836566.50 total 976.32 last 20.99) - open/close 17930 (time 1788836576.55 total 986.37 last 19.41) - open/close 18576 (time 1788836586.56 total 996.38 last 64.55) - open/close 19305 (time 1788836596.57 total 1006.39 last 72.85) - open/close 20000 (time 1788836606.55 total 1016.37 last 69.59) - open/close 20680 (time 1788836616.58 total 1026.40 last 67.80) - open/close 21378 (time 1788836626.59 total 1036.41 last 69.76) - open/close 22095 (time 1788836636.60 total 1046.42 last 71.60) - open/close 22805 (time 1788836646.61 total 1056.43 last 70.92) - open/close 23466 (time 1788836656.62 total 1066.44 last 66.03) - open/close 24176 (time 1788836666.62 total 1076.44 last 70.99) - open/close 24846 (time 1788836676.63 total 1086.45 last 66.97) - open/close 25606 (time 1788836686.63 total 1096.45 last 75.98) - open/close 26281 (time 1788836696.64 total 1106.45 last 67.47) - open/close 26976 (time 1788836706.66 total 1116.48 last 69.34) - open/close 27679 (time 1788836716.66 total 1126.48 last 70.27) - open/close 28419 (time 1788836726.67 total 1136.49 last 73.92) - open/close 29144 (time 1788836736.68 total 1146.50 last 72.46) - open/close 29898 (time 1788836746.70 total 1156.52 last 75.27) - open/close 30616 (time 1788836756.71 total 1166.53 last 71.68) - open/close 31305 (time 1788836766.72 total 1176.54 last 68.86) - open/close 32021 (time 1788836776.72 total 1186.54 last 71.56) - open/close 32720 (time 1788836786.73 total 1196.55 last 69.85) - open/close 33384 (time 1788836796.74 total 1206.56 last 66.32) - open/close 34081 (time 1788836806.75 total 1216.57 last 69.65) - open/close 34763 (time 1788836816.77 total 1226.59 last 68.07) - open/close 35372 (time 1788836826.78 total 1236.60 last 60.87) - open/close 36046 (time 1788836836.79 total 1246.61 last 67.28) - open/close 36740 (time 1788836846.80 total 1256.62 last 69.36) - open/close 37396 (time 1788836856.80 total 1266.62 last 65.57) - open/close 38101 (time 1788836866.82 total 1276.64 last 70.41) - open/close 38862 (time 1788836876.84 total 1286.66 last 75.96) - open/close 39577 (time 1788836886.85 total 1296.67 last 71.43) - open/close 40000 (time 1788836892.56 total 1302.38 last 74.01) - open/close 40717 (time 1788836902.57 total 1312.39 last 71.62) - open/close 41384 (time 1788836912.58 total 1322.40 last 66.64) - open/close 42058 (time 1788836922.62 total 1332.44 last 67.18) - open/close 42772 (time 1788836932.63 total 1342.45 last 71.32) - open/close 43523 (time 1788836942.63 total 1352.45 last 75.09) - open/close 44226 (time 1788836952.63 total 1362.45 last 70.30) - open/close 44952 (time 1788836962.64 total 1372.46 last 72.55) - open/close 45662 (time 1788836972.65 total 1382.47 last 70.87) - open/close 46416 (time 1788836982.66 total 1392.48 last 75.31) - open/close 46976 (time 1788836992.67 total 1402.49 last 55.99) - open/close 47550 (time 1788837002.67 total 1412.49 last 57.39) - open/close 48180 (time 1788837012.68 total 1422.50 last 62.92) - open/close 48822 (time 1788837022.69 total 1432.51 last 64.13) - open/close 49438 (time 1788837032.71 total 1442.53 last 61.50) total: 50000 open/close in 1450.37 seconds: 34.47 ops/second - unlinked 0 (time 1788837045 ; total 0 ; last 0) - unlinked 10000 (time 1788837191 ; total 146 ; last 146) - unlinked 20000 (time 1788837346 ; total 301 ; last 155) - unlinked 30000 (time 1788837505 ; total 460 ; last 159) - unlinked 40000 (time 1788837658 ; total 613 ; last 153) total: 50000 unlinks in 767 seconds: 65.189049 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg656-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg656-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg656-server stop mds service on oleg656-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg656-server unloading modules via unload_modules_local on: 'oleg656-server' oleg656-server: oleg656-server.virtnet: executing unload_modules_local oleg656-server: kmemleak disabled modules unloaded. pdsh@oleg656-client: oleg656-client: ssh exited with exit code 2 pdsh@oleg656-client: oleg656-server: ssh exited with exit code 2 pdsh@oleg656-client: oleg656-client: ssh exited with exit code 2 pdsh@oleg656-client: oleg656-server: ssh exited with exit code 2