== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:27:15 (1789266435) start mds service on oleg106-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 12 19:11 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg106-server' oleg106-server: oleg106-server.virtnet: executing load_modules_local oleg106-server: kmemleak disabled oleg106-server: Loading modules from /home/green/git/lustre-release/lustre oleg106-server: detected 4 online CPUs by sysfs oleg106-server: MODOPTS_LIBCFS= oleg106-server: Force libcfs to create 2 CPU partitions oleg106-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg106-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg106-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg106-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg106-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-OST0000 oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 105 (time 1789266526.71 total 10.02 last 10.48) - open/close 268 (time 1789266537.55 total 20.86 last 15.04) - open/close 425 (time 1789266547.56 total 30.87 last 15.68) - open/close 604 (time 1789266557.57 total 40.88 last 17.89) - open/close 796 (time 1789266568.84 total 52.15 last 17.03) - open/close 1028 (time 1789266578.84 total 62.15 last 23.19) - open/close 1199 (time 1789266589.15 total 72.45 last 16.60) - open/close 1329 (time 1789266599.76 total 83.07 last 12.24) - open/close 1443 (time 1789266609.80 total 93.11 last 11.36) - open/close 1522 (time 1789266620.50 total 103.81 last 7.38) - open/close 1708 (time 1789266630.52 total 113.82 last 18.58) - open/close 1932 (time 1789266640.53 total 123.84 last 22.36) - open/close 2033 (time 1789266651.27 total 134.58 last 9.40) - open/close 2223 (time 1789266661.30 total 144.61 last 18.95) - open/close 2387 (time 1789266671.32 total 154.63 last 16.37) - open/close 2417 (time 1789266684.45 total 167.76 last 2.28) - open/close 2625 (time 1789266694.50 total 177.80 last 20.70) - open/close 2826 (time 1789266705.17 total 188.48 last 18.83) - open/close 2985 (time 1789266715.33 total 198.64 last 15.65) - open/close 3088 (time 1789266728.39 total 211.70 last 7.89) - open/close 3195 (time 1789266738.40 total 221.71 last 10.69) - open/close 3259 (time 1789266748.42 total 231.72 last 6.39) - open/close 3315 (time 1789266759.36 total 242.67 last 5.12) - open/close 3568 (time 1789266769.50 total 252.81 last 24.96) - open/close 3793 (time 1789266779.51 total 262.82 last 22.47) - open/close 4035 (time 1789266789.52 total 272.83 last 24.17) - open/close 4253 (time 1789266802.37 total 285.68 last 16.96) - open/close 4455 (time 1789266812.39 total 295.70 last 20.17) - open/close 4608 (time 1789266822.56 total 305.87 last 15.04) - open/close 4782 (time 1789266832.58 total 315.89 last 17.37) - open/close 4929 (time 1789266844.82 total 328.13 last 12.01) - open/close 5114 (time 1789266854.84 total 338.15 last 18.46) - open/close 5216 (time 1789266866.64 total 349.95 last 8.65) - open/close 5432 (time 1789266878.66 total 361.97 last 17.97) - open/close 5572 (time 1789266889.17 total 372.47 last 13.33) - open/close 5693 (time 1789266899.17 total 382.48 last 12.09) - open/close 5815 (time 1789266909.25 total 392.56 last 12.11) - open/close 5956 (time 1789266919.26 total 402.56 last 14.09) - open/close 6077 (time 1789266930.21 total 413.52 last 11.05) - open/close 6185 (time 1789266941.26 total 424.57 last 9.77) - open/close 6278 (time 1789266951.26 total 434.57 last 9.29) - open/close 6424 (time 1789266961.27 total 444.58 last 14.59) - open/close 6527 (time 1789266972.31 total 455.62 last 9.33) - open/close 6634 (time 1789266982.34 total 465.65 last 10.67) - open/close 6793 (time 1789266992.39 total 475.70 last 15.82) - open/close 6904 (time 1789267002.40 total 485.71 last 11.09) - open/close 7089 (time 1789267012.40 total 495.71 last 18.50) - open/close 7302 (time 1789267022.98 total 506.29 last 20.13) - open/close 7385 (time 1789267033.05 total 516.36 last 8.25) - open/close 7548 (time 1789267043.05 total 526.36 last 16.30) - open/close 7662 (time 1789267053.07 total 536.38 last 11.38) - open/close 7782 (time 1789267064.98 total 548.29 last 10.07) - open/close 7843 (time 1789267074.99 total 558.30 last 6.10) - open/close 7979 (time 1789267085.85 total 569.15 last 12.53) - open/close 8186 (time 1789267096.64 total 579.94 last 19.19) - open/close 8365 (time 1789267106.64 total 589.95 last 17.89) - open/close 8510 (time 1789267117.20 total 600.51 last 13.74) - open/close 8573 (time 1789267127.24 total 610.55 last 6.28) - open/close 8748 (time 1789267137.56 total 620.87 last 16.95) - open/close 8988 (time 1789267147.68 total 630.98 last 23.73) - open/close 9250 (time 1789267158.08 total 641.39 last 25.18) - open/close 9513 (time 1789267168.10 total 651.41 last 26.25) - open/close 9728 (time 1789267178.96 total 662.26 last 19.80) - open/close 9895 (time 1789267190.73 total 674.04 last 14.18) - open/close 10000 (time 1789267200.72 total 684.03 last 10.52) - open/close 10111 (time 1789267210.74 total 694.05 last 11.08) - open/close 10318 (time 1789267221.67 total 704.98 last 18.94) - open/close 10543 (time 1789267231.69 total 714.99 last 22.46) - open/close 10679 (time 1789267241.72 total 725.03 last 13.56) - open/close 10782 (time 1789267252.69 total 736.00 last 9.38) - open/close 10983 (time 1789267263.77 total 747.08 last 18.15) - open/close 11139 (time 1789267273.78 total 757.09 last 15.58) - open/close 11359 (time 1789267283.88 total 767.19 last 21.79) - open/close 11611 (time 1789267294.66 total 777.97 last 23.37) - open/close 11814 (time 1789267305.51 total 788.81 last 18.72) - open/close 11926 (time 1789267316.43 total 799.74 last 10.25) - open/close 12109 (time 1789267327.06 total 810.37 last 17.21) - open/close 12264 (time 1789267337.80 total 821.11 last 14.43) - open/close 12444 (time 1789267347.85 total 831.16 last 17.92) - open/close 12635 (time 1789267358.55 total 841.86 last 17.85) - open/close 12843 (time 1789267368.97 total 852.28 last 19.96) - open/close 12940 (time 1789267378.97 total 862.28 last 9.70) - open/close 13067 (time 1789267388.99 total 872.30 last 12.67) - open/close 13270 (time 1789267399.75 total 883.06 last 18.87) - open/close 13399 (time 1789267409.76 total 893.07 last 12.89) - open/close 13509 (time 1789267419.77 total 903.08 last 10.98) - open/close 13646 (time 1789267430.21 total 913.52 last 13.12) - open/close 13802 (time 1789267440.49 total 923.80 last 15.18) - open/close 13920 (time 1789267451.07 total 934.38 last 11.15) - open/close 14080 (time 1789267461.09 total 944.40 last 15.97) - open/close 14219 (time 1789267472.15 total 955.46 last 12.57) - open/close 14423 (time 1789267483.01 total 966.32 last 18.78) - open/close 14507 (time 1789267493.44 total 976.75 last 8.05) - open/close 14681 (time 1789267503.51 total 986.82 last 17.28) - open/close 14861 (time 1789267513.51 total 996.82 last 18.00) - open/close 15070 (time 1789267524.42 total 1007.73 last 19.16) - open/close 15192 (time 1789267534.94 total 1018.25 last 11.59) - open/close 15273 (time 1789267545.30 total 1028.61 last 7.82) - open/close 15433 (time 1789267555.80 total 1039.11 last 15.24) - open/close 15532 (time 1789267565.82 total 1049.12 last 9.88) - open/close 15753 (time 1789267576.16 total 1059.47 last 21.36) - open/close 15867 (time 1789267587.48 total 1070.79 last 10.07) - open/close 16037 (time 1789267598.49 total 1081.80 last 15.43) - open/close 16690 (time 1789267608.50 total 1091.81 last 65.24) - open/close 17312 (time 1789267618.51 total 1101.82 last 62.17) - open/close 17930 (time 1789267628.51 total 1111.82 last 61.76) - open/close 18565 (time 1789267638.53 total 1121.84 last 63.40) - open/close 19212 (time 1789267648.54 total 1131.85 last 64.65) - open/close 19778 (time 1789267658.54 total 1141.85 last 56.56) - open/close 20000 (time 1789267661.84 total 1145.15 last 67.39) - open/close 20622 (time 1789267671.87 total 1155.18 last 62.01) - open/close 21197 (time 1789267681.89 total 1165.19 last 57.40) - open/close 21870 (time 1789267691.90 total 1175.20 last 67.24) - open/close 22485 (time 1789267701.90 total 1185.21 last 61.46) - open/close 23111 (time 1789267711.92 total 1195.23 last 62.47) - open/close 23674 (time 1789267721.93 total 1205.24 last 56.27) - open/close 24238 (time 1789267731.93 total 1215.24 last 56.36) - open/close 24812 (time 1789267741.94 total 1225.25 last 57.35) - open/close 25356 (time 1789267751.95 total 1235.25 last 54.39) - open/close 25908 (time 1789267761.96 total 1245.26 last 55.14) - open/close 26500 (time 1789267771.96 total 1255.27 last 59.16) - open/close 27144 (time 1789267781.96 total 1265.27 last 64.39) - open/close 27846 (time 1789267791.97 total 1275.28 last 70.14) - open/close 28503 (time 1789267801.98 total 1285.29 last 65.65) - open/close 29147 (time 1789267811.99 total 1295.30 last 64.31) - open/close 29835 (time 1789267822.00 total 1305.30 last 68.79) - open/close 30000 (time 1789267824.38 total 1307.69 last 69.08) - open/close 30668 (time 1789267834.41 total 1317.72 last 66.61) - open/close 31218 (time 1789267844.44 total 1327.75 last 54.85) - open/close 31762 (time 1789267854.44 total 1337.75 last 54.38) - open/close 32388 (time 1789267864.45 total 1347.76 last 62.54) - open/close 33086 (time 1789267874.47 total 1357.78 last 69.66) - open/close 33709 (time 1789267884.48 total 1367.79 last 62.26) - open/close 34310 (time 1789267894.49 total 1377.80 last 60.04) - open/close 34956 (time 1789267904.49 total 1387.80 last 64.59) - open/close 35571 (time 1789267914.50 total 1397.80 last 61.47) - open/close 36218 (time 1789267924.50 total 1407.81 last 64.68) - open/close 36899 (time 1789267934.50 total 1417.81 last 68.08) - open/close 37480 (time 1789267944.52 total 1427.83 last 58.01) - open/close 38112 (time 1789267954.52 total 1437.83 last 63.18) - open/close 38468 (time 1789267964.56 total 1447.87 last 35.46) - open/close 38827 (time 1789267974.57 total 1457.88 last 35.88) - open/close 39450 (time 1789267984.57 total 1467.88 last 62.30) - open/close 40000 (time 1789267992.60 total 1475.91 last 68.50) - open/close 40657 (time 1789268002.61 total 1485.92 last 65.64) - open/close 41250 (time 1789268012.62 total 1495.93 last 59.24) - open/close 41889 (time 1789268022.63 total 1505.93 last 63.85) - open/close 42554 (time 1789268032.63 total 1515.94 last 66.47) - open/close 43090 (time 1789268042.64 total 1525.95 last 53.52) - open/close 43741 (time 1789268052.65 total 1535.96 last 65.07) - open/close 44282 (time 1789268062.70 total 1546.01 last 53.79) - open/close 44788 (time 1789268072.72 total 1556.03 last 50.50) - open/close 45193 (time 1789268082.75 total 1566.06 last 40.39) - open/close 45614 (time 1789268092.77 total 1576.08 last 42.01) - open/close 46082 (time 1789268102.78 total 1586.09 last 46.78) - open/close 46761 (time 1789268112.78 total 1596.09 last 67.90) - open/close 47321 (time 1789268122.78 total 1606.09 last 55.98) - open/close 48030 (time 1789268132.79 total 1616.10 last 70.86) - open/close 48687 (time 1789268142.80 total 1626.11 last 65.61) - open/close 49270 (time 1789268152.81 total 1636.12 last 58.23) - open/close 49871 (time 1789268162.82 total 1646.13 last 60.06) total: 50000 open/close in 1648.19 seconds: 30.34 ops/second - unlinked 0 (time 1789268169 ; total 0 ; last 0) - unlinked 10000 (time 1789268327 ; total 158 ; last 158) - unlinked 20000 (time 1789268476 ; total 307 ; last 149) - unlinked 30000 (time 1789268635 ; total 466 ; last 159) - unlinked 40000 (time 1789268794 ; total 625 ; last 159) total: 50000 unlinks in 772 seconds: 64.766838 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg106-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg106-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg106-server stop mds service on oleg106-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg106-server unloading modules via unload_modules_local on: 'oleg106-server' oleg106-server: oleg106-server.virtnet: executing unload_modules_local oleg106-server: kmemleak disabled modules unloaded. pdsh@oleg106-client: oleg106-server: ssh exited with exit code 2 pdsh@oleg106-client: oleg106-client: ssh exited with exit code 2 pdsh@oleg106-client: oleg106-client: ssh exited with exit code 2 pdsh@oleg106-client: oleg106-server: ssh exited with exit code 2