== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:23:54 (1789781034) start mds service on oleg132-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 18 15:30 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing load_modules_local oleg132-server: kmemleak disabled oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: detected 4 online CPUs by sysfs oleg132-server: MODOPTS_LIBCFS= oleg132-server: Force libcfs to create 2 CPU partitions oleg132-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg132-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg132-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg132-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg132-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0000 oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 132 (time 1789781126.47 total 10.18 last 12.97) - open/close 348 (time 1789781136.48 total 20.19 last 21.57) - open/close 589 (time 1789781147.20 total 30.91 last 22.49) - open/close 782 (time 1789781157.20 total 40.91 last 19.29) - open/close 1013 (time 1789781167.75 total 51.47 last 21.89) - open/close 1165 (time 1789781177.76 total 61.47 last 15.19) - open/close 1383 (time 1789781187.85 total 71.56 last 21.60) - open/close 1613 (time 1789781199.36 total 83.07 last 19.98) - open/close 1875 (time 1789781209.84 total 93.55 last 25.01) - open/close 2110 (time 1789781220.73 total 104.44 last 21.57) - open/close 2346 (time 1789781231.49 total 115.20 last 21.93) - open/close 2546 (time 1789781242.23 total 125.95 last 18.61) - open/close 2808 (time 1789781252.26 total 135.97 last 26.13) - open/close 3104 (time 1789781263.28 total 146.99 last 26.85) - open/close 3318 (time 1789781273.59 total 157.30 last 20.76) - open/close 3544 (time 1789781284.60 total 168.31 last 20.54) - open/close 3703 (time 1789781295.54 total 179.25 last 14.53) - open/close 3887 (time 1789781306.52 total 190.23 last 16.76) - open/close 4095 (time 1789781316.52 total 200.23 last 20.80) - open/close 4292 (time 1789781326.52 total 210.23 last 19.69) - open/close 4511 (time 1789781337.19 total 220.90 last 20.53) - open/close 4788 (time 1789781347.19 total 230.90 last 27.69) - open/close 4975 (time 1789781357.62 total 241.33 last 17.92) - open/close 5139 (time 1789781367.65 total 251.36 last 16.35) - open/close 5353 (time 1789781378.69 total 262.40 last 19.39) - open/close 5552 (time 1789781388.70 total 272.41 last 19.88) - open/close 5739 (time 1789781398.71 total 282.42 last 18.68) - open/close 5952 (time 1789781409.21 total 292.92 last 20.30) - open/close 6236 (time 1789781420.15 total 303.86 last 25.95) - open/close 6433 (time 1789781430.34 total 314.05 last 19.34) - open/close 6591 (time 1789781442.23 total 325.94 last 13.28) - open/close 6736 (time 1789781452.27 total 335.98 last 14.44) - open/close 6891 (time 1789781463.17 total 346.88 last 14.22) - open/close 7075 (time 1789781473.20 total 356.91 last 18.35) - open/close 7242 (time 1789781483.22 total 366.93 last 16.66) - open/close 7361 (time 1789781494.26 total 377.97 last 10.78) - open/close 7626 (time 1789781504.27 total 387.98 last 26.49) - open/close 7855 (time 1789781514.41 total 398.12 last 22.57) - open/close 8101 (time 1789781524.86 total 408.57 last 23.56) - open/close 8328 (time 1789781535.94 total 419.65 last 20.49) - open/close 8466 (time 1789781545.96 total 429.67 last 13.77) - open/close 8790 (time 1789781555.96 total 439.67 last 32.38) - open/close 8998 (time 1789781566.02 total 449.73 last 20.67) - open/close 9194 (time 1789781576.03 total 459.74 last 19.59) - open/close 9385 (time 1789781586.03 total 469.74 last 19.09) - open/close 9564 (time 1789781598.00 total 481.71 last 14.96) - open/close 9655 (time 1789781608.46 total 492.17 last 8.70) - open/close 9819 (time 1789781618.81 total 502.52 last 15.83) - open/close 10000 (time 1789781627.98 total 511.69 last 19.74) - open/close 10194 (time 1789781638.01 total 521.72 last 19.35) - open/close 10472 (time 1789781648.02 total 531.73 last 27.77) - open/close 10701 (time 1789781658.68 total 542.39 last 21.49) - open/close 10892 (time 1789781669.44 total 553.15 last 17.75) - open/close 11045 (time 1789781680.39 total 564.10 last 13.97) - open/close 11252 (time 1789781691.32 total 575.03 last 18.93) - open/close 11438 (time 1789781701.33 total 585.04 last 18.58) - open/close 11587 (time 1789781711.57 total 595.28 last 14.55) - open/close 11712 (time 1789781721.58 total 605.29 last 12.49) - open/close 11807 (time 1789781731.97 total 615.68 last 9.14) - open/close 12044 (time 1789781741.97 total 625.68 last 23.69) - open/close 12195 (time 1789781752.38 total 636.09 last 14.51) - open/close 12396 (time 1789781762.55 total 646.26 last 19.77) - open/close 12672 (time 1789781773.55 total 657.26 last 25.08) - open/close 12856 (time 1789781784.39 total 668.10 last 16.98) - open/close 13134 (time 1789781796.29 total 680.00 last 23.37) - open/close 13352 (time 1789781806.29 total 690.00 last 21.79) - open/close 13463 (time 1789781816.30 total 700.01 last 11.09) - open/close 13637 (time 1789781826.71 total 710.42 last 16.71) - open/close 13881 (time 1789781837.63 total 721.34 last 22.33) - open/close 14066 (time 1789781848.39 total 732.10 last 17.20) - open/close 14226 (time 1789781858.57 total 742.28 last 15.71) - open/close 14411 (time 1789781869.57 total 753.28 last 16.82) - open/close 14601 (time 1789781879.60 total 763.31 last 18.95) - open/close 14758 (time 1789781889.68 total 773.39 last 15.58) - open/close 14996 (time 1789781899.68 total 783.39 last 23.80) - open/close 15093 (time 1789781909.68 total 793.40 last 9.70) - open/close 15240 (time 1789781919.69 total 803.40 last 14.70) - open/close 15422 (time 1789781929.75 total 813.47 last 18.08) - open/close 15618 (time 1789781939.76 total 823.47 last 19.59) - open/close 15833 (time 1789781950.08 total 833.79 last 20.83) - open/close 15967 (time 1789781961.03 total 844.74 last 12.24) - open/close 16062 (time 1789781971.93 total 855.64 last 8.72) - open/close 16167 (time 1789781982.91 total 866.62 last 9.56) - open/close 16394 (time 1789781992.91 total 876.62 last 22.68) - open/close 16623 (time 1789782003.90 total 887.61 last 20.84) - open/close 16722 (time 1789782013.91 total 897.62 last 9.89) - open/close 16943 (time 1789782024.29 total 908.00 last 21.29) - open/close 17065 (time 1789782035.20 total 918.91 last 11.18) - open/close 17141 (time 1789782045.42 total 929.13 last 7.43) - open/close 17245 (time 1789782055.44 total 939.15 last 10.38) - open/close 17355 (time 1789782066.53 total 950.24 last 9.92) - open/close 17572 (time 1789782076.56 total 960.27 last 21.63) - open/close 17751 (time 1789782087.53 total 971.24 last 16.31) - open/close 17820 (time 1789782097.66 total 981.37 last 6.82) - open/close 17992 (time 1789782108.16 total 991.87 last 16.38) - open/close 18263 (time 1789782118.18 total 1001.89 last 27.05) - open/close 18780 (time 1789782128.18 total 1011.89 last 51.68) - open/close 19479 (time 1789782138.20 total 1021.91 last 69.77) - open/close 20000 (time 1789782146.36 total 1030.07 last 63.87) - open/close 20671 (time 1789782156.36 total 1040.07 last 67.05) - open/close 21236 (time 1789782166.37 total 1050.08 last 56.47) - open/close 21707 (time 1789782176.44 total 1060.15 last 46.75) - open/close 22309 (time 1789782186.45 total 1070.16 last 60.19) - open/close 22985 (time 1789782196.45 total 1080.16 last 67.57) - open/close 23665 (time 1789782206.47 total 1090.18 last 67.88) - open/close 24306 (time 1789782216.47 total 1100.18 last 64.07) - open/close 25107 (time 1789782226.48 total 1110.19 last 80.01) - open/close 25854 (time 1789782236.49 total 1120.20 last 74.62) - open/close 26549 (time 1789782246.50 total 1130.21 last 69.48) - open/close 27275 (time 1789782256.50 total 1140.21 last 72.58) - open/close 28106 (time 1789782266.51 total 1150.22 last 83.04) - open/close 28830 (time 1789782276.53 total 1160.24 last 72.21) - open/close 29545 (time 1789782286.54 total 1170.25 last 71.48) - open/close 30000 (time 1789782293.07 total 1176.78 last 69.61) - open/close 30658 (time 1789782303.08 total 1186.79 last 65.78) - open/close 31338 (time 1789782313.08 total 1196.79 last 67.98) - open/close 31960 (time 1789782323.08 total 1206.79 last 62.17) - open/close 32526 (time 1789782333.08 total 1216.79 last 56.60) - open/close 33088 (time 1789782343.09 total 1226.80 last 56.14) - open/close 33573 (time 1789782353.09 total 1236.81 last 48.49) - open/close 34222 (time 1789782363.10 total 1246.81 last 64.84) - open/close 35090 (time 1789782373.11 total 1256.82 last 86.74) - open/close 35837 (time 1789782383.11 total 1266.82 last 74.69) - open/close 36585 (time 1789782393.12 total 1276.83 last 74.74) - open/close 37276 (time 1789782403.13 total 1286.84 last 69.02) - open/close 38085 (time 1789782413.14 total 1296.85 last 80.80) - open/close 38943 (time 1789782423.14 total 1306.85 last 85.79) - open/close 39691 (time 1789782433.15 total 1316.86 last 74.73) - open/close 40000 (time 1789782437.50 total 1321.21 last 71.05) - open/close 40577 (time 1789782447.50 total 1331.21 last 57.69) - open/close 41343 (time 1789782457.50 total 1341.21 last 76.59) - open/close 42109 (time 1789782467.51 total 1351.22 last 76.53) - open/close 43005 (time 1789782477.52 total 1361.23 last 89.58) - open/close 43790 (time 1789782487.52 total 1371.23 last 78.47) - open/close 44569 (time 1789782497.52 total 1381.23 last 77.89) - open/close 45407 (time 1789782507.52 total 1391.23 last 83.79) - open/close 46193 (time 1789782517.53 total 1401.24 last 78.57) - open/close 46947 (time 1789782527.53 total 1411.24 last 75.37) - open/close 47755 (time 1789782537.53 total 1421.24 last 80.78) - open/close 48571 (time 1789782547.54 total 1431.25 last 81.52) - open/close 49113 (time 1789782557.56 total 1441.27 last 54.11) - open/close 49697 (time 1789782567.56 total 1451.27 last 58.37) total: 50000 open/close in 1455.93 seconds: 34.34 ops/second - unlinked 0 (time 1789782577 ; total 0 ; last 0) - unlinked 10000 (time 1789782709 ; total 132 ; last 132) - unlinked 20000 (time 1789782846 ; total 269 ; last 137) - unlinked 30000 (time 1789782995 ; total 418 ; last 149) - unlinked 40000 (time 1789783140 ; total 563 ; last 145) total: 50000 unlinks in 695 seconds: 71.942444 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg132-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server unloading modules via unload_modules_local on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing unload_modules_local oleg132-server: kmemleak disabled modules unloaded. pdsh@oleg132-client: oleg132-client: ssh exited with exit code 2 pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2 pdsh@oleg132-client: oleg132-client: ssh exited with exit code 2 pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2