== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:42:12 (1787571732) start mds service on oleg129-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg129-server' oleg129-server: oleg129-server.virtnet: executing load_modules_local oleg129-server: Loading modules from /home/green/git/lustre-release/lustre oleg129-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg129-server: detected 4 online CPUs by sysfs oleg129-server: MODOPTS_LIBCFS= oleg129-server: Force libcfs to create 2 CPU partitions oleg129-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg129-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg129-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg129-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 239 (time 1787571819.26 total 10.02 last 23.86) - open/close 463 (time 1787571829.27 total 20.02 last 22.38) - open/close 672 (time 1787571839.28 total 30.03 last 20.89) - open/close 880 (time 1787571850.25 total 41.00 last 18.95) - open/close 1029 (time 1787571860.30 total 51.05 last 14.83) - open/close 1154 (time 1787571870.30 total 61.05 last 12.50) - open/close 1299 (time 1787571880.31 total 71.06 last 14.49) - open/close 1438 (time 1787571890.90 total 81.65 last 13.13) - open/close 1493 (time 1787571901.77 total 92.52 last 5.06) - open/close 1664 (time 1787571911.78 total 102.53 last 17.09) - open/close 1787 (time 1787571921.95 total 112.70 last 12.09) - open/close 1988 (time 1787571931.97 total 122.72 last 20.06) - open/close 2118 (time 1787571941.97 total 132.72 last 13.00) - open/close 2292 (time 1787571952.22 total 142.97 last 16.97) - open/close 2385 (time 1787571962.24 total 152.99 last 9.29) - open/close 2475 (time 1787571973.25 total 164.01 last 8.17) - open/close 2594 (time 1787571983.49 total 174.24 last 11.63) - open/close 2694 (time 1787571994.26 total 185.01 last 9.29) - open/close 2759 (time 1787572004.30 total 195.05 last 6.47) - open/close 2940 (time 1787572014.67 total 205.42 last 17.45) - open/close 3054 (time 1787572024.86 total 215.61 last 11.19) - open/close 3169 (time 1787572035.22 total 225.97 last 11.09) - open/close 3316 (time 1787572046.15 total 236.91 last 13.45) - open/close 3509 (time 1787572056.88 total 247.63 last 18.00) - open/close 3722 (time 1787572067.64 total 258.39 last 19.78) - open/close 3798 (time 1787572077.91 total 268.66 last 7.40) - open/close 3959 (time 1787572087.92 total 278.67 last 16.08) - open/close 4117 (time 1787572097.94 total 288.69 last 15.78) - open/close 4285 (time 1787572108.87 total 299.62 last 15.37) - open/close 4444 (time 1787572120.02 total 310.77 last 14.26) - open/close 4634 (time 1787572130.26 total 321.01 last 18.54) - open/close 4873 (time 1787572140.27 total 331.02 last 23.89) - open/close 5118 (time 1787572151.70 total 342.45 last 21.43) - open/close 5218 (time 1787572161.79 total 352.54 last 9.91) - open/close 5307 (time 1787572172.41 total 363.16 last 8.38) - open/close 5376 (time 1787572182.79 total 373.54 last 6.65) - open/close 5579 (time 1787572193.79 total 384.54 last 18.45) - open/close 5726 (time 1787572206.39 total 397.14 last 11.66) - open/close 5851 (time 1787572216.41 total 407.16 last 12.48) - open/close 6042 (time 1787572226.87 total 417.62 last 18.26) - open/close 6175 (time 1787572237.56 total 428.31 last 12.44) - open/close 6281 (time 1787572247.58 total 438.33 last 10.58) - open/close 6409 (time 1787572258.57 total 449.32 last 11.64) - open/close 6518 (time 1787572268.80 total 459.55 last 10.66) - open/close 6580 (time 1787572279.55 total 470.31 last 5.76) - open/close 6805 (time 1787572289.58 total 480.33 last 22.44) - open/close 7037 (time 1787572299.59 total 490.34 last 23.17) - open/close 7222 (time 1787572311.81 total 502.56 last 15.15) - open/close 7449 (time 1787572321.83 total 512.58 last 22.65) - open/close 7611 (time 1787572332.02 total 522.77 last 15.89) - open/close 7738 (time 1787572343.12 total 533.87 last 11.44) - open/close 7840 (time 1787572354.11 total 544.86 last 9.28) - open/close 7942 (time 1787572364.68 total 555.43 last 9.65) - open/close 7981 (time 1787572375.17 total 565.92 last 3.72) - open/close 8228 (time 1787572387.33 total 578.08 last 20.31) - open/close 8401 (time 1787572397.38 total 588.14 last 17.21) - open/close 8630 (time 1787572408.07 total 598.82 last 21.43) - open/close 8856 (time 1787572418.64 total 609.39 last 21.39) - open/close 8992 (time 1787572428.99 total 619.74 last 13.13) - open/close 9148 (time 1787572439.86 total 630.62 last 14.35) - open/close 9302 (time 1787572449.88 total 640.63 last 15.37) - open/close 9422 (time 1787572460.79 total 651.54 last 11.00) - open/close 9665 (time 1787572470.86 total 661.61 last 24.13) - open/close 9854 (time 1787572481.99 total 672.74 last 16.99) - open/close 9940 (time 1787572491.99 total 682.74 last 8.59) - open/close 10000 (time 1787572494.20 total 684.95 last 27.26) - open/close 10034 (time 1787572505.35 total 696.10 last 3.05) - open/close 10150 (time 1787572515.35 total 706.11 last 11.59) - open/close 10315 (time 1787572525.36 total 716.11 last 16.49) - open/close 10568 (time 1787572536.37 total 727.12 last 22.99) - open/close 10664 (time 1787572546.41 total 737.16 last 9.56) - open/close 10833 (time 1787572556.83 total 747.58 last 16.21) - open/close 11040 (time 1787572566.84 total 757.59 last 20.69) - open/close 11186 (time 1787572577.86 total 768.62 last 13.24) - open/close 11419 (time 1787572588.34 total 779.09 last 22.23) - open/close 11635 (time 1787572598.36 total 789.11 last 21.56) - open/close 11812 (time 1787572609.16 total 799.91 last 16.39) - open/close 12014 (time 1787572619.17 total 809.92 last 20.18) - open/close 12152 (time 1787572629.26 total 820.01 last 13.68) - open/close 12270 (time 1787572639.51 total 830.26 last 11.50) - open/close 12350 (time 1787572649.97 total 840.72 last 7.65) - open/close 12430 (time 1787572660.33 total 851.09 last 7.72) - open/close 12599 (time 1787572670.41 total 861.16 last 16.78) - open/close 12781 (time 1787572681.22 total 871.97 last 16.83) - open/close 13004 (time 1787572691.23 total 881.98 last 22.29) - open/close 13204 (time 1787572701.71 total 892.46 last 19.08) - open/close 13448 (time 1787572711.72 total 902.47 last 24.37) - open/close 13635 (time 1787572721.73 total 912.48 last 18.69) - open/close 13834 (time 1787572732.05 total 922.80 last 19.28) - open/close 14021 (time 1787572742.17 total 932.92 last 18.49) - open/close 14280 (time 1787572753.21 total 943.96 last 23.46) - open/close 14483 (time 1787572763.22 total 953.97 last 20.29) - open/close 14721 (time 1787572773.90 total 964.65 last 22.28) - open/close 14918 (time 1787572785.37 total 976.12 last 17.18) - open/close 15183 (time 1787572795.37 total 986.12 last 26.48) - open/close 15325 (time 1787572808.01 total 998.76 last 11.24) - open/close 15567 (time 1787572818.94 total 1009.69 last 22.14) - open/close 15766 (time 1787572828.96 total 1019.71 last 19.86) - open/close 16037 (time 1787572838.99 total 1029.74 last 27.03) - open/close 16272 (time 1787572849.00 total 1039.76 last 23.45) - open/close 16497 (time 1787572859.03 total 1049.78 last 22.45) - open/close 16661 (time 1787572869.18 total 1059.93 last 16.15) - open/close 16938 (time 1787572879.18 total 1069.93 last 27.69) - open/close 17059 (time 1787572889.28 total 1080.03 last 11.98) - open/close 17739 (time 1787572899.28 total 1090.03 last 67.99) - open/close 18593 (time 1787572909.29 total 1100.04 last 85.34) - open/close 19390 (time 1787572919.30 total 1110.05 last 79.62) - open/close 20000 (time 1787572925.57 total 1116.32 last 97.29) - open/close 20711 (time 1787572935.57 total 1126.32 last 71.10) - open/close 21420 (time 1787572945.57 total 1136.32 last 70.88) - open/close 22187 (time 1787572955.58 total 1146.33 last 76.66) - open/close 22999 (time 1787572965.59 total 1156.34 last 81.13) - open/close 23706 (time 1787572975.60 total 1166.35 last 70.63) - open/close 24408 (time 1787572985.61 total 1176.36 last 70.09) - open/close 25055 (time 1787572995.61 total 1186.36 last 64.69) - open/close 25850 (time 1787573005.62 total 1196.37 last 79.44) - open/close 26588 (time 1787573015.68 total 1206.43 last 73.39) - open/close 27284 (time 1787573025.68 total 1216.43 last 69.57) - open/close 28053 (time 1787573035.69 total 1226.44 last 76.81) - open/close 28979 (time 1787573045.70 total 1236.45 last 92.53) - open/close 29754 (time 1787573055.71 total 1246.46 last 77.42) - open/close 30000 (time 1787573058.88 total 1249.63 last 77.74) - open/close 30746 (time 1787573068.88 total 1259.63 last 74.57) - open/close 31527 (time 1787573078.89 total 1269.64 last 78.07) - open/close 32277 (time 1787573088.89 total 1279.64 last 74.97) - open/close 32967 (time 1787573098.94 total 1289.69 last 68.67) - open/close 33724 (time 1787573108.94 total 1299.69 last 75.66) - open/close 34461 (time 1787573118.95 total 1309.70 last 73.66) - open/close 35168 (time 1787573128.96 total 1319.71 last 70.62) - open/close 35851 (time 1787573138.96 total 1329.71 last 68.29) - open/close 36567 (time 1787573148.97 total 1339.72 last 71.58) - open/close 37440 (time 1787573159.02 total 1349.78 last 86.79) - open/close 38288 (time 1787573169.03 total 1359.78 last 84.74) - open/close 39033 (time 1787573179.04 total 1369.79 last 74.48) - open/close 39851 (time 1787573189.04 total 1379.79 last 81.79) - open/close 40663 (time 1787573199.04 total 1389.79 last 81.14) - open/close 41513 (time 1787573209.07 total 1399.82 last 84.82) - open/close 42255 (time 1787573219.07 total 1409.82 last 74.18) - open/close 42980 (time 1787573229.07 total 1419.82 last 72.46) - open/close 43773 (time 1787573239.08 total 1429.83 last 79.24) - open/close 44557 (time 1787573249.09 total 1439.84 last 78.34) - open/close 45279 (time 1787573259.09 total 1449.84 last 72.17) - open/close 46056 (time 1787573269.11 total 1459.86 last 77.57) - open/close 46827 (time 1787573279.13 total 1469.88 last 76.96) - open/close 47479 (time 1787573289.13 total 1479.88 last 65.20) - open/close 48187 (time 1787573299.14 total 1489.89 last 70.75) - open/close 48933 (time 1787573309.14 total 1499.89 last 74.55) - open/close 49615 (time 1787573319.16 total 1509.91 last 68.09) total: 50000 open/close in 1515.02 seconds: 33.00 ops/second - unlinked 0 (time 1787573329 ; total 0 ; last 0) - unlinked 10000 (time 1787573485 ; total 156 ; last 156) - unlinked 20000 (time 1787573632 ; total 303 ; last 147) - unlinked 30000 (time 1787573764 ; total 435 ; last 132) - unlinked 40000 (time 1787573915 ; total 586 ; last 151) total: 50000 unlinks in 713 seconds: 70.126228 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg129-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg129-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg129-server stop mds service on oleg129-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg129-server unloading modules via unload_modules_local on: 'oleg129-server' oleg129-server: oleg129-server.virtnet: executing unload_modules_local oleg129-server: modules unloaded. pdsh@oleg129-client: oleg129-client: ssh exited with exit code 2 pdsh@oleg129-client: oleg129-server: ssh exited with exit code 2 pdsh@oleg129-client: oleg129-client: ssh exited with exit code 2 pdsh@oleg129-client: oleg129-server: ssh exited with exit code 2