== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:08:35 (1786950515) start mds service on oleg258-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg258-server' oleg258-server: oleg258-server.virtnet: executing load_modules_local oleg258-server: Loading modules from /home/green/git/lustre-release/lustre oleg258-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg258-server: detected 4 online CPUs by sysfs oleg258-server: MODOPTS_LIBCFS= oleg258-server: Force libcfs to create 2 CPU partitions oleg258-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg258-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg258-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg258-server: oleg258-server.virtnet: executing set_default_debug -1 all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg258-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg258-server: oleg258-server.virtnet: executing set_default_debug -1 all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-OST0000 oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 176 (time 1786950613.86 total 10.59 last 16.61) - open/close 461 (time 1786950623.90 total 20.64 last 28.38) - open/close 624 (time 1786950633.91 total 30.64 last 16.29) - open/close 761 (time 1786950644.09 total 40.82 last 13.46) - open/close 938 (time 1786950654.10 total 50.83 last 17.69) - open/close 1090 (time 1786950664.98 total 61.72 last 13.96) - open/close 1171 (time 1786950675.54 total 72.28 last 7.67) - open/close 1355 (time 1786950685.72 total 82.45 last 18.08) - open/close 1523 (time 1786950695.74 total 92.47 last 16.77) - open/close 1690 (time 1786950705.84 total 102.57 last 16.53) - open/close 1832 (time 1786950715.84 total 112.57 last 14.20) - open/close 1957 (time 1786950726.89 total 123.62 last 11.31) - open/close 2111 (time 1786950737.84 total 134.57 last 14.07) - open/close 2250 (time 1786950748.30 total 145.03 last 13.28) - open/close 2373 (time 1786950758.50 total 155.24 last 12.05) - open/close 2427 (time 1786950768.51 total 165.24 last 5.40) - open/close 2654 (time 1786950779.25 total 175.98 last 21.14) - open/close 2830 (time 1786950789.29 total 186.02 last 17.53) - open/close 2926 (time 1786950799.30 total 196.03 last 9.59) - open/close 3121 (time 1786950809.94 total 206.67 last 18.32) - open/close 3364 (time 1786950819.94 total 216.68 last 24.30) - open/close 3537 (time 1786950829.97 total 226.70 last 17.25) - open/close 3694 (time 1786950840.51 total 237.24 last 14.90) - open/close 3842 (time 1786950851.25 total 247.98 last 13.78) - open/close 4055 (time 1786950863.98 total 260.71 last 16.73) - open/close 4203 (time 1786950874.54 total 271.27 last 14.01) - open/close 4322 (time 1786950884.56 total 281.29 last 11.88) - open/close 4423 (time 1786950894.84 total 291.57 last 9.82) - open/close 4506 (time 1786950905.39 total 302.13 last 7.86) - open/close 4581 (time 1786950915.96 total 312.69 last 7.10) - open/close 4724 (time 1786950926.77 total 323.50 last 13.22) - open/close 4864 (time 1786950936.78 total 333.51 last 13.99) - open/close 5021 (time 1786950947.55 total 344.29 last 14.57) - open/close 5152 (time 1786950957.84 total 354.57 last 12.73) - open/close 5230 (time 1786950968.09 total 364.82 last 7.61) - open/close 5353 (time 1786950978.35 total 375.08 last 11.99) - open/close 5442 (time 1786950988.87 total 385.61 last 8.46) - open/close 5557 (time 1786950999.73 total 396.46 last 10.59) - open/close 5704 (time 1786951010.61 total 407.34 last 13.51) - open/close 5803 (time 1786951022.33 total 419.06 last 8.45) - open/close 5932 (time 1786951032.94 total 429.67 last 12.16) - open/close 6121 (time 1786951043.88 total 440.62 last 17.27) - open/close 6243 (time 1786951053.88 total 450.62 last 12.20) - open/close 6321 (time 1786951066.58 total 463.31 last 6.15) - open/close 6456 (time 1786951077.28 total 474.01 last 12.61) - open/close 6648 (time 1786951087.32 total 484.05 last 19.13) - open/close 6733 (time 1786951098.13 total 494.86 last 7.86) - open/close 6864 (time 1786951109.16 total 505.89 last 11.88) - open/close 6990 (time 1786951119.45 total 516.18 last 12.25) - open/close 7186 (time 1786951129.45 total 526.18 last 19.59) - open/close 7390 (time 1786951139.48 total 536.21 last 20.35) - open/close 7541 (time 1786951149.49 total 546.22 last 15.09) - open/close 7612 (time 1786951159.75 total 556.48 last 6.92) - open/close 7749 (time 1786951169.95 total 566.68 last 13.43) - open/close 7925 (time 1786951179.98 total 576.72 last 17.54) - open/close 8088 (time 1786951189.99 total 586.72 last 16.29) - open/close 8354 (time 1786951200.00 total 596.74 last 26.56) - open/close 8543 (time 1786951210.01 total 606.74 last 18.89) - open/close 8727 (time 1786951220.01 total 616.74 last 18.40) - open/close 9090 (time 1786951230.81 total 627.54 last 33.62) - open/close 9297 (time 1786951241.18 total 637.92 last 19.95) - open/close 9503 (time 1786951251.23 total 647.96 last 20.51) - open/close 9643 (time 1786951262.12 total 658.86 last 12.85) - open/close 9816 (time 1786951272.17 total 668.90 last 17.23) - open/close 9930 (time 1786951284.19 total 680.92 last 9.48) - open/close 10000 (time 1786951290.43 total 687.16 last 11.23) - open/close 10197 (time 1786951301.21 total 697.95 last 18.26) - open/close 10426 (time 1786951311.22 total 707.95 last 22.89) - open/close 10651 (time 1786951321.69 total 718.42 last 21.48) - open/close 10897 (time 1786951331.72 total 728.45 last 24.53) - open/close 11124 (time 1786951343.83 total 740.56 last 18.75) - open/close 11293 (time 1786951354.26 total 750.99 last 16.19) - open/close 11435 (time 1786951364.27 total 761.01 last 14.19) - open/close 11622 (time 1786951374.56 total 771.30 last 18.17) - open/close 11744 (time 1786951384.93 total 781.66 last 11.77) - open/close 11953 (time 1786951394.98 total 791.72 last 20.79) - open/close 12079 (time 1786951405.00 total 801.73 last 12.58) - open/close 12239 (time 1786951415.71 total 812.44 last 14.94) - open/close 12450 (time 1786951425.73 total 822.47 last 21.05) - open/close 12564 (time 1786951435.96 total 832.69 last 11.15) - open/close 12783 (time 1786951446.43 total 843.17 last 20.90) - open/close 12932 (time 1786951457.57 total 854.30 last 13.38) - open/close 13123 (time 1786951468.20 total 864.94 last 17.96) - open/close 13266 (time 1786951479.26 total 875.99 last 12.94) - open/close 13365 (time 1786951489.63 total 886.37 last 9.54) - open/close 13514 (time 1786951500.44 total 897.17 last 13.79) - open/close 13595 (time 1786951510.79 total 907.52 last 7.83) - open/close 13780 (time 1786951520.99 total 917.72 last 18.14) - open/close 13934 (time 1786951531.94 total 928.67 last 14.06) - open/close 14098 (time 1786951542.07 total 938.80 last 16.19) - open/close 14278 (time 1786951552.42 total 949.15 last 17.38) - open/close 14386 (time 1786951562.45 total 959.18 last 10.77) - open/close 14443 (time 1786951573.17 total 969.90 last 5.32) - open/close 14604 (time 1786951583.17 total 979.90 last 16.09) - open/close 14713 (time 1786951593.19 total 989.92 last 10.88) - open/close 14831 (time 1786951603.21 total 999.94 last 11.79) - open/close 14896 (time 1786951613.21 total 1009.95 last 6.49) - open/close 14970 (time 1786951623.53 total 1020.26 last 7.18) - open/close 15162 (time 1786951633.53 total 1030.26 last 19.19) - open/close 15334 (time 1786951644.36 total 1041.09 last 15.88) - open/close 15456 (time 1786951655.19 total 1051.92 last 11.27) - open/close 15564 (time 1786951665.59 total 1062.32 last 10.38) - open/close 15672 (time 1786951676.72 total 1073.45 last 9.71) - open/close 15778 (time 1786951687.71 total 1084.44 last 9.64) - open/close 15895 (time 1786951698.15 total 1094.88 last 11.21) - open/close 15999 (time 1786951708.60 total 1105.33 last 9.95) - open/close 16136 (time 1786951719.01 total 1115.74 last 13.16) - open/close 16509 (time 1786951729.02 total 1125.75 last 37.26) - open/close 17041 (time 1786951739.04 total 1135.77 last 53.13) - open/close 17615 (time 1786951749.04 total 1145.77 last 57.39) - open/close 18197 (time 1786951759.05 total 1155.78 last 58.14) - open/close 18811 (time 1786951769.05 total 1165.79 last 61.36) - open/close 19409 (time 1786951779.06 total 1175.79 last 59.76) - open/close 19850 (time 1786951789.06 total 1185.79 last 44.10) - open/close 20000 (time 1786951791.92 total 1188.65 last 52.54) - open/close 20564 (time 1786951801.93 total 1198.66 last 56.34) - open/close 21173 (time 1786951811.96 total 1208.69 last 60.69) - open/close 21603 (time 1786951821.98 total 1218.71 last 42.91) - open/close 22006 (time 1786951832.00 total 1228.73 last 40.21) - open/close 22497 (time 1786951842.01 total 1238.74 last 49.07) - open/close 22926 (time 1786951852.01 total 1248.74 last 42.89) - open/close 23498 (time 1786951862.04 total 1258.77 last 57.05) - open/close 24078 (time 1786951872.04 total 1268.77 last 57.99) - open/close 24775 (time 1786951882.04 total 1278.77 last 69.66) - open/close 25445 (time 1786951892.07 total 1288.80 last 66.80) - open/close 26043 (time 1786951902.08 total 1298.81 last 59.77) - open/close 26544 (time 1786951912.10 total 1308.83 last 50.00) - open/close 27173 (time 1786951922.11 total 1318.84 last 62.85) - open/close 27802 (time 1786951932.12 total 1328.85 last 62.80) - open/close 28368 (time 1786951942.12 total 1338.86 last 56.58) - open/close 28890 (time 1786951952.13 total 1348.86 last 52.19) - open/close 29302 (time 1786951962.15 total 1358.88 last 41.11) - open/close 29859 (time 1786951972.15 total 1368.88 last 55.68) - open/close 30000 (time 1786951974.44 total 1371.17 last 61.62) - open/close 30631 (time 1786951984.44 total 1381.17 last 63.08) - open/close 31199 (time 1786951994.46 total 1391.19 last 56.70) - open/close 31804 (time 1786952004.46 total 1401.19 last 60.49) - open/close 32353 (time 1786952014.46 total 1411.20 last 54.88) - open/close 32949 (time 1786952024.50 total 1421.23 last 59.37) - open/close 33586 (time 1786952034.53 total 1431.26 last 63.54) - open/close 34198 (time 1786952044.54 total 1441.27 last 61.12) - open/close 34831 (time 1786952054.55 total 1451.28 last 63.26) - open/close 35448 (time 1786952064.55 total 1461.28 last 61.67) - open/close 35983 (time 1786952074.57 total 1471.30 last 53.41) - open/close 36460 (time 1786952084.58 total 1481.31 last 47.64) - open/close 37095 (time 1786952094.58 total 1491.32 last 63.48) - open/close 37798 (time 1786952104.58 total 1501.32 last 70.30) - open/close 38387 (time 1786952114.59 total 1511.32 last 58.87) - open/close 39042 (time 1786952124.59 total 1521.32 last 65.48) - open/close 39709 (time 1786952134.61 total 1531.34 last 66.60) - open/close 40000 (time 1786952139.35 total 1536.08 last 61.32) - open/close 40583 (time 1786952149.36 total 1546.09 last 58.26) - open/close 41268 (time 1786952159.37 total 1556.10 last 68.44) - open/close 41847 (time 1786952169.37 total 1566.10 last 57.89) - open/close 42314 (time 1786952179.39 total 1576.12 last 46.61) - open/close 42899 (time 1786952189.40 total 1586.13 last 58.43) - open/close 43457 (time 1786952199.41 total 1596.14 last 55.78) - open/close 44198 (time 1786952209.41 total 1606.14 last 74.06) - open/close 44756 (time 1786952219.42 total 1616.15 last 55.75) - open/close 45391 (time 1786952229.43 total 1626.16 last 63.44) - open/close 46036 (time 1786952239.45 total 1636.18 last 64.38) - open/close 46698 (time 1786952249.47 total 1646.20 last 66.07) - open/close 47273 (time 1786952259.47 total 1656.20 last 57.48) - open/close 47917 (time 1786952269.49 total 1666.22 last 64.32) - open/close 48496 (time 1786952279.49 total 1676.22 last 57.89) - open/close 49060 (time 1786952289.50 total 1686.23 last 56.32) - open/close 49589 (time 1786952299.52 total 1696.25 last 52.78) total: 50000 open/close in 1702.29 seconds: 29.37 ops/second - unlinked 0 (time 1786952310 ; total 0 ; last 0) - unlinked 10000 (time 1786952492 ; total 182 ; last 182) - unlinked 20000 (time 1786952656 ; total 346 ; last 164) - unlinked 30000 (time 1786952817 ; total 507 ; last 161) - unlinked 40000 (time 1786952975 ; total 665 ; last 158) total: 50000 unlinks in 835 seconds: 59.880241 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg258-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg258-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg258-server stop mds service on oleg258-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg258-server unloading modules via unload_modules_local on: 'oleg258-server' oleg258-server: oleg258-server.virtnet: executing unload_modules_local oleg258-server: modules unloaded. pdsh@oleg258-client: oleg258-client: ssh exited with exit code 2 pdsh@oleg258-client: oleg258-server: ssh exited with exit code 2 pdsh@oleg258-client: oleg258-client: ssh exited with exit code 2 pdsh@oleg258-client: oleg258-server: ssh exited with exit code 2