== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:10:13 (1786691413) start mds service on oleg207-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing load_modules_local oleg207-server: Loading modules from /home/green/git/lustre-release/lustre oleg207-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg207-server: detected 4 online CPUs by sysfs oleg207-server: MODOPTS_LIBCFS= oleg207-server: Force libcfs to create 2 CPU partitions oleg207-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg207-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg207-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg207-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 123 (time 1786691516.00 total 10.88 last 11.31) - open/close 224 (time 1786691526.01 total 20.88 last 10.10) - open/close 361 (time 1786691536.54 total 31.42 last 13.00) - open/close 518 (time 1786691546.94 total 41.81 last 15.10) - open/close 598 (time 1786691557.38 total 52.25 last 7.66) - open/close 682 (time 1786691567.39 total 62.26 last 8.39) - open/close 748 (time 1786691577.41 total 72.29 last 6.58) - open/close 848 (time 1786691587.43 total 82.30 last 9.99) - open/close 1003 (time 1786691597.96 total 92.84 last 14.71) - open/close 1168 (time 1786691607.98 total 102.85 last 16.47) - open/close 1310 (time 1786691617.98 total 112.86 last 14.20) - open/close 1436 (time 1786691628.87 total 123.75 last 11.57) - open/close 1571 (time 1786691639.25 total 134.12 last 13.01) - open/close 1675 (time 1786691650.25 total 145.13 last 9.45) - open/close 1826 (time 1786691660.63 total 155.51 last 14.55) - open/close 1944 (time 1786691671.38 total 166.25 last 10.98) - open/close 2071 (time 1786691681.41 total 176.28 last 12.66) - open/close 2239 (time 1786691692.31 total 187.18 last 15.42) - open/close 2300 (time 1786691702.46 total 197.34 last 6.01) - open/close 2416 (time 1786691712.49 total 207.37 last 11.57) - open/close 2480 (time 1786691723.04 total 217.92 last 6.07) - open/close 2555 (time 1786691736.01 total 230.88 last 5.79) - open/close 2680 (time 1786691746.69 total 241.57 last 11.70) - open/close 2731 (time 1786691756.76 total 251.63 last 5.07) - open/close 2820 (time 1786691766.79 total 261.66 last 8.87) - open/close 2913 (time 1786691776.79 total 271.67 last 9.30) - open/close 2993 (time 1786691787.55 total 282.42 last 7.44) - open/close 3085 (time 1786691798.46 total 293.34 last 8.43) - open/close 3195 (time 1786691809.35 total 304.22 last 10.11) - open/close 3319 (time 1786691819.35 total 314.23 last 12.40) - open/close 3426 (time 1786691829.72 total 324.59 last 10.32) - open/close 3543 (time 1786691839.75 total 334.62 last 11.66) - open/close 3618 (time 1786691850.75 total 345.63 last 6.81) - open/close 3776 (time 1786691861.30 total 356.18 last 14.98) - open/close 3903 (time 1786691871.32 total 366.19 last 12.68) - open/close 4009 (time 1786691881.83 total 376.70 last 10.08) - open/close 4099 (time 1786691891.92 total 386.79 last 8.92) - open/close 4209 (time 1786691902.32 total 397.20 last 10.57) - open/close 4323 (time 1786691912.33 total 407.20 last 11.40) - open/close 4404 (time 1786691922.33 total 417.21 last 8.10) - open/close 4496 (time 1786691932.62 total 427.49 last 8.94) - open/close 4579 (time 1786691942.81 total 437.68 last 8.15) - open/close 4675 (time 1786691952.89 total 447.77 last 9.52) - open/close 4813 (time 1786691963.01 total 457.88 last 13.65) - open/close 4927 (time 1786691973.02 total 467.89 last 11.39) - open/close 5037 (time 1786691983.34 total 478.22 last 10.65) - open/close 5177 (time 1786691993.99 total 488.87 last 13.15) - open/close 5283 (time 1786692004.22 total 499.09 last 10.37) - open/close 5400 (time 1786692014.67 total 509.54 last 11.20) - open/close 5470 (time 1786692024.90 total 519.78 last 6.84) - open/close 5578 (time 1786692035.94 total 530.81 last 9.79) - open/close 5672 (time 1786692046.97 total 541.85 last 8.52) - open/close 5746 (time 1786692057.65 total 552.53 last 6.93) - open/close 5836 (time 1786692070.80 total 565.68 last 6.84) - open/close 5958 (time 1786692080.83 total 575.71 last 12.16) - open/close 6034 (time 1786692091.23 total 586.10 last 7.31) - open/close 6138 (time 1786692101.27 total 596.14 last 10.36) - open/close 6316 (time 1786692112.31 total 607.18 last 16.12) - open/close 6416 (time 1786692123.21 total 618.08 last 9.17) - open/close 6556 (time 1786692133.58 total 628.46 last 13.49) - open/close 6600 (time 1786692144.29 total 639.16 last 4.11) - open/close 6643 (time 1786692154.54 total 649.42 last 4.19) - open/close 6797 (time 1786692165.50 total 660.37 last 14.05) - open/close 6912 (time 1786692175.68 total 670.56 last 11.29) - open/close 6955 (time 1786692186.43 total 681.31 last 4.00) - open/close 7051 (time 1786692196.55 total 691.42 last 9.49) - open/close 7226 (time 1786692207.16 total 702.04 last 16.48) - open/close 7350 (time 1786692218.76 total 713.63 last 10.69) - open/close 7441 (time 1786692230.70 total 725.57 last 7.62) - open/close 7611 (time 1786692241.62 total 736.49 last 15.57) - open/close 7727 (time 1786692252.62 total 747.49 last 10.54) - open/close 7848 (time 1786692263.56 total 758.44 last 11.06) - open/close 7910 (time 1786692273.81 total 768.68 last 6.05) - open/close 8018 (time 1786692284.35 total 779.22 last 10.25) - open/close 8099 (time 1786692295.11 total 789.98 last 7.53) - open/close 8203 (time 1786692305.12 total 800.00 last 10.38) - open/close 8309 (time 1786692315.22 total 810.09 last 10.50) - open/close 8428 (time 1786692325.91 total 820.78 last 11.14) - open/close 8616 (time 1786692336.11 total 830.99 last 18.42) - open/close 8743 (time 1786692346.25 total 841.12 last 12.53) - open/close 8815 (time 1786692357.39 total 852.27 last 6.46) - open/close 8998 (time 1786692367.40 total 862.28 last 18.28) - open/close 9091 (time 1786692377.41 total 872.28 last 9.30) - open/close 9264 (time 1786692387.66 total 882.53 last 16.87) - open/close 9367 (time 1786692398.72 total 893.59 last 9.32) - open/close 9479 (time 1786692408.72 total 903.60 last 11.19) - open/close 9562 (time 1786692419.29 total 914.16 last 7.85) - open/close 9683 (time 1786692429.77 total 924.64 last 11.55) - open/close 9839 (time 1786692439.85 total 934.73 last 15.46) - open/close 10000 (time 1786692449.74 total 944.62 last 16.28) - open/close 10173 (time 1786692460.82 total 955.69 last 15.62) - open/close 10328 (time 1786692471.11 total 965.98 last 15.06) - open/close 10483 (time 1786692481.95 total 976.82 last 14.30) - open/close 10716 (time 1786692492.31 total 987.18 last 22.49) - open/close 10952 (time 1786692503.06 total 997.93 last 21.96) - open/close 11187 (time 1786692513.72 total 1008.60 last 22.03) - open/close 11409 (time 1786692524.67 total 1019.54 last 20.29) - open/close 11568 (time 1786692535.57 total 1030.44 last 14.58) - open/close 11834 (time 1786692546.24 total 1041.12 last 24.93) - open/close 12041 (time 1786692556.26 total 1051.13 last 20.67) - open/close 12254 (time 1786692566.27 total 1061.15 last 21.26) - open/close 12485 (time 1786692576.28 total 1071.15 last 23.09) - open/close 12636 (time 1786692586.29 total 1081.16 last 15.08) - open/close 12782 (time 1786692597.15 total 1092.03 last 13.44) - open/close 12979 (time 1786692607.72 total 1102.59 last 18.64) - open/close 13247 (time 1786692617.72 total 1112.59 last 26.80) - open/close 13417 (time 1786692627.75 total 1122.62 last 16.95) - open/close 13657 (time 1786692638.59 total 1133.46 last 22.14) - open/close 13811 (time 1786692648.59 total 1143.46 last 15.40) - open/close 13989 (time 1786692658.62 total 1153.50 last 17.74) - open/close 14202 (time 1786692668.63 total 1163.50 last 21.30) - open/close 14322 (time 1786692678.73 total 1173.60 last 11.88) - open/close 14508 (time 1786692689.22 total 1184.10 last 17.72) - open/close 14715 (time 1786692701.37 total 1196.24 last 17.04) - open/close 15133 (time 1786692711.37 total 1206.25 last 41.80) - open/close 15731 (time 1786692721.38 total 1216.26 last 59.74) - open/close 16462 (time 1786692731.39 total 1226.26 last 73.03) - open/close 17149 (time 1786692741.40 total 1236.28 last 68.62) - open/close 17627 (time 1786692751.49 total 1246.37 last 47.37) - open/close 18113 (time 1786692761.50 total 1256.37 last 48.57) - open/close 18681 (time 1786692771.50 total 1266.38 last 56.78) - open/close 19234 (time 1786692781.51 total 1276.38 last 55.26) - open/close 19870 (time 1786692791.52 total 1286.39 last 63.54) - open/close 20000 (time 1786692793.62 total 1288.50 last 61.73) - open/close 20566 (time 1786692803.62 total 1298.50 last 56.59) - open/close 21276 (time 1786692813.63 total 1308.51 last 70.96) - open/close 22017 (time 1786692823.64 total 1318.52 last 74.01) - open/close 22692 (time 1786692833.66 total 1328.53 last 67.39) - open/close 23323 (time 1786692843.68 total 1338.55 last 62.98) - open/close 23896 (time 1786692853.68 total 1348.56 last 57.27) - open/close 24530 (time 1786692863.70 total 1358.58 last 63.28) - open/close 25023 (time 1786692873.71 total 1368.59 last 49.25) - open/close 25735 (time 1786692883.73 total 1378.60 last 71.07) - open/close 26274 (time 1786692893.74 total 1388.61 last 53.85) - open/close 26932 (time 1786692903.75 total 1398.63 last 65.71) - open/close 27587 (time 1786692913.76 total 1408.63 last 65.46) - open/close 28256 (time 1786692923.76 total 1418.63 last 66.90) - open/close 28958 (time 1786692933.77 total 1428.64 last 70.14) - open/close 29467 (time 1786692943.78 total 1438.65 last 50.86) - open/close 30000 (time 1786692953.27 total 1448.14 last 56.15) - open/close 30473 (time 1786692963.28 total 1458.15 last 47.27) - open/close 31142 (time 1786692973.28 total 1468.16 last 66.84) - open/close 31753 (time 1786692983.30 total 1478.18 last 61.00) - open/close 32330 (time 1786692993.30 total 1488.18 last 57.68) - open/close 32887 (time 1786693003.32 total 1498.19 last 55.63) - open/close 33427 (time 1786693013.32 total 1508.20 last 53.97) - open/close 34024 (time 1786693023.33 total 1518.21 last 59.66) - open/close 34565 (time 1786693033.34 total 1528.21 last 54.05) - open/close 35188 (time 1786693043.36 total 1538.23 last 62.18) - open/close 35856 (time 1786693053.37 total 1548.25 last 66.71) - open/close 36471 (time 1786693063.39 total 1558.26 last 61.41) - open/close 37116 (time 1786693073.39 total 1568.27 last 64.45) - open/close 37755 (time 1786693083.40 total 1578.28 last 63.86) - open/close 38321 (time 1786693093.40 total 1588.28 last 56.60) - open/close 38918 (time 1786693103.41 total 1598.29 last 59.64) - open/close 39513 (time 1786693113.41 total 1608.29 last 59.49) - open/close 40000 (time 1786693119.91 total 1614.79 last 74.93) - open/close 40712 (time 1786693129.91 total 1624.79 last 71.20) - open/close 41452 (time 1786693139.92 total 1634.80 last 73.92) - open/close 42121 (time 1786693149.94 total 1644.81 last 66.83) - open/close 42701 (time 1786693159.95 total 1654.83 last 57.91) - open/close 43250 (time 1786693169.96 total 1664.84 last 54.85) - open/close 43655 (time 1786693179.99 total 1674.87 last 40.37) - open/close 44140 (time 1786693190.00 total 1684.88 last 48.46) - open/close 44751 (time 1786693200.01 total 1694.89 last 61.04) - open/close 45367 (time 1786693210.01 total 1704.89 last 61.60) - open/close 46059 (time 1786693220.02 total 1714.89 last 69.18) - open/close 46659 (time 1786693230.02 total 1724.90 last 59.95) - open/close 47342 (time 1786693240.04 total 1734.92 last 68.18) - open/close 47956 (time 1786693250.05 total 1744.92 last 61.36) - open/close 48630 (time 1786693260.06 total 1754.93 last 67.33) - open/close 49312 (time 1786693270.07 total 1764.94 last 68.13) - open/close 49970 (time 1786693280.08 total 1774.96 last 65.71) total: 50000 open/close in 1775.41 seconds: 28.16 ops/second - unlinked 0 (time 1786693286 ; total 0 ; last 0) - unlinked 10000 (time 1786693461 ; total 175 ; last 175) - unlinked 20000 (time 1786693630 ; total 344 ; last 169) - unlinked 30000 (time 1786693796 ; total 510 ; last 166) - unlinked 40000 (time 1786693964 ; total 678 ; last 168) total: 50000 unlinks in 845 seconds: 59.171597 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server unloading modules via unload_modules_local on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing unload_modules_local oleg207-server: modules unloaded. pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2