-----============= acceptance-small: conf-sanity ============----- Thu May 1 20:35:52 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg229-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg229-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg229-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg229-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg229-server oleg229-server: oleg229-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg229-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg229-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg229-server: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg229-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 20:37:30 (1746146250) Stopping clients: oleg229-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg229-client.virtnet /mnt/lustre2 (opts:-f) oleg229-server: oleg229-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg229-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg229-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1282 (time 1746146302.96 total 10.01 last 128.11) - open/close 2575 (time 1746146312.96 total 20.01 last 129.28) - open/close 4045 (time 1746146322.96 total 30.01 last 146.98) - open/close 5312 (time 1746146332.96 total 40.01 last 126.69) - open/close 6649 (time 1746146342.97 total 50.02 last 133.62) - open/close 7968 (time 1746146352.97 total 60.02 last 131.84) - open/close 9241 (time 1746146362.97 total 70.02 last 127.26) - open/close 10000 (time 1746146369.06 total 76.11 last 124.78) - open/close 11212 (time 1746146379.06 total 86.11 last 121.13) - open/close 12417 (time 1746146389.08 total 96.13 last 120.36) - open/close 13488 (time 1746146399.09 total 106.14 last 106.99) - open/close 14721 (time 1746146409.09 total 116.14 last 123.23) - open/close 15805 (time 1746146419.10 total 126.15 last 108.32) - open/close 16837 (time 1746146429.10 total 136.15 last 103.18) - open/close 18043 (time 1746146439.11 total 146.16 last 120.53) - open/close 19262 (time 1746146449.11 total 156.16 last 121.84) - open/close 20000 (time 1746146454.79 total 161.84 last 129.91) - open/close 21307 (time 1746146464.79 total 171.84 last 130.69) - open/close 22568 (time 1746146474.80 total 181.85 last 126.00) - open/close 23711 (time 1746146484.80 total 191.85 last 114.30) - open/close 24914 (time 1746146494.80 total 201.85 last 120.27) - open/close 26061 (time 1746146504.81 total 211.86 last 114.66) - open/close 27058 (time 1746146514.81 total 221.86 last 99.66) - open/close 28200 (time 1746146524.81 total 231.86 last 114.20) - open/close 29294 (time 1746146534.82 total 241.87 last 109.36) - open/close 30000 (time 1746146542.23 total 249.28 last 95.22) - open/close 31010 (time 1746146552.23 total 259.28 last 100.99) - open/close 32078 (time 1746146562.23 total 269.28 last 106.78) - open/close 33058 (time 1746146572.23 total 279.28 last 98.00) - open/close 33994 (time 1746146582.24 total 289.29 last 93.50) - open/close 34955 (time 1746146592.25 total 299.30 last 96.06) - open/close 35970 (time 1746146602.25 total 309.30 last 101.46) - open/close 36987 (time 1746146612.27 total 319.32 last 101.56) - open/close 37888 (time 1746146622.27 total 329.32 last 90.02) - open/close 38809 (time 1746146632.28 total 339.33 last 92.07) - open/close 39750 (time 1746146642.29 total 349.34 last 94.03) - open/close 40000 (time 1746146645.04 total 352.09 last 90.87) - open/close 41003 (time 1746146655.05 total 362.10 last 100.22) - open/close 42026 (time 1746146665.05 total 372.10 last 102.22) - open/close 43107 (time 1746146675.06 total 382.11 last 108.07) - open/close 44127 (time 1746146685.06 total 392.11 last 101.95) - open/close 45087 (time 1746146695.07 total 402.12 last 95.93) - open/close 46123 (time 1746146705.07 total 412.12 last 103.55) - open/close 47147 (time 1746146715.08 total 422.13 last 102.38) - open/close 48191 (time 1746146725.08 total 432.13 last 104.39) - open/close 49345 (time 1746146735.08 total 442.13 last 115.36) - open/close 50000 (time 1746146739.64 total 446.69 last 143.59) - open/close 51386 (time 1746146749.65 total 456.70 last 138.50) - open/close 52627 (time 1746146759.65 total 466.70 last 124.03) - open/close 53878 (time 1746146769.66 total 476.71 last 124.99) - open/close 54978 (time 1746146779.67 total 486.72 last 109.90) - open/close 56001 (time 1746146789.69 total 496.74 last 102.15) - open/close 57007 (time 1746146799.70 total 506.75 last 100.51) - open/close 57967 (time 1746146809.70 total 516.75 last 95.92) - open/close 58931 (time 1746146819.71 total 526.76 last 96.34) - open/close 60000 (time 1746146829.27 total 536.32 last 111.77) - open/close 61123 (time 1746146839.29 total 546.34 last 112.17) - open/close 62477 (time 1746146849.29 total 556.34 last 135.37) - open/close 63557 (time 1746146859.29 total 566.34 last 107.95) total: 64768 open/close in 576.00 seconds: 112.45 ops/second - /unlink 2113 (time 1746146880.33 total 10.00 last 211.27) - /unlink 4532 (time 1746146890.33 total 20.00 last 241.89) - /unlink 7196 (time 1746146900.33 total 30.00 last 266.38) - /unlink 9351 (time 1746146910.33 total 40.00 last 215.46) - /unlink 10000 (time 1746146913.53 total 43.20 last 202.92) - /unlink 12098 (time 1746146923.54 total 53.21 last 209.65) - /unlink 13889 (time 1746146933.54 total 63.21 last 179.07) - /unlink 16080 (time 1746146943.54 total 73.21 last 219.05) - /unlink 17830 (time 1746146953.54 total 83.21 last 174.99) - /unlink 19535 (time 1746146963.54 total 93.22 last 170.44) - /unlink 20000 (time 1746146966.35 total 96.03 last 165.36) - /unlink 21818 (time 1746146976.35 total 106.03 last 181.80) - /unlink 23930 (time 1746146986.36 total 116.03 last 211.17) - /unlink 25740 (time 1746146996.36 total 126.03 last 180.98) - /unlink 27624 (time 1746147006.37 total 136.04 last 188.23) - /unlink 29737 (time 1746147016.37 total 146.05 last 211.21) - /unlink 31829 (time 1746147026.38 total 156.05 last 209.09) - /unlink 33586 (time 1746147036.38 total 166.06 last 175.62) - /unlink 35359 (time 1746147046.38 total 176.06 last 177.28) - /unlink 37120 (time 1746147056.39 total 186.07 last 175.92) - /unlink 39154 (time 1746147066.39 total 196.07 last 203.34) - /unlink 40000 (time 1746147070.63 total 200.30 last 199.86) - /unlink 41874 (time 1746147080.63 total 210.30 last 187.37) - /unlink 43736 (time 1746147090.63 total 220.30 last 186.19) - /unlink 45679 (time 1746147100.64 total 230.31 last 194.15) - /unlink 47599 (time 1746147110.64 total 240.32 last 191.94) - /unlink 49456 (time 1746147120.64 total 250.32 last 185.64) - /unlink 50000 (time 1746147123.81 total 253.48 last 171.82) - /unlink 51719 (time 1746147133.81 total 263.49 last 171.82) - /unlink 53623 (time 1746147143.82 total 273.49 last 190.30) - /unlink 55236 (time 1746147153.82 total 283.50 last 161.28) - /unlink 57197 (time 1746147163.82 total 293.50 last 196.07) - /unlink 58954 (time 1746147173.82 total 303.50 last 175.66) - /unlink 60000 (time 1746147179.27 total 308.95 last 192.01) - /unlink 61806 (time 1746147189.28 total 318.95 last 180.53) - /unlink 63831 (time 1746147199.28 total 328.95 last 202.44) total: 64768 /unlink in 332.69 seconds: 194.68 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1094 (time 1746147230.29 total 10.00 last 109.36) - open/close 2163 (time 1746147240.30 total 20.01 last 106.85) - open/close 3206 (time 1746147250.30 total 30.01 last 104.28) - open/close 4240 (time 1746147260.32 total 40.03 last 103.20) - open/close 5349 (time 1746147270.33 total 50.03 last 110.84) - open/close 6442 (time 1746147280.33 total 60.04 last 109.26) - open/close 7550 (time 1746147290.33 total 70.04 last 110.79) - open/close 8546 (time 1746147300.34 total 80.05 last 99.54) - open/close 9665 (time 1746147310.34 total 90.05 last 111.84) - open/close 10000 (time 1746147313.23 total 92.94 last 116.04) - open/close 11271 (time 1746147323.24 total 102.95 last 126.99) - open/close 12412 (time 1746147333.24 total 112.95 last 114.08) - open/close 13550 (time 1746147343.25 total 122.96 last 113.67) - open/close 14647 (time 1746147353.25 total 132.96 last 109.69) - open/close 15721 (time 1746147363.26 total 142.97 last 107.35) - open/close 16720 (time 1746147373.26 total 152.97 last 99.85) - open/close 17739 (time 1746147383.26 total 162.97 last 101.90) - open/close 18885 (time 1746147393.27 total 172.98 last 114.56) - open/close 19948 (time 1746147403.27 total 182.98 last 106.27) - open/close 20989 (time 1746147413.27 total 192.98 last 104.09) - open/close 22108 (time 1746147423.27 total 202.98 last 111.87) - open/close 23292 (time 1746147433.27 total 212.98 last 118.38) - open/close 24414 (time 1746147443.28 total 222.99 last 112.18) - open/close 25565 (time 1746147453.29 total 233.00 last 114.95) - open/close 26686 (time 1746147463.30 total 243.01 last 111.99) - open/close 27825 (time 1746147473.30 total 253.01 last 113.85) - open/close 28852 (time 1746147483.31 total 263.02 last 102.64) - open/close 29906 (time 1746147493.32 total 273.03 last 105.28) - open/close 31272 (time 1746147503.33 total 283.04 last 136.53) - open/close 32653 (time 1746147513.33 total 293.04 last 138.02) - open/close 33997 (time 1746147523.33 total 303.04 last 134.39) - open/close 35217 (time 1746147533.34 total 313.04 last 121.98) - open/close 36459 (time 1746147543.34 total 323.05 last 124.16) - open/close 37651 (time 1746147553.34 total 333.05 last 119.16) - open/close 38842 (time 1746147563.34 total 343.05 last 119.07) - open/close 40000 (time 1746147572.19 total 351.90 last 130.90) - open/close 41326 (time 1746147582.19 total 361.90 last 132.59) - open/close 42657 (time 1746147592.19 total 371.90 last 133.08) - open/close 43968 (time 1746147602.20 total 381.91 last 131.03) - open/close 45236 (time 1746147612.20 total 391.91 last 126.77) - open/close 46765 (time 1746147622.20 total 401.91 last 152.83) - open/close 47849 (time 1746147632.21 total 411.92 last 108.29) - open/close 48919 (time 1746147642.22 total 421.93 last 106.95) - open/close 49968 (time 1746147652.23 total 431.94 last 104.82) - open/close 50992 (time 1746147662.23 total 441.94 last 102.40) - open/close 52058 (time 1746147672.23 total 451.94 last 106.52) - open/close 53079 (time 1746147682.24 total 461.95 last 102.05) - open/close 54158 (time 1746147692.24 total 471.95 last 107.90) - open/close 55316 (time 1746147702.24 total 481.95 last 115.79) - open/close 56370 (time 1746147712.25 total 491.95 last 105.36) - open/close 57432 (time 1746147722.25 total 501.96 last 106.15) - open/close 58490 (time 1746147732.26 total 511.97 last 105.69) - open/close 59643 (time 1746147742.27 total 521.98 last 115.22) - open/close 60000 (time 1746147745.02 total 524.73 last 129.85) - open/close 61108 (time 1746147755.02 total 534.73 last 110.72) - open/close 62216 (time 1746147765.04 total 544.74 last 110.68) - open/close 63353 (time 1746147775.04 total 554.75 last 113.67) - open/close 64513 (time 1746147785.04 total 564.75 last 115.95) total: 64768 open/close in 567.07 seconds: 114.21 ops/second - /unlink 2037 (time 1746147799.23 total 10.00 last 203.65) - /unlink 4465 (time 1746147809.23 total 20.00 last 242.76) - /unlink 7819 (time 1746147819.23 total 30.00 last 335.39) - /unlink 10000 (time 1746147826.24 total 37.01 last 311.40) - /unlink 12870 (time 1746147836.24 total 47.01 last 286.87) - /unlink 15296 (time 1746147846.24 total 57.01 last 242.58) - /unlink 17388 (time 1746147856.24 total 67.01 last 209.17) - /unlink 19653 (time 1746147866.25 total 77.02 last 226.42) - /unlink 22082 (time 1746147876.25 total 87.02 last 242.76) - /unlink 24734 (time 1746147886.25 total 97.03 last 265.14) - /unlink 27522 (time 1746147896.26 total 107.03 last 278.75) - /unlink 30000 (time 1746147906.16 total 116.94 last 250.12) - /unlink 32374 (time 1746147916.17 total 126.94 last 237.30) - /unlink 34802 (time 1746147926.17 total 136.94 last 242.72) - /unlink 37482 (time 1746147936.17 total 146.94 last 267.97) - /unlink 40000 (time 1746147945.38 total 156.15 last 273.56) - /unlink 42726 (time 1746147955.38 total 166.15 last 272.51) - /unlink 46247 (time 1746147965.38 total 176.15 last 352.05) - /unlink 48758 (time 1746147975.39 total 186.16 last 250.92) - /unlink 50000 (time 1746147979.50 total 190.27 last 302.37) - /unlink 52339 (time 1746147989.50 total 200.27 last 233.89) - /unlink 54636 (time 1746147999.50 total 210.27 last 229.61) - /unlink 57099 (time 1746148009.50 total 220.27 last 246.29) - /unlink 59825 (time 1746148019.50 total 230.27 last 272.60) - /unlink 62369 (time 1746148029.51 total 240.28 last 254.24) total: 64768 /unlink in 249.83 seconds: 259.24 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1679 (time 1746148065.53 total 10.00 last 167.87) - open/close 3287 (time 1746148075.54 total 20.01 last 160.68) - open/close 4739 (time 1746148085.54 total 30.01 last 145.17) - open/close 6214 (time 1746148095.55 total 40.01 last 147.46) - open/close 7589 (time 1746148105.55 total 50.02 last 137.48) - open/close 8931 (time 1746148115.55 total 60.02 last 134.13) - open/close 10000 (time 1746148122.77 total 67.24 last 148.13) - open/close 11704 (time 1746148132.78 total 77.25 last 170.27) - open/close 13277 (time 1746148142.78 total 87.25 last 157.27) - open/close 14869 (time 1746148152.78 total 97.25 last 159.16) - open/close 16606 (time 1746148162.79 total 107.25 last 173.62) - open/close 18080 (time 1746148172.79 total 117.26 last 147.33) - open/close 19401 (time 1746148182.81 total 127.28 last 131.86) - open/close 20000 (time 1746148187.16 total 131.63 last 137.74) - open/close 21389 (time 1746148197.16 total 141.63 last 138.89) - open/close 22814 (time 1746148207.16 total 151.63 last 142.47) - open/close 24336 (time 1746148217.17 total 161.63 last 152.13) - open/close 25919 (time 1746148227.17 total 171.64 last 158.27) - open/close 27469 (time 1746148237.17 total 181.64 last 154.98) - open/close 28792 (time 1746148247.18 total 191.64 last 132.22) - open/close 30000 (time 1746148255.32 total 199.79 last 148.26) - open/close 31415 (time 1746148265.33 total 209.79 last 141.45) - open/close 33038 (time 1746148275.33 total 219.79 last 162.30) - open/close 34696 (time 1746148285.33 total 229.80 last 165.71) - open/close 36324 (time 1746148295.33 total 239.80 last 162.80) - open/close 37859 (time 1746148305.33 total 249.80 last 153.48) - open/close 39271 (time 1746148315.34 total 259.80 last 141.16) - open/close 40000 (time 1746148320.63 total 265.10 last 137.66) - open/close 41370 (time 1746148330.64 total 275.10 last 136.95) - open/close 42833 (time 1746148340.64 total 285.11 last 146.21) - open/close 44443 (time 1746148350.65 total 295.11 last 160.95) - open/close 46126 (time 1746148360.65 total 305.11 last 168.27) - open/close 47496 (time 1746148370.65 total 315.12 last 136.93) - open/close 49119 (time 1746148380.66 total 325.12 last 162.24) - open/close 50000 (time 1746148386.44 total 330.90 last 152.39) - open/close 51569 (time 1746148396.45 total 340.91 last 156.78) - open/close 53302 (time 1746148406.45 total 350.92 last 173.25) - open/close 54689 (time 1746148416.45 total 360.92 last 138.68) - open/close 56123 (time 1746148426.45 total 370.92 last 143.33) - open/close 57477 (time 1746148436.46 total 380.92 last 135.39) - open/close 58804 (time 1746148446.46 total 390.93 last 132.66) - open/close 60000 (time 1746148455.00 total 399.46 last 140.08) - open/close 61276 (time 1746148465.00 total 409.47 last 127.51) - open/close 62658 (time 1746148475.00 total 419.47 last 138.19) - open/close 63850 (time 1746148485.00 total 429.47 last 119.19) total: 64768 open/close in 435.93 seconds: 148.57 ops/second - /unlink 2174 (time 1746148502.99 total 10.01 last 217.19) - /unlink 4364 (time 1746148513.05 total 20.07 last 217.78) - /unlink 6432 (time 1746148523.05 total 30.07 last 206.69) - /unlink 8509 (time 1746148533.05 total 40.07 last 207.68) - /unlink 10000 (time 1746148540.57 total 47.59 last 198.37) - /unlink 12282 (time 1746148550.57 total 57.59 last 228.19) - /unlink 14422 (time 1746148560.57 total 67.59 last 213.91) - /unlink 16913 (time 1746148570.57 total 77.59 last 249.10) - /unlink 18961 (time 1746148580.58 total 87.60 last 204.74) - /unlink 20000 (time 1746148585.23 total 92.25 last 223.09) - /unlink 22301 (time 1746148595.24 total 102.26 last 230.03) - /unlink 25170 (time 1746148605.24 total 112.26 last 286.90) - /unlink 27206 (time 1746148615.24 total 122.26 last 203.54) - /unlink 29023 (time 1746148625.24 total 132.26 last 181.66) - /unlink 30000 (time 1746148630.00 total 137.02 last 205.47) - /unlink 32061 (time 1746148640.00 total 147.02 last 206.04) - /unlink 34023 (time 1746148650.01 total 157.03 last 196.07) - /unlink 35930 (time 1746148660.02 total 167.04 last 190.45) - /unlink 38191 (time 1746148670.02 total 177.04 last 226.10) - /unlink 40000 (time 1746148678.21 total 185.23 last 220.96) - /unlink 42043 (time 1746148688.21 total 195.23 last 204.28) - /unlink 44119 (time 1746148698.22 total 205.23 last 207.43) - /unlink 46210 (time 1746148708.22 total 215.24 last 208.97) - /unlink 48570 (time 1746148718.23 total 225.24 last 235.94) - /unlink 50000 (time 1746148723.23 total 230.25 last 285.77) - /unlink 52138 (time 1746148733.23 total 240.25 last 213.74) - /unlink 54171 (time 1746148743.23 total 250.25 last 203.28) - /unlink 56103 (time 1746148753.24 total 260.25 last 193.13) - /unlink 57922 (time 1746148763.24 total 270.26 last 181.88) - /unlink 59756 (time 1746148773.24 total 280.26 last 183.39) - /unlink 61822 (time 1746148783.24 total 290.26 last 206.57) - /unlink 63958 (time 1746148793.24 total 300.26 last 213.53) total: 64768 /unlink in 304.44 seconds: 212.74 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1365 (time 1746148827.68 total 10.00 last 136.46) - open/close 2634 (time 1746148837.68 total 20.01 last 126.86) - open/close 3877 (time 1746148847.69 total 30.02 last 124.17) - open/close 5057 (time 1746148857.71 total 40.03 last 117.84) - open/close 6255 (time 1746148867.71 total 50.03 last 119.76) - open/close 7502 (time 1746148877.72 total 60.04 last 124.63) - open/close 8716 (time 1746148887.72 total 70.04 last 121.33) - open/close 10000 (time 1746148896.88 total 79.20 last 140.20) - open/close 11514 (time 1746148906.88 total 89.21 last 151.35) - open/close 12785 (time 1746148916.88 total 99.21 last 127.10) - open/close 13979 (time 1746148926.89 total 109.21 last 119.35) - open/close 15186 (time 1746148936.89 total 119.21 last 120.68) - open/close 16441 (time 1746148946.89 total 129.22 last 125.46) - open/close 17917 (time 1746148956.89 total 139.22 last 147.58) - open/close 19485 (time 1746148966.90 total 149.22 last 156.70) - open/close 20000 (time 1746148970.43 total 152.75 last 146.01) - open/close 21337 (time 1746148980.43 total 162.75 last 133.65) - open/close 22557 (time 1746148990.43 total 172.76 last 121.97) - open/close 23921 (time 1746149000.44 total 182.76 last 136.38) - open/close 25160 (time 1746149010.44 total 192.76 last 123.85) - open/close 26380 (time 1746149020.44 total 202.77 last 121.93) - open/close 27629 (time 1746149030.45 total 212.77 last 124.87) - open/close 28965 (time 1746149040.45 total 222.77 last 133.56) - open/close 30000 (time 1746149048.11 total 230.43 last 135.13) - open/close 31344 (time 1746149058.12 total 240.44 last 134.23) - open/close 32724 (time 1746149068.13 total 250.45 last 137.91) - open/close 33912 (time 1746149078.13 total 260.45 last 118.80) - open/close 34974 (time 1746149088.13 total 270.45 last 106.19) - open/close 35989 (time 1746149098.14 total 280.46 last 101.44) - open/close 37082 (time 1746149108.14 total 290.46 last 109.28) - open/close 38334 (time 1746149118.14 total 300.47 last 125.11) - open/close 39469 (time 1746149128.15 total 310.48 last 113.40) - open/close 40000 (time 1746149132.84 total 315.16 last 113.39) - open/close 41129 (time 1746149142.84 total 325.16 last 112.83) - open/close 42359 (time 1746149152.85 total 335.17 last 122.94) - open/close 43517 (time 1746149162.85 total 345.18 last 115.74) - open/close 44462 (time 1746149172.86 total 355.18 last 94.47) - open/close 45479 (time 1746149182.86 total 365.18 last 101.64) - open/close 46628 (time 1746149192.86 total 375.19 last 114.90) - open/close 47768 (time 1746149202.87 total 385.19 last 113.93) - open/close 49024 (time 1746149212.87 total 395.19 last 125.59) - open/close 50000 (time 1746149219.86 total 402.18 last 139.66) - open/close 51492 (time 1746149229.86 total 412.18 last 149.18) - open/close 52638 (time 1746149239.87 total 422.19 last 114.53) - open/close 53773 (time 1746149249.87 total 432.20 last 113.41) - open/close 54906 (time 1746149259.88 total 442.20 last 113.22) - open/close 56043 (time 1746149269.89 total 452.21 last 113.60) - open/close 57135 (time 1746149279.89 total 462.22 last 109.17) - open/close 58280 (time 1746149289.90 total 472.22 last 114.46) - open/close 59353 (time 1746149299.90 total 482.23 last 107.23) - open/close 60000 (time 1746149305.82 total 488.14 last 109.44) - open/close 61107 (time 1746149315.82 total 498.14 last 110.64) - open/close 62348 (time 1746149325.83 total 508.15 last 123.99) - open/close 63510 (time 1746149335.83 total 518.15 last 116.17) - open/close 64605 (time 1746149345.84 total 528.16 last 109.43) total: 64768 open/close in 529.52 seconds: 122.32 ops/second - /unlink 1590 (time 1746149359.13 total 10.00 last 158.94) - /unlink 3440 (time 1746149369.14 total 20.01 last 184.94) - /unlink 5158 (time 1746149379.14 total 30.01 last 171.78) - /unlink 7018 (time 1746149389.14 total 40.01 last 185.95) - /unlink 9258 (time 1746149399.14 total 50.01 last 223.96) - /unlink 10000 (time 1746149403.41 total 54.28 last 173.83) - /unlink 11784 (time 1746149413.41 total 64.28 last 178.35) - /unlink 13625 (time 1746149423.42 total 74.29 last 184.03) - /unlink 15888 (time 1746149433.42 total 84.29 last 226.25) - /unlink 18262 (time 1746149443.42 total 94.29 last 237.37) - /unlink 20000 (time 1746149451.18 total 102.05 last 224.02) - /unlink 21776 (time 1746149461.19 total 112.06 last 177.46) - /unlink 23517 (time 1746149471.19 total 122.06 last 174.06) - /unlink 25125 (time 1746149481.19 total 132.06 last 160.79) - /unlink 26597 (time 1746149491.19 total 142.07 last 147.12) - /unlink 28134 (time 1746149501.20 total 152.07 last 153.67) - /unlink 29861 (time 1746149511.20 total 162.07 last 172.68) - /unlink 31824 (time 1746149521.20 total 172.07 last 196.23) - /unlink 33958 (time 1746149531.20 total 182.07 last 213.40) - /unlink 35934 (time 1746149541.20 total 192.07 last 197.58) - /unlink 37799 (time 1746149551.21 total 202.08 last 186.45) - /unlink 39413 (time 1746149561.21 total 212.08 last 161.34) - /unlink 40000 (time 1746149564.48 total 215.35 last 179.66) - /unlink 41682 (time 1746149574.48 total 225.35 last 168.13) - /unlink 43679 (time 1746149584.49 total 235.36 last 199.60) - /unlink 45836 (time 1746149594.50 total 245.37 last 215.43) - /unlink 47993 (time 1746149604.50 total 255.37 last 215.67) - /unlink 50000 (time 1746149612.98 total 263.85 last 236.74) - /unlink 52357 (time 1746149622.98 total 273.85 last 235.65) - /unlink 54599 (time 1746149632.98 total 283.85 last 224.14) - /unlink 56842 (time 1746149642.99 total 293.86 last 224.17) - /unlink 58912 (time 1746149652.99 total 303.87 last 206.90) - /unlink 60000 (time 1746149658.27 total 309.14 last 206.15) - /unlink 62132 (time 1746149668.27 total 319.14 last 213.16) - /unlink 64612 (time 1746149678.27 total 329.15 last 247.95) total: 64768 /unlink in 329.57 seconds: 196.52 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1190 (time 1746149703.82 total 10.01 last 118.92) - open/close 2438 (time 1746149713.82 total 20.01 last 124.79) - open/close 3719 (time 1746149723.83 total 30.01 last 128.03) - open/close 5013 (time 1746149733.83 total 40.02 last 129.29) - open/close 6389 (time 1746149743.83 total 50.02 last 137.59) - open/close 7748 (time 1746149753.84 total 60.02 last 135.89) - open/close 9177 (time 1746149763.85 total 70.03 last 142.76) - open/close 10000 (time 1746149768.77 total 74.96 last 167.13) - open/close 11348 (time 1746149778.78 total 84.96 last 134.71) - open/close 12708 (time 1746149788.78 total 94.97 last 135.94) - open/close 13978 (time 1746149798.78 total 104.97 last 126.98) - open/close 15377 (time 1746149808.79 total 114.97 last 139.85) - open/close 16727 (time 1746149818.79 total 124.98 last 134.90) - open/close 18133 (time 1746149828.79 total 134.98 last 140.58) - open/close 19448 (time 1746149838.80 total 144.99 last 131.39) - open/close 20000 (time 1746149842.87 total 149.06 last 135.55) - open/close 21477 (time 1746149852.88 total 159.07 last 147.64) - open/close 23009 (time 1746149862.88 total 169.07 last 153.19) - open/close 24576 (time 1746149872.88 total 179.07 last 156.66) - open/close 25978 (time 1746149882.89 total 189.07 last 140.14) - open/close 27368 (time 1746149892.89 total 199.08 last 138.92) - open/close 29386 (time 1746149902.90 total 209.09 last 201.59) - open/close 30000 (time 1746149906.09 total 212.28 last 192.63) - open/close 31697 (time 1746149916.10 total 222.28 last 169.61) - open/close 33214 (time 1746149926.10 total 232.29 last 151.59) - open/close 34895 (time 1746149936.11 total 242.30 last 167.99) - open/close 36129 (time 1746149946.11 total 252.30 last 123.39) - open/close 37294 (time 1746149956.11 total 262.30 last 116.46) - open/close 38322 (time 1746149966.12 total 272.31 last 102.72) - open/close 39333 (time 1746149976.12 total 282.31 last 101.06) - open/close 40000 (time 1746149981.74 total 287.93 last 118.81) - open/close 41062 (time 1746149991.75 total 297.94 last 106.05) - open/close 42124 (time 1746150001.76 total 307.95 last 106.13) - open/close 43190 (time 1746150011.76 total 317.95 last 106.54) - open/close 44269 (time 1746150021.77 total 327.96 last 107.81) - open/close 45312 (time 1746150031.78 total 337.97 last 104.21) - open/close 46488 (time 1746150041.79 total 347.98 last 117.48) - open/close 47689 (time 1746150051.79 total 357.98 last 120.09) - open/close 48939 (time 1746150061.80 total 367.99 last 124.91) - open/close 50000 (time 1746150070.30 total 376.48 last 124.89) - open/close 51100 (time 1746150080.32 total 386.50 last 109.76) - open/close 52140 (time 1746150090.34 total 396.53 last 103.77) - open/close 53243 (time 1746150100.35 total 406.54 last 110.20) - open/close 54431 (time 1746150110.36 total 416.55 last 118.65) - open/close 55541 (time 1746150120.36 total 426.55 last 110.99) - open/close 56793 (time 1746150130.36 total 436.55 last 125.19) - open/close 58032 (time 1746150140.39 total 446.57 last 123.61) - open/close 59249 (time 1746150150.39 total 456.58 last 121.67) - open/close 60000 (time 1746150155.70 total 461.88 last 141.51) - open/close 61415 (time 1746150165.70 total 471.88 last 141.47) - open/close 62734 (time 1746150175.70 total 481.89 last 131.87) - open/close 64017 (time 1746150185.70 total 491.89 last 128.29) total: 64768 open/close in 497.91 seconds: 130.08 ops/second - /unlink 2209 (time 1746150203.52 total 10.00 last 220.89) - /unlink 4191 (time 1746150213.53 total 20.01 last 197.94) - /unlink 6053 (time 1746150223.54 total 30.02 last 186.05) - /unlink 7913 (time 1746150233.54 total 40.02 last 185.94) - /unlink 9658 (time 1746150243.55 total 50.03 last 174.42) - /unlink 10000 (time 1746150245.61 total 52.09 last 165.86) - /unlink 11506 (time 1746150255.61 total 62.10 last 150.52) - /unlink 12987 (time 1746150265.62 total 72.11 last 147.98) - /unlink 14612 (time 1746150275.62 total 82.11 last 162.45) - /unlink 16309 (time 1746150285.63 total 92.11 last 169.63) - /unlink 17851 (time 1746150295.63 total 102.11 last 154.19) - /unlink 19669 (time 1746150305.63 total 112.12 last 181.74) - /unlink 21269 (time 1746150315.64 total 122.12 last 159.95) - /unlink 22830 (time 1746150325.64 total 132.12 last 156.09) - /unlink 24481 (time 1746150335.64 total 142.12 last 165.04) - /unlink 26163 (time 1746150345.68 total 152.17 last 167.47) - /unlink 27908 (time 1746150355.68 total 162.17 last 174.49) - /unlink 29851 (time 1746150365.69 total 172.17 last 194.20) - /unlink 32183 (time 1746150375.69 total 182.18 last 233.08) - /unlink 33952 (time 1746150385.70 total 192.18 last 176.82) - /unlink 35826 (time 1746150395.70 total 202.19 last 187.34) - /unlink 37603 (time 1746150405.70 total 212.19 last 177.67) - /unlink 39515 (time 1746150415.71 total 222.19 last 191.06) - /unlink 40000 (time 1746150418.04 total 224.53 last 207.86) - /unlink 42481 (time 1746150428.04 total 234.53 last 248.09) - /unlink 45000 (time 1746150438.05 total 244.53 last 251.87) - /unlink 47114 (time 1746150448.05 total 254.53 last 211.37) - /unlink 49341 (time 1746150458.05 total 264.53 last 222.67) - /unlink 50000 (time 1746150461.25 total 267.73 last 205.99) - /unlink 52115 (time 1746150471.25 total 277.74 last 211.39) - /unlink 54428 (time 1746150481.25 total 287.74 last 231.29) - /unlink 56899 (time 1746150491.26 total 297.74 last 246.99) - /unlink 59283 (time 1746150501.26 total 307.74 last 238.34) - /unlink 60000 (time 1746150504.32 total 310.80 last 234.43) - /unlink 62369 (time 1746150514.32 total 320.80 last 236.88) - /unlink 64245 (time 1746150524.32 total 330.80 last 187.57) total: 64768 /unlink in 333.23 seconds: 194.36 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server unloading modules via unload_modules_local on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 PASS 106 (4311s) Stopping clients: oleg229-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg229-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 oleg229-server: oleg229-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg229-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg229-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4440 sec ========== 21:49:53 (1746150593) === conf-sanity: start cleanup 21:49:53 (1746150593) === === conf-sanity: finish cleanup 21:49:55 (1746150595) ===