-----============= acceptance-small: conf-sanity ============----- Thu May 1 14:59:22 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg232-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg232-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg232-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg232-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg232-server oleg232-server: oleg232-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: MODOPTS_LIBCFS= oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg232-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:00:21 (1746126021) Stopping clients: oleg232-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg232-client.virtnet /mnt/lustre2 (opts:-f) oleg232-server: oleg232-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: MODOPTS_LIBCFS= oleg232-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg232-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2623 (time 1746126060.52 total 10.00 last 262.28) - open/close 5242 (time 1746126070.52 total 20.00 last 261.83) - open/close 7627 (time 1746126080.52 total 30.01 last 238.42) - open/close 9024 (time 1746126090.53 total 40.02 last 139.56) - open/close 9767 (time 1746126100.55 total 50.03 last 74.18) - open/close 10000 (time 1746126104.24 total 53.72 last 63.13) - open/close 10701 (time 1746126114.25 total 63.73 last 70.03) - open/close 11400 (time 1746126124.25 total 73.74 last 69.86) - open/close 12249 (time 1746126134.26 total 83.74 last 84.87) - open/close 13269 (time 1746126144.26 total 93.75 last 101.95) - open/close 14338 (time 1746126154.27 total 103.75 last 106.83) - open/close 15491 (time 1746126164.27 total 113.76 last 115.26) - open/close 16746 (time 1746126174.28 total 123.76 last 125.47) - open/close 17906 (time 1746126184.28 total 133.77 last 115.94) - open/close 19151 (time 1746126194.28 total 143.77 last 124.47) - open/close 20000 (time 1746126199.39 total 148.87 last 166.40) - open/close 21543 (time 1746126209.39 total 158.87 last 154.24) - open/close 23185 (time 1746126219.40 total 168.88 last 164.06) - open/close 24527 (time 1746126229.41 total 178.89 last 134.09) - open/close 25840 (time 1746126239.41 total 188.89 last 131.25) - open/close 27403 (time 1746126249.42 total 198.90 last 156.19) - open/close 28680 (time 1746126259.43 total 208.92 last 127.49) - open/close 29842 (time 1746126269.44 total 218.92 last 116.14) - open/close 30953 (time 1746126279.46 total 228.95 last 110.84) - open/close 32019 (time 1746126289.47 total 238.96 last 106.51) - open/close 33081 (time 1746126299.47 total 248.96 last 106.16) - open/close 34398 (time 1746126309.49 total 258.97 last 131.52) - open/close 35646 (time 1746126319.49 total 268.98 last 124.73) - open/close 36963 (time 1746126329.50 total 278.98 last 131.62) - open/close 38279 (time 1746126339.50 total 288.99 last 131.54) - open/close 39686 (time 1746126349.50 total 298.99 last 140.70) - open/close 40000 (time 1746126351.60 total 301.09 last 149.70) - open/close 41388 (time 1746126361.60 total 311.09 last 138.76) - open/close 42977 (time 1746126371.61 total 321.09 last 158.84) - open/close 44543 (time 1746126381.62 total 331.10 last 156.42) - open/close 45906 (time 1746126391.62 total 341.11 last 136.25) - open/close 47537 (time 1746126401.64 total 351.12 last 162.84) - open/close 49495 (time 1746126411.64 total 361.13 last 195.76) - open/close 50000 (time 1746126414.70 total 364.18 last 165.17) - open/close 51323 (time 1746126424.71 total 374.19 last 132.19) - open/close 52828 (time 1746126434.71 total 384.19 last 150.48) - open/close 54465 (time 1746126444.71 total 394.20 last 163.66) - open/close 56061 (time 1746126454.71 total 404.20 last 159.58) - open/close 57591 (time 1746126464.72 total 414.20 last 152.95) - open/close 58939 (time 1746126474.72 total 424.20 last 134.76) - open/close 60000 (time 1746126482.64 total 432.12 last 134.01) - open/close 61457 (time 1746126492.64 total 442.12 last 145.67) - open/close 62674 (time 1746126502.64 total 452.13 last 121.65) - open/close 63836 (time 1746126512.64 total 462.13 last 116.18) total: 64768 open/close in 470.58 seconds: 137.64 ops/second - /unlink 2138 (time 1746126532.85 total 10.00 last 213.74) - /unlink 4175 (time 1746126542.86 total 20.01 last 203.58) - /unlink 6253 (time 1746126552.86 total 30.01 last 207.71) - /unlink 8378 (time 1746126562.86 total 40.02 last 212.42) - /unlink 10000 (time 1746126569.02 total 46.17 last 263.52) - /unlink 12551 (time 1746126579.02 total 56.17 last 255.05) - /unlink 14615 (time 1746126589.02 total 66.18 last 206.35) - /unlink 17228 (time 1746126599.03 total 76.18 last 261.21) - /unlink 19639 (time 1746126609.03 total 86.18 last 241.10) - /unlink 22408 (time 1746126619.03 total 96.18 last 276.90) - /unlink 24830 (time 1746126629.03 total 106.18 last 242.13) - /unlink 27213 (time 1746126639.03 total 116.18 last 238.29) - /unlink 29970 (time 1746126649.03 total 126.18 last 275.67) - /unlink 32698 (time 1746126659.03 total 136.18 last 272.76) - /unlink 35535 (time 1746126669.04 total 146.19 last 283.64) - /unlink 38291 (time 1746126679.04 total 156.19 last 275.57) - /unlink 40000 (time 1746126686.05 total 163.20 last 243.76) - /unlink 42494 (time 1746126696.05 total 173.20 last 249.38) - /unlink 45242 (time 1746126706.05 total 183.20 last 274.75) - /unlink 48128 (time 1746126716.05 total 193.20 last 288.56) - /unlink 50000 (time 1746126723.13 total 200.29 last 264.28) - /unlink 52550 (time 1746126733.14 total 210.29 last 254.98) - /unlink 54992 (time 1746126743.14 total 220.29 last 244.18) - /unlink 57519 (time 1746126753.14 total 230.29 last 252.63) - /unlink 60000 (time 1746126763.09 total 240.24 last 249.35) - /unlink 62105 (time 1746126773.10 total 250.25 last 210.37) - /unlink 63910 (time 1746126783.10 total 260.25 last 180.47) total: 64768 /unlink in 264.10 seconds: 245.24 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1241 (time 1746126813.96 total 10.01 last 124.00) - open/close 2466 (time 1746126823.96 total 20.01 last 122.49) - open/close 3779 (time 1746126833.96 total 30.01 last 131.29) - open/close 5403 (time 1746126843.96 total 40.01 last 162.37) - open/close 6784 (time 1746126853.96 total 50.01 last 138.07) - open/close 8025 (time 1746126863.98 total 60.03 last 123.88) - open/close 9370 (time 1746126874.00 total 70.04 last 134.34) - open/close 10000 (time 1746126879.13 total 75.18 last 122.74) - open/close 11309 (time 1746126889.13 total 85.18 last 130.87) - open/close 12741 (time 1746126899.14 total 95.19 last 143.08) - open/close 14357 (time 1746126909.14 total 105.19 last 161.52) - open/close 15860 (time 1746126919.14 total 115.19 last 150.27) - open/close 17610 (time 1746126929.15 total 125.20 last 174.85) - open/close 19466 (time 1746126939.16 total 135.20 last 185.55) - open/close 20000 (time 1746126942.26 total 138.31 last 172.22) - open/close 21634 (time 1746126952.26 total 148.31 last 163.37) - open/close 23230 (time 1746126962.26 total 158.31 last 159.58) - open/close 24841 (time 1746126972.27 total 168.32 last 160.94) - open/close 26346 (time 1746126982.27 total 178.32 last 150.43) - open/close 27673 (time 1746126992.28 total 188.33 last 132.64) - open/close 29025 (time 1746127002.29 total 198.34 last 135.08) - open/close 30000 (time 1746127009.70 total 205.74 last 131.60) - open/close 31409 (time 1746127019.70 total 215.75 last 140.86) - open/close 32716 (time 1746127029.70 total 225.75 last 130.65) - open/close 34021 (time 1746127039.70 total 235.75 last 130.46) - open/close 35319 (time 1746127049.71 total 245.76 last 129.75) - open/close 36485 (time 1746127059.71 total 255.76 last 116.57) - open/close 37796 (time 1746127069.71 total 265.76 last 131.06) - open/close 39339 (time 1746127079.72 total 275.76 last 154.28) - open/close 40000 (time 1746127084.11 total 280.16 last 150.23) - open/close 41488 (time 1746127094.13 total 290.18 last 148.62) - open/close 42789 (time 1746127104.13 total 300.18 last 130.08) - open/close 44305 (time 1746127114.13 total 310.18 last 151.58) - open/close 45646 (time 1746127124.13 total 320.18 last 134.09) - open/close 47063 (time 1746127134.13 total 330.18 last 141.66) - open/close 48392 (time 1746127144.14 total 340.18 last 132.86) - open/close 49800 (time 1746127154.14 total 350.19 last 140.77) - open/close 51271 (time 1746127164.14 total 360.19 last 147.09) - open/close 52768 (time 1746127174.14 total 370.19 last 149.62) - open/close 54174 (time 1746127184.15 total 380.19 last 140.58) - open/close 55746 (time 1746127194.15 total 390.20 last 157.19) - open/close 57571 (time 1746127204.15 total 400.20 last 182.50) - open/close 59229 (time 1746127214.15 total 410.20 last 165.78) - open/close 60000 (time 1746127219.24 total 415.29 last 151.33) - open/close 61585 (time 1746127229.25 total 425.30 last 158.39) - open/close 63090 (time 1746127239.25 total 435.30 last 150.45) total: 64768 open/close in 443.51 seconds: 146.04 ops/second - /unlink 3549 (time 1746127258.62 total 10.00 last 354.86) - /unlink 6536 (time 1746127268.62 total 20.00 last 298.64) - /unlink 9760 (time 1746127278.62 total 30.01 last 322.26) - /unlink 12901 (time 1746127288.62 total 40.01 last 314.09) - /unlink 15448 (time 1746127298.63 total 50.01 last 254.58) - /unlink 17811 (time 1746127308.63 total 60.01 last 236.27) - /unlink 20000 (time 1746127315.88 total 67.27 last 301.76) - /unlink 22689 (time 1746127325.88 total 77.27 last 268.90) - /unlink 25346 (time 1746127335.89 total 87.27 last 265.66) - /unlink 28221 (time 1746127345.89 total 97.27 last 287.45) - /unlink 30000 (time 1746127353.31 total 104.69 last 239.67) - /unlink 32532 (time 1746127363.31 total 114.70 last 253.16) - /unlink 34996 (time 1746127373.31 total 124.70 last 246.36) - /unlink 37612 (time 1746127383.32 total 134.70 last 261.46) - /unlink 40000 (time 1746127392.04 total 143.42 last 273.95) - /unlink 42956 (time 1746127402.04 total 153.42 last 295.50) - /unlink 45693 (time 1746127412.04 total 163.42 last 273.69) - /unlink 48305 (time 1746127422.04 total 173.42 last 261.19) - /unlink 50000 (time 1746127428.58 total 179.97 last 259.06) - /unlink 52213 (time 1746127438.59 total 189.97 last 221.20) - /unlink 54128 (time 1746127448.59 total 199.97 last 191.42) - /unlink 56301 (time 1746127458.59 total 209.98 last 217.27) - /unlink 58635 (time 1746127468.59 total 219.98 last 233.39) - /unlink 60000 (time 1746127474.19 total 225.58 last 243.78) - /unlink 62492 (time 1746127484.19 total 235.58 last 249.17) total: 64768 /unlink in 244.54 seconds: 264.85 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1631 (time 1746127520.80 total 10.01 last 162.97) - open/close 3204 (time 1746127530.81 total 20.01 last 157.26) - open/close 5181 (time 1746127540.81 total 30.02 last 197.60) - open/close 6552 (time 1746127550.81 total 40.02 last 137.07) - open/close 7855 (time 1746127560.82 total 50.02 last 130.21) - open/close 9182 (time 1746127570.82 total 60.03 last 132.66) - open/close 10000 (time 1746127578.07 total 67.28 last 112.83) - open/close 11330 (time 1746127588.08 total 77.28 last 132.96) - open/close 12663 (time 1746127598.08 total 87.28 last 133.27) - open/close 13934 (time 1746127608.08 total 97.28 last 127.09) - open/close 15292 (time 1746127618.08 total 107.29 last 135.77) - open/close 16659 (time 1746127628.08 total 117.29 last 136.68) - open/close 17968 (time 1746127638.08 total 127.29 last 130.90) - open/close 19259 (time 1746127648.09 total 137.29 last 129.07) - open/close 20000 (time 1746127653.98 total 143.18 last 125.78) - open/close 21288 (time 1746127663.99 total 153.19 last 128.67) - open/close 22535 (time 1746127673.99 total 163.19 last 124.68) - open/close 23725 (time 1746127683.99 total 173.20 last 118.96) - open/close 24892 (time 1746127694.00 total 183.20 last 116.62) - open/close 26285 (time 1746127704.01 total 193.21 last 139.20) - open/close 28136 (time 1746127714.01 total 203.21 last 185.09) - open/close 29564 (time 1746127724.01 total 213.21 last 142.73) - open/close 30000 (time 1746127726.93 total 216.13 last 149.57) - open/close 31576 (time 1746127736.93 total 226.13 last 157.56) - open/close 33301 (time 1746127746.93 total 236.13 last 172.45) - open/close 35066 (time 1746127756.93 total 246.14 last 176.44) - open/close 36605 (time 1746127766.94 total 256.14 last 153.79) - open/close 37987 (time 1746127776.95 total 266.15 last 138.14) - open/close 39376 (time 1746127786.95 total 276.15 last 138.85) - open/close 40000 (time 1746127792.07 total 281.27 last 121.90) - open/close 41070 (time 1746127802.07 total 291.28 last 106.93) - open/close 42297 (time 1746127812.08 total 301.28 last 122.67) - open/close 43442 (time 1746127822.08 total 311.29 last 114.44) - open/close 44658 (time 1746127832.10 total 321.31 last 121.35) - open/close 45818 (time 1746127842.11 total 331.31 last 115.90) - open/close 47123 (time 1746127852.11 total 341.32 last 130.46) - open/close 48268 (time 1746127862.12 total 351.32 last 114.46) - open/close 49660 (time 1746127872.12 total 361.33 last 139.10) - open/close 51145 (time 1746127882.13 total 371.33 last 148.46) - open/close 52493 (time 1746127892.14 total 381.34 last 134.67) - open/close 53806 (time 1746127902.14 total 391.35 last 131.21) - open/close 55165 (time 1746127912.15 total 401.35 last 135.87) - open/close 56487 (time 1746127922.15 total 411.35 last 132.14) - open/close 57756 (time 1746127932.16 total 421.36 last 126.83) - open/close 58915 (time 1746127942.16 total 431.36 last 115.85) - open/close 60000 (time 1746127950.52 total 439.72 last 129.82) - open/close 61294 (time 1746127960.52 total 449.72 last 129.40) - open/close 62558 (time 1746127970.52 total 459.73 last 126.33) - open/close 63970 (time 1746127980.53 total 469.73 last 141.15) total: 64768 open/close in 474.65 seconds: 136.45 ops/second - /unlink 2102 (time 1746127996.83 total 10.00 last 210.13) - /unlink 4108 (time 1746128006.84 total 20.01 last 200.52) - /unlink 6123 (time 1746128016.84 total 30.01 last 201.38) - /unlink 8134 (time 1746128026.85 total 40.02 last 201.00) - /unlink 10000 (time 1746128035.98 total 49.15 last 204.36) - /unlink 11936 (time 1746128045.98 total 59.15 last 193.54) - /unlink 13903 (time 1746128055.98 total 69.16 last 196.65) - /unlink 15994 (time 1746128065.99 total 79.16 last 209.06) - /unlink 18210 (time 1746128075.99 total 89.16 last 221.48) - /unlink 20000 (time 1746128084.51 total 97.68 last 210.16) - /unlink 22015 (time 1746128094.51 total 107.68 last 201.40) - /unlink 23655 (time 1746128104.51 total 117.69 last 163.99) - /unlink 25264 (time 1746128114.52 total 127.69 last 160.78) - /unlink 26810 (time 1746128124.52 total 137.69 last 154.58) - /unlink 28442 (time 1746128134.53 total 147.70 last 163.10) - /unlink 30000 (time 1746128143.81 total 156.98 last 167.90) - /unlink 32052 (time 1746128153.81 total 166.98 last 205.16) - /unlink 34209 (time 1746128163.81 total 176.98 last 215.64) - /unlink 36124 (time 1746128173.82 total 186.99 last 191.42) - /unlink 38533 (time 1746128183.82 total 196.99 last 240.81) - /unlink 40000 (time 1746128189.52 total 202.70 last 257.21) - /unlink 41977 (time 1746128199.53 total 212.70 last 197.63) - /unlink 43893 (time 1746128209.53 total 222.70 last 191.58) - /unlink 45927 (time 1746128219.53 total 232.70 last 203.35) - /unlink 48643 (time 1746128229.53 total 242.71 last 271.50) - /unlink 50000 (time 1746128234.23 total 247.40 last 289.21) - /unlink 53183 (time 1746128244.23 total 257.40 last 318.19) - /unlink 56783 (time 1746128254.23 total 267.40 last 359.94) - /unlink 60000 (time 1746128264.04 total 277.21 last 328.00) - /unlink 63361 (time 1746128274.04 total 287.21 last 335.97) total: 64768 /unlink in 292.60 seconds: 221.36 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1863 (time 1746128310.80 total 10.01 last 186.20) - open/close 3363 (time 1746128320.80 total 20.01 last 149.92) - open/close 5110 (time 1746128330.81 total 30.01 last 174.65) - open/close 6762 (time 1746128340.81 total 40.01 last 165.20) - open/close 8235 (time 1746128350.81 total 50.02 last 147.27) - open/close 9672 (time 1746128360.81 total 60.02 last 143.63) - open/close 10000 (time 1746128363.60 total 62.81 last 117.72) - open/close 11422 (time 1746128373.61 total 72.81 last 142.10) - open/close 12646 (time 1746128383.61 total 82.82 last 122.36) - open/close 13785 (time 1746128393.61 total 92.82 last 113.89) - open/close 14918 (time 1746128403.62 total 102.82 last 113.24) - open/close 16133 (time 1746128413.62 total 112.83 last 121.41) - open/close 17343 (time 1746128423.63 total 122.84 last 120.95) - open/close 18512 (time 1746128433.63 total 132.84 last 116.86) - open/close 19699 (time 1746128443.63 total 142.84 last 118.67) - open/close 20000 (time 1746128446.07 total 145.28 last 123.59) - open/close 21083 (time 1746128456.07 total 155.28 last 108.27) - open/close 22123 (time 1746128466.08 total 165.29 last 103.86) - open/close 23143 (time 1746128476.09 total 175.29 last 101.99) - open/close 24303 (time 1746128486.09 total 185.30 last 115.92) - open/close 25317 (time 1746128496.10 total 195.31 last 101.30) - open/close 26462 (time 1746128506.13 total 205.34 last 114.22) - open/close 27446 (time 1746128516.13 total 215.34 last 98.38) - open/close 28400 (time 1746128526.14 total 225.34 last 95.33) - open/close 29428 (time 1746128536.14 total 235.35 last 102.76) - open/close 30000 (time 1746128541.61 total 240.82 last 104.51) - open/close 31086 (time 1746128551.62 total 250.83 last 108.55) - open/close 32177 (time 1746128561.63 total 260.83 last 109.03) - open/close 33211 (time 1746128571.63 total 270.84 last 103.37) - open/close 34252 (time 1746128581.64 total 280.85 last 103.98) - open/close 35336 (time 1746128591.65 total 290.86 last 108.32) - open/close 36252 (time 1746128601.65 total 300.86 last 91.57) - open/close 37194 (time 1746128611.66 total 310.87 last 94.07) - open/close 38130 (time 1746128621.67 total 320.87 last 93.59) - open/close 39070 (time 1746128631.67 total 330.88 last 93.94) - open/close 39980 (time 1746128641.68 total 340.89 last 90.94) - open/close 40986 (time 1746128651.69 total 350.90 last 100.49) - open/close 41913 (time 1746128661.69 total 360.90 last 92.68) - open/close 42851 (time 1746128671.69 total 370.90 last 93.79) - open/close 43812 (time 1746128681.70 total 380.90 last 96.05) - open/close 44761 (time 1746128691.70 total 390.90 last 94.89) - open/close 45693 (time 1746128701.72 total 400.93 last 92.99) - open/close 46624 (time 1746128711.73 total 410.94 last 92.99) - open/close 47615 (time 1746128721.74 total 420.94 last 99.04) - open/close 48584 (time 1746128731.75 total 430.96 last 96.74) - open/close 49481 (time 1746128741.76 total 440.97 last 89.61) - open/close 50000 (time 1746128747.73 total 446.94 last 87.00) - open/close 50956 (time 1746128757.74 total 456.95 last 95.45) - open/close 52050 (time 1746128767.75 total 466.95 last 109.38) - open/close 53222 (time 1746128777.76 total 476.96 last 117.09) - open/close 54316 (time 1746128787.76 total 486.97 last 109.35) - open/close 55415 (time 1746128797.76 total 496.97 last 109.86) - open/close 56637 (time 1746128807.77 total 506.97 last 122.18) - open/close 57864 (time 1746128817.77 total 516.97 last 122.70) - open/close 58884 (time 1746128827.77 total 526.98 last 101.91) - open/close 60000 (time 1746128837.02 total 536.23 last 120.68) - open/close 61179 (time 1746128847.03 total 546.23 last 117.85) - open/close 62313 (time 1746128857.03 total 556.23 last 113.39) - open/close 63534 (time 1746128867.03 total 566.24 last 122.06) - open/close 64647 (time 1746128877.04 total 576.24 last 111.23) total: 64768 open/close in 577.27 seconds: 112.20 ops/second - /unlink 2067 (time 1746128889.88 total 10.00 last 206.69) - /unlink 3718 (time 1746128899.89 total 20.00 last 165.06) - /unlink 5422 (time 1746128909.89 total 30.00 last 170.38) - /unlink 7166 (time 1746128919.89 total 40.01 last 174.39) - /unlink 8886 (time 1746128929.90 total 50.01 last 171.87) - /unlink 10000 (time 1746128936.44 total 56.55 last 170.29) - /unlink 11767 (time 1746128946.44 total 66.56 last 176.64) - /unlink 13558 (time 1746128956.44 total 76.56 last 179.08) - /unlink 15160 (time 1746128966.45 total 86.56 last 160.12) - /unlink 16801 (time 1746128976.45 total 96.56 last 164.08) - /unlink 18807 (time 1746128986.45 total 106.56 last 200.60) - /unlink 20000 (time 1746128992.57 total 112.69 last 194.80) - /unlink 21932 (time 1746129002.57 total 122.69 last 193.17) - /unlink 24027 (time 1746129012.58 total 132.70 last 209.40) - /unlink 26008 (time 1746129022.58 total 142.70 last 198.06) - /unlink 27791 (time 1746129032.59 total 152.71 last 178.11) - /unlink 29607 (time 1746129042.59 total 162.71 last 181.57) - /unlink 30000 (time 1746129044.63 total 164.75 last 192.79) - /unlink 31821 (time 1746129054.64 total 174.75 last 182.02) - /unlink 33920 (time 1746129064.64 total 184.76 last 209.81) - /unlink 35820 (time 1746129074.64 total 194.76 last 189.99) - /unlink 37850 (time 1746129084.64 total 204.76 last 202.93) - /unlink 39787 (time 1746129094.65 total 214.76 last 193.64) - /unlink 41780 (time 1746129104.65 total 224.77 last 199.17) - /unlink 43649 (time 1746129114.65 total 234.77 last 186.90) - /unlink 45319 (time 1746129124.66 total 244.77 last 166.96) - /unlink 47116 (time 1746129134.66 total 254.78 last 179.64) - /unlink 49046 (time 1746129144.66 total 264.78 last 192.97) - /unlink 50000 (time 1746129148.72 total 268.84 last 235.06) - /unlink 52406 (time 1746129158.73 total 278.84 last 240.47) - /unlink 54875 (time 1746129168.73 total 288.85 last 246.83) - /unlink 57296 (time 1746129178.74 total 298.85 last 241.94) - /unlink 59134 (time 1746129188.75 total 308.86 last 183.60) - /unlink 60000 (time 1746129192.91 total 313.03 last 208.02) - /unlink 62307 (time 1746129202.91 total 323.03 last 230.65) total: 64768 /unlink in 330.85 seconds: 195.76 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1476 (time 1746129240.91 total 10.00 last 147.58) - open/close 2982 (time 1746129250.92 total 20.00 last 150.55) - open/close 4468 (time 1746129260.92 total 30.01 last 148.50) - open/close 5804 (time 1746129270.93 total 40.01 last 133.58) - open/close 7131 (time 1746129280.93 total 50.02 last 132.67) - open/close 8676 (time 1746129290.93 total 60.02 last 154.43) - open/close 10000 (time 1746129300.66 total 69.75 last 136.12) - open/close 11521 (time 1746129310.66 total 79.75 last 152.09) - open/close 13063 (time 1746129320.66 total 89.75 last 154.15) - open/close 14653 (time 1746129330.67 total 99.76 last 158.92) - open/close 16196 (time 1746129340.67 total 109.76 last 154.22) - open/close 17819 (time 1746129350.68 total 119.77 last 162.23) - open/close 19371 (time 1746129360.69 total 129.77 last 155.10) - open/close 20000 (time 1746129365.18 total 134.26 last 140.05) - open/close 21215 (time 1746129375.18 total 144.27 last 121.42) - open/close 22722 (time 1746129385.19 total 154.28 last 150.61) - open/close 23996 (time 1746129395.20 total 164.28 last 127.31) - open/close 25293 (time 1746129405.20 total 174.28 last 129.69) - open/close 26610 (time 1746129415.20 total 184.29 last 131.65) - open/close 27878 (time 1746129425.20 total 194.29 last 126.79) - open/close 29071 (time 1746129435.21 total 204.29 last 119.24) - open/close 30000 (time 1746129442.88 total 211.97 last 120.98) - open/close 31213 (time 1746129452.89 total 221.98 last 121.23) - open/close 32688 (time 1746129462.90 total 231.99 last 147.36) - open/close 33852 (time 1746129472.91 total 242.00 last 116.24) - open/close 35187 (time 1746129482.91 total 252.00 last 133.49) - open/close 36573 (time 1746129492.92 total 262.01 last 138.53) - open/close 38030 (time 1746129502.93 total 272.02 last 145.53) - open/close 39212 (time 1746129512.93 total 282.02 last 118.16) - open/close 40000 (time 1746129519.34 total 288.43 last 122.96) - open/close 41344 (time 1746129529.35 total 298.44 last 134.28) - open/close 42980 (time 1746129539.35 total 308.44 last 163.57) - open/close 44483 (time 1746129549.36 total 318.45 last 150.20) - open/close 45780 (time 1746129559.36 total 328.45 last 129.70) - open/close 46979 (time 1746129569.36 total 338.45 last 119.89) - open/close 48131 (time 1746129579.36 total 348.45 last 115.18) - open/close 49326 (time 1746129589.36 total 358.45 last 119.48) - open/close 50000 (time 1746129594.22 total 363.31 last 138.76) - open/close 51279 (time 1746129604.22 total 373.31 last 127.86) - open/close 52348 (time 1746129614.23 total 383.32 last 106.84) - open/close 53525 (time 1746129624.23 total 393.32 last 117.69) - open/close 55361 (time 1746129634.24 total 403.32 last 183.49) - open/close 56861 (time 1746129644.24 total 413.33 last 149.91) - open/close 58295 (time 1746129654.25 total 423.34 last 143.31) - open/close 59628 (time 1746129664.25 total 433.34 last 133.24) - open/close 60000 (time 1746129666.61 total 435.69 last 158.13) - open/close 61590 (time 1746129676.61 total 445.70 last 158.97) - open/close 63115 (time 1746129686.62 total 455.70 last 152.38) - open/close 64640 (time 1746129696.62 total 465.71 last 152.39) total: 64768 open/close in 466.48 seconds: 138.85 ops/second - /unlink 2677 (time 1746129708.94 total 10.00 last 267.60) - /unlink 5400 (time 1746129718.94 total 20.00 last 272.29) - /unlink 8517 (time 1746129728.94 total 30.01 last 311.67) - /unlink 10000 (time 1746129734.23 total 35.30 last 280.21) - /unlink 12710 (time 1746129744.23 total 45.30 last 270.94) - /unlink 15154 (time 1746129754.24 total 55.30 last 244.32) - /unlink 17509 (time 1746129764.24 total 65.30 last 235.49) - /unlink 20000 (time 1746129773.28 total 74.34 last 275.54) - /unlink 22097 (time 1746129783.28 total 84.35 last 209.62) - /unlink 24559 (time 1746129793.29 total 94.35 last 246.04) - /unlink 26688 (time 1746129803.29 total 104.36 last 212.84) - /unlink 28801 (time 1746129813.29 total 114.36 last 211.27) - /unlink 30000 (time 1746129818.62 total 119.69 last 224.89) - /unlink 32552 (time 1746129828.62 total 129.69 last 255.16) - /unlink 35607 (time 1746129838.63 total 139.69 last 305.43) - /unlink 37944 (time 1746129848.63 total 149.69 last 233.69) - /unlink 40000 (time 1746129857.59 total 158.66 last 229.41) - /unlink 43267 (time 1746129867.59 total 168.66 last 326.64) - /unlink 46375 (time 1746129877.59 total 178.66 last 310.79) - /unlink 49635 (time 1746129887.59 total 188.66 last 325.94) - /unlink 52967 (time 1746129897.59 total 198.66 last 333.16) - /unlink 56660 (time 1746129907.60 total 208.67 last 369.16) - /unlink 59300 (time 1746129917.60 total 218.67 last 263.99) - /unlink 60000 (time 1746129919.95 total 221.02 last 297.70) - /unlink 62695 (time 1746129929.95 total 231.02 last 269.42) - /unlink 64671 (time 1746129939.95 total 241.02 last 197.58) total: 64768 /unlink in 241.44 seconds: 268.25 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server unloading modules via unload_modules_local on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 PASS 106 (3949s) Stopping clients: oleg232-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg232-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 oleg232-server: oleg232-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: MODOPTS_LIBCFS= oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg232-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4035 sec ========== 16:06:38 (1746129998) === conf-sanity: start cleanup 16:06:39 (1746129999) === === conf-sanity: finish cleanup 16:06:40 (1746130000) ===