-----============= acceptance-small: conf-sanity ============----- Thu May 1 15:49:02 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg305-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg305-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg305-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg305-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg305-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg305-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg305-server oleg305-server: oleg305-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg305-server' oleg305-server: oleg305-server.virtnet: executing load_modules_local oleg305-server: Loading modules from /home/green/git/lustre-release/lustre oleg305-server: detected 4 online CPUs by sysfs oleg305-server: MODOPTS_LIBCFS= oleg305-server: Force libcfs to create 2 CPU partitions oleg305-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg305-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg305-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg305-server: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg305-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg305-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg305-server stop mds service on oleg305-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg305-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:50:44 (1746129044) Stopping clients: oleg305-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg305-client.virtnet /mnt/lustre2 (opts:-f) oleg305-server: oleg305-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg305-server' oleg305-server: oleg305-server.virtnet: executing load_modules_local oleg305-server: Loading modules from /home/green/git/lustre-release/lustre oleg305-server: detected 4 online CPUs by sysfs oleg305-server: MODOPTS_LIBCFS= oleg305-server: Force libcfs to create 2 CPU partitions oleg305-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory oleg305-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg305-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg305-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg305-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1163 (time 1746129112.55 total 10.00 last 116.29) - open/close 2319 (time 1746129122.56 total 20.01 last 115.54) - open/close 3460 (time 1746129132.56 total 30.01 last 114.05) - open/close 4639 (time 1746129142.56 total 40.01 last 117.88) - open/close 5852 (time 1746129152.57 total 50.02 last 121.22) - open/close 7145 (time 1746129162.57 total 60.02 last 129.25) - open/close 8238 (time 1746129172.58 total 70.03 last 109.24) - open/close 9364 (time 1746129182.58 total 80.03 last 112.54) - open/close 10000 (time 1746129188.39 total 85.84 last 109.56) - open/close 11006 (time 1746129198.40 total 95.85 last 100.45) - open/close 12157 (time 1746129208.42 total 105.87 last 114.94) - open/close 13236 (time 1746129218.42 total 115.87 last 107.86) - open/close 14489 (time 1746129228.42 total 125.87 last 125.28) - open/close 15943 (time 1746129238.43 total 135.88 last 145.32) - open/close 17266 (time 1746129248.43 total 145.88 last 132.24) - open/close 18544 (time 1746129258.43 total 155.88 last 127.77) - open/close 19885 (time 1746129268.44 total 165.89 last 133.99) - open/close 21123 (time 1746129278.45 total 175.90 last 123.71) - open/close 22199 (time 1746129288.45 total 185.90 last 107.58) - open/close 23280 (time 1746129298.46 total 195.91 last 108.00) - open/close 24323 (time 1746129308.46 total 205.91 last 104.28) - open/close 25486 (time 1746129318.48 total 215.93 last 116.13) - open/close 26675 (time 1746129328.49 total 225.94 last 118.71) - open/close 27928 (time 1746129338.50 total 235.95 last 125.28) - open/close 29093 (time 1746129348.50 total 245.95 last 116.49) - open/close 30000 (time 1746129357.50 total 254.95 last 100.77) - open/close 31078 (time 1746129367.51 total 264.96 last 107.68) - open/close 32297 (time 1746129377.51 total 274.96 last 121.83) - open/close 33745 (time 1746129387.52 total 284.97 last 144.77) - open/close 35090 (time 1746129397.52 total 294.97 last 134.47) - open/close 36694 (time 1746129407.52 total 304.97 last 160.37) - open/close 38573 (time 1746129417.52 total 314.97 last 187.88) - open/close 39986 (time 1746129427.53 total 324.97 last 141.24) - open/close 41569 (time 1746129437.53 total 334.98 last 158.27) - open/close 43054 (time 1746129447.54 total 344.98 last 148.38) - open/close 44653 (time 1746129457.54 total 354.99 last 159.81) - open/close 45985 (time 1746129467.55 total 364.99 last 133.15) - open/close 47563 (time 1746129477.55 total 375.00 last 157.77) - open/close 49079 (time 1746129487.56 total 385.00 last 151.48) - open/close 50000 (time 1746129495.25 total 392.70 last 119.73) - open/close 51473 (time 1746129505.25 total 402.70 last 147.25) - open/close 53037 (time 1746129515.25 total 412.70 last 156.37) - open/close 54707 (time 1746129525.26 total 422.70 last 166.95) - open/close 56040 (time 1746129535.26 total 432.71 last 133.26) - open/close 57466 (time 1746129545.26 total 442.71 last 142.59) - open/close 58715 (time 1746129555.27 total 452.71 last 124.83) - open/close 59877 (time 1746129565.27 total 462.72 last 116.19) - open/close 60982 (time 1746129575.27 total 472.72 last 110.44) - open/close 62162 (time 1746129585.28 total 482.73 last 117.87) - open/close 63507 (time 1746129595.29 total 492.73 last 134.45) total: 64768 open/close in 501.21 seconds: 129.22 ops/second - /unlink 2476 (time 1746129615.14 total 10.01 last 247.45) - /unlink 4590 (time 1746129625.14 total 20.01 last 211.37) - /unlink 6580 (time 1746129635.14 total 30.01 last 198.96) - /unlink 8295 (time 1746129645.15 total 40.02 last 171.33) - /unlink 10000 (time 1746129654.70 total 49.57 last 178.51) - /unlink 11922 (time 1746129664.70 total 59.57 last 192.11) - /unlink 14880 (time 1746129674.71 total 69.58 last 295.72) - /unlink 17407 (time 1746129684.71 total 79.58 last 252.66) - /unlink 19788 (time 1746129694.71 total 89.58 last 238.07) - /unlink 21600 (time 1746129704.72 total 99.59 last 181.09) - /unlink 23438 (time 1746129714.72 total 109.59 last 183.78) - /unlink 25358 (time 1746129724.72 total 119.59 last 191.99) - /unlink 27432 (time 1746129734.72 total 129.59 last 207.38) - /unlink 29849 (time 1746129744.72 total 139.59 last 241.70) - /unlink 32150 (time 1746129754.72 total 149.59 last 230.02) - /unlink 34455 (time 1746129764.72 total 159.59 last 230.44) - /unlink 36301 (time 1746129774.73 total 169.60 last 184.59) - /unlink 38312 (time 1746129784.73 total 179.60 last 201.02) - /unlink 40000 (time 1746129793.39 total 188.26 last 194.97) - /unlink 41777 (time 1746129803.39 total 198.26 last 177.62) - /unlink 43738 (time 1746129813.40 total 208.27 last 196.04) - /unlink 45999 (time 1746129823.40 total 218.27 last 226.01) - /unlink 48419 (time 1746129833.40 total 228.27 last 241.93) - /unlink 50000 (time 1746129840.73 total 235.60 last 215.73) - /unlink 52149 (time 1746129850.73 total 245.60 last 214.87) - /unlink 54260 (time 1746129860.73 total 255.60 last 211.06) - /unlink 56230 (time 1746129870.74 total 265.61 last 196.92) - /unlink 58267 (time 1746129880.74 total 275.61 last 203.66) - /unlink 60000 (time 1746129889.72 total 284.59 last 192.89) - /unlink 61404 (time 1746129899.73 total 294.60 last 140.31) - /unlink 62992 (time 1746129909.74 total 304.61 last 158.71) - /unlink 64634 (time 1746129919.74 total 314.61 last 164.19) total: 64768 /unlink in 315.62 seconds: 205.21 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1099 (time 1746129947.64 total 10.01 last 109.82) - open/close 2335 (time 1746129957.64 total 20.01 last 123.58) - open/close 3641 (time 1746129967.65 total 30.01 last 130.53) - open/close 4991 (time 1746129977.65 total 40.02 last 134.97) - open/close 6319 (time 1746129987.65 total 50.02 last 132.75) - open/close 7590 (time 1746129997.66 total 60.02 last 127.04) - open/close 8717 (time 1746130007.66 total 70.03 last 112.66) - open/close 9954 (time 1746130017.66 total 80.03 last 123.70) - open/close 11202 (time 1746130027.66 total 90.03 last 124.78) - open/close 12291 (time 1746130037.67 total 100.04 last 108.81) - open/close 13451 (time 1746130047.68 total 110.04 last 115.92) - open/close 14496 (time 1746130057.68 total 120.05 last 104.45) - open/close 15555 (time 1746130067.68 total 130.05 last 105.88) - open/close 16536 (time 1746130077.69 total 140.06 last 98.04) - open/close 17699 (time 1746130087.69 total 150.06 last 116.26) - open/close 18752 (time 1746130097.69 total 160.06 last 105.29) - open/close 19982 (time 1746130107.70 total 170.06 last 122.97) - open/close 21119 (time 1746130117.70 total 180.06 last 113.70) - open/close 22195 (time 1746130127.70 total 190.07 last 107.55) - open/close 23347 (time 1746130137.70 total 200.07 last 115.17) - open/close 24469 (time 1746130147.71 total 210.08 last 112.12) - open/close 25689 (time 1746130157.71 total 220.08 last 121.97) - open/close 26753 (time 1746130167.72 total 230.08 last 106.35) - open/close 27870 (time 1746130177.72 total 240.09 last 111.66) - open/close 29006 (time 1746130187.72 total 250.09 last 113.58) - open/close 30000 (time 1746130197.04 total 259.40 last 106.71) - open/close 31044 (time 1746130207.05 total 269.41 last 104.32) - open/close 32217 (time 1746130217.05 total 279.41 last 117.28) - open/close 33456 (time 1746130227.05 total 289.42 last 123.87) - open/close 34547 (time 1746130237.06 total 299.43 last 109.00) - open/close 35515 (time 1746130247.06 total 309.43 last 96.75) - open/close 36758 (time 1746130257.07 total 319.43 last 124.28) - open/close 37923 (time 1746130267.07 total 329.44 last 116.41) - open/close 38843 (time 1746130277.08 total 339.45 last 91.95) - open/close 40000 (time 1746130286.39 total 348.76 last 124.27) - open/close 41314 (time 1746130296.39 total 358.76 last 131.39) - open/close 42327 (time 1746130306.39 total 368.76 last 101.27) - open/close 43457 (time 1746130316.39 total 378.76 last 112.99) - open/close 44652 (time 1746130326.40 total 388.77 last 119.44) - open/close 45580 (time 1746130336.41 total 398.78 last 92.71) - open/close 46472 (time 1746130346.41 total 408.78 last 89.19) - open/close 47553 (time 1746130356.41 total 418.78 last 108.05) - open/close 48786 (time 1746130366.42 total 428.79 last 123.21) - open/close 49884 (time 1746130376.42 total 438.79 last 109.80) - open/close 50961 (time 1746130386.43 total 448.80 last 107.61) - open/close 52055 (time 1746130396.44 total 458.81 last 109.31) - open/close 53242 (time 1746130406.45 total 468.82 last 118.55) - open/close 54228 (time 1746130416.45 total 478.82 last 98.56) - open/close 55118 (time 1746130426.46 total 488.83 last 88.96) - open/close 56090 (time 1746130436.47 total 498.83 last 97.12) - open/close 57106 (time 1746130446.47 total 508.84 last 101.59) - open/close 58140 (time 1746130456.48 total 518.84 last 103.30) - open/close 59137 (time 1746130466.48 total 528.85 last 99.66) - open/close 60000 (time 1746130475.25 total 537.61 last 98.46) - open/close 61264 (time 1746130485.25 total 547.62 last 126.36) - open/close 62465 (time 1746130495.26 total 557.63 last 119.96) - open/close 63559 (time 1746130505.28 total 567.65 last 109.22) - open/close 64709 (time 1746130515.29 total 577.66 last 114.88) total: 64768 open/close in 578.04 seconds: 112.05 ops/second - /unlink 1968 (time 1746130527.26 total 10.00 last 196.79) - /unlink 4016 (time 1746130537.27 total 20.00 last 204.73) - /unlink 6159 (time 1746130547.27 total 30.00 last 214.30) - /unlink 8122 (time 1746130557.27 total 40.01 last 196.24) - /unlink 10000 (time 1746130565.91 total 48.65 last 217.35) - /unlink 12239 (time 1746130575.91 total 58.65 last 223.88) - /unlink 14918 (time 1746130585.91 total 68.65 last 267.87) - /unlink 16999 (time 1746130595.91 total 78.65 last 208.07) - /unlink 19545 (time 1746130605.92 total 88.66 last 254.48) - /unlink 22213 (time 1746130615.92 total 98.66 last 266.75) - /unlink 24528 (time 1746130625.92 total 108.66 last 231.46) - /unlink 26580 (time 1746130635.92 total 118.66 last 205.17) - /unlink 28721 (time 1746130645.93 total 128.66 last 214.08) - /unlink 30000 (time 1746130651.50 total 134.24 last 229.45) - /unlink 31954 (time 1746130661.50 total 144.24 last 195.33) - /unlink 34092 (time 1746130671.51 total 154.24 last 213.74) - /unlink 36455 (time 1746130681.51 total 164.25 last 236.22) - /unlink 38186 (time 1746130691.52 total 174.25 last 172.98) - /unlink 40000 (time 1746130701.45 total 184.19 last 182.54) - /unlink 42202 (time 1746130711.46 total 194.20 last 220.01) - /unlink 44123 (time 1746130721.47 total 204.20 last 192.01) - /unlink 45807 (time 1746130731.47 total 214.21 last 168.35) - /unlink 47494 (time 1746130741.48 total 224.21 last 168.61) - /unlink 49187 (time 1746130751.48 total 234.21 last 169.25) - /unlink 50000 (time 1746130756.28 total 239.02 last 169.17) - /unlink 51671 (time 1746130766.29 total 249.02 last 167.09) - /unlink 53431 (time 1746130776.29 total 259.02 last 175.95) - /unlink 55340 (time 1746130786.29 total 269.03 last 190.79) - /unlink 57190 (time 1746130796.29 total 279.03 last 184.99) - /unlink 59074 (time 1746130806.30 total 289.03 last 188.38) - /unlink 60000 (time 1746130810.73 total 293.46 last 208.91) - /unlink 62068 (time 1746130820.73 total 303.47 last 206.73) - /unlink 64347 (time 1746130830.74 total 313.47 last 227.76) total: 64768 /unlink in 315.91 seconds: 205.02 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1164 (time 1746130864.06 total 10.00 last 116.38) - open/close 2225 (time 1746130874.06 total 20.01 last 106.05) - open/close 3296 (time 1746130884.06 total 30.01 last 107.09) - open/close 4500 (time 1746130894.06 total 40.01 last 120.40) - open/close 5465 (time 1746130904.09 total 50.03 last 96.29) - open/close 6408 (time 1746130914.09 total 60.03 last 94.27) - open/close 7438 (time 1746130924.10 total 70.04 last 102.90) - open/close 8490 (time 1746130934.10 total 80.05 last 105.13) - open/close 9420 (time 1746130944.10 total 90.05 last 93.00) - open/close 10000 (time 1746130949.95 total 95.89 last 99.25) - open/close 10974 (time 1746130959.95 total 105.89 last 97.39) - open/close 12031 (time 1746130969.95 total 115.90 last 105.66) - open/close 12989 (time 1746130979.96 total 125.90 last 95.76) - open/close 13945 (time 1746130989.97 total 135.91 last 95.49) - open/close 15003 (time 1746130999.97 total 145.92 last 105.78) - open/close 15974 (time 1746131009.98 total 155.93 last 97.00) - open/close 16905 (time 1746131019.98 total 165.93 last 93.08) - open/close 17952 (time 1746131029.99 total 175.94 last 104.59) - open/close 19049 (time 1746131040.00 total 185.94 last 109.64) - open/close 20000 (time 1746131048.64 total 194.58 last 110.10) - open/close 20973 (time 1746131058.64 total 204.58 last 97.27) - open/close 22178 (time 1746131068.65 total 214.59 last 120.44) - open/close 23189 (time 1746131078.65 total 224.60 last 101.02) - open/close 24124 (time 1746131088.66 total 234.61 last 93.42) - open/close 25216 (time 1746131098.67 total 244.61 last 109.15) - open/close 26499 (time 1746131108.67 total 254.62 last 128.18) - open/close 27553 (time 1746131118.69 total 264.63 last 105.29) - open/close 28608 (time 1746131128.69 total 274.63 last 105.47) - open/close 29852 (time 1746131138.69 total 284.63 last 124.38) - open/close 30867 (time 1746131148.69 total 294.64 last 101.49) - open/close 31868 (time 1746131158.69 total 304.64 last 100.07) - open/close 32833 (time 1746131168.69 total 314.64 last 96.49) - open/close 34001 (time 1746131178.70 total 324.64 last 116.77) - open/close 34944 (time 1746131188.70 total 334.65 last 94.23) - open/close 35842 (time 1746131198.71 total 344.65 last 89.76) - open/close 36839 (time 1746131208.71 total 354.66 last 99.66) - open/close 37875 (time 1746131218.72 total 364.67 last 103.49) - open/close 38916 (time 1746131228.73 total 374.67 last 104.05) - open/close 39817 (time 1746131238.73 total 384.67 last 90.10) - open/close 40961 (time 1746131248.73 total 394.68 last 114.34) - open/close 42064 (time 1746131258.75 total 404.70 last 110.10) - open/close 42992 (time 1746131268.76 total 414.70 last 92.73) - open/close 44030 (time 1746131278.77 total 424.71 last 103.68) - open/close 45120 (time 1746131288.77 total 434.72 last 108.99) - open/close 46169 (time 1746131298.78 total 444.72 last 104.84) - open/close 47170 (time 1746131308.79 total 454.73 last 99.96) - open/close 48145 (time 1746131318.80 total 464.74 last 97.42) - open/close 49200 (time 1746131328.80 total 474.75 last 105.47) - open/close 50000 (time 1746131336.54 total 482.48 last 103.40) - open/close 51019 (time 1746131346.54 total 492.48 last 101.88) - open/close 52027 (time 1746131356.55 total 502.49 last 100.75) - open/close 53019 (time 1746131366.55 total 512.49 last 99.16) - open/close 54006 (time 1746131376.55 total 522.50 last 98.65) - open/close 55054 (time 1746131386.56 total 532.50 last 104.74) - open/close 55973 (time 1746131396.56 total 542.50 last 91.90) - open/close 56953 (time 1746131406.56 total 552.51 last 97.96) - open/close 57982 (time 1746131416.57 total 562.51 last 102.86) - open/close 58951 (time 1746131426.57 total 572.51 last 96.86) - open/close 59923 (time 1746131436.58 total 582.52 last 97.15) - open/close 60968 (time 1746131446.59 total 592.53 last 104.38) - open/close 61967 (time 1746131456.59 total 602.53 last 99.89) - open/close 63004 (time 1746131466.59 total 612.53 last 103.69) - open/close 64113 (time 1746131476.59 total 622.54 last 110.87) total: 64768 open/close in 628.99 seconds: 102.97 ops/second - /unlink 974 (time 1746131495.54 total 10.01 last 97.32) - /unlink 2124 (time 1746131505.54 total 20.01 last 115.00) - /unlink 3107 (time 1746131515.54 total 30.01 last 98.29) - /unlink 4189 (time 1746131525.55 total 40.02 last 108.12) - /unlink 5195 (time 1746131535.55 total 50.02 last 100.55) - /unlink 6264 (time 1746131545.55 total 60.02 last 106.88) - /unlink 7253 (time 1746131555.56 total 70.03 last 98.85) - /unlink 8367 (time 1746131565.57 total 80.04 last 111.30) - /unlink 9342 (time 1746131575.57 total 90.04 last 97.49) - /unlink 10000 (time 1746131581.54 total 96.01 last 110.24) - /unlink 11041 (time 1746131591.54 total 106.01 last 104.06) - /unlink 11972 (time 1746131601.55 total 116.02 last 93.06) - /unlink 12925 (time 1746131611.55 total 126.02 last 95.28) - /unlink 13966 (time 1746131621.56 total 136.02 last 104.03) - /unlink 14955 (time 1746131631.56 total 146.03 last 98.88) - /unlink 15781 (time 1746131641.57 total 156.04 last 82.53) - /unlink 16691 (time 1746131651.58 total 166.05 last 90.86) - /unlink 17705 (time 1746131661.59 total 176.06 last 101.34) - /unlink 18769 (time 1746131671.59 total 186.06 last 106.34) - /unlink 19817 (time 1746131681.59 total 196.06 last 104.79) - /unlink 20826 (time 1746131691.60 total 206.07 last 100.87) - /unlink 21887 (time 1746131701.63 total 216.09 last 105.81) - /unlink 22815 (time 1746131711.63 total 226.09 last 92.80) - /unlink 23835 (time 1746131721.63 total 236.10 last 101.99) - /unlink 24803 (time 1746131731.64 total 246.11 last 96.70) - /unlink 25893 (time 1746131741.64 total 256.11 last 108.93) - /unlink 26953 (time 1746131751.65 total 266.12 last 105.95) - /unlink 27994 (time 1746131761.67 total 276.14 last 103.91) - /unlink 28879 (time 1746131771.67 total 286.14 last 88.47) - /unlink 29918 (time 1746131781.68 total 296.15 last 103.80) - /unlink 30957 (time 1746131791.68 total 306.15 last 103.84) - /unlink 32039 (time 1746131801.71 total 316.18 last 107.95) - /unlink 33010 (time 1746131811.71 total 326.18 last 97.06) - /unlink 33906 (time 1746131821.73 total 336.20 last 89.46) - /unlink 34751 (time 1746131831.74 total 346.21 last 84.41) - /unlink 35679 (time 1746131841.75 total 356.22 last 92.73) - /unlink 36648 (time 1746131851.75 total 366.22 last 96.89) - /unlink 37567 (time 1746131861.75 total 376.22 last 91.88) - /unlink 38573 (time 1746131871.75 total 386.22 last 100.59) - /unlink 39769 (time 1746131881.76 total 396.23 last 119.47) - /unlink 40809 (time 1746131891.77 total 406.23 last 103.96) - /unlink 41747 (time 1746131901.77 total 416.24 last 93.74) - /unlink 42821 (time 1746131911.78 total 426.24 last 107.36) - /unlink 43992 (time 1746131921.78 total 436.25 last 117.03) - /unlink 45067 (time 1746131931.79 total 446.26 last 107.46) - /unlink 46106 (time 1746131941.79 total 456.26 last 103.87) - /unlink 47120 (time 1746131951.79 total 466.26 last 101.39) - /unlink 48160 (time 1746131961.79 total 476.26 last 103.99) - /unlink 49265 (time 1746131971.80 total 486.27 last 110.39) - /unlink 50000 (time 1746131980.42 total 494.88 last 85.31) - /unlink 50967 (time 1746131990.43 total 504.89 last 96.61) - /unlink 51868 (time 1746132000.43 total 514.90 last 90.08) - /unlink 53073 (time 1746132010.43 total 524.90 last 120.47) - /unlink 54097 (time 1746132020.43 total 534.90 last 102.35) - /unlink 55108 (time 1746132030.46 total 544.92 last 100.89) - /unlink 56141 (time 1746132040.46 total 554.93 last 103.23) - /unlink 57161 (time 1746132050.46 total 564.93 last 101.99) - /unlink 58274 (time 1746132060.47 total 574.94 last 111.22) - /unlink 59298 (time 1746132070.48 total 584.95 last 102.33) - /unlink 60000 (time 1746132077.15 total 591.62 last 105.17) - /unlink 61150 (time 1746132087.16 total 601.63 last 114.89) - /unlink 62299 (time 1746132097.17 total 611.64 last 114.82) - /unlink 63389 (time 1746132107.18 total 621.65 last 108.85) - /unlink 64475 (time 1746132117.19 total 631.66 last 108.49) total: 64768 /unlink in 634.57 seconds: 102.07 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1164 (time 1746132150.26 total 10.01 last 116.30) - open/close 2154 (time 1746132160.26 total 20.01 last 98.95) - open/close 3084 (time 1746132170.27 total 30.02 last 92.96) - open/close 4195 (time 1746132180.27 total 40.02 last 111.09) - open/close 5333 (time 1746132190.28 total 50.03 last 113.71) - open/close 6488 (time 1746132200.28 total 60.03 last 115.43) - open/close 7548 (time 1746132210.29 total 70.04 last 105.95) - open/close 8632 (time 1746132220.30 total 80.05 last 108.27) - open/close 9857 (time 1746132230.32 total 90.07 last 122.32) - open/close 11002 (time 1746132240.32 total 100.07 last 114.47) - open/close 12079 (time 1746132250.32 total 110.08 last 107.63) - open/close 13180 (time 1746132260.33 total 120.08 last 110.05) - open/close 14242 (time 1746132270.33 total 130.08 last 106.15) - open/close 15534 (time 1746132280.34 total 140.09 last 129.12) - open/close 16779 (time 1746132290.34 total 150.09 last 124.49) - open/close 18132 (time 1746132300.34 total 160.09 last 135.29) - open/close 19299 (time 1746132310.35 total 170.10 last 116.64) - open/close 20000 (time 1746132315.52 total 175.27 last 135.46) - open/close 21096 (time 1746132325.52 total 185.27 last 109.57) - open/close 22281 (time 1746132335.53 total 195.28 last 118.46) - open/close 23452 (time 1746132345.53 total 205.28 last 117.10) - open/close 24656 (time 1746132355.54 total 215.29 last 120.24) - open/close 25790 (time 1746132365.55 total 225.30 last 113.31) - open/close 26790 (time 1746132375.55 total 235.30 last 99.98) - open/close 27819 (time 1746132385.56 total 245.31 last 102.84) - open/close 28975 (time 1746132395.56 total 255.31 last 115.55) - open/close 29939 (time 1746132405.57 total 265.32 last 96.33) - open/close 30837 (time 1746132415.57 total 275.32 last 89.79) - open/close 32011 (time 1746132425.57 total 285.32 last 117.38) - open/close 33163 (time 1746132435.58 total 295.33 last 115.15) - open/close 34112 (time 1746132445.58 total 305.33 last 94.83) - open/close 35105 (time 1746132455.59 total 315.34 last 99.20) - open/close 36183 (time 1746132465.60 total 325.35 last 107.76) - open/close 37126 (time 1746132475.60 total 335.35 last 94.30) - open/close 38023 (time 1746132485.61 total 345.36 last 89.63) - open/close 38989 (time 1746132495.61 total 355.36 last 96.56) - open/close 40000 (time 1746132504.17 total 363.92 last 118.08) - open/close 41183 (time 1746132514.17 total 373.92 last 118.28) - open/close 42202 (time 1746132524.18 total 383.93 last 101.80) - open/close 43244 (time 1746132534.19 total 393.94 last 104.16) - open/close 44264 (time 1746132544.19 total 403.94 last 101.96) - open/close 45338 (time 1746132554.21 total 413.96 last 107.24) - open/close 46218 (time 1746132564.21 total 423.96 last 87.94) - open/close 47150 (time 1746132574.22 total 433.97 last 93.15) - open/close 48279 (time 1746132584.22 total 443.97 last 112.85) - open/close 49498 (time 1746132594.23 total 453.98 last 121.84) - open/close 50000 (time 1746132599.14 total 458.89 last 102.30) - open/close 51175 (time 1746132609.14 total 468.89 last 117.40) - open/close 52249 (time 1746132619.15 total 478.90 last 107.34) - open/close 53410 (time 1746132629.16 total 488.91 last 115.99) - open/close 54590 (time 1746132639.16 total 498.91 last 117.93) - open/close 55888 (time 1746132649.17 total 508.92 last 129.73) - open/close 57246 (time 1746132659.17 total 518.92 last 135.79) - open/close 58224 (time 1746132669.18 total 528.93 last 97.70) - open/close 59214 (time 1746132679.18 total 538.94 last 98.96) - open/close 60000 (time 1746132686.89 total 546.64 last 101.98) - open/close 61176 (time 1746132696.90 total 556.65 last 117.55) - open/close 62073 (time 1746132706.90 total 566.65 last 89.68) - open/close 63085 (time 1746132716.90 total 576.65 last 101.20) - open/close 64124 (time 1746132726.90 total 586.65 last 103.86) total: 64768 open/close in 593.20 seconds: 109.18 ops/second - /unlink 1514 (time 1746132745.54 total 10.01 last 151.25) - /unlink 3241 (time 1746132755.54 total 20.01 last 172.70) - /unlink 5110 (time 1746132765.54 total 30.01 last 186.86) - /unlink 6929 (time 1746132775.54 total 40.01 last 181.85) - /unlink 8416 (time 1746132785.54 total 50.01 last 148.69) - /unlink 10000 (time 1746132793.68 total 58.15 last 194.64) - /unlink 12065 (time 1746132803.69 total 68.16 last 206.40) - /unlink 13673 (time 1746132813.69 total 78.16 last 160.70) - /unlink 15424 (time 1746132823.70 total 88.17 last 175.05) - /unlink 17341 (time 1746132833.70 total 98.17 last 191.67) - /unlink 19296 (time 1746132843.70 total 108.17 last 195.45) - /unlink 20000 (time 1746132847.71 total 112.19 last 175.35) - /unlink 21872 (time 1746132857.72 total 122.19 last 187.15) - /unlink 23866 (time 1746132867.72 total 132.19 last 199.36) - /unlink 25739 (time 1746132877.72 total 142.20 last 187.21) - /unlink 27418 (time 1746132887.73 total 152.20 last 167.84) - /unlink 29158 (time 1746132897.74 total 162.21 last 173.83) - /unlink 30000 (time 1746132901.53 total 166.01 last 221.81) - /unlink 32197 (time 1746132911.53 total 176.01 last 219.68) - /unlink 34290 (time 1746132921.54 total 186.01 last 209.29) - /unlink 36708 (time 1746132931.54 total 196.01 last 241.80) - /unlink 39104 (time 1746132941.54 total 206.01 last 239.43) - /unlink 40000 (time 1746132945.96 total 210.44 last 202.66) - /unlink 42132 (time 1746132955.96 total 220.44 last 213.19) - /unlink 44171 (time 1746132965.97 total 230.44 last 203.88) - /unlink 46828 (time 1746132975.97 total 240.44 last 265.67) - /unlink 48689 (time 1746132985.97 total 250.44 last 186.08) - /unlink 50000 (time 1746132992.07 total 256.54 last 214.90) - /unlink 52312 (time 1746133002.07 total 266.54 last 231.16) - /unlink 54907 (time 1746133012.07 total 276.54 last 259.49) - /unlink 57375 (time 1746133022.07 total 286.54 last 246.76) - /unlink 60000 (time 1746133031.76 total 296.23 last 271.02) - /unlink 61945 (time 1746133041.76 total 306.23 last 194.44) - /unlink 64058 (time 1746133051.77 total 316.24 last 211.18) total: 64768 /unlink in 319.48 seconds: 202.73 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1300 (time 1746133083.17 total 10.00 last 129.99) - open/close 2655 (time 1746133093.17 total 20.00 last 135.47) - open/close 4042 (time 1746133103.17 total 30.01 last 138.63) - open/close 5225 (time 1746133113.18 total 40.01 last 118.28) - open/close 6267 (time 1746133123.18 total 50.01 last 104.18) - open/close 7595 (time 1746133133.18 total 60.02 last 132.75) - open/close 8703 (time 1746133143.19 total 70.02 last 110.71) - open/close 9869 (time 1746133153.19 total 80.03 last 116.56) - open/close 10986 (time 1746133163.20 total 90.03 last 111.64) - open/close 12083 (time 1746133173.20 total 100.04 last 109.67) - open/close 13165 (time 1746133183.20 total 110.04 last 108.20) - open/close 14256 (time 1746133193.22 total 120.05 last 108.95) - open/close 15570 (time 1746133203.22 total 130.05 last 131.37) - open/close 16909 (time 1746133213.22 total 140.06 last 133.82) - open/close 18084 (time 1746133223.23 total 150.06 last 117.47) - open/close 19363 (time 1746133233.23 total 160.07 last 127.79) - open/close 20000 (time 1746133239.37 total 166.21 last 103.74) - open/close 21220 (time 1746133249.38 total 176.21 last 121.93) - open/close 22512 (time 1746133259.39 total 186.22 last 129.12) - open/close 23819 (time 1746133269.39 total 196.22 last 130.68) - open/close 24944 (time 1746133279.40 total 206.24 last 112.32) - open/close 26106 (time 1746133289.41 total 216.24 last 116.17) - open/close 27162 (time 1746133299.41 total 226.24 last 105.58) - open/close 28308 (time 1746133309.42 total 236.25 last 114.51) - open/close 29503 (time 1746133319.42 total 246.26 last 119.43) - open/close 30000 (time 1746133323.74 total 250.57 last 115.16) - open/close 31094 (time 1746133333.74 total 260.58 last 109.34) - open/close 32346 (time 1746133343.75 total 270.58 last 125.13) - open/close 33606 (time 1746133353.77 total 280.61 last 125.74) - open/close 34784 (time 1746133363.77 total 290.61 last 117.76) - open/close 35997 (time 1746133373.77 total 300.61 last 121.30) - open/close 37498 (time 1746133383.78 total 310.61 last 150.07) - open/close 38994 (time 1746133393.78 total 320.61 last 149.58) - open/close 40000 (time 1746133402.20 total 329.03 last 119.47) - open/close 41388 (time 1746133412.20 total 339.04 last 138.72) - open/close 42721 (time 1746133422.21 total 349.04 last 133.27) - open/close 44082 (time 1746133432.21 total 359.04 last 136.05) - open/close 45438 (time 1746133442.22 total 369.05 last 135.52) - open/close 46690 (time 1746133452.23 total 379.07 last 125.01) - open/close 47881 (time 1746133462.23 total 389.07 last 119.08) - open/close 49020 (time 1746133472.24 total 399.07 last 113.87) - open/close 50000 (time 1746133480.78 total 407.62 last 114.63) - open/close 51038 (time 1746133490.79 total 417.62 last 103.79) - open/close 52123 (time 1746133500.79 total 427.62 last 108.48) - open/close 53230 (time 1746133510.79 total 437.63 last 110.64) - open/close 54612 (time 1746133520.80 total 447.63 last 138.12) - open/close 55790 (time 1746133530.81 total 457.64 last 117.68) - open/close 56849 (time 1746133540.82 total 467.66 last 105.74) - open/close 58081 (time 1746133550.83 total 477.67 last 123.09) - open/close 59476 (time 1746133560.84 total 487.67 last 139.44) - open/close 60000 (time 1746133565.02 total 491.85 last 125.27) - open/close 61248 (time 1746133575.03 total 501.86 last 124.73) - open/close 62524 (time 1746133585.03 total 511.86 last 127.54) - open/close 63914 (time 1746133595.03 total 521.87 last 138.93) total: 64768 open/close in 527.73 seconds: 122.73 ops/second - /unlink 2043 (time 1746133612.47 total 10.00 last 204.25) - /unlink 4388 (time 1746133622.47 total 20.01 last 234.41) - /unlink 6567 (time 1746133632.47 total 30.01 last 217.84) - /unlink 8720 (time 1746133642.48 total 40.01 last 215.26) - /unlink 10000 (time 1746133648.88 total 46.41 last 199.95) - /unlink 11847 (time 1746133658.88 total 56.42 last 184.60) - /unlink 14114 (time 1746133668.88 total 66.42 last 226.69) - /unlink 16380 (time 1746133678.89 total 76.42 last 226.57) - /unlink 18635 (time 1746133688.89 total 86.42 last 225.45) - /unlink 20000 (time 1746133695.58 total 93.12 last 203.87) - /unlink 22085 (time 1746133705.59 total 103.12 last 208.39) - /unlink 23893 (time 1746133715.59 total 113.12 last 180.78) - /unlink 26005 (time 1746133725.59 total 123.13 last 211.15) - /unlink 28463 (time 1746133735.60 total 133.13 last 245.67) - /unlink 30000 (time 1746133743.56 total 141.10 last 192.94) - /unlink 32065 (time 1746133753.56 total 151.10 last 206.49) - /unlink 34070 (time 1746133763.56 total 161.10 last 200.48) - /unlink 36486 (time 1746133773.57 total 171.11 last 241.44) - /unlink 38738 (time 1746133783.57 total 181.11 last 225.18) - /unlink 40000 (time 1746133789.63 total 187.17 last 208.26) - /unlink 42363 (time 1746133799.63 total 197.17 last 236.30) - /unlink 44247 (time 1746133809.64 total 207.17 last 188.33) - /unlink 45955 (time 1746133819.64 total 217.17 last 170.77) - /unlink 47656 (time 1746133829.64 total 227.17 last 170.05) - /unlink 49407 (time 1746133839.64 total 237.17 last 175.09) - /unlink 50000 (time 1746133842.58 total 240.11 last 201.71) - /unlink 51844 (time 1746133852.59 total 250.12 last 184.31) - /unlink 53639 (time 1746133862.59 total 260.13 last 179.33) - /unlink 55736 (time 1746133872.59 total 270.13 last 209.69) - /unlink 58248 (time 1746133882.60 total 280.13 last 251.15) - /unlink 60000 (time 1746133890.73 total 288.27 last 215.30) - /unlink 61938 (time 1746133900.75 total 298.28 last 193.55) - /unlink 64237 (time 1746133910.76 total 308.29 last 229.73) total: 64768 /unlink in 310.79 seconds: 208.40 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg305-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg305-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg305-server stop mds service on oleg305-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg305-server unloading modules via unload_modules_local on: 'oleg305-server' oleg305-server: oleg305-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg305-client: oleg305-client: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-server: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-client: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-server: ssh exited with exit code 2 PASS 106 (4917s) Stopping clients: oleg305-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg305-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg305-client: oleg305-server: ssh exited with exit code 2 oleg305-server: oleg305-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg305-server' oleg305-server: oleg305-server.virtnet: executing load_modules_local oleg305-server: Loading modules from /home/green/git/lustre-release/lustre oleg305-server: detected 4 online CPUs by sysfs oleg305-server: MODOPTS_LIBCFS= oleg305-server: Force libcfs to create 2 CPU partitions oleg305-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg305-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 5057 sec ========== 17:13:20 (1746134000) === conf-sanity: start cleanup 17:13:21 (1746134001) === === conf-sanity: finish cleanup 17:13:24 (1746134004) ===