-----============= acceptance-small: conf-sanity ============----- Thu May 1 23:30:27 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg407-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg407-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg407-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg407-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg407-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg407-server oleg407-server: oleg407-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing load_modules_local oleg407-server: Loading modules from /home/green/git/lustre-release/lustre oleg407-server: detected 4 online CPUs by sysfs oleg407-server: MODOPTS_LIBCFS= oleg407-server: Force libcfs to create 2 CPU partitions oleg407-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg407-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg407-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg407-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg407-server: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg407-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg407-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server stop mds service on oleg407-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg407-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 23:31:52 (1746156712) Stopping clients: oleg407-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg407-client.virtnet /mnt/lustre2 (opts:-f) oleg407-server: oleg407-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing load_modules_local oleg407-server: Loading modules from /home/green/git/lustre-release/lustre oleg407-server: detected 4 online CPUs by sysfs oleg407-server: MODOPTS_LIBCFS= oleg407-server: Force libcfs to create 2 CPU partitions oleg407-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory oleg407-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg407-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg407-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1152 (time 1746156778.33 total 10.00 last 115.14) - open/close 2176 (time 1746156788.34 total 20.01 last 102.35) - open/close 3179 (time 1746156798.34 total 30.01 last 100.27) - open/close 4357 (time 1746156808.34 total 40.01 last 117.80) - open/close 5554 (time 1746156818.35 total 50.02 last 119.62) - open/close 6837 (time 1746156828.35 total 60.02 last 128.27) - open/close 8315 (time 1746156838.35 total 70.03 last 147.73) - open/close 9572 (time 1746156848.36 total 80.03 last 125.67) - open/close 10000 (time 1746156851.69 total 83.36 last 128.46) - open/close 11194 (time 1746156861.69 total 93.37 last 119.34) - open/close 12349 (time 1746156871.70 total 103.37 last 115.46) - open/close 13584 (time 1746156881.70 total 113.37 last 123.44) - open/close 14760 (time 1746156891.71 total 123.38 last 117.56) - open/close 16075 (time 1746156901.71 total 133.38 last 131.47) - open/close 17482 (time 1746156911.71 total 143.38 last 140.68) - open/close 19069 (time 1746156921.71 total 153.38 last 158.65) - open/close 20000 (time 1746156927.55 total 159.22 last 159.53) - open/close 21432 (time 1746156937.55 total 169.22 last 143.13) - open/close 22869 (time 1746156947.56 total 179.23 last 143.62) - open/close 24216 (time 1746156957.56 total 189.23 last 134.65) - open/close 25445 (time 1746156967.56 total 199.23 last 122.89) - open/close 26553 (time 1746156977.57 total 209.24 last 110.76) - open/close 27857 (time 1746156987.57 total 219.24 last 130.34) - open/close 29437 (time 1746156997.58 total 229.25 last 157.94) - open/close 30000 (time 1746157002.23 total 233.90 last 120.91) - open/close 31318 (time 1746157012.23 total 243.91 last 131.77) - open/close 32591 (time 1746157022.25 total 253.92 last 127.06) - open/close 33922 (time 1746157032.26 total 263.93 last 133.03) - open/close 35193 (time 1746157042.26 total 273.93 last 127.08) - open/close 36507 (time 1746157052.26 total 283.93 last 131.40) - open/close 37618 (time 1746157062.26 total 293.93 last 111.09) - open/close 38778 (time 1746157072.27 total 303.94 last 115.91) - open/close 40000 (time 1746157081.57 total 313.24 last 131.45) - open/close 41602 (time 1746157091.57 total 323.24 last 160.18) - open/close 43256 (time 1746157101.57 total 333.24 last 165.31) - open/close 44836 (time 1746157111.57 total 343.24 last 158.00) - open/close 46389 (time 1746157121.58 total 353.25 last 155.23) - open/close 47916 (time 1746157131.58 total 363.25 last 152.62) - open/close 49244 (time 1746157141.58 total 373.25 last 132.80) - open/close 50000 (time 1746157146.31 total 377.98 last 159.97) - open/close 51606 (time 1746157156.31 total 387.98 last 160.56) - open/close 53280 (time 1746157166.31 total 397.98 last 167.37) - open/close 54834 (time 1746157176.32 total 407.99 last 155.33) - open/close 56253 (time 1746157186.32 total 417.99 last 141.82) - open/close 57747 (time 1746157196.32 total 428.00 last 149.36) - open/close 59081 (time 1746157206.34 total 438.01 last 133.20) - open/close 60000 (time 1746157212.01 total 443.68 last 162.12) - open/close 61497 (time 1746157222.01 total 453.68 last 149.67) - open/close 63056 (time 1746157232.01 total 463.68 last 155.89) - open/close 64435 (time 1746157242.01 total 473.69 last 137.86) total: 64768 open/close in 476.08 seconds: 136.05 ops/second - /unlink 2633 (time 1746157255.79 total 10.00 last 263.28) - /unlink 5200 (time 1746157265.80 total 20.00 last 256.60) - /unlink 7707 (time 1746157275.80 total 30.01 last 250.65) - /unlink 10000 (time 1746157284.32 total 38.53 last 269.09) - /unlink 12701 (time 1746157294.32 total 48.53 last 270.09) - /unlink 14962 (time 1746157304.32 total 58.53 last 225.97) - /unlink 16596 (time 1746157314.33 total 68.54 last 163.31) - /unlink 18212 (time 1746157324.33 total 78.54 last 161.53) - /unlink 19821 (time 1746157334.34 total 88.55 last 160.83) - /unlink 21448 (time 1746157344.34 total 98.55 last 162.66) - /unlink 22948 (time 1746157354.35 total 108.55 last 149.93) - /unlink 24947 (time 1746157364.35 total 118.56 last 199.86) - /unlink 27375 (time 1746157374.35 total 128.56 last 242.78) - /unlink 29782 (time 1746157384.35 total 138.56 last 240.65) - /unlink 32537 (time 1746157394.35 total 148.56 last 275.49) - /unlink 34729 (time 1746157404.35 total 158.56 last 219.20) - /unlink 37014 (time 1746157414.35 total 168.56 last 228.45) - /unlink 39380 (time 1746157424.35 total 178.56 last 236.59) - /unlink 40000 (time 1746157426.69 total 180.90 last 265.26) - /unlink 42701 (time 1746157436.69 total 190.90 last 270.04) - /unlink 44892 (time 1746157446.70 total 200.91 last 219.03) - /unlink 47131 (time 1746157456.70 total 210.91 last 223.88) - /unlink 49733 (time 1746157466.70 total 220.91 last 260.07) - /unlink 52177 (time 1746157476.71 total 230.92 last 244.19) - /unlink 54350 (time 1746157486.72 total 240.93 last 217.15) - /unlink 56239 (time 1746157496.72 total 250.93 last 188.89) - /unlink 58263 (time 1746157506.72 total 260.93 last 202.34) - /unlink 60000 (time 1746157514.94 total 269.15 last 211.35) - /unlink 61930 (time 1746157524.95 total 279.16 last 192.87) - /unlink 63976 (time 1746157534.95 total 289.16 last 204.57) total: 64768 /unlink in 293.06 seconds: 221.01 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1293 (time 1746157567.83 total 10.00 last 129.24) - open/close 2511 (time 1746157577.84 total 20.01 last 121.70) - open/close 3664 (time 1746157587.84 total 30.02 last 115.25) - open/close 4974 (time 1746157597.84 total 40.02 last 130.96) - open/close 6444 (time 1746157607.85 total 50.02 last 146.94) - open/close 7680 (time 1746157617.85 total 60.03 last 123.53) - open/close 8807 (time 1746157627.86 total 70.03 last 112.66) - open/close 9972 (time 1746157637.86 total 80.04 last 116.47) - open/close 11160 (time 1746157647.87 total 90.04 last 118.74) - open/close 12345 (time 1746157657.87 total 100.04 last 118.50) - open/close 13550 (time 1746157667.87 total 110.05 last 120.41) - open/close 14801 (time 1746157677.88 total 120.05 last 125.03) - open/close 16023 (time 1746157687.88 total 130.06 last 122.15) - open/close 17166 (time 1746157697.89 total 140.06 last 114.25) - open/close 18392 (time 1746157707.89 total 150.06 last 122.59) - open/close 19643 (time 1746157717.89 total 160.06 last 125.10) - open/close 20000 (time 1746157720.92 total 163.10 last 117.54) - open/close 21200 (time 1746157730.93 total 173.10 last 119.95) - open/close 22335 (time 1746157740.94 total 183.11 last 113.41) - open/close 23553 (time 1746157750.94 total 193.11 last 121.79) - open/close 24734 (time 1746157760.94 total 203.12 last 118.06) - open/close 26032 (time 1746157770.94 total 213.12 last 129.78) - open/close 27417 (time 1746157780.95 total 223.12 last 138.45) - open/close 28698 (time 1746157790.95 total 233.13 last 128.06) - open/close 30000 (time 1746157800.80 total 242.97 last 132.24) - open/close 31219 (time 1746157810.80 total 252.97 last 121.88) - open/close 32952 (time 1746157820.80 total 262.97 last 173.29) - open/close 34565 (time 1746157830.80 total 272.98 last 161.25) - open/close 36229 (time 1746157840.80 total 282.98 last 166.36) - open/close 37605 (time 1746157850.81 total 292.98 last 137.57) - open/close 38832 (time 1746157860.81 total 302.99 last 122.65) - open/close 40000 (time 1746157869.26 total 311.44 last 138.17) - open/close 41625 (time 1746157879.27 total 321.44 last 162.42) - open/close 43435 (time 1746157889.27 total 331.45 last 180.93) - open/close 44900 (time 1746157899.27 total 341.45 last 146.48) - open/close 46524 (time 1746157909.28 total 351.46 last 162.32) - open/close 47838 (time 1746157919.28 total 361.46 last 131.38) - open/close 49415 (time 1746157929.28 total 371.46 last 157.69) - open/close 50000 (time 1746157932.74 total 374.92 last 169.17) - open/close 51303 (time 1746157942.74 total 384.92 last 130.26) - open/close 52518 (time 1746157952.75 total 394.92 last 121.44) - open/close 53750 (time 1746157962.76 total 404.93 last 123.08) - open/close 54962 (time 1746157972.76 total 414.94 last 121.15) - open/close 56105 (time 1746157982.77 total 424.95 last 114.18) - open/close 57370 (time 1746157992.77 total 434.95 last 126.50) - open/close 58589 (time 1746158002.79 total 444.97 last 121.69) - open/close 60000 (time 1746158011.21 total 453.39 last 167.50) - open/close 61689 (time 1746158021.21 total 463.39 last 168.90) - open/close 63446 (time 1746158031.22 total 473.39 last 175.64) total: 64768 open/close in 481.46 seconds: 134.52 ops/second - /unlink 3052 (time 1746158050.67 total 10.00 last 305.18) - /unlink 5804 (time 1746158060.67 total 20.00 last 275.19) - /unlink 8383 (time 1746158070.67 total 30.00 last 257.90) - /unlink 10000 (time 1746158078.48 total 37.80 last 207.21) - /unlink 12219 (time 1746158088.48 total 47.81 last 221.86) - /unlink 14291 (time 1746158098.48 total 57.81 last 207.11) - /unlink 16259 (time 1746158108.49 total 67.82 last 196.67) - /unlink 18375 (time 1746158118.49 total 77.82 last 211.53) - /unlink 20000 (time 1746158123.86 total 83.19 last 302.63) - /unlink 23108 (time 1746158133.87 total 93.19 last 310.78) - /unlink 25982 (time 1746158143.87 total 103.19 last 287.38) - /unlink 28431 (time 1746158153.87 total 113.19 last 244.89) - /unlink 30000 (time 1746158159.58 total 118.90 last 274.73) - /unlink 32294 (time 1746158169.58 total 128.90 last 229.37) - /unlink 34832 (time 1746158179.59 total 138.91 last 253.54) - /unlink 37141 (time 1746158189.59 total 148.91 last 230.90) - /unlink 39373 (time 1746158199.59 total 158.92 last 223.19) - /unlink 40000 (time 1746158201.89 total 161.22 last 272.56) - /unlink 42476 (time 1746158211.89 total 171.22 last 247.47) - /unlink 45548 (time 1746158221.90 total 181.22 last 307.14) - /unlink 48493 (time 1746158231.90 total 191.22 last 294.48) - /unlink 50000 (time 1746158238.39 total 197.71 last 232.26) - /unlink 51833 (time 1746158248.39 total 207.71 last 183.27) - /unlink 53822 (time 1746158258.39 total 217.72 last 198.82) - /unlink 56120 (time 1746158268.39 total 227.72 last 229.74) - /unlink 58687 (time 1746158278.39 total 237.72 last 256.67) - /unlink 60000 (time 1746158284.60 total 243.93 last 211.43) - /unlink 62201 (time 1746158294.61 total 253.93 last 220.10) - /unlink 64376 (time 1746158304.61 total 263.94 last 217.39) total: 64768 /unlink in 265.67 seconds: 243.79 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1484 (time 1746158334.56 total 10.00 last 148.38) - open/close 2983 (time 1746158344.56 total 20.00 last 149.85) - open/close 4459 (time 1746158354.57 total 30.01 last 147.49) - open/close 5717 (time 1746158364.57 total 40.02 last 125.74) - open/close 7057 (time 1746158374.57 total 50.02 last 133.96) - open/close 8444 (time 1746158384.58 total 60.02 last 138.67) - open/close 9818 (time 1746158394.58 total 70.02 last 137.39) - open/close 11062 (time 1746158404.58 total 80.03 last 124.35) - open/close 12349 (time 1746158414.58 total 90.03 last 128.66) - open/close 13838 (time 1746158424.59 total 100.03 last 148.83) - open/close 15262 (time 1746158434.59 total 110.03 last 142.40) - open/close 16508 (time 1746158444.59 total 120.04 last 124.58) - open/close 17837 (time 1746158454.59 total 130.04 last 132.90) - open/close 19229 (time 1746158464.59 total 140.04 last 139.16) - open/close 20000 (time 1746158470.16 total 145.60 last 138.54) - open/close 21505 (time 1746158480.17 total 155.61 last 150.40) - open/close 22958 (time 1746158490.17 total 165.61 last 145.28) - open/close 24173 (time 1746158500.17 total 175.61 last 121.47) - open/close 25400 (time 1746158510.18 total 185.62 last 122.60) - open/close 26631 (time 1746158520.18 total 195.63 last 123.04) - open/close 27893 (time 1746158530.18 total 205.63 last 126.19) - open/close 29169 (time 1746158540.19 total 215.63 last 127.56) - open/close 30000 (time 1746158546.64 total 222.08 last 128.81) - open/close 31390 (time 1746158556.64 total 232.09 last 138.94) - open/close 32705 (time 1746158566.64 total 242.09 last 131.46) - open/close 34201 (time 1746158576.65 total 252.10 last 149.51) - open/close 35579 (time 1746158586.65 total 262.10 last 137.74) - open/close 36936 (time 1746158596.66 total 272.10 last 135.65) - open/close 38434 (time 1746158606.66 total 282.10 last 149.79) - open/close 39930 (time 1746158616.66 total 292.11 last 149.56) - open/close 41616 (time 1746158626.67 total 302.12 last 168.40) - open/close 43186 (time 1746158636.68 total 312.12 last 156.98) - open/close 44603 (time 1746158646.68 total 322.13 last 141.62) - open/close 45898 (time 1746158656.68 total 332.13 last 129.47) - open/close 47173 (time 1746158666.69 total 342.13 last 127.45) - open/close 48444 (time 1746158676.69 total 352.14 last 127.06) - open/close 49785 (time 1746158686.70 total 362.15 last 133.98) - open/close 51257 (time 1746158696.70 total 372.15 last 147.18) - open/close 52725 (time 1746158706.73 total 382.18 last 146.38) - open/close 54048 (time 1746158716.73 total 392.18 last 132.28) - open/close 55287 (time 1746158726.73 total 402.18 last 123.88) - open/close 56476 (time 1746158736.74 total 412.19 last 118.82) - open/close 57641 (time 1746158746.74 total 422.19 last 116.48) - open/close 58867 (time 1746158756.75 total 432.19 last 122.55) - open/close 60000 (time 1746158766.70 total 442.15 last 113.79) - open/close 61223 (time 1746158776.71 total 452.15 last 122.24) - open/close 62598 (time 1746158786.71 total 462.15 last 137.49) - open/close 64069 (time 1746158796.72 total 472.17 last 146.93) total: 64768 open/close in 477.61 seconds: 135.61 ops/second - /unlink 2351 (time 1746158813.76 total 10.00 last 235.00) - /unlink 4432 (time 1746158823.76 total 20.01 last 208.02) - /unlink 6805 (time 1746158833.77 total 30.01 last 237.21) - /unlink 9064 (time 1746158843.77 total 40.01 last 225.85) - /unlink 10000 (time 1746158848.44 total 44.69 last 200.32) - /unlink 12081 (time 1746158858.44 total 54.69 last 208.09) - /unlink 14131 (time 1746158868.45 total 64.69 last 204.87) - /unlink 16529 (time 1746158878.46 total 74.70 last 239.61) - /unlink 18611 (time 1746158888.46 total 84.70 last 208.17) - /unlink 20000 (time 1746158894.03 total 90.27 last 249.37) - /unlink 22300 (time 1746158904.03 total 100.28 last 229.90) - /unlink 25362 (time 1746158914.04 total 110.28 last 306.08) - /unlink 28541 (time 1746158924.04 total 120.28 last 317.90) - /unlink 30000 (time 1746158929.40 total 125.64 last 272.24) - /unlink 32980 (time 1746158939.40 total 135.64 last 297.88) - /unlink 35719 (time 1746158949.40 total 145.65 last 273.87) - /unlink 38112 (time 1746158959.40 total 155.65 last 239.24) - /unlink 40000 (time 1746158967.02 total 163.27 last 247.84) - /unlink 42481 (time 1746158977.02 total 173.27 last 248.04) - /unlink 45499 (time 1746158987.03 total 183.27 last 301.61) - /unlink 48258 (time 1746158997.03 total 193.28 last 275.86) - /unlink 50000 (time 1746159003.69 total 199.93 last 261.80) - /unlink 52812 (time 1746159013.69 total 209.93 last 281.11) - /unlink 55631 (time 1746159023.69 total 219.94 last 281.84) - /unlink 58788 (time 1746159033.70 total 229.94 last 315.50) - /unlink 60000 (time 1746159036.65 total 232.90 last 410.12) - /unlink 64300 (time 1746159046.65 total 242.90 last 429.92) total: 64768 /unlink in 244.18 seconds: 265.25 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2203 (time 1746159074.88 total 10.00 last 220.28) - open/close 3725 (time 1746159084.88 total 20.00 last 152.16) - open/close 5441 (time 1746159094.89 total 30.01 last 171.48) - open/close 7503 (time 1746159104.89 total 40.01 last 206.16) - open/close 9102 (time 1746159114.89 total 50.01 last 159.89) - open/close 10000 (time 1746159120.35 total 55.47 last 164.55) - open/close 11369 (time 1746159130.36 total 65.48 last 136.73) - open/close 12624 (time 1746159140.37 total 75.49 last 125.46) - open/close 13947 (time 1746159150.37 total 85.49 last 132.21) - open/close 15371 (time 1746159160.38 total 95.50 last 142.33) - open/close 16999 (time 1746159170.38 total 105.50 last 162.80) - open/close 18576 (time 1746159180.39 total 115.50 last 157.59) - open/close 19861 (time 1746159190.39 total 125.51 last 128.47) - open/close 21468 (time 1746159200.39 total 135.51 last 160.67) - open/close 23226 (time 1746159210.40 total 145.52 last 175.66) - open/close 24798 (time 1746159220.40 total 155.52 last 157.20) - open/close 26392 (time 1746159230.40 total 165.52 last 159.33) - open/close 27913 (time 1746159240.40 total 175.52 last 152.08) - open/close 29615 (time 1746159250.41 total 185.53 last 170.13) - open/close 30000 (time 1746159252.94 total 188.06 last 151.85) - open/close 31258 (time 1746159262.95 total 198.07 last 125.71) - open/close 32741 (time 1746159272.95 total 208.07 last 148.24) - open/close 34418 (time 1746159282.96 total 218.07 last 167.69) - open/close 36118 (time 1746159292.96 total 228.08 last 169.97) - open/close 38232 (time 1746159302.96 total 238.08 last 211.38) - open/close 40000 (time 1746159312.23 total 247.35 last 190.75) - open/close 41705 (time 1746159322.23 total 257.35 last 170.39) - open/close 43905 (time 1746159332.23 total 267.35 last 219.99) - open/close 46249 (time 1746159342.24 total 277.36 last 234.23) - open/close 48136 (time 1746159352.24 total 287.36 last 188.70) - open/close 49752 (time 1746159362.24 total 297.36 last 161.58) - open/close 51451 (time 1746159372.25 total 307.36 last 169.86) - open/close 53049 (time 1746159382.25 total 317.37 last 159.73) - open/close 54941 (time 1746159392.25 total 327.37 last 189.16) - open/close 56533 (time 1746159402.25 total 337.37 last 159.15) - open/close 57850 (time 1746159412.26 total 347.38 last 131.68) - open/close 59505 (time 1746159422.26 total 357.38 last 165.50) - open/close 60000 (time 1746159425.05 total 360.17 last 177.32) - open/close 61654 (time 1746159435.05 total 370.17 last 165.36) - open/close 63315 (time 1746159445.05 total 380.17 last 166.07) total: 64768 open/close in 388.69 seconds: 166.63 ops/second - /unlink 3360 (time 1746159464.70 total 10.00 last 335.97) - /unlink 6286 (time 1746159474.71 total 20.00 last 292.56) - /unlink 8802 (time 1746159484.71 total 30.01 last 251.47) - /unlink 10000 (time 1746159489.00 total 34.29 last 279.44) - /unlink 12931 (time 1746159499.00 total 44.29 last 293.09) - /unlink 15440 (time 1746159509.00 total 54.30 last 250.86) - /unlink 17906 (time 1746159519.00 total 64.30 last 246.55) - /unlink 20000 (time 1746159527.94 total 73.24 last 234.21) - /unlink 22402 (time 1746159537.94 total 83.24 last 240.20) - /unlink 24948 (time 1746159547.94 total 93.24 last 254.59) - /unlink 27369 (time 1746159557.95 total 103.24 last 242.05) - /unlink 30000 (time 1746159566.47 total 111.77 last 308.58) - /unlink 32548 (time 1746159576.47 total 121.77 last 254.80) - /unlink 34712 (time 1746159586.47 total 131.77 last 216.37) - /unlink 37836 (time 1746159596.47 total 141.77 last 312.38) - /unlink 40000 (time 1746159604.22 total 149.51 last 279.51) - /unlink 42613 (time 1746159614.22 total 159.51 last 261.27) - /unlink 45053 (time 1746159624.22 total 169.51 last 243.99) - /unlink 48849 (time 1746159634.22 total 179.52 last 379.55) - /unlink 50000 (time 1746159636.68 total 181.98 last 466.88) - /unlink 52983 (time 1746159646.69 total 191.98 last 298.23) - /unlink 56061 (time 1746159656.69 total 201.98 last 307.74) - /unlink 59265 (time 1746159666.69 total 211.99 last 320.31) - /unlink 60000 (time 1746159669.09 total 214.39 last 306.22) - /unlink 63330 (time 1746159679.09 total 224.39 last 332.93) total: 64768 /unlink in 227.41 seconds: 284.81 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1748 (time 1746159711.04 total 10.00 last 174.75) - open/close 3267 (time 1746159721.05 total 20.01 last 151.86) - open/close 5384 (time 1746159731.05 total 30.01 last 211.66) - open/close 6868 (time 1746159741.05 total 40.01 last 148.39) - open/close 8176 (time 1746159751.05 total 50.01 last 130.77) - open/close 10000 (time 1746159758.95 total 57.91 last 230.88) - open/close 11918 (time 1746159768.95 total 67.91 last 191.74) - open/close 13585 (time 1746159778.96 total 77.92 last 166.66) - open/close 15147 (time 1746159788.96 total 87.92 last 156.14) - open/close 17353 (time 1746159798.96 total 97.92 last 220.55) - open/close 19468 (time 1746159808.97 total 107.93 last 211.42) - open/close 20000 (time 1746159812.08 total 111.04 last 170.93) - open/close 21552 (time 1746159822.09 total 121.05 last 155.02) - open/close 23328 (time 1746159832.09 total 131.05 last 177.58) - open/close 24818 (time 1746159842.10 total 141.06 last 148.92) - open/close 26547 (time 1746159852.10 total 151.06 last 172.89) - open/close 28149 (time 1746159862.10 total 161.06 last 160.19) - open/close 29645 (time 1746159872.10 total 171.06 last 149.58) - open/close 30000 (time 1746159874.36 total 173.32 last 156.99) - open/close 31391 (time 1746159884.36 total 183.32 last 139.08) - open/close 32770 (time 1746159894.37 total 193.33 last 137.85) - open/close 33933 (time 1746159904.37 total 203.33 last 116.28) - open/close 35199 (time 1746159914.38 total 213.34 last 126.41) - open/close 36385 (time 1746159924.39 total 223.35 last 118.47) - open/close 37707 (time 1746159934.40 total 233.36 last 132.15) - open/close 39045 (time 1746159944.40 total 243.36 last 133.77) - open/close 40000 (time 1746159951.67 total 250.63 last 131.30) - open/close 41265 (time 1746159961.68 total 260.64 last 126.45) - open/close 42527 (time 1746159971.68 total 270.64 last 126.16) - open/close 43655 (time 1746159981.68 total 280.64 last 112.74) - open/close 44931 (time 1746159991.69 total 290.65 last 127.51) - open/close 46172 (time 1746160001.70 total 300.66 last 124.04) - open/close 47434 (time 1746160011.70 total 310.66 last 126.11) - open/close 48756 (time 1746160021.71 total 320.67 last 132.10) - open/close 49967 (time 1746160031.71 total 330.67 last 121.06) - open/close 51214 (time 1746160041.72 total 340.68 last 124.65) - open/close 52328 (time 1746160051.72 total 350.68 last 111.33) - open/close 53492 (time 1746160061.73 total 360.69 last 116.38) - open/close 54688 (time 1746160071.73 total 370.69 last 119.54) - open/close 55817 (time 1746160081.74 total 380.70 last 112.82) - open/close 57042 (time 1746160091.74 total 390.70 last 122.45) - open/close 58290 (time 1746160101.75 total 400.71 last 124.75) - open/close 59629 (time 1746160111.75 total 410.71 last 133.89) - open/close 60000 (time 1746160115.00 total 413.96 last 113.99) - open/close 61227 (time 1746160125.00 total 423.96 last 122.70) - open/close 62426 (time 1746160135.00 total 433.96 last 119.90) - open/close 63794 (time 1746160145.01 total 443.97 last 136.71) total: 64768 open/close in 450.92 seconds: 143.63 ops/second - /unlink 2158 (time 1746160163.42 total 10.00 last 215.70) - /unlink 4192 (time 1746160173.42 total 20.01 last 203.35) - /unlink 5918 (time 1746160183.42 total 30.01 last 172.58) - /unlink 7911 (time 1746160193.42 total 40.01 last 199.22) - /unlink 9853 (time 1746160203.42 total 50.01 last 194.19) - /unlink 12184 (time 1746160213.42 total 60.01 last 233.10) - /unlink 14249 (time 1746160223.43 total 70.01 last 206.47) - /unlink 16363 (time 1746160233.43 total 80.02 last 211.37) - /unlink 18208 (time 1746160243.43 total 90.02 last 184.42) - /unlink 20000 (time 1746160252.24 total 98.83 last 203.51) - /unlink 22758 (time 1746160262.24 total 108.83 last 275.73) - /unlink 25231 (time 1746160272.24 total 118.83 last 247.29) - /unlink 28138 (time 1746160282.24 total 128.83 last 290.64) - /unlink 30000 (time 1746160290.29 total 136.88 last 231.39) - /unlink 32421 (time 1746160300.29 total 146.88 last 242.09) - /unlink 34730 (time 1746160310.29 total 156.88 last 230.85) - /unlink 36813 (time 1746160320.29 total 166.88 last 208.29) - /unlink 38971 (time 1746160330.29 total 176.88 last 215.78) - /unlink 40000 (time 1746160334.69 total 181.28 last 234.07) - /unlink 42945 (time 1746160344.69 total 191.28 last 294.48) - /unlink 45445 (time 1746160354.69 total 201.28 last 249.94) - /unlink 47995 (time 1746160364.69 total 211.28 last 254.94) - /unlink 50000 (time 1746160372.28 total 218.86 last 264.42) - /unlink 53122 (time 1746160382.28 total 228.87 last 312.15) - /unlink 57366 (time 1746160392.28 total 238.87 last 424.30) - /unlink 59753 (time 1746160402.29 total 248.88 last 238.54) - /unlink 62183 (time 1746160412.29 total 258.88 last 242.86) total: 64768 /unlink in 266.83 seconds: 242.74 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg407-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg407-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server stop mds service on oleg407-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg407-server unloading modules via unload_modules_local on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2 PASS 106 (3741s) Stopping clients: oleg407-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg407-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2 oleg407-server: oleg407-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing load_modules_local oleg407-server: Loading modules from /home/green/git/lustre-release/lustre oleg407-server: detected 4 online CPUs by sysfs oleg407-server: MODOPTS_LIBCFS= oleg407-server: Force libcfs to create 2 CPU partitions oleg407-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg407-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3849 sec ========== 00:34:37 (1746160477) === conf-sanity: start cleanup 00:34:38 (1746160478) === === conf-sanity: finish cleanup 00:34:39 (1746160479) ===