-----============= acceptance-small: conf-sanity ============----- Thu May 1 16:48:33 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg330-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg330-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg330-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg330-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg330-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg330-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg330-server oleg330-server: oleg330-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions oleg330-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg330-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg330-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg330-server: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg330-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg330-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg330-server stop mds service on oleg330-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg330-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:50:02 (1746132602) Stopping clients: oleg330-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg330-client.virtnet /mnt/lustre2 (opts:-f) oleg330-server: oleg330-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg330-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg330-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1394 (time 1746132657.38 total 10.00 last 139.38) - open/close 2453 (time 1746132667.39 total 20.00 last 105.86) - open/close 3552 (time 1746132677.39 total 30.01 last 109.88) - open/close 4699 (time 1746132687.39 total 40.01 last 114.68) - open/close 5973 (time 1746132697.39 total 50.01 last 127.35) - open/close 6937 (time 1746132707.39 total 60.01 last 96.40) - open/close 8006 (time 1746132717.40 total 70.02 last 106.83) - open/close 9011 (time 1746132727.40 total 80.02 last 100.47) - open/close 9959 (time 1746132737.41 total 90.03 last 94.72) - open/close 10895 (time 1746132747.42 total 100.04 last 93.53) - open/close 11986 (time 1746132757.43 total 110.04 last 109.04) - open/close 13244 (time 1746132767.43 total 120.04 last 125.79) - open/close 14302 (time 1746132777.43 total 130.05 last 105.77) - open/close 15367 (time 1746132787.44 total 140.06 last 106.38) - open/close 16540 (time 1746132797.44 total 150.06 last 117.28) - open/close 17748 (time 1746132807.45 total 160.07 last 120.73) - open/close 18735 (time 1746132817.46 total 170.08 last 98.61) - open/close 19763 (time 1746132827.46 total 180.08 last 102.76) - open/close 20000 (time 1746132829.72 total 182.33 last 105.17) - open/close 21185 (time 1746132839.73 total 192.35 last 118.36) - open/close 22275 (time 1746132849.74 total 202.36 last 108.90) - open/close 23483 (time 1746132859.74 total 212.36 last 120.72) - open/close 24644 (time 1746132869.76 total 222.37 last 115.96) - open/close 25647 (time 1746132879.76 total 232.38 last 100.24) - open/close 26655 (time 1746132889.77 total 242.39 last 100.70) - open/close 27693 (time 1746132899.78 total 252.39 last 103.76) - open/close 28912 (time 1746132909.78 total 262.40 last 121.88) - open/close 30000 (time 1746132919.12 total 271.74 last 116.41) - open/close 31210 (time 1746132929.13 total 281.74 last 120.97) - open/close 32530 (time 1746132939.14 total 291.75 last 131.88) - open/close 33719 (time 1746132949.14 total 301.76 last 118.79) - open/close 34926 (time 1746132959.15 total 311.77 last 120.65) - open/close 36156 (time 1746132969.15 total 321.77 last 122.95) - open/close 37502 (time 1746132979.16 total 331.77 last 134.55) - open/close 38657 (time 1746132989.17 total 341.79 last 115.30) - open/close 39872 (time 1746132999.19 total 351.80 last 121.35) - open/close 41376 (time 1746133009.19 total 361.81 last 150.33) - open/close 42758 (time 1746133019.19 total 371.81 last 138.19) - open/close 44255 (time 1746133029.19 total 381.81 last 149.68) - open/close 45489 (time 1746133039.19 total 391.81 last 123.39) - open/close 46661 (time 1746133049.20 total 401.82 last 117.14) - open/close 47955 (time 1746133059.21 total 411.83 last 129.22) - open/close 49300 (time 1746133069.22 total 421.84 last 134.43) - open/close 50000 (time 1746133075.46 total 428.08 last 112.17) - open/close 51510 (time 1746133085.46 total 438.08 last 150.96) - open/close 53022 (time 1746133095.46 total 448.08 last 151.18) - open/close 54352 (time 1746133105.47 total 458.09 last 132.88) - open/close 55463 (time 1746133115.49 total 468.11 last 110.91) - open/close 56472 (time 1746133125.50 total 478.11 last 100.82) - open/close 57781 (time 1746133135.50 total 488.11 last 130.89) - open/close 58850 (time 1746133145.50 total 498.11 last 106.90) - open/close 59845 (time 1746133155.50 total 508.12 last 99.43) - open/close 60936 (time 1746133165.51 total 518.13 last 108.99) - open/close 62017 (time 1746133175.53 total 528.14 last 107.98) - open/close 63149 (time 1746133185.53 total 538.15 last 113.17) - open/close 64276 (time 1746133195.53 total 548.15 last 112.69) total: 64768 open/close in 552.12 seconds: 117.31 ops/second - /unlink 2339 (time 1746133210.99 total 10.00 last 233.88) - /unlink 4416 (time 1746133220.99 total 20.01 last 207.55) - /unlink 6559 (time 1746133231.00 total 30.01 last 214.22) - /unlink 8332 (time 1746133241.00 total 40.01 last 177.29) - /unlink 10000 (time 1746133249.08 total 48.10 last 206.28) - /unlink 12242 (time 1746133259.09 total 58.11 last 224.03) - /unlink 14592 (time 1746133269.09 total 68.11 last 234.95) - /unlink 16632 (time 1746133279.10 total 78.11 last 203.86) - /unlink 18623 (time 1746133289.10 total 88.12 last 199.01) - /unlink 20000 (time 1746133296.91 total 95.92 last 176.48) - /unlink 21790 (time 1746133306.91 total 105.93 last 178.92) - /unlink 23751 (time 1746133316.92 total 115.93 last 195.97) - /unlink 25858 (time 1746133326.92 total 125.94 last 210.63) - /unlink 27774 (time 1746133336.92 total 135.94 last 191.52) - /unlink 29855 (time 1746133346.93 total 145.94 last 208.08) - /unlink 32052 (time 1746133356.93 total 155.94 last 219.67) - /unlink 33839 (time 1746133366.93 total 165.94 last 178.68) - /unlink 36086 (time 1746133376.93 total 175.95 last 224.66) - /unlink 39336 (time 1746133386.93 total 185.95 last 324.98) - /unlink 40000 (time 1746133389.38 total 188.40 last 270.80) - /unlink 42150 (time 1746133399.38 total 198.40 last 214.97) - /unlink 44546 (time 1746133409.38 total 208.40 last 239.60) - /unlink 46904 (time 1746133419.39 total 218.40 last 235.78) - /unlink 49365 (time 1746133429.39 total 228.40 last 246.07) - /unlink 50000 (time 1746133431.94 total 230.96 last 248.58) - /unlink 52256 (time 1746133441.94 total 240.96 last 225.57) - /unlink 54447 (time 1746133451.95 total 250.96 last 219.03) - /unlink 56457 (time 1746133461.95 total 260.96 last 200.96) - /unlink 58477 (time 1746133471.95 total 270.96 last 201.98) - /unlink 60000 (time 1746133479.16 total 278.17 last 211.25) - /unlink 61886 (time 1746133489.16 total 288.18 last 188.54) - /unlink 63879 (time 1746133499.16 total 298.18 last 199.29) total: 64768 /unlink in 302.76 seconds: 213.93 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1230 (time 1746133529.62 total 10.00 last 122.97) - open/close 2290 (time 1746133539.63 total 20.00 last 105.99) - open/close 3513 (time 1746133549.63 total 30.00 last 122.29) - open/close 5050 (time 1746133559.63 total 40.01 last 153.67) - open/close 6294 (time 1746133569.64 total 50.01 last 124.32) - open/close 7530 (time 1746133579.64 total 60.01 last 123.59) - open/close 8879 (time 1746133589.64 total 70.02 last 134.85) - open/close 10000 (time 1746133597.91 total 78.28 last 135.61) - open/close 11293 (time 1746133607.91 total 88.29 last 129.28) - open/close 12535 (time 1746133617.91 total 98.29 last 124.17) - open/close 13806 (time 1746133627.92 total 108.30 last 126.99) - open/close 15128 (time 1746133637.92 total 118.30 last 132.17) - open/close 16338 (time 1746133647.92 total 128.30 last 120.99) - open/close 17503 (time 1746133657.93 total 138.30 last 116.44) - open/close 18763 (time 1746133667.93 total 148.31 last 125.98) - open/close 20000 (time 1746133677.15 total 157.53 last 134.18) - open/close 21378 (time 1746133687.15 total 167.53 last 137.74) - open/close 22627 (time 1746133697.15 total 177.53 last 124.90) - open/close 23792 (time 1746133707.16 total 187.54 last 116.38) - open/close 24910 (time 1746133717.17 total 197.54 last 111.77) - open/close 26177 (time 1746133727.17 total 207.54 last 126.68) - open/close 27469 (time 1746133737.17 total 217.55 last 129.17) - open/close 28648 (time 1746133747.17 total 227.55 last 117.86) - open/close 29854 (time 1746133757.18 total 237.56 last 120.51) - open/close 31096 (time 1746133767.20 total 247.57 last 123.99) - open/close 32455 (time 1746133777.20 total 257.58 last 135.87) - open/close 33676 (time 1746133787.20 total 267.58 last 122.06) - open/close 34975 (time 1746133797.20 total 277.58 last 129.87) - open/close 36261 (time 1746133807.21 total 287.59 last 128.50) - open/close 37411 (time 1746133817.21 total 297.59 last 114.97) - open/close 38403 (time 1746133827.22 total 307.60 last 99.14) - open/close 39390 (time 1746133837.22 total 317.60 last 98.65) - open/close 40000 (time 1746133843.23 total 323.61 last 101.60) - open/close 41146 (time 1746133853.23 total 333.61 last 114.58) - open/close 42267 (time 1746133863.24 total 343.61 last 112.03) - open/close 43502 (time 1746133873.24 total 353.62 last 123.43) - open/close 44834 (time 1746133883.24 total 363.62 last 133.17) - open/close 45935 (time 1746133893.25 total 373.63 last 110.00) - open/close 47076 (time 1746133903.26 total 383.63 last 114.08) - open/close 48346 (time 1746133913.26 total 393.63 last 126.99) - open/close 49630 (time 1746133923.26 total 403.64 last 128.37) - open/close 50000 (time 1746133926.18 total 406.56 last 126.72) - open/close 51182 (time 1746133936.20 total 416.58 last 117.91) - open/close 52314 (time 1746133946.21 total 426.59 last 113.08) - open/close 53321 (time 1746133956.26 total 436.64 last 100.25) - open/close 54278 (time 1746133966.26 total 446.64 last 95.64) - open/close 55288 (time 1746133976.27 total 456.65 last 100.95) - open/close 56298 (time 1746133986.27 total 466.65 last 101.00) - open/close 57387 (time 1746133996.31 total 476.69 last 108.43) - open/close 58339 (time 1746134006.31 total 486.69 last 95.19) - open/close 59345 (time 1746134016.31 total 496.69 last 100.60) - open/close 60000 (time 1746134022.15 total 502.52 last 112.34) - open/close 61062 (time 1746134032.16 total 512.54 last 106.02) - open/close 62047 (time 1746134042.17 total 522.55 last 98.44) - open/close 62968 (time 1746134052.17 total 532.55 last 92.07) - open/close 63933 (time 1746134062.19 total 542.57 last 96.33) total: 64768 open/close in 552.03 seconds: 117.33 ops/second - /unlink 1571 (time 1746134083.88 total 10.01 last 157.01) - /unlink 3119 (time 1746134093.89 total 20.01 last 154.71) - /unlink 4668 (time 1746134103.89 total 30.02 last 154.84) - /unlink 6221 (time 1746134113.89 total 40.02 last 155.24) - /unlink 7789 (time 1746134123.90 total 50.02 last 156.77) - /unlink 9324 (time 1746134133.90 total 60.03 last 153.40) - /unlink 10000 (time 1746134138.11 total 64.23 last 160.64) - /unlink 11667 (time 1746134148.11 total 74.24 last 166.68) - /unlink 13383 (time 1746134158.12 total 84.24 last 171.53) - /unlink 15161 (time 1746134168.12 total 94.24 last 177.78) - /unlink 17055 (time 1746134178.12 total 104.24 last 189.38) - /unlink 18727 (time 1746134188.12 total 114.24 last 167.20) - /unlink 20000 (time 1746134194.81 total 120.93 last 190.38) - /unlink 21794 (time 1746134204.81 total 130.93 last 179.36) - /unlink 23653 (time 1746134214.81 total 140.93 last 185.89) - /unlink 25365 (time 1746134224.81 total 150.94 last 171.09) - /unlink 27028 (time 1746134234.82 total 160.95 last 166.19) - /unlink 28833 (time 1746134244.82 total 170.95 last 180.49) - /unlink 30000 (time 1746134251.09 total 177.21 last 186.16) - /unlink 31600 (time 1746134261.09 total 187.22 last 159.97) - /unlink 33444 (time 1746134271.09 total 197.22 last 184.36) - /unlink 35438 (time 1746134281.10 total 207.22 last 199.38) - /unlink 37446 (time 1746134291.10 total 217.22 last 200.73) - /unlink 39648 (time 1746134301.10 total 227.23 last 220.12) - /unlink 41666 (time 1746134311.11 total 237.23 last 201.75) - /unlink 43573 (time 1746134321.11 total 247.23 last 190.69) - /unlink 45389 (time 1746134331.11 total 257.23 last 181.60) - /unlink 47261 (time 1746134341.11 total 267.23 last 187.17) - /unlink 49273 (time 1746134351.11 total 277.23 last 201.19) - /unlink 50000 (time 1746134354.78 total 280.91 last 197.87) - /unlink 51946 (time 1746134364.78 total 290.91 last 194.60) - /unlink 53792 (time 1746134374.79 total 300.91 last 184.51) - /unlink 55802 (time 1746134384.79 total 310.92 last 200.92) - /unlink 57681 (time 1746134394.79 total 320.92 last 187.87) - /unlink 59621 (time 1746134404.79 total 330.92 last 193.98) - /unlink 60000 (time 1746134406.83 total 332.96 last 186.06) - /unlink 61842 (time 1746134416.83 total 342.96 last 184.16) - /unlink 63960 (time 1746134426.84 total 352.96 last 211.72) total: 64768 /unlink in 356.47 seconds: 181.69 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1120 (time 1746134462.23 total 10.00 last 111.95) - open/close 2373 (time 1746134472.23 total 20.01 last 125.26) - open/close 3611 (time 1746134482.24 total 30.02 last 123.69) - open/close 4820 (time 1746134492.25 total 40.03 last 120.79) - open/close 5990 (time 1746134502.25 total 50.03 last 116.98) - open/close 7176 (time 1746134512.26 total 60.04 last 118.50) - open/close 8376 (time 1746134522.26 total 70.04 last 119.99) - open/close 9554 (time 1746134532.26 total 80.04 last 117.75) - open/close 10000 (time 1746134536.37 total 84.14 last 108.68) - open/close 11069 (time 1746134546.37 total 94.15 last 106.87) - open/close 12048 (time 1746134556.40 total 104.18 last 97.59) - open/close 13051 (time 1746134566.41 total 114.19 last 100.22) - open/close 14176 (time 1746134576.41 total 124.19 last 112.50) - open/close 15353 (time 1746134586.41 total 134.19 last 117.65) - open/close 16599 (time 1746134596.42 total 144.20 last 124.55) - open/close 17978 (time 1746134606.42 total 154.20 last 137.88) - open/close 19117 (time 1746134616.42 total 164.20 last 113.90) - open/close 20000 (time 1746134624.01 total 171.79 last 116.36) - open/close 21229 (time 1746134634.01 total 181.79 last 122.84) - open/close 22463 (time 1746134644.01 total 191.79 last 123.39) - open/close 23702 (time 1746134654.02 total 201.80 last 123.83) - open/close 25177 (time 1746134664.02 total 211.80 last 147.47) - open/close 26635 (time 1746134674.02 total 221.80 last 145.77) - open/close 28054 (time 1746134684.03 total 231.81 last 141.85) - open/close 29538 (time 1746134694.03 total 241.81 last 148.34) - open/close 30000 (time 1746134696.97 total 244.75 last 157.34) - open/close 31399 (time 1746134706.97 total 254.75 last 139.88) - open/close 32765 (time 1746134716.97 total 264.75 last 136.57) - open/close 34270 (time 1746134726.97 total 274.75 last 150.46) - open/close 35627 (time 1746134736.98 total 284.76 last 135.56) - open/close 36982 (time 1746134746.99 total 294.77 last 135.45) - open/close 38670 (time 1746134756.99 total 304.77 last 168.79) - open/close 40000 (time 1746134765.54 total 313.32 last 155.56) - open/close 41513 (time 1746134775.54 total 323.32 last 151.25) - open/close 42838 (time 1746134785.54 total 333.32 last 132.44) - open/close 44522 (time 1746134795.55 total 343.33 last 168.36) - open/close 46220 (time 1746134805.55 total 353.33 last 169.77) - open/close 47650 (time 1746134815.55 total 363.33 last 142.94) - open/close 49316 (time 1746134825.56 total 373.34 last 166.52) - open/close 50000 (time 1746134829.77 total 377.55 last 162.50) - open/close 51684 (time 1746134839.77 total 387.55 last 168.34) - open/close 53320 (time 1746134849.78 total 397.56 last 163.52) - open/close 54840 (time 1746134859.78 total 407.56 last 151.95) - open/close 56254 (time 1746134869.78 total 417.56 last 141.35) - open/close 58263 (time 1746134879.78 total 427.56 last 200.87) - open/close 60000 (time 1746134889.51 total 437.29 last 178.57) - open/close 61836 (time 1746134899.52 total 447.30 last 183.47) - open/close 63356 (time 1746134909.52 total 457.30 last 151.95) total: 64768 open/close in 466.90 seconds: 138.72 ops/second - /unlink 3115 (time 1746134930.80 total 10.00 last 311.42) - /unlink 6813 (time 1746134940.80 total 20.01 last 369.69) - /unlink 10000 (time 1746134950.06 total 29.26 last 344.28) - /unlink 12811 (time 1746134960.06 total 39.26 last 281.08) - /unlink 15408 (time 1746134970.06 total 49.26 last 259.69) - /unlink 17760 (time 1746134980.06 total 59.27 last 235.10) - /unlink 20000 (time 1746134989.52 total 68.73 last 236.82) - /unlink 22677 (time 1746134999.52 total 78.73 last 267.69) - /unlink 25681 (time 1746135009.53 total 88.73 last 300.23) - /unlink 28778 (time 1746135019.53 total 98.74 last 309.56) - /unlink 30000 (time 1746135023.53 total 102.74 last 305.49) - /unlink 33171 (time 1746135033.53 total 112.74 last 317.05) - /unlink 35921 (time 1746135043.53 total 122.74 last 274.98) - /unlink 38085 (time 1746135053.54 total 132.74 last 216.36) - /unlink 40000 (time 1746135061.11 total 140.32 last 252.82) - /unlink 42865 (time 1746135071.12 total 150.32 last 286.35) - /unlink 45408 (time 1746135081.12 total 160.32 last 254.23) - /unlink 48523 (time 1746135091.12 total 170.32 last 311.48) - /unlink 50000 (time 1746135096.86 total 176.06 last 257.38) - /unlink 52713 (time 1746135106.86 total 186.07 last 271.21) - /unlink 55315 (time 1746135116.86 total 196.07 last 260.19) - /unlink 58059 (time 1746135126.86 total 206.07 last 274.40) - /unlink 60000 (time 1746135134.13 total 213.34 last 267.02) - /unlink 62987 (time 1746135144.14 total 223.34 last 298.54) total: 64768 /unlink in 230.70 seconds: 280.75 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1390 (time 1746135183.20 total 10.00 last 138.94) - open/close 2824 (time 1746135193.20 total 20.01 last 143.39) - open/close 4208 (time 1746135203.21 total 30.01 last 138.33) - open/close 5723 (time 1746135213.21 total 40.01 last 151.45) - open/close 7371 (time 1746135223.21 total 50.02 last 164.77) - open/close 8837 (time 1746135233.21 total 60.02 last 146.57) - open/close 10000 (time 1746135241.89 total 68.69 last 134.06) - open/close 11429 (time 1746135251.90 total 78.70 last 142.79) - open/close 12875 (time 1746135261.90 total 88.70 last 144.59) - open/close 14211 (time 1746135271.90 total 98.71 last 133.49) - open/close 15644 (time 1746135281.91 total 108.71 last 143.24) - open/close 16969 (time 1746135291.91 total 118.71 last 132.49) - open/close 18301 (time 1746135301.91 total 128.72 last 133.14) - open/close 19601 (time 1746135311.92 total 138.72 last 129.96) - open/close 20000 (time 1746135314.40 total 141.20 last 160.95) - open/close 21346 (time 1746135324.40 total 151.20 last 134.56) - open/close 22714 (time 1746135334.41 total 161.21 last 136.67) - open/close 23931 (time 1746135344.42 total 171.22 last 121.58) - open/close 25185 (time 1746135354.42 total 181.23 last 125.35) - open/close 26663 (time 1746135364.43 total 191.23 last 147.72) - open/close 28146 (time 1746135374.43 total 201.23 last 148.27) - open/close 29616 (time 1746135384.43 total 211.24 last 146.99) - open/close 30000 (time 1746135387.24 total 214.05 last 136.50) - open/close 31383 (time 1746135397.25 total 224.05 last 138.26) - open/close 32798 (time 1746135407.25 total 234.05 last 141.50) - open/close 34340 (time 1746135417.26 total 244.06 last 154.09) - open/close 35926 (time 1746135427.26 total 254.06 last 158.56) - open/close 37489 (time 1746135437.26 total 264.06 last 156.27) - open/close 38886 (time 1746135447.27 total 274.07 last 139.62) - open/close 40000 (time 1746135455.32 total 282.12 last 138.39) - open/close 41338 (time 1746135465.32 total 292.12 last 133.75) - open/close 42694 (time 1746135475.32 total 302.12 last 135.60) - open/close 44034 (time 1746135485.33 total 312.13 last 133.92) - open/close 45573 (time 1746135495.33 total 322.13 last 153.86) - open/close 46921 (time 1746135505.33 total 332.14 last 134.72) - open/close 48219 (time 1746135515.34 total 342.14 last 129.78) - open/close 49553 (time 1746135525.34 total 352.14 last 133.37) - open/close 50000 (time 1746135528.48 total 355.28 last 142.33) - open/close 51398 (time 1746135538.49 total 365.29 last 139.64) - open/close 52679 (time 1746135548.49 total 375.30 last 128.06) - open/close 54023 (time 1746135558.49 total 385.30 last 134.38) - open/close 55377 (time 1746135568.50 total 395.30 last 135.32) - open/close 56724 (time 1746135578.51 total 405.32 last 134.54) - open/close 58201 (time 1746135588.51 total 415.32 last 147.68) - open/close 59647 (time 1746135598.53 total 425.33 last 144.34) - open/close 60000 (time 1746135601.11 total 427.91 last 136.84) - open/close 61469 (time 1746135611.11 total 437.92 last 146.88) - open/close 62862 (time 1746135621.11 total 447.92 last 139.28) - open/close 64306 (time 1746135631.13 total 457.93 last 144.24) total: 64768 open/close in 461.18 seconds: 140.44 ops/second - /unlink 2660 (time 1746135645.99 total 10.00 last 265.95) - /unlink 4790 (time 1746135656.00 total 20.01 last 212.91) - /unlink 6865 (time 1746135666.00 total 30.01 last 207.48) - /unlink 9256 (time 1746135676.00 total 40.01 last 239.03) - /unlink 10000 (time 1746135679.23 total 43.24 last 230.64) - /unlink 12628 (time 1746135689.23 total 53.24 last 262.74) - /unlink 15085 (time 1746135699.23 total 63.24 last 245.67) - /unlink 17459 (time 1746135709.23 total 73.24 last 237.36) - /unlink 20000 (time 1746135718.16 total 82.17 last 284.70) - /unlink 22467 (time 1746135728.16 total 92.17 last 246.58) - /unlink 24225 (time 1746135738.16 total 102.17 last 175.75) - /unlink 26388 (time 1746135748.17 total 112.18 last 216.23) - /unlink 28517 (time 1746135758.17 total 122.18 last 212.87) - /unlink 30000 (time 1746135766.24 total 130.25 last 183.77) - /unlink 32207 (time 1746135776.24 total 140.25 last 220.65) - /unlink 34285 (time 1746135786.24 total 150.25 last 207.78) - /unlink 36196 (time 1746135796.24 total 160.25 last 191.07) - /unlink 38431 (time 1746135806.25 total 170.26 last 223.32) - /unlink 40000 (time 1746135812.81 total 176.82 last 239.07) - /unlink 42179 (time 1746135822.82 total 186.83 last 217.82) - /unlink 44396 (time 1746135832.82 total 196.83 last 221.66) - /unlink 46488 (time 1746135842.83 total 206.84 last 209.10) - /unlink 48784 (time 1746135852.83 total 216.84 last 229.55) - /unlink 50000 (time 1746135858.09 total 222.10 last 231.11) - /unlink 52569 (time 1746135868.09 total 232.10 last 256.87) - /unlink 55137 (time 1746135878.10 total 242.11 last 256.65) - /unlink 57026 (time 1746135888.10 total 252.11 last 188.74) - /unlink 59155 (time 1746135898.11 total 262.12 last 212.87) - /unlink 60000 (time 1746135901.95 total 265.96 last 220.05) - /unlink 62354 (time 1746135911.95 total 275.96 last 235.37) - /unlink 64681 (time 1746135921.95 total 285.96 last 232.70) total: 64768 /unlink in 286.33 seconds: 226.20 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1435 (time 1746135953.60 total 10.00 last 143.45) - open/close 2928 (time 1746135963.60 total 20.01 last 149.25) - open/close 4385 (time 1746135973.61 total 30.01 last 145.66) - open/close 6099 (time 1746135983.61 total 40.01 last 171.34) - open/close 8119 (time 1746135993.61 total 50.02 last 201.95) - open/close 10000 (time 1746136002.90 total 59.30 last 202.49) - open/close 11646 (time 1746136012.91 total 69.31 last 164.52) - open/close 13182 (time 1746136022.91 total 79.31 last 153.54) - open/close 14788 (time 1746136032.91 total 89.32 last 160.54) - open/close 16246 (time 1746136042.92 total 99.32 last 145.75) - open/close 17933 (time 1746136052.92 total 109.32 last 168.70) - open/close 19423 (time 1746136062.92 total 119.33 last 148.91) - open/close 20000 (time 1746136066.80 total 123.21 last 148.77) - open/close 21510 (time 1746136076.80 total 133.21 last 151.00) - open/close 22912 (time 1746136086.80 total 143.21 last 140.19) - open/close 24255 (time 1746136096.81 total 153.21 last 134.26) - open/close 25650 (time 1746136106.81 total 163.21 last 139.45) - open/close 27012 (time 1746136116.82 total 173.22 last 136.11) - open/close 28477 (time 1746136126.83 total 183.23 last 146.38) - open/close 29774 (time 1746136136.83 total 193.23 last 129.69) - open/close 31148 (time 1746136146.83 total 203.23 last 137.36) - open/close 32483 (time 1746136156.84 total 213.24 last 133.42) - open/close 33910 (time 1746136166.84 total 223.24 last 142.63) - open/close 35341 (time 1746136176.84 total 233.25 last 143.07) - open/close 36784 (time 1746136186.85 total 243.25 last 144.20) - open/close 38313 (time 1746136196.85 total 253.26 last 152.87) - open/close 39726 (time 1746136206.86 total 263.26 last 141.25) - open/close 41101 (time 1746136216.86 total 273.26 last 137.47) - open/close 43054 (time 1746136226.86 total 283.26 last 195.25) - open/close 44752 (time 1746136236.86 total 293.27 last 169.76) - open/close 46336 (time 1746136246.87 total 303.27 last 158.36) - open/close 48112 (time 1746136256.87 total 313.27 last 177.51) - open/close 49645 (time 1746136266.87 total 323.27 last 153.29) - open/close 50000 (time 1746136269.10 total 325.50 last 159.19) - open/close 51483 (time 1746136279.11 total 335.52 last 148.12) - open/close 52899 (time 1746136289.12 total 345.52 last 141.55) - open/close 54476 (time 1746136299.12 total 355.52 last 157.66) - open/close 56085 (time 1746136309.12 total 365.52 last 160.87) - open/close 57658 (time 1746136319.13 total 375.53 last 157.25) - open/close 59482 (time 1746136329.13 total 385.53 last 182.37) - open/close 60000 (time 1746136332.05 total 388.45 last 177.08) - open/close 61754 (time 1746136342.06 total 398.46 last 175.33) - open/close 63300 (time 1746136352.06 total 408.46 last 154.58) - open/close 64700 (time 1746136362.06 total 418.47 last 139.91) total: 64768 open/close in 418.90 seconds: 154.61 ops/second - /unlink 2305 (time 1746136374.12 total 10.00 last 230.47) - /unlink 4641 (time 1746136384.12 total 20.00 last 233.57) - /unlink 7186 (time 1746136394.13 total 30.00 last 254.45) - /unlink 9897 (time 1746136404.13 total 40.01 last 271.04) - /unlink 12629 (time 1746136414.13 total 50.01 last 273.15) - /unlink 15084 (time 1746136424.13 total 60.01 last 245.42) - /unlink 17498 (time 1746136434.13 total 70.01 last 241.35) - /unlink 20000 (time 1746136442.47 total 78.34 last 300.35) - /unlink 22943 (time 1746136452.47 total 88.34 last 294.28) - /unlink 25943 (time 1746136462.47 total 98.35 last 299.92) - /unlink 29721 (time 1746136472.47 total 108.35 last 377.80) - /unlink 33362 (time 1746136482.47 total 118.35 last 364.00) - /unlink 36416 (time 1746136492.47 total 128.35 last 305.30) - /unlink 39143 (time 1746136502.48 total 138.35 last 272.69) - /unlink 40000 (time 1746136506.82 total 142.70 last 197.19) - /unlink 41871 (time 1746136516.82 total 152.70 last 187.07) - /unlink 44035 (time 1746136526.83 total 162.71 last 216.29) - /unlink 46473 (time 1746136536.83 total 172.71 last 243.68) - /unlink 48979 (time 1746136546.83 total 182.71 last 250.55) - /unlink 50000 (time 1746136550.00 total 185.88 last 322.11) - /unlink 52579 (time 1746136560.00 total 195.88 last 257.88) - /unlink 55214 (time 1746136570.00 total 205.88 last 263.50) - /unlink 58009 (time 1746136580.01 total 215.89 last 279.46) - /unlink 60000 (time 1746136586.89 total 222.77 last 289.38) - /unlink 62848 (time 1746136596.89 total 232.77 last 284.74) total: 64768 /unlink in 238.81 seconds: 271.21 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg330-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg330-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg330-server stop mds service on oleg330-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg330-server unloading modules via unload_modules_local on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg330-client: oleg330-client: ssh exited with exit code 2 pdsh@oleg330-client: oleg330-server: ssh exited with exit code 2 pdsh@oleg330-client: oleg330-client: ssh exited with exit code 2 pdsh@oleg330-client: oleg330-server: ssh exited with exit code 2 PASS 106 (4043s) Stopping clients: oleg330-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg330-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg330-client: oleg330-server: ssh exited with exit code 2 oleg330-server: oleg330-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions oleg330-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg330-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4162 sec ========== 17:57:56 (1746136676) === conf-sanity: start cleanup 17:57:57 (1746136677) === === conf-sanity: finish cleanup 17:57:59 (1746136679) ===