-----============= acceptance-small: conf-sanity ============----- Wed Apr 30 21:54:42 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg440-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg440-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg440-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg440-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg440-server oleg440-server: oleg440-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: detected 4 online CPUs by sysfs oleg440-server: MODOPTS_LIBCFS= oleg440-server: Force libcfs to create 2 CPU partitions oleg440-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory oleg440-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg440-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg440-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg440-server: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg440-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 21:56:17 (1746064577) Stopping clients: oleg440-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg440-client.virtnet /mnt/lustre2 (opts:-f) oleg440-server: oleg440-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: detected 4 online CPUs by sysfs oleg440-server: MODOPTS_LIBCFS= oleg440-server: Force libcfs to create 2 CPU partitions oleg440-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory oleg440-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg440-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg440-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1203 (time 1746064633.89 total 10.00 last 120.24) - open/close 2270 (time 1746064643.90 total 20.01 last 106.66) - open/close 3362 (time 1746064653.90 total 30.01 last 109.17) - open/close 4768 (time 1746064663.90 total 40.01 last 140.57) - open/close 5949 (time 1746064673.91 total 50.02 last 117.99) - open/close 6928 (time 1746064683.92 total 60.03 last 97.85) - open/close 7928 (time 1746064693.92 total 70.03 last 99.97) - open/close 8976 (time 1746064703.92 total 80.03 last 104.78) - open/close 10000 (time 1746064712.74 total 88.86 last 116.05) - open/close 11003 (time 1746064722.76 total 98.87 last 100.19) - open/close 12197 (time 1746064732.77 total 108.88 last 119.28) - open/close 13307 (time 1746064742.77 total 118.88 last 110.98) - open/close 14493 (time 1746064752.77 total 128.88 last 118.57) - open/close 15699 (time 1746064762.78 total 138.89 last 120.52) - open/close 16855 (time 1746064772.78 total 148.89 last 115.54) - open/close 17987 (time 1746064782.78 total 158.89 last 113.19) - open/close 19452 (time 1746064792.79 total 168.90 last 146.39) - open/close 20000 (time 1746064797.47 total 173.59 last 117.00) - open/close 21543 (time 1746064807.48 total 183.59 last 154.25) - open/close 23304 (time 1746064817.48 total 193.60 last 175.98) - open/close 24825 (time 1746064827.49 total 203.60 last 152.05) - open/close 26430 (time 1746064837.49 total 213.60 last 160.42) - open/close 28456 (time 1746064847.49 total 223.60 last 202.59) - open/close 29762 (time 1746064857.49 total 233.61 last 130.59) - open/close 31047 (time 1746064867.50 total 243.61 last 128.39) - open/close 32341 (time 1746064877.50 total 253.61 last 129.39) - open/close 33776 (time 1746064887.50 total 263.62 last 143.47) - open/close 35531 (time 1746064897.50 total 273.62 last 175.49) - open/close 37011 (time 1746064907.51 total 283.62 last 147.92) - open/close 38377 (time 1746064917.52 total 293.63 last 136.51) - open/close 40000 (time 1746064926.79 total 302.90 last 175.10) - open/close 41795 (time 1746064936.79 total 312.90 last 179.49) - open/close 43730 (time 1746064946.79 total 322.90 last 193.42) - open/close 45404 (time 1746064956.80 total 332.91 last 167.28) - open/close 47353 (time 1746064966.80 total 342.91 last 194.84) - open/close 49101 (time 1746064976.80 total 352.92 last 174.77) - open/close 50000 (time 1746064982.51 total 358.62 last 157.64) - open/close 51669 (time 1746064992.52 total 368.63 last 166.70) - open/close 53059 (time 1746065002.52 total 378.63 last 138.97) - open/close 54068 (time 1746065012.53 total 388.64 last 100.82) - open/close 55346 (time 1746065022.53 total 398.65 last 127.74) - open/close 56818 (time 1746065032.53 total 408.65 last 147.18) - open/close 58254 (time 1746065042.54 total 418.65 last 143.54) - open/close 59768 (time 1746065052.54 total 428.65 last 151.38) - open/close 61399 (time 1746065062.54 total 438.66 last 163.04) - open/close 62876 (time 1746065072.55 total 448.66 last 147.66) - open/close 64111 (time 1746065082.55 total 458.66 last 123.46) total: 64768 open/close in 463.16 seconds: 139.84 ops/second - /unlink 2383 (time 1746065098.49 total 10.00 last 238.21) - /unlink 4564 (time 1746065108.49 total 20.00 last 218.09) - /unlink 6929 (time 1746065118.49 total 30.01 last 236.47) - /unlink 9877 (time 1746065128.49 total 40.01 last 294.70) - /unlink 12647 (time 1746065138.50 total 50.01 last 276.93) - /unlink 15076 (time 1746065148.50 total 60.02 last 242.77) - /unlink 17520 (time 1746065158.50 total 70.02 last 244.36) - /unlink 19672 (time 1746065168.51 total 80.02 last 215.12) - /unlink 21915 (time 1746065178.51 total 90.02 last 224.28) - /unlink 23918 (time 1746065188.51 total 100.02 last 200.27) - /unlink 25874 (time 1746065198.51 total 110.03 last 195.51) - /unlink 28207 (time 1746065208.52 total 120.03 last 233.21) - /unlink 30000 (time 1746065214.24 total 125.76 last 313.07) - /unlink 32743 (time 1746065224.25 total 135.76 last 274.19) - /unlink 35178 (time 1746065234.25 total 145.77 last 243.33) - /unlink 37903 (time 1746065244.26 total 155.77 last 272.40) - /unlink 40000 (time 1746065251.24 total 162.76 last 300.23) - /unlink 43417 (time 1746065261.24 total 172.76 last 341.65) - /unlink 45941 (time 1746065271.25 total 182.76 last 252.36) - /unlink 48375 (time 1746065281.25 total 192.77 last 243.31) - /unlink 50000 (time 1746065287.04 total 198.56 last 280.67) - /unlink 52798 (time 1746065297.04 total 208.56 last 279.79) - /unlink 55071 (time 1746065307.04 total 218.56 last 227.24) - /unlink 58133 (time 1746065317.04 total 228.56 last 306.14) - /unlink 60000 (time 1746065323.81 total 235.32 last 276.12) - /unlink 63182 (time 1746065333.81 total 245.32 last 318.15) total: 64768 /unlink in 249.67 seconds: 259.42 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1741 (time 1746065364.21 total 10.00 last 174.06) - open/close 3200 (time 1746065374.21 total 20.01 last 145.80) - open/close 5006 (time 1746065384.22 total 30.01 last 180.53) - open/close 6707 (time 1746065394.22 total 40.02 last 170.05) - open/close 8111 (time 1746065404.22 total 50.02 last 140.37) - open/close 9662 (time 1746065414.23 total 60.02 last 155.03) - open/close 10000 (time 1746065416.96 total 62.75 last 123.83) - open/close 11232 (time 1746065426.96 total 72.75 last 123.18) - open/close 12418 (time 1746065436.97 total 82.76 last 118.49) - open/close 13526 (time 1746065446.97 total 92.77 last 110.72) - open/close 14786 (time 1746065456.98 total 102.77 last 125.96) - open/close 16067 (time 1746065466.98 total 112.78 last 128.04) - open/close 17279 (time 1746065476.98 total 122.78 last 121.16) - open/close 18559 (time 1746065486.99 total 132.78 last 127.99) - open/close 19900 (time 1746065496.99 total 142.79 last 134.03) - open/close 21118 (time 1746065507.00 total 152.79 last 121.73) - open/close 22662 (time 1746065517.00 total 162.80 last 154.33) - open/close 24308 (time 1746065527.00 total 172.80 last 164.57) - open/close 25723 (time 1746065537.00 total 182.80 last 141.48) - open/close 27234 (time 1746065547.01 total 192.80 last 151.06) - open/close 28949 (time 1746065557.01 total 202.80 last 171.48) - open/close 30000 (time 1746065563.73 total 209.52 last 156.41) - open/close 31827 (time 1746065573.73 total 219.52 last 182.69) - open/close 34124 (time 1746065583.73 total 229.52 last 229.65) - open/close 36204 (time 1746065593.74 total 239.53 last 207.82) - open/close 38100 (time 1746065603.74 total 249.53 last 189.58) - open/close 39775 (time 1746065613.74 total 259.54 last 167.45) - open/close 41443 (time 1746065623.74 total 269.54 last 166.75) - open/close 43435 (time 1746065633.75 total 279.54 last 199.14) - open/close 45655 (time 1746065643.75 total 289.54 last 221.96) - open/close 47713 (time 1746065653.75 total 299.55 last 205.74) - open/close 49503 (time 1746065663.75 total 309.55 last 178.97) - open/close 50000 (time 1746065667.68 total 313.48 last 126.60) - open/close 51239 (time 1746065677.68 total 323.48 last 123.87) - open/close 52394 (time 1746065687.69 total 333.48 last 115.46) - open/close 53673 (time 1746065697.69 total 343.49 last 127.80) - open/close 54781 (time 1746065707.70 total 353.49 last 110.77) - open/close 56001 (time 1746065717.70 total 363.49 last 121.96) - open/close 57316 (time 1746065727.71 total 373.50 last 131.37) - open/close 58720 (time 1746065737.71 total 383.50 last 140.40) - open/close 59866 (time 1746065747.71 total 393.51 last 114.56) - open/close 60945 (time 1746065757.72 total 403.52 last 107.81) - open/close 61789 (time 1746065767.72 total 413.52 last 84.36) - open/close 62704 (time 1746065777.75 total 423.54 last 91.30) - open/close 63545 (time 1746065787.75 total 433.55 last 84.04) - open/close 64308 (time 1746065797.76 total 443.55 last 76.26) total: 64768 open/close in 447.46 seconds: 144.75 ops/second - /unlink 2186 (time 1746065813.40 total 10.00 last 218.53) - /unlink 5467 (time 1746065823.40 total 20.00 last 328.06) - /unlink 9076 (time 1746065833.41 total 30.01 last 360.80) - /unlink 10000 (time 1746065836.16 total 32.76 last 335.68) - /unlink 12391 (time 1746065846.16 total 42.76 last 239.03) - /unlink 14361 (time 1746065856.17 total 52.77 last 196.94) - /unlink 16899 (time 1746065866.17 total 62.77 last 253.62) - /unlink 19518 (time 1746065876.18 total 72.78 last 261.78) - /unlink 20000 (time 1746065878.32 total 74.92 last 225.39) - /unlink 22871 (time 1746065888.32 total 84.92 last 287.03) - /unlink 25354 (time 1746065898.32 total 94.92 last 248.15) - /unlink 27712 (time 1746065908.33 total 104.93 last 235.74) - /unlink 29753 (time 1746065918.33 total 114.93 last 204.09) - /unlink 31867 (time 1746065928.33 total 124.93 last 211.28) - /unlink 33950 (time 1746065938.34 total 134.94 last 208.25) - /unlink 35715 (time 1746065948.34 total 144.94 last 176.47) - /unlink 37472 (time 1746065958.34 total 154.94 last 175.67) - /unlink 39287 (time 1746065968.35 total 164.95 last 181.27) - /unlink 40000 (time 1746065972.49 total 169.09 last 172.11) - /unlink 41837 (time 1746065982.50 total 179.10 last 183.59) - /unlink 43792 (time 1746065992.50 total 189.10 last 195.48) - /unlink 45869 (time 1746066002.50 total 199.10 last 207.67) - /unlink 47894 (time 1746066012.51 total 209.10 last 202.47) - /unlink 49785 (time 1746066022.51 total 219.11 last 189.07) - /unlink 52068 (time 1746066032.51 total 229.11 last 228.25) - /unlink 54208 (time 1746066042.52 total 239.12 last 213.70) - /unlink 56137 (time 1746066052.52 total 249.12 last 192.88) - /unlink 58053 (time 1746066062.53 total 259.13 last 191.44) - /unlink 59995 (time 1746066072.53 total 269.13 last 194.18) - /unlink 61997 (time 1746066082.54 total 279.14 last 200.14) - /unlink 63932 (time 1746066092.54 total 289.14 last 193.45) total: 64768 /unlink in 294.08 seconds: 220.24 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1081 (time 1746066126.83 total 10.00 last 108.06) - open/close 2146 (time 1746066136.84 total 20.01 last 106.47) - open/close 3243 (time 1746066146.85 total 30.02 last 109.60) - open/close 4324 (time 1746066156.85 total 40.02 last 108.05) - open/close 5444 (time 1746066166.86 total 50.03 last 111.90) - open/close 6501 (time 1746066176.87 total 60.04 last 105.55) - open/close 7544 (time 1746066186.88 total 70.04 last 104.28) - open/close 8673 (time 1746066196.88 total 80.05 last 112.80) - open/close 9919 (time 1746066206.89 total 90.05 last 124.58) - open/close 11147 (time 1746066216.89 total 100.06 last 122.71) - open/close 12453 (time 1746066226.89 total 110.06 last 130.58) - open/close 13690 (time 1746066236.90 total 120.07 last 123.67) - open/close 14878 (time 1746066246.90 total 130.07 last 118.74) - open/close 16186 (time 1746066256.91 total 140.08 last 130.73) - open/close 17346 (time 1746066266.91 total 150.08 last 115.94) - open/close 18610 (time 1746066276.92 total 160.09 last 126.30) - open/close 19792 (time 1746066286.93 total 170.10 last 118.07) - open/close 21121 (time 1746066296.93 total 180.10 last 132.89) - open/close 22268 (time 1746066306.93 total 190.10 last 114.68) - open/close 23552 (time 1746066316.94 total 200.11 last 128.37) - open/close 24697 (time 1746066326.94 total 210.11 last 114.46) - open/close 25808 (time 1746066336.94 total 220.11 last 111.05) - open/close 27024 (time 1746066346.94 total 230.11 last 121.60) - open/close 28214 (time 1746066356.95 total 240.12 last 118.96) - open/close 29404 (time 1746066366.95 total 250.12 last 118.97) - open/close 30000 (time 1746066371.52 total 254.69 last 130.37) - open/close 31241 (time 1746066381.52 total 264.69 last 124.08) - open/close 32609 (time 1746066391.53 total 274.70 last 136.75) - open/close 33799 (time 1746066401.53 total 284.70 last 118.95) - open/close 35127 (time 1746066411.53 total 294.70 last 132.76) - open/close 36383 (time 1746066421.54 total 304.71 last 125.56) - open/close 37610 (time 1746066431.54 total 314.71 last 122.69) - open/close 38887 (time 1746066441.54 total 324.71 last 127.63) - open/close 40000 (time 1746066448.45 total 331.62 last 161.07) - open/close 41663 (time 1746066458.45 total 341.62 last 166.25) - open/close 43171 (time 1746066468.46 total 351.63 last 150.73) - open/close 44454 (time 1746066478.46 total 361.63 last 128.27) - open/close 45728 (time 1746066488.46 total 371.63 last 127.39) - open/close 46977 (time 1746066498.47 total 381.64 last 124.85) - open/close 48325 (time 1746066508.47 total 391.64 last 134.76) - open/close 49770 (time 1746066518.47 total 401.64 last 144.44) - open/close 50974 (time 1746066528.48 total 411.65 last 120.35) - open/close 52420 (time 1746066538.48 total 421.65 last 144.59) - open/close 53839 (time 1746066548.48 total 431.65 last 141.86) - open/close 55146 (time 1746066558.48 total 441.65 last 130.69) - open/close 56567 (time 1746066568.48 total 451.65 last 142.06) - open/close 58053 (time 1746066578.49 total 461.65 last 148.60) - open/close 59388 (time 1746066588.49 total 471.66 last 133.48) - open/close 60000 (time 1746066592.68 total 475.85 last 146.05) - open/close 61476 (time 1746066602.68 total 485.85 last 147.60) - open/close 62801 (time 1746066612.68 total 495.85 last 132.43) - open/close 64073 (time 1746066622.69 total 505.86 last 127.08) total: 64768 open/close in 511.43 seconds: 126.64 ops/second - /unlink 2084 (time 1746066640.01 total 10.00 last 208.32) - /unlink 4434 (time 1746066650.01 total 20.01 last 234.95) - /unlink 6969 (time 1746066660.01 total 30.01 last 253.45) - /unlink 9274 (time 1746066670.02 total 40.01 last 230.47) - /unlink 10000 (time 1746066673.04 total 43.03 last 240.29) - /unlink 12105 (time 1746066683.04 total 53.03 last 210.41) - /unlink 14215 (time 1746066693.04 total 63.04 last 210.93) - /unlink 16242 (time 1746066703.05 total 73.04 last 202.62) - /unlink 18401 (time 1746066713.05 total 83.05 last 215.77) - /unlink 20000 (time 1746066721.45 total 91.44 last 190.53) - /unlink 22028 (time 1746066731.45 total 101.45 last 202.65) - /unlink 24130 (time 1746066741.45 total 111.45 last 210.18) - /unlink 26182 (time 1746066751.46 total 121.45 last 205.12) - /unlink 28630 (time 1746066761.46 total 131.45 last 244.80) - /unlink 30000 (time 1746066767.39 total 137.38 last 231.08) - /unlink 32295 (time 1746066777.39 total 147.38 last 229.45) - /unlink 34569 (time 1746066787.39 total 157.38 last 227.39) - /unlink 36704 (time 1746066797.39 total 167.38 last 213.48) - /unlink 39136 (time 1746066807.39 total 177.39 last 243.18) - /unlink 40000 (time 1746066810.88 total 180.88 last 247.52) - /unlink 43034 (time 1746066820.88 total 190.88 last 303.39) - /unlink 45624 (time 1746066830.88 total 200.88 last 258.97) - /unlink 48337 (time 1746066840.89 total 210.88 last 271.21) - /unlink 50000 (time 1746066847.35 total 217.35 last 257.18) - /unlink 52464 (time 1746066857.35 total 227.35 last 246.39) - /unlink 54561 (time 1746066867.35 total 237.35 last 209.69) - /unlink 56719 (time 1746066877.36 total 247.35 last 215.77) - /unlink 59059 (time 1746066887.36 total 257.35 last 233.95) - /unlink 60000 (time 1746066891.80 total 261.79 last 211.98) - /unlink 61938 (time 1746066901.80 total 271.79 last 193.80) - /unlink 63713 (time 1746066911.80 total 281.79 last 177.47) total: 64768 /unlink in 286.80 seconds: 225.83 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1237 (time 1746066947.92 total 10.01 last 123.60) - open/close 2550 (time 1746066957.93 total 20.02 last 131.17) - open/close 3940 (time 1746066967.93 total 30.02 last 138.97) - open/close 5441 (time 1746066977.94 total 40.03 last 150.01) - open/close 6784 (time 1746066987.94 total 50.03 last 134.27) - open/close 8223 (time 1746066997.95 total 60.03 last 143.86) - open/close 9597 (time 1746067007.95 total 70.04 last 137.29) - open/close 10000 (time 1746067011.06 total 73.15 last 129.69) - open/close 11204 (time 1746067021.07 total 83.15 last 120.34) - open/close 12356 (time 1746067031.07 total 93.15 last 115.19) - open/close 13603 (time 1746067041.07 total 103.16 last 124.65) - open/close 14926 (time 1746067051.08 total 113.16 last 132.21) - open/close 16234 (time 1746067061.08 total 123.17 last 130.72) - open/close 17545 (time 1746067071.09 total 133.17 last 131.08) - open/close 18785 (time 1746067081.09 total 143.17 last 123.94) - open/close 20000 (time 1746067090.24 total 152.32 last 132.80) - open/close 21469 (time 1746067100.24 total 162.32 last 146.90) - open/close 22879 (time 1746067110.24 total 172.33 last 140.98) - open/close 24272 (time 1746067120.24 total 182.33 last 139.27) - open/close 25796 (time 1746067130.25 total 192.33 last 152.34) - open/close 27490 (time 1746067140.25 total 202.33 last 169.39) - open/close 29090 (time 1746067150.25 total 212.33 last 159.97) - open/close 30000 (time 1746067156.02 total 218.10 last 157.74) - open/close 32033 (time 1746067166.02 total 228.10 last 203.26) - open/close 34085 (time 1746067176.02 total 238.11 last 205.13) - open/close 35753 (time 1746067186.02 total 248.11 last 166.78) - open/close 37314 (time 1746067196.03 total 258.11 last 156.09) - open/close 38882 (time 1746067206.03 total 268.11 last 156.79) - open/close 40000 (time 1746067212.37 total 274.46 last 176.10) - open/close 41675 (time 1746067222.38 total 284.46 last 167.46) - open/close 43244 (time 1746067232.38 total 294.47 last 156.83) - open/close 45052 (time 1746067242.38 total 304.47 last 180.77) - open/close 47102 (time 1746067252.39 total 314.47 last 204.95) - open/close 48816 (time 1746067262.39 total 324.47 last 171.34) - open/close 50000 (time 1746067269.58 total 331.66 last 164.76) - open/close 51418 (time 1746067279.58 total 341.66 last 141.76) - open/close 52980 (time 1746067289.58 total 351.66 last 156.20) - open/close 54456 (time 1746067299.58 total 361.67 last 147.55) - open/close 56148 (time 1746067309.59 total 371.67 last 169.11) - open/close 57830 (time 1746067319.59 total 381.68 last 168.10) - open/close 59323 (time 1746067329.60 total 391.68 last 149.23) - open/close 60000 (time 1746067332.98 total 395.06 last 200.44) - open/close 61749 (time 1746067342.98 total 405.06 last 174.85) - open/close 63936 (time 1746067352.98 total 415.07 last 218.67) total: 64768 open/close in 419.25 seconds: 154.49 ops/second - /unlink 3378 (time 1746067368.19 total 10.00 last 337.64) - /unlink 6671 (time 1746067378.20 total 20.01 last 329.13) - /unlink 9662 (time 1746067388.20 total 30.01 last 299.02) - /unlink 12447 (time 1746067398.20 total 40.02 last 278.39) - /unlink 15633 (time 1746067408.21 total 50.02 last 318.56) - /unlink 19282 (time 1746067418.21 total 60.02 last 364.90) - /unlink 20000 (time 1746067420.80 total 62.61 last 276.81) - /unlink 23094 (time 1746067430.80 total 72.61 last 309.33) - /unlink 25202 (time 1746067440.81 total 82.62 last 210.70) - /unlink 27353 (time 1746067450.81 total 92.62 last 215.07) - /unlink 29837 (time 1746067460.81 total 102.62 last 248.36) - /unlink 32016 (time 1746067470.81 total 112.62 last 217.85) - /unlink 34503 (time 1746067480.81 total 122.63 last 248.64) - /unlink 36970 (time 1746067490.81 total 132.63 last 246.69) - /unlink 39414 (time 1746067500.82 total 142.63 last 244.39) - /unlink 40000 (time 1746067502.95 total 144.76 last 274.50) - /unlink 42862 (time 1746067512.95 total 154.76 last 286.14) - /unlink 45897 (time 1746067522.95 total 164.76 last 303.46) - /unlink 48269 (time 1746067532.95 total 174.77 last 237.18) - /unlink 50000 (time 1746067539.47 total 181.28 last 265.62) - /unlink 52496 (time 1746067549.47 total 191.28 last 249.57) - /unlink 54722 (time 1746067559.48 total 201.29 last 222.51) - /unlink 57015 (time 1746067569.48 total 211.29 last 229.30) - /unlink 59521 (time 1746067579.48 total 221.29 last 250.55) - /unlink 61968 (time 1746067589.48 total 231.29 last 244.67) total: 64768 /unlink in 240.69 seconds: 269.10 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1397 (time 1746067628.49 total 10.00 last 139.69) - open/close 2991 (time 1746067638.49 total 20.01 last 159.33) - open/close 4779 (time 1746067648.49 total 30.01 last 178.79) - open/close 6448 (time 1746067658.51 total 40.02 last 166.70) - open/close 8130 (time 1746067668.51 total 50.02 last 168.14) - open/close 10000 (time 1746067677.29 total 58.80 last 213.01) - open/close 12066 (time 1746067687.29 total 68.80 last 206.56) - open/close 13917 (time 1746067697.29 total 78.80 last 185.06) - open/close 15634 (time 1746067707.30 total 88.81 last 171.66) - open/close 17155 (time 1746067717.30 total 98.81 last 152.03) - open/close 18821 (time 1746067727.31 total 108.82 last 166.51) - open/close 20000 (time 1746067734.48 total 115.99 last 164.27) - open/close 21556 (time 1746067744.49 total 126.00 last 155.53) - open/close 22975 (time 1746067754.49 total 136.00 last 141.87) - open/close 24353 (time 1746067764.49 total 146.00 last 137.78) - open/close 25720 (time 1746067774.49 total 156.00 last 136.69) - open/close 27143 (time 1746067784.49 total 166.00 last 142.28) - open/close 28604 (time 1746067794.50 total 176.01 last 146.02) - open/close 30000 (time 1746067802.24 total 183.75 last 180.34) - open/close 31629 (time 1746067812.25 total 193.76 last 162.78) - open/close 33185 (time 1746067822.25 total 203.76 last 155.53) - open/close 34682 (time 1746067832.26 total 213.77 last 149.51) - open/close 36013 (time 1746067842.27 total 223.78 last 133.04) - open/close 37319 (time 1746067852.27 total 233.78 last 130.55) - open/close 38589 (time 1746067862.28 total 243.79 last 126.89) - open/close 40000 (time 1746067871.90 total 253.42 last 146.61) - open/close 41575 (time 1746067881.91 total 263.42 last 157.42) - open/close 43069 (time 1746067891.91 total 273.43 last 149.33) - open/close 44529 (time 1746067901.92 total 283.43 last 145.90) - open/close 45959 (time 1746067911.93 total 293.44 last 142.92) - open/close 47493 (time 1746067921.93 total 303.44 last 153.39) - open/close 49031 (time 1746067931.93 total 313.44 last 153.80) - open/close 50000 (time 1746067937.84 total 319.35 last 163.82) - open/close 51460 (time 1746067947.85 total 329.36 last 145.94) - open/close 53206 (time 1746067957.85 total 339.36 last 174.56) - open/close 54857 (time 1746067967.85 total 349.36 last 165.03) - open/close 56375 (time 1746067977.86 total 359.37 last 151.66) - open/close 58071 (time 1746067987.86 total 369.38 last 169.57) - open/close 59860 (time 1746067997.87 total 379.38 last 178.82) - open/close 61626 (time 1746068007.87 total 389.38 last 176.57) - open/close 63496 (time 1746068017.88 total 399.39 last 186.86) total: 64768 open/close in 406.99 seconds: 159.14 ops/second - /unlink 2886 (time 1746068036.82 total 10.00 last 288.59) - /unlink 5585 (time 1746068046.82 total 20.00 last 269.90) - /unlink 8678 (time 1746068056.82 total 30.00 last 309.20) - /unlink 10000 (time 1746068060.58 total 33.76 last 351.84) - /unlink 13162 (time 1746068070.58 total 43.76 last 316.15) - /unlink 16065 (time 1746068080.58 total 53.76 last 290.26) - /unlink 18877 (time 1746068090.58 total 63.77 last 281.15) - /unlink 20000 (time 1746068096.18 total 69.37 last 200.54) - /unlink 22619 (time 1746068106.18 total 79.37 last 261.85) - /unlink 25163 (time 1746068116.19 total 89.37 last 254.40) - /unlink 27721 (time 1746068126.19 total 99.37 last 255.64) - /unlink 30000 (time 1746068134.37 total 107.55 last 278.76) - /unlink 32681 (time 1746068144.37 total 117.56 last 267.91) - /unlink 35602 (time 1746068154.37 total 127.56 last 292.09) - /unlink 39222 (time 1746068164.38 total 137.56 last 361.85) - /unlink 40000 (time 1746068167.04 total 140.22 last 292.53) - /unlink 42642 (time 1746068177.04 total 150.22 last 264.20) - /unlink 45102 (time 1746068187.04 total 160.22 last 245.97) - /unlink 47695 (time 1746068197.04 total 170.22 last 259.24) - /unlink 50000 (time 1746068206.75 total 179.93 last 237.53) - /unlink 51939 (time 1746068216.75 total 189.93 last 193.90) - /unlink 54084 (time 1746068226.75 total 199.93 last 214.44) - /unlink 56257 (time 1746068236.75 total 209.94 last 217.18) - /unlink 58512 (time 1746068246.76 total 219.94 last 225.42) - /unlink 60000 (time 1746068252.22 total 225.40 last 272.41) - /unlink 62929 (time 1746068262.22 total 235.40 last 292.83) total: 64768 /unlink in 242.02 seconds: 267.61 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server unloading modules via unload_modules_local on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 PASS 106 (3720s) Stopping clients: oleg440-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg440-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 oleg440-server: oleg440-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: detected 4 online CPUs by sysfs oleg440-server: MODOPTS_LIBCFS= oleg440-server: Force libcfs to create 2 CPU partitions oleg440-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg440-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3840 sec ========== 22:58:42 (1746068322) === conf-sanity: start cleanup 22:58:44 (1746068324) === === conf-sanity: finish cleanup 22:58:45 (1746068325) ===