-----============= acceptance-small: conf-sanity ============----- Fri Dec 20 01:22:07 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg351-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 110 Stopping clients: oleg351-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg351-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg351-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg351-server oleg351-server: oleg351-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: detected 4 online CPUs by sysfs oleg351-server: MODOPTS_LIBCFS= oleg351-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg351-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg351-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg351-server: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg351-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 01:23:30 (1734675810) Stopping clients: oleg351-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg351-client.virtnet /mnt/lustre2 (opts:-f) oleg351-server: oleg351-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: detected 4 online CPUs by sysfs oleg351-server: MODOPTS_LIBCFS= oleg351-server: Force libcfs to create 2 CPU partitions oleg351-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg351-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg351-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg351-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1133 (time 1734675867.91 total 10.00 last 113.27) - open/close 2116 (time 1734675877.92 total 20.01 last 98.19) - open/close 3287 (time 1734675887.93 total 30.02 last 117.03) - open/close 4437 (time 1734675897.93 total 40.03 last 114.92) - open/close 5737 (time 1734675907.94 total 50.03 last 129.95) - open/close 6975 (time 1734675917.95 total 60.04 last 123.64) - open/close 8156 (time 1734675927.95 total 70.05 last 118.07) - open/close 9328 (time 1734675937.96 total 80.05 last 117.14) - open/close 10000 (time 1734675943.77 total 85.86 last 115.68) - open/close 11065 (time 1734675953.77 total 95.86 last 106.46) - open/close 12190 (time 1734675963.77 total 105.87 last 112.49) - open/close 13386 (time 1734675973.77 total 115.87 last 119.56) - open/close 14530 (time 1734675983.78 total 125.88 last 114.31) - open/close 15655 (time 1734675993.79 total 135.88 last 112.42) - open/close 16824 (time 1734676003.79 total 145.89 last 116.89) - open/close 18030 (time 1734676013.79 total 155.89 last 120.59) - open/close 19540 (time 1734676023.80 total 165.89 last 150.91) - open/close 20000 (time 1734676026.84 total 168.93 last 151.36) - open/close 21665 (time 1734676036.84 total 178.93 last 166.47) - open/close 23127 (time 1734676046.84 total 188.94 last 146.12) - open/close 24889 (time 1734676056.85 total 198.95 last 176.08) - open/close 26429 (time 1734676066.86 total 208.95 last 153.92) - open/close 27704 (time 1734676076.86 total 218.95 last 127.50) - open/close 28968 (time 1734676086.86 total 228.95 last 126.38) - open/close 30000 (time 1734676094.38 total 236.48 last 137.19) - open/close 31230 (time 1734676104.39 total 246.48 last 122.93) - open/close 32697 (time 1734676114.39 total 256.49 last 146.62) - open/close 34153 (time 1734676124.40 total 266.50 last 145.45) - open/close 35583 (time 1734676134.40 total 276.50 last 142.97) - open/close 37100 (time 1734676144.41 total 286.50 last 151.68) - open/close 38718 (time 1734676154.42 total 296.51 last 161.59) - open/close 40000 (time 1734676161.69 total 303.79 last 176.24) - open/close 41374 (time 1734676171.70 total 313.79 last 137.33) - open/close 42819 (time 1734676181.70 total 323.80 last 144.46) - open/close 44139 (time 1734676191.72 total 333.81 last 131.80) - open/close 45389 (time 1734676201.72 total 343.82 last 124.94) - open/close 46861 (time 1734676211.73 total 353.82 last 147.09) - open/close 48030 (time 1734676221.74 total 363.83 last 116.82) - open/close 49269 (time 1734676231.74 total 373.84 last 123.78) - open/close 50000 (time 1734676239.04 total 381.13 last 100.20) - open/close 51058 (time 1734676249.05 total 391.15 last 105.65) - open/close 52358 (time 1734676259.06 total 401.15 last 129.98) - open/close 53909 (time 1734676269.06 total 411.16 last 154.97) - open/close 55154 (time 1734676279.07 total 421.16 last 124.44) - open/close 56265 (time 1734676289.07 total 431.17 last 111.06) - open/close 57353 (time 1734676299.08 total 441.18 last 108.70) - open/close 58553 (time 1734676309.08 total 451.18 last 119.97) - open/close 59784 (time 1734676319.09 total 461.18 last 123.08) - open/close 60949 (time 1734676329.09 total 471.18 last 116.45) - open/close 62227 (time 1734676339.10 total 481.19 last 127.73) - open/close 63619 (time 1734676349.10 total 491.19 last 139.19) total: 64768 open/close in 498.73 seconds: 129.86 ops/second - /unlink 2774 (time 1734676368.06 total 10.00 last 277.36) - /unlink 5309 (time 1734676378.06 total 20.00 last 253.43) - /unlink 7531 (time 1734676388.07 total 30.01 last 222.11) - /unlink 10000 (time 1734676397.27 total 39.21 last 268.20) - /unlink 12264 (time 1734676407.28 total 49.22 last 226.30) - /unlink 14662 (time 1734676417.28 total 59.22 last 239.73) - /unlink 16645 (time 1734676427.28 total 69.22 last 198.27) - /unlink 18886 (time 1734676437.28 total 79.22 last 224.10) - /unlink 20000 (time 1734676442.58 total 84.52 last 210.25) - /unlink 22199 (time 1734676452.58 total 94.52 last 219.87) - /unlink 24688 (time 1734676462.58 total 104.52 last 248.88) - /unlink 27561 (time 1734676472.58 total 114.52 last 287.27) - /unlink 30000 (time 1734676479.93 total 121.87 last 332.16) - /unlink 33104 (time 1734676489.93 total 131.87 last 310.30) - /unlink 35818 (time 1734676499.93 total 141.87 last 271.28) - /unlink 38300 (time 1734676509.93 total 151.88 last 248.19) - /unlink 40000 (time 1734676517.14 total 159.08 last 235.88) - /unlink 42089 (time 1734676527.14 total 169.08 last 208.88) - /unlink 44810 (time 1734676537.14 total 179.08 last 272.08) - /unlink 47274 (time 1734676547.15 total 189.09 last 246.23) - /unlink 49889 (time 1734676557.15 total 199.09 last 261.45) - /unlink 52163 (time 1734676567.15 total 209.10 last 227.33) - /unlink 54269 (time 1734676577.16 total 219.10 last 210.55) - /unlink 56259 (time 1734676587.16 total 229.10 last 198.98) - /unlink 58494 (time 1734676597.16 total 239.10 last 223.45) - /unlink 60000 (time 1734676603.27 total 245.21 last 246.56) - /unlink 62435 (time 1734676613.27 total 255.21 last 243.49) total: 64768 /unlink in 264.20 seconds: 245.14 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2017 (time 1734676649.14 total 10.00 last 201.66) - open/close 3793 (time 1734676659.15 total 20.00 last 177.55) - open/close 5626 (time 1734676669.15 total 30.01 last 183.21) - open/close 7289 (time 1734676679.16 total 40.01 last 166.22) - open/close 8813 (time 1734676689.16 total 50.02 last 152.36) - open/close 10000 (time 1734676697.83 total 58.69 last 136.86) - open/close 11343 (time 1734676707.84 total 68.70 last 134.22) - open/close 12892 (time 1734676717.84 total 78.70 last 154.84) - open/close 14312 (time 1734676727.84 total 88.70 last 142.00) - open/close 15842 (time 1734676737.85 total 98.71 last 152.94) - open/close 17385 (time 1734676747.85 total 108.71 last 154.25) - open/close 18805 (time 1734676757.86 total 118.72 last 141.90) - open/close 20000 (time 1734676765.84 total 126.69 last 149.78) - open/close 21971 (time 1734676775.84 total 136.70 last 196.99) - open/close 23639 (time 1734676785.85 total 146.70 last 166.74) - open/close 25521 (time 1734676795.85 total 156.71 last 188.08) - open/close 26889 (time 1734676805.85 total 166.71 last 136.76) - open/close 28246 (time 1734676815.86 total 176.72 last 135.66) - open/close 29744 (time 1734676825.87 total 186.72 last 149.67) - open/close 31192 (time 1734676835.87 total 196.73 last 144.78) - open/close 32547 (time 1734676845.87 total 206.73 last 135.49) - open/close 33989 (time 1734676855.87 total 216.73 last 144.20) - open/close 35366 (time 1734676865.87 total 226.73 last 137.67) - open/close 36784 (time 1734676875.88 total 236.74 last 141.67) - open/close 38066 (time 1734676885.88 total 246.74 last 128.20) - open/close 39257 (time 1734676895.88 total 256.74 last 119.10) - open/close 40000 (time 1734676902.66 total 263.52 last 109.60) - open/close 41229 (time 1734676912.66 total 273.52 last 122.86) - open/close 42387 (time 1734676922.67 total 283.53 last 115.70) - open/close 43560 (time 1734676932.68 total 293.54 last 117.19) - open/close 44741 (time 1734676942.68 total 303.54 last 118.07) - open/close 45976 (time 1734676952.69 total 313.55 last 123.42) - open/close 47428 (time 1734676962.69 total 323.55 last 145.18) - open/close 48753 (time 1734676972.70 total 333.56 last 132.39) - open/close 50000 (time 1734676982.67 total 343.53 last 125.03) - open/close 51295 (time 1734676992.68 total 353.54 last 129.44) - open/close 52738 (time 1734677002.68 total 363.54 last 144.22) - open/close 54249 (time 1734677012.69 total 373.55 last 151.02) - open/close 55771 (time 1734677022.69 total 383.55 last 152.19) - open/close 57235 (time 1734677032.69 total 393.55 last 146.33) - open/close 58533 (time 1734677042.70 total 403.56 last 129.77) - open/close 59972 (time 1734677052.70 total 413.56 last 143.88) - open/close 61659 (time 1734677062.70 total 423.56 last 168.70) - open/close 63012 (time 1734677072.70 total 433.56 last 135.29) - open/close 64481 (time 1734677082.71 total 443.56 last 146.82) total: 64768 open/close in 445.51 seconds: 145.38 ops/second - /unlink 2351 (time 1734677096.23 total 10.00 last 235.03) - /unlink 5041 (time 1734677106.23 total 20.00 last 268.96) - /unlink 7628 (time 1734677116.24 total 30.01 last 258.57) - /unlink 10000 (time 1734677125.06 total 38.83 last 268.85) - /unlink 13151 (time 1734677135.06 total 48.83 last 315.10) - /unlink 15340 (time 1734677145.06 total 58.83 last 218.87) - /unlink 17307 (time 1734677155.06 total 68.83 last 196.69) - /unlink 19595 (time 1734677165.06 total 78.84 last 228.70) - /unlink 22424 (time 1734677175.07 total 88.84 last 282.89) - /unlink 24780 (time 1734677185.07 total 98.84 last 235.54) - /unlink 27970 (time 1734677195.07 total 108.85 last 318.82) - /unlink 30000 (time 1734677202.07 total 115.85 last 289.94) - /unlink 32503 (time 1734677212.08 total 125.85 last 250.25) - /unlink 35118 (time 1734677222.08 total 135.85 last 261.40) - /unlink 38432 (time 1734677232.08 total 145.85 last 331.40) - /unlink 40000 (time 1734677237.85 total 151.63 last 271.65) - /unlink 42951 (time 1734677247.86 total 161.63 last 295.03) - /unlink 45927 (time 1734677257.86 total 171.63 last 297.55) - /unlink 48380 (time 1734677267.86 total 181.63 last 245.28) - /unlink 50000 (time 1734677273.93 total 187.71 last 266.66) - /unlink 53441 (time 1734677283.93 total 197.71 last 344.10) - /unlink 56916 (time 1734677293.93 total 207.71 last 347.46) - /unlink 60000 (time 1734677301.75 total 215.52 last 394.58) - /unlink 63815 (time 1734677311.75 total 225.52 last 381.49) total: 64768 /unlink in 228.21 seconds: 283.81 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1997 (time 1734677344.39 total 10.00 last 199.68) - open/close 3501 (time 1734677354.40 total 20.00 last 150.37) - open/close 5386 (time 1734677364.40 total 30.01 last 188.40) - open/close 7813 (time 1734677374.40 total 40.01 last 242.67) - open/close 10000 (time 1734677384.25 total 49.86 last 222.01) - open/close 12110 (time 1734677394.26 total 59.86 last 210.92) - open/close 14164 (time 1734677404.26 total 69.86 last 205.39) - open/close 16529 (time 1734677414.26 total 79.86 last 236.48) - open/close 18503 (time 1734677424.26 total 89.87 last 197.33) - open/close 20000 (time 1734677432.09 total 97.70 last 191.24) - open/close 22107 (time 1734677442.09 total 107.70 last 210.65) - open/close 24030 (time 1734677452.09 total 117.70 last 192.29) - open/close 26000 (time 1734677462.09 total 127.70 last 196.99) - open/close 27841 (time 1734677472.09 total 137.70 last 184.04) - open/close 29613 (time 1734677482.10 total 147.70 last 177.18) - open/close 30000 (time 1734677484.17 total 149.78 last 186.82) - open/close 31557 (time 1734677494.17 total 159.78 last 155.66) - open/close 33278 (time 1734677504.18 total 169.78 last 172.02) - open/close 34855 (time 1734677514.18 total 179.79 last 157.63) - open/close 36215 (time 1734677524.18 total 189.79 last 135.98) - open/close 37912 (time 1734677534.18 total 199.79 last 169.68) - open/close 39910 (time 1734677544.18 total 209.79 last 199.76) - open/close 41405 (time 1734677554.18 total 219.79 last 149.50) - open/close 43200 (time 1734677564.18 total 229.79 last 179.50) - open/close 44824 (time 1734677574.19 total 239.80 last 162.35) - open/close 46435 (time 1734677584.19 total 249.80 last 161.09) - open/close 48161 (time 1734677594.19 total 259.80 last 172.54) - open/close 49791 (time 1734677604.19 total 269.80 last 162.98) - open/close 51022 (time 1734677614.20 total 279.81 last 123.01) - open/close 52295 (time 1734677624.20 total 289.81 last 127.27) - open/close 53507 (time 1734677634.21 total 299.82 last 121.12) - open/close 54781 (time 1734677644.21 total 309.82 last 127.32) - open/close 56242 (time 1734677654.22 total 319.83 last 146.03) - open/close 57530 (time 1734677664.23 total 329.83 last 128.73) - open/close 58739 (time 1734677674.23 total 339.84 last 120.80) - open/close 59926 (time 1734677684.24 total 349.85 last 118.65) - open/close 61323 (time 1734677694.24 total 359.85 last 139.63) - open/close 62953 (time 1734677704.25 total 369.86 last 162.90) - open/close 64434 (time 1734677714.25 total 379.86 last 148.07) total: 64768 open/close in 382.00 seconds: 169.55 ops/second - /unlink 2923 (time 1734677727.86 total 10.00 last 292.28) - /unlink 6085 (time 1734677737.87 total 20.00 last 316.14) - /unlink 9493 (time 1734677747.87 total 30.00 last 340.75) - /unlink 12916 (time 1734677757.87 total 40.01 last 342.26) - /unlink 16257 (time 1734677767.87 total 50.01 last 333.99) - /unlink 19396 (time 1734677777.87 total 60.01 last 313.82) - /unlink 23207 (time 1734677787.88 total 70.01 last 381.01) - /unlink 25995 (time 1734677797.88 total 80.02 last 278.75) - /unlink 28122 (time 1734677807.88 total 90.02 last 212.60) - /unlink 30000 (time 1734677816.41 total 98.55 last 220.19) - /unlink 32257 (time 1734677826.41 total 108.55 last 225.65) - /unlink 34124 (time 1734677836.42 total 118.55 last 186.67) - /unlink 36218 (time 1734677846.42 total 128.55 last 209.40) - /unlink 38563 (time 1734677856.42 total 138.56 last 234.44) - /unlink 40000 (time 1734677861.45 total 143.59 last 285.56) - /unlink 42472 (time 1734677871.45 total 153.59 last 247.17) - /unlink 44856 (time 1734677881.45 total 163.59 last 238.36) - /unlink 47793 (time 1734677891.46 total 173.59 last 293.58) - /unlink 50000 (time 1734677898.59 total 180.73 last 309.29) - /unlink 52241 (time 1734677908.60 total 190.73 last 224.06) - /unlink 54184 (time 1734677918.60 total 200.74 last 194.24) - /unlink 56753 (time 1734677928.60 total 210.74 last 256.90) - /unlink 59285 (time 1734677938.60 total 220.74 last 253.16) - /unlink 60000 (time 1734677940.99 total 223.13 last 298.88) - /unlink 63103 (time 1734677951.00 total 233.14 last 310.02) total: 64768 /unlink in 237.77 seconds: 272.39 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1667 (time 1734677986.30 total 10.00 last 166.64) - open/close 3441 (time 1734677996.30 total 20.01 last 177.24) - open/close 4845 (time 1734678006.31 total 30.01 last 140.38) - open/close 6230 (time 1734678016.31 total 40.02 last 138.44) - open/close 7929 (time 1734678026.31 total 50.02 last 169.86) - open/close 9608 (time 1734678036.32 total 60.03 last 167.77) - open/close 10000 (time 1734678038.62 total 62.33 last 170.29) - open/close 11275 (time 1734678048.63 total 72.33 last 127.43) - open/close 12466 (time 1734678058.63 total 82.34 last 119.01) - open/close 13670 (time 1734678068.64 total 92.35 last 120.35) - open/close 15116 (time 1734678078.64 total 102.35 last 144.53) - open/close 16755 (time 1734678088.64 total 112.35 last 163.89) - open/close 18246 (time 1734678098.64 total 122.35 last 149.10) - open/close 19494 (time 1734678108.65 total 132.35 last 124.79) - open/close 20000 (time 1734678112.58 total 136.29 last 128.53) - open/close 21279 (time 1734678122.59 total 146.29 last 127.86) - open/close 22473 (time 1734678132.59 total 156.30 last 119.32) - open/close 23758 (time 1734678142.60 total 166.30 last 128.45) - open/close 25124 (time 1734678152.60 total 176.31 last 136.56) - open/close 26432 (time 1734678162.60 total 186.31 last 130.78) - open/close 27749 (time 1734678172.61 total 196.31 last 131.65) - open/close 29051 (time 1734678182.61 total 206.32 last 130.12) - open/close 30000 (time 1734678188.82 total 212.53 last 152.76) - open/close 31262 (time 1734678198.84 total 222.54 last 126.05) - open/close 32467 (time 1734678208.84 total 232.55 last 120.45) - open/close 33647 (time 1734678218.84 total 242.55 last 118.00) - open/close 34910 (time 1734678228.84 total 252.55 last 126.27) - open/close 36261 (time 1734678238.84 total 262.55 last 135.06) - open/close 37552 (time 1734678248.85 total 272.55 last 129.08) - open/close 38784 (time 1734678258.85 total 282.56 last 123.16) - open/close 40000 (time 1734678267.40 total 291.11 last 142.15) - open/close 41425 (time 1734678277.41 total 301.12 last 142.42) - open/close 42818 (time 1734678287.41 total 311.12 last 139.29) - open/close 44472 (time 1734678297.41 total 321.12 last 165.38) - open/close 45941 (time 1734678307.42 total 331.12 last 146.84) - open/close 47404 (time 1734678317.42 total 341.13 last 146.21) - open/close 48786 (time 1734678327.42 total 351.13 last 138.17) - open/close 50000 (time 1734678337.05 total 360.75 last 126.15) - open/close 51479 (time 1734678347.05 total 370.76 last 147.81) - open/close 52812 (time 1734678357.06 total 380.76 last 133.27) - open/close 54173 (time 1734678367.06 total 390.77 last 136.03) - open/close 55376 (time 1734678377.08 total 400.78 last 120.12) - open/close 56692 (time 1734678387.08 total 410.78 last 131.59) - open/close 58174 (time 1734678397.08 total 420.79 last 148.18) - open/close 59480 (time 1734678407.08 total 430.79 last 130.53) - open/close 60000 (time 1734678410.88 total 434.59 last 136.97) - open/close 61243 (time 1734678420.88 total 444.59 last 124.26) - open/close 62481 (time 1734678430.88 total 454.59 last 123.78) - open/close 63660 (time 1734678440.89 total 464.60 last 117.79) total: 64768 open/close in 473.65 seconds: 136.74 ops/second - /unlink 2132 (time 1734678461.69 total 10.00 last 213.14) - /unlink 4748 (time 1734678471.69 total 20.00 last 261.55) - /unlink 7126 (time 1734678481.69 total 30.01 last 237.79) - /unlink 9445 (time 1734678491.69 total 40.01 last 231.84) - /unlink 12696 (time 1734678501.69 total 50.01 last 325.03) - /unlink 16166 (time 1734678511.70 total 60.02 last 346.80) - /unlink 19006 (time 1734678521.70 total 70.02 last 283.99) - /unlink 20000 (time 1734678525.54 total 73.86 last 258.92) - /unlink 22372 (time 1734678535.55 total 83.86 last 237.04) - /unlink 24420 (time 1734678545.55 total 93.86 last 204.74) - /unlink 26874 (time 1734678555.55 total 103.87 last 245.38) - /unlink 29004 (time 1734678565.55 total 113.87 last 212.93) - /unlink 30000 (time 1734678570.48 total 118.79 last 202.31) - /unlink 32379 (time 1734678580.48 total 128.80 last 237.80) - /unlink 34718 (time 1734678590.48 total 138.80 last 233.79) - /unlink 37186 (time 1734678600.49 total 148.80 last 246.73) - /unlink 39316 (time 1734678610.49 total 158.81 last 212.93) - /unlink 40000 (time 1734678613.69 total 162.01 last 213.52) - /unlink 42028 (time 1734678623.70 total 172.01 last 202.74) - /unlink 44203 (time 1734678633.70 total 182.01 last 217.46) - /unlink 46328 (time 1734678643.70 total 192.02 last 212.49) - /unlink 48358 (time 1734678653.70 total 202.02 last 202.96) - /unlink 50000 (time 1734678661.18 total 209.50 last 219.52) - /unlink 52245 (time 1734678671.19 total 219.50 last 224.36) - /unlink 54309 (time 1734678681.19 total 229.50 last 206.39) - /unlink 56718 (time 1734678691.19 total 239.51 last 240.81) - /unlink 58874 (time 1734678701.19 total 249.51 last 215.54) - /unlink 60000 (time 1734678706.58 total 254.90 last 208.89) - /unlink 62292 (time 1734678716.59 total 264.90 last 229.15) - /unlink 64649 (time 1734678726.59 total 274.91 last 235.64) total: 64768 /unlink in 275.35 seconds: 235.22 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1353 (time 1734678756.08 total 10.01 last 135.21) - open/close 2765 (time 1734678766.08 total 20.01 last 141.12) - open/close 4027 (time 1734678776.09 total 30.02 last 126.16) - open/close 5407 (time 1734678786.09 total 40.02 last 137.99) - open/close 6876 (time 1734678796.09 total 50.02 last 146.88) - open/close 8202 (time 1734678806.09 total 60.02 last 132.60) - open/close 9458 (time 1734678816.09 total 70.02 last 125.58) - open/close 10000 (time 1734678820.57 total 74.50 last 121.01) - open/close 11387 (time 1734678830.57 total 84.50 last 138.63) - open/close 12600 (time 1734678840.58 total 94.51 last 121.26) - open/close 13787 (time 1734678850.58 total 104.51 last 118.67) - open/close 14955 (time 1734678860.59 total 114.52 last 116.70) - open/close 16121 (time 1734678870.59 total 124.52 last 116.57) - open/close 17340 (time 1734678880.60 total 134.53 last 121.77) - open/close 18550 (time 1734678890.61 total 144.54 last 120.93) - open/close 19728 (time 1734678900.61 total 154.54 last 117.74) - open/close 20000 (time 1734678903.07 total 157.00 last 110.49) - open/close 21099 (time 1734678913.08 total 167.01 last 109.88) - open/close 22336 (time 1734678923.08 total 177.01 last 123.65) - open/close 23517 (time 1734678933.08 total 187.01 last 118.05) - open/close 24616 (time 1734678943.09 total 197.02 last 109.85) - open/close 25783 (time 1734678953.10 total 207.03 last 116.55) - open/close 26931 (time 1734678963.11 total 217.04 last 114.72) - open/close 27963 (time 1734678973.11 total 227.04 last 103.15) - open/close 29088 (time 1734678983.11 total 237.04 last 112.48) - open/close 30000 (time 1734678991.38 total 245.31 last 110.33) - open/close 31188 (time 1734679001.39 total 255.32 last 118.74) - open/close 32291 (time 1734679011.39 total 265.32 last 110.24) - open/close 33557 (time 1734679021.39 total 275.32 last 126.57) - open/close 34795 (time 1734679031.39 total 285.32 last 123.78) - open/close 36033 (time 1734679041.40 total 295.33 last 123.74) - open/close 37321 (time 1734679051.40 total 305.33 last 128.79) - open/close 38686 (time 1734679061.41 total 315.34 last 136.33) - open/close 39840 (time 1734679071.43 total 325.36 last 115.22) - open/close 40961 (time 1734679081.44 total 335.37 last 112.02) - open/close 42359 (time 1734679091.44 total 345.37 last 139.78) - open/close 43846 (time 1734679101.44 total 355.37 last 148.61) - open/close 45171 (time 1734679111.45 total 365.38 last 132.43) - open/close 46579 (time 1734679121.45 total 375.38 last 140.79) - open/close 48104 (time 1734679131.45 total 385.38 last 152.47) - open/close 49603 (time 1734679141.46 total 395.39 last 149.74) - open/close 50000 (time 1734679144.59 total 398.52 last 126.82) - open/close 51233 (time 1734679154.60 total 408.53 last 123.21) - open/close 52484 (time 1734679164.61 total 418.54 last 125.01) - open/close 53718 (time 1734679174.61 total 428.54 last 123.35) - open/close 54985 (time 1734679184.61 total 438.54 last 126.66) - open/close 56289 (time 1734679194.61 total 448.54 last 130.40) - open/close 57597 (time 1734679204.62 total 458.55 last 130.74) - open/close 58992 (time 1734679214.62 total 468.55 last 139.42) - open/close 60000 (time 1734679222.37 total 476.30 last 130.16) - open/close 61279 (time 1734679232.37 total 486.30 last 127.87) - open/close 62486 (time 1734679242.38 total 496.31 last 120.65) - open/close 63717 (time 1734679252.38 total 506.31 last 123.06) total: 64768 open/close in 514.82 seconds: 125.81 ops/second - /unlink 1912 (time 1734679272.79 total 10.00 last 191.14) - /unlink 3842 (time 1734679282.79 total 20.01 last 192.95) - /unlink 5834 (time 1734679292.80 total 30.01 last 199.14) - /unlink 7655 (time 1734679302.80 total 40.01 last 182.07) - /unlink 9542 (time 1734679312.80 total 50.01 last 188.66) - /unlink 10000 (time 1734679315.38 total 52.59 last 177.90) - /unlink 12057 (time 1734679325.38 total 62.59 last 205.62) - /unlink 13997 (time 1734679335.38 total 72.59 last 193.92) - /unlink 16049 (time 1734679345.39 total 82.60 last 205.16) - /unlink 18232 (time 1734679355.39 total 92.60 last 218.26) - /unlink 20000 (time 1734679364.18 total 101.39 last 201.08) - /unlink 21856 (time 1734679374.19 total 111.40 last 185.48) - /unlink 23973 (time 1734679384.19 total 121.40 last 211.67) - /unlink 26101 (time 1734679394.20 total 131.41 last 212.60) - /unlink 28151 (time 1734679404.20 total 141.41 last 204.93) - /unlink 30000 (time 1734679412.92 total 150.13 last 212.17) - /unlink 31935 (time 1734679422.93 total 160.14 last 193.32) - /unlink 34171 (time 1734679432.93 total 170.14 last 223.47) - /unlink 36332 (time 1734679442.93 total 180.14 last 216.05) - /unlink 38779 (time 1734679452.94 total 190.15 last 244.61) - /unlink 40000 (time 1734679458.55 total 195.76 last 217.65) - /unlink 42433 (time 1734679468.55 total 205.76 last 243.28) - /unlink 44676 (time 1734679478.55 total 215.76 last 224.25) - /unlink 46610 (time 1734679488.55 total 225.76 last 193.35) - /unlink 48824 (time 1734679498.56 total 235.77 last 221.33) - /unlink 50000 (time 1734679503.91 total 241.12 last 219.68) - /unlink 52234 (time 1734679513.91 total 251.12 last 223.36) - /unlink 54692 (time 1734679523.91 total 261.12 last 245.72) - /unlink 56963 (time 1734679533.93 total 271.14 last 226.81) - /unlink 59222 (time 1734679543.93 total 281.14 last 225.86) - /unlink 60000 (time 1734679547.07 total 284.28 last 247.34) - /unlink 62501 (time 1734679557.08 total 294.29 last 250.01) - /unlink 64665 (time 1734679567.08 total 304.29 last 216.26) total: 64768 /unlink in 304.74 seconds: 212.54 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg351-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server unloading modules via unload_modules_local on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 PASS 106 (3797s) Stopping clients: oleg351-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg351-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 oleg351-server: oleg351-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: detected 4 online CPUs by sysfs oleg351-server: MODOPTS_LIBCFS= oleg351-server: Force libcfs to create 2 CPU partitions oleg351-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg351-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3907 sec ========== 02:27:15 (1734679635) === conf-sanity: start cleanup 02:27:15 (1734679635) === === conf-sanity: finish cleanup 02:27:17 (1734679637) ===