-----============= acceptance-small: conf-sanity ============----- Thu May 1 18:02:20 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg301-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg301-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg301-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg301-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg301-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg301-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg301-server oleg301-server: oleg301-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing load_modules_local oleg301-server: Loading modules from /home/green/git/lustre-release/lustre oleg301-server: detected 4 online CPUs by sysfs oleg301-server: MODOPTS_LIBCFS= oleg301-server: Force libcfs to create 2 CPU partitions oleg301-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg301-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg301-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg301-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg301-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg301-server: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg301-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg301-server: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg301-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg301-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server stop mds service on oleg301-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg301-server stop mds service on oleg301-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg301-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:04:15 (1746137055) Stopping clients: oleg301-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg301-client.virtnet /mnt/lustre2 (opts:-f) oleg301-server: oleg301-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing load_modules_local oleg301-server: Loading modules from /home/green/git/lustre-release/lustre oleg301-server: detected 4 online CPUs by sysfs oleg301-server: MODOPTS_LIBCFS= oleg301-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg301-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg301-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg301-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1227 (time 1746137123.11 total 10.00 last 122.64) - open/close 2434 (time 1746137133.12 total 20.01 last 120.60) - open/close 3540 (time 1746137143.12 total 30.02 last 110.58) - open/close 4703 (time 1746137153.12 total 40.02 last 116.28) - open/close 5901 (time 1746137163.13 total 50.02 last 119.72) - open/close 7064 (time 1746137173.13 total 60.03 last 116.26) - open/close 8251 (time 1746137183.14 total 70.03 last 118.65) - open/close 9492 (time 1746137193.14 total 80.03 last 124.08) - open/close 10000 (time 1746137197.64 total 84.54 last 112.73) - open/close 11131 (time 1746137207.65 total 94.55 last 113.01) - open/close 12351 (time 1746137217.66 total 104.55 last 121.92) - open/close 13456 (time 1746137227.67 total 114.57 last 110.35) - open/close 14869 (time 1746137237.67 total 124.57 last 141.30) - open/close 16047 (time 1746137247.69 total 134.59 last 117.57) - open/close 17297 (time 1746137257.70 total 144.60 last 124.91) - open/close 18518 (time 1746137267.71 total 154.60 last 122.01) - open/close 19799 (time 1746137277.71 total 164.61 last 128.06) - open/close 20939 (time 1746137287.71 total 174.61 last 113.99) - open/close 22191 (time 1746137297.72 total 184.62 last 125.06) - open/close 23284 (time 1746137307.73 total 194.63 last 109.19) - open/close 24371 (time 1746137317.73 total 204.63 last 108.68) - open/close 25502 (time 1746137327.74 total 214.63 last 113.08) - open/close 26550 (time 1746137337.74 total 224.63 last 104.77) - open/close 27605 (time 1746137347.74 total 234.64 last 105.47) - open/close 28631 (time 1746137357.74 total 244.64 last 102.57) - open/close 29785 (time 1746137367.75 total 254.64 last 115.37) - open/close 30000 (time 1746137369.79 total 256.68 last 105.31) - open/close 31092 (time 1746137379.80 total 266.70 last 109.04) - open/close 32276 (time 1746137389.81 total 276.70 last 118.34) - open/close 33402 (time 1746137399.82 total 286.71 last 112.50) - open/close 34613 (time 1746137409.83 total 296.73 last 120.89) - open/close 35717 (time 1746137419.84 total 306.73 last 110.35) - open/close 36841 (time 1746137429.84 total 316.73 last 112.39) - open/close 37893 (time 1746137439.84 total 326.74 last 105.13) - open/close 39098 (time 1746137449.85 total 336.75 last 120.44) - open/close 40000 (time 1746137457.90 total 344.80 last 112.01) - open/close 41124 (time 1746137467.91 total 354.80 last 112.33) - open/close 42346 (time 1746137477.92 total 364.81 last 122.12) - open/close 43570 (time 1746137487.92 total 374.82 last 122.32) - open/close 44847 (time 1746137497.93 total 384.82 last 127.63) - open/close 46442 (time 1746137507.93 total 394.83 last 159.40) - open/close 47760 (time 1746137517.93 total 404.83 last 131.79) - open/close 49777 (time 1746137527.93 total 414.83 last 201.70) - open/close 51391 (time 1746137537.94 total 424.83 last 161.37) - open/close 53202 (time 1746137547.94 total 434.84 last 181.02) - open/close 54918 (time 1746137557.94 total 444.84 last 171.56) - open/close 56829 (time 1746137567.94 total 454.84 last 191.07) - open/close 58168 (time 1746137577.94 total 464.84 last 133.90) - open/close 59790 (time 1746137587.95 total 474.84 last 162.10) - open/close 60996 (time 1746137597.95 total 484.85 last 120.56) - open/close 62187 (time 1746137607.95 total 494.85 last 119.08) - open/close 63689 (time 1746137617.96 total 504.86 last 150.08) total: 64768 open/close in 512.79 seconds: 126.31 ops/second - /unlink 2801 (time 1746137637.12 total 10.00 last 280.06) - /unlink 5303 (time 1746137647.12 total 20.00 last 250.14) - /unlink 7958 (time 1746137657.12 total 30.01 last 265.46) - /unlink 10000 (time 1746137664.94 total 37.83 last 261.12) - /unlink 12786 (time 1746137674.94 total 47.83 last 278.58) - /unlink 14997 (time 1746137684.95 total 57.83 last 220.99) - /unlink 17269 (time 1746137694.95 total 67.83 last 227.17) - /unlink 20000 (time 1746137703.35 total 76.23 last 325.05) - /unlink 22853 (time 1746137713.35 total 86.24 last 285.24) - /unlink 25565 (time 1746137723.35 total 96.24 last 271.18) - /unlink 27597 (time 1746137733.36 total 106.24 last 203.09) - /unlink 30000 (time 1746137742.39 total 115.28 last 266.00) - /unlink 32666 (time 1746137752.39 total 125.28 last 266.50) - /unlink 34914 (time 1746137762.40 total 135.28 last 224.78) - /unlink 37822 (time 1746137772.40 total 145.29 last 290.69) - /unlink 40000 (time 1746137778.65 total 151.53 last 348.54) - /unlink 42869 (time 1746137788.65 total 161.53 last 286.89) - /unlink 46401 (time 1746137798.65 total 171.54 last 353.05) - /unlink 49071 (time 1746137808.65 total 181.54 last 266.98) - /unlink 50000 (time 1746137811.09 total 183.98 last 380.61) - /unlink 53318 (time 1746137821.10 total 193.98 last 331.68) - /unlink 55832 (time 1746137831.10 total 203.99 last 251.37) - /unlink 59023 (time 1746137841.10 total 213.99 last 319.08) - /unlink 60000 (time 1746137844.45 total 217.34 last 291.46) - /unlink 62223 (time 1746137854.46 total 227.35 last 222.14) - /unlink 64728 (time 1746137864.46 total 237.35 last 250.43) total: 64768 /unlink in 237.48 seconds: 272.73 ops/second Waiting for MDT destroys to complete - open/close 1862 (time 1746137882.94 total 10.00 last 186.18) - open/close 3722 (time 1746137892.94 total 20.00 last 185.97) - open/close 5207 (time 1746137902.95 total 30.01 last 148.43) - open/close 6616 (time 1746137912.95 total 40.01 last 140.87) - open/close 7911 (time 1746137922.95 total 50.01 last 129.45) - open/close 9313 (time 1746137932.95 total 60.01 last 140.20) - open/close 10000 (time 1746137937.16 total 64.22 last 163.28) - open/close 11877 (time 1746137947.16 total 74.22 last 187.63) - open/close 13394 (time 1746137957.17 total 84.23 last 151.60) - open/close 14822 (time 1746137967.18 total 94.23 last 142.75) - open/close 16409 (time 1746137977.18 total 104.24 last 158.56) - open/close 17965 (time 1746137987.18 total 114.24 last 155.59) - open/close 19430 (time 1746137997.19 total 124.25 last 146.46) - open/close 20000 (time 1746138001.35 total 128.41 last 137.07) - open/close 21342 (time 1746138011.35 total 138.41 last 134.11) - open/close 22913 (time 1746138021.36 total 148.41 last 157.06) - open/close 24399 (time 1746138031.36 total 158.42 last 148.54) - open/close 26158 (time 1746138041.36 total 168.42 last 175.90) - open/close 28254 (time 1746138051.36 total 178.42 last 209.60) - open/close 30000 (time 1746138060.38 total 187.44 last 193.59) - open/close 31582 (time 1746138070.39 total 197.45 last 158.07) - open/close 32869 (time 1746138080.39 total 207.45 last 128.62) - open/close 34111 (time 1746138090.40 total 217.46 last 124.09) - open/close 35390 (time 1746138100.41 total 227.47 last 127.81) - open/close 36997 (time 1746138110.41 total 237.47 last 160.70) - open/close 38467 (time 1746138120.41 total 247.47 last 146.92) - open/close 39774 (time 1746138130.42 total 257.48 last 130.63) - open/close 41218 (time 1746138140.42 total 267.48 last 144.38) - open/close 42707 (time 1746138150.42 total 277.48 last 148.85) - open/close 44381 (time 1746138160.43 total 287.49 last 167.31) - open/close 46540 (time 1746138170.43 total 297.49 last 215.89) - open/close 48689 (time 1746138180.43 total 307.49 last 214.81) - open/close 50000 (time 1746138185.48 total 312.54 last 259.61) - open/close 51896 (time 1746138195.49 total 322.55 last 189.55) - open/close 53788 (time 1746138205.49 total 332.55 last 189.17) - open/close 55535 (time 1746138215.49 total 342.55 last 174.61) - open/close 57200 (time 1746138225.50 total 352.56 last 166.48) - open/close 58991 (time 1746138235.50 total 362.56 last 179.09) - open/close 60000 (time 1746138241.30 total 368.36 last 173.97) - open/close 61896 (time 1746138251.30 total 378.36 last 189.60) - open/close 63609 (time 1746138261.30 total 388.36 last 171.29) total: 64768 open/close in 394.06 seconds: 164.36 ops/second - /unlink 2771 (time 1746138278.24 total 10.00 last 277.08) - /unlink 6309 (time 1746138288.24 total 20.00 last 353.69) - /unlink 8868 (time 1746138298.24 total 30.01 last 255.82) - /unlink 10000 (time 1746138301.88 total 33.65 last 310.80) - /unlink 12883 (time 1746138311.89 total 43.65 last 288.24) - /unlink 15267 (time 1746138321.89 total 53.65 last 238.34) - /unlink 17686 (time 1746138331.89 total 63.66 last 241.76) - /unlink 19893 (time 1746138341.90 total 73.66 last 220.62) - /unlink 21983 (time 1746138351.90 total 83.66 last 208.99) - /unlink 24279 (time 1746138361.90 total 93.67 last 229.47) - /unlink 26737 (time 1746138371.90 total 103.67 last 245.78) - /unlink 29434 (time 1746138381.90 total 113.67 last 269.69) - /unlink 30000 (time 1746138384.08 total 115.85 last 260.00) - /unlink 33627 (time 1746138394.08 total 125.85 last 362.63) - /unlink 37044 (time 1746138404.08 total 135.85 last 341.66) - /unlink 39743 (time 1746138414.09 total 145.85 last 269.81) - /unlink 42118 (time 1746138424.09 total 155.86 last 237.40) - /unlink 44136 (time 1746138434.09 total 165.86 last 201.78) - /unlink 46414 (time 1746138444.10 total 175.87 last 227.60) - /unlink 48899 (time 1746138454.11 total 185.87 last 248.38) - /unlink 50000 (time 1746138458.89 total 190.66 last 230.04) - /unlink 52341 (time 1746138468.90 total 200.66 last 234.03) - /unlink 54582 (time 1746138478.90 total 210.67 last 223.95) - /unlink 56655 (time 1746138488.90 total 220.67 last 207.30) - /unlink 58804 (time 1746138498.90 total 230.67 last 214.87) - /unlink 60000 (time 1746138504.65 total 236.41 last 208.18) - /unlink 62326 (time 1746138514.66 total 246.42 last 232.44) total: 64768 /unlink in 254.92 seconds: 254.07 ops/second Waiting for MDT destroys to complete - open/close 1642 (time 1746138538.89 total 10.00 last 164.13) - open/close 3511 (time 1746138548.89 total 20.01 last 186.88) - open/close 5701 (time 1746138558.90 total 30.01 last 218.98) - open/close 8021 (time 1746138568.90 total 40.01 last 231.94) - open/close 10000 (time 1746138578.03 total 49.14 last 216.81) - open/close 11853 (time 1746138588.03 total 59.14 last 185.25) - open/close 13676 (time 1746138598.03 total 69.14 last 182.25) - open/close 15338 (time 1746138608.03 total 79.14 last 166.20) - open/close 16686 (time 1746138618.04 total 89.15 last 134.69) - open/close 17989 (time 1746138628.04 total 99.16 last 130.24) - open/close 19168 (time 1746138638.06 total 109.17 last 117.73) - open/close 20000 (time 1746138644.73 total 115.84 last 124.80) - open/close 21218 (time 1746138654.73 total 125.84 last 121.78) - open/close 22589 (time 1746138664.73 total 135.84 last 137.07) - open/close 23954 (time 1746138674.74 total 145.85 last 136.32) - open/close 25120 (time 1746138684.75 total 155.86 last 116.57) - open/close 26379 (time 1746138694.75 total 165.86 last 125.86) - open/close 27597 (time 1746138704.75 total 175.86 last 121.80) - open/close 28725 (time 1746138714.76 total 185.87 last 112.65) - open/close 30000 (time 1746138724.43 total 195.54 last 131.91) - open/close 31157 (time 1746138734.43 total 205.54 last 115.67) - open/close 32375 (time 1746138744.44 total 215.55 last 121.71) - open/close 33507 (time 1746138754.44 total 225.55 last 113.15) - open/close 34576 (time 1746138764.45 total 235.56 last 106.80) - open/close 35749 (time 1746138774.46 total 245.57 last 117.24) - open/close 36910 (time 1746138784.46 total 255.58 last 116.00) - open/close 38181 (time 1746138794.47 total 265.58 last 127.06) - open/close 39216 (time 1746138804.47 total 275.58 last 103.48) - open/close 40000 (time 1746138812.16 total 283.27 last 101.90) - open/close 41164 (time 1746138822.17 total 293.28 last 116.38) - open/close 42192 (time 1746138832.17 total 303.28 last 102.73) - open/close 43332 (time 1746138842.19 total 313.30 last 113.84) - open/close 44564 (time 1746138852.19 total 323.30 last 123.17) - open/close 45914 (time 1746138862.19 total 333.30 last 134.92) - open/close 47104 (time 1746138872.20 total 343.31 last 118.91) - open/close 48277 (time 1746138882.21 total 353.32 last 117.24) - open/close 49628 (time 1746138892.21 total 363.32 last 135.04) - open/close 50000 (time 1746138895.02 total 366.13 last 132.25) - open/close 51199 (time 1746138905.03 total 376.14 last 119.82) - open/close 52432 (time 1746138915.04 total 386.15 last 123.19) - open/close 53585 (time 1746138925.06 total 396.17 last 115.08) - open/close 54727 (time 1746138935.07 total 406.18 last 114.09) - open/close 55932 (time 1746138945.08 total 416.19 last 120.41) - open/close 57096 (time 1746138955.08 total 426.19 last 116.33) - open/close 58170 (time 1746138965.09 total 436.20 last 107.35) - open/close 59276 (time 1746138975.09 total 446.20 last 110.58) - open/close 60000 (time 1746138981.63 total 452.74 last 110.60) - open/close 61061 (time 1746138991.63 total 462.75 last 106.10) - open/close 62245 (time 1746139001.64 total 472.75 last 118.35) - open/close 63405 (time 1746139011.64 total 482.75 last 115.97) - open/close 64529 (time 1746139021.66 total 492.77 last 112.22) total: 64768 open/close in 495.02 seconds: 130.84 ops/second - /unlink 2025 (time 1746139035.70 total 10.00 last 202.50) - /unlink 3911 (time 1746139045.70 total 20.00 last 188.57) - /unlink 5877 (time 1746139055.70 total 30.00 last 196.55) - /unlink 7815 (time 1746139065.71 total 40.01 last 193.74) - /unlink 9874 (time 1746139075.71 total 50.01 last 205.82) - /unlink 12008 (time 1746139085.72 total 60.02 last 213.20) - /unlink 14044 (time 1746139095.73 total 70.03 last 203.47) - /unlink 15929 (time 1746139105.76 total 80.05 last 187.99) - /unlink 17899 (time 1746139115.76 total 90.06 last 196.93) - /unlink 19933 (time 1746139125.76 total 100.06 last 203.38) - /unlink 22361 (time 1746139135.76 total 110.06 last 242.78) - /unlink 24410 (time 1746139145.77 total 120.07 last 204.77) - /unlink 26593 (time 1746139155.77 total 130.07 last 218.20) - /unlink 29103 (time 1746139165.78 total 140.08 last 250.89) - /unlink 30000 (time 1746139170.04 total 144.34 last 210.38) - /unlink 32431 (time 1746139180.04 total 154.34 last 243.04) - /unlink 34532 (time 1746139190.05 total 164.34 last 210.03) - /unlink 36839 (time 1746139200.05 total 174.35 last 230.68) - /unlink 39202 (time 1746139210.05 total 184.35 last 236.14) - /unlink 40000 (time 1746139213.81 total 188.11 last 212.36) - /unlink 42354 (time 1746139223.82 total 198.11 last 235.30) - /unlink 44685 (time 1746139233.82 total 208.12 last 233.08) - /unlink 46931 (time 1746139243.82 total 218.12 last 224.56) - /unlink 49499 (time 1746139253.82 total 228.12 last 256.79) - /unlink 52369 (time 1746139263.82 total 238.12 last 286.97) - /unlink 54684 (time 1746139273.82 total 248.12 last 231.46) - /unlink 57132 (time 1746139283.82 total 258.12 last 244.75) - /unlink 59676 (time 1746139293.82 total 268.12 last 254.36) - /unlink 62173 (time 1746139303.83 total 278.13 last 249.66) - /unlink 64443 (time 1746139313.83 total 288.13 last 226.97) total: 64768 /unlink in 289.56 seconds: 223.68 ops/second Waiting for MDT destroys to complete - open/close 1441 (time 1746139336.69 total 10.00 last 144.07) - open/close 3017 (time 1746139346.69 total 20.01 last 157.55) - open/close 4506 (time 1746139356.70 total 30.01 last 148.79) - open/close 5925 (time 1746139366.70 total 40.01 last 141.87) - open/close 7324 (time 1746139376.71 total 50.02 last 139.82) - open/close 8726 (time 1746139386.71 total 60.03 last 140.11) - open/close 10000 (time 1746139395.26 total 68.57 last 149.11) - open/close 11730 (time 1746139405.26 total 78.57 last 172.97) - open/close 13289 (time 1746139415.26 total 88.58 last 155.83) - open/close 14697 (time 1746139425.27 total 98.59 last 140.67) - open/close 16164 (time 1746139435.28 total 108.59 last 146.64) - open/close 17686 (time 1746139445.28 total 118.60 last 152.14) - open/close 19272 (time 1746139455.28 total 128.60 last 158.53) - open/close 20000 (time 1746139460.74 total 134.05 last 133.54) - open/close 21372 (time 1746139470.74 total 144.06 last 137.11) - open/close 22767 (time 1746139480.74 total 154.06 last 139.49) - open/close 24371 (time 1746139490.75 total 164.06 last 160.38) - open/close 25781 (time 1746139500.75 total 174.07 last 140.86) - open/close 27275 (time 1746139510.76 total 184.08 last 149.29) - open/close 28920 (time 1746139520.76 total 194.08 last 164.49) - open/close 30000 (time 1746139527.40 total 200.71 last 162.79) - open/close 31622 (time 1746139537.40 total 210.71 last 162.19) - open/close 33272 (time 1746139547.40 total 220.72 last 164.94) - open/close 34894 (time 1746139557.41 total 230.72 last 162.09) - open/close 36665 (time 1746139567.41 total 240.72 last 177.07) - open/close 38462 (time 1746139577.41 total 250.73 last 179.65) - open/close 39904 (time 1746139587.42 total 260.73 last 144.13) - open/close 41527 (time 1746139597.42 total 270.73 last 162.30) - open/close 42917 (time 1746139607.42 total 280.73 last 138.96) - open/close 44249 (time 1746139617.42 total 290.74 last 133.14) - open/close 45556 (time 1746139627.43 total 300.74 last 130.69) - open/close 46801 (time 1746139637.43 total 310.74 last 124.46) - open/close 47951 (time 1746139647.43 total 320.75 last 114.96) - open/close 49099 (time 1746139657.44 total 330.75 last 114.71) - open/close 50000 (time 1746139664.73 total 338.05 last 123.52) - open/close 51256 (time 1746139674.74 total 348.05 last 125.58) - open/close 52528 (time 1746139684.74 total 358.05 last 127.17) - open/close 53778 (time 1746139694.74 total 368.06 last 124.96) - open/close 54935 (time 1746139704.74 total 378.06 last 115.69) - open/close 56143 (time 1746139714.74 total 388.06 last 120.79) - open/close 57279 (time 1746139724.76 total 398.07 last 113.46) - open/close 58471 (time 1746139734.77 total 408.08 last 119.08) - open/close 59978 (time 1746139744.77 total 418.09 last 150.62) - open/close 61315 (time 1746139754.78 total 428.09 last 133.65) - open/close 62551 (time 1746139764.78 total 438.10 last 123.52) - open/close 63814 (time 1746139774.79 total 448.10 last 126.23) total: 64768 open/close in 456.53 seconds: 141.87 ops/second - /unlink 1892 (time 1746139795.49 total 10.01 last 189.02) - /unlink 4051 (time 1746139805.49 total 20.01 last 215.89) - /unlink 6300 (time 1746139815.49 total 30.01 last 224.80) - /unlink 8507 (time 1746139825.49 total 40.01 last 220.69) - /unlink 10000 (time 1746139832.49 total 47.02 last 213.25) - /unlink 12715 (time 1746139842.50 total 57.02 last 271.43) - /unlink 15361 (time 1746139852.50 total 67.02 last 264.57) - /unlink 17791 (time 1746139862.50 total 77.02 last 242.90) - /unlink 20000 (time 1746139870.97 total 85.50 last 260.68) - /unlink 23458 (time 1746139880.98 total 95.50 last 345.56) - /unlink 26181 (time 1746139891.03 total 105.56 last 270.93) - /unlink 28296 (time 1746139901.03 total 115.56 last 211.45) - /unlink 30000 (time 1746139907.47 total 122.00 last 264.59) - /unlink 32667 (time 1746139917.48 total 132.00 last 266.66) - /unlink 35623 (time 1746139927.48 total 142.00 last 295.51) - /unlink 37959 (time 1746139937.48 total 152.01 last 233.50) - /unlink 40000 (time 1746139945.24 total 159.77 last 263.03) - /unlink 42198 (time 1746139955.24 total 169.77 last 219.75) - /unlink 44727 (time 1746139965.25 total 179.77 last 252.82) - /unlink 46862 (time 1746139975.25 total 189.77 last 213.44) - /unlink 49216 (time 1746139985.25 total 199.78 last 235.30) - /unlink 50000 (time 1746139989.12 total 203.65 last 202.72) - /unlink 52329 (time 1746139999.12 total 213.65 last 232.84) - /unlink 54607 (time 1746140009.13 total 223.65 last 227.79) - /unlink 56950 (time 1746140019.13 total 233.65 last 234.29) - /unlink 59156 (time 1746140029.13 total 243.66 last 220.40) - /unlink 60000 (time 1746140032.96 total 247.49 last 220.44) - /unlink 62465 (time 1746140042.97 total 257.49 last 246.37) total: 64768 /unlink in 266.87 seconds: 242.69 ops/second Waiting for MDT destroys to complete - open/close 1490 (time 1746140074.54 total 10.00 last 148.96) - open/close 2927 (time 1746140084.55 total 20.01 last 143.62) - open/close 4149 (time 1746140094.55 total 30.01 last 122.16) - open/close 5390 (time 1746140104.56 total 40.02 last 124.01) - open/close 6535 (time 1746140114.56 total 50.02 last 114.49) - open/close 7745 (time 1746140124.56 total 60.02 last 121.00) - open/close 8912 (time 1746140134.56 total 70.02 last 116.64) - open/close 10000 (time 1746140143.10 total 78.56 last 127.43) - open/close 11229 (time 1746140153.11 total 88.57 last 122.84) - open/close 12469 (time 1746140163.11 total 98.57 last 123.92) - open/close 13656 (time 1746140173.12 total 108.58 last 118.68) - open/close 14974 (time 1746140183.12 total 118.58 last 131.73) - open/close 16248 (time 1746140193.12 total 128.58 last 127.36) - open/close 17435 (time 1746140203.13 total 138.59 last 118.67) - open/close 18795 (time 1746140213.13 total 148.59 last 135.96) - open/close 20000 (time 1746140222.77 total 158.23 last 124.98) - open/close 21216 (time 1746140232.77 total 168.23 last 121.56) - open/close 22394 (time 1746140242.78 total 178.24 last 117.74) - open/close 23518 (time 1746140252.79 total 188.25 last 112.32) - open/close 24662 (time 1746140262.80 total 198.26 last 114.24) - open/close 25722 (time 1746140272.81 total 208.27 last 105.89) - open/close 26963 (time 1746140282.83 total 218.29 last 123.81) - open/close 28218 (time 1746140292.87 total 228.33 last 125.01) - open/close 29468 (time 1746140302.90 total 238.36 last 124.61) - open/close 30000 (time 1746140307.05 total 242.51 last 128.47) - open/close 31205 (time 1746140317.05 total 252.51 last 120.49) - open/close 32427 (time 1746140327.05 total 262.51 last 122.18) - open/close 33562 (time 1746140337.06 total 272.52 last 113.39) - open/close 34864 (time 1746140347.07 total 282.53 last 130.09) - open/close 36032 (time 1746140357.07 total 292.53 last 116.73) - open/close 37277 (time 1746140367.08 total 302.54 last 124.44) - open/close 38585 (time 1746140377.08 total 312.54 last 130.75) - open/close 39685 (time 1746140387.08 total 322.54 last 109.99) - open/close 40000 (time 1746140390.33 total 325.79 last 97.05) - open/close 41143 (time 1746140400.34 total 335.80 last 114.20) - open/close 42343 (time 1746140410.34 total 345.80 last 119.98) - open/close 43575 (time 1746140420.34 total 355.80 last 123.19) - open/close 44899 (time 1746140430.34 total 365.81 last 132.33) - open/close 46201 (time 1746140440.35 total 375.81 last 130.13) - open/close 47386 (time 1746140450.36 total 385.82 last 118.43) - open/close 48581 (time 1746140460.36 total 395.82 last 119.49) - open/close 49757 (time 1746140470.36 total 405.82 last 117.58) - open/close 50000 (time 1746140472.41 total 407.87 last 118.26) - open/close 51243 (time 1746140482.42 total 417.88 last 124.27) - open/close 52434 (time 1746140492.42 total 427.88 last 119.10) - open/close 53671 (time 1746140502.42 total 437.89 last 123.59) - open/close 54936 (time 1746140512.43 total 447.89 last 126.49) - open/close 56085 (time 1746140522.44 total 457.90 last 114.77) - open/close 57190 (time 1746140532.45 total 467.91 last 110.40) - open/close 58378 (time 1746140542.45 total 477.91 last 118.77) - open/close 59454 (time 1746140552.46 total 487.92 last 107.50) - open/close 60000 (time 1746140557.49 total 492.95 last 108.60) - open/close 61173 (time 1746140567.49 total 502.95 last 117.28) - open/close 62311 (time 1746140577.50 total 512.96 last 113.71) - open/close 63640 (time 1746140587.50 total 522.96 last 132.84) total: 64768 open/close in 531.36 seconds: 121.89 ops/second - /unlink 2171 (time 1746140607.35 total 10.00 last 217.09) - /unlink 4375 (time 1746140617.35 total 20.00 last 220.36) - /unlink 6623 (time 1746140627.35 total 30.01 last 224.71) - /unlink 8608 (time 1746140637.36 total 40.01 last 198.39) - /unlink 10000 (time 1746140643.41 total 46.06 last 230.03) - /unlink 12921 (time 1746140653.41 total 56.06 last 292.08) - /unlink 15479 (time 1746140663.41 total 66.07 last 255.72) - /unlink 18111 (time 1746140673.42 total 76.07 last 263.17) - /unlink 20000 (time 1746140680.89 total 83.55 last 252.57) - /unlink 22181 (time 1746140690.90 total 93.56 last 217.88) - /unlink 24614 (time 1746140700.91 total 103.56 last 243.18) - /unlink 27119 (time 1746140710.91 total 113.56 last 250.48) - /unlink 29852 (time 1746140720.91 total 123.57 last 273.19) - /unlink 32456 (time 1746140730.92 total 133.57 last 260.33) - /unlink 34858 (time 1746140740.92 total 143.57 last 240.14) - /unlink 37052 (time 1746140750.93 total 153.58 last 219.20) - /unlink 38901 (time 1746140760.93 total 163.58 last 184.84) - /unlink 40000 (time 1746140765.94 total 168.59 last 219.40) - /unlink 42428 (time 1746140775.94 total 178.59 last 242.78) - /unlink 44560 (time 1746140785.95 total 188.60 last 213.12) - /unlink 46536 (time 1746140795.95 total 198.60 last 197.52) - /unlink 48563 (time 1746140805.95 total 208.60 last 202.66) - /unlink 50000 (time 1746140811.90 total 214.55 last 241.72) - /unlink 52668 (time 1746140821.90 total 224.55 last 266.76) - /unlink 55074 (time 1746140831.90 total 234.55 last 240.57) - /unlink 57285 (time 1746140841.90 total 244.55 last 221.07) - /unlink 59803 (time 1746140851.91 total 254.56 last 251.69) - /unlink 62138 (time 1746140861.91 total 264.56 last 233.43) - /unlink 64556 (time 1746140871.91 total 274.56 last 241.73) total: 64768 /unlink in 275.63 seconds: 234.98 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg301-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg301-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server stop mds service on oleg301-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg301-server stop mds service on oleg301-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg301-server unloading modules via unload_modules_local on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2 PASS 106 (3855s) Stopping clients: oleg301-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg301-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2 oleg301-server: oleg301-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing load_modules_local oleg301-server: Loading modules from /home/green/git/lustre-release/lustre oleg301-server: detected 4 online CPUs by sysfs oleg301-server: MODOPTS_LIBCFS= oleg301-server: Force libcfs to create 2 CPU partitions oleg301-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg301-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3999 sec ========== 19:09:00 (1746140940) === conf-sanity: start cleanup 19:09:01 (1746140941) === === conf-sanity: finish cleanup 19:09:02 (1746140942) ===