-----============= acceptance-small: conf-sanity ============----- Sun Jun 22 07:12:29 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg124-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg124-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg124-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg124-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg124-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg124-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg124-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg124-server oleg124-server: oleg124-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing load_modules_local oleg124-server: Loading modules from /home/green/git/lustre-release/lustre oleg124-server: detected 4 online CPUs by sysfs oleg124-server: MODOPTS_LIBCFS= oleg124-server: Force libcfs to create 2 CPU partitions oleg124-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg124-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg124-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg124-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg124-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg124-server: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg124-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg124-server: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg124-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg124-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg124-server stop mds service on oleg124-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg124-server stop mds service on oleg124-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg124-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:14:31 (1750590871) Stopping clients: oleg124-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg124-client.virtnet /mnt/lustre2 (opts:-f) oleg124-server: oleg124-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing load_modules_local oleg124-server: Loading modules from /home/green/git/lustre-release/lustre oleg124-server: detected 4 online CPUs by sysfs oleg124-server: MODOPTS_LIBCFS= oleg124-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg124-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg124-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg124-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1878 (time 1750590940.48 total 10.00 last 187.76) - open/close 3633 (time 1750590950.49 total 20.01 last 175.43) - open/close 5424 (time 1750590960.49 total 30.01 last 179.08) - open/close 7038 (time 1750590970.49 total 40.01 last 161.38) - open/close 8745 (time 1750590980.49 total 50.01 last 170.65) - open/close 10000 (time 1750590988.44 total 57.96 last 157.88) - open/close 11578 (time 1750590998.44 total 67.96 last 157.78) - open/close 12970 (time 1750591008.45 total 77.97 last 139.09) - open/close 14426 (time 1750591018.45 total 87.97 last 145.58) - open/close 15846 (time 1750591028.46 total 97.98 last 141.94) - open/close 17339 (time 1750591038.46 total 107.98 last 149.29) - open/close 18821 (time 1750591048.46 total 117.98 last 148.18) - open/close 20000 (time 1750591055.99 total 125.50 last 156.67) - open/close 21343 (time 1750591065.99 total 135.51 last 134.24) - open/close 22617 (time 1750591075.99 total 145.51 last 127.38) - open/close 24063 (time 1750591086.00 total 155.51 last 144.53) - open/close 25670 (time 1750591096.00 total 165.52 last 160.63) - open/close 27050 (time 1750591106.02 total 175.54 last 137.74) - open/close 28406 (time 1750591116.03 total 185.54 last 135.51) - open/close 29685 (time 1750591126.04 total 195.55 last 127.78) - open/close 30000 (time 1750591128.23 total 197.75 last 143.66) - open/close 31492 (time 1750591138.23 total 207.75 last 149.20) - open/close 32983 (time 1750591148.24 total 217.76 last 148.98) - open/close 34493 (time 1750591158.24 total 227.76 last 150.99) - open/close 35895 (time 1750591168.24 total 237.76 last 140.16) - open/close 37203 (time 1750591178.26 total 247.78 last 130.59) - open/close 38543 (time 1750591188.26 total 257.78 last 133.96) - open/close 40000 (time 1750591198.24 total 267.76 last 145.99) - open/close 41327 (time 1750591208.25 total 277.77 last 132.60) - open/close 42764 (time 1750591218.25 total 287.77 last 143.67) - open/close 44411 (time 1750591228.25 total 297.77 last 164.68) - open/close 45930 (time 1750591238.25 total 307.77 last 151.88) - open/close 47388 (time 1750591248.26 total 317.78 last 145.72) - open/close 48835 (time 1750591258.27 total 327.79 last 144.54) - open/close 50000 (time 1750591265.64 total 335.16 last 157.99) - open/close 51586 (time 1750591275.65 total 345.16 last 158.55) - open/close 53071 (time 1750591285.65 total 355.17 last 148.48) - open/close 54778 (time 1750591295.65 total 365.17 last 170.62) - open/close 56220 (time 1750591305.66 total 375.17 last 144.14) - open/close 57662 (time 1750591315.66 total 385.18 last 144.15) - open/close 59096 (time 1750591325.66 total 395.18 last 143.37) - open/close 60000 (time 1750591331.46 total 400.98 last 155.86) - open/close 61494 (time 1750591341.47 total 410.98 last 149.36) - open/close 62822 (time 1750591351.47 total 420.99 last 132.71) - open/close 64374 (time 1750591361.47 total 430.99 last 155.16) total: 64768 open/close in 433.79 seconds: 149.31 ops/second - /unlink 2596 (time 1750591376.07 total 10.00 last 259.54) - /unlink 4960 (time 1750591386.08 total 20.00 last 236.35) - /unlink 7625 (time 1750591396.08 total 30.01 last 266.42) - /unlink 10000 (time 1750591403.77 total 37.70 last 308.76) - /unlink 12728 (time 1750591413.77 total 47.70 last 272.80) - /unlink 15381 (time 1750591423.77 total 57.70 last 265.28) - /unlink 17770 (time 1750591433.78 total 67.70 last 238.83) - /unlink 20000 (time 1750591442.35 total 76.27 last 260.17) - /unlink 22431 (time 1750591452.35 total 86.28 last 243.08) - /unlink 24766 (time 1750591462.35 total 96.28 last 233.50) - /unlink 27261 (time 1750591472.35 total 106.28 last 249.50) - /unlink 29981 (time 1750591482.35 total 116.28 last 271.89) - /unlink 33120 (time 1750591492.36 total 126.28 last 313.78) - /unlink 36088 (time 1750591502.36 total 136.29 last 296.72) - /unlink 38801 (time 1750591512.36 total 146.29 last 271.22) - /unlink 40000 (time 1750591516.34 total 150.26 last 301.62) - /unlink 42653 (time 1750591526.34 total 160.27 last 265.23) - /unlink 45586 (time 1750591536.34 total 170.27 last 293.28) - /unlink 49165 (time 1750591546.34 total 180.27 last 357.74) - /unlink 50000 (time 1750591548.96 total 182.89 last 319.42) - /unlink 52580 (time 1750591558.96 total 192.89 last 257.98) - /unlink 54929 (time 1750591568.96 total 202.89 last 234.87) - /unlink 57543 (time 1750591578.96 total 212.89 last 261.38) - /unlink 60000 (time 1750591587.74 total 221.67 last 279.97) - /unlink 62990 (time 1750591597.74 total 231.67 last 298.96) total: 64768 /unlink in 239.00 seconds: 271.00 ops/second Waiting for MDT destroys to complete - open/close 1614 (time 1750591627.80 total 10.00 last 161.35) - open/close 3202 (time 1750591637.80 total 20.00 last 158.79) - open/close 4712 (time 1750591647.80 total 30.00 last 150.99) - open/close 6133 (time 1750591657.81 total 40.01 last 142.02) - open/close 7554 (time 1750591667.81 total 50.01 last 142.06) - open/close 9077 (time 1750591677.81 total 60.01 last 152.28) - open/close 10000 (time 1750591682.85 total 65.05 last 183.29) - open/close 11506 (time 1750591692.85 total 75.05 last 150.53) - open/close 13360 (time 1750591702.86 total 85.06 last 185.30) - open/close 14844 (time 1750591712.86 total 95.06 last 148.38) - open/close 16334 (time 1750591722.86 total 105.06 last 148.96) - open/close 17931 (time 1750591732.87 total 115.07 last 159.66) - open/close 19815 (time 1750591742.87 total 125.07 last 188.37) - open/close 21368 (time 1750591752.87 total 135.07 last 155.21) - open/close 22939 (time 1750591762.88 total 145.08 last 157.07) - open/close 24411 (time 1750591772.88 total 155.08 last 147.12) - open/close 26206 (time 1750591782.88 total 165.08 last 179.45) - open/close 28031 (time 1750591792.89 total 175.09 last 182.38) - open/close 29965 (time 1750591802.90 total 185.10 last 193.30) - open/close 31532 (time 1750591812.90 total 195.10 last 156.65) - open/close 33058 (time 1750591822.91 total 205.11 last 152.46) - open/close 34487 (time 1750591832.92 total 215.12 last 142.79) - open/close 36177 (time 1750591842.93 total 225.13 last 168.80) - open/close 37705 (time 1750591852.93 total 235.13 last 152.73) - open/close 39321 (time 1750591862.93 total 245.13 last 161.57) - open/close 40000 (time 1750591868.21 total 250.41 last 128.76) - open/close 41461 (time 1750591878.21 total 260.41 last 146.06) - open/close 42923 (time 1750591888.21 total 270.41 last 146.19) - open/close 44088 (time 1750591898.22 total 280.42 last 116.39) - open/close 45467 (time 1750591908.22 total 290.42 last 137.84) - open/close 46788 (time 1750591918.23 total 300.43 last 132.04) - open/close 48167 (time 1750591928.23 total 310.43 last 137.89) - open/close 49609 (time 1750591938.23 total 320.43 last 144.16) - open/close 50000 (time 1750591941.12 total 323.32 last 135.61) - open/close 51495 (time 1750591951.12 total 333.32 last 149.48) - open/close 52931 (time 1750591961.12 total 343.32 last 143.58) - open/close 54310 (time 1750591971.12 total 353.32 last 137.82) - open/close 55847 (time 1750591981.13 total 363.33 last 153.66) - open/close 57159 (time 1750591991.13 total 373.33 last 131.18) - open/close 58563 (time 1750592001.13 total 383.33 last 140.37) - open/close 59838 (time 1750592011.13 total 393.33 last 127.44) - open/close 61426 (time 1750592021.14 total 403.34 last 158.75) - open/close 62720 (time 1750592031.14 total 413.34 last 129.31) - open/close 64190 (time 1750592041.15 total 423.35 last 146.94) total: 64768 open/close in 427.55 seconds: 151.49 ops/second - /unlink 1880 (time 1750592057.31 total 10.00 last 187.98) - /unlink 4020 (time 1750592067.31 total 20.00 last 214.00) - /unlink 6330 (time 1750592077.31 total 30.00 last 230.92) - /unlink 8878 (time 1750592087.32 total 40.01 last 254.73) - /unlink 10000 (time 1750592091.46 total 44.16 last 270.43) - /unlink 12679 (time 1750592101.47 total 54.16 last 267.84) - /unlink 15022 (time 1750592111.47 total 64.16 last 234.29) - /unlink 17256 (time 1750592121.48 total 74.17 last 223.16) - /unlink 19413 (time 1750592131.48 total 84.17 last 215.68) - /unlink 20000 (time 1750592134.03 total 86.73 last 229.80) - /unlink 22456 (time 1750592144.04 total 96.73 last 245.53) - /unlink 24531 (time 1750592154.04 total 106.73 last 207.48) - /unlink 26717 (time 1750592164.04 total 116.73 last 218.53) - /unlink 28891 (time 1750592174.04 total 126.73 last 217.38) - /unlink 30000 (time 1750592178.90 total 131.59 last 228.14) - /unlink 32246 (time 1750592188.90 total 141.60 last 224.57) - /unlink 34643 (time 1750592198.90 total 151.60 last 239.68) - /unlink 37021 (time 1750592208.91 total 161.60 last 237.72) - /unlink 39197 (time 1750592218.91 total 171.60 last 217.60) - /unlink 40000 (time 1750592222.16 total 174.85 last 247.26) - /unlink 42557 (time 1750592232.16 total 184.85 last 255.64) - /unlink 44834 (time 1750592242.18 total 194.87 last 227.18) - /unlink 47262 (time 1750592252.18 total 204.87 last 242.79) - /unlink 49482 (time 1750592262.18 total 214.88 last 221.94) - /unlink 50000 (time 1750592264.54 total 217.23 last 220.23) - /unlink 52232 (time 1750592274.54 total 227.23 last 223.15) - /unlink 54610 (time 1750592284.55 total 237.24 last 237.56) - /unlink 57044 (time 1750592294.55 total 247.24 last 243.39) - /unlink 59537 (time 1750592304.55 total 257.24 last 249.29) - /unlink 61643 (time 1750592314.56 total 267.25 last 210.35) - /unlink 64502 (time 1750592324.56 total 277.25 last 285.88) total: 64768 /unlink in 278.24 seconds: 232.78 ops/second Waiting for MDT destroys to complete - open/close 1327 (time 1750592345.18 total 10.00 last 132.70) - open/close 2758 (time 1750592355.19 total 20.00 last 143.08) - open/close 4044 (time 1750592365.19 total 30.01 last 128.50) - open/close 5383 (time 1750592375.20 total 40.01 last 133.87) - open/close 6969 (time 1750592385.20 total 50.02 last 158.46) - open/close 8216 (time 1750592395.21 total 60.02 last 124.66) - open/close 9562 (time 1750592405.21 total 70.03 last 134.55) - open/close 10000 (time 1750592408.07 total 72.89 last 153.06) - open/close 11405 (time 1750592418.07 total 82.89 last 140.49) - open/close 12886 (time 1750592428.08 total 92.89 last 148.06) - open/close 14352 (time 1750592438.08 total 102.90 last 146.47) - open/close 15674 (time 1750592448.09 total 112.90 last 132.18) - open/close 17226 (time 1750592458.09 total 122.90 last 155.17) - open/close 18752 (time 1750592468.09 total 132.91 last 152.59) - open/close 20000 (time 1750592476.81 total 141.62 last 143.16) - open/close 21569 (time 1750592486.81 total 151.63 last 156.79) - open/close 22969 (time 1750592496.82 total 161.63 last 139.97) - open/close 24614 (time 1750592506.82 total 171.63 last 164.45) - open/close 26205 (time 1750592516.82 total 181.64 last 159.01) - open/close 27981 (time 1750592526.83 total 191.65 last 177.46) - open/close 29554 (time 1750592536.83 total 201.65 last 157.26) - open/close 30000 (time 1750592539.51 total 204.32 last 166.99) - open/close 31458 (time 1750592549.51 total 214.33 last 145.72) - open/close 32835 (time 1750592559.52 total 224.33 last 137.64) - open/close 34204 (time 1750592569.53 total 234.34 last 136.73) - open/close 35517 (time 1750592579.53 total 244.35 last 131.23) - open/close 36693 (time 1750592589.54 total 254.35 last 117.55) - open/close 38064 (time 1750592599.54 total 264.36 last 137.04) - open/close 39473 (time 1750592609.55 total 274.36 last 140.82) - open/close 40000 (time 1750592614.07 total 278.88 last 116.63) - open/close 41349 (time 1750592624.07 total 288.89 last 134.82) - open/close 42668 (time 1750592634.08 total 298.90 last 131.79) - open/close 44042 (time 1750592644.09 total 308.90 last 137.30) - open/close 45281 (time 1750592654.09 total 318.90 last 123.88) - open/close 46454 (time 1750592664.09 total 328.91 last 117.28) - open/close 47799 (time 1750592674.09 total 338.91 last 134.49) - open/close 48891 (time 1750592684.10 total 348.92 last 109.06) - open/close 50000 (time 1750592692.45 total 357.27 last 132.80) - open/close 51403 (time 1750592702.46 total 367.28 last 140.20) - open/close 52669 (time 1750592712.47 total 377.29 last 126.48) - open/close 53962 (time 1750592722.47 total 387.29 last 129.26) - open/close 55294 (time 1750592732.48 total 397.30 last 133.11) - open/close 56587 (time 1750592742.48 total 407.30 last 129.27) - open/close 57906 (time 1750592752.48 total 417.30 last 131.89) - open/close 59401 (time 1750592762.49 total 427.31 last 149.41) - open/close 60000 (time 1750592766.96 total 431.77 last 134.14) - open/close 61408 (time 1750592776.96 total 441.78 last 140.70) - open/close 62816 (time 1750592786.97 total 451.78 last 140.76) - open/close 64263 (time 1750592796.97 total 461.79 last 144.65) total: 64768 open/close in 464.88 seconds: 139.32 ops/second - /unlink 2425 (time 1750592811.89 total 10.00 last 242.49) - /unlink 5484 (time 1750592821.89 total 20.01 last 305.76) - /unlink 8031 (time 1750592831.90 total 30.01 last 254.65) - /unlink 10000 (time 1750592839.54 total 37.65 last 257.54) - /unlink 12317 (time 1750592849.55 total 47.66 last 231.61) - /unlink 14820 (time 1750592859.55 total 57.66 last 250.26) - /unlink 17273 (time 1750592869.55 total 67.66 last 245.29) - /unlink 19337 (time 1750592879.55 total 77.66 last 206.32) - /unlink 20000 (time 1750592882.83 total 80.94 last 202.31) - /unlink 22050 (time 1750592892.83 total 90.94 last 204.94) - /unlink 24239 (time 1750592902.84 total 100.95 last 218.71) - /unlink 26235 (time 1750592912.84 total 110.96 last 199.53) - /unlink 28526 (time 1750592922.85 total 120.96 last 228.98) - /unlink 30000 (time 1750592930.06 total 128.17 last 204.41) - /unlink 32067 (time 1750592940.07 total 138.18 last 206.46) - /unlink 34283 (time 1750592950.07 total 148.18 last 221.57) - /unlink 36604 (time 1750592960.08 total 158.19 last 232.02) - /unlink 38761 (time 1750592970.08 total 168.19 last 215.67) - /unlink 40000 (time 1750592975.40 total 173.51 last 232.82) - /unlink 42302 (time 1750592985.40 total 183.51 last 230.19) - /unlink 44803 (time 1750592995.40 total 193.52 last 249.98) - /unlink 47026 (time 1750593005.40 total 203.52 last 222.29) - /unlink 49453 (time 1750593015.41 total 213.52 last 242.63) - /unlink 50000 (time 1750593017.86 total 215.97 last 223.28) - /unlink 52206 (time 1750593027.86 total 225.97 last 220.50) - /unlink 54887 (time 1750593037.86 total 235.97 last 268.09) - /unlink 57324 (time 1750593047.87 total 245.98 last 243.61) - /unlink 59699 (time 1750593057.87 total 255.98 last 237.50) - /unlink 62352 (time 1750593067.87 total 265.98 last 265.29) total: 64768 /unlink in 275.83 seconds: 234.81 ops/second Waiting for MDT destroys to complete - open/close 1456 (time 1750593094.74 total 10.01 last 145.43) - open/close 2793 (time 1750593104.74 total 20.02 last 133.64) - open/close 4175 (time 1750593114.74 total 30.02 last 138.18) - open/close 5531 (time 1750593124.75 total 40.02 last 135.52) - open/close 6857 (time 1750593134.75 total 50.03 last 132.54) - open/close 8311 (time 1750593144.76 total 60.04 last 145.30) - open/close 9556 (time 1750593154.76 total 70.04 last 124.47) - open/close 10000 (time 1750593158.21 total 73.49 last 128.75) - open/close 11251 (time 1750593168.22 total 83.49 last 125.01) - open/close 12571 (time 1750593178.23 total 93.51 last 131.82) - open/close 13804 (time 1750593188.23 total 103.51 last 123.26) - open/close 15120 (time 1750593198.24 total 113.51 last 131.59) - open/close 16536 (time 1750593208.24 total 123.51 last 141.59) - open/close 17851 (time 1750593218.24 total 133.52 last 131.45) - open/close 19170 (time 1750593228.24 total 143.52 last 131.88) - open/close 20000 (time 1750593234.48 total 149.76 last 132.97) - open/close 21481 (time 1750593244.48 total 159.76 last 148.08) - open/close 22852 (time 1750593254.49 total 169.77 last 136.98) - open/close 24512 (time 1750593264.49 total 179.77 last 165.99) - open/close 26174 (time 1750593274.50 total 189.77 last 166.12) - open/close 27874 (time 1750593284.50 total 199.78 last 169.91) - open/close 29371 (time 1750593294.51 total 209.78 last 149.66) - open/close 30000 (time 1750593298.55 total 213.82 last 155.59) - open/close 31497 (time 1750593308.55 total 223.83 last 149.61) - open/close 32841 (time 1750593318.56 total 233.83 last 134.39) - open/close 34428 (time 1750593328.56 total 243.83 last 158.70) - open/close 35845 (time 1750593338.56 total 253.84 last 141.57) - open/close 37391 (time 1750593348.57 total 263.84 last 154.52) - open/close 38806 (time 1750593358.57 total 273.85 last 141.49) - open/close 40000 (time 1750593367.17 total 282.45 last 138.77) - open/close 41310 (time 1750593377.18 total 292.45 last 130.95) - open/close 42594 (time 1750593387.18 total 302.45 last 128.39) - open/close 44094 (time 1750593397.18 total 312.46 last 149.99) - open/close 45392 (time 1750593407.19 total 322.46 last 129.72) - open/close 46818 (time 1750593417.19 total 332.46 last 142.56) - open/close 48098 (time 1750593427.19 total 342.46 last 128.00) - open/close 49275 (time 1750593437.19 total 352.47 last 117.67) - open/close 50000 (time 1750593443.05 total 358.32 last 123.87) - open/close 51351 (time 1750593453.05 total 368.32 last 135.09) - open/close 52603 (time 1750593463.05 total 378.32 last 125.16) - open/close 53811 (time 1750593473.05 total 388.33 last 120.76) - open/close 55283 (time 1750593483.06 total 398.33 last 147.10) - open/close 56546 (time 1750593493.06 total 408.34 last 126.27) - open/close 57810 (time 1750593503.07 total 418.34 last 126.33) - open/close 59176 (time 1750593513.08 total 428.35 last 136.45) - open/close 60000 (time 1750593519.95 total 435.23 last 119.81) - open/close 61317 (time 1750593529.96 total 445.23 last 131.69) - open/close 62665 (time 1750593539.96 total 455.23 last 134.75) - open/close 63916 (time 1750593549.97 total 465.25 last 124.92) total: 64768 open/close in 472.08 seconds: 137.20 ops/second - /unlink 2285 (time 1750593568.66 total 10.00 last 228.44) - /unlink 4620 (time 1750593578.66 total 20.00 last 233.48) - /unlink 7443 (time 1750593588.66 total 30.00 last 282.29) - /unlink 9535 (time 1750593598.67 total 40.00 last 209.18) - /unlink 10000 (time 1750593600.88 total 42.21 last 210.41) - /unlink 12085 (time 1750593610.88 total 52.22 last 208.42) - /unlink 14612 (time 1750593620.88 total 62.22 last 252.66) - /unlink 16966 (time 1750593630.88 total 72.22 last 235.39) - /unlink 19529 (time 1750593640.88 total 82.22 last 256.23) - /unlink 20000 (time 1750593642.96 total 84.30 last 226.90) - /unlink 22260 (time 1750593652.96 total 94.30 last 225.97) - /unlink 24895 (time 1750593662.96 total 104.30 last 263.47) - /unlink 27211 (time 1750593672.97 total 114.31 last 231.42) - /unlink 29547 (time 1750593682.98 total 124.32 last 233.45) - /unlink 31902 (time 1750593692.98 total 134.32 last 235.47) - /unlink 33977 (time 1750593702.98 total 144.32 last 207.50) - /unlink 36366 (time 1750593712.98 total 154.32 last 238.89) - /unlink 38613 (time 1750593722.98 total 164.32 last 224.60) - /unlink 40000 (time 1750593729.61 total 170.95 last 209.35) - /unlink 42214 (time 1750593739.61 total 180.95 last 221.32) - /unlink 44372 (time 1750593749.64 total 190.98 last 215.28) - /unlink 46643 (time 1750593759.64 total 200.98 last 227.04) - /unlink 48880 (time 1750593769.64 total 210.98 last 223.67) - /unlink 50000 (time 1750593774.22 total 215.56 last 244.47) - /unlink 52253 (time 1750593784.23 total 225.57 last 225.16) - /unlink 54506 (time 1750593794.23 total 235.57 last 225.23) - /unlink 56919 (time 1750593804.23 total 245.57 last 241.29) - /unlink 59211 (time 1750593814.23 total 255.57 last 229.18) - /unlink 60000 (time 1750593817.90 total 259.24 last 214.88) - /unlink 62256 (time 1750593827.91 total 269.25 last 225.55) total: 64768 /unlink in 279.22 seconds: 231.96 ops/second Waiting for MDT destroys to complete - open/close 1345 (time 1750593857.24 total 10.00 last 134.48) - open/close 2703 (time 1750593867.24 total 20.01 last 135.75) - open/close 4051 (time 1750593877.24 total 30.01 last 134.79) - open/close 5380 (time 1750593887.24 total 40.01 last 132.87) - open/close 6708 (time 1750593897.25 total 50.01 last 132.76) - open/close 8042 (time 1750593907.25 total 60.01 last 133.36) - open/close 9263 (time 1750593917.25 total 70.02 last 122.07) - open/close 10000 (time 1750593922.95 total 75.72 last 129.22) - open/close 11321 (time 1750593932.96 total 85.72 last 132.06) - open/close 12578 (time 1750593942.96 total 95.73 last 125.62) - open/close 13990 (time 1750593952.97 total 105.73 last 141.18) - open/close 15507 (time 1750593962.97 total 115.74 last 151.59) - open/close 16911 (time 1750593972.98 total 125.74 last 140.37) - open/close 18564 (time 1750593982.98 total 135.74 last 165.27) - open/close 19737 (time 1750593992.98 total 145.74 last 117.29) - open/close 20000 (time 1750593995.01 total 147.77 last 129.68) - open/close 21254 (time 1750594005.02 total 157.78 last 125.29) - open/close 22562 (time 1750594015.02 total 167.78 last 130.77) - open/close 23883 (time 1750594025.02 total 177.79 last 132.05) - open/close 25321 (time 1750594035.02 total 187.79 last 143.79) - open/close 26625 (time 1750594045.03 total 197.79 last 130.34) - open/close 27883 (time 1750594055.03 total 207.79 last 125.78) - open/close 29306 (time 1750594065.03 total 217.80 last 142.27) - open/close 30000 (time 1750594069.79 total 222.56 last 145.84) - open/close 31411 (time 1750594079.79 total 232.56 last 141.10) - open/close 32912 (time 1750594089.79 total 242.56 last 150.08) - open/close 34525 (time 1750594099.79 total 252.56 last 161.24) - open/close 36031 (time 1750594109.80 total 262.57 last 150.50) - open/close 37540 (time 1750594119.81 total 272.57 last 150.83) - open/close 38960 (time 1750594129.81 total 282.57 last 141.95) - open/close 40000 (time 1750594136.69 total 289.45 last 151.18) - open/close 41411 (time 1750594146.69 total 299.46 last 141.03) - open/close 42954 (time 1750594156.70 total 309.47 last 154.21) - open/close 44313 (time 1750594166.70 total 319.47 last 135.83) - open/close 45760 (time 1750594176.71 total 329.47 last 144.68) - open/close 47249 (time 1750594186.71 total 339.48 last 148.82) - open/close 48725 (time 1750594196.71 total 349.48 last 147.55) - open/close 50000 (time 1750594205.36 total 358.12 last 147.53) - open/close 51611 (time 1750594215.36 total 368.13 last 161.03) - open/close 53106 (time 1750594225.36 total 378.13 last 149.50) - open/close 54599 (time 1750594235.36 total 388.13 last 149.29) - open/close 56226 (time 1750594245.37 total 398.13 last 162.62) - open/close 57765 (time 1750594255.37 total 408.14 last 153.85) - open/close 59148 (time 1750594265.37 total 418.14 last 138.25) - open/close 60000 (time 1750594270.84 total 423.61 last 155.80) - open/close 61449 (time 1750594280.85 total 433.61 last 144.81) - open/close 62923 (time 1750594290.86 total 443.62 last 147.26) - open/close 64418 (time 1750594300.86 total 453.63 last 149.45) total: 64768 open/close in 455.74 seconds: 142.12 ops/second - /unlink 2680 (time 1750594314.57 total 10.00 last 268.00) - /unlink 5193 (time 1750594324.57 total 20.00 last 251.26) - /unlink 8087 (time 1750594334.58 total 30.00 last 289.33) - /unlink 10000 (time 1750594341.63 total 37.06 last 271.20) - /unlink 12470 (time 1750594351.63 total 47.06 last 246.96) - /unlink 14806 (time 1750594361.63 total 57.06 last 233.56) - /unlink 17252 (time 1750594371.63 total 67.06 last 244.59) - /unlink 19622 (time 1750594381.64 total 77.06 last 236.98) - /unlink 22083 (time 1750594391.64 total 87.06 last 246.10) - /unlink 24652 (time 1750594401.64 total 97.07 last 256.69) - /unlink 27114 (time 1750594411.65 total 107.07 last 246.12) - /unlink 29690 (time 1750594421.65 total 117.08 last 257.55) - /unlink 32607 (time 1750594431.65 total 127.08 last 291.66) - /unlink 35582 (time 1750594441.65 total 137.08 last 297.48) - /unlink 38447 (time 1750594451.65 total 147.08 last 286.48) - /unlink 40000 (time 1750594457.02 total 152.45 last 289.40) - /unlink 42804 (time 1750594467.02 total 162.45 last 280.26) - /unlink 45757 (time 1750594477.02 total 172.45 last 295.30) - /unlink 48545 (time 1750594487.03 total 182.45 last 278.72) - /unlink 50000 (time 1750594492.69 total 188.11 last 257.11) - /unlink 52444 (time 1750594502.69 total 198.12 last 244.31) - /unlink 54984 (time 1750594512.70 total 208.12 last 253.84) - /unlink 57615 (time 1750594522.70 total 218.12 last 263.06) - /unlink 60000 (time 1750594531.64 total 227.07 last 266.54) - /unlink 62727 (time 1750594541.65 total 237.07 last 272.66) total: 64768 /unlink in 244.80 seconds: 264.57 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg124-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg124-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg124-server stop mds service on oleg124-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg124-server stop mds service on oleg124-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg124-server unloading modules via unload_modules_local on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg124-client: oleg124-client: ssh exited with exit code 2 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 2 pdsh@oleg124-client: oleg124-client: ssh exited with exit code 2 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 2 PASS 106 (3714s) Stopping clients: oleg124-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg124-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg124-client: oleg124-server: ssh exited with exit code 2 oleg124-server: oleg124-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing load_modules_local oleg124-server: Loading modules from /home/green/git/lustre-release/lustre oleg124-server: detected 4 online CPUs by sysfs oleg124-server: MODOPTS_LIBCFS= oleg124-server: Force libcfs to create 2 CPU partitions oleg124-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg124-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3868 sec ========== 08:16:58 (1750594618) === conf-sanity: start cleanup 08:16:59 (1750594619) === === conf-sanity: finish cleanup 08:17:00 (1750594620) ===