-----============= acceptance-small: conf-sanity ============----- Thu May 1 17:39:37 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg313-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg313-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg313-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg313-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg313-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg313-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg313-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg313-server oleg313-server: oleg313-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing load_modules_local oleg313-server: Loading modules from /home/green/git/lustre-release/lustre oleg313-server: detected 4 online CPUs by sysfs oleg313-server: MODOPTS_LIBCFS= oleg313-server: Force libcfs to create 2 CPU partitions oleg313-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg313-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg313-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg313-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg313-server: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg313-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg313-server: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg313-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg313-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg313-server stop mds service on oleg313-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg313-server stop mds service on oleg313-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg313-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:41:33 (1746135693) Stopping clients: oleg313-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg313-client.virtnet /mnt/lustre2 (opts:-f) oleg313-server: oleg313-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing load_modules_local oleg313-server: Loading modules from /home/green/git/lustre-release/lustre oleg313-server: detected 4 online CPUs by sysfs oleg313-server: MODOPTS_LIBCFS= oleg313-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg313-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg313-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg313-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg313-client.virtnet: -o user_xattr,flock 192.168.203.113@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1368 (time 1746135756.58 total 10.01 last 136.69) - open/close 2570 (time 1746135766.58 total 20.01 last 120.14) - open/close 3909 (time 1746135776.58 total 30.02 last 133.85) - open/close 5190 (time 1746135786.59 total 40.02 last 128.08) - open/close 6444 (time 1746135796.59 total 50.03 last 125.31) - open/close 7755 (time 1746135806.60 total 60.03 last 131.07) - open/close 9161 (time 1746135816.60 total 70.03 last 140.51) - open/close 10000 (time 1746135823.02 total 76.45 last 130.74) - open/close 11246 (time 1746135833.03 total 86.46 last 124.51) - open/close 12473 (time 1746135843.03 total 96.46 last 122.66) - open/close 13912 (time 1746135853.05 total 106.48 last 143.66) - open/close 15401 (time 1746135863.05 total 116.48 last 148.86) - open/close 16857 (time 1746135873.06 total 126.49 last 145.47) - open/close 18263 (time 1746135883.06 total 136.49 last 140.54) - open/close 19516 (time 1746135893.06 total 146.50 last 125.26) - open/close 20000 (time 1746135896.88 total 150.31 last 126.98) - open/close 21356 (time 1746135906.89 total 160.32 last 135.41) - open/close 22679 (time 1746135916.89 total 170.33 last 132.26) - open/close 24164 (time 1746135926.90 total 180.34 last 148.37) - open/close 25724 (time 1746135936.90 total 190.34 last 155.99) - open/close 27245 (time 1746135946.91 total 200.34 last 152.00) - open/close 28735 (time 1746135956.91 total 210.34 last 148.99) - open/close 30000 (time 1746135966.07 total 219.50 last 138.10) - open/close 31458 (time 1746135976.07 total 229.50 last 145.77) - open/close 33144 (time 1746135986.07 total 239.51 last 168.59) - open/close 35115 (time 1746135996.07 total 249.51 last 197.04) - open/close 36777 (time 1746136006.08 total 259.51 last 166.11) - open/close 38297 (time 1746136016.08 total 269.52 last 151.96) - open/close 39624 (time 1746136026.10 total 279.53 last 132.54) - open/close 40000 (time 1746136028.54 total 281.97 last 154.09) - open/close 41411 (time 1746136038.54 total 291.97 last 141.03) - open/close 42839 (time 1746136048.55 total 301.98 last 142.72) - open/close 44210 (time 1746136058.56 total 311.99 last 136.96) - open/close 45584 (time 1746136068.56 total 322.00 last 137.29) - open/close 46939 (time 1746136078.57 total 332.00 last 135.45) - open/close 48170 (time 1746136088.57 total 342.00 last 123.09) - open/close 49336 (time 1746136098.57 total 352.01 last 116.55) - open/close 50000 (time 1746136104.24 total 357.67 last 117.28) - open/close 51225 (time 1746136114.24 total 367.67 last 122.43) - open/close 52621 (time 1746136124.24 total 377.68 last 139.55) - open/close 53864 (time 1746136134.25 total 387.68 last 124.26) - open/close 55149 (time 1746136144.26 total 397.69 last 128.36) - open/close 56400 (time 1746136154.27 total 407.70 last 125.03) - open/close 57726 (time 1746136164.27 total 417.70 last 132.59) - open/close 59046 (time 1746136174.27 total 427.70 last 131.99) - open/close 60000 (time 1746136181.84 total 435.27 last 125.95) - open/close 61401 (time 1746136191.84 total 445.28 last 140.08) - open/close 62842 (time 1746136201.85 total 455.28 last 144.05) - open/close 64101 (time 1746136211.86 total 465.30 last 125.70) total: 64768 open/close in 470.37 seconds: 137.70 ops/second - /unlink 3124 (time 1746136228.40 total 10.01 last 312.22) - /unlink 5892 (time 1746136238.40 total 20.01 last 276.74) - /unlink 8306 (time 1746136248.41 total 30.01 last 241.36) - /unlink 10000 (time 1746136254.42 total 36.02 last 281.87) - /unlink 12587 (time 1746136264.42 total 46.02 last 258.60) - /unlink 15149 (time 1746136274.42 total 56.02 last 256.19) - /unlink 17435 (time 1746136284.42 total 66.03 last 228.54) - /unlink 19696 (time 1746136294.42 total 76.03 last 226.07) - /unlink 22478 (time 1746136304.43 total 86.03 last 278.18) - /unlink 25052 (time 1746136314.43 total 96.03 last 257.35) - /unlink 27997 (time 1746136324.43 total 106.03 last 294.42) - /unlink 30000 (time 1746136331.21 total 112.81 last 295.37) - /unlink 32892 (time 1746136341.21 total 122.82 last 289.19) - /unlink 35416 (time 1746136351.21 total 132.82 last 252.39) - /unlink 37674 (time 1746136361.21 total 142.82 last 225.78) - /unlink 39891 (time 1746136371.21 total 152.82 last 221.66) - /unlink 42125 (time 1746136381.22 total 162.82 last 223.27) - /unlink 44432 (time 1746136391.22 total 172.82 last 230.69) - /unlink 46991 (time 1746136401.23 total 182.83 last 255.79) - /unlink 49729 (time 1746136411.23 total 192.83 last 273.77) - /unlink 52168 (time 1746136421.23 total 202.83 last 243.83) - /unlink 54543 (time 1746136431.24 total 212.84 last 237.32) - /unlink 57295 (time 1746136441.24 total 222.84 last 275.15) - /unlink 60000 (time 1746136450.42 total 232.02 last 294.59) - /unlink 63033 (time 1746136460.42 total 242.03 last 303.22) total: 64768 /unlink in 247.09 seconds: 262.12 ops/second Waiting for MDT destroys to complete - open/close 2073 (time 1746136481.17 total 10.00 last 207.20) - open/close 3738 (time 1746136491.18 total 20.01 last 166.37) - open/close 5274 (time 1746136501.19 total 30.02 last 153.51) - open/close 6508 (time 1746136511.19 total 40.02 last 123.33) - open/close 7784 (time 1746136521.20 total 50.03 last 127.51) - open/close 9234 (time 1746136531.20 total 60.03 last 144.96) - open/close 10000 (time 1746136536.37 total 65.20 last 148.31) - open/close 11425 (time 1746136546.37 total 75.20 last 142.46) - open/close 12976 (time 1746136556.37 total 85.20 last 155.04) - open/close 14508 (time 1746136566.37 total 95.21 last 153.19) - open/close 16099 (time 1746136576.38 total 105.21 last 159.08) - open/close 17722 (time 1746136586.38 total 115.21 last 162.22) - open/close 19268 (time 1746136596.38 total 125.21 last 154.57) - open/close 20000 (time 1746136600.94 total 129.77 last 160.54) - open/close 21882 (time 1746136610.95 total 139.78 last 188.14) - open/close 23355 (time 1746136620.95 total 149.78 last 147.20) - open/close 24540 (time 1746136630.96 total 159.80 last 118.37) - open/close 25744 (time 1746136640.97 total 169.80 last 120.32) - open/close 27030 (time 1746136650.98 total 179.81 last 128.50) - open/close 28309 (time 1746136660.98 total 189.81 last 127.86) - open/close 29565 (time 1746136670.99 total 199.82 last 125.52) - open/close 30000 (time 1746136674.86 total 203.69 last 112.45) - open/close 31202 (time 1746136684.86 total 213.69 last 120.14) - open/close 32519 (time 1746136694.87 total 223.70 last 131.55) - open/close 34100 (time 1746136704.88 total 233.71 last 158.06) - open/close 35427 (time 1746136714.88 total 243.71 last 132.63) - open/close 36692 (time 1746136724.88 total 253.71 last 126.48) - open/close 37955 (time 1746136734.88 total 263.72 last 126.28) - open/close 39286 (time 1746136744.89 total 273.72 last 133.06) - open/close 40000 (time 1746136749.50 total 278.33 last 154.75) - open/close 41242 (time 1746136759.50 total 288.33 last 124.19) - open/close 42426 (time 1746136769.52 total 298.35 last 118.24) - open/close 43622 (time 1746136779.52 total 308.35 last 119.52) - open/close 44770 (time 1746136789.55 total 318.39 last 114.42) - open/close 45875 (time 1746136799.57 total 328.40 last 110.34) - open/close 46981 (time 1746136809.57 total 338.40 last 110.59) - open/close 48214 (time 1746136819.58 total 348.41 last 123.23) - open/close 49476 (time 1746136829.58 total 358.41 last 126.11) - open/close 50000 (time 1746136833.85 total 362.68 last 122.92) - open/close 51249 (time 1746136843.85 total 372.68 last 124.89) - open/close 52411 (time 1746136853.85 total 382.68 last 116.16) - open/close 53573 (time 1746136863.85 total 392.68 last 116.17) - open/close 54784 (time 1746136873.86 total 402.69 last 121.04) - open/close 56115 (time 1746136883.86 total 412.69 last 133.05) - open/close 57353 (time 1746136893.86 total 422.70 last 123.75) - open/close 58706 (time 1746136903.87 total 432.70 last 135.29) - open/close 59972 (time 1746136913.87 total 442.70 last 126.60) - open/close 61141 (time 1746136923.87 total 452.70 last 116.81) - open/close 62354 (time 1746136933.88 total 462.71 last 121.26) - open/close 63446 (time 1746136943.89 total 472.72 last 109.09) - open/close 64584 (time 1746136953.89 total 482.73 last 113.72) total: 64768 open/close in 484.39 seconds: 133.71 ops/second - /unlink 2134 (time 1746136967.43 total 10.00 last 213.35) - /unlink 4109 (time 1746136977.44 total 20.01 last 197.42) - /unlink 6047 (time 1746136987.44 total 30.01 last 193.75) - /unlink 7995 (time 1746136997.45 total 40.01 last 194.70) - /unlink 10000 (time 1746137006.88 total 49.45 last 212.56) - /unlink 11859 (time 1746137016.88 total 59.45 last 185.89) - /unlink 13739 (time 1746137026.88 total 69.45 last 187.98) - /unlink 15516 (time 1746137036.89 total 79.45 last 177.62) - /unlink 17531 (time 1746137046.89 total 89.45 last 201.50) - /unlink 19517 (time 1746137056.89 total 99.45 last 198.58) - /unlink 20000 (time 1746137059.07 total 101.64 last 221.31) - /unlink 22081 (time 1746137069.07 total 111.64 last 208.08) - /unlink 23974 (time 1746137079.07 total 121.64 last 189.27) - /unlink 26156 (time 1746137089.07 total 131.64 last 218.20) - /unlink 28273 (time 1746137099.07 total 141.64 last 211.68) - /unlink 30000 (time 1746137108.54 total 151.11 last 182.38) - /unlink 31816 (time 1746137118.55 total 161.12 last 181.49) - /unlink 33677 (time 1746137128.55 total 171.12 last 185.98) - /unlink 35643 (time 1746137138.56 total 181.12 last 196.59) - /unlink 37403 (time 1746137148.56 total 191.13 last 175.94) - /unlink 39465 (time 1746137158.56 total 201.13 last 206.15) - /unlink 40000 (time 1746137161.30 total 203.87 last 194.99) - /unlink 42042 (time 1746137171.31 total 213.87 last 204.18) - /unlink 44147 (time 1746137181.31 total 223.87 last 210.46) - /unlink 46139 (time 1746137191.31 total 233.88 last 199.18) - /unlink 47997 (time 1746137201.31 total 243.88 last 185.73) - /unlink 50000 (time 1746137210.91 total 253.48 last 208.70) - /unlink 52010 (time 1746137220.92 total 263.48 last 200.88) - /unlink 54022 (time 1746137230.92 total 273.49 last 201.08) - /unlink 56404 (time 1746137240.92 total 283.49 last 238.14) - /unlink 58380 (time 1746137250.93 total 293.50 last 197.41) - /unlink 60000 (time 1746137257.69 total 300.26 last 239.87) - /unlink 62198 (time 1746137267.69 total 310.26 last 219.78) - /unlink 64460 (time 1746137277.69 total 320.26 last 226.10) total: 64768 /unlink in 322.30 seconds: 200.96 ops/second Waiting for MDT destroys to complete - open/close 1232 (time 1746137299.49 total 10.00 last 123.17) - open/close 2372 (time 1746137309.50 total 20.01 last 113.93) - open/close 3504 (time 1746137319.57 total 30.08 last 112.39) - open/close 4634 (time 1746137329.58 total 40.09 last 112.88) - open/close 5794 (time 1746137339.58 total 50.09 last 116.00) - open/close 6844 (time 1746137349.59 total 60.10 last 104.91) - open/close 7902 (time 1746137359.59 total 70.10 last 105.79) - open/close 9037 (time 1746137369.60 total 80.11 last 113.43) - open/close 10000 (time 1746137378.77 total 89.28 last 105.01) - open/close 11169 (time 1746137388.78 total 99.28 last 116.83) - open/close 12372 (time 1746137398.78 total 109.28 last 120.30) - open/close 13577 (time 1746137408.78 total 119.29 last 120.47) - open/close 14822 (time 1746137418.79 total 129.30 last 124.29) - open/close 15896 (time 1746137428.80 total 139.31 last 107.37) - open/close 17030 (time 1746137438.80 total 149.31 last 113.38) - open/close 18252 (time 1746137448.80 total 159.31 last 122.19) - open/close 19427 (time 1746137458.81 total 169.31 last 117.43) - open/close 20000 (time 1746137463.63 total 174.13 last 118.88) - open/close 21272 (time 1746137473.63 total 184.14 last 127.13) - open/close 22562 (time 1746137483.64 total 194.14 last 128.94) - open/close 23857 (time 1746137493.64 total 204.15 last 129.42) - open/close 25405 (time 1746137503.65 total 214.15 last 154.73) - open/close 26929 (time 1746137513.65 total 224.16 last 152.31) - open/close 28649 (time 1746137523.66 total 234.16 last 171.94) - open/close 30000 (time 1746137530.90 total 241.40 last 186.58) - open/close 31883 (time 1746137540.90 total 251.40 last 188.29) - open/close 33627 (time 1746137550.90 total 261.41 last 174.35) - open/close 35878 (time 1746137560.90 total 271.41 last 225.03) - open/close 37601 (time 1746137570.91 total 281.41 last 172.26) - open/close 39083 (time 1746137580.91 total 291.42 last 148.07) - open/close 40000 (time 1746137585.89 total 296.40 last 184.11) - open/close 41344 (time 1746137595.89 total 306.40 last 134.39) - open/close 42688 (time 1746137605.90 total 316.41 last 134.37) - open/close 44254 (time 1746137615.90 total 326.41 last 156.52) - open/close 45697 (time 1746137625.90 total 336.41 last 144.29) - open/close 47396 (time 1746137635.91 total 346.41 last 169.86) - open/close 48897 (time 1746137645.91 total 356.42 last 149.99) - open/close 50000 (time 1746137653.08 total 363.59 last 153.81) - open/close 51581 (time 1746137663.08 total 373.59 last 158.09) - open/close 53120 (time 1746137673.09 total 383.59 last 153.88) - open/close 54578 (time 1746137683.09 total 393.60 last 145.73) - open/close 55852 (time 1746137693.09 total 403.60 last 127.36) - open/close 57650 (time 1746137703.10 total 413.60 last 179.73) - open/close 59307 (time 1746137713.10 total 423.61 last 165.64) - open/close 60000 (time 1746137716.95 total 427.46 last 180.12) - open/close 61302 (time 1746137726.96 total 437.47 last 130.05) - open/close 62776 (time 1746137736.96 total 447.47 last 147.40) - open/close 64304 (time 1746137746.97 total 457.48 last 152.66) total: 64768 open/close in 460.14 seconds: 140.76 ops/second - /unlink 2480 (time 1746137761.08 total 10.00 last 247.91) - /unlink 5686 (time 1746137771.08 total 20.00 last 320.59) - /unlink 9159 (time 1746137781.08 total 30.00 last 347.29) - /unlink 10000 (time 1746137783.89 total 32.82 last 299.03) - /unlink 13521 (time 1746137793.89 total 42.82 last 352.07) - /unlink 16249 (time 1746137803.89 total 52.82 last 272.77) - /unlink 19864 (time 1746137813.89 total 62.82 last 361.49) - /unlink 22725 (time 1746137823.90 total 72.82 last 285.99) - /unlink 25711 (time 1746137833.90 total 82.82 last 298.54) - /unlink 28894 (time 1746137843.90 total 92.83 last 318.24) - /unlink 30000 (time 1746137848.12 total 97.04 last 262.38) - /unlink 32228 (time 1746137858.12 total 107.04 last 222.79) - /unlink 35173 (time 1746137868.12 total 117.05 last 294.39) - /unlink 38700 (time 1746137878.12 total 127.05 last 352.69) - /unlink 40000 (time 1746137881.86 total 130.78 last 347.80) - /unlink 43402 (time 1746137891.86 total 140.78 last 340.19) - /unlink 46022 (time 1746137901.86 total 150.79 last 261.85) - /unlink 48306 (time 1746137911.87 total 160.79 last 228.36) - /unlink 50000 (time 1746137919.52 total 168.45 last 221.32) - /unlink 52549 (time 1746137929.53 total 178.45 last 254.72) - /unlink 55344 (time 1746137939.53 total 188.46 last 279.42) - /unlink 58838 (time 1746137949.53 total 198.46 last 349.37) - /unlink 60000 (time 1746137953.60 total 202.53 last 285.31) - /unlink 62283 (time 1746137963.61 total 212.53 last 228.20) total: 64768 /unlink in 220.94 seconds: 293.15 ops/second Waiting for MDT destroys to complete - open/close 1494 (time 1746137988.21 total 10.00 last 149.40) - open/close 2988 (time 1746137998.22 total 20.00 last 149.34) - open/close 4296 (time 1746138008.22 total 30.01 last 130.77) - open/close 5893 (time 1746138018.22 total 40.01 last 159.67) - open/close 7479 (time 1746138028.22 total 50.01 last 158.59) - open/close 9247 (time 1746138038.22 total 60.01 last 176.78) - open/close 10000 (time 1746138041.67 total 63.46 last 218.28) - open/close 12327 (time 1746138051.68 total 73.46 last 232.65) - open/close 14381 (time 1746138061.68 total 83.47 last 205.29) - open/close 15996 (time 1746138071.69 total 93.48 last 161.36) - open/close 17380 (time 1746138081.70 total 103.49 last 138.26) - open/close 18684 (time 1746138091.71 total 113.50 last 130.26) - open/close 20000 (time 1746138101.04 total 122.82 last 141.10) - open/close 21710 (time 1746138111.05 total 132.83 last 170.86) - open/close 23381 (time 1746138121.05 total 142.83 last 167.09) - open/close 24801 (time 1746138131.05 total 152.83 last 141.98) - open/close 26469 (time 1746138141.05 total 162.84 last 166.75) - open/close 28080 (time 1746138151.05 total 172.84 last 161.08) - open/close 30000 (time 1746138160.57 total 182.36 last 201.75) - open/close 32498 (time 1746138170.57 total 192.36 last 249.77) - open/close 35141 (time 1746138180.57 total 202.36 last 264.25) - open/close 37685 (time 1746138190.58 total 212.36 last 254.30) - open/close 39953 (time 1746138200.58 total 222.37 last 226.74) - open/close 41997 (time 1746138210.58 total 232.37 last 204.38) - open/close 43611 (time 1746138220.58 total 242.37 last 161.35) - open/close 45747 (time 1746138230.58 total 252.37 last 213.59) - open/close 47639 (time 1746138240.59 total 262.37 last 189.15) - open/close 49603 (time 1746138250.59 total 272.38 last 196.24) - open/close 50000 (time 1746138253.43 total 275.22 last 139.84) - open/close 52088 (time 1746138263.44 total 285.22 last 208.73) - open/close 53771 (time 1746138273.44 total 295.23 last 168.26) - open/close 55887 (time 1746138283.44 total 305.23 last 211.60) - open/close 57505 (time 1746138293.44 total 315.23 last 161.79) - open/close 59244 (time 1746138303.44 total 325.23 last 173.89) - open/close 60000 (time 1746138307.53 total 329.31 last 185.03) - open/close 61487 (time 1746138317.53 total 339.31 last 148.70) - open/close 62838 (time 1746138327.53 total 349.32 last 135.04) - open/close 64164 (time 1746138337.53 total 359.32 last 132.57) total: 64768 open/close in 364.54 seconds: 177.67 ops/second - /unlink 2548 (time 1746138354.76 total 10.00 last 254.75) - /unlink 5174 (time 1746138364.76 total 20.00 last 262.54) - /unlink 7804 (time 1746138374.76 total 30.01 last 262.94) - /unlink 10000 (time 1746138381.63 total 36.87 last 319.89) - /unlink 13682 (time 1746138391.63 total 46.87 last 368.16) - /unlink 17535 (time 1746138401.63 total 56.87 last 385.25) - /unlink 20000 (time 1746138408.87 total 64.11 last 340.59) - /unlink 22425 (time 1746138418.87 total 74.11 last 242.46) - /unlink 24796 (time 1746138428.87 total 84.11 last 237.08) - /unlink 26914 (time 1746138438.87 total 94.12 last 211.73) - /unlink 29626 (time 1746138448.88 total 104.12 last 271.14) - /unlink 31702 (time 1746138458.88 total 114.12 last 207.55) - /unlink 34057 (time 1746138468.88 total 124.12 last 235.47) - /unlink 36324 (time 1746138478.88 total 134.12 last 226.66) - /unlink 38514 (time 1746138488.88 total 144.13 last 218.96) - /unlink 40000 (time 1746138495.43 total 150.68 last 226.87) - /unlink 42274 (time 1746138505.44 total 160.68 last 227.36) - /unlink 44647 (time 1746138515.44 total 170.68 last 237.27) - /unlink 47924 (time 1746138525.44 total 180.68 last 327.69) - /unlink 50000 (time 1746138531.66 total 186.90 last 333.76) - /unlink 52951 (time 1746138541.66 total 196.90 last 295.06) - /unlink 57178 (time 1746138551.66 total 206.90 last 422.61) - /unlink 60000 (time 1746138558.32 total 213.56 last 423.76) - /unlink 64406 (time 1746138568.32 total 223.56 last 440.52) total: 64768 /unlink in 224.38 seconds: 288.66 ops/second Waiting for MDT destroys to complete - open/close 1881 (time 1746138584.75 total 10.00 last 188.10) - open/close 3742 (time 1746138594.76 total 20.00 last 186.02) - open/close 5559 (time 1746138604.76 total 30.01 last 181.68) - open/close 6926 (time 1746138614.76 total 40.01 last 136.69) - open/close 8341 (time 1746138624.76 total 50.01 last 141.49) - open/close 9570 (time 1746138634.76 total 60.01 last 122.86) - open/close 10000 (time 1746138638.28 total 63.53 last 122.20) - open/close 11316 (time 1746138648.29 total 73.53 last 131.53) - open/close 12603 (time 1746138658.29 total 83.54 last 128.66) - open/close 13937 (time 1746138668.29 total 93.54 last 133.35) - open/close 15221 (time 1746138678.29 total 103.54 last 128.39) - open/close 16450 (time 1746138688.30 total 113.55 last 122.84) - open/close 17788 (time 1746138698.30 total 123.55 last 133.74) - open/close 19002 (time 1746138708.31 total 133.56 last 121.33) - open/close 20000 (time 1746138716.47 total 141.71 last 122.37) - open/close 21346 (time 1746138726.47 total 151.72 last 134.50) - open/close 22579 (time 1746138736.47 total 161.72 last 123.27) - open/close 23912 (time 1746138746.48 total 171.72 last 133.29) - open/close 25145 (time 1746138756.48 total 181.73 last 123.23) - open/close 26342 (time 1746138766.49 total 191.74 last 119.61) - open/close 27662 (time 1746138776.49 total 201.74 last 131.92) - open/close 28837 (time 1746138786.50 total 211.75 last 117.44) - open/close 30000 (time 1746138794.92 total 220.17 last 138.11) - open/close 31159 (time 1746138804.92 total 230.17 last 115.86) - open/close 32251 (time 1746138814.92 total 240.17 last 109.19) - open/close 33372 (time 1746138824.92 total 250.17 last 112.09) - open/close 34452 (time 1746138834.93 total 260.18 last 107.96) - open/close 35741 (time 1746138844.93 total 270.18 last 128.89) - open/close 37004 (time 1746138854.93 total 280.18 last 126.29) - open/close 38421 (time 1746138864.93 total 290.18 last 141.67) - open/close 39626 (time 1746138874.93 total 300.18 last 120.48) - open/close 40000 (time 1746138878.26 total 303.51 last 112.31) - open/close 41276 (time 1746138888.27 total 313.51 last 127.58) - open/close 42562 (time 1746138898.27 total 323.51 last 128.60) - open/close 43751 (time 1746138908.28 total 333.52 last 118.80) - open/close 44857 (time 1746138918.28 total 343.53 last 110.53) - open/close 45967 (time 1746138928.30 total 353.54 last 110.83) - open/close 47195 (time 1746138938.30 total 363.55 last 122.76) - open/close 48320 (time 1746138948.30 total 373.55 last 112.47) - open/close 49524 (time 1746138958.30 total 383.55 last 120.37) - open/close 50000 (time 1746138962.74 total 387.98 last 107.38) - open/close 51078 (time 1746138972.74 total 397.99 last 107.77) - open/close 52229 (time 1746138982.74 total 407.99 last 115.05) - open/close 53317 (time 1746138992.75 total 417.99 last 108.78) - open/close 54515 (time 1746139002.75 total 427.99 last 119.78) - open/close 55742 (time 1746139012.77 total 438.02 last 122.44) - open/close 56824 (time 1746139022.78 total 448.02 last 108.13) - open/close 58068 (time 1746139032.78 total 458.03 last 124.35) - open/close 59293 (time 1746139042.78 total 468.03 last 122.47) - open/close 60000 (time 1746139048.93 total 474.17 last 115.08) - open/close 61198 (time 1746139058.93 total 484.18 last 119.75) - open/close 62449 (time 1746139068.93 total 494.18 last 125.09) - open/close 63703 (time 1746139078.94 total 504.18 last 125.32) total: 64768 open/close in 513.10 seconds: 126.23 ops/second - /unlink 1756 (time 1746139100.12 total 10.01 last 175.51) - /unlink 3537 (time 1746139110.12 total 20.01 last 178.07) - /unlink 5652 (time 1746139120.12 total 30.01 last 211.47) - /unlink 8126 (time 1746139130.12 total 40.01 last 247.33) - /unlink 10000 (time 1746139137.63 total 47.52 last 249.73) - /unlink 11897 (time 1746139147.63 total 57.52 last 189.67) - /unlink 14237 (time 1746139157.63 total 67.52 last 233.99) - /unlink 16623 (time 1746139167.63 total 77.52 last 238.57) - /unlink 19014 (time 1746139177.63 total 87.52 last 239.05) - /unlink 20000 (time 1746139181.99 total 91.88 last 226.21) - /unlink 22214 (time 1746139192.00 total 101.88 last 221.28) - /unlink 24668 (time 1746139202.00 total 111.89 last 245.29) - /unlink 26973 (time 1746139212.00 total 121.89 last 230.45) - /unlink 29064 (time 1746139222.01 total 131.89 last 209.05) - /unlink 30000 (time 1746139226.46 total 136.35 last 209.95) - /unlink 32388 (time 1746139236.47 total 146.35 last 238.74) - /unlink 34860 (time 1746139246.47 total 156.36 last 247.16) - /unlink 37671 (time 1746139256.47 total 166.36 last 281.04) - /unlink 40000 (time 1746139264.14 total 174.03 last 303.50) - /unlink 42412 (time 1746139274.14 total 184.03 last 241.20) - /unlink 45045 (time 1746139284.15 total 194.04 last 263.24) - /unlink 47878 (time 1746139294.15 total 204.04 last 283.22) - /unlink 50000 (time 1746139302.87 total 212.76 last 243.21) - /unlink 52371 (time 1746139312.88 total 222.77 last 237.03) - /unlink 54753 (time 1746139322.88 total 232.77 last 238.20) - /unlink 57059 (time 1746139332.88 total 242.77 last 230.60) - /unlink 59874 (time 1746139342.88 total 252.77 last 281.45) - /unlink 62586 (time 1746139352.88 total 262.77 last 271.15) total: 64768 /unlink in 270.61 seconds: 239.34 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg313-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg313-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg313-server stop mds service on oleg313-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg313-server stop mds service on oleg313-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg313-server unloading modules via unload_modules_local on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg313-client: oleg313-client: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-client: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2 PASS 106 (3703s) Stopping clients: oleg313-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg313-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2 oleg313-server: oleg313-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing load_modules_local oleg313-server: Loading modules from /home/green/git/lustre-release/lustre oleg313-server: detected 4 online CPUs by sysfs oleg313-server: MODOPTS_LIBCFS= oleg313-server: Force libcfs to create 2 CPU partitions oleg313-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory oleg313-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg313-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3851 sec ========== 18:43:48 (1746139428) === conf-sanity: start cleanup 18:43:48 (1746139428) === === conf-sanity: finish cleanup 18:43:49 (1746139429) ===