-----============= acceptance-small: conf-sanity ============----- Thu May 1 17:07:38 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg306-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg306-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg306-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg306-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg306-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg306-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg306-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg306-server oleg306-server: oleg306-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing load_modules_local oleg306-server: Loading modules from /home/green/git/lustre-release/lustre oleg306-server: detected 4 online CPUs by sysfs oleg306-server: MODOPTS_LIBCFS= oleg306-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg306-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg306-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg306-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg306-server: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg306-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg306-server: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg306-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg306-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg306-server stop mds service on oleg306-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg306-server stop mds service on oleg306-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg306-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:09:24 (1746133764) Stopping clients: oleg306-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg306-client.virtnet /mnt/lustre2 (opts:-f) oleg306-server: oleg306-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing load_modules_local oleg306-server: Loading modules from /home/green/git/lustre-release/lustre oleg306-server: detected 4 online CPUs by sysfs oleg306-server: MODOPTS_LIBCFS= oleg306-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg306-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg306-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg306-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1116 (time 1746133828.26 total 10.00 last 111.60) - open/close 2370 (time 1746133838.26 total 20.00 last 125.39) - open/close 3614 (time 1746133848.26 total 30.01 last 124.31) - open/close 4927 (time 1746133858.27 total 40.01 last 131.23) - open/close 6229 (time 1746133868.28 total 50.03 last 130.03) - open/close 7650 (time 1746133878.28 total 60.03 last 142.08) - open/close 9023 (time 1746133888.29 total 70.03 last 137.23) - open/close 10000 (time 1746133895.97 total 77.72 last 127.12) - open/close 11410 (time 1746133905.98 total 87.72 last 140.94) - open/close 12753 (time 1746133915.98 total 97.73 last 134.24) - open/close 14109 (time 1746133925.98 total 107.73 last 135.60) - open/close 15481 (time 1746133935.99 total 117.74 last 137.08) - open/close 16820 (time 1746133946.00 total 127.74 last 133.80) - open/close 18034 (time 1746133956.01 total 137.75 last 121.31) - open/close 19176 (time 1746133966.01 total 147.75 last 114.17) - open/close 20000 (time 1746133973.04 total 154.78 last 117.26) - open/close 21208 (time 1746133983.04 total 164.78 last 120.79) - open/close 22355 (time 1746133993.04 total 174.79 last 114.65) - open/close 23352 (time 1746134003.06 total 184.80 last 99.51) - open/close 24390 (time 1746134013.06 total 194.81 last 103.75) - open/close 25549 (time 1746134023.06 total 204.81 last 115.90) - open/close 26645 (time 1746134033.07 total 214.81 last 109.60) - open/close 27688 (time 1746134043.07 total 224.82 last 104.21) - open/close 28723 (time 1746134053.08 total 234.83 last 103.41) - open/close 29847 (time 1746134063.09 total 244.84 last 112.31) - open/close 30979 (time 1746134073.09 total 254.84 last 113.18) - open/close 32130 (time 1746134083.09 total 264.84 last 115.08) - open/close 33146 (time 1746134093.11 total 274.85 last 101.45) - open/close 34274 (time 1746134103.11 total 284.86 last 112.76) - open/close 35274 (time 1746134113.12 total 294.86 last 99.97) - open/close 36363 (time 1746134123.12 total 304.86 last 108.86) - open/close 37487 (time 1746134133.12 total 314.87 last 112.35) - open/close 38561 (time 1746134143.13 total 324.87 last 107.35) - open/close 39667 (time 1746134153.13 total 334.88 last 110.54) - open/close 40000 (time 1746134156.07 total 337.82 last 113.32) - open/close 41265 (time 1746134166.07 total 347.82 last 126.48) - open/close 42482 (time 1746134176.07 total 357.82 last 121.70) - open/close 43685 (time 1746134186.08 total 367.82 last 120.26) - open/close 44933 (time 1746134196.09 total 377.83 last 124.69) - open/close 46110 (time 1746134206.09 total 387.83 last 117.66) - open/close 47355 (time 1746134216.10 total 397.84 last 124.38) - open/close 48514 (time 1746134226.10 total 407.85 last 115.83) - open/close 49658 (time 1746134236.11 total 417.86 last 114.28) - open/close 50000 (time 1746134238.91 total 420.65 last 122.36) - open/close 51299 (time 1746134248.91 total 430.66 last 129.84) - open/close 52337 (time 1746134258.92 total 440.66 last 103.73) - open/close 53482 (time 1746134268.93 total 450.67 last 114.43) - open/close 54791 (time 1746134278.93 total 460.67 last 130.89) - open/close 56097 (time 1746134288.93 total 470.68 last 130.55) - open/close 57441 (time 1746134298.94 total 480.69 last 134.27) - open/close 58737 (time 1746134308.95 total 490.69 last 129.52) - open/close 60000 (time 1746134318.88 total 500.62 last 127.18) - open/close 61282 (time 1746134328.89 total 510.63 last 128.08) - open/close 62595 (time 1746134338.89 total 520.63 last 131.27) - open/close 63891 (time 1746134348.89 total 530.64 last 129.57) total: 64768 open/close in 537.33 seconds: 120.54 ops/second - /unlink 2208 (time 1746134367.39 total 10.00 last 220.78) - /unlink 4243 (time 1746134377.39 total 20.00 last 203.44) - /unlink 6341 (time 1746134387.40 total 30.01 last 209.74) - /unlink 8300 (time 1746134397.40 total 40.01 last 195.81) - /unlink 10000 (time 1746134406.20 total 48.81 last 193.11) - /unlink 11912 (time 1746134416.21 total 58.82 last 191.10) - /unlink 14161 (time 1746134426.21 total 68.82 last 224.90) - /unlink 16846 (time 1746134436.21 total 78.82 last 268.46) - /unlink 18966 (time 1746134446.21 total 88.82 last 211.97) - /unlink 20000 (time 1746134450.85 total 93.46 last 223.10) - /unlink 21932 (time 1746134460.85 total 103.46 last 193.06) - /unlink 23848 (time 1746134470.86 total 113.47 last 191.55) - /unlink 25957 (time 1746134480.86 total 123.47 last 210.89) - /unlink 28063 (time 1746134490.86 total 133.47 last 210.48) - /unlink 30000 (time 1746134500.12 total 142.73 last 209.27) - /unlink 32049 (time 1746134510.12 total 152.73 last 204.88) - /unlink 34202 (time 1746134520.13 total 162.74 last 215.20) - /unlink 36131 (time 1746134530.13 total 172.74 last 192.80) - /unlink 37995 (time 1746134540.13 total 182.74 last 186.33) - /unlink 39800 (time 1746134550.14 total 192.75 last 180.47) - /unlink 41509 (time 1746134560.14 total 202.75 last 170.88) - /unlink 43306 (time 1746134570.14 total 212.75 last 179.64) - /unlink 45183 (time 1746134580.15 total 222.76 last 187.61) - /unlink 47265 (time 1746134590.15 total 232.76 last 208.15) - /unlink 49780 (time 1746134600.15 total 242.76 last 251.46) - /unlink 52109 (time 1746134610.15 total 252.76 last 232.85) - /unlink 54238 (time 1746134620.15 total 262.76 last 212.89) - /unlink 56512 (time 1746134630.15 total 272.76 last 227.36) - /unlink 58850 (time 1746134640.16 total 282.77 last 233.74) - /unlink 60000 (time 1746134645.38 total 287.99 last 220.05) - /unlink 62390 (time 1746134655.39 total 298.00 last 238.87) total: 64768 /unlink in 306.60 seconds: 211.24 ops/second Waiting for MDT destroys to complete - open/close 1456 (time 1746134683.61 total 10.00 last 145.56) - open/close 2946 (time 1746134693.61 total 20.00 last 148.99) - open/close 4489 (time 1746134703.61 total 30.01 last 154.27) - open/close 5835 (time 1746134713.61 total 40.01 last 134.57) - open/close 7284 (time 1746134723.62 total 50.01 last 144.86) - open/close 8769 (time 1746134733.62 total 60.01 last 148.50) - open/close 9968 (time 1746134743.62 total 70.02 last 119.82) - open/close 11529 (time 1746134753.63 total 80.03 last 155.92) - open/close 13058 (time 1746134763.63 total 90.03 last 152.88) - open/close 14466 (time 1746134773.63 total 100.03 last 140.80) - open/close 15821 (time 1746134783.64 total 110.03 last 135.43) - open/close 17335 (time 1746134793.64 total 120.04 last 151.35) - open/close 18960 (time 1746134803.65 total 130.04 last 162.42) - open/close 20000 (time 1746134811.25 total 137.64 last 136.82) - open/close 21467 (time 1746134821.25 total 147.64 last 146.70) - open/close 23049 (time 1746134831.25 total 157.64 last 158.19) - open/close 24630 (time 1746134841.25 total 167.65 last 158.08) - open/close 26179 (time 1746134851.25 total 177.65 last 154.89) - open/close 27613 (time 1746134861.25 total 187.65 last 143.40) - open/close 29039 (time 1746134871.25 total 197.65 last 142.59) - open/close 30000 (time 1746134876.50 total 202.89 last 183.27) - open/close 31876 (time 1746134886.50 total 212.89 last 187.59) - open/close 33465 (time 1746134896.50 total 222.89 last 158.90) - open/close 34970 (time 1746134906.50 total 232.89 last 150.48) - open/close 36259 (time 1746134916.50 total 242.90 last 128.88) - open/close 37571 (time 1746134926.50 total 252.90 last 131.16) - open/close 39390 (time 1746134936.51 total 262.90 last 181.81) - open/close 40000 (time 1746134940.07 total 266.46 last 171.27) - open/close 41624 (time 1746134950.07 total 276.47 last 162.34) - open/close 43073 (time 1746134960.08 total 286.47 last 144.86) - open/close 44440 (time 1746134970.08 total 296.47 last 136.68) - open/close 45691 (time 1746134980.09 total 306.48 last 124.98) - open/close 46977 (time 1746134990.10 total 316.49 last 128.48) - open/close 48364 (time 1746135000.10 total 326.50 last 138.62) - open/close 49819 (time 1746135010.11 total 336.50 last 145.40) - open/close 51345 (time 1746135020.11 total 346.51 last 152.56) - open/close 52823 (time 1746135030.12 total 356.51 last 147.72) - open/close 54341 (time 1746135040.12 total 366.52 last 151.75) - open/close 55544 (time 1746135050.13 total 376.52 last 120.24) - open/close 56851 (time 1746135060.13 total 386.52 last 130.67) - open/close 58302 (time 1746135070.13 total 396.53 last 145.01) - open/close 59662 (time 1746135080.14 total 406.53 last 135.93) - open/close 60000 (time 1746135082.32 total 408.71 last 155.21) - open/close 61569 (time 1746135092.33 total 418.73 last 156.69) - open/close 62931 (time 1746135102.34 total 428.74 last 136.04) - open/close 64367 (time 1746135112.35 total 438.74 last 143.56) total: 64768 open/close in 441.41 seconds: 146.73 ops/second - /unlink 2339 (time 1746135126.62 total 10.00 last 233.89) - /unlink 4809 (time 1746135136.63 total 20.00 last 246.93) - /unlink 7305 (time 1746135146.63 total 30.00 last 249.57) - /unlink 9639 (time 1746135156.63 total 40.01 last 233.35) - /unlink 11785 (time 1746135166.63 total 50.01 last 214.52) - /unlink 13887 (time 1746135176.63 total 60.01 last 210.19) - /unlink 16193 (time 1746135186.63 total 70.01 last 230.59) - /unlink 18273 (time 1746135196.63 total 80.01 last 207.98) - /unlink 20000 (time 1746135204.12 total 87.49 last 230.83) - /unlink 22372 (time 1746135214.12 total 97.50 last 237.09) - /unlink 24979 (time 1746135224.12 total 107.50 last 260.60) - /unlink 27264 (time 1746135234.13 total 117.50 last 228.48) - /unlink 29416 (time 1746135244.13 total 127.51 last 215.11) - /unlink 30000 (time 1746135246.91 total 130.29 last 209.98) - /unlink 32429 (time 1746135256.91 total 140.29 last 242.87) - /unlink 34744 (time 1746135266.91 total 150.29 last 231.46) - /unlink 36932 (time 1746135276.92 total 160.29 last 218.77) - /unlink 39248 (time 1746135286.92 total 170.30 last 231.53) - /unlink 40000 (time 1746135290.20 total 173.58 last 228.93) - /unlink 42201 (time 1746135300.21 total 183.58 last 220.02) - /unlink 44326 (time 1746135310.21 total 193.58 last 212.49) - /unlink 46656 (time 1746135320.21 total 203.59 last 232.97) - /unlink 48927 (time 1746135330.21 total 213.59 last 226.98) - /unlink 50000 (time 1746135335.07 total 218.45 last 220.96) - /unlink 52229 (time 1746135345.08 total 228.45 last 222.77) - /unlink 54225 (time 1746135355.08 total 238.46 last 199.50) - /unlink 56531 (time 1746135365.08 total 248.46 last 230.52) - /unlink 58818 (time 1746135375.09 total 258.46 last 228.66) - /unlink 60000 (time 1746135380.34 total 263.72 last 225.06) - /unlink 62440 (time 1746135390.34 total 273.72 last 244.00) - /unlink 64695 (time 1746135400.34 total 283.72 last 225.50) total: 64768 /unlink in 284.04 seconds: 228.02 ops/second Waiting for MDT destroys to complete - open/close 1548 (time 1746135424.02 total 10.00 last 154.75) - open/close 3104 (time 1746135434.02 total 20.01 last 155.57) - open/close 4492 (time 1746135444.03 total 30.01 last 138.76) - open/close 5801 (time 1746135454.03 total 40.01 last 130.88) - open/close 7124 (time 1746135464.03 total 50.01 last 132.27) - open/close 8363 (time 1746135474.04 total 60.02 last 123.81) - open/close 9606 (time 1746135484.04 total 70.02 last 124.26) - open/close 10000 (time 1746135487.01 total 72.99 last 132.69) - open/close 11398 (time 1746135497.01 total 82.99 last 139.79) - open/close 12677 (time 1746135507.01 total 92.99 last 127.89) - open/close 13953 (time 1746135517.02 total 103.00 last 127.52) - open/close 15241 (time 1746135527.02 total 113.00 last 128.79) - open/close 16589 (time 1746135537.02 total 123.00 last 134.75) - open/close 17786 (time 1746135547.02 total 133.01 last 119.68) - open/close 19086 (time 1746135557.03 total 143.01 last 129.95) - open/close 20000 (time 1746135564.03 total 150.01 last 130.51) - open/close 21319 (time 1746135574.04 total 160.02 last 131.82) - open/close 22756 (time 1746135584.04 total 170.02 last 143.66) - open/close 24100 (time 1746135594.05 total 180.03 last 134.32) - open/close 25419 (time 1746135604.05 total 190.03 last 131.87) - open/close 26914 (time 1746135614.05 total 200.03 last 149.46) - open/close 28217 (time 1746135624.06 total 210.04 last 130.27) - open/close 29598 (time 1746135634.06 total 220.04 last 138.08) - open/close 30000 (time 1746135637.05 total 223.03 last 134.36) - open/close 31396 (time 1746135647.05 total 233.03 last 139.58) - open/close 32634 (time 1746135657.06 total 243.04 last 123.74) - open/close 33866 (time 1746135667.06 total 253.04 last 123.13) - open/close 35246 (time 1746135677.06 total 263.04 last 137.99) - open/close 36623 (time 1746135687.06 total 273.04 last 137.70) - open/close 38000 (time 1746135697.07 total 283.05 last 137.62) - open/close 39257 (time 1746135707.07 total 293.05 last 125.68) - open/close 40000 (time 1746135712.30 total 298.28 last 142.00) - open/close 41492 (time 1746135722.31 total 308.29 last 149.11) - open/close 42676 (time 1746135732.31 total 318.29 last 118.39) - open/close 43837 (time 1746135742.31 total 328.29 last 116.07) - open/close 45048 (time 1746135752.31 total 338.29 last 121.10) - open/close 46273 (time 1746135762.31 total 348.29 last 122.47) - open/close 47462 (time 1746135772.32 total 358.30 last 118.84) - open/close 48769 (time 1746135782.32 total 368.30 last 130.67) - open/close 49957 (time 1746135792.33 total 378.31 last 118.69) - open/close 51142 (time 1746135802.34 total 388.32 last 118.39) - open/close 52410 (time 1746135812.34 total 398.32 last 126.76) - open/close 53660 (time 1746135822.34 total 408.33 last 124.98) - open/close 54943 (time 1746135832.35 total 418.33 last 128.22) - open/close 56127 (time 1746135842.36 total 428.34 last 118.34) - open/close 57417 (time 1746135852.36 total 438.35 last 128.90) - open/close 58805 (time 1746135862.37 total 448.35 last 138.72) - open/close 60000 (time 1746135871.11 total 457.09 last 136.79) - open/close 61362 (time 1746135881.11 total 467.09 last 136.12) - open/close 62494 (time 1746135891.11 total 477.09 last 113.19) - open/close 63681 (time 1746135901.12 total 487.11 last 118.57) total: 64768 open/close in 496.03 seconds: 130.57 ops/second - /unlink 2146 (time 1746135921.62 total 10.00 last 214.52) - /unlink 4661 (time 1746135931.63 total 20.01 last 251.46) - /unlink 7323 (time 1746135941.63 total 30.01 last 266.15) - /unlink 9612 (time 1746135951.63 total 40.01 last 228.77) - /unlink 11909 (time 1746135961.64 total 50.01 last 229.66) - /unlink 14169 (time 1746135971.64 total 60.02 last 225.94) - /unlink 16867 (time 1746135981.64 total 70.02 last 269.75) - /unlink 19824 (time 1746135991.65 total 80.02 last 295.55) - /unlink 23111 (time 1746136001.65 total 90.03 last 328.65) - /unlink 25793 (time 1746136011.65 total 100.03 last 268.04) - /unlink 28337 (time 1746136021.65 total 110.03 last 254.39) - /unlink 30000 (time 1746136029.07 total 117.45 last 224.13) - /unlink 32473 (time 1746136039.08 total 127.45 last 247.25) - /unlink 35075 (time 1746136049.08 total 137.45 last 260.19) - /unlink 37479 (time 1746136059.08 total 147.46 last 240.23) - /unlink 39985 (time 1746136069.08 total 157.46 last 250.58) - /unlink 42418 (time 1746136079.09 total 167.47 last 243.22) - /unlink 44665 (time 1746136089.09 total 177.47 last 224.68) - /unlink 46710 (time 1746136099.09 total 187.47 last 204.49) - /unlink 48922 (time 1746136109.09 total 197.47 last 221.17) - /unlink 50000 (time 1746136113.96 total 202.34 last 221.21) - /unlink 52214 (time 1746136123.96 total 212.34 last 221.40) - /unlink 54446 (time 1746136133.96 total 222.34 last 223.19) - /unlink 56686 (time 1746136143.97 total 232.34 last 223.97) - /unlink 58857 (time 1746136153.98 total 242.36 last 216.72) - /unlink 60000 (time 1746136159.16 total 247.53 last 220.94) - /unlink 62368 (time 1746136169.16 total 257.54 last 236.73) - /unlink 64681 (time 1746136179.16 total 267.54 last 231.28) total: 64768 /unlink in 267.93 seconds: 241.73 ops/second Waiting for MDT destroys to complete - open/close 1444 (time 1746136195.20 total 10.00 last 144.33) - open/close 2860 (time 1746136205.20 total 20.01 last 141.59) - open/close 4185 (time 1746136215.20 total 30.01 last 132.49) - open/close 6001 (time 1746136225.21 total 40.01 last 181.49) - open/close 7543 (time 1746136235.21 total 50.01 last 154.20) - open/close 9007 (time 1746136245.21 total 60.01 last 146.40) - open/close 10000 (time 1746136251.31 total 66.11 last 162.86) - open/close 11600 (time 1746136261.31 total 76.11 last 159.95) - open/close 13013 (time 1746136271.31 total 86.12 last 141.25) - open/close 14424 (time 1746136281.32 total 96.12 last 141.01) - open/close 15751 (time 1746136291.33 total 106.14 last 132.53) - open/close 17249 (time 1746136301.34 total 116.14 last 149.74) - open/close 18770 (time 1746136311.35 total 126.15 last 151.93) - open/close 20000 (time 1746136319.76 total 134.56 last 146.30) - open/close 21662 (time 1746136329.76 total 144.56 last 166.14) - open/close 23241 (time 1746136339.76 total 154.57 last 157.84) - open/close 24720 (time 1746136349.77 total 164.57 last 147.81) - open/close 26018 (time 1746136359.77 total 174.58 last 129.74) - open/close 27367 (time 1746136369.78 total 184.58 last 134.81) - open/close 28588 (time 1746136379.78 total 194.58 last 122.09) - open/close 29888 (time 1746136389.79 total 204.59 last 129.93) - open/close 31313 (time 1746136399.79 total 214.59 last 142.49) - open/close 32821 (time 1746136409.79 total 224.59 last 150.79) - open/close 34256 (time 1746136419.79 total 234.59 last 143.49) - open/close 35565 (time 1746136429.79 total 244.59 last 130.87) - open/close 37008 (time 1746136439.79 total 254.59 last 144.29) - open/close 38614 (time 1746136449.80 total 264.60 last 160.51) - open/close 40000 (time 1746136458.69 total 273.49 last 155.97) - open/close 41761 (time 1746136468.69 total 283.49 last 176.08) - open/close 43803 (time 1746136478.69 total 293.49 last 204.09) - open/close 45488 (time 1746136488.69 total 303.49 last 168.48) - open/close 47075 (time 1746136498.70 total 313.50 last 158.56) - open/close 48290 (time 1746136508.70 total 323.51 last 121.48) - open/close 49425 (time 1746136518.71 total 333.51 last 113.40) - open/close 50000 (time 1746136523.22 total 338.03 last 127.44) - open/close 51375 (time 1746136533.23 total 348.03 last 137.46) - open/close 52770 (time 1746136543.23 total 358.03 last 139.50) - open/close 54240 (time 1746136553.23 total 368.03 last 146.97) - open/close 55593 (time 1746136563.24 total 378.04 last 135.20) - open/close 56997 (time 1746136573.24 total 388.05 last 140.30) - open/close 58431 (time 1746136583.25 total 398.05 last 143.38) - open/close 59848 (time 1746136593.25 total 408.05 last 141.69) - open/close 61380 (time 1746136603.25 total 418.05 last 153.12) - open/close 63160 (time 1746136613.26 total 428.06 last 177.93) - open/close 64455 (time 1746136623.26 total 438.06 last 129.42) total: 64768 open/close in 440.87 seconds: 146.91 ops/second - /unlink 1887 (time 1746136638.10 total 10.01 last 188.59) - /unlink 3910 (time 1746136648.10 total 20.01 last 202.26) - /unlink 6099 (time 1746136658.11 total 30.01 last 218.83) - /unlink 8379 (time 1746136668.11 total 40.01 last 227.95) - /unlink 10000 (time 1746136676.52 total 48.43 last 192.63) - /unlink 12099 (time 1746136686.52 total 58.43 last 209.87) - /unlink 14530 (time 1746136696.53 total 68.43 last 243.00) - /unlink 17136 (time 1746136706.53 total 78.44 last 260.50) - /unlink 19370 (time 1746136716.53 total 88.44 last 223.39) - /unlink 20000 (time 1746136719.43 total 91.33 last 217.74) - /unlink 22179 (time 1746136729.43 total 101.33 last 217.83) - /unlink 24306 (time 1746136739.44 total 111.34 last 212.58) - /unlink 26726 (time 1746136749.44 total 121.34 last 241.96) - /unlink 28951 (time 1746136759.44 total 131.34 last 222.46) - /unlink 30000 (time 1746136764.74 total 136.64 last 197.98) - /unlink 31857 (time 1746136774.74 total 146.65 last 185.62) - /unlink 33887 (time 1746136784.75 total 156.65 last 202.91) - /unlink 35758 (time 1746136794.75 total 166.66 last 186.99) - /unlink 37649 (time 1746136804.75 total 176.66 last 189.09) - /unlink 39567 (time 1746136814.76 total 186.66 last 191.71) - /unlink 40000 (time 1746136816.80 total 188.70 last 212.12) - /unlink 41977 (time 1746136826.80 total 198.71 last 197.57) - /unlink 44208 (time 1746136836.80 total 208.71 last 223.09) - /unlink 46102 (time 1746136846.81 total 218.72 last 189.30) - /unlink 48037 (time 1746136856.81 total 228.72 last 193.42) - /unlink 50000 (time 1746136866.52 total 238.43 last 202.25) - /unlink 52096 (time 1746136876.52 total 248.43 last 209.54) - /unlink 54347 (time 1746136886.52 total 258.43 last 225.09) - /unlink 56408 (time 1746136896.53 total 268.43 last 206.03) - /unlink 58772 (time 1746136906.53 total 278.43 last 236.33) - /unlink 60000 (time 1746136912.32 total 284.22 last 212.19) - /unlink 62003 (time 1746136922.32 total 294.22 last 200.26) - /unlink 64140 (time 1746136932.32 total 304.23 last 213.65) total: 64768 /unlink in 307.37 seconds: 210.71 ops/second Waiting for MDT destroys to complete - open/close 1221 (time 1746136957.18 total 10.00 last 122.08) - open/close 2426 (time 1746136967.18 total 20.01 last 120.43) - open/close 3635 (time 1746136977.19 total 30.02 last 120.81) - open/close 4845 (time 1746136987.20 total 40.02 last 120.89) - open/close 5951 (time 1746136997.20 total 50.03 last 110.55) - open/close 7139 (time 1746137007.20 total 60.03 last 118.79) - open/close 8237 (time 1746137017.22 total 70.04 last 109.63) - open/close 9300 (time 1746137027.23 total 80.05 last 106.20) - open/close 10000 (time 1746137033.36 total 86.18 last 114.26) - open/close 11192 (time 1746137043.37 total 96.19 last 119.03) - open/close 12399 (time 1746137053.38 total 106.20 last 120.61) - open/close 13635 (time 1746137063.38 total 116.20 last 123.56) - open/close 14874 (time 1746137073.39 total 126.21 last 123.83) - open/close 16054 (time 1746137083.39 total 136.21 last 117.95) - open/close 17363 (time 1746137093.40 total 146.22 last 130.82) - open/close 18527 (time 1746137103.40 total 156.22 last 116.36) - open/close 19622 (time 1746137113.40 total 166.23 last 109.46) - open/close 20000 (time 1746137116.72 total 169.54 last 114.05) - open/close 21087 (time 1746137126.72 total 179.55 last 108.63) - open/close 22214 (time 1746137136.73 total 189.55 last 112.68) - open/close 23262 (time 1746137146.73 total 199.55 last 104.80) - open/close 24412 (time 1746137156.73 total 209.56 last 114.91) - open/close 25531 (time 1746137166.74 total 219.56 last 111.87) - open/close 26718 (time 1746137176.74 total 229.56 last 118.65) - open/close 27817 (time 1746137186.74 total 239.57 last 109.86) - open/close 28937 (time 1746137196.75 total 249.57 last 111.93) - open/close 29984 (time 1746137206.76 total 259.58 last 104.61) - open/close 31164 (time 1746137216.76 total 269.58 last 117.99) - open/close 32217 (time 1746137226.76 total 279.59 last 105.29) - open/close 33498 (time 1746137236.76 total 289.59 last 128.06) - open/close 34646 (time 1746137246.76 total 299.59 last 114.78) - open/close 35871 (time 1746137256.77 total 309.59 last 122.44) - open/close 37088 (time 1746137266.77 total 319.59 last 121.69) - open/close 38363 (time 1746137276.77 total 329.60 last 127.45) - open/close 39504 (time 1746137286.79 total 339.62 last 113.89) - open/close 40000 (time 1746137291.15 total 343.97 last 113.93) - open/close 41200 (time 1746137301.18 total 354.00 last 119.61) - open/close 42295 (time 1746137311.18 total 364.01 last 109.48) - open/close 43409 (time 1746137321.18 total 374.01 last 111.39) - open/close 44497 (time 1746137331.18 total 384.01 last 108.79) - open/close 45603 (time 1746137341.19 total 394.01 last 110.56) - open/close 46551 (time 1746137351.20 total 404.02 last 94.70) - open/close 47576 (time 1746137361.20 total 414.02 last 102.47) - open/close 48592 (time 1746137371.21 total 424.03 last 101.53) - open/close 49704 (time 1746137381.21 total 434.04 last 111.16) - open/close 50000 (time 1746137383.69 total 436.52 last 119.22) - open/close 51115 (time 1746137393.70 total 446.52 last 111.48) - open/close 52208 (time 1746137403.70 total 456.53 last 109.24) - open/close 53333 (time 1746137413.70 total 466.53 last 112.49) - open/close 54383 (time 1746137423.70 total 476.53 last 104.98) - open/close 55445 (time 1746137433.70 total 486.53 last 106.20) - open/close 56561 (time 1746137443.71 total 496.53 last 111.54) - open/close 57665 (time 1746137453.71 total 506.54 last 110.38) - open/close 58762 (time 1746137463.71 total 516.54 last 109.67) - open/close 59945 (time 1746137473.72 total 526.54 last 118.25) - open/close 61157 (time 1746137483.73 total 536.55 last 121.11) - open/close 62355 (time 1746137493.74 total 546.56 last 119.69) - open/close 63813 (time 1746137503.74 total 556.56 last 145.79) total: 64768 open/close in 562.71 seconds: 115.10 ops/second - /unlink 2432 (time 1746137521.30 total 10.00 last 243.12) - /unlink 5910 (time 1746137531.30 total 20.00 last 347.77) - /unlink 9080 (time 1746137541.30 total 30.00 last 316.99) - /unlink 10000 (time 1746137543.67 total 32.38 last 388.06) - /unlink 13020 (time 1746137553.67 total 42.38 last 301.99) - /unlink 16721 (time 1746137563.67 total 52.38 last 370.04) - /unlink 19388 (time 1746137573.68 total 62.38 last 266.56) - /unlink 20000 (time 1746137576.24 total 64.95 last 238.40) - /unlink 22904 (time 1746137586.24 total 74.95 last 290.32) - /unlink 25246 (time 1746137596.25 total 84.95 last 234.16) - /unlink 27427 (time 1746137606.25 total 94.96 last 218.08) - /unlink 30000 (time 1746137615.87 total 104.58 last 267.31) - /unlink 32516 (time 1746137625.88 total 114.58 last 251.52) - /unlink 35568 (time 1746137635.88 total 124.59 last 305.07) - /unlink 38317 (time 1746137645.88 total 134.59 last 274.83) - /unlink 40000 (time 1746137651.79 total 140.49 last 285.10) - /unlink 42701 (time 1746137661.79 total 150.50 last 270.04) - /unlink 45351 (time 1746137671.79 total 160.50 last 264.86) - /unlink 48060 (time 1746137681.80 total 170.51 last 270.75) - /unlink 50000 (time 1746137690.46 total 179.16 last 224.10) - /unlink 52909 (time 1746137700.46 total 189.17 last 290.85) - /unlink 55988 (time 1746137710.46 total 199.17 last 307.89) - /unlink 58821 (time 1746137720.46 total 209.17 last 283.24) - /unlink 60000 (time 1746137725.85 total 214.55 last 218.94) - /unlink 62424 (time 1746137735.85 total 224.55 last 242.36) total: 64768 /unlink in 233.54 seconds: 277.34 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg306-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg306-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg306-server stop mds service on oleg306-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg306-server stop mds service on oleg306-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg306-server unloading modules via unload_modules_local on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg306-client: oleg306-server: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-client: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-client: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 2 PASS 106 (4016s) Stopping clients: oleg306-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg306-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg306-client: oleg306-server: ssh exited with exit code 2 oleg306-server: oleg306-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing load_modules_local oleg306-server: Loading modules from /home/green/git/lustre-release/lustre oleg306-server: detected 4 online CPUs by sysfs oleg306-server: MODOPTS_LIBCFS= oleg306-server: Force libcfs to create 2 CPU partitions oleg306-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg306-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 4148 sec ========== 18:16:47 (1746137807) === conf-sanity: start cleanup 18:16:47 (1746137807) === === conf-sanity: finish cleanup 18:16:49 (1746137809) ===