-----============= acceptance-small: conf-sanity ============----- Thu May 1 21:01:46 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg352-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg352-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg352-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg352-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg352-server oleg352-server: oleg352-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg352-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg352-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg352-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg352-server: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg352-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg352-server: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg352-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg352-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 21:03:26 (1746147806) Stopping clients: oleg352-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg352-client.virtnet /mnt/lustre2 (opts:-f) oleg352-server: oleg352-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg352-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg352-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg352-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg352-client.virtnet: -o user_xattr,flock 192.168.203.152@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1403 (time 1746147870.77 total 10.01 last 140.20) - open/close 2762 (time 1746147880.78 total 20.02 last 135.78) - open/close 4071 (time 1746147890.79 total 30.02 last 130.87) - open/close 5553 (time 1746147900.79 total 40.02 last 148.14) - open/close 6947 (time 1746147910.79 total 50.03 last 139.34) - open/close 8432 (time 1746147920.80 total 60.03 last 148.46) - open/close 9934 (time 1746147930.80 total 70.03 last 150.16) - open/close 11500 (time 1746147940.81 total 80.04 last 156.47) - open/close 12945 (time 1746147950.81 total 90.04 last 144.43) - open/close 14353 (time 1746147960.82 total 100.05 last 140.76) - open/close 15948 (time 1746147970.82 total 110.05 last 159.43) - open/close 17157 (time 1746147980.82 total 120.06 last 120.86) - open/close 18610 (time 1746147990.83 total 130.06 last 145.23) - open/close 20000 (time 1746148000.37 total 139.61 last 145.59) - open/close 21493 (time 1746148010.38 total 149.61 last 149.23) - open/close 22889 (time 1746148020.38 total 159.62 last 139.56) - open/close 24175 (time 1746148030.39 total 169.62 last 128.53) - open/close 25490 (time 1746148040.39 total 179.62 last 131.49) - open/close 27016 (time 1746148050.39 total 189.62 last 152.57) - open/close 28376 (time 1746148060.40 total 199.63 last 135.89) - open/close 29557 (time 1746148070.40 total 209.63 last 118.07) - open/close 30000 (time 1746148074.54 total 213.77 last 107.08) - open/close 31191 (time 1746148084.54 total 223.77 last 119.08) - open/close 32336 (time 1746148094.55 total 233.78 last 114.45) - open/close 33499 (time 1746148104.55 total 243.78 last 116.28) - open/close 34702 (time 1746148114.55 total 253.78 last 120.29) - open/close 35910 (time 1746148124.56 total 263.79 last 120.64) - open/close 37106 (time 1746148134.58 total 273.81 last 119.41) - open/close 38176 (time 1746148144.58 total 283.82 last 106.95) - open/close 39529 (time 1746148154.60 total 293.83 last 135.06) - open/close 40000 (time 1746148158.17 total 297.40 last 132.03) - open/close 41397 (time 1746148168.17 total 307.40 last 139.66) - open/close 42629 (time 1746148178.18 total 317.41 last 123.14) - open/close 44020 (time 1746148188.18 total 327.41 last 139.06) - open/close 45152 (time 1746148198.18 total 337.41 last 113.16) - open/close 46215 (time 1746148208.18 total 347.42 last 106.28) - open/close 47588 (time 1746148218.18 total 357.42 last 137.28) - open/close 48683 (time 1746148228.19 total 367.42 last 109.45) - open/close 49704 (time 1746148238.20 total 377.43 last 102.01) - open/close 50000 (time 1746148241.09 total 380.32 last 102.52) - open/close 51118 (time 1746148251.09 total 390.32 last 111.78) - open/close 52344 (time 1746148261.09 total 400.33 last 122.51) - open/close 53351 (time 1746148271.10 total 410.33 last 100.69) - open/close 54456 (time 1746148281.10 total 420.33 last 110.45) - open/close 55733 (time 1746148291.10 total 430.33 last 127.69) - open/close 56757 (time 1746148301.10 total 440.34 last 102.38) - open/close 57773 (time 1746148311.11 total 450.35 last 101.50) - open/close 58823 (time 1746148321.12 total 460.36 last 104.90) - open/close 60000 (time 1746148330.68 total 469.91 last 123.16) - open/close 61023 (time 1746148340.72 total 479.95 last 101.93) - open/close 62016 (time 1746148350.72 total 489.95 last 99.29) - open/close 63082 (time 1746148360.73 total 499.96 last 106.49) - open/close 64327 (time 1746148370.73 total 509.96 last 124.46) total: 64768 open/close in 514.21 seconds: 125.96 ops/second - /unlink 1755 (time 1746148386.89 total 10.00 last 175.46) - /unlink 3573 (time 1746148396.89 total 20.00 last 181.76) - /unlink 5825 (time 1746148406.89 total 30.01 last 225.10) - /unlink 7661 (time 1746148416.90 total 40.01 last 183.53) - /unlink 9413 (time 1746148426.90 total 50.01 last 175.18) - /unlink 10000 (time 1746148430.04 total 53.15 last 187.14) - /unlink 12149 (time 1746148440.04 total 63.15 last 214.85) - /unlink 13833 (time 1746148450.04 total 73.15 last 168.37) - /unlink 15613 (time 1746148460.04 total 83.16 last 177.92) - /unlink 17796 (time 1746148470.05 total 93.16 last 218.26) - /unlink 19618 (time 1746148480.05 total 103.16 last 182.20) - /unlink 20000 (time 1746148482.20 total 105.32 last 177.04) - /unlink 21729 (time 1746148492.21 total 115.32 last 172.83) - /unlink 23676 (time 1746148502.22 total 125.33 last 194.54) - /unlink 25891 (time 1746148512.22 total 135.33 last 221.46) - /unlink 27641 (time 1746148522.22 total 145.34 last 174.91) - /unlink 29386 (time 1746148532.23 total 155.34 last 174.44) - /unlink 30000 (time 1746148535.34 total 158.46 last 196.91) - /unlink 32225 (time 1746148545.35 total 168.47 last 222.33) - /unlink 33911 (time 1746148555.35 total 178.47 last 168.57) - /unlink 35581 (time 1746148565.36 total 188.47 last 166.93) - /unlink 37431 (time 1746148575.36 total 198.48 last 184.89) - /unlink 39633 (time 1746148585.37 total 208.48 last 220.17) - /unlink 40000 (time 1746148587.45 total 210.56 last 176.37) - /unlink 41787 (time 1746148597.45 total 220.56 last 178.70) - /unlink 43583 (time 1746148607.45 total 230.57 last 179.49) - /unlink 45782 (time 1746148617.46 total 240.57 last 219.82) - /unlink 47600 (time 1746148627.46 total 250.57 last 181.75) - /unlink 49393 (time 1746148637.46 total 260.58 last 179.28) - /unlink 50000 (time 1746148640.62 total 263.73 last 192.16) - /unlink 52155 (time 1746148650.62 total 273.74 last 215.46) - /unlink 54408 (time 1746148660.63 total 283.74 last 225.18) - /unlink 56172 (time 1746148670.63 total 293.74 last 176.38) - /unlink 58198 (time 1746148680.63 total 303.75 last 202.49) - /unlink 60000 (time 1746148689.47 total 312.58 last 203.95) - /unlink 61850 (time 1746148699.47 total 322.59 last 184.98) - /unlink 63583 (time 1746148709.48 total 332.59 last 173.19) total: 64768 /unlink in 338.66 seconds: 191.25 ops/second Waiting for MDT destroys to complete - open/close 1172 (time 1746148736.93 total 10.01 last 117.14) - open/close 2278 (time 1746148746.93 total 20.01 last 110.59) - open/close 3377 (time 1746148756.93 total 30.01 last 109.88) - open/close 4591 (time 1746148766.94 total 40.02 last 121.27) - open/close 5823 (time 1746148776.95 total 50.02 last 123.15) - open/close 6878 (time 1746148786.95 total 60.02 last 105.49) - open/close 8016 (time 1746148796.96 total 70.03 last 113.74) - open/close 9375 (time 1746148806.96 total 80.04 last 135.79) - open/close 10000 (time 1746148811.96 total 85.03 last 125.20) - open/close 11124 (time 1746148821.96 total 95.03 last 112.39) - open/close 12387 (time 1746148831.96 total 105.03 last 126.29) - open/close 13970 (time 1746148841.96 total 115.03 last 158.27) - open/close 15270 (time 1746148851.97 total 125.04 last 129.90) - open/close 16477 (time 1746148861.97 total 135.04 last 120.69) - open/close 17761 (time 1746148871.97 total 145.05 last 128.33) - open/close 18750 (time 1746148881.99 total 155.06 last 98.75) - open/close 19823 (time 1746148891.99 total 165.07 last 107.25) - open/close 21079 (time 1746148902.00 total 175.08 last 125.47) - open/close 22261 (time 1746148912.01 total 185.08 last 118.17) - open/close 23270 (time 1746148922.01 total 195.09 last 100.82) - open/close 24394 (time 1746148932.02 total 205.10 last 112.32) - open/close 25740 (time 1746148942.02 total 215.10 last 134.59) - open/close 26815 (time 1746148952.02 total 225.10 last 107.45) - open/close 27962 (time 1746148962.03 total 235.10 last 114.67) - open/close 29339 (time 1746148972.03 total 245.10 last 137.69) - open/close 30000 (time 1746148977.47 total 250.55 last 121.42) - open/close 31220 (time 1746148987.47 total 260.55 last 122.00) - open/close 32646 (time 1746148997.48 total 270.55 last 142.56) - open/close 33958 (time 1746149007.48 total 280.55 last 131.18) - open/close 35131 (time 1746149017.48 total 290.56 last 117.24) - open/close 36364 (time 1746149027.49 total 300.56 last 123.23) - open/close 37618 (time 1746149037.49 total 310.57 last 125.34) - open/close 38724 (time 1746149047.50 total 320.58 last 110.51) - open/close 39818 (time 1746149057.50 total 330.58 last 109.39) - open/close 40990 (time 1746149067.50 total 340.58 last 117.20) - open/close 42005 (time 1746149077.51 total 350.58 last 101.48) - open/close 43043 (time 1746149087.51 total 360.59 last 103.71) - open/close 44040 (time 1746149097.52 total 370.59 last 99.67) - open/close 45201 (time 1746149107.52 total 380.60 last 116.02) - open/close 46308 (time 1746149117.53 total 390.61 last 110.61) - open/close 47384 (time 1746149127.54 total 400.61 last 107.55) - open/close 48478 (time 1746149137.54 total 410.62 last 109.32) - open/close 49653 (time 1746149147.55 total 420.62 last 117.44) - open/close 50000 (time 1746149150.47 total 423.54 last 118.93) - open/close 50948 (time 1746149160.47 total 433.55 last 94.76) - open/close 51937 (time 1746149170.48 total 443.55 last 98.86) - open/close 53109 (time 1746149180.48 total 453.55 last 117.18) - open/close 54309 (time 1746149190.48 total 463.55 last 119.99) - open/close 55334 (time 1746149200.48 total 473.56 last 102.46) - open/close 56380 (time 1746149210.48 total 483.56 last 104.59) - open/close 57511 (time 1746149220.48 total 493.56 last 113.08) - open/close 58674 (time 1746149230.49 total 503.57 last 116.21) - open/close 59683 (time 1746149240.49 total 513.57 last 100.87) - open/close 60000 (time 1746149243.39 total 516.47 last 109.47) - open/close 61110 (time 1746149253.40 total 526.47 last 110.94) - open/close 62396 (time 1746149263.41 total 536.49 last 128.40) - open/close 63429 (time 1746149273.41 total 546.49 last 103.27) - open/close 64502 (time 1746149283.42 total 556.49 last 107.29) total: 64768 open/close in 559.00 seconds: 115.86 ops/second - /unlink 2210 (time 1746149297.76 total 10.01 last 220.88) - /unlink 4156 (time 1746149307.76 total 20.01 last 194.60) - /unlink 6230 (time 1746149317.77 total 30.01 last 207.36) - /unlink 8663 (time 1746149327.77 total 40.01 last 243.25) - /unlink 10000 (time 1746149335.08 total 47.32 last 182.79) - /unlink 11833 (time 1746149345.08 total 57.32 last 183.29) - /unlink 13853 (time 1746149355.08 total 67.32 last 201.99) - /unlink 16312 (time 1746149365.09 total 77.33 last 245.80) - /unlink 18244 (time 1746149375.09 total 87.33 last 193.14) - /unlink 20000 (time 1746149384.18 total 96.42 last 193.28) - /unlink 22361 (time 1746149394.18 total 106.42 last 236.05) - /unlink 24086 (time 1746149404.18 total 116.42 last 172.40) - /unlink 25926 (time 1746149414.19 total 126.43 last 183.92) - /unlink 28125 (time 1746149424.19 total 136.43 last 219.86) - /unlink 29908 (time 1746149434.19 total 146.43 last 178.28) - /unlink 31735 (time 1746149444.19 total 156.43 last 182.70) - /unlink 33603 (time 1746149454.19 total 166.43 last 186.80) - /unlink 35791 (time 1746149464.19 total 176.43 last 218.76) - /unlink 37716 (time 1746149474.20 total 186.44 last 192.42) - /unlink 39788 (time 1746149484.20 total 196.44 last 207.16) - /unlink 41893 (time 1746149494.20 total 206.44 last 210.49) - /unlink 43831 (time 1746149504.20 total 216.44 last 193.77) - /unlink 45778 (time 1746149514.20 total 226.45 last 194.64) - /unlink 48366 (time 1746149524.21 total 236.45 last 258.75) - /unlink 50000 (time 1746149531.12 total 243.36 last 236.46) - /unlink 51945 (time 1746149541.12 total 253.36 last 194.43) - /unlink 54234 (time 1746149551.12 total 263.36 last 228.83) - /unlink 56586 (time 1746149561.13 total 273.37 last 235.12) - /unlink 58469 (time 1746149571.13 total 283.37 last 188.19) - /unlink 60000 (time 1746149579.17 total 291.41 last 190.47) - /unlink 62340 (time 1746149589.17 total 301.41 last 233.97) - /unlink 64317 (time 1746149599.18 total 311.42 last 197.56) total: 64768 /unlink in 314.12 seconds: 206.19 ops/second Waiting for MDT destroys to complete - open/close 1336 (time 1746149623.45 total 10.00 last 133.54) - open/close 2393 (time 1746149633.46 total 20.01 last 105.68) - open/close 3461 (time 1746149643.46 total 30.01 last 106.77) - open/close 4639 (time 1746149653.47 total 40.02 last 117.70) - open/close 5934 (time 1746149663.47 total 50.02 last 129.48) - open/close 6978 (time 1746149673.47 total 60.02 last 104.38) - open/close 8069 (time 1746149683.48 total 70.03 last 109.05) - open/close 9447 (time 1746149693.48 total 80.03 last 137.71) - open/close 10000 (time 1746149697.88 total 84.43 last 125.74) - open/close 11104 (time 1746149707.89 total 94.44 last 110.34) - open/close 12186 (time 1746149717.89 total 104.44 last 108.14) - open/close 13462 (time 1746149727.90 total 114.44 last 127.55) - open/close 14592 (time 1746149737.90 total 124.45 last 112.99) - open/close 15675 (time 1746149747.90 total 134.45 last 108.27) - open/close 16935 (time 1746149757.90 total 144.45 last 125.94) - open/close 18213 (time 1746149767.91 total 154.46 last 127.74) - open/close 19292 (time 1746149777.92 total 164.47 last 107.75) - open/close 20000 (time 1746149784.74 total 171.29 last 103.86) - open/close 21352 (time 1746149794.74 total 181.29 last 135.14) - open/close 22644 (time 1746149804.75 total 191.30 last 129.13) - open/close 23788 (time 1746149814.75 total 201.30 last 114.38) - open/close 25008 (time 1746149824.76 total 211.31 last 121.89) - open/close 26192 (time 1746149834.76 total 221.31 last 118.37) - open/close 27178 (time 1746149844.77 total 231.32 last 98.54) - open/close 28217 (time 1746149854.77 total 241.32 last 103.86) - open/close 29537 (time 1746149864.78 total 251.33 last 131.94) - open/close 30000 (time 1746149869.19 total 255.74 last 105.01) - open/close 31050 (time 1746149879.19 total 265.74 last 104.95) - open/close 32267 (time 1746149889.20 total 275.75 last 121.57) - open/close 33715 (time 1746149899.21 total 285.76 last 144.72) - open/close 34941 (time 1746149909.21 total 295.76 last 122.55) - open/close 36238 (time 1746149919.22 total 305.77 last 129.62) - open/close 37477 (time 1746149929.22 total 315.77 last 123.88) - open/close 38524 (time 1746149939.22 total 325.77 last 104.69) - open/close 39754 (time 1746149949.22 total 335.77 last 122.97) - open/close 41126 (time 1746149959.23 total 345.78 last 137.06) - open/close 42159 (time 1746149969.25 total 355.80 last 103.13) - open/close 43357 (time 1746149979.25 total 365.80 last 119.76) - open/close 44749 (time 1746149989.26 total 375.81 last 139.08) - open/close 45813 (time 1746149999.27 total 385.82 last 106.34) - open/close 46965 (time 1746150009.27 total 395.82 last 115.17) - open/close 48252 (time 1746150019.27 total 405.82 last 128.68) - open/close 49326 (time 1746150029.28 total 415.83 last 107.28) - open/close 50000 (time 1746150036.07 total 422.62 last 99.24) - open/close 51018 (time 1746150046.08 total 432.63 last 101.76) - open/close 52143 (time 1746150056.09 total 442.64 last 112.39) - open/close 53253 (time 1746150066.09 total 452.64 last 110.93) - open/close 54239 (time 1746150076.10 total 462.65 last 98.55) - open/close 55297 (time 1746150086.10 total 472.65 last 105.79) - open/close 56514 (time 1746150096.10 total 482.65 last 121.66) - open/close 57475 (time 1746150106.11 total 492.66 last 96.06) - open/close 58521 (time 1746150116.11 total 502.66 last 104.59) - open/close 59463 (time 1746150126.12 total 512.67 last 94.12) - open/close 60000 (time 1746150130.99 total 517.54 last 110.19) - open/close 61218 (time 1746150140.99 total 527.54 last 121.79) - open/close 62215 (time 1746150150.99 total 537.54 last 99.69) - open/close 63234 (time 1746150160.99 total 547.54 last 101.88) - open/close 64359 (time 1746150171.00 total 557.55 last 112.44) total: 64768 open/close in 560.74 seconds: 115.50 ops/second - /unlink 1694 (time 1746150186.14 total 10.00 last 169.37) - /unlink 3355 (time 1746150196.15 total 20.02 last 165.87) - /unlink 5201 (time 1746150206.16 total 30.02 last 184.55) - /unlink 7382 (time 1746150216.16 total 40.02 last 218.04) - /unlink 8986 (time 1746150226.16 total 50.02 last 160.35) - /unlink 10000 (time 1746150232.75 total 56.61 last 153.93) - /unlink 11695 (time 1746150242.76 total 66.62 last 169.42) - /unlink 13726 (time 1746150252.76 total 76.62 last 203.04) - /unlink 15273 (time 1746150262.76 total 86.62 last 154.64) - /unlink 16971 (time 1746150272.77 total 96.63 last 169.74) - /unlink 18662 (time 1746150282.77 total 106.63 last 169.01) - /unlink 20000 (time 1746150289.79 total 113.65 last 190.66) - /unlink 21757 (time 1746150299.79 total 123.65 last 175.63) - /unlink 23572 (time 1746150309.79 total 133.65 last 181.49) - /unlink 25716 (time 1746150319.80 total 143.66 last 214.34) - /unlink 27737 (time 1746150329.80 total 153.66 last 202.05) - /unlink 29592 (time 1746150339.80 total 163.66 last 185.49) - /unlink 30000 (time 1746150341.94 total 165.80 last 190.46) - /unlink 31964 (time 1746150351.94 total 175.80 last 196.35) - /unlink 34374 (time 1746150361.95 total 185.81 last 240.89) - /unlink 36142 (time 1746150371.95 total 195.81 last 176.80) - /unlink 38044 (time 1746150381.95 total 205.82 last 190.09) - /unlink 40000 (time 1746150389.33 total 213.19 last 265.15) - /unlink 41833 (time 1746150399.33 total 223.19 last 183.29) - /unlink 43771 (time 1746150409.33 total 233.19 last 193.78) - /unlink 45948 (time 1746150419.33 total 243.20 last 217.66) - /unlink 48124 (time 1746150429.33 total 253.20 last 217.60) - /unlink 49918 (time 1746150439.34 total 263.20 last 179.31) - /unlink 51721 (time 1746150449.34 total 273.20 last 180.22) - /unlink 54067 (time 1746150459.34 total 283.21 last 234.59) - /unlink 55838 (time 1746150469.35 total 293.21 last 177.01) - /unlink 57715 (time 1746150479.35 total 303.21 last 187.69) - /unlink 59910 (time 1746150489.35 total 313.21 last 219.45) - /unlink 62333 (time 1746150499.35 total 323.22 last 242.23) - /unlink 64257 (time 1746150509.36 total 333.22 last 192.38) total: 64768 /unlink in 335.91 seconds: 192.81 ops/second Waiting for MDT destroys to complete - open/close 1622 (time 1746150529.07 total 10.01 last 162.02) - open/close 2718 (time 1746150539.07 total 20.01 last 109.59) - open/close 3856 (time 1746150549.08 total 30.01 last 113.78) - open/close 5336 (time 1746150559.09 total 40.03 last 147.83) - open/close 6550 (time 1746150569.10 total 50.03 last 121.32) - open/close 7762 (time 1746150579.10 total 60.04 last 121.13) - open/close 9105 (time 1746150589.10 total 70.04 last 134.29) - open/close 10000 (time 1746150596.56 total 77.50 last 119.98) - open/close 11124 (time 1746150606.57 total 87.50 last 112.34) - open/close 12459 (time 1746150616.57 total 97.51 last 133.49) - open/close 13733 (time 1746150626.57 total 107.51 last 127.33) - open/close 14950 (time 1746150636.58 total 117.52 last 121.59) - open/close 16200 (time 1746150646.58 total 127.52 last 125.00) - open/close 17417 (time 1746150656.59 total 137.53 last 121.63) - open/close 18643 (time 1746150666.59 total 147.53 last 122.56) - open/close 19787 (time 1746150676.60 total 157.54 last 114.32) - open/close 21054 (time 1746150686.60 total 167.54 last 126.69) - open/close 22293 (time 1746150696.61 total 177.55 last 123.71) - open/close 23369 (time 1746150706.62 total 187.56 last 107.52) - open/close 24554 (time 1746150716.63 total 197.56 last 118.44) - open/close 25758 (time 1746150726.63 total 207.56 last 120.40) - open/close 26962 (time 1746150736.63 total 217.57 last 120.33) - open/close 28113 (time 1746150746.64 total 227.58 last 114.99) - open/close 29377 (time 1746150756.64 total 237.58 last 126.37) - open/close 30000 (time 1746150762.03 total 242.96 last 115.75) - open/close 31021 (time 1746150772.04 total 252.97 last 102.01) - open/close 32241 (time 1746150782.04 total 262.98 last 121.97) - open/close 33423 (time 1746150792.04 total 272.98 last 118.17) - open/close 34487 (time 1746150802.04 total 282.98 last 106.39) - open/close 35549 (time 1746150812.06 total 292.99 last 106.03) - open/close 36765 (time 1746150822.06 total 303.00 last 121.59) - open/close 37894 (time 1746150832.07 total 313.01 last 112.79) - open/close 38926 (time 1746150842.07 total 323.01 last 103.15) - open/close 40000 (time 1746150851.99 total 332.93 last 108.30) - open/close 41141 (time 1746150862.00 total 342.94 last 113.98) - open/close 42351 (time 1746150872.01 total 352.95 last 120.87) - open/close 43358 (time 1746150882.02 total 362.95 last 100.64) - open/close 44479 (time 1746150892.02 total 372.95 last 112.10) - open/close 45607 (time 1746150902.02 total 382.96 last 112.72) - open/close 46730 (time 1746150912.02 total 392.96 last 112.29) - open/close 47739 (time 1746150922.03 total 402.97 last 100.81) - open/close 48838 (time 1746150932.05 total 412.99 last 109.71) - open/close 49917 (time 1746150942.06 total 423.00 last 107.79) - open/close 51035 (time 1746150952.07 total 433.00 last 111.75) - open/close 52170 (time 1746150962.07 total 443.00 last 113.49) - open/close 53309 (time 1746150972.07 total 453.01 last 113.88) - open/close 54454 (time 1746150982.08 total 463.01 last 114.40) - open/close 55646 (time 1746150992.09 total 473.02 last 119.08) - open/close 56753 (time 1746151002.10 total 483.03 last 110.62) - open/close 57905 (time 1746151012.10 total 493.04 last 115.17) - open/close 59024 (time 1746151022.10 total 503.04 last 111.86) - open/close 60000 (time 1746151029.41 total 510.35 last 133.48) - open/close 61323 (time 1746151039.42 total 520.35 last 132.25) - open/close 62748 (time 1746151049.42 total 530.36 last 142.48) - open/close 63942 (time 1746151059.42 total 540.36 last 119.37) total: 64768 open/close in 547.28 seconds: 118.35 ops/second - /unlink 2157 (time 1746151078.54 total 10.01 last 215.57) - /unlink 4091 (time 1746151088.54 total 20.01 last 193.34) - /unlink 6132 (time 1746151098.54 total 30.01 last 204.02) - /unlink 8375 (time 1746151108.55 total 40.02 last 224.22) - /unlink 10000 (time 1746151116.00 total 47.47 last 217.88) - /unlink 12098 (time 1746151126.02 total 57.49 last 209.57) - /unlink 14250 (time 1746151136.02 total 67.49 last 215.16) - /unlink 16687 (time 1746151146.02 total 77.49 last 243.65) - /unlink 19157 (time 1746151156.02 total 87.49 last 246.96) - /unlink 20000 (time 1746151159.20 total 90.68 last 264.76) - /unlink 22574 (time 1746151169.21 total 100.68 last 257.33) - /unlink 24967 (time 1746151179.21 total 110.68 last 239.20) - /unlink 27669 (time 1746151189.21 total 120.68 last 270.17) - /unlink 29996 (time 1746151199.21 total 130.69 last 232.64) - /unlink 32103 (time 1746151209.22 total 140.69 last 210.57) - /unlink 34233 (time 1746151219.22 total 150.69 last 212.94) - /unlink 36497 (time 1746151229.23 total 160.70 last 226.35) - /unlink 38736 (time 1746151239.23 total 170.70 last 223.88) - /unlink 40000 (time 1746151243.89 total 175.36 last 271.08) - /unlink 42568 (time 1746151253.89 total 185.36 last 256.77) - /unlink 44729 (time 1746151263.89 total 195.36 last 216.08) - /unlink 46950 (time 1746151273.89 total 205.37 last 222.05) - /unlink 49077 (time 1746151283.90 total 215.37 last 212.69) - /unlink 50000 (time 1746151288.39 total 219.86 last 205.25) - /unlink 52177 (time 1746151298.39 total 229.86 last 217.67) - /unlink 54349 (time 1746151308.40 total 239.87 last 217.12) - /unlink 56842 (time 1746151318.40 total 249.87 last 249.25) - /unlink 59172 (time 1746151328.40 total 259.87 last 232.95) - /unlink 60000 (time 1746151331.59 total 263.06 last 260.05) - /unlink 62622 (time 1746151341.59 total 273.06 last 262.15) total: 64768 /unlink in 280.35 seconds: 231.02 ops/second Waiting for MDT destroys to complete - open/close 1410 (time 1746151365.61 total 10.01 last 140.86) - open/close 2802 (time 1746151375.62 total 20.02 last 139.10) - open/close 4385 (time 1746151385.62 total 30.02 last 158.27) - open/close 5689 (time 1746151395.63 total 40.03 last 130.28) - open/close 7310 (time 1746151405.63 total 50.03 last 162.07) - open/close 8899 (time 1746151415.64 total 60.03 last 158.85) - open/close 10000 (time 1746151422.63 total 67.02 last 157.45) - open/close 11553 (time 1746151432.63 total 77.03 last 155.23) - open/close 13237 (time 1746151442.64 total 87.04 last 168.25) - open/close 14828 (time 1746151452.65 total 97.04 last 159.05) - open/close 16357 (time 1746151462.65 total 107.05 last 152.81) - open/close 17946 (time 1746151472.65 total 117.05 last 158.88) - open/close 19561 (time 1746151482.65 total 127.05 last 161.46) - open/close 20000 (time 1746151485.51 total 129.91 last 153.60) - open/close 21545 (time 1746151495.52 total 139.91 last 154.42) - open/close 23068 (time 1746151505.52 total 149.92 last 152.26) - open/close 24794 (time 1746151515.53 total 159.92 last 172.51) - open/close 26386 (time 1746151525.53 total 169.92 last 159.17) - open/close 27898 (time 1746151535.53 total 179.93 last 151.15) - open/close 29748 (time 1746151545.53 total 189.93 last 184.96) - open/close 31380 (time 1746151555.53 total 199.93 last 163.19) - open/close 33002 (time 1746151565.53 total 209.93 last 162.19) - open/close 34727 (time 1746151575.53 total 219.93 last 172.50) - open/close 36546 (time 1746151585.54 total 229.93 last 181.83) - open/close 38328 (time 1746151595.54 total 239.93 last 178.19) - open/close 39899 (time 1746151605.54 total 249.94 last 157.08) - open/close 41473 (time 1746151615.55 total 259.94 last 157.31) - open/close 42975 (time 1746151625.55 total 269.94 last 150.15) - open/close 44473 (time 1746151635.55 total 279.95 last 149.75) - open/close 46001 (time 1746151645.56 total 289.96 last 152.64) - open/close 47579 (time 1746151655.57 total 299.96 last 157.71) - open/close 49140 (time 1746151665.57 total 309.97 last 156.08) - open/close 50000 (time 1746151671.81 total 316.20 last 137.87) - open/close 51368 (time 1746151681.81 total 326.21 last 136.76) - open/close 52848 (time 1746151691.82 total 336.21 last 147.91) - open/close 54135 (time 1746151701.82 total 346.22 last 128.65) - open/close 55558 (time 1746151711.82 total 356.22 last 142.29) - open/close 57029 (time 1746151721.82 total 366.22 last 147.07) - open/close 58434 (time 1746151731.83 total 376.22 last 140.43) - open/close 59817 (time 1746151741.83 total 386.23 last 138.25) - open/close 61362 (time 1746151751.83 total 396.23 last 154.44) - open/close 62844 (time 1746151761.84 total 406.23 last 148.19) - open/close 64507 (time 1746151771.84 total 416.23 last 166.27) total: 64768 open/close in 418.10 seconds: 154.91 ops/second - /unlink 2597 (time 1746151785.51 total 10.01 last 259.55) - /unlink 5110 (time 1746151795.51 total 20.01 last 251.23) - /unlink 7462 (time 1746151805.51 total 30.01 last 235.19) - /unlink 10000 (time 1746151815.40 total 39.90 last 256.64) - /unlink 12451 (time 1746151825.41 total 49.90 last 245.04) - /unlink 14893 (time 1746151835.41 total 59.90 last 244.10) - /unlink 17474 (time 1746151845.41 total 69.91 last 258.10) - /unlink 19986 (time 1746151855.41 total 79.91 last 251.13) - /unlink 22369 (time 1746151865.41 total 89.91 last 238.28) - /unlink 24998 (time 1746151875.42 total 99.92 last 262.63) - /unlink 27367 (time 1746151885.42 total 109.92 last 236.90) - /unlink 29894 (time 1746151895.43 total 119.92 last 252.66) - /unlink 32232 (time 1746151905.43 total 129.92 last 233.75) - /unlink 34805 (time 1746151915.43 total 139.93 last 257.22) - /unlink 37386 (time 1746151925.44 total 149.93 last 257.98) - /unlink 39684 (time 1746151935.44 total 159.93 last 229.77) - /unlink 42101 (time 1746151945.44 total 169.93 last 241.68) - /unlink 44692 (time 1746151955.45 total 179.94 last 258.84) - /unlink 47380 (time 1746151965.45 total 189.94 last 268.79) - /unlink 49855 (time 1746151975.45 total 199.94 last 247.48) - /unlink 52337 (time 1746151985.45 total 209.95 last 248.16) - /unlink 54938 (time 1746151995.46 total 219.95 last 259.96) - /unlink 57396 (time 1746152005.46 total 229.95 last 245.79) - /unlink 59771 (time 1746152015.46 total 239.95 last 237.47) - /unlink 62281 (time 1746152025.46 total 249.96 last 250.93) total: 64768 /unlink in 259.27 seconds: 249.81 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg352-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg352-server unloading modules via unload_modules_local on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg352-client: oleg352-client: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-client: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 PASS 106 (4268s) Stopping clients: oleg352-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg352-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 oleg352-server: oleg352-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions oleg352-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg352-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 4406 sec ========== 22:15:13 (1746152113) === conf-sanity: start cleanup 22:15:14 (1746152114) === === conf-sanity: finish cleanup 22:15:16 (1746152116) ===