-----============= acceptance-small: conf-sanity ============----- Fri Apr 25 18:24:25 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg348-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg348-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg348-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg348-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg348-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg348-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg348-server oleg348-server: oleg348-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg348-server' oleg348-server: oleg348-server.virtnet: executing load_modules_local oleg348-server: Loading modules from /home/green/git/lustre-release/lustre oleg348-server: detected 4 online CPUs by sysfs oleg348-server: MODOPTS_LIBCFS= oleg348-server: Force libcfs to create 2 CPU partitions oleg348-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg348-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg348-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg348-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg348-server: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg348-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg348-server: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg348-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg348-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server stop mds service on oleg348-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg348-server stop mds service on oleg348-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg348-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:26:36 (1745619996) Stopping clients: oleg348-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg348-client.virtnet /mnt/lustre2 (opts:-f) oleg348-server: oleg348-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg348-server' oleg348-server: oleg348-server.virtnet: executing load_modules_local oleg348-server: Loading modules from /home/green/git/lustre-release/lustre oleg348-server: detected 4 online CPUs by sysfs oleg348-server: MODOPTS_LIBCFS= oleg348-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg348-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg348-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg348-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg348-client.virtnet: -o user_xattr,flock 192.168.203.148@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 862 (time 1745620077.44 total 10.01 last 86.13) - open/close 1634 (time 1745620087.45 total 20.02 last 77.13) - open/close 2430 (time 1745620097.46 total 30.03 last 79.48) - open/close 3206 (time 1745620107.47 total 40.04 last 77.56) - open/close 4027 (time 1745620117.47 total 50.05 last 82.04) - open/close 5042 (time 1745620127.48 total 60.05 last 101.40) - open/close 5809 (time 1745620137.49 total 70.07 last 76.62) - open/close 6580 (time 1745620147.50 total 80.07 last 77.05) - open/close 7392 (time 1745620157.50 total 90.07 last 81.19) - open/close 8160 (time 1745620167.51 total 100.08 last 76.78) - open/close 9432 (time 1745620177.51 total 110.08 last 127.17) - open/close 10000 (time 1745620183.79 total 116.36 last 90.39) - open/close 10861 (time 1745620193.79 total 126.36 last 86.08) - open/close 11793 (time 1745620203.80 total 136.37 last 93.10) - open/close 12710 (time 1745620213.81 total 146.38 last 91.65) - open/close 13613 (time 1745620223.81 total 156.38 last 90.29) - open/close 15008 (time 1745620233.82 total 166.39 last 139.43) - open/close 16008 (time 1745620243.83 total 176.40 last 99.91) - open/close 16887 (time 1745620253.83 total 186.40 last 87.88) - open/close 17781 (time 1745620263.83 total 196.40 last 89.39) - open/close 18691 (time 1745620273.83 total 206.40 last 90.95) - open/close 19948 (time 1745620283.84 total 216.41 last 125.66) - open/close 20950 (time 1745620293.85 total 226.42 last 100.11) - open/close 21807 (time 1745620303.85 total 236.42 last 85.70) - open/close 22643 (time 1745620313.85 total 246.42 last 83.56) - open/close 23484 (time 1745620323.85 total 256.42 last 84.09) - open/close 24537 (time 1745620333.85 total 266.42 last 105.28) - open/close 25345 (time 1745620343.86 total 276.43 last 80.74) - open/close 26134 (time 1745620353.87 total 286.44 last 78.85) - open/close 26849 (time 1745620363.87 total 296.44 last 71.47) - open/close 27601 (time 1745620373.88 total 306.45 last 75.18) - open/close 28549 (time 1745620383.88 total 316.45 last 94.75) - open/close 29552 (time 1745620393.88 total 326.45 last 100.29) - open/close 30000 (time 1745620398.91 total 331.48 last 89.07) - open/close 30847 (time 1745620408.91 total 341.48 last 84.70) - open/close 31917 (time 1745620418.92 total 351.49 last 106.96) - open/close 33495 (time 1745620428.92 total 361.49 last 157.68) - open/close 34562 (time 1745620438.93 total 371.50 last 106.63) - open/close 35759 (time 1745620448.94 total 381.51 last 119.62) - open/close 37169 (time 1745620458.94 total 391.51 last 140.95) - open/close 39056 (time 1745620468.95 total 401.52 last 188.59) - open/close 40000 (time 1745620475.78 total 408.35 last 138.06) - open/close 41004 (time 1745620485.80 total 418.37 last 100.26) - open/close 42047 (time 1745620495.80 total 428.37 last 104.28) - open/close 43657 (time 1745620505.81 total 438.38 last 160.88) - open/close 44819 (time 1745620515.81 total 448.38 last 116.16) - open/close 46039 (time 1745620525.81 total 458.38 last 121.98) - open/close 47476 (time 1745620535.82 total 468.39 last 143.63) - open/close 48765 (time 1745620545.82 total 478.39 last 128.83) - open/close 49670 (time 1745620555.83 total 488.40 last 90.40) - open/close 50000 (time 1745620559.50 total 492.07 last 90.05) - open/close 50922 (time 1745620569.50 total 502.07 last 92.15) - open/close 51996 (time 1745620579.51 total 512.08 last 107.38) - open/close 53321 (time 1745620589.51 total 522.08 last 132.43) - open/close 54419 (time 1745620599.52 total 532.09 last 109.70) - open/close 55598 (time 1745620609.52 total 542.09 last 117.87) - open/close 56819 (time 1745620619.52 total 552.09 last 122.08) - open/close 57798 (time 1745620629.53 total 562.10 last 97.87) - open/close 58878 (time 1745620639.54 total 572.11 last 107.90) - open/close 60000 (time 1745620649.24 total 581.81 last 115.58) - open/close 61120 (time 1745620659.25 total 591.82 last 111.95) - open/close 62258 (time 1745620669.28 total 601.85 last 113.48) - open/close 63725 (time 1745620679.28 total 611.85 last 146.59) total: 64768 open/close in 619.07 seconds: 104.62 ops/second - /unlink 1367 (time 1745620698.32 total 10.01 last 136.61) - /unlink 2812 (time 1745620708.32 total 20.01 last 144.42) - /unlink 4061 (time 1745620718.33 total 30.02 last 124.83) - /unlink 6122 (time 1745620728.33 total 40.02 last 206.07) - /unlink 8246 (time 1745620738.33 total 50.02 last 212.38) - /unlink 10000 (time 1745620746.49 total 58.17 last 215.11) - /unlink 12535 (time 1745620756.49 total 68.18 last 253.47) - /unlink 15282 (time 1745620766.49 total 78.18 last 274.60) - /unlink 17847 (time 1745620776.49 total 88.18 last 256.47) - /unlink 20000 (time 1745620785.15 total 96.83 last 248.80) - /unlink 22692 (time 1745620795.15 total 106.84 last 269.14) - /unlink 25193 (time 1745620805.15 total 116.84 last 250.03) - /unlink 27597 (time 1745620815.15 total 126.84 last 240.29) - /unlink 30000 (time 1745620824.83 total 136.51 last 248.48) - /unlink 32492 (time 1745620834.83 total 146.52 last 249.08) - /unlink 35079 (time 1745620844.83 total 156.52 last 258.61) - /unlink 37577 (time 1745620854.83 total 166.52 last 249.79) - /unlink 40000 (time 1745620864.62 total 176.31 last 247.58) - /unlink 42701 (time 1745620874.62 total 186.31 last 270.10) - /unlink 45165 (time 1745620884.63 total 196.32 last 246.27) - /unlink 47726 (time 1745620894.63 total 206.32 last 256.10) - /unlink 50000 (time 1745620903.55 total 215.24 last 254.79) - /unlink 53128 (time 1745620913.55 total 225.24 last 312.70) - /unlink 55948 (time 1745620923.55 total 235.24 last 281.99) - /unlink 59192 (time 1745620933.56 total 245.25 last 324.33) - /unlink 60000 (time 1745620936.39 total 248.08 last 284.91) - /unlink 63122 (time 1745620946.40 total 258.09 last 311.94) total: 64768 /unlink in 264.83 seconds: 244.56 ops/second Waiting for MDT destroys to complete - open/close 1350 (time 1745620973.85 total 10.00 last 134.95) - open/close 3080 (time 1745620983.86 total 20.01 last 172.95) - open/close 5190 (time 1745620993.86 total 30.01 last 210.99) - open/close 7020 (time 1745621003.86 total 40.01 last 183.00) - open/close 8978 (time 1745621013.86 total 50.01 last 195.76) - open/close 10000 (time 1745621018.78 total 54.93 last 207.55) - open/close 12111 (time 1745621028.79 total 64.94 last 211.03) - open/close 14209 (time 1745621038.79 total 74.94 last 209.70) - open/close 16218 (time 1745621048.79 total 84.94 last 200.86) - open/close 18608 (time 1745621058.81 total 94.96 last 238.70) - open/close 20000 (time 1745621063.13 total 99.28 last 322.09) - open/close 22701 (time 1745621073.13 total 109.28 last 269.94) - open/close 24760 (time 1745621083.13 total 119.28 last 205.88) - open/close 26761 (time 1745621093.14 total 129.29 last 200.03) - open/close 28673 (time 1745621103.14 total 139.29 last 191.16) - open/close 30000 (time 1745621110.06 total 146.21 last 191.85) - open/close 31892 (time 1745621120.06 total 156.21 last 189.19) - open/close 33943 (time 1745621130.06 total 166.21 last 205.05) - open/close 36012 (time 1745621140.06 total 176.21 last 206.86) - open/close 38178 (time 1745621150.06 total 186.21 last 216.58) - open/close 40000 (time 1745621159.35 total 195.49 last 196.29) - open/close 41949 (time 1745621169.35 total 205.50 last 194.87) - open/close 43678 (time 1745621179.35 total 215.50 last 172.84) - open/close 45480 (time 1745621189.35 total 225.50 last 180.13) - open/close 47299 (time 1745621199.36 total 235.51 last 181.83) - open/close 49169 (time 1745621209.36 total 245.51 last 186.91) - open/close 50000 (time 1745621214.21 total 250.36 last 171.46) - open/close 51685 (time 1745621224.21 total 260.36 last 168.48) - open/close 53201 (time 1745621234.21 total 270.36 last 151.59) - open/close 54936 (time 1745621244.21 total 280.36 last 173.45) - open/close 56636 (time 1745621254.22 total 290.36 last 169.97) - open/close 58310 (time 1745621264.22 total 300.37 last 167.36) - open/close 59859 (time 1745621274.22 total 310.37 last 154.83) - open/close 61295 (time 1745621284.22 total 320.37 last 143.60) - open/close 62767 (time 1745621294.23 total 330.38 last 147.10) - open/close 64350 (time 1745621304.24 total 340.39 last 158.20) total: 64768 open/close in 342.90 seconds: 188.88 ops/second - /unlink 2521 (time 1745621318.59 total 10.00 last 252.05) - /unlink 5093 (time 1745621328.59 total 20.01 last 257.07) - /unlink 7644 (time 1745621338.60 total 30.01 last 254.96) - /unlink 10000 (time 1745621348.28 total 39.70 last 243.29) - /unlink 12455 (time 1745621358.28 total 49.70 last 245.48) - /unlink 15944 (time 1745621368.28 total 59.70 last 348.87) - /unlink 19436 (time 1745621378.28 total 69.70 last 349.20) - /unlink 22421 (time 1745621388.28 total 79.70 last 298.48) - /unlink 25619 (time 1745621398.29 total 89.70 last 319.75) - /unlink 28474 (time 1745621408.31 total 99.73 last 284.70) - /unlink 30000 (time 1745621413.99 total 105.40 last 269.05) - /unlink 32530 (time 1745621423.99 total 115.40 last 253.00) - /unlink 35354 (time 1745621433.99 total 125.40 last 282.31) - /unlink 38153 (time 1745621443.99 total 135.41 last 279.84) - /unlink 40000 (time 1745621450.39 total 141.80 last 288.65) - /unlink 42729 (time 1745621460.39 total 151.81 last 272.85) - /unlink 45586 (time 1745621470.40 total 161.81 last 285.53) - /unlink 48665 (time 1745621480.40 total 171.81 last 307.82) - /unlink 50000 (time 1745621484.88 total 176.30 last 297.71) - /unlink 52906 (time 1745621494.89 total 186.30 last 290.54) - /unlink 55822 (time 1745621504.89 total 196.30 last 291.54) - /unlink 58334 (time 1745621514.89 total 206.31 last 251.04) - /unlink 60000 (time 1745621521.25 total 212.66 last 262.24) - /unlink 62788 (time 1745621531.25 total 222.66 last 278.77) total: 64768 /unlink in 230.11 seconds: 281.46 ops/second Waiting for MDT destroys to complete - open/close 1752 (time 1745621560.48 total 10.00 last 175.15) - open/close 3468 (time 1745621570.48 total 20.00 last 171.57) - open/close 5131 (time 1745621580.48 total 30.01 last 166.26) - open/close 6792 (time 1745621590.48 total 40.01 last 166.04) - open/close 8345 (time 1745621600.49 total 50.02 last 155.22) - open/close 9903 (time 1745621610.49 total 60.02 last 155.75) - open/close 11498 (time 1745621620.50 total 70.02 last 159.41) - open/close 13082 (time 1745621630.50 total 80.03 last 158.38) - open/close 14922 (time 1745621640.50 total 90.03 last 183.95) - open/close 16731 (time 1745621650.51 total 100.03 last 180.86) - open/close 18720 (time 1745621660.51 total 110.03 last 198.86) - open/close 20000 (time 1745621667.63 total 117.15 last 179.75) - open/close 21729 (time 1745621677.63 total 127.16 last 172.82) - open/close 23495 (time 1745621687.64 total 137.16 last 176.52) - open/close 25217 (time 1745621697.64 total 147.17 last 172.16) - open/close 27086 (time 1745621707.64 total 157.17 last 186.89) - open/close 28760 (time 1745621717.64 total 167.17 last 167.38) - open/close 30000 (time 1745621725.68 total 175.21 last 154.20) - open/close 31571 (time 1745621735.68 total 185.21 last 157.06) - open/close 33255 (time 1745621745.69 total 195.21 last 168.33) - open/close 34827 (time 1745621755.69 total 205.22 last 157.11) - open/close 36407 (time 1745621765.70 total 215.22 last 157.95) - open/close 37854 (time 1745621775.70 total 225.23 last 144.66) - open/close 39268 (time 1745621785.71 total 235.23 last 141.27) - open/close 40000 (time 1745621790.51 total 240.04 last 152.48) - open/close 41536 (time 1745621800.51 total 250.04 last 153.55) - open/close 42810 (time 1745621810.52 total 260.04 last 127.37) - open/close 44034 (time 1745621820.52 total 270.04 last 122.40) - open/close 45391 (time 1745621830.52 total 280.04 last 135.70) - open/close 46820 (time 1745621840.52 total 290.04 last 142.89) - open/close 48358 (time 1745621850.52 total 300.05 last 153.75) - open/close 49740 (time 1745621860.53 total 310.06 last 138.06) - open/close 51287 (time 1745621870.53 total 320.06 last 154.69) - open/close 52944 (time 1745621880.53 total 330.06 last 165.68) - open/close 54411 (time 1745621890.53 total 340.06 last 146.67) - open/close 55779 (time 1745621900.54 total 350.07 last 136.72) - open/close 57206 (time 1745621910.54 total 360.07 last 142.70) - open/close 58599 (time 1745621920.55 total 370.08 last 139.12) - open/close 59926 (time 1745621930.56 total 380.09 last 132.61) - open/close 61274 (time 1745621940.56 total 390.09 last 134.78) - open/close 62586 (time 1745621950.56 total 400.09 last 131.18) - open/close 63930 (time 1745621960.57 total 410.10 last 134.32) total: 64768 open/close in 416.50 seconds: 155.50 ops/second - /unlink 2426 (time 1745621978.67 total 10.00 last 242.58) - /unlink 4578 (time 1745621988.67 total 20.00 last 215.17) - /unlink 6989 (time 1745621998.67 total 30.00 last 241.08) - /unlink 9264 (time 1745622008.67 total 40.00 last 227.47) - /unlink 10000 (time 1745622012.06 total 43.39 last 217.37) - /unlink 12546 (time 1745622022.06 total 53.39 last 254.60) - /unlink 15158 (time 1745622032.06 total 63.40 last 261.03) - /unlink 17685 (time 1745622042.06 total 73.40 last 252.70) - /unlink 19929 (time 1745622052.07 total 83.40 last 224.33) - /unlink 22375 (time 1745622062.07 total 93.40 last 244.53) - /unlink 24778 (time 1745622072.07 total 103.41 last 240.22) - /unlink 27071 (time 1745622082.07 total 113.41 last 229.25) - /unlink 29309 (time 1745622092.07 total 123.41 last 223.78) - /unlink 30000 (time 1745622094.98 total 126.32 last 237.49) - /unlink 32461 (time 1745622104.98 total 136.32 last 246.10) - /unlink 34955 (time 1745622114.99 total 146.32 last 249.30) - /unlink 37422 (time 1745622124.99 total 156.33 last 246.66) - /unlink 39830 (time 1745622134.99 total 166.33 last 240.75) - /unlink 42424 (time 1745622144.99 total 176.33 last 259.36) - /unlink 44946 (time 1745622155.00 total 186.34 last 251.97) - /unlink 47418 (time 1745622165.01 total 196.34 last 247.04) - /unlink 50000 (time 1745622174.39 total 205.73 last 275.11) - /unlink 52698 (time 1745622184.40 total 215.73 last 269.74) - /unlink 55213 (time 1745622194.42 total 225.76 last 250.82) - /unlink 57818 (time 1745622204.43 total 235.76 last 260.41) - /unlink 60000 (time 1745622213.01 total 244.35 last 254.21) - /unlink 62747 (time 1745622223.01 total 254.35 last 274.67) total: 64768 /unlink in 261.24 seconds: 247.93 ops/second Waiting for MDT destroys to complete - open/close 1555 (time 1745622245.51 total 10.00 last 155.46) - open/close 3210 (time 1745622255.52 total 20.01 last 165.42) - open/close 4891 (time 1745622265.52 total 30.01 last 168.05) - open/close 6653 (time 1745622275.52 total 40.01 last 176.20) - open/close 8258 (time 1745622285.52 total 50.01 last 160.47) - open/close 10000 (time 1745622295.40 total 59.89 last 176.32) - open/close 11836 (time 1745622305.41 total 69.90 last 183.50) - open/close 13513 (time 1745622315.41 total 79.90 last 167.61) - open/close 15376 (time 1745622325.42 total 89.91 last 186.18) - open/close 17102 (time 1745622335.42 total 99.91 last 172.54) - open/close 18816 (time 1745622345.43 total 109.92 last 171.30) - open/close 20000 (time 1745622352.54 total 117.03 last 166.46) - open/close 21777 (time 1745622362.55 total 127.04 last 177.60) - open/close 23432 (time 1745622372.55 total 137.04 last 165.45) - open/close 25319 (time 1745622382.56 total 147.05 last 188.58) - open/close 26956 (time 1745622392.56 total 157.05 last 163.63) - open/close 28592 (time 1745622402.57 total 167.06 last 163.50) - open/close 30000 (time 1745622410.86 total 175.35 last 169.85) - open/close 31828 (time 1745622420.86 total 185.35 last 182.73) - open/close 33703 (time 1745622430.86 total 195.35 last 187.45) - open/close 35539 (time 1745622440.86 total 205.35 last 183.57) - open/close 37315 (time 1745622450.86 total 215.35 last 177.59) - open/close 39133 (time 1745622460.87 total 225.36 last 181.66) - open/close 40000 (time 1745622465.77 total 230.26 last 177.09) - open/close 42096 (time 1745622475.77 total 240.26 last 209.49) - open/close 44073 (time 1745622485.78 total 250.27 last 197.62) - open/close 46378 (time 1745622495.78 total 260.27 last 230.48) - open/close 48535 (time 1745622505.78 total 270.27 last 215.57) - open/close 50000 (time 1745622513.25 total 277.74 last 196.32) - open/close 51944 (time 1745622523.25 total 287.74 last 194.39) - open/close 53912 (time 1745622533.25 total 297.74 last 196.75) - open/close 56078 (time 1745622543.25 total 307.74 last 216.59) - open/close 59252 (time 1745622553.25 total 317.74 last 317.36) - open/close 60000 (time 1745622555.82 total 320.31 last 291.10) - open/close 62518 (time 1745622565.82 total 330.31 last 251.80) total: 64768 open/close in 339.09 seconds: 191.01 ops/second - /unlink 4656 (time 1745622585.71 total 10.00 last 465.52) - /unlink 8908 (time 1745622595.71 total 20.00 last 425.20) - /unlink 10000 (time 1745622598.42 total 22.71 last 403.77) - /unlink 14106 (time 1745622608.42 total 32.71 last 410.55) - /unlink 17744 (time 1745622618.42 total 42.71 last 363.68) - /unlink 20000 (time 1745622625.18 total 49.47 last 334.00) - /unlink 23522 (time 1745622635.18 total 59.47 last 352.07) - /unlink 27964 (time 1745622645.18 total 69.47 last 444.13) - /unlink 30000 (time 1745622649.27 total 73.56 last 497.65) - /unlink 34433 (time 1745622659.27 total 83.56 last 443.29) - /unlink 38639 (time 1745622669.27 total 93.56 last 420.54) - /unlink 40000 (time 1745622673.11 total 97.40 last 354.46) - /unlink 45002 (time 1745622683.12 total 107.40 last 500.12) - /unlink 48715 (time 1745622693.12 total 117.41 last 371.23) - /unlink 50000 (time 1745622696.61 total 120.90 last 367.71) - /unlink 53384 (time 1745622706.61 total 130.90 last 338.38) - /unlink 57427 (time 1745622716.61 total 140.90 last 404.25) - /unlink 60000 (time 1745622722.39 total 146.68 last 445.54) total: 64768 /unlink in 156.30 seconds: 414.40 ops/second Waiting for MDT destroys to complete - open/close 2024 (time 1745622751.90 total 10.00 last 202.40) - open/close 3943 (time 1745622761.90 total 20.00 last 191.85) - open/close 5931 (time 1745622771.90 total 30.00 last 198.77) - open/close 8333 (time 1745622781.90 total 40.01 last 240.18) - open/close 10000 (time 1745622788.95 total 47.05 last 236.60) - open/close 12215 (time 1745622798.96 total 57.06 last 221.37) - open/close 14387 (time 1745622808.96 total 67.06 last 217.16) - open/close 16668 (time 1745622818.96 total 77.06 last 228.00) - open/close 19174 (time 1745622828.96 total 87.06 last 250.57) - open/close 20000 (time 1745622832.87 total 90.97 last 211.26) - open/close 22293 (time 1745622842.88 total 100.98 last 229.20) - open/close 24626 (time 1745622852.88 total 110.98 last 233.23) - open/close 27031 (time 1745622862.88 total 120.99 last 240.38) - open/close 29271 (time 1745622872.89 total 130.99 last 223.94) - open/close 30000 (time 1745622876.14 total 134.24 last 224.43) - open/close 32677 (time 1745622886.14 total 144.24 last 267.68) - open/close 35238 (time 1745622896.14 total 154.24 last 256.10) - open/close 37594 (time 1745622906.14 total 164.24 last 235.58) - open/close 40000 (time 1745622914.44 total 172.54 last 289.90) - open/close 42942 (time 1745622924.44 total 182.54 last 294.16) - open/close 46343 (time 1745622934.44 total 192.54 last 340.09) - open/close 49278 (time 1745622944.44 total 202.54 last 293.44) - open/close 50000 (time 1745622947.20 total 205.30 last 261.94) - open/close 53181 (time 1745622957.20 total 215.30 last 318.06) - open/close 56093 (time 1745622967.20 total 225.30 last 291.15) - open/close 58509 (time 1745622977.20 total 235.30 last 241.59) - open/close 60000 (time 1745622982.82 total 240.93 last 265.09) - open/close 62698 (time 1745622992.82 total 250.93 last 269.80) total: 64768 open/close in 259.18 seconds: 249.89 ops/second - /unlink 4324 (time 1745623012.16 total 10.00 last 432.32) - /unlink 8460 (time 1745623022.18 total 20.02 last 412.94) - /unlink 10000 (time 1745623026.16 total 24.01 last 386.12) - /unlink 13775 (time 1745623036.17 total 34.01 last 377.37) - /unlink 17524 (time 1745623046.17 total 44.01 last 374.81) - /unlink 20000 (time 1745623052.92 total 50.76 last 366.93) - /unlink 23924 (time 1745623062.92 total 60.76 last 392.38) - /unlink 27411 (time 1745623072.92 total 70.76 last 348.58) - /unlink 30000 (time 1745623080.60 total 78.44 last 337.38) - /unlink 33476 (time 1745623090.60 total 88.44 last 347.49) - /unlink 36983 (time 1745623100.60 total 98.44 last 350.66) - /unlink 40000 (time 1745623106.22 total 104.06 last 537.12) - /unlink 44626 (time 1745623116.22 total 114.06 last 462.58) - /unlink 49020 (time 1745623126.22 total 124.06 last 439.38) - /unlink 50000 (time 1745623128.75 total 126.59 last 387.59) - /unlink 54639 (time 1745623138.75 total 136.59 last 463.89) - /unlink 58793 (time 1745623148.75 total 146.59 last 415.35) - /unlink 60000 (time 1745623151.39 total 149.23 last 457.45) - /unlink 64428 (time 1745623161.39 total 159.23 last 442.73) total: 64768 /unlink in 159.95 seconds: 404.92 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg348-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg348-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server stop mds service on oleg348-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg348-server stop mds service on oleg348-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg348-server unloading modules via unload_modules_local on: 'oleg348-server' oleg348-server: oleg348-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg348-client: oleg348-client: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-server: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-client: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-server: ssh exited with exit code 2 PASS 106 (3194s) Stopping clients: oleg348-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg348-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg348-client: oleg348-server: ssh exited with exit code 2 oleg348-server: oleg348-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg348-server' oleg348-server: oleg348-server.virtnet: executing load_modules_local oleg348-server: Loading modules from /home/green/git/lustre-release/lustre oleg348-server: detected 4 online CPUs by sysfs oleg348-server: MODOPTS_LIBCFS= oleg348-server: Force libcfs to create 2 CPU partitions oleg348-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg348-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3345 sec ========== 19:20:11 (1745623211) === conf-sanity: start cleanup 19:20:11 (1745623211) === === conf-sanity: finish cleanup 19:20:12 (1745623212) ===