-----============= acceptance-small: conf-sanity ============----- Wed Apr 30 21:55:01 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg430-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg430-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg430-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg430-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg430-server oleg430-server: oleg430-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing load_modules_local oleg430-server: Loading modules from /home/green/git/lustre-release/lustre oleg430-server: detected 4 online CPUs by sysfs oleg430-server: MODOPTS_LIBCFS= oleg430-server: Force libcfs to create 2 CPU partitions oleg430-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg430-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg430-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg430-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg430-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg430-server: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg430-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg430-server: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg430-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg430-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 21:56:59 (1746064619) Stopping clients: oleg430-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg430-client.virtnet /mnt/lustre2 (opts:-f) oleg430-server: oleg430-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing load_modules_local oleg430-server: Loading modules from /home/green/git/lustre-release/lustre oleg430-server: detected 4 online CPUs by sysfs oleg430-server: MODOPTS_LIBCFS= oleg430-server: Force libcfs to create 2 CPU partitions oleg430-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg430-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg430-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg430-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1297 (time 1746064688.27 total 10.01 last 129.59) - open/close 2665 (time 1746064698.28 total 20.01 last 136.76) - open/close 4044 (time 1746064708.28 total 30.01 last 137.89) - open/close 5456 (time 1746064718.28 total 40.02 last 141.11) - open/close 6799 (time 1746064728.30 total 50.03 last 134.14) - open/close 8212 (time 1746064738.30 total 60.03 last 141.27) - open/close 9575 (time 1746064748.30 total 70.04 last 136.24) - open/close 10000 (time 1746064751.09 total 72.82 last 152.69) - open/close 11355 (time 1746064761.09 total 82.83 last 135.40) - open/close 12782 (time 1746064771.09 total 92.83 last 142.69) - open/close 14205 (time 1746064781.10 total 102.83 last 142.28) - open/close 15849 (time 1746064791.10 total 112.83 last 164.36) - open/close 17253 (time 1746064801.10 total 122.84 last 140.32) - open/close 19143 (time 1746064811.11 total 132.84 last 188.97) - open/close 20000 (time 1746064815.29 total 137.03 last 204.63) - open/close 21866 (time 1746064825.30 total 147.03 last 186.55) - open/close 23684 (time 1746064835.30 total 157.04 last 181.68) - open/close 25872 (time 1746064845.30 total 167.04 last 218.77) - open/close 27663 (time 1746064855.31 total 177.04 last 179.01) - open/close 29216 (time 1746064865.31 total 187.05 last 155.24) - open/close 30000 (time 1746064870.54 total 192.27 last 150.06) - open/close 31586 (time 1746064880.54 total 202.27 last 158.59) - open/close 33361 (time 1746064890.54 total 212.27 last 177.49) - open/close 35199 (time 1746064900.54 total 222.28 last 183.72) - open/close 36926 (time 1746064910.54 total 232.28 last 172.69) - open/close 38615 (time 1746064920.54 total 242.28 last 168.89) - open/close 40000 (time 1746064927.81 total 249.54 last 190.71) - open/close 42033 (time 1746064937.81 total 259.55 last 203.18) - open/close 44188 (time 1746064947.82 total 269.55 last 215.42) - open/close 46038 (time 1746064957.82 total 279.55 last 184.94) - open/close 48182 (time 1746064967.82 total 289.55 last 214.39) - open/close 50000 (time 1746064977.02 total 298.75 last 197.63) - open/close 51786 (time 1746064987.02 total 308.75 last 178.57) - open/close 53647 (time 1746064997.02 total 318.76 last 186.07) - open/close 55114 (time 1746065007.02 total 328.76 last 146.70) - open/close 56383 (time 1746065017.02 total 338.76 last 126.87) - open/close 58030 (time 1746065027.03 total 348.76 last 164.63) - open/close 59752 (time 1746065037.03 total 358.76 last 172.19) - open/close 61403 (time 1746065047.03 total 368.77 last 165.04) - open/close 62992 (time 1746065057.04 total 378.77 last 158.85) total: 64768 open/close in 388.04 seconds: 166.91 ops/second - /unlink 2689 (time 1746065077.82 total 10.00 last 268.83) - /unlink 5344 (time 1746065087.82 total 20.00 last 265.50) - /unlink 8139 (time 1746065097.82 total 30.00 last 279.45) - /unlink 10000 (time 1746065104.59 total 36.78 last 274.84) - /unlink 12817 (time 1746065114.59 total 46.78 last 281.66) - /unlink 15903 (time 1746065124.59 total 56.78 last 308.57) - /unlink 19008 (time 1746065134.60 total 66.78 last 310.43) - /unlink 20000 (time 1746065137.07 total 69.25 last 401.28) - /unlink 23046 (time 1746065147.07 total 79.26 last 304.48) - /unlink 25775 (time 1746065157.08 total 89.26 last 272.67) - /unlink 28543 (time 1746065167.08 total 99.27 last 276.79) - /unlink 30000 (time 1746065173.23 total 105.42 last 236.76) - /unlink 32554 (time 1746065183.24 total 115.42 last 255.37) - /unlink 34937 (time 1746065193.24 total 125.42 last 238.22) - /unlink 37276 (time 1746065203.24 total 135.43 last 233.83) - /unlink 40000 (time 1746065212.32 total 144.51 last 300.06) - /unlink 43363 (time 1746065222.32 total 154.51 last 336.27) - /unlink 46163 (time 1746065232.32 total 164.51 last 279.91) - /unlink 49011 (time 1746065242.33 total 174.51 last 284.76) - /unlink 50000 (time 1746065245.30 total 177.48 last 333.06) - /unlink 53727 (time 1746065255.30 total 187.48 last 372.68) - /unlink 57213 (time 1746065265.30 total 197.48 last 348.59) - /unlink 60000 (time 1746065274.80 total 206.98 last 293.31) - /unlink 62948 (time 1746065284.80 total 216.99 last 294.65) total: 64768 /unlink in 222.37 seconds: 291.27 ops/second Waiting for MDT destroys to complete - open/close 1878 (time 1746065312.74 total 10.00 last 187.71) - open/close 3688 (time 1746065322.76 total 20.02 last 180.80) - open/close 5597 (time 1746065332.76 total 30.02 last 190.83) - open/close 7822 (time 1746065342.76 total 40.02 last 222.50) - open/close 9848 (time 1746065352.76 total 50.02 last 202.53) - open/close 11906 (time 1746065362.76 total 60.03 last 205.77) - open/close 13537 (time 1746065372.77 total 70.03 last 163.08) - open/close 15361 (time 1746065382.77 total 80.03 last 182.34) - open/close 17446 (time 1746065392.77 total 90.03 last 208.47) - open/close 19040 (time 1746065402.77 total 100.03 last 159.35) - open/close 20000 (time 1746065408.06 total 105.32 last 181.55) - open/close 21657 (time 1746065418.06 total 115.32 last 165.68) - open/close 23146 (time 1746065428.06 total 125.33 last 148.87) - open/close 24529 (time 1746065438.07 total 135.33 last 138.23) - open/close 25809 (time 1746065448.07 total 145.33 last 127.96) - open/close 27206 (time 1746065458.07 total 155.33 last 139.69) - open/close 28591 (time 1746065468.07 total 165.34 last 138.49) - open/close 29891 (time 1746065478.08 total 175.34 last 129.95) - open/close 31290 (time 1746065488.08 total 185.34 last 139.89) - open/close 32695 (time 1746065498.08 total 195.35 last 140.43) - open/close 34026 (time 1746065508.09 total 205.35 last 133.00) - open/close 35823 (time 1746065518.10 total 215.36 last 179.62) - open/close 37696 (time 1746065528.10 total 225.37 last 187.16) - open/close 39287 (time 1746065538.11 total 235.37 last 159.03) - open/close 40000 (time 1746065542.46 total 239.73 last 163.68) - open/close 41781 (time 1746065552.46 total 249.73 last 178.10) - open/close 43795 (time 1746065562.47 total 259.73 last 201.35) - open/close 45640 (time 1746065572.47 total 269.73 last 184.42) - open/close 48366 (time 1746065582.47 total 279.73 last 272.59) - open/close 50000 (time 1746065589.13 total 286.39 last 245.47) - open/close 52199 (time 1746065599.13 total 296.39 last 219.83) - open/close 54127 (time 1746065609.14 total 306.40 last 192.65) - open/close 56150 (time 1746065619.14 total 316.40 last 202.22) - open/close 58030 (time 1746065629.14 total 326.40 last 187.99) - open/close 60000 (time 1746065637.24 total 334.51 last 243.19) - open/close 62569 (time 1746065647.25 total 344.51 last 256.78) - open/close 64746 (time 1746065657.25 total 354.51 last 217.68) total: 64768 open/close in 354.61 seconds: 182.65 ops/second - /unlink 3112 (time 1746065668.70 total 10.00 last 311.07) - /unlink 5524 (time 1746065678.70 total 20.01 last 241.16) - /unlink 7948 (time 1746065688.70 total 30.01 last 242.36) - /unlink 10000 (time 1746065697.38 total 38.68 last 236.48) - /unlink 12031 (time 1746065707.38 total 48.69 last 203.03) - /unlink 14121 (time 1746065717.38 total 58.69 last 208.98) - /unlink 16175 (time 1746065727.39 total 68.69 last 205.37) - /unlink 18590 (time 1746065737.39 total 78.69 last 241.48) - /unlink 20000 (time 1746065744.31 total 85.62 last 203.65) - /unlink 22036 (time 1746065754.31 total 95.62 last 203.51) - /unlink 23805 (time 1746065764.33 total 105.63 last 176.68) - /unlink 25520 (time 1746065774.34 total 115.64 last 171.33) - /unlink 27182 (time 1746065784.34 total 125.65 last 166.13) - /unlink 28690 (time 1746065794.34 total 135.65 last 150.76) - /unlink 30000 (time 1746065801.74 total 143.04 last 177.13) - /unlink 32280 (time 1746065811.74 total 153.05 last 227.92) - /unlink 35866 (time 1746065821.75 total 163.05 last 358.43) - /unlink 39749 (time 1746065831.75 total 173.05 last 388.29) - /unlink 43225 (time 1746065841.75 total 183.06 last 347.54) - /unlink 45969 (time 1746065851.75 total 193.06 last 274.40) - /unlink 48417 (time 1746065861.75 total 203.06 last 244.78) - /unlink 50000 (time 1746065866.68 total 207.99 last 320.94) - /unlink 52888 (time 1746065876.69 total 217.99 last 288.77) - /unlink 56033 (time 1746065886.69 total 227.99 last 314.38) - /unlink 58806 (time 1746065896.69 total 238.00 last 277.20) - /unlink 60000 (time 1746065901.41 total 242.72 last 252.89) - /unlink 62829 (time 1746065911.41 total 252.72 last 282.90) total: 64768 /unlink in 261.66 seconds: 247.53 ops/second Waiting for MDT destroys to complete - open/close 1353 (time 1746065938.56 total 10.00 last 135.29) - open/close 2591 (time 1746065948.57 total 20.01 last 123.74) - open/close 3702 (time 1746065958.57 total 30.01 last 111.08) - open/close 4840 (time 1746065968.58 total 40.02 last 113.63) - open/close 6103 (time 1746065978.58 total 50.02 last 126.27) - open/close 7344 (time 1746065988.59 total 60.03 last 124.04) - open/close 8693 (time 1746065998.59 total 70.03 last 134.89) - open/close 10000 (time 1746066008.03 total 79.47 last 138.52) - open/close 11407 (time 1746066018.03 total 89.47 last 140.65) - open/close 12791 (time 1746066028.03 total 99.47 last 138.34) - open/close 14349 (time 1746066038.04 total 109.48 last 155.66) - open/close 15721 (time 1746066048.05 total 119.49 last 137.15) - open/close 17035 (time 1746066058.05 total 129.49 last 131.34) - open/close 18383 (time 1746066068.05 total 139.49 last 134.75) - open/close 19862 (time 1746066078.06 total 149.49 last 147.88) - open/close 21104 (time 1746066088.07 total 159.51 last 124.02) - open/close 22326 (time 1746066098.07 total 169.51 last 122.15) - open/close 23536 (time 1746066108.08 total 179.52 last 120.97) - open/close 24823 (time 1746066118.08 total 189.52 last 128.70) - open/close 26018 (time 1746066128.08 total 199.52 last 119.45) - open/close 27194 (time 1746066138.10 total 209.54 last 117.40) - open/close 28370 (time 1746066148.10 total 219.54 last 117.58) - open/close 29546 (time 1746066158.10 total 229.54 last 117.55) - open/close 30000 (time 1746066161.92 total 233.36 last 119.09) - open/close 31198 (time 1746066171.92 total 243.36 last 119.74) - open/close 32410 (time 1746066181.93 total 253.37 last 121.10) - open/close 33675 (time 1746066191.93 total 263.37 last 126.48) - open/close 35003 (time 1746066201.93 total 273.37 last 132.76) - open/close 36349 (time 1746066211.93 total 283.37 last 134.59) - open/close 37743 (time 1746066221.94 total 293.38 last 139.32) - open/close 39098 (time 1746066231.94 total 303.38 last 135.47) - open/close 40000 (time 1746066238.67 total 310.11 last 134.06) - open/close 41333 (time 1746066248.67 total 320.11 last 133.30) - open/close 42719 (time 1746066258.68 total 330.12 last 138.53) - open/close 44043 (time 1746066268.68 total 340.12 last 132.38) - open/close 45365 (time 1746066278.68 total 350.12 last 132.15) - open/close 46705 (time 1746066288.68 total 360.12 last 134.00) - open/close 48138 (time 1746066298.68 total 370.12 last 143.27) - open/close 49463 (time 1746066308.68 total 380.12 last 132.49) - open/close 50000 (time 1746066312.24 total 383.68 last 151.11) - open/close 51459 (time 1746066322.24 total 393.68 last 145.88) - open/close 52750 (time 1746066332.24 total 403.68 last 129.06) - open/close 53983 (time 1746066342.25 total 413.69 last 123.24) - open/close 55344 (time 1746066352.25 total 423.69 last 136.04) - open/close 56650 (time 1746066362.25 total 433.69 last 130.58) - open/close 58077 (time 1746066372.26 total 443.70 last 142.59) - open/close 59450 (time 1746066382.27 total 453.71 last 137.23) - open/close 60000 (time 1746066386.10 total 457.54 last 143.54) - open/close 61456 (time 1746066396.10 total 467.54 last 145.56) - open/close 62833 (time 1746066406.10 total 477.54 last 137.66) - open/close 64296 (time 1746066416.11 total 487.55 last 146.24) total: 64768 open/close in 490.84 seconds: 131.95 ops/second - /unlink 2294 (time 1746066431.16 total 10.00 last 229.38) - /unlink 4483 (time 1746066441.16 total 20.00 last 218.85) - /unlink 7427 (time 1746066451.16 total 30.01 last 294.32) - /unlink 10000 (time 1746066458.61 total 37.46 last 345.36) - /unlink 12988 (time 1746066468.62 total 47.46 last 298.67) - /unlink 15472 (time 1746066478.62 total 57.46 last 248.36) - /unlink 17940 (time 1746066488.62 total 67.46 last 246.75) - /unlink 20000 (time 1746066497.50 total 76.34 last 231.98) - /unlink 22581 (time 1746066507.50 total 86.35 last 258.07) - /unlink 25441 (time 1746066517.50 total 96.35 last 285.93) - /unlink 27938 (time 1746066527.51 total 106.35 last 249.66) - /unlink 30000 (time 1746066535.15 total 113.99 last 269.75) - /unlink 32650 (time 1746066545.15 total 123.99 last 264.97) - /unlink 35458 (time 1746066555.16 total 134.00 last 280.68) - /unlink 38161 (time 1746066565.16 total 144.01 last 270.11) - /unlink 40000 (time 1746066571.96 total 150.80 last 270.72) - /unlink 42798 (time 1746066581.96 total 160.80 last 279.67) - /unlink 45556 (time 1746066591.96 total 170.80 last 275.78) - /unlink 48501 (time 1746066601.96 total 180.81 last 294.40) - /unlink 50000 (time 1746066607.42 total 186.26 last 274.76) - /unlink 52441 (time 1746066617.43 total 196.27 last 243.98) - /unlink 55109 (time 1746066627.43 total 206.27 last 266.74) - /unlink 57676 (time 1746066637.43 total 216.27 last 256.68) - /unlink 60000 (time 1746066646.06 total 224.90 last 269.21) - /unlink 62795 (time 1746066656.06 total 234.91 last 279.44) total: 64768 /unlink in 241.75 seconds: 267.92 ops/second Waiting for MDT destroys to complete - open/close 1464 (time 1746066680.25 total 10.01 last 146.28) - open/close 2852 (time 1746066690.25 total 20.01 last 138.78) - open/close 4253 (time 1746066700.25 total 30.01 last 140.06) - open/close 5659 (time 1746066710.26 total 40.02 last 140.50) - open/close 7050 (time 1746066720.26 total 50.02 last 139.06) - open/close 8394 (time 1746066730.26 total 60.02 last 134.40) - open/close 9819 (time 1746066740.26 total 70.03 last 142.45) - open/close 11130 (time 1746066750.27 total 80.03 last 131.01) - open/close 12618 (time 1746066760.27 total 90.04 last 148.79) - open/close 14101 (time 1746066770.27 total 100.04 last 148.29) - open/close 15655 (time 1746066780.28 total 110.04 last 155.33) - open/close 17121 (time 1746066790.28 total 120.04 last 146.56) - open/close 18494 (time 1746066800.29 total 130.05 last 137.22) - open/close 20000 (time 1746066810.24 total 140.00 last 151.28) - open/close 21791 (time 1746066820.26 total 150.02 last 178.83) - open/close 23515 (time 1746066830.26 total 160.02 last 172.36) - open/close 25172 (time 1746066840.26 total 170.02 last 165.68) - open/close 26703 (time 1746066850.27 total 180.03 last 153.02) - open/close 28240 (time 1746066860.28 total 190.05 last 153.41) - open/close 29582 (time 1746066870.29 total 200.05 last 134.19) - open/close 30000 (time 1746066873.49 total 203.25 last 130.47) - open/close 31442 (time 1746066883.49 total 213.25 last 144.16) - open/close 32822 (time 1746066893.50 total 223.26 last 137.94) - open/close 34149 (time 1746066903.51 total 233.27 last 132.53) - open/close 35407 (time 1746066913.51 total 243.27 last 125.77) - open/close 36877 (time 1746066923.52 total 253.28 last 146.95) - open/close 38262 (time 1746066933.52 total 263.28 last 138.47) - open/close 39553 (time 1746066943.53 total 273.29 last 128.99) - open/close 40000 (time 1746066947.82 total 277.58 last 104.14) - open/close 41341 (time 1746066957.82 total 287.58 last 134.07) - open/close 42773 (time 1746066967.83 total 297.59 last 143.13) - open/close 44300 (time 1746066977.83 total 307.59 last 152.64) - open/close 45728 (time 1746066987.83 total 317.60 last 142.75) - open/close 47134 (time 1746066997.84 total 327.60 last 140.57) - open/close 48532 (time 1746067007.84 total 337.60 last 139.79) - open/close 49869 (time 1746067017.84 total 347.60 last 133.67) - open/close 51152 (time 1746067027.84 total 357.60 last 128.28) - open/close 52505 (time 1746067037.84 total 367.61 last 135.25) - open/close 53894 (time 1746067047.85 total 377.61 last 138.86) - open/close 55239 (time 1746067057.85 total 387.62 last 134.41) - open/close 56569 (time 1746067067.86 total 397.62 last 132.88) - open/close 57962 (time 1746067077.87 total 407.63 last 139.24) - open/close 59271 (time 1746067087.87 total 417.63 last 130.85) - open/close 60000 (time 1746067092.91 total 422.67 last 144.79) - open/close 61501 (time 1746067102.91 total 432.67 last 150.10) - open/close 62939 (time 1746067112.91 total 442.67 last 143.76) - open/close 64410 (time 1746067122.91 total 452.67 last 147.07) total: 64768 open/close in 455.28 seconds: 142.26 ops/second - /unlink 2806 (time 1746067136.97 total 10.00 last 280.53) - /unlink 5517 (time 1746067146.97 total 20.00 last 271.10) - /unlink 8327 (time 1746067156.97 total 30.00 last 280.99) - /unlink 10000 (time 1746067162.77 total 35.81 last 288.27) - /unlink 14230 (time 1746067172.77 total 45.81 last 422.98) - /unlink 17240 (time 1746067182.77 total 55.81 last 300.95) - /unlink 20000 (time 1746067192.54 total 65.57 last 282.62) - /unlink 22763 (time 1746067202.54 total 75.58 last 276.26) - /unlink 25764 (time 1746067212.54 total 85.58 last 300.04) - /unlink 28751 (time 1746067222.55 total 95.58 last 298.58) - /unlink 30000 (time 1746067226.62 total 99.66 last 306.44) - /unlink 32788 (time 1746067236.62 total 109.66 last 278.80) - /unlink 36646 (time 1746067246.63 total 119.66 last 385.70) - /unlink 40000 (time 1746067256.18 total 129.22 last 350.99) - /unlink 43099 (time 1746067266.18 total 139.22 last 309.87) - /unlink 45959 (time 1746067276.19 total 149.22 last 285.86) - /unlink 48663 (time 1746067286.19 total 159.22 last 270.39) - /unlink 50000 (time 1746067290.88 total 163.92 last 284.71) - /unlink 52737 (time 1746067300.89 total 173.92 last 273.57) - /unlink 55850 (time 1746067310.89 total 183.92 last 311.28) - /unlink 59041 (time 1746067320.89 total 193.93 last 319.03) - /unlink 60000 (time 1746067324.35 total 197.38 last 277.53) - /unlink 63187 (time 1746067334.35 total 207.38 last 318.64) total: 64768 /unlink in 212.29 seconds: 305.09 ops/second Waiting for MDT destroys to complete - open/close 2317 (time 1746067360.84 total 10.00 last 231.64) - open/close 4176 (time 1746067370.84 total 20.00 last 185.88) - open/close 6150 (time 1746067380.84 total 30.01 last 197.37) - open/close 7998 (time 1746067390.85 total 40.01 last 184.77) - open/close 9679 (time 1746067400.85 total 50.01 last 168.06) - open/close 11593 (time 1746067410.85 total 60.01 last 191.39) - open/close 13703 (time 1746067420.85 total 70.01 last 210.92) - open/close 15557 (time 1746067430.85 total 80.01 last 185.38) - open/close 16930 (time 1746067440.85 total 90.02 last 137.28) - open/close 18330 (time 1746067450.86 total 100.02 last 139.96) - open/close 19809 (time 1746067460.86 total 110.02 last 147.85) - open/close 21237 (time 1746067470.86 total 120.02 last 142.77) - open/close 22744 (time 1746067480.87 total 130.03 last 150.66) - open/close 24182 (time 1746067490.87 total 140.03 last 143.76) - open/close 25704 (time 1746067500.87 total 150.04 last 152.12) - open/close 27285 (time 1746067510.88 total 160.04 last 158.04) - open/close 29111 (time 1746067520.88 total 170.04 last 182.53) - open/close 30000 (time 1746067526.32 total 175.48 last 163.54) - open/close 31509 (time 1746067536.32 total 185.48 last 150.81) - open/close 32970 (time 1746067546.33 total 195.49 last 146.07) - open/close 34463 (time 1746067556.33 total 205.49 last 149.29) - open/close 35798 (time 1746067566.33 total 215.49 last 133.46) - open/close 37275 (time 1746067576.33 total 225.49 last 147.68) - open/close 38681 (time 1746067586.34 total 235.50 last 140.51) - open/close 40000 (time 1746067594.65 total 243.81 last 158.60) - open/close 41657 (time 1746067604.66 total 253.82 last 165.64) - open/close 43387 (time 1746067614.67 total 263.83 last 172.85) - open/close 44830 (time 1746067624.67 total 273.83 last 144.26) - open/close 46372 (time 1746067634.68 total 283.84 last 154.06) - open/close 47967 (time 1746067644.68 total 293.84 last 159.42) - open/close 49711 (time 1746067654.68 total 303.85 last 174.38) - open/close 51324 (time 1746067664.69 total 313.85 last 161.26) - open/close 53232 (time 1746067674.69 total 323.85 last 190.78) - open/close 55344 (time 1746067684.69 total 333.85 last 211.12) - open/close 57302 (time 1746067694.69 total 343.85 last 195.79) - open/close 58982 (time 1746067704.70 total 353.86 last 167.93) - open/close 60000 (time 1746067711.12 total 360.28 last 158.54) - open/close 61497 (time 1746067721.12 total 370.28 last 149.64) - open/close 63155 (time 1746067731.12 total 380.28 last 165.80) - open/close 64680 (time 1746067741.12 total 390.28 last 152.48) total: 64768 open/close in 390.80 seconds: 165.73 ops/second - /unlink 2437 (time 1746067753.07 total 10.00 last 243.66) - /unlink 4594 (time 1746067763.08 total 20.00 last 215.63) - /unlink 6896 (time 1746067773.08 total 30.01 last 230.16) - /unlink 9206 (time 1746067783.08 total 40.01 last 230.89) - /unlink 10000 (time 1746067786.13 total 43.06 last 260.65) - /unlink 12347 (time 1746067796.13 total 53.06 last 234.67) - /unlink 15509 (time 1746067806.13 total 63.06 last 316.18) - /unlink 18232 (time 1746067816.13 total 73.06 last 272.25) - /unlink 20000 (time 1746067822.50 total 79.43 last 277.80) - /unlink 22699 (time 1746067832.50 total 89.43 last 269.83) - /unlink 24895 (time 1746067842.50 total 99.43 last 219.50) - /unlink 27256 (time 1746067852.51 total 109.44 last 235.99) - /unlink 29485 (time 1746067862.51 total 119.44 last 222.87) - /unlink 30000 (time 1746067865.15 total 122.08 last 195.25) - /unlink 32676 (time 1746067875.15 total 132.08 last 267.59) - /unlink 35358 (time 1746067885.15 total 142.08 last 268.15) - /unlink 38161 (time 1746067895.15 total 152.08 last 280.21) - /unlink 40000 (time 1746067903.16 total 160.09 last 229.54) - /unlink 42443 (time 1746067913.17 total 170.09 last 244.29) - /unlink 45131 (time 1746067923.17 total 180.10 last 268.75) - /unlink 47898 (time 1746067933.17 total 190.10 last 276.62) - /unlink 50000 (time 1746067940.79 total 197.72 last 275.91) - /unlink 52769 (time 1746067950.79 total 207.72 last 276.79) - /unlink 55785 (time 1746067960.80 total 217.72 last 301.51) - /unlink 58772 (time 1746067970.80 total 227.72 last 298.69) - /unlink 60000 (time 1746067975.16 total 232.09 last 281.29) - /unlink 62797 (time 1746067985.16 total 242.09 last 279.68) total: 64768 /unlink in 247.62 seconds: 261.56 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg430-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg430-server unloading modules via unload_modules_local on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg430-client: oleg430-client: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-client: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 PASS 106 (3406s) Stopping clients: oleg430-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg430-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 oleg430-server: oleg430-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing load_modules_local oleg430-server: Loading modules from /home/green/git/lustre-release/lustre oleg430-server: detected 4 online CPUs by sysfs oleg430-server: MODOPTS_LIBCFS= oleg430-server: Force libcfs to create 2 CPU partitions oleg430-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg430-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3550 sec ========== 22:54:12 (1746068052) === conf-sanity: start cleanup 22:54:13 (1746068053) === === conf-sanity: finish cleanup 22:54:14 (1746068054) ===