-----============= acceptance-small: conf-sanity ============----- Thu May 1 15:29:07 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg352-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg352-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg352-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg352-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg352-server oleg352-server: oleg352-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg352-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg352-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg352-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg352-server: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg352-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg352-server: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg352-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg352-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:31:00 (1746127860) Stopping clients: oleg352-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg352-client.virtnet /mnt/lustre2 (opts:-f) oleg352-server: oleg352-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions oleg352-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg352-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg352-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg352-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg352-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg352-client.virtnet: -o user_xattr,flock 192.168.203.152@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1265 (time 1746127926.20 total 10.01 last 126.43) - open/close 2397 (time 1746127936.21 total 20.02 last 113.08) - open/close 3575 (time 1746127946.21 total 30.02 last 117.76) - open/close 4701 (time 1746127956.21 total 40.02 last 112.55) - open/close 5763 (time 1746127966.21 total 50.02 last 106.19) - open/close 6858 (time 1746127976.22 total 60.03 last 109.47) - open/close 8033 (time 1746127986.23 total 70.04 last 117.39) - open/close 9199 (time 1746127996.23 total 80.04 last 116.59) - open/close 10000 (time 1746128002.85 total 86.66 last 120.99) - open/close 11180 (time 1746128012.85 total 96.66 last 117.94) - open/close 12291 (time 1746128022.86 total 106.67 last 110.99) - open/close 13484 (time 1746128032.87 total 116.68 last 119.28) - open/close 14630 (time 1746128042.87 total 126.68 last 114.60) - open/close 15871 (time 1746128052.89 total 136.70 last 123.80) - open/close 16992 (time 1746128062.90 total 146.71 last 112.01) - open/close 18096 (time 1746128072.90 total 156.71 last 110.37) - open/close 19168 (time 1746128082.91 total 166.72 last 107.05) - open/close 20000 (time 1746128091.16 total 174.97 last 100.86) - open/close 21074 (time 1746128101.16 total 184.97 last 107.38) - open/close 22067 (time 1746128111.17 total 194.98 last 99.22) - open/close 23114 (time 1746128121.18 total 204.99 last 104.65) - open/close 24206 (time 1746128131.18 total 214.99 last 109.19) - open/close 25324 (time 1746128141.18 total 224.99 last 111.78) - open/close 26412 (time 1746128151.18 total 234.99 last 108.76) - open/close 27411 (time 1746128161.19 total 245.00 last 99.82) - open/close 28370 (time 1746128171.20 total 255.01 last 95.83) - open/close 29210 (time 1746128181.20 total 265.01 last 83.98) - open/close 30000 (time 1746128189.67 total 273.48 last 93.25) - open/close 31176 (time 1746128199.68 total 283.49 last 117.56) - open/close 32155 (time 1746128209.68 total 293.49 last 97.86) - open/close 33103 (time 1746128219.69 total 303.50 last 94.75) - open/close 33986 (time 1746128229.70 total 313.51 last 88.22) - open/close 35077 (time 1746128239.70 total 323.51 last 109.03) - open/close 36109 (time 1746128249.71 total 333.52 last 103.16) - open/close 37065 (time 1746128259.71 total 343.52 last 95.51) - open/close 38065 (time 1746128269.72 total 353.53 last 99.98) - open/close 39043 (time 1746128279.72 total 363.53 last 97.79) - open/close 40000 (time 1746128289.43 total 373.24 last 98.49) - open/close 40974 (time 1746128299.44 total 383.25 last 97.36) - open/close 41978 (time 1746128309.44 total 393.25 last 100.37) - open/close 43109 (time 1746128319.45 total 403.26 last 112.97) - open/close 44214 (time 1746128329.46 total 413.27 last 110.45) - open/close 45336 (time 1746128339.47 total 423.28 last 112.07) - open/close 46442 (time 1746128349.47 total 433.28 last 110.54) - open/close 47607 (time 1746128359.49 total 443.30 last 116.35) - open/close 48639 (time 1746128369.49 total 453.30 last 103.15) - open/close 49631 (time 1746128379.50 total 463.31 last 99.11) - open/close 50000 (time 1746128382.87 total 466.68 last 109.48) - open/close 50991 (time 1746128392.88 total 476.69 last 99.01) - open/close 52100 (time 1746128402.88 total 486.69 last 110.85) - open/close 53220 (time 1746128412.89 total 496.70 last 111.91) - open/close 54448 (time 1746128422.90 total 506.71 last 122.74) - open/close 55743 (time 1746128432.90 total 516.71 last 129.48) - open/close 57040 (time 1746128442.91 total 526.72 last 129.53) - open/close 58281 (time 1746128452.92 total 536.73 last 124.07) - open/close 59536 (time 1746128462.92 total 546.73 last 125.47) - open/close 60000 (time 1746128466.65 total 550.46 last 124.28) - open/close 61170 (time 1746128476.65 total 560.46 last 116.99) - open/close 62174 (time 1746128486.65 total 570.46 last 100.38) - open/close 63320 (time 1746128496.66 total 580.47 last 114.55) - open/close 64415 (time 1746128506.66 total 590.47 last 109.45) total: 64768 open/close in 593.65 seconds: 109.10 ops/second - /unlink 1924 (time 1746128522.11 total 10.00 last 192.36) - /unlink 3737 (time 1746128532.12 total 20.01 last 181.23) - /unlink 5684 (time 1746128542.12 total 30.01 last 194.68) - /unlink 7796 (time 1746128552.12 total 40.01 last 211.09) - /unlink 9867 (time 1746128562.13 total 50.02 last 207.05) - /unlink 11843 (time 1746128572.13 total 60.02 last 197.56) - /unlink 13791 (time 1746128582.13 total 70.02 last 194.76) - /unlink 15783 (time 1746128592.13 total 80.02 last 199.16) - /unlink 17489 (time 1746128602.13 total 90.02 last 170.59) - /unlink 19089 (time 1746128612.14 total 100.03 last 159.87) - /unlink 20000 (time 1746128617.19 total 105.07 last 180.59) - /unlink 21923 (time 1746128627.19 total 115.08 last 192.24) - /unlink 23775 (time 1746128637.19 total 125.08 last 185.15) - /unlink 25734 (time 1746128647.20 total 135.09 last 195.72) - /unlink 27656 (time 1746128657.20 total 145.09 last 192.18) - /unlink 29659 (time 1746128667.21 total 155.09 last 200.23) - /unlink 31641 (time 1746128677.21 total 165.09 last 198.18) - /unlink 34020 (time 1746128687.21 total 175.10 last 237.76) - /unlink 36196 (time 1746128697.21 total 185.10 last 217.59) - /unlink 38466 (time 1746128707.22 total 195.11 last 226.83) - /unlink 40000 (time 1746128715.59 total 203.48 last 183.30) - /unlink 41910 (time 1746128725.59 total 213.48 last 190.99) - /unlink 43896 (time 1746128735.59 total 223.48 last 198.59) - /unlink 46084 (time 1746128745.59 total 233.48 last 218.74) - /unlink 48034 (time 1746128755.59 total 243.48 last 194.96) - /unlink 50000 (time 1746128764.47 total 252.36 last 221.58) - /unlink 51917 (time 1746128774.47 total 262.36 last 191.65) - /unlink 53773 (time 1746128784.47 total 272.36 last 185.57) - /unlink 55606 (time 1746128794.49 total 282.38 last 182.93) - /unlink 57544 (time 1746128804.49 total 292.38 last 193.79) - /unlink 59592 (time 1746128814.49 total 302.38 last 204.75) - /unlink 60000 (time 1746128816.59 total 304.48 last 194.95) - /unlink 61982 (time 1746128826.59 total 314.48 last 198.11) - /unlink 63882 (time 1746128836.60 total 324.48 last 189.92) total: 64768 /unlink in 330.17 seconds: 196.16 ops/second Waiting for MDT destroys to complete - open/close 1194 (time 1746128864.13 total 10.01 last 119.29) - open/close 2296 (time 1746128874.15 total 20.03 last 109.98) - open/close 3207 (time 1746128884.15 total 30.03 last 91.05) - open/close 4317 (time 1746128894.16 total 40.04 last 110.98) - open/close 5409 (time 1746128904.17 total 50.05 last 109.02) - open/close 6456 (time 1746128914.18 total 60.06 last 104.67) - open/close 7463 (time 1746128924.18 total 70.06 last 100.62) - open/close 8439 (time 1746128934.20 total 80.08 last 97.45) - open/close 9541 (time 1746128944.20 total 90.08 last 110.19) - open/close 10000 (time 1746128948.61 total 94.49 last 104.04) - open/close 11063 (time 1746128958.61 total 104.49 last 106.27) - open/close 12251 (time 1746128968.62 total 114.50 last 118.75) - open/close 13378 (time 1746128978.62 total 124.51 last 112.63) - open/close 14376 (time 1746128988.62 total 134.51 last 99.80) - open/close 15455 (time 1746128998.63 total 144.51 last 107.82) - open/close 16560 (time 1746129008.63 total 154.51 last 110.47) - open/close 17704 (time 1746129018.65 total 164.53 last 114.25) - open/close 18764 (time 1746129028.66 total 174.54 last 105.86) - open/close 19847 (time 1746129038.66 total 184.54 last 108.27) - open/close 20883 (time 1746129048.67 total 194.55 last 103.50) - open/close 21917 (time 1746129058.68 total 204.56 last 103.31) - open/close 22995 (time 1746129068.69 total 214.57 last 107.74) - open/close 24168 (time 1746129078.69 total 224.57 last 117.27) - open/close 25374 (time 1746129088.69 total 234.57 last 120.55) - open/close 26628 (time 1746129098.70 total 244.58 last 125.33) - open/close 27770 (time 1746129108.70 total 254.58 last 114.15) - open/close 28959 (time 1746129118.71 total 264.59 last 118.88) - open/close 30000 (time 1746129127.39 total 273.27 last 119.87) - open/close 31340 (time 1746129137.39 total 283.27 last 133.94) - open/close 32636 (time 1746129147.40 total 293.28 last 129.58) - open/close 34029 (time 1746129157.40 total 303.28 last 139.26) - open/close 35268 (time 1746129167.40 total 313.28 last 123.85) - open/close 36551 (time 1746129177.41 total 323.29 last 128.24) - open/close 37724 (time 1746129187.41 total 333.29 last 117.27) - open/close 38988 (time 1746129197.41 total 343.29 last 126.36) - open/close 40000 (time 1746129205.66 total 351.54 last 122.70) - open/close 41221 (time 1746129215.67 total 361.55 last 122.02) - open/close 42558 (time 1746129225.67 total 371.55 last 133.69) - open/close 44013 (time 1746129235.67 total 381.55 last 145.44) - open/close 45419 (time 1746129245.67 total 391.56 last 140.55) - open/close 46719 (time 1746129255.68 total 401.56 last 129.99) - open/close 48093 (time 1746129265.68 total 411.56 last 137.32) - open/close 49349 (time 1746129275.69 total 421.57 last 125.53) - open/close 50000 (time 1746129281.36 total 427.24 last 114.70) - open/close 51151 (time 1746129291.36 total 437.24 last 115.10) - open/close 52253 (time 1746129301.37 total 447.25 last 110.12) - open/close 53447 (time 1746129311.38 total 457.26 last 119.24) - open/close 54732 (time 1746129321.40 total 467.28 last 128.26) - open/close 56054 (time 1746129331.40 total 477.28 last 132.19) - open/close 57311 (time 1746129341.42 total 487.30 last 125.45) - open/close 58524 (time 1746129351.44 total 497.32 last 121.04) - open/close 59686 (time 1746129361.45 total 507.33 last 116.13) - open/close 60000 (time 1746129364.02 total 509.90 last 122.08) - open/close 61328 (time 1746129374.03 total 519.91 last 132.66) - open/close 62728 (time 1746129384.03 total 529.91 last 139.98) - open/close 64173 (time 1746129394.03 total 539.92 last 144.48) total: 64768 open/close in 544.07 seconds: 119.04 ops/second - /unlink 3077 (time 1746129409.79 total 10.00 last 307.64) - /unlink 6055 (time 1746129419.80 total 20.01 last 297.66) - /unlink 8615 (time 1746129429.80 total 30.01 last 255.98) - /unlink 10000 (time 1746129434.84 total 35.04 last 275.04) - /unlink 12931 (time 1746129444.85 total 45.05 last 292.77) - /unlink 15798 (time 1746129454.85 total 55.06 last 286.64) - /unlink 18332 (time 1746129464.85 total 65.06 last 253.35) - /unlink 20000 (time 1746129471.69 total 71.90 last 243.81) - /unlink 22879 (time 1746129481.70 total 81.90 last 287.80) - /unlink 25211 (time 1746129491.70 total 91.91 last 233.09) - /unlink 27734 (time 1746129501.70 total 101.91 last 252.21) - /unlink 30000 (time 1746129510.49 total 110.69 last 258.00) - /unlink 32976 (time 1746129520.49 total 120.70 last 297.55) - /unlink 35517 (time 1746129530.49 total 130.70 last 254.08) - /unlink 38179 (time 1746129540.49 total 140.70 last 266.17) - /unlink 40000 (time 1746129547.64 total 147.84 last 254.84) - /unlink 42349 (time 1746129557.64 total 157.85 last 234.85) - /unlink 44438 (time 1746129567.65 total 167.85 last 208.75) - /unlink 46552 (time 1746129577.65 total 177.86 last 211.32) - /unlink 48947 (time 1746129587.65 total 187.86 last 239.42) - /unlink 50000 (time 1746129592.35 total 192.55 last 224.33) - /unlink 52724 (time 1746129602.35 total 202.56 last 272.36) - /unlink 55634 (time 1746129612.35 total 212.56 last 290.94) - /unlink 58084 (time 1746129622.35 total 222.56 last 244.94) - /unlink 60000 (time 1746129630.14 total 230.35 last 245.97) - /unlink 62073 (time 1746129640.17 total 240.37 last 206.78) - /unlink 64196 (time 1746129650.17 total 250.38 last 212.24) total: 64768 /unlink in 253.20 seconds: 255.80 ops/second Waiting for MDT destroys to complete - open/close 1738 (time 1746129673.59 total 10.00 last 173.76) - open/close 3329 (time 1746129683.60 total 20.01 last 158.96) - open/close 4769 (time 1746129693.60 total 30.01 last 143.99) - open/close 5999 (time 1746129703.60 total 40.02 last 122.95) - open/close 7310 (time 1746129713.60 total 50.02 last 131.08) - open/close 8641 (time 1746129723.61 total 60.02 last 133.01) - open/close 9982 (time 1746129733.62 total 70.03 last 134.02) - open/close 11446 (time 1746129743.62 total 80.03 last 146.38) - open/close 12850 (time 1746129753.62 total 90.03 last 140.36) - open/close 14284 (time 1746129763.62 total 100.04 last 143.35) - open/close 15511 (time 1746129773.63 total 110.04 last 122.62) - open/close 16782 (time 1746129783.63 total 120.05 last 127.07) - open/close 18043 (time 1746129793.64 total 130.05 last 126.09) - open/close 19166 (time 1746129803.64 total 140.05 last 112.24) - open/close 20000 (time 1746129810.68 total 147.09 last 118.52) - open/close 21322 (time 1746129820.68 total 157.09 last 132.16) - open/close 22677 (time 1746129830.69 total 167.10 last 135.41) - open/close 23927 (time 1746129840.69 total 177.10 last 124.98) - open/close 25275 (time 1746129850.69 total 187.11 last 134.71) - open/close 26589 (time 1746129860.70 total 197.11 last 131.32) - open/close 27805 (time 1746129870.70 total 207.12 last 121.58) - open/close 29087 (time 1746129880.71 total 217.12 last 128.14) - open/close 30000 (time 1746129887.78 total 224.20 last 128.99) - open/close 31003 (time 1746129897.79 total 234.20 last 100.25) - open/close 32110 (time 1746129907.79 total 244.21 last 110.67) - open/close 33249 (time 1746129917.80 total 254.21 last 113.84) - open/close 34471 (time 1746129927.80 total 264.21 last 122.17) - open/close 35691 (time 1746129937.80 total 274.22 last 121.97) - open/close 36817 (time 1746129947.81 total 284.22 last 112.56) - open/close 38117 (time 1746129957.81 total 294.22 last 129.99) - open/close 39400 (time 1746129967.81 total 304.23 last 128.21) - open/close 40000 (time 1746129971.93 total 308.34 last 145.72) - open/close 41310 (time 1746129981.94 total 318.36 last 130.86) - open/close 42575 (time 1746129991.95 total 328.36 last 126.39) - open/close 43928 (time 1746130001.96 total 338.37 last 135.23) - open/close 45190 (time 1746130011.96 total 348.37 last 126.19) - open/close 46471 (time 1746130021.96 total 358.37 last 128.04) - open/close 47643 (time 1746130031.97 total 368.38 last 117.16) - open/close 48850 (time 1746130041.97 total 378.38 last 120.67) - open/close 50000 (time 1746130050.74 total 387.15 last 131.13) - open/close 51187 (time 1746130060.74 total 397.15 last 118.69) - open/close 52419 (time 1746130070.74 total 407.16 last 123.16) - open/close 53555 (time 1746130080.75 total 417.16 last 113.54) - open/close 54773 (time 1746130090.75 total 427.16 last 121.76) - open/close 56030 (time 1746130100.75 total 437.17 last 125.67) - open/close 57434 (time 1746130110.76 total 447.17 last 140.33) - open/close 58680 (time 1746130120.77 total 457.18 last 124.48) - open/close 59930 (time 1746130130.77 total 467.18 last 124.97) - open/close 61179 (time 1746130140.77 total 477.19 last 124.83) - open/close 62442 (time 1746130150.78 total 487.19 last 126.23) - open/close 63712 (time 1746130160.78 total 497.20 last 126.95) total: 64768 open/close in 505.66 seconds: 128.09 ops/second - /unlink 2136 (time 1746130180.98 total 10.00 last 213.60) - /unlink 4217 (time 1746130190.98 total 20.00 last 208.07) - /unlink 6263 (time 1746130200.98 total 30.00 last 204.57) - /unlink 8414 (time 1746130211.00 total 40.02 last 214.69) - /unlink 10000 (time 1746130218.08 total 47.10 last 224.05) - /unlink 12245 (time 1746130228.09 total 57.11 last 224.33) - /unlink 14306 (time 1746130238.09 total 67.11 last 206.02) - /unlink 16120 (time 1746130248.09 total 77.11 last 181.40) - /unlink 18592 (time 1746130258.09 total 87.11 last 247.20) - /unlink 20000 (time 1746130263.92 total 92.93 last 241.86) - /unlink 21874 (time 1746130273.92 total 102.94 last 187.31) - /unlink 24079 (time 1746130283.92 total 112.94 last 220.45) - /unlink 26662 (time 1746130293.93 total 122.95 last 258.20) - /unlink 28674 (time 1746130303.93 total 132.95 last 201.18) - /unlink 30000 (time 1746130310.82 total 139.84 last 192.36) - /unlink 32271 (time 1746130320.83 total 149.85 last 226.94) - /unlink 34528 (time 1746130330.84 total 159.86 last 225.51) - /unlink 36421 (time 1746130340.84 total 169.86 last 189.23) - /unlink 38278 (time 1746130350.84 total 179.86 last 185.69) - /unlink 40000 (time 1746130357.90 total 186.92 last 244.03) - /unlink 42256 (time 1746130367.90 total 196.92 last 225.52) - /unlink 44600 (time 1746130377.90 total 206.92 last 234.34) - /unlink 46894 (time 1746130387.90 total 216.92 last 229.37) - /unlink 49301 (time 1746130397.91 total 226.93 last 240.58) - /unlink 50000 (time 1746130400.98 total 230.00 last 227.94) - /unlink 52179 (time 1746130410.98 total 240.00 last 217.85) - /unlink 54125 (time 1746130420.98 total 250.00 last 194.56) - /unlink 55991 (time 1746130430.99 total 260.01 last 186.49) - /unlink 58082 (time 1746130440.99 total 270.01 last 209.06) - /unlink 60000 (time 1746130450.07 total 279.09 last 211.22) - /unlink 62185 (time 1746130460.07 total 289.09 last 218.47) - /unlink 64322 (time 1746130470.07 total 299.09 last 213.66) total: 64768 /unlink in 301.01 seconds: 215.17 ops/second Waiting for MDT destroys to complete - open/close 1317 (time 1746130495.27 total 10.00 last 131.64) - open/close 2555 (time 1746130505.28 total 20.01 last 123.74) - open/close 3852 (time 1746130515.28 total 30.01 last 129.69) - open/close 5171 (time 1746130525.28 total 40.01 last 131.86) - open/close 6416 (time 1746130535.29 total 50.02 last 124.39) - open/close 7779 (time 1746130545.29 total 60.03 last 136.25) - open/close 9125 (time 1746130555.31 total 70.04 last 134.36) - open/close 10000 (time 1746130562.00 total 76.73 last 130.78) - open/close 11446 (time 1746130572.01 total 86.74 last 144.54) - open/close 12952 (time 1746130582.01 total 96.74 last 150.54) - open/close 14420 (time 1746130592.01 total 106.75 last 146.75) - open/close 15917 (time 1746130602.02 total 116.75 last 149.67) - open/close 17313 (time 1746130612.02 total 126.75 last 139.55) - open/close 18778 (time 1746130622.02 total 136.75 last 146.49) - open/close 20000 (time 1746130631.23 total 145.96 last 132.66) - open/close 21177 (time 1746130641.23 total 155.97 last 117.68) - open/close 22508 (time 1746130651.24 total 165.97 last 133.04) - open/close 23746 (time 1746130661.24 total 175.98 last 123.72) - open/close 25084 (time 1746130671.25 total 185.98 last 133.74) - open/close 26470 (time 1746130681.25 total 195.98 last 138.59) - open/close 27563 (time 1746130691.25 total 205.98 last 109.28) - open/close 28761 (time 1746130701.26 total 215.99 last 119.75) - open/close 30000 (time 1746130709.68 total 224.41 last 147.09) - open/close 31213 (time 1746130719.69 total 234.42 last 121.19) - open/close 32309 (time 1746130729.69 total 244.42 last 109.56) - open/close 33463 (time 1746130739.70 total 254.43 last 115.30) - open/close 34683 (time 1746130749.71 total 264.44 last 121.94) - open/close 35819 (time 1746130759.71 total 274.44 last 113.56) - open/close 37010 (time 1746130769.72 total 284.45 last 119.03) - open/close 38194 (time 1746130779.72 total 294.45 last 118.35) - open/close 39497 (time 1746130789.72 total 304.46 last 130.25) - open/close 40000 (time 1746130793.98 total 308.71 last 118.14) - open/close 41196 (time 1746130803.99 total 318.72 last 119.55) - open/close 42531 (time 1746130813.99 total 328.72 last 133.44) - open/close 43830 (time 1746130824.00 total 338.73 last 129.81) - open/close 44965 (time 1746130834.01 total 348.74 last 113.41) - open/close 46102 (time 1746130844.01 total 358.74 last 113.67) - open/close 47341 (time 1746130854.02 total 368.75 last 123.77) - open/close 48527 (time 1746130864.02 total 378.76 last 118.53) - open/close 49644 (time 1746130874.03 total 388.76 last 111.69) - open/close 50000 (time 1746130877.08 total 391.81 last 116.56) - open/close 51239 (time 1746130887.09 total 401.82 last 123.82) - open/close 52412 (time 1746130897.09 total 411.82 last 117.23) - open/close 53484 (time 1746130907.10 total 421.83 last 107.17) - open/close 54558 (time 1746130917.10 total 431.83 last 107.33) - open/close 55636 (time 1746130927.11 total 441.84 last 107.71) - open/close 56673 (time 1746130937.11 total 451.85 last 103.66) - open/close 57689 (time 1746130947.12 total 461.85 last 101.56) - open/close 58708 (time 1746130957.12 total 471.85 last 101.89) - open/close 59815 (time 1746130967.12 total 481.85 last 110.67) - open/close 60949 (time 1746130977.12 total 491.85 last 113.39) - open/close 62004 (time 1746130987.13 total 501.86 last 105.45) - open/close 63062 (time 1746130997.13 total 511.86 last 105.75) - open/close 64137 (time 1746131007.14 total 521.87 last 107.41) total: 64768 open/close in 528.38 seconds: 122.58 ops/second - /unlink 1831 (time 1746131025.62 total 10.00 last 183.04) - /unlink 3827 (time 1746131035.62 total 20.01 last 199.55) - /unlink 5895 (time 1746131045.62 total 30.01 last 206.79) - /unlink 8015 (time 1746131055.62 total 40.01 last 212.00) - /unlink 10000 (time 1746131064.84 total 49.22 last 215.37) - /unlink 12191 (time 1746131074.84 total 59.22 last 219.08) - /unlink 13993 (time 1746131084.84 total 69.23 last 180.19) - /unlink 16029 (time 1746131094.84 total 79.23 last 203.51) - /unlink 18595 (time 1746131104.85 total 89.24 last 256.47) - /unlink 20000 (time 1746131112.08 total 96.47 last 194.23) - /unlink 22041 (time 1746131122.09 total 106.47 last 204.00) - /unlink 24226 (time 1746131132.09 total 116.47 last 218.50) - /unlink 26614 (time 1746131142.09 total 126.47 last 238.78) - /unlink 28608 (time 1746131152.09 total 136.48 last 199.34) - /unlink 30000 (time 1746131159.02 total 143.41 last 200.91) - /unlink 31938 (time 1746131169.02 total 153.41 last 193.72) - /unlink 34352 (time 1746131179.03 total 163.41 last 241.29) - /unlink 36320 (time 1746131189.03 total 173.42 last 196.74) - /unlink 38066 (time 1746131199.03 total 183.42 last 174.58) - /unlink 40000 (time 1746131208.18 total 192.57 last 211.32) - /unlink 42044 (time 1746131218.18 total 202.57 last 204.37) - /unlink 44126 (time 1746131228.19 total 212.58 last 208.14) - /unlink 45886 (time 1746131238.19 total 222.58 last 175.95) - /unlink 48125 (time 1746131248.19 total 232.58 last 223.89) - /unlink 50000 (time 1746131256.89 total 241.28 last 215.48) - /unlink 51944 (time 1746131266.90 total 251.29 last 194.28) - /unlink 53856 (time 1746131276.90 total 261.29 last 191.15) - /unlink 55919 (time 1746131286.90 total 271.29 last 206.26) - /unlink 58039 (time 1746131296.91 total 281.29 last 211.93) - /unlink 59806 (time 1746131306.91 total 291.29 last 176.69) - /unlink 61673 (time 1746131316.91 total 301.29 last 186.69) - /unlink 63738 (time 1746131326.92 total 311.31 last 206.17) total: 64768 /unlink in 316.46 seconds: 204.67 ops/second Waiting for MDT destroys to complete - open/close 1051 (time 1746131352.23 total 10.00 last 105.05) - open/close 2126 (time 1746131362.23 total 20.01 last 107.42) - open/close 3265 (time 1746131372.24 total 30.02 last 113.86) - open/close 4495 (time 1746131382.24 total 40.02 last 122.92) - open/close 5553 (time 1746131392.25 total 50.03 last 105.75) - open/close 6591 (time 1746131402.26 total 60.03 last 103.74) - open/close 7728 (time 1746131412.26 total 70.03 last 113.69) - open/close 8891 (time 1746131422.26 total 80.04 last 116.24) - open/close 10000 (time 1746131431.99 total 89.77 last 114.02) - open/close 11151 (time 1746131442.00 total 99.77 last 115.01) - open/close 12323 (time 1746131452.00 total 109.78 last 117.17) - open/close 13549 (time 1746131462.01 total 119.78 last 122.50) - open/close 14756 (time 1746131472.01 total 129.79 last 120.64) - open/close 15975 (time 1746131482.02 total 139.79 last 121.85) - open/close 17179 (time 1746131492.03 total 149.80 last 120.29) - open/close 18376 (time 1746131502.03 total 159.80 last 119.69) - open/close 19446 (time 1746131512.03 total 169.81 last 106.92) - open/close 20000 (time 1746131516.97 total 174.74 last 112.29) - open/close 21210 (time 1746131526.97 total 184.75 last 120.94) - open/close 22339 (time 1746131536.97 total 194.75 last 112.88) - open/close 23519 (time 1746131546.99 total 204.76 last 117.88) - open/close 24720 (time 1746131556.99 total 214.77 last 120.06) - open/close 25915 (time 1746131566.99 total 224.77 last 119.48) - open/close 27066 (time 1746131576.99 total 234.77 last 115.09) - open/close 28265 (time 1746131586.99 total 244.77 last 119.88) - open/close 29335 (time 1746131596.99 total 254.77 last 106.98) - open/close 30000 (time 1746131603.49 total 261.27 last 102.34) - open/close 31096 (time 1746131613.50 total 271.27 last 109.55) - open/close 32244 (time 1746131623.50 total 281.28 last 114.78) - open/close 33293 (time 1746131633.51 total 291.28 last 104.81) - open/close 34250 (time 1746131643.52 total 301.29 last 95.62) - open/close 35274 (time 1746131653.52 total 311.29 last 102.39) - open/close 36349 (time 1746131663.52 total 321.30 last 107.44) - open/close 37566 (time 1746131673.54 total 331.31 last 121.54) - open/close 38554 (time 1746131683.54 total 341.32 last 98.78) - open/close 39562 (time 1746131693.54 total 351.32 last 100.75) - open/close 40000 (time 1746131697.54 total 355.31 last 109.65) - open/close 41025 (time 1746131707.55 total 365.33 last 102.35) - open/close 42178 (time 1746131717.56 total 375.34 last 115.22) - open/close 43161 (time 1746131727.57 total 385.35 last 98.20) - open/close 44226 (time 1746131737.58 total 395.35 last 106.42) - open/close 45351 (time 1746131747.58 total 405.36 last 112.44) - open/close 46538 (time 1746131757.59 total 415.37 last 118.63) - open/close 47642 (time 1746131767.61 total 425.38 last 110.22) - open/close 48697 (time 1746131777.61 total 435.39 last 105.45) - open/close 49793 (time 1746131787.61 total 445.39 last 109.56) - open/close 50000 (time 1746131789.84 total 447.62 last 92.86) - open/close 51097 (time 1746131799.84 total 457.62 last 109.69) - open/close 52142 (time 1746131809.85 total 467.62 last 104.48) - open/close 53162 (time 1746131819.85 total 477.63 last 101.93) - open/close 54196 (time 1746131829.86 total 487.63 last 103.37) - open/close 55230 (time 1746131839.86 total 497.64 last 103.37) - open/close 56296 (time 1746131849.87 total 507.65 last 106.48) - open/close 57360 (time 1746131859.87 total 517.65 last 106.37) - open/close 58415 (time 1746131869.88 total 527.65 last 105.47) - open/close 59647 (time 1746131879.88 total 537.66 last 123.13) - open/close 60000 (time 1746131882.47 total 540.25 last 136.20) - open/close 61099 (time 1746131892.48 total 550.26 last 109.80) - open/close 62195 (time 1746131902.48 total 560.26 last 109.59) - open/close 63330 (time 1746131912.49 total 570.26 last 113.46) - open/close 64601 (time 1746131922.49 total 580.27 last 127.04) total: 64768 open/close in 581.60 seconds: 111.36 ops/second - /unlink 1774 (time 1746131935.84 total 10.00 last 177.35) - /unlink 3714 (time 1746131945.85 total 20.01 last 193.84) - /unlink 5623 (time 1746131955.85 total 30.01 last 190.86) - /unlink 7735 (time 1746131965.85 total 40.02 last 211.14) - /unlink 9506 (time 1746131975.85 total 50.02 last 177.04) - /unlink 10000 (time 1746131978.54 total 52.70 last 184.30) - /unlink 11861 (time 1746131988.54 total 62.70 last 186.10) - /unlink 13669 (time 1746131998.54 total 72.70 last 180.80) - /unlink 15754 (time 1746132008.54 total 82.70 last 208.43) - /unlink 17646 (time 1746132018.55 total 92.72 last 188.96) - /unlink 19361 (time 1746132028.55 total 102.72 last 171.46) - /unlink 20000 (time 1746132032.19 total 106.35 last 175.71) - /unlink 21817 (time 1746132042.20 total 116.37 last 181.52) - /unlink 23542 (time 1746132052.20 total 126.37 last 172.48) - /unlink 25384 (time 1746132062.20 total 136.37 last 184.17) - /unlink 27202 (time 1746132072.21 total 146.37 last 181.71) - /unlink 29094 (time 1746132082.22 total 156.38 last 189.06) - /unlink 30000 (time 1746132086.74 total 160.91 last 200.05) - /unlink 32098 (time 1746132096.75 total 170.91 last 209.76) - /unlink 34024 (time 1746132106.75 total 180.91 last 192.57) - /unlink 35824 (time 1746132116.75 total 190.92 last 179.94) - /unlink 37614 (time 1746132126.76 total 200.92 last 178.87) - /unlink 39553 (time 1746132136.76 total 210.92 last 193.88) - /unlink 40000 (time 1746132139.07 total 213.24 last 193.38) - /unlink 41921 (time 1746132149.08 total 223.24 last 192.02) - /unlink 43641 (time 1746132159.08 total 233.24 last 171.99) - /unlink 45480 (time 1746132169.08 total 243.24 last 183.90) - /unlink 47457 (time 1746132179.08 total 253.24 last 197.70) - /unlink 49486 (time 1746132189.09 total 263.25 last 202.67) - /unlink 50000 (time 1746132191.55 total 265.71 last 208.89) - /unlink 52169 (time 1746132201.55 total 275.72 last 216.76) - /unlink 54114 (time 1746132211.55 total 285.72 last 194.49) - /unlink 56251 (time 1746132221.56 total 295.72 last 213.68) - /unlink 58511 (time 1746132231.56 total 305.72 last 226.00) - /unlink 60000 (time 1746132238.59 total 312.76 last 211.54) - /unlink 61977 (time 1746132248.60 total 322.76 last 197.66) - /unlink 63867 (time 1746132258.61 total 332.77 last 188.81) total: 64768 /unlink in 336.54 seconds: 192.45 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg352-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg352-server unloading modules via unload_modules_local on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-client: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-client: ssh exited with exit code 2 PASS 106 (4441s) Stopping clients: oleg352-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg352-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 oleg352-server: oleg352-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions oleg352-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg352-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 4586 sec ========== 16:45:33 (1746132333) === conf-sanity: start cleanup 16:45:34 (1746132334) === === conf-sanity: finish cleanup 16:45:36 (1746132336) ===