-----============= acceptance-small: conf-sanity ============----- Thu May 1 19:16:46 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg217-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg217-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg217-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg217-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg217-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg217-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg217-server oleg217-server: oleg217-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg217-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg217-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg217-server: oleg217-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg217-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg217-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg217-server stop mds service on oleg217-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg217-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 19:18:00 (1746141480) Stopping clients: oleg217-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg217-client.virtnet /mnt/lustre2 (opts:-f) oleg217-server: oleg217-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions oleg217-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg217-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg217-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg217-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1551 (time 1746141532.72 total 10.00 last 155.05) - open/close 3333 (time 1746141542.72 total 20.01 last 178.12) - open/close 4661 (time 1746141552.73 total 30.01 last 132.78) - open/close 5998 (time 1746141562.73 total 40.02 last 133.61) - open/close 7218 (time 1746141572.74 total 50.02 last 121.92) - open/close 8434 (time 1746141582.74 total 60.02 last 121.57) - open/close 9646 (time 1746141592.74 total 70.03 last 121.18) - open/close 10000 (time 1746141595.48 total 72.77 last 129.21) - open/close 11253 (time 1746141605.48 total 82.77 last 125.29) - open/close 12734 (time 1746141615.49 total 92.77 last 148.02) - open/close 14282 (time 1746141625.50 total 102.78 last 154.69) - open/close 15607 (time 1746141635.50 total 112.78 last 132.45) - open/close 16833 (time 1746141645.50 total 122.78 last 122.59) - open/close 18079 (time 1746141655.50 total 132.79 last 124.55) - open/close 19214 (time 1746141665.51 total 142.79 last 113.43) - open/close 20000 (time 1746141671.83 total 149.12 last 124.30) - open/close 21255 (time 1746141681.84 total 159.12 last 125.45) - open/close 22530 (time 1746141691.84 total 169.12 last 127.46) - open/close 23788 (time 1746141701.85 total 179.13 last 125.75) - open/close 25067 (time 1746141711.85 total 189.13 last 127.83) - open/close 26119 (time 1746141721.86 total 199.14 last 105.09) - open/close 27425 (time 1746141731.87 total 209.15 last 130.50) - open/close 28420 (time 1746141741.87 total 219.15 last 99.48) - open/close 29559 (time 1746141751.87 total 229.15 last 113.90) - open/close 30000 (time 1746141756.07 total 233.35 last 105.08) - open/close 31180 (time 1746141766.07 total 243.36 last 117.93) - open/close 32199 (time 1746141776.07 total 253.36 last 101.90) - open/close 33262 (time 1746141786.08 total 263.36 last 106.28) - open/close 34183 (time 1746141796.08 total 273.36 last 92.05) - open/close 35148 (time 1746141806.08 total 283.37 last 96.48) - open/close 36153 (time 1746141816.09 total 293.37 last 100.43) - open/close 37110 (time 1746141826.09 total 303.38 last 95.67) - open/close 38042 (time 1746141836.10 total 313.38 last 93.15) - open/close 39057 (time 1746141846.11 total 323.39 last 101.41) - open/close 40000 (time 1746141856.13 total 333.41 last 94.08) - open/close 41001 (time 1746141866.14 total 343.42 last 100.03) - open/close 42000 (time 1746141876.14 total 353.42 last 99.87) - open/close 43125 (time 1746141886.15 total 363.43 last 112.36) - open/close 44244 (time 1746141896.16 total 373.44 last 111.87) - open/close 45370 (time 1746141906.16 total 383.44 last 112.59) - open/close 46762 (time 1746141916.16 total 393.44 last 139.19) - open/close 48021 (time 1746141926.16 total 403.44 last 125.86) - open/close 49147 (time 1746141936.17 total 413.45 last 112.54) - open/close 50000 (time 1746141943.85 total 421.13 last 111.05) - open/close 51036 (time 1746141953.85 total 431.14 last 103.53) - open/close 52049 (time 1746141963.86 total 441.14 last 101.23) - open/close 53201 (time 1746141973.86 total 451.15 last 115.17) - open/close 54216 (time 1746141983.86 total 461.15 last 101.49) - open/close 55380 (time 1746141993.87 total 471.15 last 116.34) - open/close 56518 (time 1746142003.87 total 481.16 last 113.74) - open/close 57597 (time 1746142013.88 total 491.16 last 107.86) - open/close 58653 (time 1746142023.89 total 501.17 last 105.50) - open/close 59861 (time 1746142033.89 total 511.17 last 120.79) - open/close 61163 (time 1746142043.89 total 521.18 last 130.13) - open/close 62250 (time 1746142053.90 total 531.18 last 108.63) - open/close 63581 (time 1746142063.90 total 541.18 last 133.10) total: 64768 open/close in 550.92 seconds: 117.56 ops/second - /unlink 2657 (time 1746142085.00 total 10.00 last 265.68) - /unlink 4844 (time 1746142095.00 total 20.00 last 218.68) - /unlink 6736 (time 1746142105.00 total 30.01 last 189.12) - /unlink 8536 (time 1746142115.01 total 40.01 last 179.86) - /unlink 10000 (time 1746142122.58 total 47.58 last 193.35) - /unlink 12511 (time 1746142132.59 total 57.59 last 250.95) - /unlink 15048 (time 1746142142.59 total 67.60 last 253.57) - /unlink 17403 (time 1746142152.60 total 77.60 last 235.43) - /unlink 19506 (time 1746142162.60 total 87.60 last 210.19) - /unlink 20000 (time 1746142165.06 total 90.06 last 201.26) - /unlink 22191 (time 1746142175.06 total 100.06 last 219.08) - /unlink 24089 (time 1746142185.06 total 110.06 last 189.78) - /unlink 25897 (time 1746142195.06 total 120.06 last 180.74) - /unlink 27969 (time 1746142205.07 total 130.07 last 207.05) - /unlink 30000 (time 1746142214.98 total 139.98 last 204.89) - /unlink 32021 (time 1746142224.99 total 149.99 last 202.02) - /unlink 34121 (time 1746142234.99 total 159.99 last 209.99) - /unlink 35949 (time 1746142244.99 total 169.99 last 182.71) - /unlink 37886 (time 1746142254.99 total 180.00 last 193.62) - /unlink 39813 (time 1746142265.00 total 190.00 last 192.69) - /unlink 41494 (time 1746142275.00 total 200.00 last 168.06) - /unlink 43103 (time 1746142285.00 total 210.00 last 160.88) - /unlink 44561 (time 1746142295.00 total 220.01 last 145.72) - /unlink 46056 (time 1746142305.01 total 230.02 last 149.36) - /unlink 47422 (time 1746142315.01 total 240.02 last 136.59) - /unlink 49232 (time 1746142325.02 total 250.03 last 180.82) - /unlink 50000 (time 1746142329.17 total 254.17 last 185.45) - /unlink 52050 (time 1746142339.17 total 264.17 last 204.94) - /unlink 54089 (time 1746142349.17 total 274.17 last 203.87) - /unlink 56004 (time 1746142359.17 total 284.17 last 191.48) - /unlink 58055 (time 1746142369.17 total 294.18 last 205.09) - /unlink 59929 (time 1746142379.18 total 304.18 last 187.33) - /unlink 61981 (time 1746142389.18 total 314.18 last 205.19) - /unlink 64195 (time 1746142399.18 total 324.18 last 221.40) total: 64768 /unlink in 327.21 seconds: 197.94 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1036 (time 1746142429.97 total 10.01 last 103.50) - open/close 2150 (time 1746142439.97 total 20.01 last 111.36) - open/close 3149 (time 1746142449.97 total 30.01 last 99.88) - open/close 4342 (time 1746142459.97 total 40.02 last 119.28) - open/close 5565 (time 1746142469.98 total 50.02 last 122.27) - open/close 6741 (time 1746142479.98 total 60.02 last 117.57) - open/close 8100 (time 1746142489.98 total 70.03 last 135.82) - open/close 9510 (time 1746142499.99 total 80.03 last 140.90) - open/close 10000 (time 1746142503.88 total 83.92 last 125.96) - open/close 11431 (time 1746142513.88 total 93.93 last 143.08) - open/close 12615 (time 1746142523.88 total 103.93 last 118.39) - open/close 13727 (time 1746142533.88 total 113.93 last 111.19) - open/close 14841 (time 1746142543.89 total 123.93 last 111.38) - open/close 15883 (time 1746142553.89 total 133.93 last 104.16) - open/close 17273 (time 1746142563.89 total 143.93 last 138.99) - open/close 18522 (time 1746142573.90 total 153.94 last 124.78) - open/close 19828 (time 1746142583.90 total 163.94 last 130.58) - open/close 21058 (time 1746142593.91 total 173.95 last 122.96) - open/close 22143 (time 1746142603.91 total 183.95 last 108.49) - open/close 23059 (time 1746142613.92 total 193.96 last 91.46) - open/close 23977 (time 1746142623.92 total 203.97 last 91.78) - open/close 24876 (time 1746142633.93 total 213.97 last 89.85) - open/close 26076 (time 1746142643.93 total 223.97 last 119.98) - open/close 27360 (time 1746142653.94 total 233.98 last 128.28) - open/close 28354 (time 1746142663.94 total 243.99 last 99.35) - open/close 29593 (time 1746142673.94 total 253.99 last 123.89) - open/close 30000 (time 1746142677.67 total 257.71 last 109.38) - open/close 31044 (time 1746142687.67 total 267.71 last 104.38) - open/close 32120 (time 1746142697.67 total 277.72 last 107.52) - open/close 33265 (time 1746142707.69 total 287.73 last 114.34) - open/close 34312 (time 1746142717.69 total 297.73 last 104.68) - open/close 35538 (time 1746142727.70 total 307.74 last 122.51) - open/close 36741 (time 1746142737.70 total 317.74 last 120.26) - open/close 37782 (time 1746142747.71 total 327.75 last 104.00) - open/close 38756 (time 1746142757.72 total 337.76 last 97.31) - open/close 39715 (time 1746142767.73 total 347.77 last 95.82) - open/close 40000 (time 1746142770.67 total 350.72 last 96.75) - open/close 41026 (time 1746142780.68 total 360.72 last 102.55) - open/close 42088 (time 1746142790.68 total 370.73 last 106.16) - open/close 43100 (time 1746142800.68 total 380.73 last 101.19) - open/close 44136 (time 1746142810.69 total 390.73 last 103.58) - open/close 45287 (time 1746142820.69 total 400.73 last 115.10) - open/close 46448 (time 1746142830.69 total 410.74 last 116.01) - open/close 47447 (time 1746142840.70 total 420.74 last 99.87) - open/close 48692 (time 1746142850.70 total 430.74 last 124.43) - open/close 49872 (time 1746142860.70 total 440.74 last 118.00) - open/close 51010 (time 1746142870.70 total 450.75 last 113.76) - open/close 52110 (time 1746142880.71 total 460.75 last 109.97) - open/close 53284 (time 1746142890.72 total 470.76 last 117.31) - open/close 54465 (time 1746142900.72 total 480.76 last 118.05) - open/close 55646 (time 1746142910.72 total 490.77 last 118.06) - open/close 56776 (time 1746142920.73 total 500.77 last 112.93) - open/close 57687 (time 1746142930.74 total 510.78 last 91.01) - open/close 58629 (time 1746142940.74 total 520.78 last 94.20) - open/close 59715 (time 1746142950.74 total 530.78 last 108.59) - open/close 60000 (time 1746142953.35 total 533.39 last 109.23) - open/close 61061 (time 1746142963.36 total 543.40 last 106.03) - open/close 62173 (time 1746142973.36 total 553.40 last 111.14) - open/close 63202 (time 1746142983.36 total 563.41 last 102.87) - open/close 64189 (time 1746142993.37 total 573.41 last 98.68) total: 64768 open/close in 579.97 seconds: 111.68 ops/second - /unlink 2258 (time 1746143011.72 total 10.00 last 225.75) - /unlink 4015 (time 1746143021.72 total 20.01 last 175.62) - /unlink 5611 (time 1746143031.73 total 30.01 last 159.51) - /unlink 7409 (time 1746143041.73 total 40.01 last 179.75) - /unlink 9118 (time 1746143051.73 total 50.02 last 170.88) - /unlink 10000 (time 1746143057.26 total 55.55 last 159.40) - /unlink 11605 (time 1746143067.26 total 65.55 last 160.48) - /unlink 13216 (time 1746143077.27 total 75.55 last 161.06) - /unlink 14603 (time 1746143087.27 total 85.55 last 138.69) - /unlink 16202 (time 1746143097.27 total 95.56 last 159.87) - /unlink 17855 (time 1746143107.28 total 105.56 last 165.18) - /unlink 19554 (time 1746143117.28 total 115.57 last 169.85) - /unlink 20000 (time 1746143120.05 total 118.34 last 160.78) - /unlink 21605 (time 1746143130.06 total 128.35 last 160.39) - /unlink 23263 (time 1746143140.06 total 138.35 last 165.75) - /unlink 24821 (time 1746143150.07 total 148.36 last 155.64) - /unlink 26363 (time 1746143160.08 total 158.37 last 154.08) - /unlink 27840 (time 1746143170.09 total 168.37 last 147.62) - /unlink 29321 (time 1746143180.13 total 178.42 last 147.44) - /unlink 30000 (time 1746143184.23 total 182.52 last 165.55) - /unlink 31875 (time 1746143194.24 total 192.52 last 187.40) - /unlink 33435 (time 1746143204.24 total 202.53 last 155.96) - /unlink 35077 (time 1746143214.24 total 212.53 last 164.16) - /unlink 36772 (time 1746143224.25 total 222.53 last 169.47) - /unlink 38621 (time 1746143234.25 total 232.53 last 184.84) - /unlink 40000 (time 1746143242.25 total 240.53 last 172.43) - /unlink 41563 (time 1746143252.25 total 250.54 last 156.23) - /unlink 43327 (time 1746143262.25 total 260.54 last 176.39) - /unlink 45067 (time 1746143272.26 total 270.55 last 173.86) - /unlink 46693 (time 1746143282.27 total 280.55 last 162.50) - /unlink 48422 (time 1746143292.27 total 290.56 last 172.83) - /unlink 50000 (time 1746143301.04 total 299.32 last 180.01) - /unlink 51496 (time 1746143311.04 total 309.33 last 149.54) - /unlink 53260 (time 1746143321.04 total 319.33 last 176.31) - /unlink 54815 (time 1746143331.05 total 329.34 last 155.41) - /unlink 56461 (time 1746143341.06 total 339.34 last 164.50) - /unlink 57998 (time 1746143351.06 total 349.35 last 153.63) - /unlink 59681 (time 1746143361.07 total 359.35 last 168.19) - /unlink 61406 (time 1746143371.09 total 369.37 last 172.14) - /unlink 63116 (time 1746143381.09 total 379.38 last 170.92) total: 64768 /unlink in 388.55 seconds: 166.69 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 903 (time 1746143421.54 total 10.01 last 90.24) - open/close 1920 (time 1746143431.56 total 20.02 last 101.55) - open/close 2994 (time 1746143441.56 total 30.03 last 107.36) - open/close 4131 (time 1746143451.57 total 40.03 last 113.68) - open/close 5204 (time 1746143461.57 total 50.04 last 107.22) - open/close 6295 (time 1746143471.58 total 60.04 last 109.04) - open/close 7440 (time 1746143481.58 total 70.05 last 114.45) - open/close 8515 (time 1746143491.58 total 80.05 last 107.50) - open/close 9598 (time 1746143501.59 total 90.05 last 108.26) - open/close 10000 (time 1746143505.39 total 93.85 last 105.78) - open/close 11060 (time 1746143515.39 total 103.85 last 105.96) - open/close 12140 (time 1746143525.39 total 113.86 last 107.96) - open/close 13291 (time 1746143535.39 total 123.86 last 115.09) - open/close 14393 (time 1746143545.40 total 133.86 last 110.16) - open/close 15406 (time 1746143555.41 total 143.87 last 101.22) - open/close 16328 (time 1746143565.41 total 153.87 last 92.20) - open/close 17312 (time 1746143575.41 total 163.87 last 98.34) - open/close 18383 (time 1746143585.42 total 173.88 last 107.05) - open/close 19407 (time 1746143595.42 total 183.88 last 102.39) - open/close 20000 (time 1746143601.32 total 189.78 last 100.47) - open/close 21079 (time 1746143611.33 total 199.79 last 107.83) - open/close 22141 (time 1746143621.33 total 209.80 last 106.13) - open/close 23126 (time 1746143631.33 total 219.80 last 98.50) - open/close 24248 (time 1746143641.33 total 229.80 last 112.19) - open/close 25290 (time 1746143651.34 total 239.80 last 104.14) - open/close 26338 (time 1746143661.35 total 249.82 last 104.65) - open/close 27373 (time 1746143671.36 total 259.82 last 103.43) - open/close 28396 (time 1746143681.36 total 269.82 last 102.28) - open/close 29356 (time 1746143691.37 total 279.83 last 95.94) - open/close 30000 (time 1746143698.10 total 286.56 last 95.64) - open/close 31086 (time 1746143708.11 total 296.57 last 108.56) - open/close 32240 (time 1746143718.11 total 306.57 last 115.40) - open/close 33285 (time 1746143728.12 total 316.58 last 104.39) - open/close 34318 (time 1746143738.12 total 326.58 last 103.29) - open/close 35403 (time 1746143748.12 total 336.58 last 108.50) - open/close 36495 (time 1746143758.13 total 346.59 last 109.09) - open/close 37716 (time 1746143768.13 total 356.59 last 122.07) - open/close 38949 (time 1746143778.14 total 366.60 last 123.16) - open/close 40000 (time 1746143787.05 total 375.52 last 117.94) - open/close 41181 (time 1746143797.05 total 385.52 last 118.09) - open/close 42492 (time 1746143807.06 total 395.52 last 131.06) - open/close 43825 (time 1746143817.06 total 405.53 last 133.21) - open/close 45165 (time 1746143827.07 total 415.53 last 133.96) - open/close 46323 (time 1746143837.07 total 425.53 last 115.76) - open/close 47463 (time 1746143847.07 total 435.54 last 113.96) - open/close 48582 (time 1746143857.08 total 445.54 last 111.89) - open/close 49734 (time 1746143867.09 total 455.55 last 115.03) - open/close 50000 (time 1746143869.67 total 458.13 last 103.09) - open/close 51257 (time 1746143879.68 total 468.14 last 125.63) - open/close 52652 (time 1746143889.68 total 478.14 last 139.49) - open/close 53958 (time 1746143899.68 total 488.14 last 130.59) - open/close 55375 (time 1746143909.68 total 498.15 last 141.62) - open/close 56538 (time 1746143919.68 total 508.15 last 116.30) - open/close 57646 (time 1746143929.69 total 518.15 last 110.75) - open/close 58805 (time 1746143939.69 total 528.16 last 115.83) - open/close 59980 (time 1746143949.70 total 538.16 last 117.47) - open/close 61090 (time 1746143959.70 total 548.16 last 110.98) - open/close 62391 (time 1746143969.70 total 558.16 last 130.06) - open/close 63481 (time 1746143979.70 total 568.17 last 108.97) total: 64768 open/close in 577.90 seconds: 112.07 ops/second - /unlink 2202 (time 1746144001.16 total 10.01 last 220.01) - /unlink 4368 (time 1746144011.18 total 20.02 last 216.26) - /unlink 6155 (time 1746144021.18 total 30.03 last 178.61) - /unlink 8008 (time 1746144031.19 total 40.04 last 185.16) - /unlink 9803 (time 1746144041.19 total 50.04 last 179.38) - /unlink 11647 (time 1746144051.20 total 60.05 last 184.36) - /unlink 13325 (time 1746144061.20 total 70.05 last 167.76) - /unlink 14821 (time 1746144071.21 total 80.06 last 149.49) - /unlink 16482 (time 1746144081.21 total 90.06 last 166.03) - /unlink 18237 (time 1746144091.21 total 100.06 last 175.49) - /unlink 19918 (time 1746144101.23 total 110.08 last 167.79) - /unlink 21823 (time 1746144111.26 total 120.11 last 189.86) - /unlink 23821 (time 1746144121.27 total 130.12 last 199.74) - /unlink 25639 (time 1746144131.27 total 140.12 last 181.70) - /unlink 27529 (time 1746144141.28 total 150.13 last 188.82) - /unlink 29599 (time 1746144151.28 total 160.13 last 206.96) - /unlink 31698 (time 1746144161.29 total 170.14 last 209.82) - /unlink 33548 (time 1746144171.29 total 180.14 last 184.94) - /unlink 35480 (time 1746144181.29 total 190.14 last 193.16) - /unlink 37458 (time 1746144191.29 total 200.14 last 197.78) - /unlink 39589 (time 1746144201.30 total 210.14 last 213.05) - /unlink 41465 (time 1746144211.30 total 220.15 last 187.58) - /unlink 43367 (time 1746144221.30 total 230.15 last 190.19) - /unlink 45236 (time 1746144231.30 total 240.15 last 186.89) - /unlink 47282 (time 1746144241.30 total 250.15 last 204.58) - /unlink 49654 (time 1746144251.30 total 260.15 last 237.19) - /unlink 51682 (time 1746144261.30 total 270.15 last 202.76) - /unlink 53620 (time 1746144271.30 total 280.15 last 193.73) - /unlink 55762 (time 1746144281.31 total 290.16 last 214.16) - /unlink 57861 (time 1746144291.31 total 300.16 last 209.81) - /unlink 59725 (time 1746144301.31 total 310.16 last 186.34) - /unlink 61490 (time 1746144311.32 total 320.17 last 176.45) - /unlink 63320 (time 1746144321.32 total 330.17 last 182.95) total: 64768 /unlink in 336.06 seconds: 192.73 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1395 (time 1746144357.78 total 10.01 last 139.41) - open/close 2762 (time 1746144367.78 total 20.01 last 136.70) - open/close 3989 (time 1746144377.79 total 30.01 last 122.63) - open/close 5158 (time 1746144387.80 total 40.02 last 116.83) - open/close 6370 (time 1746144397.80 total 50.02 last 121.17) - open/close 7730 (time 1746144407.80 total 60.03 last 135.92) - open/close 9073 (time 1746144417.81 total 70.04 last 134.17) - open/close 10000 (time 1746144424.94 total 77.16 last 130.15) - open/close 11251 (time 1746144434.94 total 87.16 last 125.05) - open/close 12368 (time 1746144444.94 total 97.17 last 111.66) - open/close 13593 (time 1746144454.95 total 107.17 last 122.43) - open/close 14966 (time 1746144464.95 total 117.18 last 137.26) - open/close 16511 (time 1746144474.95 total 127.18 last 154.50) - open/close 18243 (time 1746144484.96 total 137.18 last 173.16) - open/close 20000 (time 1746144492.74 total 144.97 last 225.59) - open/close 22254 (time 1746144502.75 total 154.97 last 225.27) - open/close 24156 (time 1746144512.75 total 164.97 last 190.16) - open/close 25777 (time 1746144522.76 total 174.98 last 162.00) - open/close 27280 (time 1746144532.76 total 184.98 last 150.29) - open/close 28997 (time 1746144542.77 total 194.99 last 171.57) - open/close 30000 (time 1746144549.06 total 201.29 last 159.30) - open/close 31501 (time 1746144559.06 total 211.29 last 150.09) - open/close 33178 (time 1746144569.07 total 221.29 last 167.61) - open/close 34607 (time 1746144579.07 total 231.29 last 142.87) - open/close 36059 (time 1746144589.08 total 241.30 last 145.14) - open/close 37448 (time 1746144599.08 total 251.30 last 138.87) - open/close 38997 (time 1746144609.08 total 261.30 last 154.88) - open/close 40000 (time 1746144616.22 total 268.44 last 140.42) - open/close 41633 (time 1746144626.22 total 278.45 last 163.27) - open/close 43210 (time 1746144636.23 total 288.45 last 157.64) - open/close 44877 (time 1746144646.23 total 298.46 last 166.59) - open/close 46956 (time 1746144656.24 total 308.46 last 207.79) - open/close 48333 (time 1746144666.24 total 318.46 last 137.66) - open/close 49688 (time 1746144676.25 total 328.47 last 135.38) - open/close 50000 (time 1746144678.44 total 330.66 last 142.36) - open/close 51452 (time 1746144688.45 total 340.67 last 145.14) - open/close 52950 (time 1746144698.46 total 350.68 last 149.67) - open/close 54304 (time 1746144708.46 total 360.68 last 135.40) - open/close 55675 (time 1746144718.46 total 370.68 last 137.02) - open/close 57196 (time 1746144728.47 total 380.69 last 151.98) - open/close 58818 (time 1746144738.47 total 390.70 last 162.15) - open/close 60000 (time 1746144745.25 total 397.47 last 174.53) - open/close 61506 (time 1746144755.25 total 407.47 last 150.54) - open/close 62768 (time 1746144765.25 total 417.48 last 126.14) - open/close 64279 (time 1746144775.26 total 427.48 last 150.98) total: 64768 open/close in 430.83 seconds: 150.33 ops/second - /unlink 2661 (time 1746144790.12 total 10.00 last 266.00) - /unlink 5247 (time 1746144800.12 total 20.01 last 258.55) - /unlink 7397 (time 1746144810.12 total 30.01 last 214.92) - /unlink 9483 (time 1746144820.13 total 40.01 last 208.49) - /unlink 10000 (time 1746144822.33 total 42.22 last 234.58) - /unlink 12142 (time 1746144832.33 total 52.22 last 214.13) - /unlink 14725 (time 1746144842.34 total 62.22 last 258.29) - /unlink 17173 (time 1746144852.34 total 72.22 last 244.75) - /unlink 19338 (time 1746144862.34 total 82.23 last 216.38) - /unlink 20000 (time 1746144866.36 total 86.25 last 164.78) - /unlink 21946 (time 1746144876.36 total 96.25 last 194.56) - /unlink 24118 (time 1746144886.36 total 106.25 last 217.17) - /unlink 26114 (time 1746144896.37 total 116.25 last 199.55) - /unlink 28539 (time 1746144906.37 total 126.25 last 242.45) - /unlink 30000 (time 1746144912.38 total 132.27 last 243.03) - /unlink 32047 (time 1746144922.38 total 142.27 last 204.70) - /unlink 34142 (time 1746144932.38 total 152.27 last 209.47) - /unlink 36340 (time 1746144942.38 total 162.27 last 219.78) - /unlink 38716 (time 1746144952.38 total 172.27 last 237.60) - /unlink 40000 (time 1746144957.56 total 177.45 last 247.97) - /unlink 42228 (time 1746144967.56 total 187.45 last 222.79) - /unlink 44372 (time 1746144977.56 total 197.45 last 214.35) - /unlink 46560 (time 1746144987.56 total 207.45 last 218.75) - /unlink 48777 (time 1746144997.57 total 217.45 last 221.67) - /unlink 50000 (time 1746145003.74 total 223.62 last 198.19) - /unlink 52374 (time 1746145013.74 total 233.63 last 237.31) - /unlink 54659 (time 1746145023.74 total 243.63 last 228.43) - /unlink 56866 (time 1746145033.75 total 253.64 last 220.54) - /unlink 58969 (time 1746145043.75 total 263.64 last 210.28) - /unlink 60000 (time 1746145048.45 total 268.34 last 219.34) - /unlink 62155 (time 1746145058.46 total 278.34 last 215.43) - /unlink 64498 (time 1746145068.46 total 288.35 last 234.21) total: 64768 /unlink in 289.57 seconds: 223.67 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1572 (time 1746145098.79 total 10.01 last 157.11) - open/close 3069 (time 1746145108.79 total 20.01 last 149.68) - open/close 4434 (time 1746145118.79 total 30.01 last 136.40) - open/close 5955 (time 1746145128.80 total 40.02 last 152.09) - open/close 7535 (time 1746145138.80 total 50.02 last 157.95) - open/close 9036 (time 1746145148.80 total 60.02 last 150.07) - open/close 10000 (time 1746145154.65 total 65.87 last 164.84) - open/close 11491 (time 1746145164.66 total 75.88 last 148.86) - open/close 12856 (time 1746145174.67 total 85.89 last 136.44) - open/close 14399 (time 1746145184.67 total 95.89 last 154.30) - open/close 16289 (time 1746145194.67 total 105.89 last 188.98) - open/close 18175 (time 1746145204.67 total 115.89 last 188.60) - open/close 19859 (time 1746145214.67 total 125.89 last 168.37) - open/close 21235 (time 1746145224.68 total 135.90 last 137.55) - open/close 22813 (time 1746145234.68 total 145.90 last 157.67) - open/close 24628 (time 1746145244.69 total 155.91 last 181.44) - open/close 26047 (time 1746145254.69 total 165.91 last 141.85) - open/close 27552 (time 1746145264.69 total 175.91 last 150.50) - open/close 29161 (time 1746145274.70 total 185.92 last 160.79) - open/close 30000 (time 1746145280.44 total 191.66 last 146.15) - open/close 31381 (time 1746145290.44 total 201.66 last 138.06) - open/close 32683 (time 1746145300.45 total 211.67 last 130.14) - open/close 34148 (time 1746145310.45 total 221.67 last 146.43) - open/close 35712 (time 1746145320.45 total 231.67 last 156.36) - open/close 37380 (time 1746145330.46 total 241.68 last 166.76) - open/close 39156 (time 1746145340.46 total 251.68 last 177.55) - open/close 40000 (time 1746145345.96 total 257.18 last 153.31) - open/close 41549 (time 1746145355.97 total 267.19 last 154.87) - open/close 43078 (time 1746145365.97 total 277.19 last 152.80) - open/close 44699 (time 1746145375.98 total 287.20 last 162.01) - open/close 46359 (time 1746145385.98 total 297.20 last 165.99) - open/close 47881 (time 1746145395.98 total 307.20 last 152.19) - open/close 49671 (time 1746145405.99 total 317.20 last 178.90) - open/close 50000 (time 1746145408.09 total 319.31 last 156.21) - open/close 51396 (time 1746145418.09 total 329.31 last 139.56) - open/close 52873 (time 1746145428.10 total 339.32 last 147.68) - open/close 54330 (time 1746145438.10 total 349.32 last 145.66) - open/close 56172 (time 1746145448.10 total 359.32 last 184.19) - open/close 57770 (time 1746145458.10 total 369.32 last 159.75) - open/close 59224 (time 1746145468.11 total 379.33 last 145.34) - open/close 60000 (time 1746145473.40 total 384.62 last 146.49) - open/close 61562 (time 1746145483.41 total 394.62 last 156.17) - open/close 63062 (time 1746145493.42 total 404.64 last 149.84) - open/close 64528 (time 1746145503.42 total 414.64 last 146.47) total: 64768 open/close in 416.35 seconds: 155.56 ops/second - /unlink 2101 (time 1746145516.75 total 10.00 last 210.08) - /unlink 4315 (time 1746145526.75 total 20.00 last 221.37) - /unlink 6319 (time 1746145536.75 total 30.00 last 200.38) - /unlink 8073 (time 1746145546.75 total 40.01 last 175.35) - /unlink 9935 (time 1746145556.75 total 50.01 last 186.17) - /unlink 12374 (time 1746145566.76 total 60.01 last 243.86) - /unlink 14422 (time 1746145576.76 total 70.01 last 204.78) - /unlink 16924 (time 1746145586.76 total 80.01 last 250.17) - /unlink 19318 (time 1746145596.76 total 90.01 last 239.33) - /unlink 20000 (time 1746145601.01 total 94.26 last 160.63) - /unlink 22016 (time 1746145611.01 total 104.26 last 201.53) - /unlink 24031 (time 1746145621.01 total 114.26 last 201.48) - /unlink 26241 (time 1746145631.01 total 124.26 last 220.98) - /unlink 28556 (time 1746145641.01 total 134.27 last 231.44) - /unlink 30000 (time 1746145647.63 total 140.88 last 218.39) - /unlink 32330 (time 1746145657.63 total 150.88 last 232.97) - /unlink 34850 (time 1746145667.63 total 160.88 last 251.91) - /unlink 37078 (time 1746145677.64 total 170.89 last 222.70) - /unlink 39129 (time 1746145687.64 total 180.89 last 205.05) - /unlink 40000 (time 1746145690.84 total 184.09 last 272.38) - /unlink 42166 (time 1746145700.84 total 194.09 last 216.55) - /unlink 44602 (time 1746145710.84 total 204.09 last 243.51) - /unlink 47158 (time 1746145720.84 total 214.10 last 255.56) - /unlink 49587 (time 1746145730.84 total 224.10 last 242.89) - /unlink 51978 (time 1746145740.84 total 234.10 last 239.09) - /unlink 54395 (time 1746145750.84 total 244.10 last 241.69) - /unlink 56629 (time 1746145760.84 total 254.10 last 223.39) - /unlink 58686 (time 1746145770.85 total 264.10 last 205.64) - /unlink 60000 (time 1746145777.67 total 270.92 last 192.65) - /unlink 62340 (time 1746145787.67 total 280.92 last 233.92) - /unlink 64678 (time 1746145797.67 total 290.93 last 233.75) total: 64768 /unlink in 291.39 seconds: 222.27 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg217-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg217-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg217-server stop mds service on oleg217-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg217-server unloading modules via unload_modules_local on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg217-client: oleg217-client: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-client: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2 PASS 106 (4348s) Stopping clients: oleg217-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg217-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2 oleg217-server: oleg217-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions oleg217-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg217-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4449 sec ========== 20:30:56 (1746145856) === conf-sanity: start cleanup 20:30:57 (1746145857) === === conf-sanity: finish cleanup 20:30:58 (1746145858) ===