-----============= acceptance-small: conf-sanity ============----- Thu May 1 18:19:07 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg419-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg419-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg419-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg419-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg419-server oleg419-server: oleg419-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: detected 4 online CPUs by sysfs oleg419-server: MODOPTS_LIBCFS= oleg419-server: Force libcfs to create 2 CPU partitions oleg419-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg419-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg419-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg419-server: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg419-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:20:16 (1746138016) Stopping clients: oleg419-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg419-client.virtnet /mnt/lustre2 (opts:-f) oleg419-server: oleg419-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: detected 4 online CPUs by sysfs oleg419-server: MODOPTS_LIBCFS= oleg419-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg419-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg419-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1455 (time 1746138064.31 total 10.00 last 145.46) - open/close 2696 (time 1746138074.32 total 20.01 last 124.01) - open/close 4281 (time 1746138084.32 total 30.01 last 158.46) - open/close 5833 (time 1746138094.32 total 40.02 last 155.14) - open/close 7268 (time 1746138104.33 total 50.02 last 143.47) - open/close 8893 (time 1746138114.33 total 60.02 last 162.48) - open/close 10000 (time 1746138122.01 total 67.70 last 144.10) - open/close 11734 (time 1746138132.01 total 77.71 last 173.32) - open/close 13598 (time 1746138142.02 total 87.71 last 186.33) - open/close 14996 (time 1746138152.02 total 97.71 last 139.76) - open/close 16355 (time 1746138162.02 total 107.72 last 135.85) - open/close 17702 (time 1746138172.03 total 117.72 last 134.63) - open/close 19007 (time 1746138182.03 total 127.73 last 130.43) - open/close 20000 (time 1746138189.63 total 135.32 last 130.74) - open/close 21237 (time 1746138199.63 total 145.33 last 123.66) - open/close 22614 (time 1746138209.64 total 155.33 last 137.66) - open/close 24147 (time 1746138219.65 total 165.34 last 153.16) - open/close 25735 (time 1746138229.65 total 175.35 last 158.65) - open/close 27778 (time 1746138239.66 total 185.36 last 204.13) - open/close 29669 (time 1746138249.67 total 195.37 last 188.90) - open/close 30000 (time 1746138251.75 total 197.45 last 159.09) - open/close 31561 (time 1746138261.76 total 207.45 last 156.03) - open/close 33268 (time 1746138271.76 total 217.45 last 170.70) - open/close 34779 (time 1746138281.76 total 227.46 last 151.03) - open/close 36292 (time 1746138291.77 total 237.46 last 151.24) - open/close 37459 (time 1746138301.77 total 247.46 last 116.66) - open/close 38668 (time 1746138311.78 total 257.47 last 120.83) - open/close 39875 (time 1746138321.78 total 267.47 last 120.64) - open/close 41082 (time 1746138331.78 total 277.48 last 120.68) - open/close 42207 (time 1746138341.79 total 287.48 last 112.48) - open/close 43607 (time 1746138351.79 total 297.48 last 139.99) - open/close 45130 (time 1746138361.79 total 307.48 last 152.22) - open/close 46561 (time 1746138371.79 total 317.49 last 143.08) - open/close 48011 (time 1746138381.79 total 327.49 last 144.99) - open/close 49539 (time 1746138391.79 total 337.49 last 152.79) - open/close 50000 (time 1746138394.23 total 339.92 last 189.25) - open/close 51712 (time 1746138404.23 total 349.93 last 171.13) - open/close 53725 (time 1746138414.24 total 359.93 last 201.23) - open/close 55462 (time 1746138424.24 total 369.93 last 173.63) - open/close 57196 (time 1746138434.25 total 379.94 last 173.34) - open/close 59050 (time 1746138444.25 total 389.94 last 185.40) - open/close 60000 (time 1746138449.20 total 394.89 last 191.88) - open/close 61548 (time 1746138459.20 total 404.89 last 154.70) - open/close 63165 (time 1746138469.20 total 414.90 last 161.66) total: 64768 open/close in 423.59 seconds: 152.90 ops/second - /unlink 2268 (time 1746138489.31 total 10.01 last 226.66) - /unlink 4853 (time 1746138499.31 total 20.01 last 258.42) - /unlink 7215 (time 1746138509.31 total 30.01 last 236.20) - /unlink 9620 (time 1746138519.32 total 40.01 last 240.44) - /unlink 11987 (time 1746138529.32 total 50.02 last 236.62) - /unlink 14312 (time 1746138539.32 total 60.02 last 232.44) - /unlink 16948 (time 1746138549.32 total 70.02 last 263.60) - /unlink 19096 (time 1746138559.32 total 80.02 last 214.79) - /unlink 20000 (time 1746138563.08 total 83.77 last 240.69) - /unlink 22928 (time 1746138573.08 total 93.78 last 292.74) - /unlink 25176 (time 1746138583.08 total 103.78 last 224.71) - /unlink 27573 (time 1746138593.09 total 113.79 last 239.52) - /unlink 29619 (time 1746138603.09 total 123.79 last 204.57) - /unlink 32112 (time 1746138613.10 total 133.79 last 249.18) - /unlink 34038 (time 1746138623.11 total 143.80 last 192.44) - /unlink 36914 (time 1746138633.11 total 153.80 last 287.56) - /unlink 39358 (time 1746138643.11 total 163.81 last 244.21) - /unlink 40000 (time 1746138645.29 total 165.99 last 294.94) - /unlink 43424 (time 1746138655.29 total 175.99 last 342.37) - /unlink 46062 (time 1746138665.29 total 185.99 last 263.80) - /unlink 48743 (time 1746138675.30 total 195.99 last 267.99) - /unlink 50000 (time 1746138679.88 total 200.58 last 273.97) - /unlink 52390 (time 1746138689.89 total 210.58 last 238.95) - /unlink 54437 (time 1746138699.89 total 220.59 last 204.56) - /unlink 56750 (time 1746138709.89 total 230.59 last 231.27) - /unlink 58570 (time 1746138719.90 total 240.60 last 181.90) - /unlink 60000 (time 1746138727.25 total 247.95 last 194.50) - /unlink 61999 (time 1746138737.26 total 257.96 last 199.76) - /unlink 64655 (time 1746138747.27 total 267.96 last 265.39) total: 64768 /unlink in 268.49 seconds: 241.23 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1290 (time 1746138776.31 total 10.00 last 128.94) - open/close 2818 (time 1746138786.31 total 20.01 last 152.77) - open/close 4164 (time 1746138796.31 total 30.01 last 134.55) - open/close 5327 (time 1746138806.32 total 40.01 last 116.25) - open/close 6648 (time 1746138816.32 total 50.02 last 132.03) - open/close 7864 (time 1746138826.32 total 60.02 last 121.60) - open/close 9095 (time 1746138836.33 total 70.02 last 123.07) - open/close 10000 (time 1746138842.68 total 76.38 last 142.42) - open/close 11354 (time 1746138852.69 total 86.38 last 135.33) - open/close 12516 (time 1746138862.69 total 96.39 last 116.15) - open/close 13760 (time 1746138872.70 total 106.40 last 124.25) - open/close 14930 (time 1746138882.71 total 116.40 last 116.96) - open/close 16176 (time 1746138892.72 total 126.41 last 124.48) - open/close 17319 (time 1746138902.72 total 136.41 last 114.28) - open/close 18507 (time 1746138912.73 total 146.42 last 118.67) - open/close 19594 (time 1746138922.74 total 156.43 last 108.59) - open/close 20000 (time 1746138926.90 total 160.60 last 97.55) - open/close 21236 (time 1746138936.91 total 170.61 last 123.48) - open/close 22532 (time 1746138946.91 total 180.61 last 129.59) - open/close 23825 (time 1746138956.91 total 190.61 last 129.28) - open/close 25230 (time 1746138966.91 total 200.61 last 140.48) - open/close 26362 (time 1746138976.92 total 210.61 last 113.16) - open/close 27385 (time 1746138986.92 total 220.62 last 102.29) - open/close 28448 (time 1746138996.93 total 230.62 last 106.22) - open/close 29640 (time 1746139006.93 total 240.62 last 119.19) - open/close 30000 (time 1746139010.00 total 243.69 last 117.37) - open/close 31209 (time 1746139020.00 total 253.69 last 120.87) - open/close 32517 (time 1746139030.00 total 263.69 last 130.79) - open/close 33820 (time 1746139040.00 total 273.70 last 130.27) - open/close 35072 (time 1746139050.00 total 283.70 last 125.17) - open/close 36234 (time 1746139060.01 total 293.71 last 116.09) - open/close 37343 (time 1746139070.02 total 303.72 last 110.82) - open/close 38496 (time 1746139080.02 total 313.72 last 115.28) - open/close 39779 (time 1746139090.02 total 323.72 last 128.29) - open/close 41250 (time 1746139100.03 total 333.72 last 147.05) - open/close 42611 (time 1746139110.03 total 343.73 last 136.04) - open/close 44237 (time 1746139120.03 total 353.73 last 162.55) - open/close 46183 (time 1746139130.04 total 363.73 last 194.56) - open/close 47469 (time 1746139140.04 total 373.74 last 128.55) - open/close 48684 (time 1746139150.05 total 383.74 last 121.42) - open/close 50000 (time 1746139159.32 total 393.02 last 141.83) - open/close 51551 (time 1746139169.33 total 403.02 last 155.10) - open/close 52764 (time 1746139179.33 total 413.03 last 121.25) - open/close 53950 (time 1746139189.33 total 423.03 last 118.58) - open/close 55253 (time 1746139199.33 total 433.03 last 130.26) - open/close 56555 (time 1746139209.34 total 443.03 last 130.19) - open/close 57691 (time 1746139219.34 total 453.03 last 113.56) - open/close 59006 (time 1746139229.34 total 463.04 last 131.43) - open/close 60000 (time 1746139236.86 total 470.55 last 132.29) - open/close 61294 (time 1746139246.86 total 480.56 last 129.39) - open/close 62620 (time 1746139256.86 total 490.56 last 132.59) - open/close 63874 (time 1746139266.86 total 500.56 last 125.35) total: 64768 open/close in 506.33 seconds: 127.92 ops/second - /unlink 2676 (time 1746139284.14 total 10.00 last 267.58) - /unlink 4993 (time 1746139294.14 total 20.00 last 231.65) - /unlink 6747 (time 1746139304.14 total 30.00 last 175.40) - /unlink 8594 (time 1746139314.15 total 40.01 last 184.59) - /unlink 10000 (time 1746139321.35 total 47.22 last 195.04) - /unlink 12467 (time 1746139331.36 total 57.22 last 246.68) - /unlink 14793 (time 1746139341.36 total 67.22 last 232.57) - /unlink 16830 (time 1746139351.36 total 77.22 last 203.66) - /unlink 18724 (time 1746139361.37 total 87.23 last 189.27) - /unlink 20000 (time 1746139368.40 total 94.26 last 181.38) - /unlink 21786 (time 1746139378.41 total 104.27 last 178.47) - /unlink 23679 (time 1746139388.41 total 114.27 last 189.25) - /unlink 25806 (time 1746139398.41 total 124.28 last 212.64) - /unlink 28148 (time 1746139408.42 total 134.28 last 234.13) - /unlink 30000 (time 1746139416.30 total 142.17 last 234.84) - /unlink 32189 (time 1746139426.30 total 152.17 last 218.88) - /unlink 34089 (time 1746139436.31 total 162.17 last 189.98) - /unlink 36012 (time 1746139446.31 total 172.17 last 192.25) - /unlink 37757 (time 1746139456.33 total 182.19 last 174.17) - /unlink 39534 (time 1746139466.33 total 192.19 last 177.64) - /unlink 40000 (time 1746139468.63 total 194.49 last 202.55) - /unlink 41995 (time 1746139478.64 total 204.50 last 199.37) - /unlink 43540 (time 1746139488.64 total 214.50 last 154.49) - /unlink 45297 (time 1746139498.64 total 224.50 last 175.67) - /unlink 47072 (time 1746139508.64 total 234.50 last 177.47) - /unlink 49068 (time 1746139518.64 total 244.51 last 199.59) - /unlink 50000 (time 1746139522.60 total 248.46 last 235.60) - /unlink 51982 (time 1746139532.61 total 258.47 last 197.96) - /unlink 54013 (time 1746139542.61 total 268.47 last 203.09) - /unlink 56229 (time 1746139552.61 total 278.47 last 221.58) - /unlink 59190 (time 1746139562.62 total 288.48 last 295.99) - /unlink 60000 (time 1746139566.00 total 291.86 last 239.43) - /unlink 62425 (time 1746139576.00 total 301.86 last 242.44) total: 64768 /unlink in 309.67 seconds: 209.15 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1157 (time 1746139613.93 total 10.01 last 115.61) - open/close 2567 (time 1746139623.94 total 20.02 last 140.87) - open/close 3924 (time 1746139633.95 total 30.02 last 135.61) - open/close 5104 (time 1746139643.95 total 40.02 last 118.00) - open/close 6423 (time 1746139653.95 total 50.02 last 131.88) - open/close 7797 (time 1746139663.95 total 60.03 last 137.35) - open/close 9085 (time 1746139673.96 total 70.03 last 128.75) - open/close 10000 (time 1746139680.81 total 76.89 last 133.48) - open/close 11312 (time 1746139690.81 total 86.89 last 131.19) - open/close 12557 (time 1746139700.82 total 96.89 last 124.46) - open/close 13820 (time 1746139710.82 total 106.89 last 126.29) - open/close 14994 (time 1746139720.83 total 116.90 last 117.30) - open/close 16135 (time 1746139730.83 total 126.90 last 114.05) - open/close 17250 (time 1746139740.84 total 136.91 last 111.42) - open/close 18438 (time 1746139750.84 total 146.92 last 118.73) - open/close 19660 (time 1746139760.85 total 156.92 last 122.17) - open/close 20000 (time 1746139763.80 total 159.87 last 115.24) - open/close 21255 (time 1746139773.80 total 169.88 last 125.41) - open/close 22474 (time 1746139783.81 total 179.88 last 121.82) - open/close 23833 (time 1746139793.82 total 189.89 last 135.80) - open/close 24944 (time 1746139803.82 total 199.90 last 111.03) - open/close 26173 (time 1746139813.83 total 209.90 last 122.89) - open/close 27757 (time 1746139823.83 total 219.91 last 158.27) - open/close 29252 (time 1746139833.84 total 229.91 last 149.43) - open/close 30000 (time 1746139839.49 total 235.56 last 132.39) - open/close 31282 (time 1746139849.49 total 245.56 last 128.19) - open/close 32730 (time 1746139859.50 total 255.57 last 144.70) - open/close 34087 (time 1746139869.51 total 265.58 last 135.55) - open/close 35445 (time 1746139879.51 total 275.58 last 135.79) - open/close 36942 (time 1746139889.51 total 285.58 last 149.69) - open/close 38326 (time 1746139899.51 total 295.59 last 138.34) - open/close 39774 (time 1746139909.51 total 305.59 last 144.79) - open/close 40975 (time 1746139919.51 total 315.59 last 120.08) - open/close 42209 (time 1746139929.52 total 325.59 last 123.36) - open/close 43567 (time 1746139939.52 total 335.60 last 135.72) - open/close 44687 (time 1746139949.53 total 345.60 last 111.99) - open/close 45748 (time 1746139959.54 total 355.61 last 105.99) - open/close 46723 (time 1746139969.54 total 365.61 last 97.47) - open/close 47747 (time 1746139979.55 total 375.62 last 102.32) - open/close 48764 (time 1746139989.55 total 385.63 last 101.64) - open/close 49916 (time 1746139999.56 total 395.63 last 115.14) - open/close 51053 (time 1746140009.56 total 405.63 last 113.65) - open/close 52220 (time 1746140019.57 total 415.64 last 116.60) - open/close 53239 (time 1746140029.57 total 425.64 last 101.89) - open/close 54303 (time 1746140039.57 total 435.65 last 106.36) - open/close 55305 (time 1746140049.58 total 445.65 last 100.13) - open/close 56448 (time 1746140059.59 total 455.66 last 114.25) - open/close 57439 (time 1746140069.59 total 465.66 last 99.05) - open/close 58431 (time 1746140079.63 total 475.70 last 98.83) - open/close 59521 (time 1746140089.64 total 485.71 last 108.93) - open/close 60000 (time 1746140093.70 total 489.77 last 117.82) - open/close 61088 (time 1746140103.71 total 499.78 last 108.73) - open/close 62170 (time 1746140113.71 total 509.79 last 108.13) - open/close 63224 (time 1746140123.72 total 519.79 last 105.37) - open/close 64285 (time 1746140133.72 total 529.79 last 106.09) total: 64768 open/close in 535.11 seconds: 121.04 ops/second - /unlink 1621 (time 1746140151.14 total 10.01 last 161.96) - /unlink 3150 (time 1746140161.15 total 20.02 last 152.80) - /unlink 4692 (time 1746140171.16 total 30.03 last 153.95) - /unlink 6305 (time 1746140181.16 total 40.03 last 161.28) - /unlink 8059 (time 1746140191.21 total 50.08 last 174.63) - /unlink 9604 (time 1746140201.21 total 60.08 last 154.45) - /unlink 10000 (time 1746140204.04 total 62.91 last 139.91) - /unlink 11562 (time 1746140214.05 total 72.91 last 156.14) - /unlink 13166 (time 1746140224.05 total 82.91 last 160.40) - /unlink 14844 (time 1746140234.05 total 92.91 last 167.79) - /unlink 16521 (time 1746140244.05 total 102.91 last 167.70) - /unlink 18027 (time 1746140254.05 total 112.92 last 150.54) - /unlink 19621 (time 1746140264.05 total 122.92 last 159.36) - /unlink 20000 (time 1746140266.50 total 125.37 last 154.71) - /unlink 21601 (time 1746140276.50 total 135.37 last 160.09) - /unlink 23215 (time 1746140286.51 total 145.37 last 161.35) - /unlink 24851 (time 1746140296.51 total 155.37 last 163.60) - /unlink 26469 (time 1746140306.51 total 165.38 last 161.75) - /unlink 28117 (time 1746140316.51 total 175.38 last 164.79) - /unlink 29643 (time 1746140326.51 total 185.38 last 152.54) - /unlink 30000 (time 1746140328.77 total 187.63 last 158.56) - /unlink 31566 (time 1746140338.77 total 197.64 last 156.57) - /unlink 33157 (time 1746140348.77 total 207.64 last 159.09) - /unlink 34795 (time 1746140358.77 total 217.64 last 163.71) - /unlink 36447 (time 1746140368.78 total 227.64 last 165.18) - /unlink 38012 (time 1746140378.78 total 237.65 last 156.45) - /unlink 39736 (time 1746140388.78 total 247.65 last 172.33) - /unlink 41579 (time 1746140398.79 total 257.65 last 184.23) - /unlink 43157 (time 1746140408.79 total 267.65 last 157.78) - /unlink 44743 (time 1746140418.79 total 277.66 last 158.53) - /unlink 46439 (time 1746140428.80 total 287.67 last 169.46) - /unlink 48064 (time 1746140438.80 total 297.67 last 162.47) - /unlink 49789 (time 1746140448.80 total 307.67 last 172.47) - /unlink 51631 (time 1746140458.81 total 317.68 last 184.06) - /unlink 53545 (time 1746140468.81 total 327.68 last 191.35) - /unlink 55269 (time 1746140478.82 total 337.68 last 172.38) - /unlink 57074 (time 1746140488.82 total 347.68 last 180.48) - /unlink 58812 (time 1746140498.82 total 357.69 last 173.77) - /unlink 60000 (time 1746140505.50 total 364.37 last 177.80) - /unlink 61733 (time 1746140515.50 total 374.37 last 173.25) - /unlink 63452 (time 1746140525.51 total 384.38 last 171.78) total: 64768 /unlink in 392.63 seconds: 164.96 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1095 (time 1746140564.38 total 10.01 last 109.42) - open/close 2173 (time 1746140574.38 total 20.01 last 107.79) - open/close 3308 (time 1746140584.39 total 30.02 last 113.41) - open/close 4320 (time 1746140594.40 total 40.02 last 101.11) - open/close 5312 (time 1746140604.40 total 50.03 last 99.18) - open/close 6302 (time 1746140614.41 total 60.04 last 98.87) - open/close 7379 (time 1746140624.43 total 70.05 last 107.54) - open/close 8409 (time 1746140634.44 total 80.06 last 102.93) - open/close 9436 (time 1746140644.44 total 90.07 last 102.60) - open/close 10000 (time 1746140649.08 total 94.70 last 121.81) - open/close 11282 (time 1746140659.08 total 104.70 last 128.15) - open/close 12496 (time 1746140669.08 total 114.71 last 121.35) - open/close 13913 (time 1746140679.09 total 124.71 last 141.61) - open/close 15508 (time 1746140689.10 total 134.72 last 159.40) - open/close 16783 (time 1746140699.10 total 144.73 last 127.42) - open/close 17942 (time 1746140709.11 total 154.73 last 115.85) - open/close 19012 (time 1746140719.11 total 164.73 last 106.98) - open/close 20000 (time 1746140727.27 total 172.89 last 121.06) - open/close 21094 (time 1746140737.27 total 182.90 last 109.35) - open/close 22266 (time 1746140747.28 total 192.90 last 117.16) - open/close 23515 (time 1746140757.28 total 202.91 last 124.83) - open/close 24891 (time 1746140767.28 total 212.91 last 137.59) - open/close 26153 (time 1746140777.29 total 222.91 last 126.17) - open/close 27413 (time 1746140787.29 total 232.92 last 125.91) - open/close 28654 (time 1746140797.30 total 242.92 last 124.03) - open/close 30000 (time 1746140806.44 total 252.06 last 147.31) - open/close 31647 (time 1746140816.44 total 262.06 last 164.66) - open/close 33059 (time 1746140826.44 total 272.07 last 141.14) - open/close 34444 (time 1746140836.44 total 282.07 last 138.48) - open/close 35660 (time 1746140846.45 total 292.07 last 121.58) - open/close 37157 (time 1746140856.45 total 302.07 last 149.67) - open/close 38328 (time 1746140866.45 total 312.08 last 117.06) - open/close 39850 (time 1746140876.46 total 322.08 last 152.10) - open/close 41326 (time 1746140886.46 total 332.08 last 147.57) - open/close 43151 (time 1746140896.46 total 342.09 last 182.42) - open/close 44650 (time 1746140906.47 total 352.09 last 149.88) - open/close 46098 (time 1746140916.47 total 362.09 last 144.79) - open/close 47394 (time 1746140926.47 total 372.09 last 129.56) - open/close 48656 (time 1746140936.47 total 382.10 last 126.19) - open/close 49975 (time 1746140946.48 total 392.10 last 131.83) - open/close 51447 (time 1746140956.48 total 402.10 last 147.14) - open/close 52827 (time 1746140966.48 total 412.11 last 137.99) - open/close 54769 (time 1746140976.48 total 422.11 last 194.17) - open/close 56359 (time 1746140986.49 total 432.11 last 158.91) - open/close 57915 (time 1746140996.49 total 442.12 last 155.52) - open/close 59761 (time 1746141006.50 total 452.12 last 184.54) - open/close 61385 (time 1746141016.50 total 462.13 last 162.35) - open/close 62937 (time 1746141026.50 total 472.13 last 155.16) total: 64768 open/close in 481.74 seconds: 134.45 ops/second - /unlink 2109 (time 1746141047.71 total 10.00 last 210.88) - /unlink 3878 (time 1746141057.72 total 20.01 last 176.72) - /unlink 5951 (time 1746141067.72 total 30.01 last 207.30) - /unlink 8103 (time 1746141077.72 total 40.01 last 215.18) - /unlink 10000 (time 1746141087.49 total 49.78 last 194.29) - /unlink 11950 (time 1746141097.50 total 59.78 last 194.84) - /unlink 13632 (time 1746141107.50 total 69.79 last 168.15) - /unlink 15259 (time 1746141117.50 total 79.79 last 162.69) - /unlink 17061 (time 1746141127.51 total 89.80 last 180.04) - /unlink 18879 (time 1746141137.51 total 99.80 last 181.76) - /unlink 20000 (time 1746141143.55 total 105.84 last 185.71) - /unlink 22148 (time 1746141153.55 total 115.84 last 214.74) - /unlink 24251 (time 1746141163.56 total 125.85 last 210.10) - /unlink 26205 (time 1746141173.56 total 135.85 last 195.31) - /unlink 28234 (time 1746141183.57 total 145.86 last 202.81) - /unlink 29992 (time 1746141193.59 total 155.88 last 175.37) - /unlink 32134 (time 1746141203.60 total 165.88 last 214.16) - /unlink 34091 (time 1746141213.60 total 175.89 last 195.59) - /unlink 35934 (time 1746141223.60 total 185.89 last 184.25) - /unlink 37741 (time 1746141233.61 total 195.90 last 180.62) - /unlink 39948 (time 1746141243.61 total 205.90 last 220.70) - /unlink 42224 (time 1746141253.61 total 215.90 last 227.58) - /unlink 44443 (time 1746141263.61 total 225.90 last 221.89) - /unlink 46664 (time 1746141273.61 total 235.90 last 221.97) - /unlink 48742 (time 1746141283.62 total 245.90 last 207.79) - /unlink 50000 (time 1746141290.03 total 252.32 last 196.18) - /unlink 51785 (time 1746141300.03 total 262.32 last 178.46) - /unlink 54012 (time 1746141310.03 total 272.32 last 222.64) - /unlink 56314 (time 1746141320.04 total 282.32 last 230.13) - /unlink 58787 (time 1746141330.04 total 292.33 last 247.23) - /unlink 60000 (time 1746141334.82 total 297.11 last 253.72) - /unlink 62551 (time 1746141344.82 total 307.11 last 255.07) total: 64768 /unlink in 315.50 seconds: 205.29 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1429 (time 1746141386.26 total 10.01 last 142.79) - open/close 3049 (time 1746141396.26 total 20.01 last 161.97) - open/close 4808 (time 1746141406.26 total 30.01 last 175.87) - open/close 6208 (time 1746141416.26 total 40.01 last 139.98) - open/close 7532 (time 1746141426.26 total 50.02 last 132.37) - open/close 9034 (time 1746141436.27 total 60.02 last 150.16) - open/close 10000 (time 1746141443.00 total 66.75 last 143.58) - open/close 11244 (time 1746141453.00 total 76.75 last 124.36) - open/close 12747 (time 1746141463.00 total 86.75 last 150.21) - open/close 14548 (time 1746141473.01 total 96.76 last 180.02) - open/close 16061 (time 1746141483.02 total 106.77 last 151.19) - open/close 17550 (time 1746141493.02 total 116.77 last 148.81) - open/close 18989 (time 1746141503.02 total 126.77 last 143.87) - open/close 20000 (time 1746141509.99 total 133.74 last 145.19) - open/close 21519 (time 1746141519.99 total 143.74 last 151.89) - open/close 23165 (time 1746141529.99 total 153.74 last 164.57) - open/close 24421 (time 1746141539.99 total 163.74 last 125.57) - open/close 25874 (time 1746141549.99 total 173.74 last 145.27) - open/close 27245 (time 1746141560.00 total 183.75 last 136.96) - open/close 28657 (time 1746141570.01 total 193.76 last 141.18) - open/close 30000 (time 1746141579.95 total 203.71 last 134.98) - open/close 31229 (time 1746141589.96 total 213.71 last 122.84) - open/close 32464 (time 1746141599.97 total 223.72 last 123.37) - open/close 33670 (time 1746141609.97 total 233.72 last 120.60) - open/close 34918 (time 1746141619.98 total 243.73 last 124.73) - open/close 36205 (time 1746141629.98 total 253.73 last 128.65) - open/close 37566 (time 1746141639.99 total 263.74 last 136.01) - open/close 39085 (time 1746141649.99 total 273.74 last 151.84) - open/close 40000 (time 1746141657.63 total 281.38 last 119.85) - open/close 41146 (time 1746141667.63 total 291.38 last 114.53) - open/close 42783 (time 1746141677.63 total 301.38 last 163.67) - open/close 44386 (time 1746141687.64 total 311.39 last 160.26) - open/close 45641 (time 1746141697.64 total 321.39 last 125.49) - open/close 46920 (time 1746141707.64 total 331.39 last 127.88) - open/close 48103 (time 1746141717.64 total 341.39 last 118.29) - open/close 49331 (time 1746141727.64 total 351.39 last 122.76) - open/close 50000 (time 1746141732.97 total 356.72 last 125.62) - open/close 51366 (time 1746141742.98 total 366.73 last 136.52) - open/close 52661 (time 1746141752.98 total 376.73 last 129.43) - open/close 53870 (time 1746141762.98 total 386.73 last 120.88) - open/close 55093 (time 1746141772.99 total 396.74 last 122.21) - open/close 56152 (time 1746141783.00 total 406.75 last 105.83) - open/close 57167 (time 1746141793.00 total 416.75 last 101.44) - open/close 58445 (time 1746141803.00 total 426.75 last 127.79) - open/close 60000 (time 1746141812.80 total 436.55 last 158.68) - open/close 61227 (time 1746141822.80 total 446.55 last 122.68) - open/close 62511 (time 1746141832.81 total 456.56 last 128.35) - open/close 63733 (time 1746141842.81 total 466.56 last 122.12) total: 64768 open/close in 474.87 seconds: 136.39 ops/second - /unlink 1787 (time 1746141862.92 total 10.00 last 178.66) - /unlink 3573 (time 1746141872.93 total 20.01 last 178.48) - /unlink 5399 (time 1746141882.93 total 30.01 last 182.57) - /unlink 7135 (time 1746141892.93 total 40.01 last 173.58) - /unlink 9687 (time 1746141902.94 total 50.01 last 255.14) - /unlink 11321 (time 1746141912.94 total 60.02 last 163.34) - /unlink 12874 (time 1746141922.95 total 70.03 last 155.13) - /unlink 14333 (time 1746141932.95 total 80.03 last 145.89) - /unlink 16002 (time 1746141942.95 total 90.03 last 166.88) - /unlink 17523 (time 1746141952.95 total 100.03 last 152.05) - /unlink 19044 (time 1746141962.96 total 110.04 last 152.08) - /unlink 20000 (time 1746141967.31 total 114.39 last 219.43) - /unlink 21828 (time 1746141977.31 total 124.39 last 182.77) - /unlink 23987 (time 1746141987.32 total 134.40 last 215.86) - /unlink 26478 (time 1746141997.32 total 144.40 last 249.04) - /unlink 28599 (time 1746142007.32 total 154.40 last 212.05) - /unlink 30000 (time 1746142013.12 total 160.20 last 241.70) - /unlink 32354 (time 1746142023.12 total 170.20 last 235.36) - /unlink 34246 (time 1746142033.12 total 180.20 last 189.15) - /unlink 36337 (time 1746142043.13 total 190.21 last 208.99) - /unlink 38833 (time 1746142053.13 total 200.21 last 249.55) - /unlink 40000 (time 1746142056.83 total 203.91 last 315.68) - /unlink 42839 (time 1746142066.83 total 213.91 last 283.81) - /unlink 45427 (time 1746142076.83 total 223.91 last 258.74) - /unlink 47630 (time 1746142086.83 total 233.91 last 220.26) - /unlink 49802 (time 1746142096.84 total 243.92 last 217.11) - /unlink 51723 (time 1746142106.84 total 253.92 last 192.09) - /unlink 54170 (time 1746142116.84 total 263.92 last 244.62) - /unlink 56544 (time 1746142126.85 total 273.92 last 237.32) - /unlink 59508 (time 1746142136.85 total 283.93 last 296.18) - /unlink 61799 (time 1746142146.85 total 293.93 last 229.08) - /unlink 63875 (time 1746142156.86 total 303.93 last 207.56) total: 64768 /unlink in 308.54 seconds: 209.92 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg419-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server unloading modules via unload_modules_local on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 PASS 106 (4182s) Stopping clients: oleg419-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg419-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 oleg419-server: oleg419-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: detected 4 online CPUs by sysfs oleg419-server: MODOPTS_LIBCFS= oleg419-server: Force libcfs to create 2 CPU partitions oleg419-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg419-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4278 sec ========== 19:30:25 (1746142225) === conf-sanity: start cleanup 19:30:26 (1746142226) === === conf-sanity: finish cleanup 19:30:27 (1746142227) ===