-----============= acceptance-small: conf-sanity ============----- Thu May 1 20:09:36 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg221-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg221-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg221-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg221-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg221-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg221-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg221-server oleg221-server: oleg221-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing load_modules_local oleg221-server: Loading modules from /home/green/git/lustre-release/lustre oleg221-server: detected 4 online CPUs by sysfs oleg221-server: MODOPTS_LIBCFS= oleg221-server: Force libcfs to create 2 CPU partitions oleg221-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg221-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg221-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg221-server: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg221-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg221-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg221-server stop mds service on oleg221-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg221-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 20:11:01 (1746144661) Stopping clients: oleg221-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg221-client.virtnet /mnt/lustre2 (opts:-f) oleg221-server: oleg221-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing load_modules_local oleg221-server: Loading modules from /home/green/git/lustre-release/lustre oleg221-server: detected 4 online CPUs by sysfs oleg221-server: MODOPTS_LIBCFS= oleg221-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg221-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg221-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg221-client.virtnet: -o user_xattr,flock 192.168.202.121@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1210 (time 1746144715.67 total 10.00 last 120.99) - open/close 2664 (time 1746144725.67 total 20.00 last 145.40) - open/close 4241 (time 1746144735.68 total 30.00 last 157.65) - open/close 5962 (time 1746144745.68 total 40.01 last 172.05) - open/close 7415 (time 1746144755.68 total 50.01 last 145.26) - open/close 8621 (time 1746144765.68 total 60.01 last 120.60) - open/close 10000 (time 1746144775.26 total 69.59 last 143.98) - open/close 11468 (time 1746144785.26 total 79.59 last 146.73) - open/close 12973 (time 1746144795.27 total 89.60 last 150.37) - open/close 14273 (time 1746144805.27 total 99.60 last 129.98) - open/close 15437 (time 1746144815.28 total 109.61 last 116.32) - open/close 16718 (time 1746144825.28 total 119.61 last 128.06) - open/close 18020 (time 1746144835.29 total 129.62 last 130.11) - open/close 19466 (time 1746144845.31 total 139.64 last 144.34) - open/close 20000 (time 1746144849.23 total 143.56 last 136.04) - open/close 21383 (time 1746144859.24 total 153.57 last 138.20) - open/close 22542 (time 1746144869.25 total 163.58 last 115.83) - open/close 23755 (time 1746144879.25 total 173.58 last 121.28) - open/close 25074 (time 1746144889.25 total 183.58 last 131.88) - open/close 26244 (time 1746144899.25 total 193.58 last 116.95) - open/close 27746 (time 1746144909.26 total 203.59 last 150.14) - open/close 29040 (time 1746144919.26 total 213.59 last 129.36) - open/close 30000 (time 1746144926.75 total 221.07 last 128.28) - open/close 31244 (time 1746144936.75 total 231.08 last 124.38) - open/close 32565 (time 1746144946.75 total 241.08 last 132.06) - open/close 34015 (time 1746144956.75 total 251.08 last 144.97) - open/close 35338 (time 1746144966.76 total 261.09 last 132.23) - open/close 36602 (time 1746144976.76 total 271.09 last 126.35) - open/close 37892 (time 1746144986.76 total 281.09 last 128.96) - open/close 39186 (time 1746144996.77 total 291.10 last 129.35) - open/close 40000 (time 1746145003.20 total 297.53 last 126.55) - open/close 41356 (time 1746145013.21 total 307.54 last 135.51) - open/close 42665 (time 1746145023.21 total 317.54 last 130.85) - open/close 43933 (time 1746145033.21 total 327.54 last 126.76) - open/close 45154 (time 1746145043.21 total 337.54 last 122.10) - open/close 46355 (time 1746145053.22 total 347.55 last 120.03) - open/close 47593 (time 1746145063.22 total 357.55 last 123.78) - open/close 48843 (time 1746145073.23 total 367.55 last 124.96) - open/close 50000 (time 1746145082.05 total 376.38 last 131.13) - open/close 51635 (time 1746145092.05 total 386.38 last 163.46) - open/close 53142 (time 1746145102.05 total 396.38 last 150.68) - open/close 54490 (time 1746145112.06 total 406.39 last 134.72) - open/close 55791 (time 1746145122.06 total 416.39 last 130.09) - open/close 57267 (time 1746145132.06 total 426.39 last 147.53) - open/close 58747 (time 1746145142.07 total 436.40 last 147.93) - open/close 60000 (time 1746145150.02 total 444.35 last 157.56) - open/close 61365 (time 1746145160.02 total 454.35 last 136.48) - open/close 62810 (time 1746145170.03 total 464.36 last 144.35) - open/close 64014 (time 1746145180.04 total 474.37 last 120.32) total: 64768 open/close in 478.77 seconds: 135.28 ops/second - /unlink 3062 (time 1746145195.52 total 10.00 last 306.15) - /unlink 6309 (time 1746145205.53 total 20.00 last 324.60) - /unlink 9155 (time 1746145215.53 total 30.01 last 284.47) - /unlink 10000 (time 1746145219.59 total 34.06 last 208.35) - /unlink 12447 (time 1746145229.59 total 44.07 last 244.65) - /unlink 15074 (time 1746145239.59 total 54.07 last 262.65) - /unlink 17787 (time 1746145249.59 total 64.07 last 271.20) - /unlink 20000 (time 1746145259.35 total 73.83 last 226.84) - /unlink 22251 (time 1746145269.35 total 83.83 last 225.04) - /unlink 24776 (time 1746145279.36 total 93.84 last 252.37) - /unlink 26836 (time 1746145289.36 total 103.84 last 205.93) - /unlink 28862 (time 1746145299.36 total 113.84 last 202.52) - /unlink 30000 (time 1746145304.14 total 118.62 last 238.20) - /unlink 32162 (time 1746145314.15 total 128.63 last 216.01) - /unlink 34961 (time 1746145324.16 total 138.64 last 279.56) - /unlink 37803 (time 1746145334.17 total 148.65 last 284.07) - /unlink 40000 (time 1746145341.24 total 155.72 last 310.52) - /unlink 42457 (time 1746145351.24 total 165.72 last 245.68) - /unlink 45154 (time 1746145361.25 total 175.72 last 269.63) - /unlink 47653 (time 1746145371.25 total 185.73 last 249.88) - /unlink 50000 (time 1746145380.14 total 194.62 last 264.02) - /unlink 52536 (time 1746145390.14 total 204.62 last 253.59) - /unlink 55820 (time 1746145400.14 total 214.62 last 328.33) - /unlink 58480 (time 1746145410.16 total 224.64 last 265.54) - /unlink 60000 (time 1746145416.51 total 230.99 last 239.33) - /unlink 62537 (time 1746145426.51 total 240.99 last 253.69) total: 64768 /unlink in 250.22 seconds: 258.84 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1354 (time 1746145462.85 total 10.00 last 135.38) - open/close 2846 (time 1746145472.87 total 20.03 last 148.80) - open/close 4363 (time 1746145482.88 total 30.03 last 151.64) - open/close 5881 (time 1746145492.88 total 40.04 last 151.71) - open/close 7292 (time 1746145502.90 total 50.05 last 140.92) - open/close 8540 (time 1746145512.90 total 60.06 last 124.71) - open/close 9799 (time 1746145522.91 total 70.06 last 125.87) - open/close 10988 (time 1746145532.92 total 80.07 last 118.80) - open/close 12024 (time 1746145542.92 total 90.07 last 103.60) - open/close 13123 (time 1746145552.92 total 100.07 last 109.87) - open/close 14450 (time 1746145562.92 total 110.08 last 132.64) - open/close 15745 (time 1746145572.93 total 120.09 last 129.38) - open/close 17013 (time 1746145582.94 total 130.09 last 126.70) - open/close 18432 (time 1746145592.94 total 140.10 last 141.89) - open/close 19571 (time 1746145602.96 total 150.12 last 113.65) - open/close 20000 (time 1746145605.98 total 153.13 last 142.18) - open/close 21131 (time 1746145615.99 total 163.14 last 113.00) - open/close 22427 (time 1746145625.99 total 173.14 last 129.59) - open/close 23752 (time 1746145635.99 total 183.15 last 132.49) - open/close 25035 (time 1746145646.00 total 193.15 last 128.22) - open/close 26296 (time 1746145656.00 total 203.15 last 126.10) - open/close 27713 (time 1746145666.00 total 213.16 last 141.65) - open/close 28973 (time 1746145676.01 total 223.16 last 125.94) - open/close 30000 (time 1746145684.38 total 231.54 last 122.58) - open/close 31312 (time 1746145694.39 total 241.55 last 131.08) - open/close 32585 (time 1746145704.40 total 251.55 last 127.24) - open/close 34063 (time 1746145714.41 total 261.56 last 147.64) - open/close 35486 (time 1746145724.41 total 271.56 last 142.30) - open/close 36828 (time 1746145734.41 total 281.57 last 134.17) - open/close 38171 (time 1746145744.42 total 291.57 last 134.22) - open/close 39562 (time 1746145754.42 total 301.57 last 139.08) - open/close 40000 (time 1746145757.97 total 305.12 last 123.50) - open/close 41172 (time 1746145767.97 total 315.12 last 117.15) - open/close 42288 (time 1746145777.97 total 325.12 last 111.60) - open/close 43589 (time 1746145787.99 total 335.14 last 129.91) - open/close 44890 (time 1746145797.99 total 345.14 last 130.04) - open/close 46268 (time 1746145808.00 total 355.15 last 137.68) - open/close 47712 (time 1746145818.00 total 365.16 last 144.33) - open/close 49272 (time 1746145828.00 total 375.16 last 156.00) - open/close 50000 (time 1746145832.43 total 379.58 last 164.44) - open/close 51435 (time 1746145842.43 total 389.59 last 143.48) - open/close 52869 (time 1746145852.43 total 399.59 last 143.36) - open/close 54191 (time 1746145862.44 total 409.59 last 132.17) - open/close 55482 (time 1746145872.44 total 419.59 last 129.09) - open/close 56779 (time 1746145882.44 total 429.60 last 129.65) - open/close 57980 (time 1746145892.45 total 439.60 last 120.03) - open/close 59280 (time 1746145902.45 total 449.60 last 129.99) - open/close 60000 (time 1746145908.36 total 455.52 last 121.73) - open/close 61144 (time 1746145918.37 total 465.52 last 114.36) - open/close 62435 (time 1746145928.37 total 475.53 last 129.00) - open/close 63655 (time 1746145938.39 total 485.54 last 121.83) total: 64768 open/close in 494.88 seconds: 130.88 ops/second - /unlink 2350 (time 1746145959.48 total 10.00 last 234.95) - /unlink 4534 (time 1746145969.49 total 20.01 last 218.33) - /unlink 6656 (time 1746145979.49 total 30.01 last 212.17) - /unlink 8409 (time 1746145989.49 total 40.01 last 175.21) - /unlink 10000 (time 1746145995.97 total 46.49 last 245.62) - /unlink 12809 (time 1746146005.97 total 56.49 last 280.87) - /unlink 14990 (time 1746146015.97 total 66.49 last 218.09) - /unlink 17112 (time 1746146025.98 total 76.49 last 212.17) - /unlink 19265 (time 1746146035.99 total 86.50 last 215.06) - /unlink 20000 (time 1746146039.46 total 89.98 last 211.71) - /unlink 22736 (time 1746146049.46 total 99.98 last 273.58) - /unlink 25283 (time 1746146059.46 total 109.98 last 254.69) - /unlink 27886 (time 1746146069.46 total 119.98 last 260.28) - /unlink 30000 (time 1746146077.87 total 128.38 last 251.49) - /unlink 32584 (time 1746146087.87 total 138.38 last 258.39) - /unlink 34679 (time 1746146097.87 total 148.39 last 209.39) - /unlink 36815 (time 1746146107.87 total 158.39 last 213.53) - /unlink 39092 (time 1746146117.88 total 168.39 last 227.67) - /unlink 40000 (time 1746146121.56 total 172.08 last 246.63) - /unlink 42560 (time 1746146131.56 total 182.08 last 255.94) - /unlink 44883 (time 1746146141.56 total 192.08 last 232.29) - /unlink 47278 (time 1746146151.56 total 202.08 last 239.41) - /unlink 49437 (time 1746146161.57 total 212.09 last 215.76) - /unlink 50000 (time 1746146164.34 total 214.86 last 203.24) - /unlink 52052 (time 1746146174.35 total 224.86 last 205.08) - /unlink 54559 (time 1746146184.35 total 234.87 last 250.61) - /unlink 56810 (time 1746146194.35 total 244.87 last 225.04) - /unlink 60000 (time 1746146204.16 total 254.68 last 325.14) - /unlink 62586 (time 1746146214.16 total 264.68 last 258.58) total: 64768 /unlink in 273.73 seconds: 236.61 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1520 (time 1746146253.08 total 10.01 last 151.90) - open/close 2839 (time 1746146263.08 total 20.01 last 131.87) - open/close 4391 (time 1746146273.08 total 30.01 last 155.15) - open/close 6046 (time 1746146283.08 total 40.01 last 165.48) - open/close 7555 (time 1746146293.09 total 50.01 last 150.89) - open/close 9097 (time 1746146303.09 total 60.02 last 154.15) - open/close 10000 (time 1746146308.65 total 65.58 last 162.31) - open/close 11568 (time 1746146318.65 total 75.58 last 156.76) - open/close 13146 (time 1746146328.66 total 85.59 last 157.74) - open/close 14693 (time 1746146338.66 total 95.59 last 154.63) - open/close 16303 (time 1746146348.66 total 105.59 last 160.96) - open/close 17954 (time 1746146358.67 total 115.60 last 165.04) - open/close 19445 (time 1746146368.68 total 125.61 last 148.97) - open/close 20000 (time 1746146372.65 total 129.58 last 139.61) - open/close 21468 (time 1746146382.66 total 139.59 last 146.73) - open/close 22825 (time 1746146392.66 total 149.59 last 135.60) - open/close 24200 (time 1746146402.67 total 159.60 last 137.40) - open/close 25683 (time 1746146412.67 total 169.60 last 148.27) - open/close 26911 (time 1746146422.68 total 179.61 last 122.70) - open/close 28252 (time 1746146432.68 total 189.61 last 134.09) - open/close 29712 (time 1746146442.69 total 199.62 last 145.89) - open/close 30000 (time 1746146444.80 total 201.73 last 136.52) - open/close 31429 (time 1746146454.80 total 211.73 last 142.89) - open/close 32973 (time 1746146464.80 total 221.73 last 154.33) - open/close 34327 (time 1746146474.81 total 231.74 last 135.32) - open/close 35778 (time 1746146484.81 total 241.74 last 145.10) - open/close 37118 (time 1746146494.82 total 251.74 last 133.93) - open/close 38449 (time 1746146504.82 total 261.75 last 132.99) - open/close 39636 (time 1746146514.83 total 271.75 last 118.68) - open/close 40000 (time 1746146517.45 total 274.38 last 138.70) - open/close 41338 (time 1746146527.46 total 284.38 last 133.72) - open/close 42569 (time 1746146537.46 total 294.39 last 123.08) - open/close 43744 (time 1746146547.47 total 304.40 last 117.39) - open/close 44879 (time 1746146557.47 total 314.40 last 113.45) - open/close 46157 (time 1746146567.48 total 324.41 last 127.72) - open/close 47224 (time 1746146577.48 total 334.41 last 106.69) - open/close 48326 (time 1746146587.48 total 344.41 last 110.17) - open/close 49493 (time 1746146597.49 total 354.42 last 116.61) - open/close 50000 (time 1746146601.71 total 358.64 last 120.20) - open/close 51188 (time 1746146611.71 total 368.64 last 118.77) - open/close 52334 (time 1746146621.72 total 378.65 last 114.46) - open/close 53484 (time 1746146631.73 total 388.65 last 114.95) - open/close 54569 (time 1746146641.73 total 398.66 last 108.45) - open/close 55789 (time 1746146651.74 total 408.66 last 121.95) - open/close 56983 (time 1746146661.75 total 418.67 last 119.28) - open/close 58183 (time 1746146671.75 total 428.68 last 119.95) - open/close 59394 (time 1746146681.75 total 438.68 last 121.07) - open/close 60000 (time 1746146686.43 total 443.36 last 129.43) - open/close 61179 (time 1746146696.44 total 453.37 last 117.82) - open/close 62371 (time 1746146706.45 total 463.38 last 119.04) - open/close 63507 (time 1746146716.46 total 473.38 last 113.57) - open/close 64748 (time 1746146726.46 total 483.39 last 124.06) total: 64768 open/close in 483.57 seconds: 133.94 ops/second - /unlink 2452 (time 1746146738.16 total 10.00 last 245.16) - /unlink 5218 (time 1746146748.16 total 20.01 last 276.50) - /unlink 7710 (time 1746146758.17 total 30.01 last 249.06) - /unlink 10000 (time 1746146767.43 total 39.27 last 247.31) - /unlink 12317 (time 1746146777.43 total 49.27 last 231.65) - /unlink 14405 (time 1746146787.43 total 59.28 last 208.69) - /unlink 16510 (time 1746146797.44 total 69.28 last 210.44) - /unlink 18465 (time 1746146807.44 total 79.28 last 195.48) - /unlink 20000 (time 1746146814.97 total 86.81 last 203.77) - /unlink 21851 (time 1746146824.97 total 96.82 last 185.06) - /unlink 24119 (time 1746146834.97 total 106.82 last 226.80) - /unlink 26637 (time 1746146844.98 total 116.82 last 251.73) - /unlink 29112 (time 1746146854.98 total 126.82 last 247.42) - /unlink 30000 (time 1746146858.90 total 130.75 last 226.26) - /unlink 32689 (time 1746146868.90 total 140.75 last 268.87) - /unlink 35279 (time 1746146878.90 total 150.75 last 259.00) - /unlink 37971 (time 1746146888.91 total 160.75 last 269.17) - /unlink 40000 (time 1746146895.39 total 167.24 last 312.83) - /unlink 42725 (time 1746146905.40 total 177.24 last 272.37) - /unlink 45151 (time 1746146915.40 total 187.24 last 242.59) - /unlink 47466 (time 1746146925.40 total 197.24 last 231.42) - /unlink 49704 (time 1746146935.40 total 207.25 last 223.72) - /unlink 52254 (time 1746146945.41 total 217.26 last 254.81) - /unlink 54282 (time 1746146955.41 total 227.26 last 202.80) - /unlink 56267 (time 1746146965.41 total 237.26 last 198.44) - /unlink 58347 (time 1746146975.42 total 247.26 last 207.91) - /unlink 60000 (time 1746146981.96 total 253.80 last 252.87) - /unlink 62378 (time 1746146991.96 total 263.80 last 237.71) - /unlink 64646 (time 1746147001.96 total 273.81 last 226.76) total: 64768 /unlink in 274.36 seconds: 236.07 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1185 (time 1746147034.52 total 10.00 last 118.48) - open/close 2310 (time 1746147044.53 total 20.01 last 112.38) - open/close 3627 (time 1746147054.54 total 30.02 last 131.63) - open/close 4982 (time 1746147064.54 total 40.02 last 135.48) - open/close 6249 (time 1746147074.54 total 50.02 last 126.69) - open/close 7579 (time 1746147084.55 total 60.03 last 132.93) - open/close 8891 (time 1746147094.55 total 70.03 last 131.18) - open/close 10000 (time 1746147102.89 total 78.37 last 132.90) - open/close 11292 (time 1746147112.90 total 88.38 last 129.13) - open/close 12581 (time 1746147122.90 total 98.38 last 128.85) - open/close 13829 (time 1746147132.91 total 108.39 last 124.69) - open/close 15079 (time 1746147142.91 total 118.39 last 124.97) - open/close 16272 (time 1746147152.91 total 128.39 last 119.26) - open/close 17612 (time 1746147162.92 total 138.40 last 133.91) - open/close 18773 (time 1746147172.92 total 148.40 last 116.06) - open/close 20000 (time 1746147182.38 total 157.86 last 129.77) - open/close 21194 (time 1746147192.38 total 167.86 last 119.40) - open/close 22623 (time 1746147202.38 total 177.86 last 142.85) - open/close 24015 (time 1746147212.39 total 187.87 last 139.14) - open/close 25549 (time 1746147222.39 total 197.87 last 153.35) - open/close 26803 (time 1746147232.39 total 207.87 last 125.35) - open/close 28272 (time 1746147242.41 total 217.89 last 146.71) - open/close 29658 (time 1746147252.41 total 227.89 last 138.57) - open/close 30000 (time 1746147255.19 total 230.67 last 123.07) - open/close 31252 (time 1746147265.20 total 240.68 last 125.09) - open/close 32508 (time 1746147275.20 total 250.68 last 125.59) - open/close 33724 (time 1746147285.20 total 260.68 last 121.53) - open/close 34899 (time 1746147295.21 total 270.69 last 117.48) - open/close 36164 (time 1746147305.21 total 280.69 last 126.50) - open/close 37491 (time 1746147315.21 total 290.69 last 132.70) - open/close 38903 (time 1746147325.21 total 300.69 last 141.14) - open/close 40000 (time 1746147333.86 total 309.34 last 126.86) - open/close 41296 (time 1746147343.86 total 319.34 last 129.51) - open/close 42473 (time 1746147353.87 total 329.35 last 117.68) - open/close 43668 (time 1746147363.87 total 339.35 last 119.48) - open/close 44834 (time 1746147373.87 total 349.35 last 116.52) - open/close 46093 (time 1746147383.88 total 359.36 last 125.88) - open/close 47375 (time 1746147393.88 total 369.36 last 128.18) - open/close 48566 (time 1746147403.88 total 379.36 last 119.06) - open/close 49731 (time 1746147413.89 total 389.37 last 116.43) - open/close 50000 (time 1746147416.19 total 391.66 last 117.08) - open/close 51269 (time 1746147426.19 total 401.67 last 126.88) - open/close 52477 (time 1746147436.19 total 411.67 last 120.79) - open/close 53690 (time 1746147446.20 total 421.67 last 121.21) - open/close 54867 (time 1746147456.21 total 431.69 last 117.58) - open/close 56157 (time 1746147466.21 total 441.69 last 128.97) - open/close 57478 (time 1746147476.21 total 451.69 last 132.10) - open/close 58597 (time 1746147486.21 total 461.69 last 111.86) - open/close 59825 (time 1746147496.21 total 471.69 last 122.77) - open/close 61522 (time 1746147506.22 total 481.70 last 169.62) - open/close 62924 (time 1746147516.22 total 491.70 last 140.14) - open/close 64458 (time 1746147526.23 total 501.71 last 153.35) total: 64768 open/close in 504.20 seconds: 128.46 ops/second - /unlink 2342 (time 1746147540.28 total 10.00 last 234.14) - /unlink 4409 (time 1746147550.28 total 20.00 last 206.68) - /unlink 6853 (time 1746147560.29 total 30.01 last 244.27) - /unlink 9371 (time 1746147570.29 total 40.01 last 251.76) - /unlink 10000 (time 1746147572.64 total 42.36 last 267.48) - /unlink 12564 (time 1746147582.64 total 52.36 last 256.37) - /unlink 15145 (time 1746147592.65 total 62.37 last 257.99) - /unlink 17752 (time 1746147602.65 total 72.37 last 260.69) - /unlink 20000 (time 1746147611.54 total 81.27 last 252.64) - /unlink 23051 (time 1746147621.55 total 91.27 last 305.05) - /unlink 25128 (time 1746147631.55 total 101.27 last 207.70) - /unlink 27046 (time 1746147641.55 total 111.27 last 191.80) - /unlink 28934 (time 1746147651.55 total 121.27 last 188.73) - /unlink 30000 (time 1746147657.28 total 127.00 last 186.08) - /unlink 32052 (time 1746147667.28 total 137.00 last 205.18) - /unlink 34034 (time 1746147677.28 total 147.00 last 198.16) - /unlink 36157 (time 1746147687.29 total 157.01 last 212.20) - /unlink 38476 (time 1746147697.29 total 167.01 last 231.84) - /unlink 40000 (time 1746147704.00 total 173.73 last 226.95) - /unlink 42266 (time 1746147714.01 total 183.73 last 226.47) - /unlink 44350 (time 1746147724.01 total 193.73 last 208.35) - /unlink 46482 (time 1746147734.01 total 203.73 last 213.20) - /unlink 48924 (time 1746147744.01 total 213.73 last 244.17) - /unlink 50000 (time 1746147748.12 total 217.84 last 262.30) - /unlink 52070 (time 1746147758.12 total 227.84 last 206.95) - /unlink 54420 (time 1746147768.12 total 237.84 last 234.95) - /unlink 56665 (time 1746147778.12 total 247.84 last 224.47) - /unlink 58723 (time 1746147788.12 total 257.84 last 205.79) - /unlink 60000 (time 1746147794.51 total 264.23 last 199.98) - /unlink 62615 (time 1746147804.51 total 274.23 last 261.48) total: 64768 /unlink in 281.46 seconds: 230.11 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1577 (time 1746147841.97 total 10.00 last 157.68) - open/close 2855 (time 1746147851.98 total 20.01 last 127.75) - open/close 4040 (time 1746147861.98 total 30.01 last 118.49) - open/close 5454 (time 1746147871.98 total 40.01 last 141.35) - open/close 6846 (time 1746147881.99 total 50.02 last 138.99) - open/close 8210 (time 1746147892.00 total 60.03 last 136.33) - open/close 9496 (time 1746147902.00 total 70.03 last 128.54) - open/close 10000 (time 1746147905.75 total 73.78 last 134.44) - open/close 11312 (time 1746147915.76 total 83.79 last 131.07) - open/close 12665 (time 1746147925.77 total 93.80 last 135.17) - open/close 14087 (time 1746147935.78 total 103.81 last 142.14) - open/close 15549 (time 1746147945.78 total 113.81 last 146.11) - open/close 17173 (time 1746147955.78 total 123.82 last 162.38) - open/close 19180 (time 1746147965.79 total 133.82 last 200.64) - open/close 20000 (time 1746147971.15 total 139.18 last 152.80) - open/close 21571 (time 1746147981.16 total 149.19 last 156.98) - open/close 22975 (time 1746147991.16 total 159.19 last 140.40) - open/close 24405 (time 1746148001.17 total 169.20 last 142.93) - open/close 25877 (time 1746148011.17 total 179.20 last 147.19) - open/close 27516 (time 1746148021.17 total 189.20 last 163.90) - open/close 28911 (time 1746148031.17 total 199.20 last 139.44) - open/close 30000 (time 1746148038.59 total 206.62 last 146.76) - open/close 31896 (time 1746148048.60 total 216.63 last 189.53) - open/close 33928 (time 1746148058.60 total 226.63 last 203.09) - open/close 35454 (time 1746148068.61 total 236.64 last 152.56) - open/close 36959 (time 1746148078.61 total 246.64 last 150.45) - open/close 38331 (time 1746148088.61 total 256.64 last 137.16) - open/close 39819 (time 1746148098.61 total 266.64 last 148.79) - open/close 41069 (time 1746148108.62 total 276.65 last 124.91) - open/close 42404 (time 1746148118.63 total 286.66 last 133.39) - open/close 43957 (time 1746148128.63 total 296.66 last 155.24) - open/close 45730 (time 1746148138.64 total 306.67 last 177.18) - open/close 47277 (time 1746148148.65 total 316.68 last 154.49) - open/close 48822 (time 1746148158.66 total 326.69 last 154.43) - open/close 50000 (time 1746148166.35 total 334.38 last 153.05) - open/close 51314 (time 1746148176.36 total 344.39 last 131.37) - open/close 52627 (time 1746148186.36 total 354.39 last 131.23) - open/close 53956 (time 1746148196.36 total 364.39 last 132.86) - open/close 55387 (time 1746148206.37 total 374.40 last 143.01) - open/close 56918 (time 1746148216.37 total 384.40 last 153.08) - open/close 58415 (time 1746148226.38 total 394.41 last 149.56) - open/close 59961 (time 1746148236.39 total 404.42 last 154.53) - open/close 61207 (time 1746148246.39 total 414.42 last 124.57) - open/close 62633 (time 1746148256.39 total 424.42 last 142.54) - open/close 64124 (time 1746148266.40 total 434.43 last 149.01) total: 64768 open/close in 438.59 seconds: 147.67 ops/second - /unlink 2922 (time 1746148282.03 total 10.00 last 292.19) - /unlink 5639 (time 1746148292.03 total 20.00 last 271.70) - /unlink 8337 (time 1746148302.03 total 30.00 last 269.79) - /unlink 10000 (time 1746148309.43 total 37.41 last 224.57) - /unlink 12449 (time 1746148319.44 total 47.41 last 244.77) - /unlink 14777 (time 1746148329.44 total 57.41 last 232.76) - /unlink 17351 (time 1746148339.44 total 67.41 last 257.39) - /unlink 20000 (time 1746148349.00 total 76.98 last 276.98) - /unlink 22921 (time 1746148359.00 total 86.98 last 292.10) - /unlink 25474 (time 1746148369.01 total 96.98 last 255.24) - /unlink 28227 (time 1746148379.01 total 106.98 last 275.25) - /unlink 30000 (time 1746148385.59 total 113.56 last 269.44) - /unlink 32742 (time 1746148395.59 total 123.56 last 274.20) - /unlink 35708 (time 1746148405.59 total 133.56 last 296.59) - /unlink 37958 (time 1746148415.59 total 143.57 last 224.90) - /unlink 40000 (time 1746148424.15 total 152.12 last 238.66) - /unlink 42319 (time 1746148434.15 total 162.12 last 231.87) - /unlink 44314 (time 1746148444.15 total 172.13 last 199.46) - /unlink 46636 (time 1746148454.16 total 182.13 last 232.12) - /unlink 48751 (time 1746148464.16 total 192.13 last 211.42) - /unlink 50000 (time 1746148469.47 total 197.44 last 235.18) - /unlink 52101 (time 1746148479.47 total 207.45 last 210.03) - /unlink 54218 (time 1746148489.48 total 217.45 last 211.67) - /unlink 56400 (time 1746148499.48 total 227.45 last 218.16) - /unlink 58552 (time 1746148509.48 total 237.45 last 215.18) - /unlink 60000 (time 1746148516.02 total 243.99 last 221.42) - /unlink 62174 (time 1746148526.02 total 253.99 last 217.40) - /unlink 64250 (time 1746148536.02 total 263.99 last 207.57) total: 64768 /unlink in 266.40 seconds: 243.12 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg221-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg221-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg221-server stop mds service on oleg221-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg221-server unloading modules via unload_modules_local on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg221-client: oleg221-client: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-client: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2 PASS 106 (3921s) Stopping clients: oleg221-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg221-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2 oleg221-server: oleg221-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing load_modules_local oleg221-server: Loading modules from /home/green/git/lustre-release/lustre oleg221-server: detected 4 online CPUs by sysfs oleg221-server: MODOPTS_LIBCFS= oleg221-server: Force libcfs to create 2 CPU partitions oleg221-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg221-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4034 sec ========== 21:16:51 (1746148611) === conf-sanity: start cleanup 21:16:52 (1746148612) === === conf-sanity: finish cleanup 21:16:53 (1746148613) ===