-----============= acceptance-small: conf-sanity ============----- Thu May 1 22:52:50 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg203-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg203-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg203-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg203-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg203-server oleg203-server: oleg203-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing load_modules_local oleg203-server: Loading modules from /home/green/git/lustre-release/lustre oleg203-server: detected 4 online CPUs by sysfs oleg203-server: MODOPTS_LIBCFS= oleg203-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg203-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg203-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg203-server: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg203-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server stop mds service on oleg203-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 22:54:13 (1746154453) Stopping clients: oleg203-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg203-client.virtnet /mnt/lustre2 (opts:-f) oleg203-server: oleg203-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing load_modules_local oleg203-server: Loading modules from /home/green/git/lustre-release/lustre oleg203-server: detected 4 online CPUs by sysfs oleg203-server: MODOPTS_LIBCFS= oleg203-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg203-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg203-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1216 (time 1746154508.37 total 10.00 last 121.57) - open/close 2460 (time 1746154518.37 total 20.00 last 124.37) - open/close 3661 (time 1746154528.37 total 30.01 last 120.09) - open/close 4881 (time 1746154538.38 total 40.01 last 121.91) - open/close 6062 (time 1746154548.39 total 50.03 last 117.91) - open/close 7045 (time 1746154558.40 total 60.04 last 98.24) - open/close 8082 (time 1746154568.40 total 70.04 last 103.68) - open/close 9084 (time 1746154578.40 total 80.04 last 100.19) - open/close 10000 (time 1746154586.26 total 87.90 last 116.56) - open/close 11106 (time 1746154596.27 total 97.90 last 110.54) - open/close 12258 (time 1746154606.27 total 107.90 last 115.20) - open/close 13405 (time 1746154616.27 total 117.90 last 114.67) - open/close 14548 (time 1746154626.27 total 127.91 last 114.27) - open/close 15789 (time 1746154636.27 total 137.91 last 124.06) - open/close 17239 (time 1746154646.28 total 147.91 last 144.94) - open/close 18517 (time 1746154656.28 total 157.92 last 127.72) - open/close 19674 (time 1746154666.29 total 167.92 last 115.69) - open/close 20000 (time 1746154668.53 total 170.16 last 145.32) - open/close 21281 (time 1746154678.53 total 180.17 last 128.07) - open/close 22605 (time 1746154688.54 total 190.17 last 132.33) - open/close 23871 (time 1746154698.54 total 200.18 last 126.53) - open/close 25117 (time 1746154708.54 total 210.18 last 124.56) - open/close 26318 (time 1746154718.55 total 220.19 last 120.01) - open/close 27498 (time 1746154728.56 total 230.19 last 117.95) - open/close 28647 (time 1746154738.56 total 240.20 last 114.82) - open/close 29915 (time 1746154748.56 total 250.20 last 126.79) - open/close 31115 (time 1746154758.57 total 260.20 last 119.95) - open/close 32268 (time 1746154768.57 total 270.21 last 115.26) - open/close 33389 (time 1746154778.58 total 280.21 last 112.02) - open/close 34520 (time 1746154788.58 total 290.22 last 113.04) - open/close 35735 (time 1746154798.58 total 300.22 last 121.49) - open/close 37160 (time 1746154808.59 total 310.23 last 142.43) - open/close 38352 (time 1746154818.59 total 320.23 last 119.15) - open/close 39675 (time 1746154828.60 total 330.23 last 132.26) - open/close 40000 (time 1746154830.70 total 332.34 last 154.22) - open/close 41563 (time 1746154840.71 total 342.35 last 156.23) - open/close 42956 (time 1746154850.71 total 352.35 last 139.29) - open/close 44302 (time 1746154860.71 total 362.35 last 134.55) - open/close 45449 (time 1746154870.72 total 372.35 last 114.64) - open/close 46517 (time 1746154880.72 total 382.35 last 106.79) - open/close 47634 (time 1746154890.72 total 392.36 last 111.67) - open/close 48956 (time 1746154900.72 total 402.36 last 132.17) - open/close 50000 (time 1746154909.45 total 411.08 last 119.68) - open/close 51239 (time 1746154919.46 total 421.09 last 123.76) - open/close 52627 (time 1746154929.46 total 431.10 last 138.72) - open/close 53857 (time 1746154939.47 total 441.10 last 122.98) - open/close 55091 (time 1746154949.47 total 451.10 last 123.37) - open/close 56339 (time 1746154959.47 total 461.11 last 124.75) - open/close 57538 (time 1746154969.47 total 471.11 last 119.89) - open/close 58859 (time 1746154979.48 total 481.12 last 131.96) - open/close 60000 (time 1746154987.68 total 489.32 last 139.15) - open/close 61429 (time 1746154997.68 total 499.32 last 142.89) - open/close 62860 (time 1746155007.68 total 509.32 last 143.10) - open/close 64136 (time 1746155017.69 total 519.32 last 127.55) total: 64768 open/close in 524.49 seconds: 123.49 ops/second - /unlink 2163 (time 1746155034.82 total 10.00 last 216.22) - /unlink 4422 (time 1746155044.82 total 20.01 last 225.82) - /unlink 6585 (time 1746155054.82 total 30.01 last 216.26) - /unlink 8498 (time 1746155064.83 total 40.01 last 191.25) - /unlink 10000 (time 1746155072.19 total 47.38 last 203.87) - /unlink 12053 (time 1746155082.19 total 57.38 last 205.27) - /unlink 14070 (time 1746155092.20 total 67.38 last 201.68) - /unlink 15986 (time 1746155102.20 total 77.38 last 191.60) - /unlink 17936 (time 1746155112.20 total 87.38 last 195.00) - /unlink 20000 (time 1746155121.97 total 97.15 last 211.19) - /unlink 22064 (time 1746155131.97 total 107.16 last 206.36) - /unlink 23864 (time 1746155141.98 total 117.16 last 179.89) - /unlink 25505 (time 1746155151.98 total 127.16 last 164.10) - /unlink 27277 (time 1746155161.98 total 137.17 last 177.15) - /unlink 29204 (time 1746155171.98 total 147.17 last 192.64) - /unlink 30000 (time 1746155175.23 total 150.42 last 244.86) - /unlink 32490 (time 1746155185.24 total 160.42 last 248.90) - /unlink 34640 (time 1746155195.24 total 170.43 last 214.95) - /unlink 36734 (time 1746155205.24 total 180.43 last 209.39) - /unlink 38463 (time 1746155215.24 total 190.43 last 172.89) - /unlink 40000 (time 1746155223.41 total 198.60 last 188.14) - /unlink 41749 (time 1746155233.41 total 208.60 last 174.85) - /unlink 43510 (time 1746155243.42 total 218.60 last 176.05) - /unlink 45431 (time 1746155253.42 total 228.60 last 192.07) - /unlink 47472 (time 1746155263.42 total 238.60 last 204.09) - /unlink 49763 (time 1746155273.42 total 248.61 last 229.06) - /unlink 51815 (time 1746155283.43 total 258.61 last 205.08) - /unlink 53828 (time 1746155293.43 total 268.61 last 201.26) - /unlink 55793 (time 1746155303.43 total 278.62 last 196.42) - /unlink 57884 (time 1746155313.43 total 288.62 last 209.08) - /unlink 60000 (time 1746155323.40 total 298.59 last 212.29) - /unlink 62251 (time 1746155333.40 total 308.59 last 225.07) - /unlink 64564 (time 1746155343.41 total 318.60 last 231.11) total: 64768 /unlink in 319.46 seconds: 202.74 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1331 (time 1746155374.31 total 10.01 last 133.03) - open/close 2651 (time 1746155384.33 total 20.02 last 131.81) - open/close 3906 (time 1746155394.34 total 30.03 last 125.39) - open/close 5166 (time 1746155404.34 total 40.03 last 125.94) - open/close 6472 (time 1746155414.35 total 50.04 last 130.53) - open/close 7778 (time 1746155424.35 total 60.04 last 130.57) - open/close 9020 (time 1746155434.35 total 70.04 last 124.14) - open/close 10000 (time 1746155442.74 total 78.43 last 116.88) - open/close 11355 (time 1746155452.74 total 88.43 last 135.44) - open/close 12644 (time 1746155462.74 total 98.44 last 128.89) - open/close 13841 (time 1746155472.75 total 108.44 last 119.66) - open/close 15125 (time 1746155482.76 total 118.45 last 128.23) - open/close 16432 (time 1746155492.77 total 128.46 last 130.58) - open/close 17623 (time 1746155502.78 total 138.47 last 119.03) - open/close 18900 (time 1746155512.78 total 148.47 last 127.64) - open/close 20000 (time 1746155521.17 total 156.86 last 131.17) - open/close 21247 (time 1746155531.17 total 166.86 last 124.68) - open/close 22531 (time 1746155541.18 total 176.87 last 128.31) - open/close 23696 (time 1746155551.18 total 186.87 last 116.46) - open/close 24964 (time 1746155561.18 total 196.87 last 126.76) - open/close 26134 (time 1746155571.18 total 206.87 last 116.97) - open/close 27267 (time 1746155581.19 total 216.88 last 113.20) - open/close 28429 (time 1746155591.20 total 226.89 last 116.15) - open/close 29838 (time 1746155601.20 total 236.89 last 140.87) - open/close 31347 (time 1746155611.20 total 246.89 last 150.84) - open/close 32669 (time 1746155621.20 total 256.89 last 132.19) - open/close 34077 (time 1746155631.21 total 266.90 last 140.75) - open/close 35583 (time 1746155641.21 total 276.90 last 150.59) - open/close 37128 (time 1746155651.21 total 286.90 last 154.50) - open/close 38485 (time 1746155661.21 total 296.90 last 135.64) - open/close 39796 (time 1746155671.22 total 306.91 last 131.03) - open/close 41137 (time 1746155681.22 total 316.91 last 134.06) - open/close 42663 (time 1746155691.23 total 326.92 last 152.50) - open/close 43905 (time 1746155701.24 total 336.93 last 124.07) - open/close 45144 (time 1746155711.24 total 346.93 last 123.87) - open/close 46380 (time 1746155721.24 total 356.93 last 123.56) - open/close 47629 (time 1746155731.25 total 366.94 last 124.83) - open/close 48935 (time 1746155741.25 total 376.94 last 130.56) - open/close 50000 (time 1746155749.22 total 384.91 last 133.60) - open/close 51270 (time 1746155759.22 total 394.91 last 127.00) - open/close 52648 (time 1746155769.23 total 404.92 last 137.75) - open/close 53993 (time 1746155779.23 total 414.92 last 134.49) - open/close 55468 (time 1746155789.23 total 424.92 last 147.45) - open/close 57122 (time 1746155799.24 total 434.93 last 165.34) - open/close 58706 (time 1746155809.24 total 444.93 last 158.33) - open/close 60000 (time 1746155816.79 total 452.48 last 171.33) - open/close 61516 (time 1746155826.80 total 462.49 last 151.51) - open/close 62735 (time 1746155836.80 total 472.49 last 121.85) - open/close 64195 (time 1746155846.81 total 482.50 last 145.96) total: 64768 open/close in 487.38 seconds: 132.89 ops/second - /unlink 1840 (time 1746155863.76 total 10.01 last 183.89) - /unlink 3446 (time 1746155873.76 total 20.01 last 160.58) - /unlink 5133 (time 1746155883.76 total 30.01 last 168.68) - /unlink 7013 (time 1746155893.78 total 40.03 last 187.66) - /unlink 9062 (time 1746155903.79 total 50.03 last 204.83) - /unlink 10000 (time 1746155907.24 total 53.49 last 271.28) - /unlink 11958 (time 1746155917.25 total 63.49 last 195.69) - /unlink 13584 (time 1746155927.25 total 73.50 last 162.57) - /unlink 15228 (time 1746155937.25 total 83.50 last 164.37) - /unlink 16903 (time 1746155947.25 total 93.50 last 167.47) - /unlink 18549 (time 1746155957.26 total 103.50 last 164.53) - /unlink 20000 (time 1746155965.86 total 112.11 last 168.62) - /unlink 21623 (time 1746155975.87 total 122.11 last 162.25) - /unlink 23265 (time 1746155985.87 total 132.11 last 164.20) - /unlink 25041 (time 1746155995.87 total 142.12 last 177.53) - /unlink 26756 (time 1746156005.87 total 152.12 last 171.50) - /unlink 28489 (time 1746156015.87 total 162.12 last 173.26) - /unlink 30000 (time 1746156024.88 total 171.13 last 167.74) - /unlink 31821 (time 1746156034.89 total 181.13 last 182.01) - /unlink 33846 (time 1746156044.89 total 191.13 last 202.50) - /unlink 35915 (time 1746156054.89 total 201.14 last 206.74) - /unlink 38101 (time 1746156064.90 total 211.14 last 218.54) - /unlink 40000 (time 1746156073.86 total 220.11 last 211.78) - /unlink 42221 (time 1746156083.86 total 230.11 last 222.08) - /unlink 44416 (time 1746156093.87 total 240.11 last 219.43) - /unlink 47125 (time 1746156103.87 total 250.12 last 270.79) - /unlink 49219 (time 1746156113.87 total 260.12 last 209.35) - /unlink 50000 (time 1746156117.63 total 263.87 last 208.10) - /unlink 52756 (time 1746156127.63 total 273.87 last 275.53) - /unlink 55460 (time 1746156137.63 total 283.88 last 270.36) - /unlink 58075 (time 1746156147.63 total 293.88 last 261.44) - /unlink 60000 (time 1746156156.69 total 302.94 last 212.49) - /unlink 62075 (time 1746156166.70 total 312.95 last 207.31) - /unlink 64411 (time 1746156176.70 total 322.95 last 233.58) total: 64768 /unlink in 324.72 seconds: 199.46 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1251 (time 1746156206.67 total 10.01 last 125.01) - open/close 2616 (time 1746156216.67 total 20.01 last 136.45) - open/close 4012 (time 1746156226.68 total 30.02 last 139.47) - open/close 5330 (time 1746156236.69 total 40.02 last 131.75) - open/close 6652 (time 1746156246.69 total 50.03 last 132.15) - open/close 7899 (time 1746156256.69 total 60.03 last 124.69) - open/close 9239 (time 1746156266.70 total 70.03 last 133.92) - open/close 10000 (time 1746156272.47 total 75.80 last 131.90) - open/close 11224 (time 1746156282.47 total 85.81 last 122.33) - open/close 12517 (time 1746156292.47 total 95.81 last 129.27) - open/close 13841 (time 1746156302.48 total 105.82 last 132.29) - open/close 15154 (time 1746156312.49 total 115.82 last 131.26) - open/close 16476 (time 1746156322.49 total 125.83 last 132.16) - open/close 17719 (time 1746156332.49 total 135.83 last 124.27) - open/close 19064 (time 1746156342.49 total 145.83 last 134.49) - open/close 20000 (time 1746156350.03 total 153.37 last 124.11) - open/close 21251 (time 1746156360.04 total 163.38 last 125.01) - open/close 22639 (time 1746156370.04 total 173.38 last 138.77) - open/close 23898 (time 1746156380.04 total 183.38 last 125.86) - open/close 25106 (time 1746156390.05 total 193.38 last 120.77) - open/close 26304 (time 1746156400.05 total 203.39 last 119.80) - open/close 27438 (time 1746156410.05 total 213.39 last 113.33) - open/close 28616 (time 1746156420.06 total 223.39 last 117.77) - open/close 29877 (time 1746156430.06 total 233.39 last 126.09) - open/close 31189 (time 1746156440.06 total 243.40 last 131.13) - open/close 32355 (time 1746156450.07 total 253.41 last 116.48) - open/close 33691 (time 1746156460.08 total 263.41 last 133.55) - open/close 34902 (time 1746156470.08 total 273.42 last 121.01) - open/close 36144 (time 1746156480.09 total 283.43 last 124.07) - open/close 37408 (time 1746156490.10 total 293.44 last 126.31) - open/close 38762 (time 1746156500.11 total 303.44 last 135.33) - open/close 40000 (time 1746156510.06 total 313.40 last 124.35) - open/close 41341 (time 1746156520.07 total 323.40 last 134.02) - open/close 42574 (time 1746156530.07 total 333.41 last 123.25) - open/close 43710 (time 1746156540.07 total 343.41 last 113.57) - open/close 44974 (time 1746156550.08 total 353.42 last 126.34) - open/close 46134 (time 1746156560.09 total 363.43 last 115.89) - open/close 47334 (time 1746156570.09 total 373.43 last 119.99) - open/close 48658 (time 1746156580.09 total 383.43 last 132.37) - open/close 50000 (time 1746156589.49 total 392.82 last 142.84) - open/close 51603 (time 1746156599.49 total 402.83 last 160.26) - open/close 53477 (time 1746156609.49 total 412.83 last 187.33) - open/close 55123 (time 1746156619.49 total 422.83 last 164.57) - open/close 56661 (time 1746156629.49 total 432.83 last 153.80) - open/close 58439 (time 1746156639.50 total 442.84 last 177.71) - open/close 59903 (time 1746156649.51 total 452.84 last 146.28) - open/close 61438 (time 1746156659.51 total 462.85 last 153.48) - open/close 62892 (time 1746156669.51 total 472.85 last 145.38) - open/close 64313 (time 1746156679.51 total 482.85 last 142.04) total: 64768 open/close in 485.87 seconds: 133.30 ops/second - /unlink 2571 (time 1746156693.85 total 10.00 last 257.04) - /unlink 5581 (time 1746156703.85 total 20.01 last 300.91) - /unlink 8900 (time 1746156713.85 total 30.01 last 331.84) - /unlink 10000 (time 1746156718.42 total 34.57 last 240.97) - /unlink 12683 (time 1746156728.42 total 44.57 last 268.27) - /unlink 15202 (time 1746156738.42 total 54.57 last 251.89) - /unlink 17766 (time 1746156748.43 total 64.58 last 256.22) - /unlink 20000 (time 1746156757.52 total 73.68 last 245.61) - /unlink 22329 (time 1746156767.53 total 83.68 last 232.74) - /unlink 25021 (time 1746156777.53 total 93.68 last 269.17) - /unlink 28554 (time 1746156787.53 total 103.69 last 353.22) - /unlink 30000 (time 1746156791.22 total 107.37 last 392.41) - /unlink 33594 (time 1746156801.22 total 117.38 last 359.22) - /unlink 37820 (time 1746156811.22 total 127.38 last 422.51) - /unlink 40000 (time 1746156815.48 total 131.63 last 512.34) - /unlink 43657 (time 1746156825.48 total 141.63 last 365.69) - /unlink 48053 (time 1746156835.48 total 151.63 last 439.58) - /unlink 50000 (time 1746156840.30 total 156.46 last 403.45) - /unlink 53677 (time 1746156850.31 total 166.46 last 367.65) - /unlink 57806 (time 1746156860.31 total 176.46 last 412.89) - /unlink 60000 (time 1746156865.00 total 181.15 last 467.54) - /unlink 63600 (time 1746156875.00 total 191.16 last 359.83) total: 64768 /unlink in 194.89 seconds: 332.32 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1489 (time 1746156906.65 total 10.00 last 148.87) - open/close 3334 (time 1746156916.65 total 20.00 last 184.45) - open/close 4990 (time 1746156926.66 total 30.01 last 165.51) - open/close 6977 (time 1746156936.66 total 40.01 last 198.62) - open/close 8554 (time 1746156946.66 total 50.02 last 157.68) - open/close 10000 (time 1746156955.69 total 59.04 last 160.27) - open/close 11766 (time 1746156965.69 total 69.04 last 176.51) - open/close 13739 (time 1746156975.69 total 79.05 last 197.23) - open/close 16118 (time 1746156985.70 total 89.05 last 237.88) - open/close 17867 (time 1746156995.70 total 99.05 last 174.88) - open/close 19264 (time 1746157005.72 total 109.07 last 139.40) - open/close 20000 (time 1746157010.88 total 114.23 last 142.54) - open/close 21741 (time 1746157020.88 total 124.23 last 174.06) - open/close 23124 (time 1746157030.89 total 134.24 last 138.26) - open/close 24553 (time 1746157040.89 total 144.24 last 142.89) - open/close 25741 (time 1746157050.89 total 154.24 last 118.79) - open/close 27141 (time 1746157060.89 total 164.24 last 139.97) - open/close 28572 (time 1746157070.90 total 174.25 last 143.02) - open/close 30000 (time 1746157079.63 total 182.98 last 163.56) - open/close 31385 (time 1746157089.63 total 192.99 last 138.40) - open/close 32756 (time 1746157099.64 total 202.99 last 137.05) - open/close 34321 (time 1746157109.64 total 212.99 last 156.44) - open/close 35688 (time 1746157119.64 total 223.00 last 136.65) - open/close 37135 (time 1746157129.65 total 233.00 last 144.68) - open/close 38536 (time 1746157139.65 total 243.00 last 140.04) - open/close 40000 (time 1746157148.51 total 251.86 last 165.26) - open/close 42036 (time 1746157158.51 total 261.86 last 203.55) - open/close 43580 (time 1746157168.52 total 271.87 last 154.25) - open/close 45326 (time 1746157178.52 total 281.87 last 174.59) - open/close 47081 (time 1746157188.52 total 291.88 last 175.46) - open/close 48845 (time 1746157198.53 total 301.88 last 176.31) - open/close 50000 (time 1746157205.70 total 309.05 last 161.04) - open/close 51557 (time 1746157215.71 total 319.06 last 155.63) - open/close 53038 (time 1746157225.71 total 329.06 last 148.03) - open/close 54492 (time 1746157235.72 total 339.07 last 145.30) - open/close 55862 (time 1746157245.72 total 349.07 last 136.94) - open/close 57171 (time 1746157255.72 total 359.08 last 130.87) - open/close 58490 (time 1746157265.73 total 369.08 last 131.81) - open/close 59919 (time 1746157275.74 total 379.10 last 142.70) - open/close 61325 (time 1746157285.75 total 389.10 last 140.57) - open/close 62809 (time 1746157295.75 total 399.10 last 148.38) - open/close 64101 (time 1746157305.75 total 409.11 last 129.13) total: 64768 open/close in 414.43 seconds: 156.28 ops/second - /unlink 2107 (time 1746157322.69 total 10.01 last 210.51) - /unlink 4337 (time 1746157332.69 total 20.01 last 222.93) - /unlink 6773 (time 1746157342.69 total 30.01 last 243.60) - /unlink 8639 (time 1746157352.70 total 40.02 last 186.50) - /unlink 10000 (time 1746157360.57 total 47.89 last 172.87) - /unlink 11836 (time 1746157370.57 total 57.89 last 183.58) - /unlink 13577 (time 1746157380.58 total 67.90 last 173.96) - /unlink 15415 (time 1746157390.58 total 77.91 last 183.69) - /unlink 17421 (time 1746157400.59 total 87.91 last 200.44) - /unlink 19293 (time 1746157410.60 total 97.92 last 187.12) - /unlink 20000 (time 1746157413.62 total 100.95 last 233.54) - /unlink 21970 (time 1746157423.63 total 110.95 last 196.88) - /unlink 24798 (time 1746157433.63 total 120.95 last 282.71) - /unlink 27661 (time 1746157443.63 total 130.96 last 286.27) - /unlink 30000 (time 1746157452.05 total 139.37 last 277.85) - /unlink 32273 (time 1746157462.06 total 149.38 last 227.16) - /unlink 34634 (time 1746157472.06 total 159.38 last 236.09) - /unlink 37047 (time 1746157482.06 total 169.38 last 241.24) - /unlink 39250 (time 1746157492.06 total 179.38 last 220.27) - /unlink 40000 (time 1746157495.88 total 183.20 last 196.56) - /unlink 42378 (time 1746157505.88 total 193.20 last 237.75) - /unlink 44876 (time 1746157515.88 total 203.20 last 249.78) - /unlink 47233 (time 1746157525.88 total 213.21 last 235.61) - /unlink 50000 (time 1746157535.04 total 222.37 last 302.11) - /unlink 52999 (time 1746157545.04 total 232.37 last 299.86) - /unlink 55362 (time 1746157555.05 total 242.37 last 236.26) - /unlink 57541 (time 1746157565.05 total 252.37 last 217.83) - /unlink 59889 (time 1746157575.05 total 262.37 last 234.79) - /unlink 63275 (time 1746157585.05 total 272.37 last 338.58) total: 64768 /unlink in 276.55 seconds: 234.20 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1767 (time 1746157618.48 total 10.00 last 176.68) - open/close 3966 (time 1746157628.48 total 20.00 last 219.85) - open/close 6710 (time 1746157638.49 total 30.01 last 274.29) - open/close 8338 (time 1746157648.49 total 40.01 last 162.80) - open/close 9949 (time 1746157658.49 total 50.01 last 161.00) - open/close 12100 (time 1746157668.49 total 60.01 last 215.09) - open/close 13964 (time 1746157678.49 total 70.02 last 186.38) - open/close 15439 (time 1746157688.51 total 80.03 last 147.32) - open/close 16774 (time 1746157698.51 total 90.03 last 133.48) - open/close 18024 (time 1746157708.52 total 100.04 last 124.89) - open/close 19261 (time 1746157718.52 total 110.04 last 123.66) - open/close 20000 (time 1746157725.13 total 116.65 last 111.79) - open/close 21206 (time 1746157735.13 total 126.65 last 120.57) - open/close 22337 (time 1746157745.14 total 136.66 last 113.06) - open/close 23202 (time 1746157755.14 total 146.66 last 86.44) - open/close 24046 (time 1746157765.15 total 156.68 last 84.31) - open/close 24896 (time 1746157775.16 total 166.68 last 84.98) - open/close 26206 (time 1746157785.16 total 176.68 last 130.99) - open/close 28036 (time 1746157795.16 total 186.68 last 182.93) - open/close 29720 (time 1746157805.16 total 196.69 last 168.34) - open/close 31221 (time 1746157815.17 total 206.69 last 150.08) - open/close 32927 (time 1746157825.17 total 216.69 last 170.59) - open/close 34860 (time 1746157835.17 total 226.69 last 193.26) - open/close 36163 (time 1746157845.18 total 236.70 last 130.15) - open/close 37296 (time 1746157855.19 total 246.71 last 113.22) - open/close 38803 (time 1746157865.19 total 256.71 last 150.62) - open/close 40000 (time 1746157873.46 total 264.98 last 144.77) - open/close 41376 (time 1746157883.46 total 274.98 last 137.58) - open/close 42579 (time 1746157893.46 total 284.98 last 120.28) - open/close 43850 (time 1746157903.47 total 294.99 last 127.03) - open/close 45380 (time 1746157913.47 total 304.99 last 152.96) - open/close 46625 (time 1746157923.47 total 314.99 last 124.48) - open/close 47810 (time 1746157933.47 total 324.99 last 118.50) - open/close 48937 (time 1746157943.48 total 335.00 last 112.61) - open/close 50000 (time 1746157952.57 total 344.09 last 116.99) - open/close 51321 (time 1746157962.57 total 354.09 last 132.07) - open/close 52524 (time 1746157972.58 total 364.10 last 120.18) - open/close 53685 (time 1746157982.59 total 374.11 last 116.04) - open/close 54815 (time 1746157992.60 total 384.12 last 112.86) - open/close 55637 (time 1746158002.61 total 394.13 last 82.11) - open/close 56431 (time 1746158012.62 total 404.14 last 79.28) - open/close 57196 (time 1746158022.63 total 414.15 last 76.49) - open/close 58105 (time 1746158032.63 total 424.15 last 90.88) - open/close 59036 (time 1746158042.65 total 434.17 last 92.93) - open/close 60000 (time 1746158052.07 total 443.59 last 102.34) - open/close 61098 (time 1746158062.08 total 453.60 last 109.68) - open/close 61999 (time 1746158072.09 total 463.61 last 90.02) - open/close 62968 (time 1746158082.09 total 473.61 last 96.86) - open/close 64005 (time 1746158092.10 total 483.62 last 103.63) total: 64768 open/close in 490.76 seconds: 131.97 ops/second - /unlink 1706 (time 1746158111.38 total 10.01 last 170.49) - /unlink 3783 (time 1746158121.38 total 20.01 last 207.68) - /unlink 5640 (time 1746158131.38 total 30.01 last 185.63) - /unlink 7342 (time 1746158141.41 total 40.04 last 169.69) - /unlink 9348 (time 1746158151.42 total 50.05 last 200.48) - /unlink 10000 (time 1746158155.08 total 53.71 last 178.20) - /unlink 11634 (time 1746158165.09 total 63.72 last 163.19) - /unlink 13265 (time 1746158175.10 total 73.73 last 162.96) - /unlink 14987 (time 1746158185.10 total 83.73 last 172.19) - /unlink 16372 (time 1746158195.10 total 93.73 last 138.50) - /unlink 17946 (time 1746158205.10 total 103.73 last 157.37) - /unlink 19620 (time 1746158215.11 total 113.73 last 167.36) - /unlink 20000 (time 1746158217.20 total 115.83 last 181.22) - /unlink 21721 (time 1746158227.20 total 125.83 last 172.09) - /unlink 23632 (time 1746158237.21 total 135.84 last 190.99) - /unlink 25843 (time 1746158247.21 total 145.84 last 221.06) - /unlink 28173 (time 1746158257.21 total 155.84 last 232.91) - /unlink 30000 (time 1746158265.69 total 164.32 last 215.43) - /unlink 32178 (time 1746158275.70 total 174.32 last 217.78) - /unlink 34268 (time 1746158285.70 total 184.32 last 208.98) - /unlink 36788 (time 1746158295.70 total 194.33 last 251.97) - /unlink 39436 (time 1746158305.70 total 204.33 last 264.71) - /unlink 40000 (time 1746158308.29 total 206.92 last 217.76) - /unlink 42391 (time 1746158318.30 total 216.92 last 239.02) - /unlink 44745 (time 1746158328.30 total 226.92 last 235.36) - /unlink 47131 (time 1746158338.30 total 236.93 last 238.52) - /unlink 49666 (time 1746158348.30 total 246.93 last 253.46) - /unlink 52118 (time 1746158358.30 total 256.93 last 245.15) - /unlink 55054 (time 1746158368.31 total 266.94 last 293.43) - /unlink 57240 (time 1746158378.31 total 276.94 last 218.58) - /unlink 59475 (time 1746158388.31 total 286.94 last 223.46) - /unlink 61810 (time 1746158398.31 total 296.94 last 233.49) - /unlink 64391 (time 1746158408.32 total 306.94 last 258.01) total: 64768 /unlink in 308.26 seconds: 210.11 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg203-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server stop mds service on oleg203-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server unloading modules via unload_modules_local on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 PASS 106 (3991s) Stopping clients: oleg203-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg203-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 oleg203-server: oleg203-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing load_modules_local oleg203-server: Loading modules from /home/green/git/lustre-release/lustre oleg203-server: detected 4 online CPUs by sysfs oleg203-server: MODOPTS_LIBCFS= oleg203-server: Force libcfs to create 2 CPU partitions oleg203-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg203-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4103 sec ========== 00:01:13 (1746158473) === conf-sanity: start cleanup 00:01:14 (1746158474) === === conf-sanity: finish cleanup 00:01:15 (1746158475) ===