-----============= acceptance-small: conf-sanity ============----- Thu Dec 19 23:58:59 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg122-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 110 Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg122-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg122-server oleg122-server: oleg122-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing load_modules_local oleg122-server: Loading modules from /home/green/git/lustre-release/lustre oleg122-server: detected 4 online CPUs by sysfs oleg122-server: MODOPTS_LIBCFS= oleg122-server: Force libcfs to create 2 CPU partitions oleg122-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg122-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg122-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg122-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server stop mds service on oleg122-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 00:00:14 (1734670814) Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f) oleg122-server: oleg122-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing load_modules_local oleg122-server: Loading modules from /home/green/git/lustre-release/lustre oleg122-server: detected 4 online CPUs by sysfs oleg122-server: MODOPTS_LIBCFS= oleg122-server: Force libcfs to create 2 CPU partitions oleg122-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg122-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg122-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg122-client.virtnet: -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1396 (time 1734670867.45 total 10.01 last 139.50) - open/close 2902 (time 1734670877.46 total 20.01 last 150.51) - open/close 4274 (time 1734670887.47 total 30.02 last 137.07) - open/close 6009 (time 1734670897.47 total 40.03 last 173.42) - open/close 7532 (time 1734670907.47 total 50.03 last 152.28) - open/close 8837 (time 1734670917.48 total 60.03 last 130.45) - open/close 10000 (time 1734670925.79 total 68.34 last 139.96) - open/close 11403 (time 1734670935.79 total 78.35 last 140.23) - open/close 12749 (time 1734670945.80 total 88.35 last 134.53) - open/close 14088 (time 1734670955.80 total 98.35 last 133.89) - open/close 15500 (time 1734670965.80 total 108.36 last 141.13) - open/close 16998 (time 1734670975.80 total 118.36 last 149.79) - open/close 18306 (time 1734670985.80 total 128.36 last 130.79) - open/close 19764 (time 1734670995.81 total 138.36 last 145.73) - open/close 21220 (time 1734671005.81 total 148.36 last 145.59) - open/close 22639 (time 1734671015.82 total 158.37 last 141.79) - open/close 24381 (time 1734671025.82 total 168.37 last 174.15) - open/close 26153 (time 1734671035.82 total 178.38 last 177.19) - open/close 27816 (time 1734671045.82 total 188.38 last 166.24) - open/close 29282 (time 1734671055.83 total 198.38 last 146.57) - open/close 30000 (time 1734671061.00 total 203.55 last 138.86) - open/close 31380 (time 1734671071.00 total 213.56 last 137.92) - open/close 32972 (time 1734671081.00 total 223.56 last 159.17) - open/close 34556 (time 1734671091.01 total 233.57 last 158.28) - open/close 35970 (time 1734671101.02 total 243.57 last 141.32) - open/close 37515 (time 1734671111.02 total 253.57 last 154.50) - open/close 38898 (time 1734671121.02 total 263.57 last 138.30) - open/close 40000 (time 1734671129.08 total 271.63 last 136.77) - open/close 41353 (time 1734671139.08 total 281.63 last 135.28) - open/close 42624 (time 1734671149.08 total 291.63 last 127.09) - open/close 43950 (time 1734671159.08 total 301.64 last 132.55) - open/close 45477 (time 1734671169.08 total 311.64 last 152.66) - open/close 47025 (time 1734671179.09 total 321.64 last 154.73) - open/close 48376 (time 1734671189.09 total 331.65 last 135.04) - open/close 49768 (time 1734671199.09 total 341.65 last 139.18) - open/close 51078 (time 1734671209.10 total 351.65 last 130.99) - open/close 52777 (time 1734671219.10 total 361.65 last 169.89) - open/close 54504 (time 1734671229.10 total 371.66 last 172.60) - open/close 56113 (time 1734671239.11 total 381.66 last 160.81) - open/close 57402 (time 1734671249.11 total 391.67 last 128.86) - open/close 58746 (time 1734671259.11 total 401.67 last 134.35) - open/close 60000 (time 1734671268.62 total 411.17 last 131.97) - open/close 61187 (time 1734671278.62 total 421.17 last 118.68) - open/close 62455 (time 1734671288.62 total 431.18 last 126.73) - open/close 63752 (time 1734671298.63 total 441.18 last 129.63) total: 64768 open/close in 449.58 seconds: 144.06 ops/second - /unlink 2021 (time 1734671319.00 total 10.00 last 202.06) - /unlink 4495 (time 1734671329.01 total 20.01 last 247.29) - /unlink 6364 (time 1734671339.01 total 30.01 last 186.78) - /unlink 8002 (time 1734671349.01 total 40.01 last 163.77) - /unlink 9750 (time 1734671359.02 total 50.02 last 174.75) - /unlink 11513 (time 1734671369.02 total 60.02 last 176.29) - /unlink 13265 (time 1734671379.02 total 70.02 last 175.15) - /unlink 15104 (time 1734671389.02 total 80.02 last 183.90) - /unlink 17031 (time 1734671399.02 total 90.02 last 192.67) - /unlink 19174 (time 1734671409.02 total 100.02 last 214.28) - /unlink 20000 (time 1734671412.94 total 103.94 last 210.91) - /unlink 22231 (time 1734671422.95 total 113.95 last 222.93) - /unlink 24257 (time 1734671432.95 total 123.95 last 202.51) - /unlink 26324 (time 1734671442.96 total 133.96 last 206.61) - /unlink 28325 (time 1734671452.96 total 143.96 last 200.07) - /unlink 30000 (time 1734671461.43 total 152.43 last 197.76) - /unlink 32222 (time 1734671471.43 total 162.43 last 222.19) - /unlink 34889 (time 1734671481.43 total 172.43 last 266.64) - /unlink 37686 (time 1734671491.44 total 182.44 last 279.52) - /unlink 40000 (time 1734671499.92 total 190.92 last 272.88) - /unlink 42652 (time 1734671509.92 total 200.92 last 265.18) - /unlink 45827 (time 1734671519.92 total 210.92 last 317.48) - /unlink 48371 (time 1734671529.92 total 220.92 last 254.40) - /unlink 50000 (time 1734671536.75 total 227.75 last 238.27) - /unlink 53101 (time 1734671546.75 total 237.75 last 310.09) - /unlink 55786 (time 1734671556.76 total 247.75 last 268.48) - /unlink 58513 (time 1734671566.76 total 257.76 last 272.61) - /unlink 60000 (time 1734671573.45 total 264.45 last 222.10) - /unlink 62147 (time 1734671583.46 total 274.46 last 214.64) - /unlink 64529 (time 1734671593.46 total 284.46 last 238.16) total: 64768 /unlink in 285.50 seconds: 226.86 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1150 (time 1734671624.56 total 10.00 last 114.97) - open/close 2343 (time 1734671634.56 total 20.01 last 119.26) - open/close 3417 (time 1734671644.57 total 30.01 last 107.35) - open/close 4575 (time 1734671654.57 total 40.01 last 115.78) - open/close 5775 (time 1734671664.57 total 50.02 last 119.96) - open/close 6905 (time 1734671674.57 total 60.02 last 112.98) - open/close 8031 (time 1734671684.58 total 70.02 last 112.51) - open/close 9153 (time 1734671694.59 total 80.03 last 112.10) - open/close 10000 (time 1734671702.46 total 87.90 last 107.62) - open/close 11272 (time 1734671712.46 total 97.91 last 127.17) - open/close 12778 (time 1734671722.46 total 107.91 last 150.55) - open/close 14232 (time 1734671732.47 total 117.91 last 145.36) - open/close 15696 (time 1734671742.47 total 127.92 last 146.34) - open/close 17260 (time 1734671752.48 total 137.92 last 156.29) - open/close 18889 (time 1734671762.48 total 147.92 last 162.86) - open/close 20000 (time 1734671771.19 total 156.63 last 127.59) - open/close 21385 (time 1734671781.19 total 166.63 last 138.47) - open/close 22724 (time 1734671791.20 total 176.64 last 133.83) - open/close 24293 (time 1734671801.20 total 186.65 last 156.82) - open/close 25942 (time 1734671811.20 total 196.65 last 164.86) - open/close 27614 (time 1734671821.20 total 206.65 last 167.20) - open/close 29492 (time 1734671831.21 total 216.65 last 187.74) - open/close 30000 (time 1734671833.24 total 218.68 last 250.18) - open/close 32046 (time 1734671843.24 total 228.68 last 204.54) - open/close 34137 (time 1734671853.24 total 238.69 last 209.05) - open/close 35733 (time 1734671863.25 total 248.69 last 159.57) - open/close 37572 (time 1734671873.25 total 258.69 last 183.85) - open/close 39987 (time 1734671883.25 total 268.69 last 241.48) - open/close 41908 (time 1734671893.25 total 278.69 last 192.09) - open/close 43511 (time 1734671903.25 total 288.70 last 160.26) - open/close 44980 (time 1734671913.25 total 298.70 last 146.88) - open/close 46501 (time 1734671923.25 total 308.70 last 152.08) - open/close 47968 (time 1734671933.26 total 318.70 last 146.64) - open/close 49507 (time 1734671943.26 total 328.71 last 153.85) - open/close 50000 (time 1734671947.07 total 332.52 last 129.42) - open/close 51622 (time 1734671957.08 total 342.52 last 162.13) - open/close 53483 (time 1734671967.08 total 352.52 last 186.10) - open/close 55022 (time 1734671977.08 total 362.52 last 153.88) - open/close 56268 (time 1734671987.08 total 372.52 last 124.59) - open/close 57953 (time 1734671997.08 total 382.53 last 168.43) - open/close 59718 (time 1734672007.09 total 392.53 last 176.41) - open/close 60000 (time 1734672009.16 total 394.60 last 136.36) - open/close 61732 (time 1734672019.16 total 404.60 last 173.12) - open/close 63906 (time 1734672029.16 total 414.61 last 217.35) total: 64768 open/close in 419.42 seconds: 154.42 ops/second - /unlink 2808 (time 1734672045.49 total 10.00 last 280.80) - /unlink 6024 (time 1734672055.49 total 20.00 last 321.52) - /unlink 8890 (time 1734672065.50 total 30.01 last 286.27) - /unlink 10000 (time 1734672069.12 total 33.63 last 307.20) - /unlink 12764 (time 1734672079.12 total 43.63 last 276.33) - /unlink 15027 (time 1734672089.12 total 53.63 last 226.20) - /unlink 17335 (time 1734672099.13 total 63.64 last 230.73) - /unlink 19747 (time 1734672109.13 total 73.64 last 241.15) - /unlink 22071 (time 1734672119.13 total 83.64 last 232.33) - /unlink 24279 (time 1734672129.13 total 93.64 last 220.79) - /unlink 26669 (time 1734672139.13 total 103.64 last 238.96) - /unlink 29090 (time 1734672149.14 total 113.65 last 242.07) - /unlink 30000 (time 1734672153.33 total 117.84 last 217.10) - /unlink 32462 (time 1734672163.33 total 127.84 last 246.18) - /unlink 34584 (time 1734672173.33 total 137.84 last 212.14) - /unlink 36902 (time 1734672183.33 total 147.84 last 231.76) - /unlink 39507 (time 1734672193.34 total 157.85 last 260.36) - /unlink 42093 (time 1734672203.34 total 167.85 last 258.52) - /unlink 44422 (time 1734672213.35 total 177.86 last 232.80) - /unlink 46818 (time 1734672223.35 total 187.86 last 239.52) - /unlink 49105 (time 1734672233.35 total 197.86 last 228.69) - /unlink 50000 (time 1734672236.87 total 201.38 last 254.21) - /unlink 52548 (time 1734672246.87 total 211.38 last 254.75) - /unlink 55584 (time 1734672256.87 total 221.38 last 303.56) - /unlink 58573 (time 1734672266.88 total 231.39 last 298.82) - /unlink 60000 (time 1734672272.59 total 237.10 last 249.55) - /unlink 62504 (time 1734672282.60 total 247.11 last 250.34) total: 64768 /unlink in 256.92 seconds: 252.10 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1284 (time 1734672320.35 total 10.00 last 128.34) - open/close 2632 (time 1734672330.35 total 20.01 last 134.78) - open/close 3964 (time 1734672340.35 total 30.01 last 133.13) - open/close 5294 (time 1734672350.36 total 40.02 last 132.94) - open/close 6663 (time 1734672360.36 total 50.02 last 136.87) - open/close 8098 (time 1734672370.36 total 60.02 last 143.48) - open/close 9567 (time 1734672380.37 total 70.02 last 146.86) - open/close 10000 (time 1734672383.07 total 72.72 last 160.30) - open/close 11292 (time 1734672393.07 total 82.72 last 129.19) - open/close 12522 (time 1734672403.07 total 92.73 last 122.97) - open/close 13740 (time 1734672413.08 total 102.73 last 121.74) - open/close 15081 (time 1734672423.08 total 112.74 last 134.00) - open/close 16348 (time 1734672433.09 total 122.74 last 126.66) - open/close 17637 (time 1734672443.09 total 132.74 last 128.89) - open/close 18810 (time 1734672453.09 total 142.75 last 117.26) - open/close 19938 (time 1734672463.09 total 152.75 last 112.79) - open/close 21157 (time 1734672473.10 total 162.75 last 121.85) - open/close 22323 (time 1734672483.10 total 172.76 last 116.53) - open/close 23439 (time 1734672493.11 total 182.76 last 111.55) - open/close 24582 (time 1734672503.11 total 192.77 last 114.25) - open/close 25680 (time 1734672513.12 total 202.77 last 109.76) - open/close 26733 (time 1734672523.12 total 212.78 last 105.22) - open/close 27766 (time 1734672533.12 total 222.78 last 103.28) - open/close 29082 (time 1734672543.13 total 232.79 last 131.52) - open/close 30000 (time 1734672547.91 total 237.57 last 191.95) - open/close 32237 (time 1734672557.91 total 247.57 last 223.68) - open/close 34068 (time 1734672567.92 total 257.57 last 183.03) - open/close 35425 (time 1734672577.92 total 267.58 last 135.60) - open/close 36779 (time 1734672587.93 total 277.58 last 135.38) - open/close 38265 (time 1734672597.93 total 287.58 last 148.57) - open/close 39608 (time 1734672607.94 total 297.60 last 134.16) - open/close 40000 (time 1734672610.13 total 299.79 last 178.88) - open/close 41842 (time 1734672620.14 total 309.79 last 184.09) - open/close 43685 (time 1734672630.14 total 319.80 last 184.18) - open/close 45258 (time 1734672640.15 total 329.81 last 157.20) - open/close 46740 (time 1734672650.15 total 339.81 last 148.13) - open/close 48789 (time 1734672660.16 total 349.82 last 204.78) - open/close 50000 (time 1734672667.84 total 357.50 last 157.64) - open/close 51878 (time 1734672677.85 total 367.50 last 187.72) - open/close 54440 (time 1734672687.85 total 377.50 last 256.19) - open/close 56742 (time 1734672697.85 total 387.50 last 230.15) - open/close 58827 (time 1734672707.85 total 397.51 last 208.48) - open/close 60000 (time 1734672714.32 total 403.98 last 181.29) - open/close 61645 (time 1734672724.32 total 413.98 last 164.49) - open/close 63134 (time 1734672734.32 total 423.98 last 148.83) - open/close 64662 (time 1734672744.33 total 433.99 last 152.70) total: 64768 open/close in 434.79 seconds: 148.96 ops/second - /unlink 2561 (time 1734672756.72 total 10.00 last 256.03) - /unlink 5350 (time 1734672766.72 total 20.00 last 278.90) - /unlink 9339 (time 1734672776.72 total 30.00 last 398.83) - /unlink 13915 (time 1734672786.72 total 40.01 last 457.54) - /unlink 17930 (time 1734672796.72 total 50.01 last 401.47) - /unlink 20000 (time 1734672802.02 total 55.30 last 390.87) - /unlink 23321 (time 1734672812.02 total 65.30 last 332.03) - /unlink 25644 (time 1734672822.02 total 75.31 last 232.26) - /unlink 28284 (time 1734672832.02 total 85.31 last 264.00) - /unlink 30000 (time 1734672838.57 total 91.86 last 261.97) - /unlink 32866 (time 1734672848.57 total 101.86 last 286.59) - /unlink 36692 (time 1734672858.57 total 111.86 last 382.55) - /unlink 40000 (time 1734672865.70 total 118.99 last 463.85) - /unlink 44138 (time 1734672875.70 total 128.99 last 413.80) - /unlink 46907 (time 1734672885.71 total 138.99 last 276.88) - /unlink 49444 (time 1734672895.71 total 148.99 last 253.63) - /unlink 50000 (time 1734672898.29 total 151.58 last 215.12) - /unlink 52186 (time 1734672908.30 total 161.58 last 218.54) - /unlink 55024 (time 1734672918.30 total 171.58 last 283.73) - /unlink 57551 (time 1734672928.30 total 181.59 last 252.61) - /unlink 59815 (time 1734672938.31 total 191.59 last 226.28) - /unlink 61766 (time 1734672948.31 total 201.59 last 195.04) - /unlink 63820 (time 1734672958.32 total 211.60 last 205.23) total: 64768 /unlink in 216.35 seconds: 299.36 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1426 (time 1734672991.58 total 10.00 last 142.55) - open/close 2881 (time 1734673001.59 total 20.01 last 145.45) - open/close 4676 (time 1734673011.59 total 30.01 last 179.48) - open/close 6435 (time 1734673021.59 total 40.01 last 175.84) - open/close 8245 (time 1734673031.59 total 50.01 last 180.96) - open/close 9788 (time 1734673041.60 total 60.02 last 154.21) - open/close 11423 (time 1734673051.60 total 70.02 last 163.48) - open/close 13276 (time 1734673061.61 total 80.03 last 185.17) - open/close 14838 (time 1734673071.62 total 90.04 last 156.05) - open/close 16400 (time 1734673081.62 total 100.04 last 156.15) - open/close 18347 (time 1734673091.62 total 110.04 last 194.69) - open/close 20000 (time 1734673098.38 total 116.80 last 244.50) - open/close 21631 (time 1734673108.39 total 126.81 last 163.05) - open/close 23203 (time 1734673118.39 total 136.81 last 157.20) - open/close 24848 (time 1734673128.39 total 146.81 last 164.41) - open/close 26383 (time 1734673138.40 total 156.81 last 153.45) - open/close 28056 (time 1734673148.40 total 166.82 last 167.17) - open/close 29585 (time 1734673158.40 total 176.82 last 152.90) - open/close 30000 (time 1734673161.47 total 179.89 last 135.19) - open/close 31412 (time 1734673171.48 total 189.90 last 141.10) - open/close 32870 (time 1734673181.48 total 199.90 last 145.80) - open/close 34333 (time 1734673191.48 total 209.90 last 146.26) - open/close 35813 (time 1734673201.49 total 219.91 last 147.83) - open/close 37358 (time 1734673211.50 total 229.92 last 154.47) - open/close 38831 (time 1734673221.50 total 239.92 last 147.28) - open/close 40000 (time 1734673229.59 total 248.01 last 144.49) - open/close 41500 (time 1734673239.59 total 258.01 last 149.98) - open/close 42970 (time 1734673249.60 total 268.02 last 146.88) - open/close 44350 (time 1734673259.61 total 278.02 last 137.87) - open/close 45728 (time 1734673269.61 total 288.03 last 137.80) - open/close 47151 (time 1734673279.61 total 298.03 last 142.23) - open/close 48588 (time 1734673289.62 total 308.04 last 143.57) - open/close 49882 (time 1734673299.62 total 318.04 last 129.35) - open/close 51125 (time 1734673309.63 total 328.05 last 124.27) - open/close 52403 (time 1734673319.63 total 338.05 last 127.75) - open/close 53674 (time 1734673329.64 total 348.06 last 127.00) - open/close 55065 (time 1734673339.65 total 358.07 last 138.98) - open/close 56393 (time 1734673349.65 total 368.07 last 132.77) - open/close 57592 (time 1734673359.65 total 378.07 last 119.86) - open/close 58786 (time 1734673369.66 total 388.07 last 119.36) - open/close 60000 (time 1734673379.21 total 397.63 last 127.09) - open/close 61298 (time 1734673389.21 total 407.63 last 129.78) - open/close 62467 (time 1734673399.22 total 417.63 last 116.83) - open/close 63719 (time 1734673409.22 total 427.64 last 125.18) total: 64768 open/close in 435.76 seconds: 148.63 ops/second - /unlink 1963 (time 1734673428.95 total 10.00 last 196.25) - /unlink 4000 (time 1734673438.96 total 20.00 last 203.66) - /unlink 6641 (time 1734673448.96 total 30.01 last 264.05) - /unlink 9614 (time 1734673458.96 total 40.01 last 297.20) - /unlink 11930 (time 1734673468.97 total 50.01 last 231.52) - /unlink 14370 (time 1734673478.97 total 60.02 last 243.92) - /unlink 17021 (time 1734673488.97 total 70.02 last 265.03) - /unlink 19554 (time 1734673498.97 total 80.02 last 253.27) - /unlink 22001 (time 1734673508.97 total 90.02 last 244.67) - /unlink 24239 (time 1734673518.98 total 100.03 last 223.66) - /unlink 26647 (time 1734673528.98 total 110.03 last 240.77) - /unlink 29237 (time 1734673538.98 total 120.03 last 258.95) - /unlink 30000 (time 1734673541.95 total 123.00 last 256.67) - /unlink 33130 (time 1734673551.96 total 133.01 last 312.90) - /unlink 36479 (time 1734673561.96 total 143.01 last 334.85) - /unlink 38770 (time 1734673571.96 total 153.01 last 229.03) - /unlink 40000 (time 1734673576.67 total 157.72 last 261.34) - /unlink 42497 (time 1734673586.68 total 167.72 last 249.54) - /unlink 45524 (time 1734673596.68 total 177.72 last 302.66) - /unlink 48032 (time 1734673606.68 total 187.73 last 250.75) - /unlink 50000 (time 1734673614.07 total 195.11 last 266.43) - /unlink 52507 (time 1734673624.07 total 205.11 last 250.69) - /unlink 54983 (time 1734673634.07 total 215.12 last 247.56) - /unlink 57741 (time 1734673644.07 total 225.12 last 275.65) - /unlink 60000 (time 1734673651.92 total 232.97 last 287.95) - /unlink 63382 (time 1734673661.92 total 242.97 last 338.19) total: 64768 /unlink in 248.00 seconds: 261.16 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1507 (time 1734673697.70 total 10.00 last 150.67) - open/close 3201 (time 1734673707.70 total 20.01 last 169.33) - open/close 4700 (time 1734673717.70 total 30.01 last 149.87) - open/close 6578 (time 1734673727.70 total 40.01 last 187.79) - open/close 8211 (time 1734673737.71 total 50.01 last 163.27) - open/close 9557 (time 1734673747.71 total 60.01 last 134.59) - open/close 10000 (time 1734673751.07 total 63.38 last 131.62) - open/close 11529 (time 1734673761.07 total 73.38 last 152.89) - open/close 13012 (time 1734673771.08 total 83.38 last 148.21) - open/close 14600 (time 1734673781.08 total 93.38 last 158.78) - open/close 16155 (time 1734673791.08 total 103.39 last 155.44) - open/close 17919 (time 1734673801.09 total 113.39 last 176.33) - open/close 19573 (time 1734673811.09 total 123.40 last 165.32) - open/close 20000 (time 1734673813.94 total 126.24 last 150.00) - open/close 21434 (time 1734673823.94 total 136.24 last 143.39) - open/close 22867 (time 1734673833.95 total 146.25 last 143.23) - open/close 24310 (time 1734673843.95 total 156.25 last 144.30) - open/close 26225 (time 1734673853.95 total 166.25 last 191.43) - open/close 28683 (time 1734673863.95 total 176.25 last 245.80) - open/close 30000 (time 1734673869.33 total 181.63 last 244.98) - open/close 32813 (time 1734673879.33 total 191.63 last 281.18) - open/close 34834 (time 1734673889.33 total 201.64 last 202.05) - open/close 36711 (time 1734673899.33 total 211.64 last 187.67) - open/close 39409 (time 1734673909.34 total 221.64 last 269.67) - open/close 40000 (time 1734673912.37 total 224.67 last 195.14) - open/close 41951 (time 1734673922.37 total 234.68 last 195.00) - open/close 43672 (time 1734673932.38 total 244.68 last 172.02) - open/close 45057 (time 1734673942.38 total 254.68 last 138.50) - open/close 46504 (time 1734673952.39 total 264.69 last 144.58) - open/close 48864 (time 1734673962.40 total 274.70 last 235.72) - open/close 50000 (time 1734673968.97 total 281.28 last 172.78) - open/close 51996 (time 1734673978.98 total 291.28 last 199.52) - open/close 53685 (time 1734673988.98 total 301.28 last 168.85) - open/close 55625 (time 1734673998.98 total 311.29 last 193.96) - open/close 57953 (time 1734674008.98 total 321.29 last 232.79) - open/close 59943 (time 1734674018.99 total 331.29 last 198.95) - open/close 61585 (time 1734674028.99 total 341.29 last 164.17) - open/close 63201 (time 1734674038.99 total 351.29 last 161.56) total: 64768 open/close in 361.01 seconds: 179.41 ops/second - /unlink 3315 (time 1734674059.93 total 10.00 last 331.36) - /unlink 6514 (time 1734674069.94 total 20.01 last 319.83) - /unlink 9643 (time 1734674079.94 total 30.01 last 312.83) - /unlink 12990 (time 1734674089.94 total 40.01 last 334.66) - /unlink 17132 (time 1734674099.94 total 50.01 last 414.15) - /unlink 20000 (time 1734674107.54 total 57.62 last 377.17) - /unlink 24271 (time 1734674117.54 total 67.62 last 427.08) - /unlink 29893 (time 1734674127.55 total 77.62 last 562.18) - /unlink 35158 (time 1734674137.55 total 87.62 last 526.39) - /unlink 39670 (time 1734674147.55 total 97.62 last 451.17) - /unlink 44108 (time 1734674157.55 total 107.62 last 443.77) - /unlink 47863 (time 1734674167.55 total 117.62 last 375.39) - /unlink 50000 (time 1734674174.80 total 124.87 last 294.96) - /unlink 55464 (time 1734674184.80 total 134.87 last 546.39) - /unlink 60000 (time 1734674192.77 total 142.84 last 568.71) - /unlink 64187 (time 1734674202.77 total 152.84 last 418.69) total: 64768 /unlink in 154.42 seconds: 419.43 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg122-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg122-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server stop mds service on oleg122-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server unloading modules via unload_modules_local on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2 PASS 106 (3427s) Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2 oleg122-server: oleg122-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing load_modules_local oleg122-server: Loading modules from /home/green/git/lustre-release/lustre oleg122-server: detected 4 online CPUs by sysfs oleg122-server: MODOPTS_LIBCFS= oleg122-server: Force libcfs to create 2 CPU partitions oleg122-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg122-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3527 sec ========== 00:57:46 (1734674266) === conf-sanity: start cleanup 00:57:46 (1734674266) === === conf-sanity: finish cleanup 00:57:47 (1734674267) ===