-----============= acceptance-small: conf-sanity ============----- Fri Sep 20 13:20:19 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 32newtarball 110 Stopping clients: oleg211-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg211-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg211-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg211-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg211-server oleg211-server: oleg211-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg211-server' oleg211-server: oleg211-server.virtnet: executing load_modules_local oleg211-server: Loading modules from /home/green/git/lustre-release/lustre oleg211-server: detected 4 online CPUs by sysfs oleg211-server: Force libcfs to create 2 CPU partitions oleg211-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg211-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg211-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg211-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg211-server: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg211-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg211-server: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg211-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg211-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server stop mds service on oleg211-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server stop mds service on oleg211-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg211-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:22:01 (1726852921) Stopping clients: oleg211-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg211-client.virtnet /mnt/lustre2 (opts:-f) oleg211-server: oleg211-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg211-server' oleg211-server: oleg211-server.virtnet: executing load_modules_local oleg211-server: Loading modules from /home/green/git/lustre-release/lustre oleg211-server: detected 4 online CPUs by sysfs oleg211-server: Force libcfs to create 2 CPU partitions oleg211-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg211-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg211-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg211-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg211-client.virtnet: -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16131 291 15840 2% /mnt/lustre[OST:0] filesystem_summary: 16483 643 15840 4% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1376 (time 1726852988.73 total 10.01 last 137.52) - open/close 2907 (time 1726852998.73 total 20.01 last 153.05) - open/close 4376 (time 1726853008.74 total 30.02 last 146.71) - open/close 5754 (time 1726853018.74 total 40.02 last 137.79) - open/close 7219 (time 1726853028.74 total 50.02 last 146.49) - open/close 8545 (time 1726853038.74 total 60.03 last 132.57) - open/close 9859 (time 1726853048.75 total 70.03 last 131.32) - open/close 11069 (time 1726853058.75 total 80.03 last 120.98) - open/close 12460 (time 1726853068.76 total 90.04 last 139.01) - open/close 13830 (time 1726853078.76 total 100.04 last 136.98) - open/close 15183 (time 1726853088.77 total 110.05 last 135.18) - open/close 16572 (time 1726853098.77 total 120.05 last 138.82) - open/close 18081 (time 1726853108.78 total 130.06 last 150.81) - open/close 19477 (time 1726853118.79 total 140.07 last 139.48) - open/close 20000 (time 1726853122.85 total 144.14 last 128.65) - open/close 21247 (time 1726853132.86 total 154.14 last 124.68) - open/close 22513 (time 1726853142.86 total 164.14 last 126.56) - open/close 23783 (time 1726853152.86 total 174.14 last 126.93) - open/close 24955 (time 1726853162.87 total 184.15 last 117.18) - open/close 26064 (time 1726853172.88 total 194.16 last 110.77) - open/close 27337 (time 1726853182.88 total 204.16 last 127.26) - open/close 28724 (time 1726853192.89 total 214.17 last 138.62) - open/close 30000 (time 1726853202.48 total 223.76 last 133.04) - open/close 31251 (time 1726853212.48 total 233.76 last 125.07) - open/close 32512 (time 1726853222.48 total 243.77 last 126.04) - open/close 33854 (time 1726853232.49 total 253.77 last 134.19) - open/close 35363 (time 1726853242.49 total 263.77 last 150.86) - open/close 36883 (time 1726853252.49 total 273.77 last 151.97) - open/close 38299 (time 1726853262.50 total 283.78 last 141.52) - open/close 39900 (time 1726853272.50 total 293.78 last 160.05) - open/close 41454 (time 1726853282.50 total 303.79 last 155.31) - open/close 42927 (time 1726853292.51 total 313.79 last 147.28) - open/close 44335 (time 1726853302.51 total 323.79 last 140.73) - open/close 45961 (time 1726853312.51 total 333.80 last 162.54) - open/close 47455 (time 1726853322.52 total 343.80 last 149.38) - open/close 48892 (time 1726853332.52 total 353.80 last 143.66) - open/close 50000 (time 1726853340.24 total 361.52 last 143.51) - open/close 51490 (time 1726853350.25 total 371.53 last 148.88) - open/close 53029 (time 1726853360.25 total 381.53 last 153.89) - open/close 54609 (time 1726853370.25 total 391.53 last 157.96) - open/close 56112 (time 1726853380.25 total 401.53 last 150.28) - open/close 57652 (time 1726853390.26 total 411.54 last 153.92) - open/close 59231 (time 1726853400.26 total 421.54 last 157.90) - open/close 60000 (time 1726853405.48 total 426.76 last 147.36) - open/close 61462 (time 1726853415.48 total 436.76 last 146.16) - open/close 62816 (time 1726853425.48 total 446.76 last 135.35) - open/close 64128 (time 1726853435.49 total 456.77 last 131.14) total: 64768 open/close in 461.31 seconds: 140.40 ops/second - /unlink 2526 (time 1726853451.92 total 10.00 last 252.53) - /unlink 5175 (time 1726853461.92 total 20.01 last 264.77) - /unlink 7678 (time 1726853471.93 total 30.01 last 250.12) - /unlink 10000 (time 1726853481.55 total 39.64 last 241.24) - /unlink 12404 (time 1726853491.56 total 49.64 last 240.31) - /unlink 15176 (time 1726853501.56 total 59.64 last 277.17) - /unlink 18064 (time 1726853511.56 total 69.65 last 288.73) - /unlink 20000 (time 1726853518.89 total 76.98 last 264.18) - /unlink 22601 (time 1726853528.89 total 86.98 last 260.04) - /unlink 24933 (time 1726853538.89 total 96.98 last 233.14) - /unlink 27257 (time 1726853548.90 total 106.98 last 232.33) - /unlink 29448 (time 1726853558.90 total 116.99 last 219.05) - /unlink 30000 (time 1726853561.27 total 119.35 last 233.11) - /unlink 32326 (time 1726853571.27 total 129.35 last 232.60) - /unlink 35103 (time 1726853581.27 total 139.36 last 277.65) - /unlink 37622 (time 1726853591.27 total 149.36 last 251.83) - /unlink 39923 (time 1726853601.28 total 159.36 last 229.97) - /unlink 42441 (time 1726853611.28 total 169.37 last 251.74) - /unlink 44917 (time 1726853621.28 total 179.37 last 247.54) - /unlink 47590 (time 1726853631.28 total 189.37 last 267.27) - /unlink 50000 (time 1726853641.11 total 199.20 last 245.19) - /unlink 52655 (time 1726853651.11 total 209.20 last 265.49) - /unlink 55809 (time 1726853661.12 total 219.20 last 315.31) - /unlink 58385 (time 1726853671.12 total 229.21 last 257.53) - /unlink 60000 (time 1726853678.31 total 236.40 last 224.44) - /unlink 62514 (time 1726853688.32 total 246.40 last 251.37) total: 64768 /unlink in 254.55 seconds: 254.44 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 1478 (time 1726853725.20 total 10.00 last 147.79) - open/close 2831 (time 1726853735.21 total 20.00 last 135.28) - open/close 4214 (time 1726853745.21 total 30.01 last 138.26) - open/close 5872 (time 1726853755.21 total 40.01 last 165.73) - open/close 7468 (time 1726853765.21 total 50.01 last 159.57) - open/close 9325 (time 1726853775.22 total 60.02 last 185.62) - open/close 10000 (time 1726853778.93 total 63.73 last 181.77) - open/close 11639 (time 1726853788.94 total 73.73 last 163.84) - open/close 13089 (time 1726853798.94 total 83.73 last 144.97) - open/close 14510 (time 1726853808.94 total 93.74 last 142.09) - open/close 16057 (time 1726853818.95 total 103.74 last 154.59) - open/close 17802 (time 1726853828.95 total 113.75 last 174.44) - open/close 19608 (time 1726853838.95 total 123.75 last 180.51) - open/close 20000 (time 1726853841.77 total 126.56 last 139.35) - open/close 21496 (time 1726853851.77 total 136.57 last 149.58) - open/close 23673 (time 1726853861.77 total 146.57 last 217.67) - open/close 25895 (time 1726853871.77 total 156.57 last 222.18) - open/close 28057 (time 1726853881.77 total 166.57 last 216.17) - open/close 30000 (time 1726853890.57 total 175.36 last 220.90) - open/close 31560 (time 1726853900.57 total 185.37 last 155.95) - open/close 33281 (time 1726853910.57 total 195.37 last 172.07) - open/close 35007 (time 1726853920.58 total 205.37 last 172.55) - open/close 36697 (time 1726853930.58 total 215.38 last 168.92) - open/close 38432 (time 1726853940.58 total 225.38 last 173.42) - open/close 40000 (time 1726853948.78 total 233.58 last 191.36) - open/close 41792 (time 1726853958.78 total 243.58 last 179.14) - open/close 43564 (time 1726853968.78 total 253.58 last 177.14) - open/close 45381 (time 1726853978.79 total 263.59 last 181.60) - open/close 46932 (time 1726853988.79 total 273.59 last 155.06) - open/close 48496 (time 1726853998.79 total 283.59 last 156.37) - open/close 50000 (time 1726854007.52 total 292.32 last 172.29) - open/close 51855 (time 1726854017.53 total 302.32 last 185.48) - open/close 53608 (time 1726854027.53 total 312.33 last 175.25) - open/close 55143 (time 1726854037.53 total 322.33 last 153.42) - open/close 56617 (time 1726854047.54 total 332.33 last 147.35) - open/close 58341 (time 1726854057.54 total 342.34 last 172.32) - open/close 59807 (time 1726854067.55 total 352.34 last 146.54) - open/close 61337 (time 1726854077.55 total 362.35 last 152.88) - open/close 62924 (time 1726854087.56 total 372.35 last 158.64) - open/close 64429 (time 1726854097.56 total 382.36 last 150.38) total: 64768 open/close in 384.70 seconds: 168.36 ops/second - /unlink 2715 (time 1726854111.39 total 10.00 last 271.49) - /unlink 5634 (time 1726854121.39 total 20.00 last 291.86) - /unlink 8339 (time 1726854131.39 total 30.00 last 270.49) - /unlink 10000 (time 1726854137.62 total 36.23 last 266.54) - /unlink 12483 (time 1726854147.62 total 46.24 last 248.26) - /unlink 15012 (time 1726854157.63 total 56.24 last 252.72) - /unlink 17551 (time 1726854167.63 total 66.25 last 253.84) - /unlink 20000 (time 1726854176.80 total 75.41 last 267.17) - /unlink 22725 (time 1726854186.80 total 85.41 last 272.47) - /unlink 25798 (time 1726854196.80 total 95.42 last 307.21) - /unlink 28851 (time 1726854206.80 total 105.42 last 305.25) - /unlink 30000 (time 1726854210.93 total 109.54 last 278.62) - /unlink 32720 (time 1726854220.93 total 119.54 last 271.94) - /unlink 35367 (time 1726854230.93 total 129.54 last 264.66) - /unlink 38119 (time 1726854240.93 total 139.55 last 275.12) - /unlink 40000 (time 1726854249.47 total 148.09 last 220.27) - /unlink 42471 (time 1726854259.48 total 158.09 last 247.04) - /unlink 45203 (time 1726854269.48 total 168.09 last 273.15) - /unlink 48277 (time 1726854279.48 total 178.09 last 307.36) - /unlink 50000 (time 1726854285.56 total 184.17 last 283.39) - /unlink 52802 (time 1726854295.56 total 194.18 last 280.13) - /unlink 55668 (time 1726854305.56 total 204.18 last 286.60) - /unlink 58479 (time 1726854315.56 total 214.18 last 281.06) - /unlink 60000 (time 1726854321.01 total 219.62 last 279.29) - /unlink 63487 (time 1726854331.01 total 229.63 last 348.62) total: 64768 /unlink in 233.69 seconds: 277.15 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 1702 (time 1726854365.88 total 10.00 last 170.18) - open/close 3401 (time 1726854375.88 total 20.01 last 169.83) - open/close 4965 (time 1726854385.89 total 30.01 last 156.32) - open/close 6425 (time 1726854395.89 total 40.01 last 145.94) - open/close 7889 (time 1726854405.89 total 50.02 last 146.37) - open/close 9482 (time 1726854415.90 total 60.02 last 159.28) - open/close 10000 (time 1726854418.98 total 63.10 last 168.14) - open/close 11649 (time 1726854428.98 total 73.10 last 164.83) - open/close 13565 (time 1726854438.98 total 83.11 last 191.51) - open/close 15545 (time 1726854448.99 total 93.11 last 197.94) - open/close 17463 (time 1726854458.99 total 103.11 last 191.73) - open/close 19240 (time 1726854468.99 total 113.11 last 177.67) - open/close 20000 (time 1726854473.69 total 117.82 last 161.65) - open/close 21943 (time 1726854483.70 total 127.82 last 194.26) - open/close 24594 (time 1726854493.70 total 137.82 last 265.08) - open/close 26616 (time 1726854503.70 total 147.82 last 202.17) - open/close 28404 (time 1726854513.70 total 157.82 last 178.78) - open/close 30000 (time 1726854523.01 total 167.14 last 171.37) - open/close 31618 (time 1726854533.01 total 177.14 last 161.79) - open/close 33617 (time 1726854543.02 total 187.14 last 199.83) - open/close 35797 (time 1726854553.02 total 197.14 last 217.95) - open/close 38212 (time 1726854563.02 total 207.14 last 241.43) - open/close 40000 (time 1726854569.97 total 214.09 last 257.24) - open/close 42032 (time 1726854579.98 total 224.10 last 203.13) - open/close 43664 (time 1726854589.98 total 234.10 last 163.13) - open/close 45580 (time 1726854599.98 total 244.10 last 191.58) - open/close 47614 (time 1726854609.98 total 254.11 last 203.37) - open/close 49947 (time 1726854619.99 total 264.11 last 233.19) - open/close 52927 (time 1726854629.99 total 274.11 last 297.95) - open/close 55985 (time 1726854640.00 total 284.12 last 305.63) - open/close 58827 (time 1726854650.00 total 294.12 last 284.14) - open/close 60000 (time 1726854653.65 total 297.78 last 320.74) - open/close 63895 (time 1726854663.65 total 307.78 last 389.49) total: 64768 open/close in 310.63 seconds: 208.51 ops/second - /unlink 4290 (time 1726854677.54 total 10.00 last 428.96) - /unlink 9344 (time 1726854687.54 total 20.00 last 505.34) - /unlink 14535 (time 1726854697.54 total 30.00 last 519.07) - /unlink 19355 (time 1726854707.54 total 40.00 last 481.97) - /unlink 23823 (time 1726854717.54 total 50.00 last 446.79) - /unlink 28895 (time 1726854727.54 total 60.00 last 507.14) - /unlink 30000 (time 1726854730.10 total 62.56 last 431.65) - /unlink 34168 (time 1726854740.11 total 72.57 last 416.74) - /unlink 38101 (time 1726854750.11 total 82.57 last 393.26) - /unlink 40000 (time 1726854754.57 total 87.03 last 425.97) - /unlink 44012 (time 1726854764.57 total 97.03 last 401.14) - /unlink 48055 (time 1726854774.57 total 107.03 last 404.23) - /unlink 50000 (time 1726854779.08 total 111.54 last 430.90) - /unlink 53735 (time 1726854789.09 total 121.55 last 373.32) - /unlink 57323 (time 1726854799.09 total 131.55 last 358.79) - /unlink 60000 (time 1726854805.59 total 138.05 last 411.44) total: 64768 /unlink in 146.26 seconds: 442.82 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2657 (time 1726854842.12 total 10.00 last 265.67) - open/close 5541 (time 1726854852.12 total 20.00 last 288.37) - open/close 9368 (time 1726854862.13 total 30.01 last 382.53) - open/close 12430 (time 1726854872.13 total 40.01 last 306.17) - open/close 15425 (time 1726854882.13 total 50.01 last 299.42) - open/close 18419 (time 1726854892.13 total 60.02 last 299.25) - open/close 20000 (time 1726854898.99 total 66.88 last 230.45) - open/close 22934 (time 1726854909.00 total 76.88 last 293.39) - open/close 26090 (time 1726854919.00 total 86.88 last 315.46) - open/close 27940 (time 1726854929.00 total 96.88 last 185.00) - open/close 30000 (time 1726854936.64 total 104.52 last 269.50) - open/close 32429 (time 1726854946.65 total 114.53 last 242.81) - open/close 34559 (time 1726854956.65 total 124.53 last 212.94) - open/close 36942 (time 1726854966.65 total 134.53 last 238.28) - open/close 38902 (time 1726854976.65 total 144.53 last 195.96) - open/close 40000 (time 1726854980.96 total 148.84 last 254.90) - open/close 42275 (time 1726854990.96 total 158.84 last 227.48) - open/close 44369 (time 1726855000.97 total 168.85 last 209.23) - open/close 46065 (time 1726855010.98 total 178.86 last 169.43) - open/close 48303 (time 1726855020.98 total 188.86 last 223.79) - open/close 50000 (time 1726855025.81 total 193.69 last 351.38) - open/close 52584 (time 1726855035.81 total 203.69 last 258.31) - open/close 54734 (time 1726855045.85 total 213.74 last 214.12) - open/close 57173 (time 1726855055.86 total 223.75 last 243.66) - open/close 58967 (time 1726855065.87 total 233.75 last 179.34) - open/close 60000 (time 1726855072.21 total 240.09 last 162.92) - open/close 61746 (time 1726855082.22 total 250.10 last 174.46) - open/close 63465 (time 1726855092.22 total 260.10 last 171.89) total: 64768 open/close in 266.22 seconds: 243.29 ops/second - /unlink 5129 (time 1726855109.24 total 10.00 last 512.83) - /unlink 9980 (time 1726855119.24 total 20.00 last 485.05) - /unlink 15137 (time 1726855129.24 total 30.00 last 515.65) - /unlink 19384 (time 1726855139.24 total 40.00 last 424.69) - /unlink 23896 (time 1726855149.24 total 50.00 last 451.18) - /unlink 28436 (time 1726855159.25 total 60.01 last 453.93) - /unlink 30000 (time 1726855163.15 total 63.91 last 400.95) - /unlink 34372 (time 1726855173.15 total 73.91 last 437.20) - /unlink 38779 (time 1726855183.15 total 83.91 last 440.61) - /unlink 40000 (time 1726855186.14 total 86.91 last 407.43) - /unlink 44248 (time 1726855196.15 total 96.91 last 424.70) - /unlink 48751 (time 1726855206.15 total 106.91 last 450.19) - /unlink 50000 (time 1726855208.93 total 109.69 last 449.11) - /unlink 54783 (time 1726855218.93 total 119.69 last 478.29) - /unlink 60000 (time 1726855226.25 total 127.01 last 713.16) total: 64768 /unlink in 133.25 seconds: 486.08 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4328 (time 1726855262.07 total 10.00 last 432.74) - open/close 8541 (time 1726855272.07 total 20.00 last 421.24) - open/close 10000 (time 1726855275.40 total 23.33 last 438.67) - open/close 13200 (time 1726855285.40 total 33.33 last 319.96) - open/close 15982 (time 1726855295.40 total 43.33 last 278.16) - open/close 20000 (time 1726855304.64 total 52.58 last 434.66) - open/close 24639 (time 1726855314.64 total 62.58 last 463.87) - open/close 29336 (time 1726855324.65 total 72.58 last 469.62) - open/close 33575 (time 1726855334.65 total 82.58 last 423.89) - open/close 37617 (time 1726855344.70 total 92.64 last 401.89) - open/close 40000 (time 1726855350.88 total 98.81 last 386.02) - open/close 44911 (time 1726855360.88 total 108.81 last 491.06) - open/close 49572 (time 1726855370.88 total 118.81 last 466.10) - open/close 54859 (time 1726855380.88 total 128.81 last 528.62) - open/close 59547 (time 1726855390.88 total 138.81 last 468.76) - open/close 64426 (time 1726855400.88 total 148.81 last 487.85) total: 64768 open/close in 149.57 seconds: 433.02 ops/second - /unlink 6702 (time 1726855412.47 total 10.00 last 670.19) - /unlink 10000 (time 1726855417.69 total 15.22 last 632.16) - /unlink 18460 (time 1726855427.69 total 25.22 last 845.97) - /unlink 27917 (time 1726855437.69 total 35.22 last 945.62) - /unlink 30000 (time 1726855440.06 total 37.59 last 878.90) - /unlink 38012 (time 1726855450.06 total 47.59 last 801.13) - /unlink 40000 (time 1726855452.75 total 50.28 last 739.39) - /unlink 48351 (time 1726855462.75 total 60.28 last 835.08) - /unlink 57151 (time 1726855472.75 total 70.28 last 879.98) - /unlink 60000 (time 1726855475.69 total 73.22 last 968.52) total: 64768 /unlink in 78.70 seconds: 823.01 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg211-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg211-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server stop mds service on oleg211-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server stop mds service on oleg211-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg211-server unloading modules on: 'oleg211-server' oleg211-server: oleg211-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2 PASS 106 (2599s) Stopping clients: oleg211-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg211-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2 oleg211-server: oleg211-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg211-server' oleg211-server: oleg211-server.virtnet: executing load_modules_local oleg211-server: Loading modules from /home/green/git/lustre-release/lustre oleg211-server: detected 4 online CPUs by sysfs oleg211-server: Force libcfs to create 2 CPU partitions oleg211-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg211-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 2721 sec ========== 14:05:40 (1726855540) === conf-sanity: start cleanup 14:05:41 (1726855541) === === conf-sanity: finish cleanup 14:05:41 (1726855541) ===