-----============= acceptance-small: conf-sanity ============----- Thu May 1 23:21:48 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg246-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg246-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg246-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg246-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg246-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg246-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg246-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg246-server oleg246-server: oleg246-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing load_modules_local oleg246-server: Loading modules from /home/green/git/lustre-release/lustre oleg246-server: detected 4 online CPUs by sysfs oleg246-server: MODOPTS_LIBCFS= oleg246-server: Force libcfs to create 2 CPU partitions oleg246-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg246-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg246-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg246-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg246-server: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg246-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg246-server: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg246-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg246-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg246-server stop mds service on oleg246-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg246-server stop mds service on oleg246-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg246-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 23:23:34 (1746156214) Stopping clients: oleg246-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg246-client.virtnet /mnt/lustre2 (opts:-f) oleg246-server: oleg246-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing load_modules_local oleg246-server: Loading modules from /home/green/git/lustre-release/lustre oleg246-server: detected 4 online CPUs by sysfs oleg246-server: MODOPTS_LIBCFS= oleg246-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg246-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg246-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg246-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg246-client.virtnet: -o user_xattr,flock 192.168.202.146@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1388 (time 1746156278.54 total 10.00 last 138.77) - open/close 2786 (time 1746156288.54 total 20.00 last 139.77) - open/close 4266 (time 1746156298.54 total 30.01 last 147.94) - open/close 5647 (time 1746156308.55 total 40.02 last 137.97) - open/close 7005 (time 1746156318.55 total 50.02 last 135.79) - open/close 8482 (time 1746156328.56 total 60.02 last 147.67) - open/close 9913 (time 1746156338.56 total 70.03 last 143.02) - open/close 11384 (time 1746156348.57 total 80.03 last 147.01) - open/close 12693 (time 1746156358.58 total 90.04 last 130.82) - open/close 14198 (time 1746156368.58 total 100.04 last 150.44) - open/close 15614 (time 1746156378.59 total 110.05 last 141.50) - open/close 16962 (time 1746156388.60 total 120.06 last 134.63) - open/close 18282 (time 1746156398.60 total 130.06 last 131.98) - open/close 19497 (time 1746156408.61 total 140.07 last 121.40) - open/close 20000 (time 1746156412.49 total 143.95 last 129.57) - open/close 21329 (time 1746156422.50 total 153.96 last 132.83) - open/close 22789 (time 1746156432.50 total 163.97 last 145.90) - open/close 24203 (time 1746156442.51 total 173.97 last 141.36) - open/close 25621 (time 1746156452.51 total 183.97 last 141.79) - open/close 27096 (time 1746156462.51 total 193.97 last 147.44) - open/close 28446 (time 1746156472.51 total 203.97 last 134.98) - open/close 29793 (time 1746156482.53 total 213.99 last 134.48) - open/close 31244 (time 1746156492.56 total 224.03 last 144.60) - open/close 32691 (time 1746156502.57 total 234.03 last 144.65) - open/close 34000 (time 1746156512.57 total 244.03 last 130.89) - open/close 35447 (time 1746156522.57 total 254.03 last 144.63) - open/close 36810 (time 1746156532.58 total 264.04 last 136.23) - open/close 38171 (time 1746156542.58 total 274.04 last 136.06) - open/close 39446 (time 1746156552.58 total 284.05 last 127.47) - open/close 40000 (time 1746156556.63 total 288.09 last 136.85) - open/close 41456 (time 1746156566.63 total 298.10 last 145.58) - open/close 42954 (time 1746156576.64 total 308.10 last 149.75) - open/close 44512 (time 1746156586.64 total 318.10 last 155.76) - open/close 46136 (time 1746156596.64 total 328.10 last 162.37) - open/close 47808 (time 1746156606.64 total 338.10 last 167.19) - open/close 49774 (time 1746156616.64 total 348.10 last 196.60) - open/close 51433 (time 1746156626.64 total 358.11 last 165.84) - open/close 53402 (time 1746156636.65 total 368.11 last 196.82) - open/close 55040 (time 1746156646.65 total 378.12 last 163.73) - open/close 56695 (time 1746156656.66 total 388.12 last 165.43) - open/close 58340 (time 1746156666.66 total 398.13 last 164.41) - open/close 59888 (time 1746156676.67 total 408.13 last 154.73) - open/close 61420 (time 1746156686.67 total 418.13 last 153.19) - open/close 63027 (time 1746156696.67 total 428.13 last 160.68) total: 64768 open/close in 438.02 seconds: 147.87 ops/second - /unlink 3464 (time 1746156717.88 total 10.00 last 346.32) - /unlink 6135 (time 1746156727.88 total 20.00 last 267.10) - /unlink 8960 (time 1746156737.89 total 30.00 last 282.45) - /unlink 10000 (time 1746156741.07 total 33.19 last 326.49) - /unlink 12634 (time 1746156751.07 total 43.19 last 263.38) - /unlink 15287 (time 1746156761.07 total 53.19 last 265.25) - /unlink 17806 (time 1746156771.08 total 63.20 last 251.82) - /unlink 20000 (time 1746156778.36 total 70.48 last 301.20) - /unlink 23647 (time 1746156788.36 total 80.48 last 364.65) - /unlink 27597 (time 1746156798.37 total 90.48 last 394.87) - /unlink 30000 (time 1746156803.92 total 96.04 last 432.88) - /unlink 34328 (time 1746156813.92 total 106.04 last 432.80) - /unlink 38872 (time 1746156823.92 total 116.04 last 454.36) - /unlink 40000 (time 1746156827.04 total 119.16 last 360.87) - /unlink 44781 (time 1746156837.05 total 129.16 last 478.02) - /unlink 48846 (time 1746156847.05 total 139.17 last 406.44) - /unlink 50000 (time 1746156849.77 total 141.89 last 423.43) - /unlink 54141 (time 1746156859.78 total 151.89 last 413.99) - /unlink 58683 (time 1746156869.78 total 161.90 last 454.11) - /unlink 60000 (time 1746156873.20 total 165.32 last 384.53) - /unlink 63336 (time 1746156883.20 total 175.32 last 333.60) total: 64768 /unlink in 179.07 seconds: 361.68 ops/second Waiting for MDT destroys to complete - open/close 1602 (time 1746156905.09 total 10.00 last 160.17) - open/close 3560 (time 1746156915.09 total 20.00 last 195.74) - open/close 5529 (time 1746156925.10 total 30.01 last 196.78) - open/close 7533 (time 1746156935.11 total 40.02 last 200.25) - open/close 9327 (time 1746156945.11 total 50.02 last 179.37) - open/close 10000 (time 1746156949.07 total 53.98 last 169.96) - open/close 11677 (time 1746156959.07 total 63.98 last 167.62) - open/close 13699 (time 1746156969.07 total 73.99 last 202.16) - open/close 15714 (time 1746156979.08 total 83.99 last 201.48) - open/close 18129 (time 1746156989.08 total 93.99 last 241.48) - open/close 19866 (time 1746156999.08 total 103.99 last 173.67) - open/close 21431 (time 1746157009.08 total 113.99 last 156.42) - open/close 23262 (time 1746157019.08 total 124.00 last 183.05) - open/close 24835 (time 1746157029.10 total 134.01 last 157.13) - open/close 26415 (time 1746157039.10 total 144.01 last 157.99) - open/close 27778 (time 1746157049.10 total 154.01 last 136.27) - open/close 29282 (time 1746157059.10 total 164.01 last 150.39) - open/close 30000 (time 1746157063.85 total 168.76 last 151.26) - open/close 31594 (time 1746157073.85 total 178.76 last 159.35) - open/close 33297 (time 1746157083.85 total 188.77 last 170.23) - open/close 34761 (time 1746157093.86 total 198.77 last 146.30) - open/close 36418 (time 1746157103.86 total 208.78 last 165.63) - open/close 38039 (time 1746157113.87 total 218.78 last 162.04) - open/close 39458 (time 1746157123.87 total 228.78 last 141.84) - open/close 40000 (time 1746157127.00 total 231.91 last 173.37) - open/close 41465 (time 1746157137.00 total 241.91 last 146.49) - open/close 43012 (time 1746157147.00 total 251.91 last 154.64) - open/close 45173 (time 1746157157.00 total 261.92 last 216.07) - open/close 46986 (time 1746157167.01 total 271.92 last 181.25) - open/close 48714 (time 1746157177.01 total 281.92 last 172.79) - open/close 50000 (time 1746157183.58 total 288.49 last 195.58) - open/close 51791 (time 1746157193.58 total 298.49 last 179.10) - open/close 53539 (time 1746157203.58 total 308.50 last 174.77) - open/close 55176 (time 1746157213.59 total 318.50 last 163.67) - open/close 56756 (time 1746157223.59 total 328.50 last 157.92) - open/close 58150 (time 1746157233.59 total 338.50 last 139.39) - open/close 59786 (time 1746157243.59 total 348.51 last 163.58) - open/close 61087 (time 1746157253.59 total 358.51 last 130.09) - open/close 62374 (time 1746157263.60 total 368.51 last 128.59) - open/close 63859 (time 1746157273.60 total 378.52 last 148.49) total: 64768 open/close in 385.13 seconds: 168.17 ops/second - /unlink 2524 (time 1746157291.67 total 10.01 last 252.26) - /unlink 5116 (time 1746157301.67 total 20.01 last 259.16) - /unlink 7366 (time 1746157311.67 total 30.01 last 224.97) - /unlink 9797 (time 1746157321.67 total 40.01 last 243.10) - /unlink 12148 (time 1746157331.68 total 50.02 last 234.94) - /unlink 14728 (time 1746157341.68 total 60.02 last 257.99) - /unlink 16910 (time 1746157351.68 total 70.02 last 218.12) - /unlink 18732 (time 1746157361.68 total 80.02 last 182.14) - /unlink 20000 (time 1746157367.62 total 85.96 last 213.56) - /unlink 22032 (time 1746157377.63 total 95.97 last 203.08) - /unlink 24224 (time 1746157387.63 total 105.97 last 219.11) - /unlink 26523 (time 1746157397.63 total 115.97 last 229.89) - /unlink 28598 (time 1746157407.63 total 125.97 last 207.43) - /unlink 30000 (time 1746157413.91 total 132.25 last 223.35) - /unlink 32421 (time 1746157423.91 total 142.25 last 242.08) - /unlink 35311 (time 1746157433.91 total 152.25 last 289.00) - /unlink 38445 (time 1746157443.92 total 162.25 last 313.31) - /unlink 40000 (time 1746157449.15 total 167.48 last 297.31) - /unlink 42734 (time 1746157459.15 total 177.49 last 273.17) - /unlink 45162 (time 1746157469.16 total 187.50 last 242.63) - /unlink 47703 (time 1746157479.16 total 197.50 last 254.08) - /unlink 50000 (time 1746157488.15 total 206.49 last 255.49) - /unlink 52492 (time 1746157498.15 total 216.49 last 249.15) - /unlink 55345 (time 1746157508.16 total 226.49 last 285.25) - /unlink 58167 (time 1746157518.16 total 236.50 last 282.17) - /unlink 60000 (time 1746157525.19 total 243.53 last 260.69) - /unlink 63405 (time 1746157535.19 total 253.53 last 340.31) total: 64768 /unlink in 257.29 seconds: 251.73 ops/second Waiting for MDT destroys to complete - open/close 1494 (time 1746157561.68 total 10.01 last 149.26) - open/close 2982 (time 1746157571.68 total 20.01 last 148.75) - open/close 4860 (time 1746157581.69 total 30.02 last 187.72) - open/close 6886 (time 1746157591.69 total 40.02 last 202.57) - open/close 8711 (time 1746157601.69 total 50.02 last 182.44) - open/close 10000 (time 1746157608.13 total 56.46 last 200.15) - open/close 11825 (time 1746157618.14 total 66.46 last 182.46) - open/close 13970 (time 1746157628.14 total 76.47 last 214.47) - open/close 16912 (time 1746157638.14 total 86.47 last 294.15) - open/close 18759 (time 1746157648.14 total 96.47 last 184.69) - open/close 20000 (time 1746157655.74 total 104.07 last 163.26) - open/close 22020 (time 1746157665.74 total 114.07 last 201.97) - open/close 24109 (time 1746157675.74 total 124.07 last 208.83) - open/close 25968 (time 1746157685.75 total 134.08 last 185.80) - open/close 27329 (time 1746157695.75 total 144.08 last 136.08) - open/close 28713 (time 1746157705.76 total 154.08 last 138.34) - open/close 29986 (time 1746157715.76 total 164.08 last 127.29) - open/close 31305 (time 1746157725.76 total 174.09 last 131.87) - open/close 32570 (time 1746157735.76 total 184.09 last 126.50) - open/close 33809 (time 1746157745.76 total 194.09 last 123.83) - open/close 34757 (time 1746157755.77 total 204.09 last 94.78) - open/close 35656 (time 1746157765.77 total 214.09 last 89.89) - open/close 36573 (time 1746157775.78 total 224.11 last 91.59) - open/close 37867 (time 1746157785.79 total 234.12 last 129.30) - open/close 39788 (time 1746157795.79 total 244.12 last 192.06) - open/close 41516 (time 1746157805.79 total 254.12 last 172.78) - open/close 43179 (time 1746157815.80 total 264.13 last 166.12) - open/close 44952 (time 1746157825.81 total 274.14 last 177.12) - open/close 46946 (time 1746157835.82 total 284.14 last 199.30) - open/close 48368 (time 1746157845.82 total 294.15 last 142.17) - open/close 49527 (time 1746157855.82 total 304.15 last 115.89) - open/close 50000 (time 1746157859.21 total 307.54 last 139.60) - open/close 51512 (time 1746157869.21 total 317.54 last 151.18) - open/close 53006 (time 1746157879.21 total 327.54 last 149.40) - open/close 54267 (time 1746157889.22 total 337.54 last 126.01) - open/close 55490 (time 1746157899.22 total 347.55 last 122.29) - open/close 56857 (time 1746157909.22 total 357.55 last 136.67) - open/close 58490 (time 1746157919.22 total 367.55 last 163.21) - open/close 59659 (time 1746157929.23 total 377.56 last 116.83) - open/close 60000 (time 1746157931.85 total 380.18 last 129.95) - open/close 61297 (time 1746157941.86 total 390.19 last 129.61) - open/close 62515 (time 1746157951.87 total 400.19 last 121.75) - open/close 63888 (time 1746157961.87 total 410.19 last 137.29) total: 64768 open/close in 417.30 seconds: 155.21 ops/second - /unlink 2151 (time 1746157980.56 total 10.00 last 215.08) - /unlink 4265 (time 1746157990.57 total 20.01 last 211.18) - /unlink 5869 (time 1746158000.57 total 30.01 last 160.37) - /unlink 7278 (time 1746158010.57 total 40.02 last 140.86) - /unlink 8755 (time 1746158020.57 total 50.02 last 147.68) - /unlink 10000 (time 1746158029.18 total 58.62 last 144.70) - /unlink 11755 (time 1746158039.18 total 68.62 last 175.48) - /unlink 13408 (time 1746158049.18 total 78.63 last 165.24) - /unlink 15393 (time 1746158059.19 total 88.63 last 198.41) - /unlink 17100 (time 1746158069.19 total 98.63 last 170.69) - /unlink 18839 (time 1746158079.19 total 108.63 last 173.85) - /unlink 20000 (time 1746158085.24 total 114.68 last 191.89) - /unlink 21874 (time 1746158095.24 total 124.68 last 187.40) - /unlink 23709 (time 1746158105.24 total 134.69 last 183.47) - /unlink 25671 (time 1746158115.24 total 144.69 last 196.20) - /unlink 28081 (time 1746158125.24 total 154.69 last 240.97) - /unlink 29919 (time 1746158135.25 total 164.69 last 183.75) - /unlink 31886 (time 1746158145.25 total 174.69 last 196.67) - /unlink 34057 (time 1746158155.26 total 184.70 last 216.93) - /unlink 35829 (time 1746158165.26 total 194.70 last 177.11) - /unlink 37583 (time 1746158175.27 total 204.71 last 175.28) - /unlink 39564 (time 1746158185.27 total 214.71 last 198.06) - /unlink 40000 (time 1746158187.68 total 217.12 last 180.78) - /unlink 41711 (time 1746158197.69 total 227.13 last 171.02) - /unlink 43558 (time 1746158207.69 total 237.14 last 184.57) - /unlink 45509 (time 1746158217.71 total 247.15 last 194.82) - /unlink 47414 (time 1746158227.71 total 257.15 last 190.45) - /unlink 49558 (time 1746158237.71 total 267.15 last 214.38) - /unlink 52127 (time 1746158247.71 total 277.15 last 256.89) - /unlink 54676 (time 1746158257.71 total 287.16 last 254.89) - /unlink 56889 (time 1746158267.71 total 297.16 last 221.25) - /unlink 59499 (time 1746158277.72 total 307.16 last 260.90) - /unlink 60000 (time 1746158280.14 total 309.59 last 206.52) - /unlink 62504 (time 1746158290.15 total 319.59 last 250.37) total: 64768 /unlink in 327.22 seconds: 197.93 ops/second Waiting for MDT destroys to complete - open/close 1519 (time 1746158321.81 total 10.00 last 151.86) - open/close 2905 (time 1746158331.81 total 20.01 last 138.53) - open/close 4363 (time 1746158341.82 total 30.01 last 145.74) - open/close 5917 (time 1746158351.82 total 40.01 last 155.37) - open/close 7572 (time 1746158361.82 total 50.01 last 165.49) - open/close 9255 (time 1746158371.82 total 60.02 last 168.25) - open/close 10000 (time 1746158377.81 total 66.00 last 124.50) - open/close 11306 (time 1746158387.81 total 76.01 last 130.55) - open/close 12736 (time 1746158397.81 total 86.01 last 142.96) - open/close 14221 (time 1746158407.82 total 96.02 last 148.40) - open/close 15674 (time 1746158417.83 total 106.02 last 145.16) - open/close 16994 (time 1746158427.84 total 116.03 last 131.94) - open/close 18373 (time 1746158437.84 total 126.03 last 137.86) - open/close 19717 (time 1746158447.84 total 136.03 last 134.38) - open/close 20000 (time 1746158449.98 total 138.17 last 132.44) - open/close 21636 (time 1746158459.98 total 148.18 last 163.52) - open/close 22929 (time 1746158469.98 total 158.18 last 129.27) - open/close 24284 (time 1746158479.99 total 168.18 last 135.46) - open/close 25629 (time 1746158489.99 total 178.19 last 134.44) - open/close 26701 (time 1746158500.00 total 188.19 last 107.13) - open/close 27919 (time 1746158510.00 total 198.20 last 121.73) - open/close 29242 (time 1746158520.01 total 208.20 last 132.23) - open/close 30000 (time 1746158525.51 total 213.70 last 137.81) - open/close 31314 (time 1746158535.52 total 223.71 last 131.27) - open/close 32461 (time 1746158545.52 total 233.72 last 114.67) - open/close 33524 (time 1746158555.53 total 243.72 last 106.26) - open/close 34700 (time 1746158565.55 total 253.74 last 117.35) - open/close 35838 (time 1746158575.55 total 263.74 last 113.80) - open/close 36891 (time 1746158585.55 total 273.75 last 105.24) - open/close 38091 (time 1746158595.56 total 283.75 last 119.98) - open/close 39307 (time 1746158605.56 total 293.75 last 121.57) - open/close 40000 (time 1746158611.97 total 300.16 last 108.12) - open/close 41081 (time 1746158621.97 total 310.16 last 108.09) - open/close 42315 (time 1746158631.97 total 320.16 last 123.37) - open/close 43401 (time 1746158641.97 total 330.17 last 108.59) - open/close 44528 (time 1746158651.98 total 340.18 last 112.59) - open/close 45721 (time 1746158661.99 total 350.18 last 119.24) - open/close 46820 (time 1746158671.99 total 360.18 last 109.89) - open/close 47908 (time 1746158681.99 total 370.18 last 108.78) - open/close 49054 (time 1746158691.99 total 380.19 last 114.53) - open/close 50000 (time 1746158700.48 total 388.67 last 111.48) - open/close 51081 (time 1746158710.49 total 398.68 last 108.02) - open/close 52268 (time 1746158720.49 total 408.68 last 118.68) - open/close 53399 (time 1746158730.49 total 418.69 last 113.06) - open/close 54548 (time 1746158740.50 total 428.69 last 114.83) - open/close 55737 (time 1746158750.51 total 438.70 last 118.84) - open/close 56906 (time 1746158760.51 total 448.70 last 116.85) - open/close 58177 (time 1746158770.51 total 458.70 last 127.09) - open/close 59271 (time 1746158780.51 total 468.71 last 109.38) - open/close 60000 (time 1746158787.51 total 475.70 last 104.22) - open/close 61179 (time 1746158797.51 total 485.71 last 117.82) - open/close 62366 (time 1746158807.52 total 495.71 last 118.63) - open/close 63446 (time 1746158817.53 total 505.72 last 107.88) - open/close 64585 (time 1746158827.54 total 515.73 last 113.84) total: 64768 open/close in 517.09 seconds: 125.25 ops/second - /unlink 1679 (time 1746158840.97 total 10.00 last 167.86) - /unlink 3177 (time 1746158850.97 total 20.00 last 149.78) - /unlink 4502 (time 1746158860.97 total 30.00 last 132.49) - /unlink 6021 (time 1746158870.98 total 40.01 last 151.75) - /unlink 7306 (time 1746158881.00 total 50.03 last 128.34) - /unlink 8506 (time 1746158891.00 total 60.03 last 119.97) - /unlink 9735 (time 1746158901.00 total 70.03 last 122.88) - /unlink 11174 (time 1746158911.01 total 80.04 last 143.83) - /unlink 12926 (time 1746158921.01 total 90.04 last 175.12) - /unlink 14766 (time 1746158931.01 total 100.04 last 183.99) - /unlink 16494 (time 1746158941.01 total 110.04 last 172.78) - /unlink 18350 (time 1746158951.02 total 120.05 last 185.45) - /unlink 19799 (time 1746158961.02 total 130.05 last 144.87) - /unlink 21587 (time 1746158971.03 total 140.06 last 178.74) - /unlink 23436 (time 1746158981.03 total 150.06 last 184.88) - /unlink 25432 (time 1746158991.03 total 160.06 last 199.50) - /unlink 27296 (time 1746159001.04 total 170.07 last 186.28) - /unlink 29065 (time 1746159011.04 total 180.07 last 176.85) - /unlink 30000 (time 1746159016.38 total 185.41 last 175.27) - /unlink 31753 (time 1746159026.38 total 195.41 last 175.25) - /unlink 33594 (time 1746159036.39 total 205.41 last 183.99) - /unlink 35511 (time 1746159046.39 total 215.42 last 191.68) - /unlink 37188 (time 1746159056.39 total 225.42 last 167.69) - /unlink 38773 (time 1746159066.41 total 235.43 last 158.21) - /unlink 40000 (time 1746159073.58 total 242.61 last 171.03) - /unlink 41852 (time 1746159083.58 total 252.61 last 185.19) - /unlink 43477 (time 1746159093.59 total 262.62 last 162.41) - /unlink 45179 (time 1746159103.59 total 272.62 last 170.19) - /unlink 47245 (time 1746159113.59 total 282.62 last 206.53) - /unlink 49289 (time 1746159123.60 total 292.63 last 204.21) - /unlink 50000 (time 1746159127.33 total 296.35 last 190.81) - /unlink 52102 (time 1746159137.33 total 306.36 last 210.10) - /unlink 54134 (time 1746159147.33 total 316.36 last 203.14) - /unlink 56293 (time 1746159157.33 total 326.36 last 215.89) - /unlink 58368 (time 1746159167.34 total 336.36 last 207.46) - /unlink 60000 (time 1746159174.22 total 343.25 last 237.13) - /unlink 62414 (time 1746159184.22 total 353.25 last 241.39) - /unlink 64449 (time 1746159194.22 total 363.25 last 203.48) total: 64768 /unlink in 365.01 seconds: 177.44 ops/second Waiting for MDT destroys to complete - open/close 1160 (time 1746159219.17 total 10.00 last 115.97) - open/close 2281 (time 1746159229.17 total 20.01 last 112.05) - open/close 3427 (time 1746159239.17 total 30.01 last 114.54) - open/close 4677 (time 1746159249.18 total 40.02 last 124.89) - open/close 5775 (time 1746159259.19 total 50.03 last 109.75) - open/close 6950 (time 1746159269.19 total 60.03 last 117.45) - open/close 8332 (time 1746159279.20 total 70.04 last 138.04) - open/close 9396 (time 1746159289.22 total 80.06 last 106.20) - open/close 10000 (time 1746159295.58 total 86.42 last 95.03) - open/close 11127 (time 1746159305.59 total 96.43 last 112.60) - open/close 12393 (time 1746159315.59 total 106.43 last 126.57) - open/close 13531 (time 1746159325.60 total 116.44 last 113.72) - open/close 14673 (time 1746159335.60 total 126.44 last 114.15) - open/close 15936 (time 1746159345.60 total 136.44 last 126.27) - open/close 17184 (time 1746159355.61 total 146.45 last 124.71) - open/close 18356 (time 1746159365.62 total 156.45 last 117.15) - open/close 19642 (time 1746159375.62 total 166.46 last 128.56) - open/close 20000 (time 1746159378.16 total 168.99 last 141.13) - open/close 21089 (time 1746159388.16 total 179.00 last 108.86) - open/close 22294 (time 1746159398.17 total 189.00 last 120.43) - open/close 23596 (time 1746159408.17 total 199.00 last 130.20) - open/close 24804 (time 1746159418.17 total 209.01 last 120.72) - open/close 25979 (time 1746159428.18 total 219.01 last 117.46) - open/close 27075 (time 1746159438.18 total 229.02 last 109.56) - open/close 28180 (time 1746159448.19 total 239.02 last 110.42) - open/close 29318 (time 1746159458.19 total 249.03 last 113.73) - open/close 30000 (time 1746159464.19 total 255.03 last 113.71) - open/close 31187 (time 1746159474.19 total 265.03 last 118.67) - open/close 32427 (time 1746159484.20 total 275.03 last 123.94) - open/close 33583 (time 1746159494.20 total 285.04 last 115.57) - open/close 34710 (time 1746159504.20 total 295.04 last 112.64) - open/close 35961 (time 1746159514.21 total 305.05 last 125.02) - open/close 37200 (time 1746159524.21 total 315.05 last 123.84) - open/close 38278 (time 1746159534.22 total 325.05 last 107.77) - open/close 39410 (time 1746159544.22 total 335.06 last 113.19) - open/close 40000 (time 1746159549.64 total 340.47 last 108.89) - open/close 41261 (time 1746159559.67 total 350.50 last 125.71) - open/close 42286 (time 1746159569.68 total 360.51 last 102.40) - open/close 43467 (time 1746159579.68 total 370.52 last 118.04) - open/close 44755 (time 1746159589.69 total 380.53 last 128.65) - open/close 45962 (time 1746159599.70 total 390.53 last 120.68) - open/close 47164 (time 1746159609.70 total 400.54 last 120.17) - open/close 48464 (time 1746159619.71 total 410.55 last 129.83) - open/close 49557 (time 1746159629.71 total 420.55 last 109.26) - open/close 50000 (time 1746159633.26 total 424.10 last 125.01) - open/close 51295 (time 1746159643.26 total 434.10 last 129.43) - open/close 52390 (time 1746159653.27 total 444.11 last 109.44) - open/close 53590 (time 1746159663.27 total 454.11 last 119.94) - open/close 54942 (time 1746159673.28 total 464.12 last 135.15) - open/close 56029 (time 1746159683.29 total 474.12 last 108.61) - open/close 57202 (time 1746159693.29 total 484.13 last 117.25) - open/close 58479 (time 1746159703.30 total 494.14 last 127.57) - open/close 59537 (time 1746159713.31 total 504.15 last 105.68) - open/close 60000 (time 1746159717.55 total 508.38 last 109.30) - open/close 61166 (time 1746159727.55 total 518.39 last 116.54) - open/close 62315 (time 1746159737.56 total 528.40 last 114.82) - open/close 63428 (time 1746159747.56 total 538.40 last 111.30) - open/close 64657 (time 1746159757.57 total 548.41 last 122.79) total: 64768 open/close in 549.23 seconds: 117.92 ops/second - /unlink 1916 (time 1746159770.23 total 10.00 last 191.60) - /unlink 3656 (time 1746159780.23 total 20.01 last 173.90) - /unlink 5508 (time 1746159790.24 total 30.01 last 185.13) - /unlink 7465 (time 1746159800.25 total 40.02 last 195.57) - /unlink 9261 (time 1746159810.25 total 50.02 last 179.54) - /unlink 10000 (time 1746159814.74 total 54.51 last 164.39) - /unlink 11823 (time 1746159824.75 total 64.52 last 182.14) - /unlink 13678 (time 1746159834.76 total 74.53 last 185.36) - /unlink 15352 (time 1746159844.76 total 84.53 last 167.34) - /unlink 17060 (time 1746159854.76 total 94.53 last 170.80) - /unlink 18834 (time 1746159864.77 total 104.54 last 177.31) - /unlink 20000 (time 1746159871.75 total 111.52 last 166.94) - /unlink 21842 (time 1746159881.76 total 121.53 last 184.09) - /unlink 23635 (time 1746159891.76 total 131.53 last 179.26) - /unlink 25397 (time 1746159901.76 total 141.53 last 176.18) - /unlink 27522 (time 1746159911.77 total 151.54 last 212.45) - /unlink 29486 (time 1746159921.77 total 161.54 last 196.36) - /unlink 30000 (time 1746159924.85 total 164.62 last 166.90) - /unlink 31831 (time 1746159934.85 total 174.62 last 182.97) - /unlink 33848 (time 1746159944.85 total 184.63 last 201.68) - /unlink 35744 (time 1746159954.86 total 194.63 last 189.56) - /unlink 37534 (time 1746159964.86 total 204.63 last 178.97) - /unlink 39309 (time 1746159974.86 total 214.63 last 177.41) - /unlink 40000 (time 1746159978.35 total 218.12 last 198.02) - /unlink 41752 (time 1746159988.35 total 228.13 last 175.17) - /unlink 43447 (time 1746159998.36 total 238.13 last 169.45) - /unlink 45172 (time 1746160008.36 total 248.13 last 172.49) - /unlink 47213 (time 1746160018.36 total 258.13 last 204.10) - /unlink 49000 (time 1746160028.37 total 268.14 last 178.56) - /unlink 50000 (time 1746160033.75 total 273.52 last 185.66) - /unlink 51801 (time 1746160043.76 total 283.53 last 180.02) - /unlink 53701 (time 1746160053.76 total 293.53 last 189.98) - /unlink 55313 (time 1746160063.76 total 303.53 last 161.19) - /unlink 57167 (time 1746160073.76 total 313.53 last 185.34) - /unlink 59009 (time 1746160083.80 total 323.57 last 183.53) - /unlink 60000 (time 1746160089.42 total 329.19 last 176.30) - /unlink 61907 (time 1746160099.42 total 339.19 last 190.65) - /unlink 63693 (time 1746160109.42 total 349.19 last 178.58) total: 64768 /unlink in 355.34 seconds: 182.27 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg246-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg246-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg246-server stop mds service on oleg246-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg246-server stop mds service on oleg246-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg246-server unloading modules via unload_modules_local on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg246-client: oleg246-server: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-client: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-client: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 2 PASS 106 (3939s) Stopping clients: oleg246-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg246-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg246-client: oleg246-server: ssh exited with exit code 2 oleg246-server: oleg246-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing load_modules_local oleg246-server: Loading modules from /home/green/git/lustre-release/lustre oleg246-server: detected 4 online CPUs by sysfs oleg246-server: MODOPTS_LIBCFS= oleg246-server: Force libcfs to create 2 CPU partitions oleg246-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg246-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg246-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 4085 sec ========== 00:29:54 (1746160194) === conf-sanity: start cleanup 00:29:55 (1746160195) === === conf-sanity: finish cleanup 00:29:57 (1746160197) ===