-----============= acceptance-small: conf-sanity ============----- Fri May 16 08:43:33 EDT 2025 oleg341-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 110 Stopping clients: oleg341-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg341-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg341-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg341-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg341-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg341-server oleg341-server: oleg341-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing load_modules_local oleg341-server: Loading modules from /home/green/git/lustre-release/lustre oleg341-server: detected 4 online CPUs by sysfs oleg341-server: Force libcfs to create 2 CPU partitions oleg341-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg341-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg341-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg341-server: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg341-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg341-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg341-server stop mds service on oleg341-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg341-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 08:45:08 (1747399508) Stopping clients: oleg341-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg341-client.virtnet /mnt/lustre2 (opts:-f) oleg341-server: oleg341-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing load_modules_local oleg341-server: Loading modules from /home/green/git/lustre-release/lustre oleg341-server: detected 4 online CPUs by sysfs oleg341-server: Force libcfs to create 2 CPU partitions oleg341-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg341-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg341-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg341-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg341-client.virtnet: -o user_xattr,flock oleg341-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16166 358 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1128 (time 1747399573.29 total 10.00 last 112.76) - open/close 2351 (time 1747399583.30 total 20.00 last 122.29) - open/close 3526 (time 1747399593.30 total 30.01 last 117.44) - open/close 4726 (time 1747399603.30 total 40.01 last 119.99) - open/close 5928 (time 1747399613.30 total 50.01 last 120.19) - open/close 7208 (time 1747399623.31 total 60.01 last 127.96) - open/close 8575 (time 1747399633.31 total 70.02 last 136.66) - open/close 9918 (time 1747399643.31 total 80.02 last 134.29) - open/close 11166 (time 1747399653.31 total 90.02 last 124.73) - open/close 12503 (time 1747399663.32 total 100.03 last 133.66) - open/close 13816 (time 1747399673.32 total 110.03 last 131.26) - open/close 15220 (time 1747399683.32 total 120.03 last 140.34) - open/close 16517 (time 1747399693.33 total 130.03 last 129.69) - open/close 17968 (time 1747399703.33 total 140.04 last 145.06) - open/close 19486 (time 1747399713.33 total 150.04 last 151.71) - open/close 20000 (time 1747399717.12 total 153.82 last 135.95) - open/close 21401 (time 1747399727.12 total 163.83 last 140.07) - open/close 22866 (time 1747399737.12 total 173.83 last 146.48) - open/close 24209 (time 1747399747.12 total 183.83 last 134.27) - open/close 25480 (time 1747399757.13 total 193.84 last 127.03) - open/close 26872 (time 1747399767.13 total 203.84 last 139.13) - open/close 28381 (time 1747399777.13 total 213.84 last 150.89) - open/close 29915 (time 1747399787.14 total 223.85 last 153.33) - open/close 31530 (time 1747399797.14 total 233.85 last 161.46) - open/close 33239 (time 1747399807.15 total 243.85 last 170.78) - open/close 34821 (time 1747399817.15 total 253.86 last 158.13) - open/close 36310 (time 1747399827.16 total 263.87 last 148.75) - open/close 37680 (time 1747399837.16 total 273.87 last 136.95) - open/close 39026 (time 1747399847.19 total 283.90 last 134.27) - open/close 40000 (time 1747399854.47 total 291.18 last 133.77) - open/close 41400 (time 1747399864.47 total 301.18 last 139.99) - open/close 42996 (time 1747399874.47 total 311.18 last 159.54) - open/close 44682 (time 1747399884.48 total 321.19 last 168.53) - open/close 46247 (time 1747399894.48 total 331.19 last 156.44) - open/close 47750 (time 1747399904.49 total 341.20 last 150.24) - open/close 49486 (time 1747399914.49 total 351.20 last 173.57) - open/close 50000 (time 1747399917.29 total 354.00 last 183.19) - open/close 51719 (time 1747399927.30 total 364.01 last 171.82) - open/close 53709 (time 1747399937.30 total 374.01 last 199.00) - open/close 55480 (time 1747399947.30 total 384.01 last 177.05) - open/close 57197 (time 1747399957.31 total 394.01 last 171.62) - open/close 59144 (time 1747399967.31 total 404.02 last 194.67) - open/close 60000 (time 1747399972.46 total 409.17 last 166.04) - open/close 61779 (time 1747399982.47 total 419.17 last 177.85) - open/close 63606 (time 1747399992.47 total 429.18 last 182.67) total: 64768 open/close in 435.23 seconds: 148.81 ops/second - /unlink 3511 (time 1747400009.68 total 10.00 last 351.00) - /unlink 6398 (time 1747400019.68 total 20.01 last 288.58) - /unlink 9162 (time 1747400029.68 total 30.01 last 276.33) - /unlink 10000 (time 1747400032.75 total 33.07 last 273.41) - /unlink 13208 (time 1747400042.75 total 43.08 last 320.78) - /unlink 15802 (time 1747400052.75 total 53.08 last 259.38) - /unlink 18975 (time 1747400062.75 total 63.08 last 317.27) - /unlink 20000 (time 1747400065.83 total 66.16 last 332.98) - /unlink 23348 (time 1747400075.83 total 76.16 last 334.76) - /unlink 26742 (time 1747400085.83 total 86.16 last 339.38) - /unlink 30000 (time 1747400094.78 total 95.10 last 364.16) - /unlink 33236 (time 1747400104.78 total 105.10 last 323.59) - /unlink 36119 (time 1747400114.78 total 115.10 last 288.29) - /unlink 39126 (time 1747400124.78 total 125.11 last 300.61) - /unlink 40000 (time 1747400127.71 total 128.03 last 298.83) - /unlink 43177 (time 1747400137.71 total 138.03 last 317.62) - /unlink 46325 (time 1747400147.71 total 148.04 last 314.70) - /unlink 49978 (time 1747400157.71 total 158.04 last 365.21) - /unlink 53593 (time 1747400167.72 total 168.04 last 361.49) - /unlink 58017 (time 1747400177.72 total 178.04 last 442.40) - /unlink 60000 (time 1747400181.46 total 181.79 last 528.90) total: 64768 /unlink in 191.66 seconds: 337.94 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2827 (time 1747400221.68 total 10.00 last 282.66) - open/close 5517 (time 1747400231.68 total 20.00 last 268.98) - open/close 7774 (time 1747400241.69 total 30.00 last 225.66) - open/close 10000 (time 1747400251.27 total 39.59 last 232.18) - open/close 12531 (time 1747400261.27 total 49.59 last 253.10) - open/close 15068 (time 1747400271.28 total 59.60 last 253.61) - open/close 17200 (time 1747400281.28 total 69.60 last 213.17) - open/close 19528 (time 1747400291.28 total 79.60 last 232.79) - open/close 21881 (time 1747400301.28 total 89.60 last 235.29) - open/close 24786 (time 1747400311.28 total 99.60 last 290.48) - open/close 27304 (time 1747400321.28 total 109.60 last 251.71) - open/close 29925 (time 1747400331.28 total 119.60 last 262.07) - open/close 32752 (time 1747400341.28 total 129.60 last 282.67) - open/close 35433 (time 1747400351.29 total 139.61 last 268.03) - open/close 38328 (time 1747400361.29 total 149.61 last 289.49) - open/close 40000 (time 1747400366.45 total 154.76 last 324.12) - open/close 43088 (time 1747400376.45 total 164.77 last 308.63) - open/close 46266 (time 1747400386.45 total 174.77 last 317.75) - open/close 49528 (time 1747400396.45 total 184.77 last 326.19) - open/close 53094 (time 1747400406.45 total 194.77 last 356.54) - open/close 56845 (time 1747400416.46 total 204.78 last 375.02) - open/close 60000 (time 1747400425.16 total 213.48 last 362.44) - open/close 63367 (time 1747400435.16 total 223.48 last 336.65) total: 64768 open/close in 227.82 seconds: 284.30 ops/second - /unlink 6114 (time 1747400450.44 total 10.00 last 611.36) - /unlink 10000 (time 1747400456.12 total 15.68 last 684.36) - /unlink 17490 (time 1747400466.12 total 25.68 last 748.92) - /unlink 20000 (time 1747400469.38 total 28.93 last 771.52) - /unlink 28038 (time 1747400479.38 total 38.93 last 803.70) - /unlink 30000 (time 1747400482.27 total 41.82 last 679.19) - /unlink 36188 (time 1747400492.27 total 51.82 last 618.74) - /unlink 40000 (time 1747400498.88 total 58.43 last 576.81) - /unlink 46775 (time 1747400508.88 total 68.43 last 677.47) - /unlink 50000 (time 1747400514.22 total 73.77 last 603.92) - /unlink 56332 (time 1747400524.22 total 83.77 last 633.15) - /unlink 60000 (time 1747400528.78 total 88.33 last 804.68) total: 64768 /unlink in 94.66 seconds: 684.22 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5154 (time 1747400567.41 total 10.00 last 515.35) - open/close 10000 (time 1747400577.28 total 19.87 last 491.02) - open/close 12833 (time 1747400587.28 total 29.87 last 283.20) - open/close 14368 (time 1747400597.29 total 39.88 last 153.37) - open/close 16070 (time 1747400607.29 total 49.88 last 170.18) - open/close 18817 (time 1747400617.29 total 59.89 last 274.65) - open/close 20000 (time 1747400621.36 total 63.96 last 290.65) - open/close 22612 (time 1747400631.36 total 73.96 last 261.19) - open/close 24830 (time 1747400641.37 total 83.96 last 221.72) - open/close 27131 (time 1747400651.37 total 93.96 last 230.06) - open/close 29271 (time 1747400661.37 total 103.96 last 213.95) - open/close 30000 (time 1747400664.51 total 107.11 last 231.95) - open/close 32874 (time 1747400674.52 total 117.11 last 287.31) - open/close 35234 (time 1747400684.52 total 127.11 last 235.94) - open/close 37341 (time 1747400694.52 total 137.11 last 210.66) - open/close 39606 (time 1747400704.52 total 147.11 last 226.47) - open/close 40000 (time 1747400706.91 total 149.50 last 165.25) - open/close 41919 (time 1747400716.91 total 159.50 last 191.89) - open/close 44378 (time 1747400726.91 total 169.50 last 245.80) - open/close 47317 (time 1747400736.91 total 179.51 last 293.83) - open/close 50000 (time 1747400745.82 total 188.41 last 301.27) - open/close 52732 (time 1747400755.82 total 198.41 last 273.16) - open/close 56163 (time 1747400765.82 total 208.41 last 343.04) - open/close 59852 (time 1747400775.82 total 218.42 last 368.87) - open/close 63432 (time 1747400785.82 total 228.42 last 357.98) total: 64768 open/close in 232.33 seconds: 278.77 ops/second - /unlink 6981 (time 1747400800.72 total 10.00 last 698.03) - /unlink 10000 (time 1747400806.65 total 15.93 last 509.47) - /unlink 14950 (time 1747400816.65 total 25.93 last 495.00) - /unlink 20000 (time 1747400823.93 total 33.21 last 693.67) - /unlink 27191 (time 1747400833.93 total 43.21 last 719.06) - /unlink 30000 (time 1747400838.39 total 47.67 last 629.69) - /unlink 37365 (time 1747400848.39 total 57.67 last 736.39) - /unlink 40000 (time 1747400851.97 total 61.25 last 736.61) - /unlink 47284 (time 1747400861.97 total 71.25 last 728.35) - /unlink 50000 (time 1747400865.95 total 75.23 last 682.02) - /unlink 57496 (time 1747400875.95 total 85.23 last 749.54) - /unlink 60000 (time 1747400879.23 total 88.51 last 764.19) total: 64768 /unlink in 93.92 seconds: 689.58 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3649 (time 1747400918.18 total 10.00 last 364.83) - open/close 8413 (time 1747400928.18 total 20.00 last 476.35) - open/close 10000 (time 1747400931.17 total 22.99 last 531.35) - open/close 14911 (time 1747400941.17 total 32.99 last 491.01) - open/close 19694 (time 1747400951.17 total 42.99 last 478.26) - open/close 24682 (time 1747400961.17 total 52.99 last 498.77) - open/close 30000 (time 1747400970.51 total 62.33 last 569.80) - open/close 35295 (time 1747400980.51 total 72.33 last 529.43) - open/close 40000 (time 1747400989.23 total 81.04 last 539.73) - open/close 45662 (time 1747400999.23 total 91.05 last 566.10) - open/close 50000 (time 1747401006.70 total 98.52 last 580.66) - open/close 56007 (time 1747401016.70 total 108.52 last 600.61) - open/close 60000 (time 1747401025.75 total 117.57 last 441.21) - open/close 64037 (time 1747401035.75 total 127.57 last 403.67) total: 64768 open/close in 129.40 seconds: 500.52 ops/second - /unlink 9357 (time 1747401048.52 total 10.00 last 935.70) - /unlink 17850 (time 1747401058.52 total 20.00 last 849.27) - /unlink 20000 (time 1747401060.92 total 22.40 last 896.91) - /unlink 28155 (time 1747401070.92 total 32.40 last 815.45) - /unlink 30000 (time 1747401072.99 total 34.47 last 892.22) - /unlink 39146 (time 1747401082.99 total 44.47 last 914.50) - /unlink 47518 (time 1747401092.99 total 54.47 last 837.07) - /unlink 50000 (time 1747401096.08 total 57.56 last 802.78) - /unlink 57202 (time 1747401106.08 total 67.56 last 720.14) - /unlink 60000 (time 1747401109.50 total 70.97 last 820.18) total: 64768 /unlink in 76.11 seconds: 850.97 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5581 (time 1747401148.30 total 10.00 last 558.06) - open/close 10000 (time 1747401155.64 total 17.34 last 602.14) - open/close 16275 (time 1747401165.64 total 27.34 last 627.45) - open/close 20000 (time 1747401172.46 total 34.16 last 546.19) - open/close 25149 (time 1747401182.46 total 44.16 last 514.88) - open/close 30000 (time 1747401192.05 total 53.75 last 505.93) - open/close 35165 (time 1747401202.05 total 63.75 last 516.42) - open/close 40000 (time 1747401211.04 total 72.74 last 537.65) - open/close 45475 (time 1747401221.04 total 82.74 last 547.46) - open/close 50000 (time 1747401229.51 total 91.21 last 534.23) - open/close 55334 (time 1747401239.51 total 101.22 last 533.32) - open/close 60000 (time 1747401248.34 total 110.04 last 528.76) total: 64768 open/close in 119.08 seconds: 543.91 ops/second - /unlink 9374 (time 1747401268.25 total 10.00 last 937.39) - /unlink 18836 (time 1747401278.25 total 20.00 last 946.15) - /unlink 28246 (time 1747401288.25 total 30.00 last 940.99) - /unlink 37645 (time 1747401298.25 total 40.00 last 939.89) - /unlink 40000 (time 1747401300.77 total 42.53 last 932.99) - /unlink 49064 (time 1747401310.77 total 52.53 last 906.37) - /unlink 58753 (time 1747401320.77 total 62.53 last 968.88) total: 64768 /unlink in 68.78 seconds: 941.71 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg341-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg341-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg341-server stop mds service on oleg341-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg341-server LNET ready to unload unloading modules on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing unload_modules_local oleg341-server: LNET ready to unload modules unloaded. pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2 PASS 106 (1864s) Stopping clients: oleg341-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg341-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2 oleg341-server: oleg341-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing load_modules_local oleg341-server: Loading modules from /home/green/git/lustre-release/lustre oleg341-server: detected 4 online CPUs by sysfs oleg341-server: Force libcfs to create 2 CPU partitions oleg341-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg341-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg341-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1977 sec ========== 09:16:31 (1747401391)