-----============= acceptance-small: conf-sanity ============----- Wed Jan 15 18:36:33 EST 2025 excepting tests: 32newtarball 110 Stopping clients: oleg254-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg254-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg254-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg254-server oleg254-server: oleg254-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: detected 4 online CPUs by sysfs oleg254-server: Force libcfs to create 2 CPU partitions oleg254-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg254-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg254-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg254-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:37:40 (1736984260) Stopping clients: oleg254-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg254-client.virtnet /mnt/lustre2 (opts:-f) oleg254-server: oleg254-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: detected 4 online CPUs by sysfs oleg254-server: Force libcfs to create 2 CPU partitions oleg254-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg254-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg254-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg254-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg254-client.virtnet: -o user_xattr,flock oleg254-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16131 291 15840 2% /mnt/lustre[OST:0] filesystem_summary: 16169 329 15840 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 5398 (time 1736984295.17 total 10.00 last 539.71) - open/close 10000 (time 1736984304.77 total 19.61 last 478.99) - open/close 15463 (time 1736984314.77 total 29.61 last 546.23) - open/close 20000 (time 1736984323.64 total 38.47 last 512.02) - open/close 24965 (time 1736984333.64 total 48.47 last 496.49) - open/close 29875 (time 1736984343.64 total 58.47 last 490.91) - open/close 35044 (time 1736984353.64 total 68.47 last 516.89) - open/close 40000 (time 1736984363.25 total 78.09 last 515.55) - open/close 45119 (time 1736984373.25 total 88.09 last 511.82) - open/close 50000 (time 1736984382.31 total 97.14 last 539.08) - open/close 55224 (time 1736984392.31 total 107.14 last 522.37) - open/close 59749 (time 1736984402.31 total 117.15 last 452.42) total: 64768 open/close in 126.77 seconds: 510.90 ops/second - /unlink 9161 (time 1736984422.84 total 10.00 last 916.01) - /unlink 18381 (time 1736984432.84 total 20.00 last 921.91) - /unlink 24463 (time 1736984442.84 total 30.00 last 608.16) - /unlink 26963 (time 1736984452.84 total 40.01 last 249.92) - /unlink 29142 (time 1736984462.84 total 50.01 last 217.89) - /unlink 34868 (time 1736984472.84 total 60.01 last 572.54) - /unlink 39828 (time 1736984482.85 total 70.01 last 495.94) - /unlink 43015 (time 1736984492.85 total 80.01 last 318.54) - /unlink 45682 (time 1736984502.85 total 90.01 last 266.70) - /unlink 49193 (time 1736984512.85 total 100.02 last 351.02) - /unlink 50000 (time 1736984515.52 total 102.69 last 302.23) - /unlink 53861 (time 1736984525.52 total 112.69 last 386.05) - /unlink 59393 (time 1736984535.52 total 122.69 last 553.17) - /unlink 64186 (time 1736984545.53 total 132.69 last 479.25) total: 64768 /unlink in 134.52 seconds: 481.46 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3286 (time 1736984576.01 total 10.00 last 328.53) - open/close 7418 (time 1736984586.01 total 20.00 last 413.20) - open/close 10000 (time 1736984592.03 total 26.02 last 429.11) - open/close 15214 (time 1736984602.03 total 36.02 last 521.31) - open/close 20000 (time 1736984611.55 total 45.54 last 502.66) - open/close 23889 (time 1736984621.55 total 55.54 last 388.82) - open/close 27543 (time 1736984631.55 total 65.54 last 365.39) - open/close 30000 (time 1736984639.15 total 73.14 last 323.46) - open/close 33134 (time 1736984649.15 total 83.14 last 313.33) - open/close 35485 (time 1736984659.15 total 93.15 last 235.01) - open/close 38666 (time 1736984669.15 total 103.15 last 318.09) - open/close 40000 (time 1736984672.35 total 106.35 last 416.93) - open/close 43907 (time 1736984682.35 total 116.35 last 390.70) - open/close 47262 (time 1736984692.35 total 126.35 last 335.47) - open/close 50000 (time 1736984697.87 total 131.87 last 495.92) - open/close 54788 (time 1736984707.88 total 141.87 last 478.79) - open/close 59975 (time 1736984717.93 total 151.93 last 515.77) total: 64768 open/close in 160.82 seconds: 402.74 ops/second - /unlink 8531 (time 1736984737.65 total 10.00 last 853.06) - /unlink 18418 (time 1736984747.65 total 20.00 last 988.67) - /unlink 28698 (time 1736984757.65 total 30.00 last 1028.00) - /unlink 37030 (time 1736984767.65 total 40.00 last 833.18) - /unlink 39965 (time 1736984777.65 total 50.00 last 293.49) - /unlink 43443 (time 1736984787.65 total 60.00 last 347.79) - /unlink 50000 (time 1736984797.18 total 69.53 last 688.11) - /unlink 57036 (time 1736984807.19 total 79.53 last 703.50) - /unlink 60000 (time 1736984814.24 total 86.59 last 420.06) - /unlink 64120 (time 1736984824.24 total 96.59 last 411.98) total: 64768 /unlink in 97.68 seconds: 663.06 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3441 (time 1736984855.72 total 10.00 last 344.05) - open/close 7729 (time 1736984865.73 total 20.00 last 428.74) - open/close 9135 (time 1736984875.73 total 30.00 last 140.59) - open/close 10000 (time 1736984883.18 total 37.46 last 116.03) - open/close 11046 (time 1736984893.18 total 47.46 last 104.56) - open/close 12295 (time 1736984903.19 total 57.46 last 124.87) - open/close 13787 (time 1736984913.19 total 67.46 last 149.20) - open/close 15256 (time 1736984923.19 total 77.47 last 146.85) - open/close 16756 (time 1736984933.19 total 87.47 last 149.97) - open/close 18069 (time 1736984943.20 total 97.47 last 131.26) - open/close 19530 (time 1736984953.20 total 107.48 last 146.07) - open/close 20000 (time 1736984956.28 total 110.56 last 152.29) - open/close 21772 (time 1736984966.29 total 120.57 last 177.12) - open/close 23678 (time 1736984976.29 total 130.57 last 190.60) - open/close 26237 (time 1736984986.29 total 140.57 last 255.87) - open/close 27716 (time 1736984996.29 total 150.57 last 147.89) - open/close 29604 (time 1736985006.29 total 160.57 last 188.76) - open/close 30000 (time 1736985008.64 total 162.92 last 168.72) - open/close 31421 (time 1736985018.64 total 172.92 last 142.08) - open/close 32859 (time 1736985028.64 total 182.92 last 143.77) - open/close 34416 (time 1736985038.65 total 192.93 last 155.58) - open/close 35883 (time 1736985048.65 total 202.93 last 146.65) - open/close 37359 (time 1736985058.65 total 212.93 last 147.59) - open/close 38820 (time 1736985068.66 total 222.94 last 146.02) - open/close 40000 (time 1736985075.92 total 230.20 last 162.39) - open/close 41726 (time 1736985085.93 total 240.20 last 172.58) - open/close 43328 (time 1736985095.93 total 250.20 last 160.18) - open/close 45122 (time 1736985105.93 total 260.21 last 179.29) - open/close 46719 (time 1736985115.94 total 270.22 last 159.63) - open/close 48186 (time 1736985125.94 total 280.22 last 146.68) - open/close 50000 (time 1736985135.87 total 290.15 last 182.60) - open/close 51646 (time 1736985145.88 total 300.15 last 164.57) - open/close 53247 (time 1736985155.88 total 310.16 last 160.02) - open/close 55007 (time 1736985165.88 total 320.16 last 175.99) - open/close 56600 (time 1736985175.89 total 330.16 last 159.23) - open/close 58251 (time 1736985185.89 total 340.16 last 165.08) - open/close 59834 (time 1736985195.89 total 350.17 last 158.24) - open/close 61344 (time 1736985205.90 total 360.17 last 150.92) - open/close 63007 (time 1736985215.90 total 370.18 last 166.27) - open/close 64570 (time 1736985225.90 total 380.18 last 156.25) total: 64768 open/close in 381.52 seconds: 169.76 ops/second - /unlink 2970 (time 1736985238.86 total 10.00 last 296.96) - /unlink 5438 (time 1736985248.87 total 20.00 last 246.72) - /unlink 8280 (time 1736985258.87 total 30.01 last 284.16) - /unlink 10000 (time 1736985264.36 total 35.50 last 313.24) - /unlink 13832 (time 1736985274.36 total 45.50 last 383.11) - /unlink 17050 (time 1736985284.36 total 55.50 last 321.76) - /unlink 20000 (time 1736985292.90 total 64.04 last 345.47) - /unlink 22781 (time 1736985302.90 total 74.04 last 277.97) - /unlink 26793 (time 1736985312.91 total 84.05 last 401.16) - /unlink 30000 (time 1736985320.89 total 92.03 last 401.43) - /unlink 33320 (time 1736985330.90 total 102.04 last 331.97) - /unlink 36776 (time 1736985340.90 total 112.04 last 345.54) - /unlink 40000 (time 1736985349.86 total 121.00 last 359.75) - /unlink 43888 (time 1736985359.86 total 131.00 last 388.72) - /unlink 47260 (time 1736985369.86 total 141.00 last 337.12) - /unlink 50000 (time 1736985376.57 total 147.71 last 408.74) - /unlink 54345 (time 1736985386.57 total 157.71 last 434.43) - /unlink 58175 (time 1736985396.57 total 167.71 last 382.96) - /unlink 60000 (time 1736985401.02 total 172.16 last 409.76) - /unlink 64107 (time 1736985411.02 total 182.16 last 410.68) total: 64768 /unlink in 183.67 seconds: 352.64 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 1950 (time 1736985441.61 total 10.00 last 194.94) - open/close 4374 (time 1736985451.62 total 20.01 last 242.34) - open/close 6541 (time 1736985461.62 total 30.01 last 216.68) - open/close 8826 (time 1736985471.62 total 40.01 last 228.39) - open/close 10000 (time 1736985477.07 total 45.46 last 215.55) - open/close 11878 (time 1736985487.07 total 55.46 last 187.80) - open/close 14167 (time 1736985497.07 total 65.46 last 228.87) - open/close 16642 (time 1736985507.07 total 75.46 last 247.44) - open/close 19019 (time 1736985517.14 total 85.53 last 236.10) - open/close 20000 (time 1736985522.25 total 90.64 last 192.11) - open/close 22031 (time 1736985532.25 total 100.64 last 203.08) - open/close 24021 (time 1736985542.25 total 110.64 last 198.99) - open/close 26439 (time 1736985552.25 total 120.64 last 241.77) - open/close 28408 (time 1736985562.25 total 130.64 last 196.85) - open/close 30000 (time 1736985569.40 total 137.79 last 222.59) - open/close 32075 (time 1736985579.41 total 147.80 last 207.48) - open/close 34267 (time 1736985589.41 total 157.80 last 219.15) - open/close 37381 (time 1736985599.41 total 167.80 last 311.31) - open/close 40000 (time 1736985607.31 total 175.70 last 331.54) - open/close 43550 (time 1736985617.31 total 185.70 last 354.99) - open/close 47014 (time 1736985627.31 total 195.70 last 346.39) - open/close 49485 (time 1736985637.31 total 205.70 last 247.01) - open/close 52246 (time 1736985647.32 total 215.71 last 276.05) - open/close 54936 (time 1736985657.32 total 225.71 last 268.97) - open/close 57653 (time 1736985667.32 total 235.71 last 271.69) - open/close 60000 (time 1736985675.36 total 243.75 last 291.71) - open/close 63001 (time 1736985685.37 total 253.76 last 300.04) total: 64768 open/close in 258.64 seconds: 250.42 ops/second - /unlink 6132 (time 1736985701.28 total 10.00 last 613.18) - /unlink 10000 (time 1736985708.33 total 17.05 last 548.95) - /unlink 14888 (time 1736985718.33 total 27.05 last 488.75) - /unlink 19872 (time 1736985728.33 total 37.05 last 498.35) - /unlink 25571 (time 1736985738.33 total 47.05 last 569.82) - /unlink 30000 (time 1736985746.78 total 55.50 last 524.31) - /unlink 35483 (time 1736985756.78 total 65.50 last 548.24) - /unlink 40000 (time 1736985763.63 total 72.35 last 659.48) - /unlink 46692 (time 1736985773.63 total 82.35 last 669.13) - /unlink 50000 (time 1736985779.12 total 87.84 last 601.89) - /unlink 54632 (time 1736985789.12 total 97.85 last 463.16) - /unlink 60000 (time 1736985798.09 total 106.82 last 598.47) total: 64768 /unlink in 116.43 seconds: 556.30 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3312 (time 1736985836.87 total 10.00 last 331.17) - open/close 6637 (time 1736985846.87 total 20.01 last 332.36) - open/close 9737 (time 1736985856.87 total 30.01 last 309.94) - open/close 13470 (time 1736985866.88 total 40.01 last 373.25) - open/close 17119 (time 1736985876.88 total 50.01 last 364.88) - open/close 19988 (time 1736985886.88 total 60.01 last 286.86) - open/close 24726 (time 1736985896.88 total 70.01 last 473.73) - open/close 28455 (time 1736985906.88 total 80.01 last 372.89) - open/close 30000 (time 1736985911.46 total 84.59 last 337.42) - open/close 34021 (time 1736985921.46 total 94.59 last 401.94) - open/close 37750 (time 1736985931.46 total 104.60 last 372.80) - open/close 40000 (time 1736985936.59 total 109.73 last 438.71) - open/close 44774 (time 1736985946.59 total 119.73 last 477.35) - open/close 49227 (time 1736985956.60 total 129.73 last 445.19) - open/close 54009 (time 1736985966.60 total 139.73 last 478.13) - open/close 59214 (time 1736985976.60 total 149.73 last 520.35) - open/close 64421 (time 1736985986.60 total 159.73 last 520.67) total: 64768 open/close in 160.54 seconds: 403.44 ops/second - /unlink 8859 (time 1736985998.28 total 10.00 last 885.81) - /unlink 16788 (time 1736986008.28 total 20.00 last 792.83) - /unlink 20000 (time 1736986013.15 total 24.87 last 659.57) - /unlink 26901 (time 1736986023.15 total 34.87 last 690.09) - /unlink 30000 (time 1736986026.96 total 38.68 last 813.23) - /unlink 39046 (time 1736986036.96 total 48.68 last 904.56) - /unlink 47037 (time 1736986046.96 total 58.68 last 799.03) - /unlink 50000 (time 1736986051.06 total 62.78 last 723.19) - /unlink 57828 (time 1736986061.06 total 72.78 last 782.77) - /unlink 60000 (time 1736986063.69 total 75.41 last 826.44) total: 64768 /unlink in 80.88 seconds: 800.77 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg254-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server LNET ready to unload unloading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing unload_modules_local oleg254-server: LNET ready to unload modules unloaded. pdsh@oleg254-client: oleg254-client: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-client: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 PASS 106 (1850s) Stopping clients: oleg254-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg254-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 oleg254-server: oleg254-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: detected 4 online CPUs by sysfs oleg254-server: Force libcfs to create 2 CPU partitions oleg254-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg254-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1935 sec ========== 19:08:48 (1736986128)