== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 08:47:10 (1747399630) Stopping clients: oleg231-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg231-client.virtnet /mnt/lustre2 (opts:-f) oleg231-server: oleg231-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing load_modules_local oleg231-server: Loading modules from /home/green/git/lustre-release/lustre oleg231-server: detected 4 online CPUs by sysfs oleg231-server: Force libcfs to create 2 CPU partitions oleg231-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg231-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg231-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg231-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg231-client.virtnet: -o user_xattr,flock oleg231-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 2046 (time 1747399691.53 total 10.00 last 204.54) - open/close 3777 (time 1747399701.54 total 20.01 last 173.04) - open/close 5548 (time 1747399711.54 total 30.01 last 177.05) - open/close 7433 (time 1747399721.54 total 40.01 last 188.42) - open/close 9251 (time 1747399731.55 total 50.02 last 181.73) - open/close 10000 (time 1747399735.99 total 54.46 last 168.46) - open/close 12125 (time 1747399746.00 total 64.47 last 212.41) - open/close 14259 (time 1747399756.00 total 74.47 last 213.34) - open/close 16410 (time 1747399766.00 total 84.47 last 215.09) - open/close 18188 (time 1747399776.01 total 94.48 last 177.69) - open/close 20000 (time 1747399785.79 total 104.27 last 185.11) - open/close 21634 (time 1747399795.80 total 114.27 last 163.38) - open/close 23143 (time 1747399805.80 total 124.27 last 150.86) - open/close 24797 (time 1747399815.80 total 134.27 last 165.40) - open/close 26670 (time 1747399825.80 total 144.27 last 187.24) - open/close 28494 (time 1747399835.81 total 154.28 last 182.33) - open/close 30000 (time 1747399843.44 total 161.91 last 197.32) - open/close 31712 (time 1747399853.44 total 171.91 last 171.12) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 177.64 seconds: 183.48 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 2892 (time 1747399870.70 total 10.00 last 289.12) - /unlink 5541 (time 1747399880.70 total 20.01 last 264.82) - /unlink 8205 (time 1747399890.70 total 30.01 last 266.36) - /unlink 10000 (time 1747399897.06 total 36.36 last 282.46) - /unlink 12855 (time 1747399907.06 total 46.36 last 285.47) - /unlink 15911 (time 1747399917.06 total 56.37 last 305.50) - /unlink 18495 (time 1747399927.06 total 66.37 last 258.26) - /unlink 20000 (time 1747399932.25 total 71.56 last 290.06) - /unlink 22807 (time 1747399942.25 total 81.56 last 280.70) - /unlink 25915 (time 1747399952.25 total 91.56 last 310.79) - /unlink 28847 (time 1747399962.26 total 101.56 last 293.15) - /unlink 30000 (time 1747399966.20 total 105.51 last 292.23) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 115.02 seconds: 283.39 ops/second Waiting for MDT destroys to complete - open/close 1728 (time 1747399998.02 total 10.00 last 172.74) - open/close 3302 (time 1747400008.03 total 20.00 last 157.38) - open/close 4935 (time 1747400018.03 total 30.01 last 163.29) - open/close 6678 (time 1747400028.03 total 40.01 last 174.26) - open/close 8356 (time 1747400038.03 total 50.01 last 167.76) - open/close 9960 (time 1747400048.03 total 60.01 last 160.38) - open/close 11696 (time 1747400058.03 total 70.01 last 173.60) - open/close 13361 (time 1747400068.03 total 80.01 last 166.49) - open/close 15019 (time 1747400078.03 total 90.01 last 165.78) - open/close 16648 (time 1747400088.04 total 100.02 last 162.77) - open/close 18527 (time 1747400098.04 total 110.02 last 187.89) - open/close 20000 (time 1747400105.69 total 117.67 last 192.59) - open/close 22148 (time 1747400115.69 total 127.67 last 214.78) - open/close 24296 (time 1747400125.70 total 137.68 last 214.69) - open/close 26151 (time 1747400135.70 total 147.68 last 185.42) - open/close 27843 (time 1747400145.70 total 157.68 last 169.19) - open/close 29568 (time 1747400155.71 total 167.68 last 172.46) - open/close 30000 (time 1747400158.45 total 170.42 last 157.65) - open/close 31712 (time 1747400168.45 total 180.43 last 171.17) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 184.57 seconds: 176.59 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 3418 (time 1747400183.80 total 10.00 last 341.70) - /unlink 7947 (time 1747400193.80 total 20.00 last 452.89) - /unlink 10000 (time 1747400198.42 total 24.63 last 444.00) - /unlink 13911 (time 1747400208.42 total 34.63 last 391.08) - /unlink 17385 (time 1747400218.42 total 44.63 last 347.38) - /unlink 20000 (time 1747400226.07 total 52.27 last 341.99) - /unlink 22887 (time 1747400236.07 total 62.28 last 288.64) - /unlink 26006 (time 1747400246.07 total 72.28 last 311.86) - /unlink 29706 (time 1747400256.08 total 82.28 last 369.91) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 90.01 seconds: 362.12 ops/second Waiting for MDT destroys to complete - open/close 2383 (time 1747400284.78 total 10.00 last 238.24) - open/close 4410 (time 1747400294.79 total 20.01 last 202.65) - open/close 6651 (time 1747400304.79 total 30.01 last 224.07) - open/close 8926 (time 1747400314.79 total 40.01 last 227.44) - open/close 10000 (time 1747400319.24 total 44.45 last 241.68) - open/close 11918 (time 1747400329.24 total 54.45 last 191.79) - open/close 13996 (time 1747400339.24 total 64.46 last 207.75) - open/close 16767 (time 1747400349.24 total 74.46 last 277.06) - open/close 19687 (time 1747400359.24 total 84.46 last 291.98) - open/close 22621 (time 1747400369.24 total 94.46 last 293.37) - open/close 25215 (time 1747400379.25 total 104.46 last 259.28) - open/close 27723 (time 1747400389.25 total 114.47 last 250.75) - open/close 29612 (time 1747400399.25 total 124.47 last 188.85) - open/close 30000 (time 1747400401.40 total 126.62 last 180.23) - open/close 32148 (time 1747400411.40 total 136.62 last 214.75) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 138.73 seconds: 234.94 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 3850 (time 1747400424.73 total 10.00 last 384.93) - /unlink 7260 (time 1747400434.73 total 20.00 last 341.00) - /unlink 10000 (time 1747400442.07 total 27.34 last 373.53) - /unlink 14063 (time 1747400452.07 total 37.34 last 406.22) - /unlink 18708 (time 1747400462.07 total 47.34 last 464.49) - /unlink 20000 (time 1747400464.56 total 49.83 last 519.54) - /unlink 25162 (time 1747400474.56 total 59.83 last 516.20) - /unlink 29613 (time 1747400484.56 total 69.83 last 445.02) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 76.21 seconds: 427.71 ops/second Waiting for MDT destroys to complete - open/close 3703 (time 1747400509.53 total 10.00 last 370.27) - open/close 6962 (time 1747400519.53 total 20.00 last 325.88) - open/close 10000 (time 1747400529.41 total 29.88 last 307.50) - open/close 12792 (time 1747400539.41 total 39.88 last 279.16) - open/close 15763 (time 1747400549.41 total 49.89 last 297.03) - open/close 19364 (time 1747400559.41 total 59.89 last 360.06) - open/close 22852 (time 1747400569.41 total 69.89 last 348.78) - open/close 26564 (time 1747400579.41 total 79.89 last 371.17) - open/close 28915 (time 1747400589.42 total 89.89 last 235.02) - open/close 30000 (time 1747400597.48 total 97.95 last 134.61) - open/close 31849 (time 1747400607.48 total 107.95 last 184.86) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 111.15 seconds: 293.24 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 4146 (time 1747400621.83 total 10.00 last 414.59) - /unlink 8228 (time 1747400631.83 total 20.00 last 408.19) - /unlink 10000 (time 1747400637.05 total 25.21 last 339.87) - /unlink 13762 (time 1747400647.05 total 35.21 last 376.19) - /unlink 17186 (time 1747400657.05 total 45.22 last 342.37) - /unlink 20000 (time 1747400665.52 total 53.68 last 332.26) - /unlink 22872 (time 1747400675.52 total 63.69 last 287.17) - /unlink 25944 (time 1747400685.52 total 73.69 last 307.17) - /unlink 28463 (time 1747400695.52 total 83.69 last 251.89) - /unlink 30000 (time 1747400701.01 total 89.18 last 279.77) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 98.20 seconds: 331.93 ops/second Waiting for MDT destroys to complete - open/close 1432 (time 1747400731.29 total 10.00 last 143.16) - open/close 3343 (time 1747400741.29 total 20.01 last 191.05) - open/close 5280 (time 1747400751.29 total 30.01 last 193.67) - open/close 7562 (time 1747400761.29 total 40.01 last 228.12) - open/close 10000 (time 1747400770.80 total 49.52 last 256.49) - open/close 12199 (time 1747400780.80 total 59.52 last 219.89) - open/close 14856 (time 1747400790.80 total 69.52 last 265.66) - open/close 18678 (time 1747400800.80 total 79.52 last 382.11) - open/close 20000 (time 1747400804.21 total 82.93 last 387.90) - open/close 22883 (time 1747400814.21 total 92.93 last 288.29) - open/close 25217 (time 1747400824.22 total 102.93 last 233.31) - open/close 27776 (time 1747400834.22 total 112.93 last 255.86) - open/close 30000 (time 1747400840.73 total 119.45 last 341.34) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 128.35 seconds: 253.94 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 5067 (time 1747400860.62 total 10.00 last 506.56) - /unlink 10000 (time 1747400870.06 total 19.43 last 523.02) - /unlink 14974 (time 1747400880.06 total 29.43 last 497.38) - /unlink 20000 (time 1747400889.61 total 38.99 last 525.83) - /unlink 26552 (time 1747400899.61 total 48.99 last 655.18) - /unlink 30000 (time 1747400903.86 total 53.24 last 811.19) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 56.51 seconds: 576.82 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg231-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg231-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg231-server stop mds service on oleg231-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg231-server stop mds service on oleg231-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg231-server LNET ready to unload unloading modules on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing unload_modules_local oleg231-server: LNET ready to unload modules unloaded. pdsh@oleg231-client: oleg231-client: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-client: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2