== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:38:33 (1736984313) Stopping clients: oleg328-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg328-client.virtnet /mnt/lustre2 (opts:-f) oleg328-server: oleg328-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg328-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg328-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg328-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 2935 (time 1736984358.04 total 10.00 last 293.43) - open/close 5919 (time 1736984368.04 total 20.01 last 298.31) - open/close 8722 (time 1736984378.04 total 30.01 last 280.23) - open/close 10000 (time 1736984382.63 total 34.60 last 278.54) - open/close 13331 (time 1736984392.63 total 44.60 last 333.06) - open/close 16783 (time 1736984402.63 total 54.60 last 345.19) - open/close 20000 (time 1736984412.32 total 64.29 last 331.98) - open/close 23085 (time 1736984422.32 total 74.29 last 308.48) - open/close 25769 (time 1736984432.32 total 84.29 last 268.40) - open/close 27551 (time 1736984442.33 total 94.29 last 178.11) - open/close 28673 (time 1736984452.34 total 104.30 last 112.12) - open/close 29810 (time 1736984462.34 total 114.30 last 113.67) - open/close 31167 (time 1736984472.34 total 124.31 last 135.65) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 133.67 seconds: 243.83 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 2627 (time 1736984493.24 total 10.00 last 262.65) - /unlink 5485 (time 1736984503.24 total 20.00 last 285.75) - /unlink 8305 (time 1736984513.24 total 30.00 last 282.00) - /unlink 10000 (time 1736984519.13 total 35.89 last 287.81) - /unlink 13899 (time 1736984529.13 total 45.90 last 389.80) - /unlink 17343 (time 1736984539.14 total 55.90 last 344.33) - /unlink 20000 (time 1736984545.63 total 62.39 last 409.35) - /unlink 24463 (time 1736984555.63 total 72.39 last 446.22) - /unlink 29194 (time 1736984565.63 total 82.39 last 473.01) - /unlink 30000 (time 1736984567.70 total 84.46 last 388.90) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 90.18 seconds: 361.45 ops/second Waiting for MDT destroys to complete - open/close 3082 (time 1736984596.46 total 10.00 last 308.12) - open/close 5564 (time 1736984606.46 total 20.01 last 248.13) - open/close 8239 (time 1736984616.46 total 30.01 last 267.48) - open/close 10000 (time 1736984622.69 total 36.23 last 282.73) - open/close 12304 (time 1736984632.69 total 46.24 last 230.39) - open/close 14468 (time 1736984642.70 total 56.24 last 216.32) - open/close 16130 (time 1736984652.70 total 66.24 last 166.12) - open/close 17935 (time 1736984662.70 total 76.25 last 180.46) - open/close 20000 (time 1736984672.46 total 86.01 last 211.57) - open/close 21831 (time 1736984682.47 total 96.01 last 183.06) - open/close 23953 (time 1736984692.47 total 106.01 last 212.17) - open/close 26307 (time 1736984702.47 total 116.01 last 235.35) - open/close 29103 (time 1736984712.47 total 126.01 last 279.59) - open/close 30000 (time 1736984716.31 total 129.85 last 233.56) - open/close 32413 (time 1736984726.31 total 139.86 last 241.23) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 140.64 seconds: 231.75 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 4445 (time 1736984738.27 total 10.00 last 444.46) - /unlink 10000 (time 1736984747.54 total 19.27 last 599.15) - /unlink 16407 (time 1736984757.54 total 29.27 last 640.62) - /unlink 20000 (time 1736984763.73 total 35.46 last 580.52) - /unlink 23169 (time 1736984773.73 total 45.46 last 316.87) - /unlink 25702 (time 1736984783.74 total 55.47 last 253.21) - /unlink 29320 (time 1736984793.74 total 65.47 last 361.69) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 72.84 seconds: 447.48 ops/second Waiting for MDT destroys to complete - open/close 1964 (time 1736984823.09 total 10.00 last 196.35) - open/close 4003 (time 1736984833.09 total 20.00 last 203.85) - open/close 6229 (time 1736984843.10 total 30.01 last 222.58) - open/close 8538 (time 1736984853.10 total 40.01 last 230.84) - open/close 10000 (time 1736984859.66 total 46.57 last 222.66) - open/close 11716 (time 1736984869.67 total 56.58 last 171.54) - open/close 12528 (time 1736984879.67 total 66.58 last 81.17) - open/close 13239 (time 1736984889.68 total 76.59 last 71.06) - open/close 13947 (time 1736984899.69 total 86.60 last 70.71) - open/close 15019 (time 1736984909.69 total 96.60 last 107.19) - open/close 16158 (time 1736984919.70 total 106.61 last 113.78) - open/close 17395 (time 1736984929.70 total 116.62 last 123.65) - open/close 18626 (time 1736984939.71 total 126.62 last 123.01) - open/close 19706 (time 1736984949.72 total 136.63 last 107.87) - open/close 20000 (time 1736984952.64 total 139.55 last 100.72) - open/close 21121 (time 1736984962.64 total 149.55 last 112.09) - open/close 22234 (time 1736984972.65 total 159.56 last 111.29) - open/close 23345 (time 1736984982.65 total 169.56 last 111.01) - open/close 24591 (time 1736984992.66 total 179.57 last 124.55) - open/close 25827 (time 1736985002.66 total 189.57 last 123.57) - open/close 27015 (time 1736985012.67 total 199.58 last 118.72) - open/close 28268 (time 1736985022.67 total 209.58 last 125.27) - open/close 29664 (time 1736985032.67 total 219.58 last 139.52) - open/close 30000 (time 1736985034.93 total 221.84 last 148.85) - open/close 31105 (time 1736985044.94 total 231.85 last 110.42) - open/close 32220 (time 1736985054.94 total 241.85 last 111.44) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 245.24 seconds: 132.91 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1917 (time 1736985070.31 total 10.00 last 191.63) - /unlink 3805 (time 1736985080.32 total 20.00 last 188.78) - /unlink 5670 (time 1736985090.32 total 30.01 last 186.49) - /unlink 7432 (time 1736985100.32 total 40.01 last 176.19) - /unlink 9214 (time 1736985110.32 total 50.01 last 178.13) - /unlink 10000 (time 1736985114.69 total 54.38 last 179.97) - /unlink 11877 (time 1736985124.69 total 64.38 last 187.64) - /unlink 13791 (time 1736985134.69 total 74.38 last 191.39) - /unlink 15727 (time 1736985144.69 total 84.38 last 193.60) - /unlink 17583 (time 1736985154.70 total 94.39 last 185.51) - /unlink 19455 (time 1736985164.70 total 104.39 last 187.16) - /unlink 20000 (time 1736985167.45 total 107.13 last 198.42) - /unlink 21962 (time 1736985177.45 total 117.14 last 196.09) - /unlink 23975 (time 1736985187.45 total 127.14 last 201.22) - /unlink 26137 (time 1736985197.46 total 137.14 last 216.18) - /unlink 28548 (time 1736985207.46 total 147.15 last 240.99) - /unlink 30000 (time 1736985213.03 total 152.71 last 260.92) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 161.99 seconds: 201.21 ops/second Waiting for MDT destroys to complete - open/close 1464 (time 1736985239.00 total 10.00 last 146.37) - open/close 2851 (time 1736985249.00 total 20.01 last 138.63) - open/close 4378 (time 1736985259.01 total 30.01 last 152.69) - open/close 5882 (time 1736985269.01 total 40.01 last 150.38) - open/close 7421 (time 1736985279.01 total 50.01 last 153.88) - open/close 8810 (time 1736985289.02 total 60.02 last 138.77) - open/close 10000 (time 1736985297.87 total 68.87 last 134.45) - open/close 11306 (time 1736985307.87 total 78.87 last 130.56) - open/close 12631 (time 1736985317.87 total 88.87 last 132.49) - open/close 14000 (time 1736985327.87 total 98.88 last 136.85) - open/close 15500 (time 1736985337.88 total 108.88 last 149.98) - open/close 16884 (time 1736985347.88 total 118.88 last 138.31) - open/close 18271 (time 1736985357.88 total 128.89 last 138.69) - open/close 19812 (time 1736985367.89 total 138.89 last 154.05) - open/close 21355 (time 1736985377.89 total 148.89 last 154.30) - open/close 23097 (time 1736985387.89 total 158.89 last 174.19) - open/close 24674 (time 1736985397.89 total 168.89 last 157.62) - open/close 26126 (time 1736985407.89 total 178.90 last 145.19) - open/close 27612 (time 1736985417.90 total 188.90 last 148.50) - open/close 29090 (time 1736985427.91 total 198.91 last 147.71) - open/close 30000 (time 1736985434.11 total 205.11 last 146.75) - open/close 31576 (time 1736985444.11 total 215.11 last 157.59) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 221.18 seconds: 147.36 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 2992 (time 1736985461.45 total 10.00 last 299.17) - /unlink 5958 (time 1736985471.45 total 20.00 last 296.56) - /unlink 8631 (time 1736985481.46 total 30.01 last 267.06) - /unlink 10000 (time 1736985487.15 total 35.71 last 240.33) - /unlink 12927 (time 1736985497.15 total 45.71 last 292.65) - /unlink 15848 (time 1736985507.15 total 55.71 last 292.09) - /unlink 18955 (time 1736985517.16 total 65.71 last 310.66) - /unlink 20000 (time 1736985520.59 total 69.15 last 303.86) - /unlink 23179 (time 1736985530.60 total 79.15 last 317.84) - /unlink 26378 (time 1736985540.60 total 89.15 last 319.83) - /unlink 29558 (time 1736985550.60 total 99.15 last 317.99) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 108.89 seconds: 299.34 ops/second Waiting for MDT destroys to complete - open/close 1825 (time 1736985583.36 total 10.00 last 182.41) - open/close 3441 (time 1736985593.36 total 20.01 last 161.54) - open/close 5023 (time 1736985603.37 total 30.01 last 158.13) - open/close 6958 (time 1736985613.37 total 40.01 last 193.48) - open/close 9135 (time 1736985623.37 total 50.01 last 217.68) - open/close 10000 (time 1736985627.46 total 54.10 last 211.66) - open/close 12148 (time 1736985637.46 total 64.10 last 214.79) - open/close 14169 (time 1736985647.46 total 74.11 last 202.04) - open/close 16212 (time 1736985657.46 total 84.11 last 204.28) - open/close 18152 (time 1736985667.46 total 94.11 last 193.94) - open/close 20000 (time 1736985676.04 total 102.69 last 215.48) - open/close 22645 (time 1736985686.05 total 112.69 last 264.33) - open/close 24693 (time 1736985696.05 total 122.70 last 204.71) - open/close 26406 (time 1736985706.05 total 132.70 last 171.27) - open/close 28535 (time 1736985716.05 total 142.70 last 212.85) - open/close 30000 (time 1736985723.59 total 150.24 last 194.36) - open/close 32056 (time 1736985733.59 total 160.24 last 205.56) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 162.70 seconds: 200.33 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 4317 (time 1736985747.27 total 10.00 last 431.64) - /unlink 8606 (time 1736985757.27 total 20.00 last 428.82) - /unlink 10000 (time 1736985760.83 total 23.57 last 390.99) - /unlink 13680 (time 1736985770.84 total 33.57 last 367.92) - /unlink 18615 (time 1736985780.84 total 43.57 last 493.41) - /unlink 20000 (time 1736985784.10 total 46.84 last 424.35) - /unlink 25333 (time 1736985794.10 total 56.84 last 533.26) - /unlink 29609 (time 1736985804.10 total 66.84 last 427.57) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 74.20 seconds: 439.25 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg328-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg328-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server stop mds service on oleg328-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server stop mds service on oleg328-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg328-server LNET ready to unload unloading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing unload_modules_local oleg328-server: LNET ready to unload modules unloaded. pdsh@oleg328-client: oleg328-client: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-client: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2