== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:16:32 (1736982992) Stopping clients: oleg120-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg120-client.virtnet /mnt/lustre2 (opts:-f) oleg120-server: oleg120-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: detected 4 online CPUs by sysfs oleg120-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg120-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg120-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg120-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg120-client.virtnet: -o user_xattr,flock oleg120-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 302 32594 1% /mnt/lustre[OST:0] filesystem_summary: 33123 529 32594 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 2307 (time 1736983040.82 total 10.00 last 230.58) - open/close 4965 (time 1736983050.82 total 20.01 last 265.80) - open/close 7366 (time 1736983060.82 total 30.01 last 240.09) - open/close 8839 (time 1736983070.82 total 40.01 last 147.25) - open/close 10000 (time 1736983078.98 total 48.17 last 142.31) - open/close 11548 (time 1736983088.98 total 58.17 last 154.73) - open/close 13544 (time 1736983098.99 total 68.17 last 199.58) - open/close 15312 (time 1736983108.99 total 78.18 last 176.73) - open/close 16996 (time 1736983118.99 total 88.18 last 168.36) - open/close 18712 (time 1736983128.99 total 98.18 last 171.57) - open/close 20000 (time 1736983134.95 total 104.13 last 216.39) - open/close 22384 (time 1736983144.95 total 114.14 last 238.32) - open/close 24569 (time 1736983154.95 total 124.14 last 218.41) - open/close 26147 (time 1736983164.96 total 134.15 last 157.62) - open/close 27675 (time 1736983174.97 total 144.15 last 152.76) - open/close 29182 (time 1736983184.97 total 154.16 last 150.62) - open/close 30000 (time 1736983189.52 total 158.71 last 179.95) - open/close 31871 (time 1736983199.52 total 168.71 last 187.08) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 172.05 seconds: 189.45 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 3584 (time 1736983214.11 total 10.00 last 358.33) - /unlink 7210 (time 1736983224.11 total 20.00 last 362.52) - /unlink 10000 (time 1736983232.51 total 28.40 last 332.30) - /unlink 13825 (time 1736983242.51 total 38.40 last 382.48) - /unlink 17555 (time 1736983252.51 total 48.40 last 372.96) - /unlink 20000 (time 1736983257.49 total 53.38 last 491.24) - /unlink 24337 (time 1736983267.49 total 63.38 last 433.66) - /unlink 29658 (time 1736983277.49 total 73.38 last 532.10) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 78.45 seconds: 415.49 ops/second Waiting for MDT destroys to complete - open/close 2043 (time 1736983305.23 total 10.00 last 204.22) - open/close 4406 (time 1736983315.23 total 20.01 last 236.25) - open/close 6227 (time 1736983325.24 total 30.01 last 182.04) - open/close 7889 (time 1736983335.24 total 40.01 last 166.20) - open/close 10000 (time 1736983344.12 total 48.90 last 237.50) - open/close 12409 (time 1736983354.12 total 58.90 last 240.90) - open/close 15171 (time 1736983364.13 total 68.90 last 276.18) - open/close 17802 (time 1736983374.13 total 78.90 last 263.09) - open/close 19719 (time 1736983384.13 total 88.90 last 191.69) - open/close 22354 (time 1736983394.13 total 98.90 last 263.47) - open/close 24688 (time 1736983404.13 total 108.90 last 233.36) - open/close 27575 (time 1736983414.13 total 118.90 last 288.69) - open/close 30000 (time 1736983423.06 total 127.84 last 271.44) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 137.72 seconds: 236.68 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 5144 (time 1736983444.07 total 10.00 last 514.36) - /unlink 9435 (time 1736983454.07 total 20.00 last 429.06) - /unlink 13854 (time 1736983464.07 total 30.00 last 441.88) - /unlink 18262 (time 1736983474.07 total 40.00 last 440.74) - /unlink 20000 (time 1736983477.38 total 43.31 last 525.74) - /unlink 24554 (time 1736983487.38 total 53.31 last 455.32) - /unlink 30000 (time 1736983496.78 total 62.71 last 579.29) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 67.85 seconds: 480.39 ops/second Waiting for MDT destroys to complete - open/close 2723 (time 1736983519.75 total 10.00 last 272.26) - open/close 5820 (time 1736983529.75 total 20.00 last 309.69) - open/close 8596 (time 1736983539.76 total 30.00 last 277.56) - open/close 10000 (time 1736983544.06 total 34.31 last 326.00) - open/close 13627 (time 1736983554.06 total 44.31 last 362.63) - open/close 16732 (time 1736983564.07 total 54.32 last 310.38) - open/close 19552 (time 1736983574.07 total 64.32 last 281.99) - open/close 22615 (time 1736983584.07 total 74.32 last 306.28) - open/close 25268 (time 1736983594.07 total 84.32 last 265.19) - open/close 27739 (time 1736983604.08 total 94.32 last 247.03) - open/close 30000 (time 1736983612.90 total 103.15 last 256.14) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 112.79 seconds: 288.99 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 6708 (time 1736983633.56 total 10.00 last 670.77) - /unlink 10000 (time 1736983639.08 total 15.52 last 596.22) - /unlink 15372 (time 1736983649.08 total 25.52 last 537.16) - /unlink 20000 (time 1736983656.23 total 32.67 last 647.76) - /unlink 27304 (time 1736983666.23 total 42.67 last 730.30) - /unlink 30000 (time 1736983670.39 total 46.83 last 648.55) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 51.87 seconds: 628.36 ops/second Waiting for MDT destroys to complete - open/close 3902 (time 1736983695.50 total 10.00 last 390.17) - open/close 8790 (time 1736983705.50 total 20.00 last 488.76) - open/close 10000 (time 1736983707.77 total 22.27 last 532.35) - open/close 14496 (time 1736983717.77 total 32.28 last 449.54) - open/close 18605 (time 1736983727.78 total 42.28 last 410.82) - open/close 20000 (time 1736983731.06 total 45.57 last 424.22) - open/close 23552 (time 1736983741.07 total 55.57 last 355.14) - open/close 27295 (time 1736983751.07 total 65.57 last 374.28) - open/close 30000 (time 1736983757.22 total 71.72 last 439.96) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 77.96 seconds: 418.07 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 8019 (time 1736983774.37 total 10.00 last 801.87) - /unlink 10000 (time 1736983777.00 total 12.64 last 751.67) - /unlink 17086 (time 1736983787.00 total 22.64 last 708.56) - /unlink 20000 (time 1736983790.25 total 25.88 last 898.49) - /unlink 27789 (time 1736983800.25 total 35.88 last 778.81) - /unlink 30000 (time 1736983802.81 total 38.44 last 863.60) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 41.17 seconds: 791.68 ops/second Waiting for MDT destroys to complete - open/close 5116 (time 1736983844.68 total 10.00 last 511.55) - open/close 10000 (time 1736983853.61 total 18.93 last 547.10) - open/close 14733 (time 1736983863.61 total 28.93 last 473.19) - open/close 19529 (time 1736983873.61 total 38.93 last 479.54) - open/close 24750 (time 1736983883.62 total 48.93 last 522.05) - open/close 30000 (time 1736983892.97 total 58.28 last 561.46) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 63.52 seconds: 513.17 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 9739 (time 1736983909.17 total 10.00 last 973.78) - /unlink 19172 (time 1736983919.17 total 20.00 last 943.19) - /unlink 27920 (time 1736983929.17 total 30.00 last 874.78) - /unlink 30000 (time 1736983931.46 total 32.30 last 907.08) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 34.95 seconds: 932.53 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg120-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg120-server LNET ready to unload unloading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing unload_modules_local oleg120-server: LNET ready to unload modules unloaded. pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2