== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:24:52 (1736263492) Stopping clients: oleg447-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg447-client.virtnet /mnt/lustre2 (opts:-f) oleg447-server: oleg447-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing load_modules_local oleg447-server: Loading modules from /home/green/git/lustre-release/lustre oleg447-server: detected 4 online CPUs by sysfs oleg447-server: Force libcfs to create 2 CPU partitions oleg447-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg447-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg447-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg447-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg447-client.virtnet: -o user_xattr,flock oleg447-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1581 (time 1736263555.69 total 10.00 last 158.07) - open/close 3132 (time 1736263565.69 total 20.00 last 155.07) - open/close 4665 (time 1736263575.70 total 30.01 last 153.23) - open/close 6200 (time 1736263585.71 total 40.01 last 153.40) - open/close 7673 (time 1736263595.71 total 50.02 last 147.24) - open/close 9021 (time 1736263605.71 total 60.02 last 134.78) - open/close 10000 (time 1736263612.27 total 66.58 last 149.32) - open/close 11428 (time 1736263622.28 total 76.58 last 142.68) - open/close 12889 (time 1736263632.28 total 86.59 last 146.09) - open/close 14536 (time 1736263642.28 total 96.59 last 164.70) - open/close 15937 (time 1736263652.28 total 106.59 last 140.03) - open/close 17770 (time 1736263662.29 total 116.60 last 183.18) - open/close 19881 (time 1736263672.29 total 126.60 last 211.06) - open/close 22359 (time 1736263682.29 total 136.60 last 247.70) - open/close 24598 (time 1736263692.29 total 146.60 last 223.87) - open/close 26773 (time 1736263702.30 total 156.60 last 217.49) - open/close 29407 (time 1736263712.30 total 166.61 last 263.34) - open/close 30000 (time 1736263715.04 total 169.35 last 215.97) - open/close 31889 (time 1736263725.04 total 179.35 last 188.88) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 183.48 seconds: 177.64 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 3500 (time 1736263740.54 total 10.00 last 349.97) - /unlink 7082 (time 1736263750.54 total 20.00 last 358.17) - /unlink 10000 (time 1736263759.04 total 28.50 last 343.38) - /unlink 13734 (time 1736263769.04 total 38.50 last 373.38) - /unlink 16704 (time 1736263779.05 total 48.50 last 296.90) - /unlink 20000 (time 1736263788.92 total 58.38 last 333.63) - /unlink 23969 (time 1736263798.93 total 68.38 last 396.83) - /unlink 27464 (time 1736263808.93 total 78.39 last 349.39) - /unlink 30000 (time 1736263816.38 total 85.84 last 340.28) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 92.46 seconds: 352.54 ops/second Waiting for MDT destroys to complete - open/close 2070 (time 1736263841.06 total 10.00 last 206.95) - open/close 4582 (time 1736263851.06 total 20.00 last 251.16) - open/close 7214 (time 1736263861.06 total 30.01 last 263.18) - open/close 9690 (time 1736263871.06 total 40.01 last 247.60) - open/close 11893 (time 1736263881.06 total 50.01 last 220.28) - open/close 14510 (time 1736263891.06 total 60.01 last 261.68) - open/close 17571 (time 1736263901.07 total 70.01 last 306.05) - open/close 20000 (time 1736263909.99 total 78.93 last 272.18) - open/close 23246 (time 1736263919.99 total 88.93 last 324.57) - open/close 26563 (time 1736263929.99 total 98.94 last 331.61) - open/close 29581 (time 1736263940.00 total 108.94 last 301.68) - open/close 31752 (time 1736263950.00 total 118.94 last 217.01) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 122.62 seconds: 265.80 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 5026 (time 1736263964.87 total 10.00 last 502.55) - /unlink 9728 (time 1736263974.87 total 20.00 last 470.15) - /unlink 14224 (time 1736263984.87 total 30.00 last 449.59) - /unlink 18743 (time 1736263994.88 total 40.00 last 451.85) - /unlink 20000 (time 1736263996.90 total 42.02 last 621.98) - /unlink 26476 (time 1736264006.90 total 52.03 last 647.49) - /unlink 30000 (time 1736264013.58 total 58.71 last 527.52) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 64.22 seconds: 507.54 ops/second Waiting for MDT destroys to complete - open/close 2548 (time 1736264036.99 total 10.00 last 254.78) - open/close 5240 (time 1736264046.99 total 20.00 last 269.17) - open/close 8067 (time 1736264057.00 total 30.00 last 282.64) - open/close 10000 (time 1736264065.54 total 38.55 last 226.14) - open/close 12516 (time 1736264075.54 total 48.55 last 251.59) - open/close 15036 (time 1736264085.55 total 58.55 last 251.97) - open/close 17457 (time 1736264095.55 total 68.56 last 242.07) - open/close 19872 (time 1736264105.55 total 78.56 last 241.42) - open/close 21914 (time 1736264115.56 total 88.56 last 204.09) - open/close 24557 (time 1736264125.56 total 98.56 last 264.30) - open/close 27077 (time 1736264135.56 total 108.57 last 251.93) - open/close 29337 (time 1736264145.56 total 118.57 last 225.91) - open/close 30000 (time 1736264148.26 total 121.26 last 246.24) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 130.55 seconds: 249.66 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 4659 (time 1736264168.68 total 10.00 last 465.88) - /unlink 8698 (time 1736264178.69 total 20.00 last 403.85) - /unlink 10000 (time 1736264181.16 total 22.48 last 526.38) - /unlink 15869 (time 1736264191.16 total 32.48 last 586.83) - /unlink 20000 (time 1736264198.87 total 40.18 last 536.14) - /unlink 27109 (time 1736264208.87 total 50.18 last 710.89) - /unlink 30000 (time 1736264213.49 total 54.80 last 625.82) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 58.92 seconds: 553.18 ops/second Waiting for MDT destroys to complete - open/close 3727 (time 1736264247.63 total 10.00 last 372.69) - open/close 8052 (time 1736264257.63 total 20.00 last 432.41) - open/close 10000 (time 1736264262.03 total 24.41 last 442.14) - open/close 13920 (time 1736264272.04 total 34.41 last 391.93) - open/close 17145 (time 1736264282.04 total 44.41 last 322.45) - open/close 20000 (time 1736264288.19 total 50.57 last 463.89) - open/close 24305 (time 1736264298.19 total 60.57 last 430.45) - open/close 29813 (time 1736264308.19 total 70.57 last 550.79) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 76.08 seconds: 428.43 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 9298 (time 1736264324.56 total 10.00 last 929.74) - /unlink 19541 (time 1736264334.56 total 20.00 last 1024.24) - /unlink 30000 (time 1736264344.31 total 29.75 last 1072.65) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 31.67 seconds: 1029.23 ops/second Waiting for MDT destroys to complete - open/close 5262 (time 1736264377.26 total 10.00 last 526.14) - open/close 10000 (time 1736264386.90 total 19.64 last 491.65) - open/close 15064 (time 1736264396.90 total 29.64 last 506.35) - open/close 20000 (time 1736264405.65 total 38.39 last 564.13) - open/close 25032 (time 1736264415.65 total 48.39 last 503.13) - open/close 29958 (time 1736264425.65 total 58.39 last 492.57) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 63.13 seconds: 516.29 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 9085 (time 1736264441.29 total 10.00 last 908.44) - /unlink 18259 (time 1736264451.29 total 20.00 last 917.30) - /unlink 20000 (time 1736264453.69 total 22.40 last 725.65) - /unlink 26636 (time 1736264463.69 total 32.40 last 663.52) - /unlink 30000 (time 1736264468.11 total 36.82 last 762.07) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 39.33 seconds: 828.73 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg447-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server stop mds service on oleg447-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg447-server stop mds service on oleg447-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg447-server LNET ready to unload unloading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing unload_modules_local oleg447-server: LNET ready to unload modules unloaded. pdsh@oleg447-client: oleg447-client: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-client: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2