== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:16:54 (1743527814) Stopping clients: oleg453-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg453-client.virtnet /mnt/lustre2 (opts:-f) oleg453-server: oleg453-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg453-server' oleg453-server: oleg453-server.virtnet: executing load_modules_local oleg453-server: Loading modules from /home/green/git/lustre-release/lustre oleg453-server: detected 4 online CPUs by sysfs oleg453-server: Force libcfs to create 2 CPU partitions oleg453-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg453-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg453-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg453-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg453-client.virtnet: -o user_xattr,flock oleg453-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3704 (time 1743527856.54 total 10.00 last 370.36) - open/close 8652 (time 1743527866.54 total 20.00 last 494.72) - open/close 10000 (time 1743527869.91 total 23.38 last 399.24) - open/close 15167 (time 1743527879.92 total 33.38 last 516.62) - open/close 19791 (time 1743527889.92 total 43.38 last 462.26) - open/close 24033 (time 1743527899.92 total 53.39 last 424.14) - open/close 28793 (time 1743527909.92 total 63.39 last 475.91) - open/close 30000 (time 1743527913.13 total 66.59 last 376.36) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 73.03 seconds: 445.84 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7482 (time 1743527930.53 total 10.00 last 748.10) - /unlink 10000 (time 1743527934.11 total 13.58 last 702.65) - /unlink 17498 (time 1743527944.11 total 23.59 last 749.79) - /unlink 20000 (time 1743527947.37 total 26.84 last 768.45) - /unlink 26027 (time 1743527957.37 total 36.84 last 602.61) - /unlink 30000 (time 1743527962.78 total 42.26 last 734.00) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 45.03 seconds: 723.02 ops/second Waiting for MDT destroys to complete - open/close 3770 (time 1743527995.62 total 10.00 last 376.99) - open/close 7995 (time 1743528005.62 total 20.00 last 422.43) - open/close 10000 (time 1743528010.75 total 25.13 last 390.66) - open/close 14291 (time 1743528020.75 total 35.14 last 429.06) - open/close 18098 (time 1743528030.75 total 45.14 last 380.70) - open/close 20000 (time 1743528036.22 total 50.60 last 348.13) - open/close 24480 (time 1743528046.22 total 60.60 last 447.91) - open/close 29026 (time 1743528056.22 total 70.60 last 454.54) - open/close 30000 (time 1743528059.64 total 74.03 last 284.44) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 81.45 seconds: 399.77 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7286 (time 1743528077.96 total 10.00 last 728.55) - /unlink 10000 (time 1743528081.99 total 14.03 last 673.39) - /unlink 17313 (time 1743528092.00 total 24.03 last 731.23) - /unlink 20000 (time 1743528095.45 total 27.49 last 776.94) - /unlink 28710 (time 1743528105.45 total 37.49 last 870.95) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 41.23 seconds: 789.75 ops/second Waiting for MDT destroys to complete - open/close 4445 (time 1743528138.68 total 10.00 last 444.44) - open/close 9471 (time 1743528148.69 total 20.00 last 502.51) - open/close 15007 (time 1743528158.69 total 30.00 last 553.50) - open/close 20000 (time 1743528167.47 total 38.79 last 568.36) - open/close 25965 (time 1743528177.47 total 48.79 last 596.46) - open/close 30000 (time 1743528185.19 total 56.51 last 522.83) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 61.55 seconds: 528.97 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1743528200.94 total 9.85 last 1014.94) - /unlink 20000 (time 1743528210.17 total 19.08 last 1083.79) - /unlink 30000 (time 1743528219.46 total 28.37 last 1076.20) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 30.78 seconds: 1057.72 ops/second Waiting for MDT destroys to complete - open/close 5107 (time 1743528266.74 total 10.00 last 510.63) - open/close 10000 (time 1743528276.16 total 19.42 last 519.36) - open/close 15195 (time 1743528286.16 total 29.42 last 519.42) - open/close 20000 (time 1743528295.25 total 38.52 last 528.46) - open/close 24139 (time 1743528305.25 total 48.52 last 413.89) - open/close 29213 (time 1743528315.25 total 58.52 last 507.35) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 64.76 seconds: 502.80 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1743528331.69 total 9.36 last 1068.75) - /unlink 18750 (time 1743528341.69 total 19.36 last 874.96) - /unlink 28078 (time 1743528351.69 total 29.36 last 932.76) - /unlink 30000 (time 1743528353.75 total 31.42 last 934.10) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 34.12 seconds: 954.34 ops/second Waiting for MDT destroys to complete - open/close 5894 (time 1743528395.41 total 10.00 last 589.36) - open/close 10000 (time 1743528402.39 total 16.98 last 588.33) - open/close 15778 (time 1743528412.39 total 26.98 last 577.78) - open/close 20000 (time 1743528419.67 total 34.26 last 580.13) - open/close 26279 (time 1743528429.67 total 44.26 last 627.81) - open/close 30000 (time 1743528435.89 total 50.48 last 597.71) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 54.99 seconds: 592.11 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1743528450.81 total 9.55 last 1046.95) - /unlink 20000 (time 1743528460.24 total 18.98 last 1060.60) - /unlink 30000 (time 1743528469.77 total 28.51 last 1049.34) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 30.80 seconds: 1057.02 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg453-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg453-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg453-server stop mds service on oleg453-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg453-server stop mds service on oleg453-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg453-server unloading modules on: 'oleg453-server' oleg453-server: oleg453-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg453-client: oleg453-client: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-client: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 2