== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 01:15:20 (1732428920) Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f) oleg238-server: oleg238-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing load_modules_local oleg238-server: Loading modules from /home/green/git/lustre-release/lustre oleg238-server: detected 4 online CPUs by sysfs oleg238-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg238-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg238-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg238-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5550 (time 1732428959.85 total 10.00 last 554.95) - open/close 10000 (time 1732428967.90 total 18.05 last 552.57) - open/close 15523 (time 1732428977.90 total 28.06 last 552.26) - open/close 20000 (time 1732428985.85 total 36.00 last 563.51) - open/close 25415 (time 1732428995.85 total 46.00 last 541.47) - open/close 30000 (time 1732429004.65 total 54.80 last 520.80) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 59.65 seconds: 545.89 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9835 (time 1732429020.39 total 10.00 last 983.42) - /unlink 19842 (time 1732429030.39 total 20.00 last 1000.67) - /unlink 29779 (time 1732429040.39 total 30.00 last 993.70) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 32.73 seconds: 994.76 ops/second Waiting for MDT destroys to complete oleg238-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg238-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 - open/close 5568 (time 1732429082.37 total 10.00 last 556.73) - open/close 10000 (time 1732429090.37 total 18.00 last 554.02) - open/close 15366 (time 1732429100.37 total 28.00 last 536.56) - open/close 20000 (time 1732429109.20 total 36.82 last 525.20) - open/close 25456 (time 1732429119.20 total 46.83 last 545.52) - open/close 30000 (time 1732429127.47 total 55.10 last 549.47) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 59.71 seconds: 545.26 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1732429142.82 total 9.89 last 1010.73) - /unlink 20000 (time 1732429152.76 total 19.83 last 1006.49) - /unlink 30000 (time 1732429162.69 total 29.76 last 1007.08) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 32.38 seconds: 1005.59 ops/second Waiting for MDT destroys to complete oleg238-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg238-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 - open/close 5734 (time 1732429207.55 total 10.00 last 573.39) - open/close 10000 (time 1732429215.05 total 17.50 last 568.88) - open/close 15714 (time 1732429225.05 total 27.50 last 571.31) - open/close 20000 (time 1732429232.60 total 35.05 last 567.75) - open/close 25384 (time 1732429242.60 total 45.05 last 538.38) - open/close 30000 (time 1732429250.97 total 53.42 last 551.30) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 59.06 seconds: 551.26 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7988 (time 1732429267.75 total 10.00 last 798.65) - /unlink 10000 (time 1732429271.66 total 13.91 last 515.22) - /unlink 15201 (time 1732429281.66 total 23.91 last 520.04) - /unlink 19907 (time 1732429291.66 total 33.91 last 470.50) - /unlink 24836 (time 1732429301.66 total 43.91 last 492.89) - /unlink 29993 (time 1732429311.66 total 53.91 last 515.66) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 57.58 seconds: 565.46 ops/second Waiting for MDT destroys to complete oleg238-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg238-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 - open/close 3098 (time 1732429335.50 total 10.00 last 309.79) - open/close 6722 (time 1732429345.51 total 20.00 last 362.36) - open/close 10000 (time 1732429354.74 total 29.24 last 355.00) - open/close 13306 (time 1732429364.74 total 39.24 last 330.54) - open/close 16487 (time 1732429374.74 total 49.24 last 318.03) - open/close 20000 (time 1732429384.60 total 59.10 last 356.28) - open/close 24721 (time 1732429394.61 total 69.10 last 472.02) - open/close 27962 (time 1732429404.61 total 79.10 last 324.08) - open/close 30000 (time 1732429411.58 total 86.07 last 292.38) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 93.11 seconds: 349.70 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5872 (time 1732429429.85 total 10.00 last 587.17) - /unlink 10000 (time 1732429436.68 total 16.83 last 604.28) - /unlink 15806 (time 1732429446.68 total 26.83 last 580.57) - /unlink 20000 (time 1732429454.28 total 34.44 last 551.51) - /unlink 25538 (time 1732429464.28 total 44.44 last 553.78) - /unlink 30000 (time 1732429473.00 total 53.16 last 511.72) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 57.94 seconds: 561.94 ops/second Waiting for MDT destroys to complete oleg238-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg238-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 - open/close 2790 (time 1732429496.58 total 10.00 last 278.94) - open/close 5596 (time 1732429506.58 total 20.00 last 280.60) - open/close 8632 (time 1732429516.59 total 30.01 last 303.49) - open/close 10000 (time 1732429521.39 total 34.81 last 284.86) - open/close 12707 (time 1732429531.39 total 44.81 last 270.62) - open/close 15468 (time 1732429541.40 total 54.82 last 275.99) - open/close 18202 (time 1732429551.40 total 64.82 last 273.17) - open/close 20000 (time 1732429557.54 total 70.96 last 292.87) - open/close 22650 (time 1732429567.55 total 80.97 last 264.93) - open/close 25945 (time 1732429577.55 total 90.97 last 329.48) - open/close 28578 (time 1732429587.55 total 100.97 last 263.22) - open/close 30000 (time 1732429592.41 total 105.83 last 292.84) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 113.35 seconds: 287.24 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4800 (time 1732429611.18 total 10.00 last 479.98) - /unlink 10000 (time 1732429616.80 total 15.62 last 925.34) - /unlink 19301 (time 1732429626.80 total 25.62 last 930.09) - /unlink 28851 (time 1732429636.80 total 35.62 last 954.95) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 39.42 seconds: 825.94 ops/second Waiting for MDT destroys to complete oleg238-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg238-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server stop mds service on oleg238-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server stop mds service on oleg238-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server unloading modules via unload_modules_local on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2