== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 23:38:35 (1721878715) Stopping clients: oleg324-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg324-client.virtnet /mnt/lustre2 (opts:-f) oleg324-server: oleg324-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: detected 4 online CPUs by sysfs oleg324-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg324-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg324-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg324-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg324-client.virtnet: -o user_xattr,flock oleg324-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 336 32560 2% /mnt/lustre[OST:0] filesystem_summary: 33089 529 32560 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5779 (time 1721878751.97 total 10.00 last 577.84) - open/close 10000 (time 1721878759.33 total 17.36 last 573.32) - open/close 15747 (time 1721878769.33 total 27.36 last 574.61) - open/close 20000 (time 1721878779.07 total 37.10 last 437.02) - open/close 24080 (time 1721878789.07 total 47.10 last 407.98) - open/close 29574 (time 1721878799.07 total 57.10 last 549.39) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 62.63 seconds: 519.86 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9851 (time 1721878815.38 total 10.00 last 985.04) - /unlink 19804 (time 1721878825.38 total 20.00 last 995.21) - /unlink 28837 (time 1721878835.38 total 30.00 last 903.27) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 34.84 seconds: 934.67 ops/second Waiting for MDT destroys to complete oleg324-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg324-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 - open/close 5686 (time 1721878895.73 total 10.00 last 568.49) - open/close 8351 (time 1721878905.73 total 20.00 last 266.42) - open/close 10000 (time 1721878915.22 total 29.49 last 173.81) - open/close 14402 (time 1721878925.22 total 39.49 last 440.15) - open/close 19202 (time 1721878935.22 total 49.49 last 479.99) - open/close 22195 (time 1721878945.23 total 59.51 last 298.93) - open/close 24465 (time 1721878955.23 total 69.51 last 226.89) - open/close 28647 (time 1721878965.24 total 79.51 last 418.12) - open/close 30000 (time 1721878967.56 total 81.84 last 582.39) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 86.23 seconds: 377.61 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1721878982.50 total 9.75 last 1025.28) - /unlink 18920 (time 1721878992.50 total 19.75 last 891.97) - /unlink 28059 (time 1721879002.50 total 29.75 last 913.81) - /unlink 30000 (time 1721879004.62 total 31.88 last 913.59) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 34.49 seconds: 944.11 ops/second Waiting for MDT destroys to complete oleg324-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 oleg324-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 - open/close 1416 (time 1721879050.01 total 10.00 last 141.57) - open/close 3592 (time 1721879060.02 total 20.01 last 217.52) - open/close 6155 (time 1721879070.02 total 30.01 last 256.29) - open/close 9206 (time 1721879080.02 total 40.01 last 305.06) - open/close 10000 (time 1721879082.37 total 42.36 last 336.99) - open/close 13340 (time 1721879092.38 total 52.37 last 333.95) - open/close 16623 (time 1721879102.38 total 62.37 last 328.27) - open/close 20000 (time 1721879111.54 total 71.53 last 368.47) - open/close 24789 (time 1721879121.54 total 81.53 last 478.87) - open/close 29190 (time 1721879131.55 total 91.53 last 439.97) - open/close 30000 (time 1721879133.77 total 93.76 last 363.87) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 100.86 seconds: 322.84 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7372 (time 1721879151.89 total 10.00 last 737.15) - /unlink 10000 (time 1721879154.78 total 12.90 last 907.73) - /unlink 19873 (time 1721879164.78 total 22.90 last 987.29) - /unlink 30000 (time 1721879174.18 total 32.29 last 1077.79) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 34.60 seconds: 940.99 ops/second Waiting for MDT destroys to complete oleg324-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg324-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 - open/close 5583 (time 1721879218.01 total 10.00 last 558.23) - open/close 10000 (time 1721879225.77 total 17.77 last 568.78) - open/close 15621 (time 1721879235.77 total 27.77 last 562.08) - open/close 20000 (time 1721879243.86 total 35.86 last 541.28) - open/close 25524 (time 1721879253.86 total 45.86 last 552.39) - open/close 30000 (time 1721879262.30 total 54.29 last 530.56) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 59.08 seconds: 551.10 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 8913 (time 1721879278.01 total 10.01 last 890.59) - /unlink 18564 (time 1721879288.01 total 20.01 last 965.07) - /unlink 29548 (time 1721879298.01 total 30.01 last 1098.31) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 32.80 seconds: 992.56 ops/second Waiting for MDT destroys to complete oleg324-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg324-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 - open/close 6201 (time 1721879340.44 total 10.00 last 620.05) - open/close 10000 (time 1721879346.68 total 16.23 last 609.69) - open/close 16054 (time 1721879356.68 total 26.23 last 605.36) - open/close 20000 (time 1721879362.81 total 32.36 last 643.81) - open/close 26112 (time 1721879372.81 total 42.36 last 611.16) - open/close 30000 (time 1721879379.26 total 48.82 last 602.32) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 52.98 seconds: 614.57 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32807 47185 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33062 33062 0 100% /mnt/lustre - /unlink 10000 (time 1721879393.46 total 9.13 last 1094.79) - /unlink 19523 (time 1721879403.46 total 19.14 last 952.17) - /unlink 29049 (time 1721879413.46 total 29.14 last 952.52) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 32.50 seconds: 1001.70 ops/second Waiting for MDT destroys to complete oleg324-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg324-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg324-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg324-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg324-server stop mds service on oleg324-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg324-server stop mds service on oleg324-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg324-server unloading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2