== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 14:11:24 (1722795084) Stopping clients: oleg307-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg307-client.virtnet /mnt/lustre2 (opts:-f) oleg307-server: oleg307-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing load_modules_local oleg307-server: Loading modules from /home/green/git/lustre-release/lustre oleg307-server: detected 4 online CPUs by sysfs oleg307-server: Force libcfs to create 2 CPU partitions oleg307-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg307-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg307-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg307-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg307-client.virtnet: -o user_xattr,flock oleg307-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4029 (time 1722795124.53 total 10.00 last 402.87) - open/close 7717 (time 1722795134.53 total 20.00 last 368.73) - open/close 10000 (time 1722795140.61 total 26.08 last 375.73) - open/close 14555 (time 1722795150.61 total 36.08 last 455.49) - open/close 19566 (time 1722795160.61 total 46.08 last 501.01) - open/close 24930 (time 1722795170.61 total 56.08 last 536.25) - open/close 29707 (time 1722795180.61 total 66.09 last 477.62) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 71.20 seconds: 457.29 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1722795196.40 total 9.78 last 1022.47) - /unlink 19753 (time 1722795206.40 total 19.78 last 975.29) - /unlink 28766 (time 1722795216.40 total 29.78 last 901.22) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 34.11 seconds: 954.49 ops/second Waiting for MDT destroys to complete oleg307-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg307-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 - open/close 5301 (time 1722795278.10 total 10.00 last 530.04) - open/close 10000 (time 1722795286.78 total 18.67 last 541.84) - open/close 14844 (time 1722795296.78 total 28.68 last 484.25) - open/close 20000 (time 1722795306.76 total 38.66 last 516.36) - open/close 25860 (time 1722795316.76 total 48.66 last 585.97) - open/close 30000 (time 1722795324.35 total 56.25 last 545.50) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 61.03 seconds: 533.50 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9799 (time 1722795340.02 total 10.00 last 979.87) - /unlink 19813 (time 1722795350.02 total 20.00 last 1001.33) - /unlink 29582 (time 1722795360.02 total 30.00 last 976.84) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 32.97 seconds: 987.49 ops/second Waiting for MDT destroys to complete oleg307-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg307-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 - open/close 5733 (time 1722795405.63 total 10.00 last 573.25) - open/close 10000 (time 1722795413.51 total 17.89 last 541.03) - open/close 15561 (time 1722795423.51 total 27.89 last 556.07) - open/close 20000 (time 1722795431.33 total 35.70 last 568.15) - open/close 25506 (time 1722795441.33 total 45.70 last 550.57) - open/close 30000 (time 1722795449.30 total 53.68 last 563.59) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 58.23 seconds: 559.14 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9917 (time 1722795464.71 total 10.00 last 991.65) - /unlink 20000 (time 1722795474.55 total 19.84 last 1024.81) - /unlink 30000 (time 1722795484.20 total 29.48 last 1037.05) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 32.06 seconds: 1015.55 ops/second Waiting for MDT destroys to complete oleg307-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg307-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 - open/close 5677 (time 1722795528.07 total 10.00 last 567.62) - open/close 10000 (time 1722795535.67 total 17.60 last 569.03) - open/close 15474 (time 1722795545.67 total 27.60 last 547.36) - open/close 20000 (time 1722795554.07 total 36.00 last 538.67) - open/close 24842 (time 1722795564.07 total 46.00 last 484.20) - open/close 29422 (time 1722795574.07 total 56.00 last 457.96) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 63.53 seconds: 512.54 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 8884 (time 1722795592.54 total 10.00 last 888.32) - /unlink 15780 (time 1722795602.54 total 20.00 last 689.53) - /unlink 20000 (time 1722795608.35 total 25.81 last 726.28) - /unlink 27043 (time 1722795618.35 total 35.81 last 704.25) - /unlink 30000 (time 1722795621.72 total 39.18 last 877.54) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 41.99 seconds: 775.38 ops/second Waiting for MDT destroys to complete oleg307-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg307-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 - open/close 4101 (time 1722795653.06 total 10.00 last 409.99) - open/close 8562 (time 1722795663.06 total 20.00 last 446.06) - open/close 10000 (time 1722795665.78 total 22.72 last 530.19) - open/close 14714 (time 1722795675.78 total 32.72 last 471.34) - open/close 20000 (time 1722795685.34 total 42.29 last 552.48) - open/close 24936 (time 1722795695.35 total 52.29 last 493.56) - open/close 28685 (time 1722795705.35 total 62.29 last 374.78) - open/close 30000 (time 1722795709.36 total 66.30 last 328.21) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 73.06 seconds: 445.66 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7459 (time 1722795727.00 total 10.00 last 745.83) - /unlink 10000 (time 1722795730.85 total 13.85 last 660.70) - /unlink 18044 (time 1722795740.85 total 23.85 last 804.25) - /unlink 20000 (time 1722795743.67 total 26.68 last 691.90) - /unlink 27411 (time 1722795753.68 total 36.68 last 740.98) - /unlink 30000 (time 1722795757.79 total 40.79 last 629.94) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 44.34 seconds: 734.30 ops/second Waiting for MDT destroys to complete oleg307-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg307-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg307-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg307-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg307-server unloading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2