== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 23:57:18 (1721879838) Stopping clients: oleg455-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg455-client.virtnet /mnt/lustre2 (opts:-f) oleg455-server: oleg455-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing load_modules_local oleg455-server: Loading modules from /home/green/git/lustre-release/lustre oleg455-server: detected 4 online CPUs by sysfs oleg455-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg455-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg455-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg455-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg455-client.virtnet: -o user_xattr,flock oleg455-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1868 (time 1721879881.82 total 10.01 last 186.67) - open/close 3612 (time 1721879891.82 total 20.01 last 174.32) - open/close 6288 (time 1721879901.83 total 30.02 last 267.42) - open/close 8653 (time 1721879911.83 total 40.02 last 236.43) - open/close 10000 (time 1721879917.92 total 46.10 last 221.42) - open/close 12323 (time 1721879927.92 total 56.11 last 232.29) - open/close 15583 (time 1721879937.92 total 66.11 last 325.80) - open/close 18113 (time 1721879947.93 total 76.12 last 252.84) - open/close 20000 (time 1721879957.17 total 85.36 last 204.16) - open/close 22252 (time 1721879967.17 total 95.36 last 225.19) - open/close 25637 (time 1721879977.17 total 105.36 last 338.49) - open/close 28564 (time 1721879987.18 total 115.36 last 292.61) - open/close 30000 (time 1721879992.28 total 120.47 last 281.42) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 129.16 seconds: 252.09 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 6510 (time 1721880011.95 total 10.00 last 651.00) - /unlink 10000 (time 1721880016.71 total 14.76 last 733.19) - /unlink 17321 (time 1721880026.71 total 24.76 last 732.07) - /unlink 20000 (time 1721880030.83 total 28.88 last 650.32) - /unlink 26882 (time 1721880040.83 total 38.88 last 688.14) - /unlink 30000 (time 1721880046.07 total 44.12 last 595.22) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 48.35 seconds: 673.47 ops/second Waiting for MDT destroys to complete oleg455-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg455-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 - open/close 3093 (time 1721880079.56 total 10.00 last 309.17) - open/close 6106 (time 1721880089.56 total 20.01 last 301.26) - open/close 9116 (time 1721880099.56 total 30.01 last 300.93) - open/close 10000 (time 1721880102.58 total 33.02 last 293.32) - open/close 12753 (time 1721880112.58 total 43.02 last 275.26) - open/close 15782 (time 1721880122.58 total 53.03 last 302.80) - open/close 19158 (time 1721880132.58 total 63.03 last 337.54) - open/close 20000 (time 1721880135.13 total 65.58 last 330.17) - open/close 23499 (time 1721880145.14 total 75.58 last 349.80) - open/close 26842 (time 1721880155.14 total 85.58 last 334.28) - open/close 29458 (time 1721880165.14 total 95.58 last 261.60) - open/close 30000 (time 1721880167.33 total 97.78 last 246.97) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 105.54 seconds: 308.50 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 6663 (time 1721880186.16 total 10.00 last 666.25) - /unlink 10000 (time 1721880192.11 total 15.94 last 561.62) - /unlink 15388 (time 1721880202.11 total 25.94 last 538.73) - /unlink 20000 (time 1721880209.65 total 33.49 last 611.49) - /unlink 26609 (time 1721880219.65 total 43.49 last 660.89) - /unlink 30000 (time 1721880225.06 total 48.89 last 626.94) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 53.65 seconds: 606.87 ops/second Waiting for MDT destroys to complete oleg455-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg455-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 - open/close 3059 (time 1721880274.30 total 10.00 last 305.87) - open/close 6253 (time 1721880284.30 total 20.00 last 319.36) - open/close 9949 (time 1721880294.30 total 30.00 last 369.57) - open/close 13892 (time 1721880304.31 total 40.00 last 394.25) - open/close 18513 (time 1721880314.31 total 50.01 last 462.07) - open/close 20000 (time 1721880317.74 total 53.44 last 433.10) - open/close 24614 (time 1721880327.74 total 63.44 last 461.30) - open/close 30000 (time 1721880337.73 total 73.43 last 539.40) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 78.33 seconds: 415.67 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7670 (time 1721880353.60 total 10.00 last 766.99) - /unlink 10000 (time 1721880356.82 total 13.23 last 722.22) - /unlink 17242 (time 1721880366.82 total 23.23 last 724.17) - /unlink 20000 (time 1721880370.68 total 27.08 last 715.06) - /unlink 29966 (time 1721880380.68 total 37.08 last 996.56) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 39.47 seconds: 824.86 ops/second Waiting for MDT destroys to complete oleg455-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg455-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 - open/close 4398 (time 1721880417.17 total 10.00 last 439.75) - open/close 8942 (time 1721880427.17 total 20.00 last 454.38) - open/close 10000 (time 1721880429.21 total 22.04 last 517.93) - open/close 15957 (time 1721880439.21 total 32.05 last 595.66) - open/close 20000 (time 1721880447.43 total 40.26 last 491.97) - open/close 24431 (time 1721880457.43 total 50.26 last 443.08) - open/close 28602 (time 1721880467.43 total 60.27 last 417.04) - open/close 30000 (time 1721880469.84 total 62.68 last 579.88) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 67.27 seconds: 484.05 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1721880484.85 total 9.54 last 1047.95) - /unlink 18786 (time 1721880494.85 total 19.54 last 878.56) - /unlink 28318 (time 1721880504.85 total 29.54 last 953.19) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 33.29 seconds: 978.05 ops/second Waiting for MDT destroys to complete oleg455-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg455-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 - open/close 5274 (time 1721880536.61 total 10.00 last 527.30) - open/close 10000 (time 1721880545.54 total 18.93 last 529.31) - open/close 15775 (time 1721880555.54 total 28.93 last 577.42) - open/close 20000 (time 1721880563.23 total 36.62 last 549.46) - open/close 25256 (time 1721880573.23 total 46.62 last 525.52) - open/close 30000 (time 1721880583.13 total 56.52 last 479.28) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 62.12 seconds: 524.15 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 8794 (time 1721880599.75 total 10.00 last 879.35) - /unlink 17660 (time 1721880609.75 total 20.00 last 886.54) - /unlink 20000 (time 1721880612.10 total 22.35 last 995.76) - /unlink 29107 (time 1721880622.10 total 32.35 last 910.68) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 35.98 seconds: 905.01 ops/second Waiting for MDT destroys to complete oleg455-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg455-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg455-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg455-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg455-server stop mds service on oleg455-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg455-server stop mds service on oleg455-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg455-server unloading modules on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg455-client: oleg455-client: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-client: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2