== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 01:29:36 (1724304576) Stopping clients: oleg130-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg130-client.virtnet /mnt/lustre2 (opts:-f) oleg130-server: oleg130-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg130-server' oleg130-server: oleg130-server.virtnet: executing load_modules_local oleg130-server: Loading modules from /home/green/git/lustre-release/lustre oleg130-server: detected 4 online CPUs by sysfs oleg130-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg130-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg130-server: oleg130-server.virtnet: executing set_default_debug -1 all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg130-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg130-server: oleg130-server.virtnet: executing set_default_debug -1 all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg130-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg130-server: oleg130-server.virtnet: executing set_default_debug -1 all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4542 (time 1724304617.80 total 10.00 last 454.15) - open/close 9522 (time 1724304627.80 total 20.00 last 497.99) - open/close 14308 (time 1724304637.80 total 30.00 last 478.51) - open/close 19408 (time 1724304647.80 total 40.00 last 509.94) - open/close 25383 (time 1724304657.80 total 50.01 last 597.44) - open/close 30000 (time 1724304665.96 total 58.16 last 565.93) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 62.42 seconds: 521.66 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1724304680.51 total 9.46 last 1056.79) - /unlink 19713 (time 1724304690.51 total 19.46 last 971.24) - /unlink 28081 (time 1724304700.51 total 29.46 last 836.80) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 33.90 seconds: 960.38 ops/second Waiting for MDT destroys to complete oleg130-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg130-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 - open/close 5196 (time 1724304741.35 total 10.00 last 519.50) - open/close 10000 (time 1724304750.24 total 18.89 last 540.25) - open/close 15506 (time 1724304760.24 total 28.89 last 550.57) - open/close 20000 (time 1724304768.51 total 37.16 last 543.89) - open/close 25534 (time 1724304778.51 total 47.16 last 553.40) - open/close 30000 (time 1724304786.53 total 55.18 last 556.53) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 59.82 seconds: 544.33 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9899 (time 1724304802.04 total 10.00 last 989.89) - /unlink 19934 (time 1724304812.04 total 20.00 last 1003.45) - /unlink 29785 (time 1724304822.04 total 30.00 last 985.08) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 32.91 seconds: 989.36 ops/second Waiting for MDT destroys to complete oleg130-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg130-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 - open/close 6262 (time 1724304861.08 total 10.00 last 626.17) - open/close 10000 (time 1724304867.40 total 16.32 last 591.47) - open/close 12228 (time 1724304877.40 total 26.32 last 222.76) - open/close 13671 (time 1724304887.41 total 36.32 last 144.26) - open/close 16659 (time 1724304897.41 total 46.33 last 298.56) - open/close 19357 (time 1724304907.42 total 56.34 last 269.66) - open/close 20000 (time 1724304909.71 total 58.63 last 280.99) - open/close 22718 (time 1724304919.71 total 68.63 last 271.68) - open/close 24738 (time 1724304929.71 total 78.63 last 201.96) - open/close 27474 (time 1724304939.72 total 88.63 last 273.55) - open/close 30000 (time 1724304946.29 total 95.21 last 384.38) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 100.99 seconds: 322.42 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 6322 (time 1724304963.06 total 10.00 last 632.12) - /unlink 10000 (time 1724304968.04 total 14.98 last 738.89) - /unlink 15732 (time 1724304978.04 total 24.98 last 573.13) - /unlink 20000 (time 1724304985.76 total 32.70 last 552.76) - /unlink 26918 (time 1724304995.77 total 42.70 last 691.61) - /unlink 30000 (time 1724305001.46 total 48.40 last 541.43) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 52.72 seconds: 617.63 ops/second Waiting for MDT destroys to complete oleg130-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg130-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 - open/close 2757 (time 1724305025.42 total 10.00 last 275.63) - open/close 5241 (time 1724305035.43 total 20.00 last 248.37) - open/close 7477 (time 1724305045.44 total 30.02 last 223.16) - open/close 8509 (time 1724305055.45 total 40.03 last 103.15) - open/close 9437 (time 1724305065.46 total 50.03 last 92.74) - open/close 10000 (time 1724305070.15 total 54.73 last 119.84) - open/close 11733 (time 1724305080.16 total 64.73 last 173.28) - open/close 13394 (time 1724305090.16 total 74.73 last 166.09) - open/close 15278 (time 1724305100.16 total 84.74 last 188.36) - open/close 16948 (time 1724305110.16 total 94.74 last 166.92) - open/close 18474 (time 1724305120.18 total 104.75 last 152.41) - open/close 20000 (time 1724305130.05 total 114.63 last 154.55) - open/close 21832 (time 1724305140.05 total 124.63 last 183.11) - open/close 23629 (time 1724305150.06 total 134.64 last 179.59) - open/close 25731 (time 1724305160.06 total 144.64 last 210.14) - open/close 27197 (time 1724305170.07 total 154.65 last 146.55) - open/close 28837 (time 1724305180.07 total 164.65 last 163.96) - open/close 30000 (time 1724305186.42 total 171.00 last 183.20) - open/close 31612 (time 1724305196.42 total 181.00 last 161.17) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 187.06 seconds: 174.06 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 2777 (time 1724305214.29 total 10.00 last 277.64) - /unlink 5686 (time 1724305224.29 total 20.01 last 290.79) - /unlink 8806 (time 1724305234.29 total 30.01 last 311.97) - /unlink 10000 (time 1724305237.75 total 33.46 last 345.75) - /unlink 13181 (time 1724305247.75 total 43.46 last 318.03) - /unlink 16706 (time 1724305257.75 total 53.46 last 352.46) - /unlink 20000 (time 1724305266.71 total 62.42 last 367.67) - /unlink 23048 (time 1724305276.71 total 72.42 last 304.73) - /unlink 26178 (time 1724305286.71 total 82.43 last 312.97) - /unlink 29292 (time 1724305296.72 total 92.43 last 311.36) - /unlink 30000 (time 1724305299.20 total 94.91 last 284.74) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 103.21 seconds: 315.47 ops/second Waiting for MDT destroys to complete oleg130-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 oleg130-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 - open/close 1878 (time 1724305327.76 total 10.00 last 187.72) - open/close 3685 (time 1724305337.76 total 20.01 last 180.60) - open/close 5513 (time 1724305347.77 total 30.02 last 182.62) - open/close 7455 (time 1724305357.77 total 40.02 last 194.19) - open/close 9441 (time 1724305367.78 total 50.02 last 198.51) - open/close 10000 (time 1724305372.38 total 54.63 last 121.30) - open/close 10947 (time 1724305382.39 total 64.64 last 94.67) - open/close 11821 (time 1724305392.40 total 74.65 last 87.32) - open/close 13237 (time 1724305402.40 total 84.65 last 141.54) - open/close 14642 (time 1724305412.41 total 94.65 last 140.44) - open/close 16103 (time 1724305422.41 total 104.66 last 146.08) - open/close 17426 (time 1724305432.41 total 114.66 last 132.29) - open/close 18782 (time 1724305442.41 total 124.66 last 135.58) - open/close 20000 (time 1724305450.30 total 132.55 last 154.36) - open/close 21379 (time 1724305460.30 total 142.55 last 137.88) - open/close 22776 (time 1724305470.33 total 152.58 last 139.33) - open/close 24149 (time 1724305480.34 total 162.59 last 137.18) - open/close 25641 (time 1724305490.36 total 172.60 last 148.92) - open/close 27082 (time 1724305500.37 total 182.62 last 143.86) - open/close 28567 (time 1724305510.38 total 192.62 last 148.45) - open/close 30000 (time 1724305520.24 total 202.49 last 145.26) - open/close 31062 (time 1724305530.24 total 212.49 last 106.19) - open/close 32221 (time 1724305540.25 total 222.50 last 115.77) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 225.24 seconds: 144.56 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 2105 (time 1724305555.22 total 10.00 last 210.49) - /unlink 4389 (time 1724305565.22 total 20.00 last 228.35) - /unlink 6911 (time 1724305575.22 total 30.00 last 252.18) - /unlink 9431 (time 1724305585.22 total 40.01 last 251.94) - /unlink 10000 (time 1724305588.11 total 42.89 last 197.14) - /unlink 12213 (time 1724305598.12 total 52.90 last 221.04) - /unlink 14244 (time 1724305608.12 total 62.90 last 203.08) - /unlink 16433 (time 1724305618.13 total 72.91 last 218.79) - /unlink 18474 (time 1724305628.13 total 82.91 last 204.02) - /unlink 20000 (time 1724305635.76 total 90.54 last 200.13) - /unlink 22248 (time 1724305645.77 total 100.55 last 224.58) - /unlink 24559 (time 1724305655.78 total 110.56 last 230.82) - /unlink 27126 (time 1724305665.78 total 120.57 last 256.52) - /unlink 29735 (time 1724305675.79 total 130.57 last 260.85) - /unlink 32454 (time 1724305685.79 total 140.57 last 271.84) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 141.04 seconds: 230.86 ops/second Waiting for MDT destroys to complete oleg130-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 oleg130-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg130-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg130-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg130-server stop mds service on oleg130-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg130-server stop mds service on oleg130-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg130-server unloading modules on: 'oleg130-server' oleg130-server: oleg130-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg130-client: oleg130-client: ssh exited with exit code 2 pdsh@oleg130-client: oleg130-server: ssh exited with exit code 2 pdsh@oleg130-client: oleg130-client: ssh exited with exit code 2 pdsh@oleg130-client: oleg130-server: ssh exited with exit code 2