== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:26:54 (1734258414) Stopping clients: oleg121-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg121-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2 oleg121-server: oleg121-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg121-server' oleg121-server: oleg121-server.virtnet: executing load_modules_local oleg121-server: Loading modules from /home/green/git/lustre-release/lustre oleg121-server: detected 4 online CPUs by sysfs oleg121-server: Force libcfs to create 2 CPU partitions oleg121-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg121-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg121-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg121-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 272 79720 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 302 32594 1% /mnt/lustre[OST:0] filesystem_summary: 32866 272 32594 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1931 (time 1734258473.22 total 10.01 last 192.96) - open/close 3894 (time 1734258483.23 total 20.01 last 196.22) - open/close 6051 (time 1734258493.23 total 30.01 last 215.68) - open/close 8296 (time 1734258503.23 total 40.02 last 224.44) - open/close 10000 (time 1734258512.40 total 49.18 last 185.93) - open/close 11708 (time 1734258522.41 total 59.19 last 170.64) - open/close 13359 (time 1734258532.41 total 69.19 last 165.08) - open/close 14897 (time 1734258542.41 total 79.20 last 153.73) - open/close 16491 (time 1734258552.41 total 89.20 last 159.36) - open/close 17978 (time 1734258562.42 total 99.20 last 148.63) - open/close 19597 (time 1734258572.42 total 109.20 last 161.88) - open/close 20000 (time 1734258574.90 total 111.69 last 162.21) - open/close 21550 (time 1734258584.91 total 121.69 last 154.98) - open/close 23019 (time 1734258594.91 total 131.69 last 146.86) - open/close 24820 (time 1734258604.92 total 141.70 last 179.90) - open/close 26381 (time 1734258614.92 total 151.71 last 156.06) - open/close 27949 (time 1734258624.92 total 161.71 last 156.80) - open/close 29494 (time 1734258634.92 total 171.71 last 154.47) - open/close 30000 (time 1734258638.44 total 175.22 last 143.90) - open/close 31541 (time 1734258648.44 total 185.23 last 154.04) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 191.85 seconds: 170.06 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3090 (time 1734258666.73 total 10.00 last 308.96) - /unlink 6271 (time 1734258676.73 total 20.00 last 318.04) - /unlink 9372 (time 1734258686.73 total 30.01 last 309.97) - /unlink 12990 (time 1734258696.73 total 40.01 last 361.78) - /unlink 16061 (time 1734258706.74 total 50.01 last 307.00) - /unlink 19230 (time 1734258716.74 total 60.01 last 316.84) - /unlink 20000 (time 1734258719.46 total 62.73 last 283.13) - /unlink 22612 (time 1734258729.46 total 72.73 last 261.15) - /unlink 25475 (time 1734258739.46 total 82.74 last 286.21) - /unlink 28203 (time 1734258749.47 total 92.74 last 272.75) - /unlink 30000 (time 1734258755.37 total 98.65 last 304.14) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 107.86 seconds: 302.47 ops/second Waiting for MDT destroys to complete oleg121-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 oleg121-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 - open/close 1663 (time 1734258780.94 total 10.01 last 166.20) - open/close 3413 (time 1734258790.95 total 20.01 last 174.94) - open/close 5412 (time 1734258800.95 total 30.01 last 199.83) - open/close 7153 (time 1734258810.95 total 40.01 last 174.09) - open/close 8856 (time 1734258820.95 total 50.02 last 170.26) - open/close 10000 (time 1734258826.56 total 55.63 last 203.88) - open/close 11938 (time 1734258836.56 total 65.63 last 193.79) - open/close 13743 (time 1734258846.56 total 75.63 last 180.50) - open/close 15498 (time 1734258856.57 total 85.63 last 175.41) - open/close 17150 (time 1734258866.57 total 95.63 last 165.16) - open/close 18927 (time 1734258876.57 total 105.64 last 177.67) - open/close 20000 (time 1734258882.85 total 111.91 last 170.94) - open/close 21703 (time 1734258892.86 total 121.92 last 170.20) - open/close 23479 (time 1734258902.86 total 131.92 last 177.60) - open/close 25146 (time 1734258912.86 total 141.92 last 166.68) - open/close 26917 (time 1734258922.86 total 151.92 last 177.08) - open/close 28918 (time 1734258932.86 total 161.93 last 200.02) - open/close 30000 (time 1734258938.12 total 167.18 last 205.93) - open/close 32096 (time 1734258948.12 total 177.18 last 209.53) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 179.91 seconds: 181.34 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3738 (time 1734258962.26 total 10.00 last 373.72) - /unlink 6684 (time 1734258972.26 total 20.00 last 294.52) - /unlink 9792 (time 1734258982.26 total 30.01 last 310.68) - /unlink 12863 (time 1734258992.26 total 40.01 last 307.06) - /unlink 16060 (time 1734259002.27 total 50.01 last 319.58) - /unlink 19286 (time 1734259012.27 total 60.01 last 322.56) - /unlink 20000 (time 1734259014.44 total 62.19 last 328.58) - /unlink 23483 (time 1734259024.44 total 72.19 last 348.28) - /unlink 26825 (time 1734259034.45 total 82.19 last 334.14) - /unlink 29963 (time 1734259044.45 total 92.19 last 313.80) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 100.41 seconds: 324.92 ops/second Waiting for MDT destroys to complete oleg121-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 oleg121-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 - open/close 1644 (time 1734259075.80 total 10.00 last 164.37) - open/close 3364 (time 1734259085.80 total 20.01 last 171.91) - open/close 5170 (time 1734259095.80 total 30.01 last 180.53) - open/close 6807 (time 1734259105.81 total 40.01 last 163.66) - open/close 8449 (time 1734259115.81 total 50.01 last 164.20) - open/close 10000 (time 1734259125.78 total 59.99 last 155.49) - open/close 11535 (time 1734259135.78 total 69.99 last 153.50) - open/close 13126 (time 1734259145.79 total 79.99 last 159.04) - open/close 14623 (time 1734259155.79 total 90.00 last 149.63) - open/close 16085 (time 1734259165.79 total 100.00 last 146.17) - open/close 17620 (time 1734259175.80 total 110.00 last 153.43) - open/close 19165 (time 1734259185.80 total 120.01 last 154.42) - open/close 20000 (time 1734259191.31 total 125.52 last 151.47) - open/close 21535 (time 1734259201.32 total 135.53 last 153.40) - open/close 23128 (time 1734259211.32 total 145.53 last 159.28) - open/close 24669 (time 1734259221.33 total 155.53 last 154.00) - open/close 26267 (time 1734259231.33 total 165.54 last 159.78) - open/close 27807 (time 1734259241.33 total 175.54 last 154.00) - open/close 29449 (time 1734259251.34 total 185.54 last 164.12) - open/close 30000 (time 1734259254.50 total 188.71 last 173.89) - open/close 31748 (time 1734259264.51 total 198.71 last 174.74) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 204.10 seconds: 159.85 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3490 (time 1734259281.41 total 10.00 last 348.98) - /unlink 7038 (time 1734259291.41 total 20.00 last 354.79) - /unlink 10000 (time 1734259301.08 total 29.67 last 306.28) - /unlink 13339 (time 1734259311.08 total 39.67 last 333.82) - /unlink 16367 (time 1734259321.08 total 49.67 last 302.78) - /unlink 19538 (time 1734259331.08 total 59.68 last 317.08) - /unlink 22147 (time 1734259341.08 total 69.68 last 260.86) - /unlink 25386 (time 1734259351.08 total 79.68 last 323.87) - /unlink 28584 (time 1734259361.09 total 89.68 last 319.72) - /unlink 30000 (time 1734259365.35 total 93.94 last 332.41) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 102.81 seconds: 317.33 ops/second Waiting for MDT destroys to complete oleg121-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg121-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 - open/close 2023 (time 1734259391.02 total 10.00 last 202.28) - open/close 3941 (time 1734259401.02 total 20.00 last 191.77) - open/close 5823 (time 1734259411.02 total 30.00 last 188.19) - open/close 7880 (time 1734259421.02 total 40.00 last 205.70) - open/close 9977 (time 1734259431.03 total 50.01 last 209.66) - open/close 11955 (time 1734259441.03 total 60.01 last 197.75) - open/close 13753 (time 1734259451.03 total 70.01 last 179.80) - open/close 15777 (time 1734259461.03 total 80.01 last 202.33) - open/close 18019 (time 1734259471.03 total 90.01 last 224.17) - open/close 20000 (time 1734259479.51 total 98.49 last 233.66) - open/close 22278 (time 1734259489.52 total 108.49 last 227.72) - open/close 24684 (time 1734259499.52 total 118.50 last 240.55) - open/close 26978 (time 1734259509.52 total 128.50 last 229.40) - open/close 29313 (time 1734259519.52 total 138.50 last 233.41) - open/close 30000 (time 1734259522.53 total 141.51 last 228.50) - open/close 32265 (time 1734259532.53 total 151.51 last 226.44) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 153.35 seconds: 212.76 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3557 (time 1734259545.81 total 10.00 last 355.69) - /unlink 6581 (time 1734259555.81 total 20.00 last 302.34) - /unlink 9829 (time 1734259565.81 total 30.00 last 324.79) - /unlink 13307 (time 1734259575.81 total 40.00 last 347.78) - /unlink 16678 (time 1734259585.81 total 50.00 last 337.10) - /unlink 20000 (time 1734259595.52 total 59.71 last 342.25) - /unlink 23532 (time 1734259605.52 total 69.71 last 353.18) - /unlink 27636 (time 1734259615.52 total 79.71 last 410.23) - /unlink 30000 (time 1734259621.37 total 85.56 last 404.31) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 92.95 seconds: 351.02 ops/second Waiting for MDT destroys to complete oleg121-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 oleg121-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 - open/close 1998 (time 1734259645.60 total 10.00 last 199.75) - open/close 4010 (time 1734259655.60 total 20.00 last 201.20) - open/close 5788 (time 1734259665.60 total 30.01 last 177.71) - open/close 7782 (time 1734259675.60 total 40.01 last 199.38) - open/close 9764 (time 1734259685.60 total 50.01 last 198.16) - open/close 11875 (time 1734259695.61 total 60.01 last 211.04) - open/close 13749 (time 1734259705.61 total 70.02 last 187.33) - open/close 15631 (time 1734259715.61 total 80.02 last 188.14) - open/close 17362 (time 1734259725.62 total 90.02 last 173.02) - open/close 19193 (time 1734259735.62 total 100.03 last 183.04) - open/close 20000 (time 1734259739.91 total 104.32 last 188.07) - open/close 22069 (time 1734259749.91 total 114.32 last 206.86) - open/close 24021 (time 1734259759.92 total 124.32 last 195.16) - open/close 25920 (time 1734259769.92 total 134.33 last 189.84) - open/close 28189 (time 1734259779.93 total 144.33 last 226.78) - open/close 30000 (time 1734259787.97 total 152.37 last 225.22) - open/close 32447 (time 1734259797.97 total 162.38 last 244.56) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 163.35 seconds: 199.72 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3988 (time 1734259810.32 total 10.00 last 398.76) - /unlink 8405 (time 1734259820.32 total 20.00 last 441.69) - /unlink 10000 (time 1734259824.13 total 23.80 last 419.45) - /unlink 13676 (time 1734259834.14 total 33.81 last 367.29) - /unlink 17292 (time 1734259844.14 total 43.81 last 361.59) - /unlink 20000 (time 1734259851.50 total 51.18 last 367.58) - /unlink 24074 (time 1734259861.50 total 61.18 last 407.38) - /unlink 27522 (time 1734259871.50 total 71.18 last 344.78) - /unlink 30000 (time 1734259878.91 total 78.58 last 334.80) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 85.21 seconds: 382.88 ops/second Waiting for MDT destroys to complete oleg121-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg121-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg121-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg121-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg121-server stop mds service on oleg121-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg121-server unloading modules via unload_modules_local on: 'oleg121-server' oleg121-server: oleg121-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg121-client: oleg121-client: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-client: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2