== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:29:19 (1734258559) Stopping clients: oleg137-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg137-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2 oleg137-server: oleg137-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing load_modules_local oleg137-server: Loading modules from /home/green/git/lustre-release/lustre oleg137-server: detected 4 online CPUs by sysfs oleg137-server: Force libcfs to create 2 CPU partitions oleg137-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg137-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg137-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg137-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 272 79720 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 302 32594 1% /mnt/lustre[OST:0] filesystem_summary: 32866 272 32594 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1405 (time 1734258624.18 total 10.00 last 140.47) - open/close 2823 (time 1734258634.18 total 20.01 last 141.71) - open/close 4149 (time 1734258644.19 total 30.01 last 132.53) - open/close 5593 (time 1734258654.19 total 40.02 last 144.36) - open/close 7092 (time 1734258664.19 total 50.02 last 149.88) - open/close 8635 (time 1734258674.20 total 60.03 last 154.19) - open/close 10000 (time 1734258683.59 total 69.41 last 145.44) - open/close 11666 (time 1734258693.59 total 79.42 last 166.47) - open/close 13197 (time 1734258703.60 total 89.42 last 153.05) - open/close 14720 (time 1734258713.60 total 99.43 last 152.23) - open/close 16020 (time 1734258723.61 total 109.43 last 129.91) - open/close 17326 (time 1734258733.61 total 119.44 last 130.57) - open/close 18656 (time 1734258743.62 total 129.44 last 132.94) - open/close 19972 (time 1734258753.62 total 139.45 last 131.48) - open/close 21320 (time 1734258763.63 total 149.45 last 134.73) - open/close 22669 (time 1734258773.63 total 159.46 last 134.86) - open/close 24148 (time 1734258783.64 total 169.46 last 147.85) - open/close 25729 (time 1734258793.64 total 179.46 last 158.06) - open/close 27358 (time 1734258803.64 total 189.46 last 162.88) - open/close 28863 (time 1734258813.64 total 199.46 last 150.49) - open/close 30000 (time 1734258821.18 total 207.00 last 150.81) - open/close 31750 (time 1734258831.18 total 217.00 last 175.00) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 222.52 seconds: 146.62 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 2997 (time 1734258848.22 total 10.00 last 299.64) - /unlink 5790 (time 1734258858.22 total 20.01 last 279.20) - /unlink 8608 (time 1734258868.22 total 30.01 last 281.74) - /unlink 10000 (time 1734258873.10 total 34.89 last 285.08) - /unlink 12851 (time 1734258883.11 total 44.89 last 285.02) - /unlink 15749 (time 1734258893.11 total 54.89 last 289.76) - /unlink 18723 (time 1734258903.11 total 64.89 last 297.39) - /unlink 20000 (time 1734258907.54 total 69.33 last 287.99) - /unlink 22867 (time 1734258917.54 total 79.33 last 286.69) - /unlink 26085 (time 1734258927.54 total 89.33 last 321.79) - /unlink 29603 (time 1734258937.54 total 99.33 last 351.79) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 107.47 seconds: 303.58 ops/second Waiting for MDT destroys to complete oleg137-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg137-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 - open/close 1703 (time 1734258963.40 total 10.00 last 170.26) - open/close 3103 (time 1734258973.41 total 20.01 last 139.94) - open/close 4562 (time 1734258983.41 total 30.01 last 145.82) - open/close 6043 (time 1734258993.42 total 40.02 last 148.03) - open/close 7557 (time 1734259003.42 total 50.02 last 151.33) - open/close 9091 (time 1734259013.42 total 60.03 last 153.33) - open/close 10000 (time 1734259019.20 total 65.80 last 157.40) - open/close 11668 (time 1734259029.21 total 75.81 last 166.71) - open/close 13114 (time 1734259039.21 total 85.81 last 144.58) - open/close 14599 (time 1734259049.21 total 95.81 last 148.48) - open/close 16163 (time 1734259059.21 total 105.81 last 156.36) - open/close 17592 (time 1734259069.21 total 115.81 last 142.85) - open/close 19054 (time 1734259079.22 total 125.82 last 146.16) - open/close 20000 (time 1734259085.35 total 131.95 last 154.24) - open/close 21552 (time 1734259095.35 total 141.95 last 155.16) - open/close 23009 (time 1734259105.36 total 151.96 last 145.59) - open/close 24462 (time 1734259115.37 total 161.97 last 145.10) - open/close 25823 (time 1734259125.38 total 171.98 last 136.00) - open/close 27153 (time 1734259135.38 total 181.98 last 132.97) - open/close 28540 (time 1734259145.38 total 191.98 last 138.69) - open/close 29842 (time 1734259155.39 total 201.99 last 130.17) - open/close 31118 (time 1734259165.39 total 211.99 last 127.50) - open/close 32461 (time 1734259175.40 total 222.00 last 134.27) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 223.31 seconds: 146.10 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 2580 (time 1734259188.59 total 10.00 last 257.95) - /unlink 4890 (time 1734259198.59 total 20.00 last 230.98) - /unlink 7452 (time 1734259208.59 total 30.01 last 256.12) - /unlink 9836 (time 1734259218.60 total 40.01 last 238.30) - /unlink 12261 (time 1734259228.60 total 50.01 last 242.44) - /unlink 14724 (time 1734259238.60 total 60.01 last 246.28) - /unlink 17276 (time 1734259248.60 total 70.02 last 255.16) - /unlink 20000 (time 1734259258.08 total 79.49 last 287.45) - /unlink 22868 (time 1734259268.08 total 89.49 last 286.79) - /unlink 26015 (time 1734259278.08 total 99.49 last 314.64) - /unlink 29276 (time 1734259288.08 total 109.50 last 326.04) - /unlink 30000 (time 1734259290.50 total 111.91 last 299.32) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 121.67 seconds: 268.14 ops/second Waiting for MDT destroys to complete oleg137-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 oleg137-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 - open/close 1475 (time 1734259324.78 total 10.01 last 147.42) - open/close 2913 (time 1734259334.78 total 20.01 last 143.77) - open/close 4224 (time 1734259344.79 total 30.01 last 131.02) - open/close 5804 (time 1734259354.79 total 40.02 last 157.94) - open/close 7250 (time 1734259364.80 total 50.03 last 144.47) - open/close 8666 (time 1734259374.80 total 60.03 last 141.59) - open/close 10000 (time 1734259382.91 total 68.14 last 164.49) - open/close 11664 (time 1734259392.91 total 78.14 last 166.37) - open/close 13297 (time 1734259402.92 total 88.14 last 163.24) - open/close 14962 (time 1734259412.92 total 98.14 last 166.48) - open/close 16722 (time 1734259422.92 total 108.14 last 176.00) - open/close 18450 (time 1734259432.92 total 118.15 last 172.78) - open/close 20000 (time 1734259442.03 total 127.25 last 170.22) - open/close 21551 (time 1734259452.03 total 137.25 last 155.04) - open/close 23339 (time 1734259462.03 total 147.26 last 178.75) - open/close 25297 (time 1734259472.04 total 157.26 last 195.71) - open/close 27222 (time 1734259482.04 total 167.27 last 192.44) - open/close 29264 (time 1734259492.05 total 177.27 last 204.10) - open/close 30000 (time 1734259495.39 total 180.61 last 220.33) - open/close 31911 (time 1734259505.39 total 190.62 last 191.01) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 194.21 seconds: 168.00 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3797 (time 1734259520.19 total 10.00 last 379.66) - /unlink 7664 (time 1734259530.19 total 20.00 last 386.65) - /unlink 10000 (time 1734259537.27 total 27.08 last 330.13) - /unlink 13054 (time 1734259547.27 total 37.08 last 305.38) - /unlink 15678 (time 1734259557.28 total 47.08 last 262.27) - /unlink 18595 (time 1734259567.28 total 57.09 last 291.60) - /unlink 20000 (time 1734259571.85 total 61.66 last 307.53) - /unlink 23135 (time 1734259581.85 total 71.66 last 313.43) - /unlink 26039 (time 1734259591.85 total 81.66 last 290.32) - /unlink 29134 (time 1734259601.86 total 91.67 last 309.37) - /unlink 30000 (time 1734259604.47 total 94.28 last 331.79) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 101.64 seconds: 320.99 ops/second Waiting for MDT destroys to complete oleg137-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg137-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 - open/close 1659 (time 1734259632.63 total 10.00 last 165.89) - open/close 3393 (time 1734259642.63 total 20.00 last 173.38) - open/close 5174 (time 1734259652.64 total 30.01 last 177.97) - open/close 6906 (time 1734259662.65 total 40.01 last 173.10) - open/close 8658 (time 1734259672.65 total 50.02 last 175.12) - open/close 10000 (time 1734259680.86 total 58.23 last 163.50) - open/close 11881 (time 1734259690.86 total 68.23 last 188.07) - open/close 13592 (time 1734259700.86 total 78.23 last 171.01) - open/close 15264 (time 1734259710.87 total 88.24 last 167.10) - open/close 16782 (time 1734259720.88 total 98.25 last 151.69) - open/close 18382 (time 1734259730.88 total 108.25 last 159.93) - open/close 20000 (time 1734259740.65 total 118.02 last 165.68) - open/close 21826 (time 1734259750.66 total 128.03 last 182.40) - open/close 23564 (time 1734259760.66 total 138.03 last 173.78) - open/close 25244 (time 1734259770.66 total 148.03 last 167.93) - open/close 27266 (time 1734259780.66 total 158.03 last 202.19) - open/close 29249 (time 1734259790.67 total 168.04 last 198.22) - open/close 30000 (time 1734259793.98 total 171.35 last 226.72) - open/close 31812 (time 1734259803.98 total 181.35 last 181.14) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 185.74 seconds: 175.66 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3846 (time 1734259819.80 total 10.00 last 384.57) - /unlink 7408 (time 1734259829.80 total 20.00 last 356.12) - /unlink 10000 (time 1734259837.85 total 28.05 last 322.17) - /unlink 13269 (time 1734259847.85 total 38.05 last 326.84) - /unlink 16941 (time 1734259857.85 total 48.05 last 367.16) - /unlink 20000 (time 1734259867.41 total 57.61 last 319.93) - /unlink 23182 (time 1734259877.41 total 67.62 last 318.08) - /unlink 26654 (time 1734259887.41 total 77.62 last 347.19) - /unlink 30000 (time 1734259896.52 total 86.72 last 367.63) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 94.26 seconds: 346.14 ops/second Waiting for MDT destroys to complete oleg137-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 oleg137-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 - open/close 2182 (time 1734259922.68 total 10.01 last 218.06) - open/close 4242 (time 1734259932.69 total 20.01 last 205.94) - open/close 6390 (time 1734259942.69 total 30.01 last 214.71) - open/close 8522 (time 1734259952.69 total 40.02 last 213.13) - open/close 10000 (time 1734259959.04 total 46.37 last 232.81) - open/close 12237 (time 1734259969.05 total 56.37 last 223.59) - open/close 14375 (time 1734259979.05 total 66.37 last 213.75) - open/close 16510 (time 1734259989.05 total 76.37 last 213.46) - open/close 18744 (time 1734259999.05 total 86.38 last 223.35) - open/close 20000 (time 1734260004.39 total 91.72 last 235.16) - open/close 22204 (time 1734260014.40 total 101.72 last 220.30) - open/close 24385 (time 1734260024.40 total 111.72 last 218.08) - open/close 26619 (time 1734260034.40 total 121.73 last 223.33) - open/close 28735 (time 1734260044.41 total 131.73 last 211.50) - open/close 30000 (time 1734260049.90 total 137.23 last 230.19) - open/close 32022 (time 1734260059.91 total 147.23 last 202.15) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 150.20 seconds: 217.21 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32878 18 100% /mnt/lustre[OST:0] filesystem_summary: 32917 32899 18 100% /mnt/lustre - /unlink 3340 (time 1734260074.24 total 10.00 last 334.00) - /unlink 6740 (time 1734260084.24 total 20.00 last 339.94) - /unlink 10000 (time 1734260092.67 total 28.43 last 386.72) - /unlink 13634 (time 1734260102.67 total 38.43 last 363.30) - /unlink 17518 (time 1734260112.67 total 48.44 last 388.36) - /unlink 20000 (time 1734260119.04 total 54.80 last 390.00) - /unlink 24264 (time 1734260129.04 total 64.80 last 426.33) - /unlink 28860 (time 1734260139.04 total 74.80 last 459.58) - /unlink 30000 (time 1734260141.43 total 77.19 last 477.38) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 82.88 seconds: 393.67 ops/second Waiting for MDT destroys to complete oleg137-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg137-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg137-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg137-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg137-server stop mds service on oleg137-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg137-server unloading modules via unload_modules_local on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg137-client: oleg137-client: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-client: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2