== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:30:39 (1734258639) Stopping clients: oleg203-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg203-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 oleg203-server: oleg203-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing load_modules_local oleg203-server: Loading modules from /home/green/git/lustre-release/lustre oleg203-server: detected 4 online CPUs by sysfs oleg203-server: Force libcfs to create 2 CPU partitions oleg203-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg203-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg203-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg203-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 272 79720 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 302 32594 1% /mnt/lustre[OST:0] filesystem_summary: 32866 272 32594 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1678 (time 1734258703.21 total 10.00 last 167.80) - open/close 3500 (time 1734258713.21 total 20.00 last 182.14) - open/close 5037 (time 1734258723.21 total 30.01 last 153.66) - open/close 6513 (time 1734258733.21 total 40.01 last 147.60) - open/close 8146 (time 1734258743.22 total 50.01 last 163.25) - open/close 9667 (time 1734258753.22 total 60.01 last 152.05) - open/close 11706 (time 1734258763.22 total 70.01 last 203.85) - open/close 13690 (time 1734258773.23 total 80.02 last 198.21) - open/close 15300 (time 1734258783.24 total 90.03 last 160.89) - open/close 17346 (time 1734258793.24 total 100.04 last 204.52) - open/close 19285 (time 1734258803.24 total 110.04 last 193.89) - open/close 20000 (time 1734258808.11 total 114.90 last 146.85) - open/close 21383 (time 1734258818.12 total 124.91 last 138.21) - open/close 23081 (time 1734258828.12 total 134.91 last 169.75) - open/close 24673 (time 1734258838.12 total 144.92 last 159.15) - open/close 26433 (time 1734258848.13 total 154.92 last 175.91) - open/close 28418 (time 1734258858.14 total 164.93 last 198.38) - open/close 30000 (time 1734258865.02 total 171.81 last 229.94) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 181.38 seconds: 179.87 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 4328 (time 1734258885.75 total 10.00 last 432.73) - /unlink 8716 (time 1734258895.75 total 20.00 last 438.75) - /unlink 10000 (time 1734258898.84 total 23.10 last 415.11) - /unlink 13486 (time 1734258908.84 total 33.10 last 348.57) - /unlink 16732 (time 1734258918.84 total 43.10 last 324.57) - /unlink 20000 (time 1734258927.96 total 52.21 last 358.47) - /unlink 24176 (time 1734258937.96 total 62.22 last 417.50) - /unlink 28872 (time 1734258947.96 total 72.22 last 469.57) - /unlink 30000 (time 1734258950.49 total 74.74 last 446.74) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 80.32 seconds: 406.18 ops/second Waiting for MDT destroys to complete oleg203-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg203-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 - open/close 2266 (time 1734258974.81 total 10.00 last 226.59) - open/close 4440 (time 1734258984.81 total 20.00 last 217.30) - open/close 6582 (time 1734258994.81 total 30.01 last 214.17) - open/close 8892 (time 1734259004.81 total 40.01 last 230.97) - open/close 10000 (time 1734259009.26 total 44.45 last 249.43) - open/close 12114 (time 1734259019.26 total 54.45 last 211.36) - open/close 13861 (time 1734259029.26 total 64.46 last 174.61) - open/close 15577 (time 1734259039.27 total 74.46 last 171.55) - open/close 17508 (time 1734259049.27 total 84.46 last 193.04) - open/close 19284 (time 1734259059.27 total 94.46 last 177.58) - open/close 20000 (time 1734259062.71 total 97.90 last 208.12) - open/close 22062 (time 1734259072.71 total 107.91 last 206.17) - open/close 24032 (time 1734259082.72 total 117.91 last 196.93) - open/close 25910 (time 1734259092.72 total 127.91 last 187.70) - open/close 27816 (time 1734259102.72 total 137.92 last 190.57) - open/close 29597 (time 1734259112.72 total 147.92 last 178.08) - open/close 31875 (time 1734259122.73 total 157.92 last 227.72) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 161.84 seconds: 201.60 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3685 (time 1734259137.82 total 10.00 last 368.50) - /unlink 7627 (time 1734259147.82 total 20.00 last 394.17) - /unlink 10000 (time 1734259154.26 total 26.44 last 368.34) - /unlink 14012 (time 1734259164.26 total 36.44 last 401.19) - /unlink 18110 (time 1734259174.26 total 46.44 last 409.74) - /unlink 20000 (time 1734259179.41 total 51.59 last 367.03) - /unlink 23870 (time 1734259189.41 total 61.60 last 386.97) - /unlink 28045 (time 1734259199.41 total 71.60 last 417.46) - /unlink 30000 (time 1734259204.80 total 76.98 last 363.17) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 83.56 seconds: 390.44 ops/second Waiting for MDT destroys to complete oleg203-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 oleg203-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 - open/close 1901 (time 1734259232.93 total 10.00 last 190.09) - open/close 3931 (time 1734259242.93 total 20.00 last 202.99) - open/close 6117 (time 1734259252.93 total 30.00 last 218.59) - open/close 8268 (time 1734259262.93 total 40.00 last 215.05) - open/close 10000 (time 1734259270.89 total 47.96 last 217.62) - open/close 12579 (time 1734259280.90 total 57.97 last 257.82) - open/close 15258 (time 1734259290.90 total 67.97 last 267.85) - open/close 17597 (time 1734259300.90 total 77.97 last 233.88) - open/close 19615 (time 1734259310.90 total 87.97 last 201.76) - open/close 20000 (time 1734259313.21 total 90.28 last 166.88) - open/close 21893 (time 1734259323.21 total 100.28 last 189.24) - open/close 23930 (time 1734259333.21 total 110.28 last 203.67) - open/close 26057 (time 1734259343.21 total 120.28 last 212.67) - open/close 28302 (time 1734259353.22 total 130.28 last 224.49) - open/close 30000 (time 1734259360.88 total 137.95 last 221.61) - open/close 31953 (time 1734259370.88 total 147.95 last 195.18) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 151.54 seconds: 215.30 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 4110 (time 1734259385.76 total 10.00 last 410.94) - /unlink 8296 (time 1734259395.76 total 20.00 last 418.58) - /unlink 10000 (time 1734259399.89 total 24.14 last 411.77) - /unlink 14133 (time 1734259409.90 total 34.14 last 413.28) - /unlink 18045 (time 1734259419.90 total 44.14 last 391.09) - /unlink 20000 (time 1734259425.21 total 49.46 last 367.90) - /unlink 23790 (time 1734259435.21 total 59.46 last 378.94) - /unlink 27344 (time 1734259445.22 total 69.46 last 355.35) - /unlink 30000 (time 1734259452.03 total 76.27 last 389.88) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 83.18 seconds: 392.22 ops/second Waiting for MDT destroys to complete oleg203-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg203-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 - open/close 2141 (time 1734259479.40 total 10.00 last 214.08) - open/close 4058 (time 1734259489.40 total 20.00 last 191.66) - open/close 6137 (time 1734259499.41 total 30.01 last 207.85) - open/close 7902 (time 1734259509.41 total 40.01 last 176.49) - open/close 9910 (time 1734259519.41 total 50.01 last 200.70) - open/close 11741 (time 1734259529.42 total 60.02 last 182.97) - open/close 13486 (time 1734259539.42 total 70.02 last 174.49) - open/close 15315 (time 1734259549.42 total 80.02 last 182.85) - open/close 17174 (time 1734259559.43 total 90.03 last 185.83) - open/close 19285 (time 1734259569.43 total 100.03 last 211.04) - open/close 20000 (time 1734259572.97 total 103.57 last 202.03) - open/close 21755 (time 1734259582.97 total 113.57 last 175.44) - open/close 23693 (time 1734259592.97 total 123.57 last 193.80) - open/close 25859 (time 1734259602.97 total 133.57 last 216.57) - open/close 27867 (time 1734259612.97 total 143.57 last 200.77) - open/close 29931 (time 1734259622.97 total 153.57 last 206.38) - open/close 32262 (time 1734259632.98 total 163.58 last 233.03) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 165.36 seconds: 197.31 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 4811 (time 1734259645.94 total 10.00 last 481.06) - /unlink 9293 (time 1734259655.94 total 20.00 last 448.17) - /unlink 13187 (time 1734259665.94 total 30.00 last 389.39) - /unlink 16973 (time 1734259675.94 total 40.00 last 378.56) - /unlink 20000 (time 1734259682.98 total 47.04 last 430.08) - /unlink 24330 (time 1734259692.98 total 57.04 last 432.98) - /unlink 29524 (time 1734259702.98 total 67.04 last 519.31) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 73.09 seconds: 446.40 ops/second Waiting for MDT destroys to complete oleg203-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 oleg203-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 - open/close 2087 (time 1734259724.81 total 10.00 last 208.63) - open/close 4333 (time 1734259734.82 total 20.01 last 224.50) - open/close 6196 (time 1734259744.82 total 30.01 last 186.28) - open/close 8423 (time 1734259754.82 total 40.01 last 222.59) - open/close 10000 (time 1734259763.03 total 48.22 last 192.07) - open/close 11795 (time 1734259773.04 total 58.23 last 179.43) - open/close 13685 (time 1734259783.04 total 68.23 last 188.94) - open/close 15688 (time 1734259793.04 total 78.23 last 200.26) - open/close 17907 (time 1734259803.05 total 88.24 last 221.81) - open/close 20000 (time 1734259811.25 total 96.44 last 255.20) - open/close 22355 (time 1734259821.25 total 106.44 last 235.42) - open/close 24766 (time 1734259831.25 total 116.44 last 241.04) - open/close 26984 (time 1734259841.26 total 126.45 last 221.73) - open/close 29349 (time 1734259851.26 total 136.45 last 236.45) - open/close 30000 (time 1734259855.06 total 140.25 last 171.11) - open/close 31969 (time 1734259865.06 total 150.26 last 196.89) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 153.20 seconds: 212.96 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3946 (time 1734259879.22 total 10.00 last 394.55) - /unlink 8059 (time 1734259889.22 total 20.00 last 411.28) - /unlink 10000 (time 1734259892.99 total 23.77 last 515.15) - /unlink 14401 (time 1734259902.99 total 33.77 last 440.00) - /unlink 18932 (time 1734259912.99 total 43.77 last 453.07) - /unlink 20000 (time 1734259915.56 total 46.34 last 415.67) - /unlink 25243 (time 1734259925.56 total 56.34 last 524.19) - /unlink 30000 (time 1734259933.67 total 64.45 last 586.72) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 68.62 seconds: 475.46 ops/second Waiting for MDT destroys to complete oleg203-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg203-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg203-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server stop mds service on oleg203-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server unloading modules via unload_modules_local on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2