== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:28:42 (1734258522) Stopping clients: oleg120-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg120-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 oleg120-server: oleg120-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: detected 4 online CPUs by sysfs oleg120-server: Force libcfs to create 2 CPU partitions oleg120-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg120-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg120-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg120-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 272 79720 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 302 32594 1% /mnt/lustre[OST:0] filesystem_summary: 32866 272 32594 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1355 (time 1734258586.02 total 10.00 last 135.48) - open/close 2715 (time 1734258596.02 total 20.00 last 135.99) - open/close 4264 (time 1734258606.03 total 30.00 last 154.87) - open/close 5562 (time 1734258616.03 total 40.01 last 129.75) - open/close 6952 (time 1734258626.04 total 50.01 last 138.91) - open/close 8316 (time 1734258636.04 total 60.02 last 136.32) - open/close 9604 (time 1734258646.04 total 70.02 last 128.76) - open/close 10000 (time 1734258648.64 total 72.62 last 152.57) - open/close 11387 (time 1734258658.64 total 82.62 last 138.64) - open/close 12912 (time 1734258668.65 total 92.63 last 152.39) - open/close 14321 (time 1734258678.66 total 102.64 last 140.82) - open/close 15864 (time 1734258688.66 total 112.64 last 154.29) - open/close 17498 (time 1734258698.66 total 122.64 last 163.40) - open/close 18998 (time 1734258708.66 total 132.64 last 149.95) - open/close 20000 (time 1734258716.01 total 139.99 last 136.39) - open/close 21298 (time 1734258726.01 total 149.99 last 129.76) - open/close 22644 (time 1734258736.02 total 160.00 last 134.52) - open/close 23967 (time 1734258746.02 total 170.00 last 132.25) - open/close 25334 (time 1734258756.03 total 180.01 last 136.62) - open/close 26623 (time 1734258766.03 total 190.01 last 128.84) - open/close 28030 (time 1734258776.04 total 200.01 last 140.65) - open/close 29543 (time 1734258786.04 total 210.02 last 151.26) - open/close 30000 (time 1734258789.01 total 212.99 last 153.97) - open/close 31709 (time 1734258799.01 total 222.99 last 170.81) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 228.66 seconds: 142.68 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 2791 (time 1734258816.35 total 10.00 last 279.05) - /unlink 5945 (time 1734258826.35 total 20.00 last 315.32) - /unlink 9119 (time 1734258836.35 total 30.01 last 317.28) - /unlink 10000 (time 1734258839.19 total 32.84 last 310.72) - /unlink 13002 (time 1734258849.19 total 42.85 last 300.07) - /unlink 15830 (time 1734258859.19 total 52.85 last 282.75) - /unlink 18819 (time 1734258869.20 total 62.85 last 298.81) - /unlink 20000 (time 1734258873.32 total 66.97 last 286.84) - /unlink 23015 (time 1734258883.32 total 76.97 last 301.49) - /unlink 26042 (time 1734258893.32 total 86.97 last 302.63) - /unlink 29090 (time 1734258903.32 total 96.98 last 304.70) - /unlink 30000 (time 1734258906.39 total 100.04 last 296.50) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 108.92 seconds: 299.55 ops/second Waiting for MDT destroys to complete oleg120-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 oleg120-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 - open/close 1821 (time 1734258936.68 total 10.00 last 182.04) - open/close 3713 (time 1734258946.69 total 20.01 last 189.09) - open/close 5519 (time 1734258956.69 total 30.01 last 180.57) - open/close 7101 (time 1734258966.69 total 40.01 last 158.16) - open/close 8606 (time 1734258976.69 total 50.01 last 150.48) - open/close 10000 (time 1734258986.03 total 59.35 last 149.30) - open/close 11442 (time 1734258996.04 total 69.36 last 144.11) - open/close 12953 (time 1734259006.04 total 79.36 last 151.04) - open/close 14554 (time 1734259016.05 total 89.37 last 160.02) - open/close 16209 (time 1734259026.05 total 99.37 last 165.47) - open/close 17682 (time 1734259036.05 total 109.37 last 147.28) - open/close 19183 (time 1734259046.05 total 119.37 last 150.10) - open/close 20000 (time 1734259051.57 total 124.89 last 148.05) - open/close 21554 (time 1734259061.57 total 134.89 last 155.31) - open/close 22963 (time 1734259071.58 total 144.90 last 140.86) - open/close 24385 (time 1734259081.58 total 154.90 last 142.13) - open/close 26044 (time 1734259091.58 total 164.91 last 165.83) - open/close 27458 (time 1734259101.59 total 174.91 last 141.32) - open/close 28966 (time 1734259111.60 total 184.92 last 150.70) - open/close 30000 (time 1734259119.21 total 192.53 last 135.82) - open/close 31415 (time 1734259129.22 total 202.54 last 141.41) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 211.79 seconds: 154.05 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 2541 (time 1734259150.34 total 10.00 last 254.04) - /unlink 4903 (time 1734259160.35 total 20.00 last 236.16) - /unlink 7359 (time 1734259170.35 total 30.01 last 245.56) - /unlink 9765 (time 1734259180.35 total 40.01 last 240.49) - /unlink 12433 (time 1734259190.35 total 50.01 last 266.79) - /unlink 14795 (time 1734259200.35 total 60.01 last 236.14) - /unlink 17341 (time 1734259210.36 total 70.02 last 254.44) - /unlink 19862 (time 1734259220.36 total 80.02 last 252.08) - /unlink 22330 (time 1734259230.36 total 90.02 last 246.78) - /unlink 24905 (time 1734259240.37 total 100.03 last 257.35) - /unlink 27624 (time 1734259250.37 total 110.03 last 271.89) - /unlink 30000 (time 1734259258.74 total 118.40 last 283.70) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 127.22 seconds: 256.46 ops/second Waiting for MDT destroys to complete oleg120-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg120-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 - open/close 1729 (time 1734259287.70 total 10.01 last 172.79) - open/close 3195 (time 1734259297.70 total 20.01 last 146.52) - open/close 4797 (time 1734259307.71 total 30.02 last 160.10) - open/close 6247 (time 1734259317.71 total 40.02 last 144.98) - open/close 7693 (time 1734259327.71 total 50.02 last 144.58) - open/close 9075 (time 1734259337.72 total 60.02 last 138.14) - open/close 10000 (time 1734259344.70 total 67.00 last 132.56) - open/close 11571 (time 1734259354.70 total 77.01 last 157.06) - open/close 12982 (time 1734259364.70 total 87.01 last 141.03) - open/close 14388 (time 1734259374.70 total 97.01 last 140.59) - open/close 16079 (time 1734259384.71 total 107.01 last 169.06) - open/close 17751 (time 1734259394.71 total 117.02 last 167.09) - open/close 19416 (time 1734259404.72 total 127.02 last 166.43) - open/close 20000 (time 1734259408.44 total 130.74 last 156.97) - open/close 21694 (time 1734259418.45 total 140.75 last 169.27) - open/close 23538 (time 1734259428.45 total 150.76 last 184.32) - open/close 25237 (time 1734259438.45 total 160.76 last 169.83) - open/close 26830 (time 1734259448.46 total 170.76 last 159.25) - open/close 28507 (time 1734259458.46 total 180.77 last 167.66) - open/close 30000 (time 1734259466.34 total 188.65 last 189.37) - open/close 31978 (time 1734259476.35 total 198.65 last 197.76) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 202.17 seconds: 161.38 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3932 (time 1734259491.20 total 10.00 last 393.16) - /unlink 7859 (time 1734259501.20 total 20.00 last 392.67) - /unlink 10000 (time 1734259506.76 total 25.56 last 384.96) - /unlink 13891 (time 1734259516.76 total 35.57 last 389.01) - /unlink 17911 (time 1734259526.76 total 45.57 last 401.94) - /unlink 20000 (time 1734259532.34 total 51.14 last 374.93) - /unlink 23381 (time 1734259542.34 total 61.14 last 338.06) - /unlink 26238 (time 1734259552.34 total 71.15 last 285.54) - /unlink 29164 (time 1734259562.34 total 81.15 last 292.53) - /unlink 30000 (time 1734259565.05 total 83.86 last 308.68) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 92.17 seconds: 353.97 ops/second Waiting for MDT destroys to complete oleg120-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 oleg120-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 - open/close 1542 (time 1734259593.20 total 10.00 last 154.19) - open/close 3219 (time 1734259603.21 total 20.00 last 167.64) - open/close 5076 (time 1734259613.21 total 30.00 last 185.69) - open/close 6901 (time 1734259623.21 total 40.01 last 182.37) - open/close 8543 (time 1734259633.22 total 50.02 last 164.10) - open/close 10000 (time 1734259641.57 total 58.37 last 174.46) - open/close 11772 (time 1734259651.57 total 68.37 last 177.14) - open/close 13485 (time 1734259661.58 total 78.38 last 171.22) - open/close 15179 (time 1734259671.58 total 88.38 last 169.33) - open/close 16855 (time 1734259681.58 total 98.38 last 167.56) - open/close 18762 (time 1734259691.59 total 108.38 last 190.68) - open/close 20000 (time 1734259698.74 total 115.54 last 173.01) - open/close 21667 (time 1734259708.75 total 125.55 last 166.60) - open/close 23175 (time 1734259718.75 total 135.55 last 150.71) - open/close 24692 (time 1734259728.76 total 145.55 last 151.68) - open/close 26305 (time 1734259738.76 total 155.56 last 161.27) - open/close 28157 (time 1734259748.76 total 165.56 last 185.14) - open/close 29868 (time 1734259758.76 total 175.56 last 171.03) - open/close 31475 (time 1734259768.77 total 185.56 last 160.69) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 191.74 seconds: 170.16 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3603 (time 1734259786.13 total 10.00 last 360.24) - /unlink 7573 (time 1734259796.13 total 20.00 last 396.90) - /unlink 10000 (time 1734259803.46 total 27.33 last 331.43) - /unlink 13685 (time 1734259813.46 total 37.33 last 368.43) - /unlink 17355 (time 1734259823.46 total 47.33 last 366.99) - /unlink 20000 (time 1734259831.38 total 55.25 last 333.97) - /unlink 23171 (time 1734259841.38 total 65.25 last 317.00) - /unlink 26363 (time 1734259851.38 total 75.25 last 319.14) - /unlink 29914 (time 1734259861.39 total 85.26 last 354.92) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 94.10 seconds: 346.73 ops/second Waiting for MDT destroys to complete oleg120-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 oleg120-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 - open/close 1698 (time 1734259886.04 total 10.00 last 169.73) - open/close 3540 (time 1734259896.04 total 20.01 last 184.14) - open/close 5409 (time 1734259906.04 total 30.01 last 186.89) - open/close 7470 (time 1734259916.04 total 40.01 last 206.07) - open/close 9600 (time 1734259926.04 total 50.01 last 212.93) - open/close 11670 (time 1734259936.05 total 60.01 last 207.00) - open/close 13714 (time 1734259946.05 total 70.01 last 204.36) - open/close 15836 (time 1734259956.05 total 80.02 last 212.10) - open/close 18092 (time 1734259966.05 total 90.02 last 225.59) - open/close 20000 (time 1734259974.99 total 98.95 last 213.54) - open/close 22213 (time 1734259984.99 total 108.96 last 221.21) - open/close 24396 (time 1734259995.00 total 118.96 last 218.21) - open/close 26678 (time 1734260005.00 total 128.97 last 228.11) - open/close 28801 (time 1734260015.00 total 138.97 last 212.25) - open/close 30000 (time 1734260020.76 total 144.73 last 208.10) - open/close 32133 (time 1734260030.76 total 154.73 last 213.28) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 157.10 seconds: 207.68 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3962 (time 1734260044.38 total 10.00 last 396.19) - /unlink 8051 (time 1734260054.38 total 20.00 last 408.87) - /unlink 10000 (time 1734260059.59 total 25.21 last 374.13) - /unlink 13368 (time 1734260069.59 total 35.21 last 336.69) - /unlink 16803 (time 1734260079.60 total 45.22 last 343.45) - /unlink 20000 (time 1734260088.42 total 54.04 last 362.15) - /unlink 23785 (time 1734260098.43 total 64.05 last 378.42) - /unlink 27489 (time 1734260108.43 total 74.05 last 370.34) - /unlink 30000 (time 1734260114.87 total 80.49 last 389.92) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 87.42 seconds: 373.23 ops/second Waiting for MDT destroys to complete oleg120-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 oleg120-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg120-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server unloading modules via unload_modules_local on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2