== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:28:29 (1734258509) Stopping clients: oleg133-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg133-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: oleg133-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing load_modules_local oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: detected 4 online CPUs by sysfs oleg133-server: Force libcfs to create 2 CPU partitions oleg133-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg133-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg133-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg133-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 272 79720 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 302 32594 1% /mnt/lustre[OST:0] filesystem_summary: 32866 272 32594 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1452 (time 1734258572.41 total 10.00 last 145.18) - open/close 2819 (time 1734258582.41 total 20.00 last 136.69) - open/close 4091 (time 1734258592.41 total 30.01 last 127.11) - open/close 5602 (time 1734258602.42 total 40.01 last 151.05) - open/close 6969 (time 1734258612.42 total 50.01 last 136.69) - open/close 8383 (time 1734258622.42 total 60.02 last 141.36) - open/close 9792 (time 1734258632.42 total 70.02 last 140.86) - open/close 11112 (time 1734258642.42 total 80.02 last 131.99) - open/close 12517 (time 1734258652.43 total 90.02 last 140.45) - open/close 13944 (time 1734258662.44 total 100.03 last 142.57) - open/close 15501 (time 1734258672.44 total 110.03 last 155.67) - open/close 16875 (time 1734258682.44 total 120.04 last 137.34) - open/close 18501 (time 1734258692.44 total 130.04 last 162.56) - open/close 20000 (time 1734258702.28 total 139.87 last 152.46) - open/close 21526 (time 1734258712.28 total 149.88 last 152.53) - open/close 22870 (time 1734258722.29 total 159.88 last 134.34) - open/close 24161 (time 1734258732.29 total 169.89 last 129.03) - open/close 25537 (time 1734258742.29 total 179.89 last 137.58) - open/close 26832 (time 1734258752.30 total 189.89 last 129.44) - open/close 28158 (time 1734258762.30 total 199.90 last 132.52) - open/close 29440 (time 1734258772.31 total 209.91 last 128.10) - open/close 30000 (time 1734258776.20 total 213.79 last 144.08) - open/close 31457 (time 1734258786.20 total 223.80 last 145.66) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 231.13 seconds: 141.16 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3052 (time 1734258804.94 total 10.00 last 305.15) - /unlink 5784 (time 1734258814.94 total 20.00 last 273.20) - /unlink 8822 (time 1734258824.94 total 30.00 last 303.79) - /unlink 10000 (time 1734258828.21 total 33.27 last 360.00) - /unlink 13060 (time 1734258838.22 total 43.28 last 305.83) - /unlink 16081 (time 1734258848.22 total 53.28 last 302.06) - /unlink 19005 (time 1734258858.22 total 63.28 last 292.36) - /unlink 20000 (time 1734258861.93 total 67.00 last 267.87) - /unlink 22937 (time 1734258871.94 total 77.00 last 293.61) - /unlink 25871 (time 1734258881.94 total 87.00 last 293.34) - /unlink 28777 (time 1734258891.94 total 97.01 last 290.48) - /unlink 30000 (time 1734258895.79 total 100.85 last 318.37) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 109.69 seconds: 297.43 ops/second Waiting for MDT destroys to complete oleg133-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg133-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 - open/close 1623 (time 1734258926.21 total 10.00 last 162.25) - open/close 3439 (time 1734258936.21 total 20.01 last 181.53) - open/close 5317 (time 1734258946.22 total 30.02 last 187.64) - open/close 7099 (time 1734258956.22 total 40.02 last 178.18) - open/close 8711 (time 1734258966.22 total 50.02 last 161.19) - open/close 10000 (time 1734258975.29 total 59.08 last 142.25) - open/close 11494 (time 1734258985.29 total 69.08 last 149.33) - open/close 12961 (time 1734258995.29 total 79.09 last 146.66) - open/close 14494 (time 1734259005.30 total 89.09 last 153.22) - open/close 16079 (time 1734259015.30 total 99.10 last 158.44) - open/close 17696 (time 1734259025.31 total 109.10 last 161.61) - open/close 19237 (time 1734259035.31 total 119.10 last 154.06) - open/close 20000 (time 1734259040.76 total 124.55 last 140.10) - open/close 21489 (time 1734259050.76 total 134.55 last 148.87) - open/close 23022 (time 1734259060.76 total 144.56 last 153.25) - open/close 24368 (time 1734259070.77 total 154.56 last 134.48) - open/close 25793 (time 1734259080.78 total 164.57 last 142.43) - open/close 27416 (time 1734259090.78 total 174.57 last 162.28) - open/close 28830 (time 1734259100.78 total 184.57 last 141.37) - open/close 30000 (time 1734259108.55 total 192.34 last 150.60) - open/close 31367 (time 1734259118.55 total 202.35 last 136.65) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 211.63 seconds: 154.17 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 2521 (time 1734259139.71 total 10.00 last 252.07) - /unlink 5018 (time 1734259149.71 total 20.00 last 249.64) - /unlink 7282 (time 1734259159.71 total 30.01 last 226.34) - /unlink 9691 (time 1734259169.72 total 40.01 last 240.75) - /unlink 12081 (time 1734259179.72 total 50.01 last 238.96) - /unlink 14649 (time 1734259189.72 total 60.02 last 256.71) - /unlink 17059 (time 1734259199.72 total 70.02 last 241.00) - /unlink 19679 (time 1734259209.73 total 80.02 last 261.87) - /unlink 22098 (time 1734259219.73 total 90.02 last 241.85) - /unlink 24563 (time 1734259229.73 total 100.03 last 246.45) - /unlink 27114 (time 1734259239.73 total 110.03 last 255.08) - /unlink 29645 (time 1734259249.73 total 120.03 last 253.09) - /unlink 32575 (time 1734259259.73 total 130.03 last 292.96) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 130.20 seconds: 250.59 ops/second Waiting for MDT destroys to complete oleg133-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 oleg133-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 - open/close 1646 (time 1734259278.61 total 10.00 last 164.54) - open/close 3339 (time 1734259288.61 total 20.01 last 169.22) - open/close 4826 (time 1734259298.61 total 30.01 last 148.66) - open/close 6427 (time 1734259308.62 total 40.01 last 160.09) - open/close 7872 (time 1734259318.63 total 50.02 last 144.33) - open/close 9328 (time 1734259328.63 total 60.02 last 145.60) - open/close 10000 (time 1734259333.43 total 64.82 last 140.05) - open/close 11291 (time 1734259343.43 total 74.83 last 129.02) - open/close 12872 (time 1734259353.43 total 84.83 last 158.08) - open/close 14332 (time 1734259363.44 total 94.83 last 145.94) - open/close 15703 (time 1734259373.44 total 104.84 last 137.04) - open/close 17360 (time 1734259383.44 total 114.84 last 165.67) - open/close 19047 (time 1734259393.44 total 124.84 last 168.67) - open/close 20000 (time 1734259399.36 total 130.75 last 161.22) - open/close 21626 (time 1734259409.36 total 140.75 last 162.57) - open/close 23332 (time 1734259419.36 total 150.76 last 170.49) - open/close 25085 (time 1734259429.36 total 160.76 last 175.29) - open/close 26732 (time 1734259439.37 total 170.77 last 164.63) - open/close 28290 (time 1734259449.37 total 180.77 last 155.78) - open/close 29960 (time 1734259459.37 total 190.77 last 166.96) - open/close 31872 (time 1734259469.38 total 200.77 last 191.16) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 204.43 seconds: 159.60 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3709 (time 1734259484.44 total 10.00 last 370.88) - /unlink 7767 (time 1734259494.44 total 20.00 last 405.77) - /unlink 10000 (time 1734259500.55 total 26.11 last 365.71) - /unlink 13915 (time 1734259510.55 total 36.11 last 391.45) - /unlink 17734 (time 1734259520.55 total 46.11 last 381.88) - /unlink 20000 (time 1734259526.21 total 51.77 last 400.17) - /unlink 23583 (time 1734259536.21 total 61.77 last 358.26) - /unlink 26675 (time 1734259546.22 total 71.77 last 309.16) - /unlink 29427 (time 1734259556.22 total 81.78 last 275.15) - /unlink 32427 (time 1734259566.22 total 91.78 last 299.88) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 92.42 seconds: 353.03 ops/second Waiting for MDT destroys to complete oleg133-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 oleg133-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 - open/close 1566 (time 1734259586.84 total 10.00 last 156.53) - open/close 3197 (time 1734259596.85 total 20.01 last 163.02) - open/close 4885 (time 1734259606.85 total 30.01 last 168.72) - open/close 6790 (time 1734259616.86 total 40.02 last 190.46) - open/close 8499 (time 1734259626.86 total 50.02 last 170.87) - open/close 10000 (time 1734259635.91 total 59.07 last 165.89) - open/close 11785 (time 1734259645.91 total 69.07 last 178.42) - open/close 13527 (time 1734259655.92 total 79.08 last 174.12) - open/close 15137 (time 1734259665.92 total 89.08 last 160.95) - open/close 16887 (time 1734259675.92 total 99.08 last 174.94) - open/close 18659 (time 1734259685.92 total 109.08 last 177.16) - open/close 20000 (time 1734259693.19 total 116.35 last 184.61) - open/close 21678 (time 1734259703.19 total 126.35 last 167.75) - open/close 23336 (time 1734259713.19 total 136.35 last 165.79) - open/close 24797 (time 1734259723.20 total 146.36 last 146.00) - open/close 26365 (time 1734259733.21 total 156.37 last 156.68) - open/close 28085 (time 1734259743.21 total 166.37 last 171.89) - open/close 29862 (time 1734259753.21 total 176.37 last 177.68) - open/close 31530 (time 1734259763.22 total 186.38 last 166.73) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 193.25 seconds: 168.83 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3951 (time 1734259781.41 total 10.00 last 395.05) - /unlink 7932 (time 1734259791.41 total 20.00 last 398.03) - /unlink 10000 (time 1734259796.43 total 25.03 last 411.72) - /unlink 13544 (time 1734259806.43 total 35.03 last 354.34) - /unlink 17516 (time 1734259816.44 total 45.03 last 397.14) - /unlink 20000 (time 1734259823.24 total 51.83 last 365.23) - /unlink 23339 (time 1734259833.24 total 61.83 last 333.84) - /unlink 26553 (time 1734259843.24 total 71.83 last 321.39) - /unlink 29911 (time 1734259853.24 total 81.83 last 335.78) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 89.17 seconds: 365.88 ops/second Waiting for MDT destroys to complete oleg133-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 oleg133-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 - open/close 1642 (time 1734259880.51 total 10.00 last 164.19) - open/close 3476 (time 1734259890.52 total 20.00 last 183.35) - open/close 5290 (time 1734259900.52 total 30.01 last 181.30) - open/close 7300 (time 1734259910.52 total 40.01 last 200.97) - open/close 9425 (time 1734259920.53 total 50.01 last 212.41) - open/close 10000 (time 1734259923.26 total 52.75 last 210.31) - open/close 12010 (time 1734259933.26 total 62.75 last 200.99) - open/close 14089 (time 1734259943.27 total 72.75 last 207.81) - open/close 16190 (time 1734259953.27 total 82.76 last 210.04) - open/close 18410 (time 1734259963.27 total 92.76 last 221.94) - open/close 20000 (time 1734259970.99 total 100.48 last 205.89) - open/close 22171 (time 1734259981.00 total 110.48 last 217.04) - open/close 24277 (time 1734259991.00 total 120.49 last 210.57) - open/close 26413 (time 1734260001.00 total 130.49 last 213.54) - open/close 28770 (time 1734260011.01 total 140.49 last 235.60) - open/close 30000 (time 1734260016.32 total 145.81 last 231.25) - open/close 32299 (time 1734260026.33 total 155.81 last 229.85) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 157.27 seconds: 207.46 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 4431 (time 1734260038.96 total 10.00 last 443.03) - /unlink 8854 (time 1734260048.96 total 20.00 last 442.21) - /unlink 10000 (time 1734260051.69 total 22.73 last 419.94) - /unlink 14207 (time 1734260061.69 total 32.73 last 420.69) - /unlink 17747 (time 1734260071.69 total 42.74 last 353.92) - /unlink 20000 (time 1734260077.68 total 48.73 last 375.84) - /unlink 23888 (time 1734260087.69 total 58.73 last 388.71) - /unlink 28195 (time 1734260097.69 total 68.73 last 430.67) - /unlink 30000 (time 1734260102.31 total 73.36 last 390.16) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 79.50 seconds: 410.38 ops/second Waiting for MDT destroys to complete oleg133-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 oleg133-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg133-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg133-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server stop mds service on oleg133-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg133-server unloading modules via unload_modules_local on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2