== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:28:32 (1734258512) Stopping clients: oleg247-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg247-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2 oleg247-server: oleg247-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing load_modules_local oleg247-server: Loading modules from /home/green/git/lustre-release/lustre oleg247-server: detected 4 online CPUs by sysfs oleg247-server: Force libcfs to create 2 CPU partitions oleg247-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg247-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg247-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg247-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 272 79720 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 302 32594 1% /mnt/lustre[OST:0] filesystem_summary: 32866 272 32594 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1685 (time 1734258575.95 total 10.01 last 168.40) - open/close 3452 (time 1734258585.96 total 20.01 last 176.58) - open/close 5140 (time 1734258595.96 total 30.02 last 168.74) - open/close 6872 (time 1734258605.96 total 40.02 last 173.20) - open/close 8548 (time 1734258615.97 total 50.02 last 167.52) - open/close 10000 (time 1734258624.45 total 58.50 last 171.24) - open/close 11471 (time 1734258634.45 total 68.51 last 147.01) - open/close 12902 (time 1734258644.47 total 78.53 last 142.76) - open/close 14481 (time 1734258654.49 total 88.54 last 157.68) - open/close 16140 (time 1734258664.50 total 98.56 last 165.67) - open/close 17684 (time 1734258674.52 total 108.58 last 154.12) - open/close 19628 (time 1734258684.52 total 118.58 last 194.32) - open/close 20000 (time 1734258686.66 total 120.71 last 174.34) - open/close 21877 (time 1734258696.66 total 130.72 last 187.65) - open/close 23964 (time 1734258706.66 total 140.72 last 208.65) - open/close 26024 (time 1734258716.67 total 150.72 last 205.91) - open/close 27703 (time 1734258726.67 total 160.73 last 167.83) - open/close 29458 (time 1734258736.67 total 170.73 last 175.50) - open/close 30000 (time 1734258739.54 total 173.59 last 189.05) - open/close 31776 (time 1734258749.54 total 183.60 last 177.56) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 188.43 seconds: 173.15 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 4411 (time 1734258765.65 total 10.00 last 441.05) - /unlink 8828 (time 1734258775.65 total 20.00 last 441.61) - /unlink 10000 (time 1734258778.86 total 23.22 last 364.61) - /unlink 14215 (time 1734258788.87 total 33.22 last 421.45) - /unlink 18962 (time 1734258798.87 total 43.22 last 474.66) - /unlink 20000 (time 1734258801.45 total 45.80 last 401.94) - /unlink 23326 (time 1734258811.45 total 55.81 last 332.50) - /unlink 26652 (time 1734258821.45 total 65.81 last 332.56) - /unlink 30000 (time 1734258830.19 total 74.54 last 383.21) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 81.39 seconds: 400.88 ops/second Waiting for MDT destroys to complete oleg247-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg247-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 - open/close 2429 (time 1734258859.48 total 10.00 last 242.81) - open/close 5362 (time 1734258869.48 total 20.00 last 293.29) - open/close 8396 (time 1734258879.49 total 30.01 last 303.33) - open/close 10000 (time 1734258886.43 total 36.95 last 230.89) - open/close 12483 (time 1734258896.43 total 46.95 last 248.28) - open/close 14576 (time 1734258906.44 total 56.96 last 209.25) - open/close 16500 (time 1734258916.44 total 66.96 last 192.34) - open/close 18447 (time 1734258926.44 total 76.96 last 194.69) - open/close 20000 (time 1734258932.92 total 83.44 last 239.63) - open/close 22345 (time 1734258942.92 total 93.44 last 234.46) - open/close 25005 (time 1734258952.92 total 103.44 last 265.95) - open/close 27747 (time 1734258962.93 total 113.45 last 274.16) - open/close 30000 (time 1734258971.96 total 122.48 last 249.48) - open/close 32397 (time 1734258981.96 total 132.48 last 239.62) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 133.44 seconds: 244.50 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 4907 (time 1734258994.17 total 10.00 last 490.68) - /unlink 10000 (time 1734259003.84 total 19.67 last 526.66) - /unlink 15488 (time 1734259013.84 total 29.67 last 548.78) - /unlink 19886 (time 1734259023.84 total 39.67 last 439.71) - /unlink 23820 (time 1734259033.85 total 49.68 last 393.32) - /unlink 28126 (time 1734259043.85 total 59.68 last 430.58) - /unlink 30000 (time 1734259047.99 total 63.82 last 452.35) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 69.63 seconds: 468.58 ops/second Waiting for MDT destroys to complete oleg247-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 oleg247-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 - open/close 2524 (time 1734259072.97 total 10.00 last 252.35) - open/close 4980 (time 1734259082.98 total 20.01 last 245.46) - open/close 7263 (time 1734259092.98 total 30.01 last 228.25) - open/close 9594 (time 1734259102.99 total 40.01 last 233.02) - open/close 10000 (time 1734259105.00 total 42.03 last 201.11) - open/close 12481 (time 1734259115.01 total 52.03 last 248.07) - open/close 15107 (time 1734259125.01 total 62.04 last 262.47) - open/close 17516 (time 1734259135.01 total 72.04 last 240.87) - open/close 19902 (time 1734259145.02 total 82.04 last 238.52) - open/close 22221 (time 1734259155.02 total 92.04 last 231.86) - open/close 24748 (time 1734259165.02 total 102.05 last 252.60) - open/close 27162 (time 1734259175.02 total 112.05 last 241.38) - open/close 29352 (time 1734259185.03 total 122.05 last 218.92) - open/close 30000 (time 1734259187.71 total 124.74 last 241.42) - open/close 32613 (time 1734259197.71 total 134.74 last 261.23) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 134.80 seconds: 242.04 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 4149 (time 1734259208.97 total 10.00 last 414.89) - /unlink 9288 (time 1734259218.98 total 20.00 last 513.81) - /unlink 13247 (time 1734259228.98 total 30.00 last 395.86) - /unlink 17288 (time 1734259238.98 total 40.00 last 404.06) - /unlink 20000 (time 1734259244.70 total 45.73 last 473.93) - /unlink 24763 (time 1734259254.70 total 55.73 last 476.20) - /unlink 29159 (time 1734259264.70 total 65.73 last 439.53) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 72.93 seconds: 447.36 ops/second Waiting for MDT destroys to complete oleg247-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg247-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 - open/close 2747 (time 1734259294.31 total 10.00 last 274.68) - open/close 5189 (time 1734259304.31 total 20.00 last 244.15) - open/close 7331 (time 1734259314.31 total 30.00 last 214.19) - open/close 9429 (time 1734259324.31 total 40.00 last 209.79) - open/close 10000 (time 1734259326.83 total 42.52 last 226.89) - open/close 12247 (time 1734259336.83 total 52.52 last 224.62) - open/close 14923 (time 1734259346.83 total 62.53 last 267.51) - open/close 17617 (time 1734259356.84 total 72.53 last 269.34) - open/close 20000 (time 1734259365.87 total 81.57 last 263.67) - open/close 22406 (time 1734259375.88 total 91.57 last 240.55) - open/close 24969 (time 1734259385.88 total 101.57 last 256.24) - open/close 27481 (time 1734259395.88 total 111.57 last 251.16) - open/close 29855 (time 1734259405.88 total 121.57 last 237.37) - open/close 32316 (time 1734259415.88 total 131.58 last 246.04) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 132.89 seconds: 245.52 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 4112 (time 1734259428.32 total 10.00 last 411.19) - /unlink 8538 (time 1734259438.32 total 20.00 last 442.59) - /unlink 10000 (time 1734259441.71 total 23.40 last 430.46) - /unlink 13985 (time 1734259451.72 total 33.40 last 398.31) - /unlink 18170 (time 1734259461.72 total 43.40 last 418.45) - /unlink 20000 (time 1734259465.56 total 47.24 last 476.61) - /unlink 25387 (time 1734259475.56 total 57.24 last 538.69) - /unlink 30000 (time 1734259485.36 total 67.04 last 470.81) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 72.48 seconds: 450.16 ops/second Waiting for MDT destroys to complete oleg247-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 oleg247-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 - open/close 2195 (time 1734259509.26 total 10.00 last 219.45) - open/close 4803 (time 1734259519.26 total 20.00 last 260.80) - open/close 7120 (time 1734259529.27 total 30.01 last 231.63) - open/close 9357 (time 1734259539.27 total 40.01 last 223.69) - open/close 10000 (time 1734259542.15 total 42.89 last 222.90) - open/close 12226 (time 1734259552.16 total 52.89 last 222.54) - open/close 14539 (time 1734259562.16 total 62.90 last 231.22) - open/close 17060 (time 1734259572.16 total 72.90 last 252.02) - open/close 19437 (time 1734259582.16 total 82.90 last 237.63) - open/close 20000 (time 1734259584.34 total 85.08 last 258.82) - open/close 22610 (time 1734259594.34 total 95.08 last 260.99) - open/close 25436 (time 1734259604.34 total 105.08 last 282.55) - open/close 28084 (time 1734259614.35 total 115.08 last 264.71) - open/close 30000 (time 1734259621.67 total 122.40 last 261.68) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 130.75 seconds: 249.53 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 6272 (time 1734259641.00 total 10.00 last 627.02) - /unlink 10000 (time 1734259647.09 total 16.10 last 611.37) - /unlink 16224 (time 1734259657.09 total 26.10 last 622.37) - /unlink 20000 (time 1734259664.56 total 33.56 last 506.03) - /unlink 25440 (time 1734259674.56 total 43.56 last 543.95) - /unlink 30000 (time 1734259682.67 total 51.68 last 562.03) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 56.30 seconds: 579.53 ops/second Waiting for MDT destroys to complete oleg247-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 oleg247-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg247-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server unloading modules via unload_modules_local on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg247-client: oleg247-client: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-client: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2