== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:30:30 (1734258630) Stopping clients: oleg352-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg352-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 oleg352-server: oleg352-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: detected 4 online CPUs by sysfs oleg352-server: Force libcfs to create 2 CPU partitions oleg352-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg352-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg352-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg352-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg352-client.virtnet: -o user_xattr,flock 192.168.203.152@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 272 79720 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 302 32594 1% /mnt/lustre[OST:0] filesystem_summary: 32866 272 32594 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1519 (time 1734258688.13 total 10.00 last 151.83) - open/close 3044 (time 1734258698.13 total 20.01 last 152.40) - open/close 4479 (time 1734258708.13 total 30.01 last 143.49) - open/close 5992 (time 1734258718.14 total 40.01 last 151.25) - open/close 7661 (time 1734258728.14 total 50.02 last 166.88) - open/close 9340 (time 1734258738.14 total 60.02 last 167.83) - open/close 10000 (time 1734258741.75 total 63.63 last 183.07) - open/close 11788 (time 1734258751.75 total 73.63 last 178.77) - open/close 13424 (time 1734258761.75 total 83.63 last 163.59) - open/close 15268 (time 1734258771.75 total 93.63 last 184.35) - open/close 16865 (time 1734258781.75 total 103.63 last 159.69) - open/close 18382 (time 1734258791.76 total 113.63 last 151.66) - open/close 19814 (time 1734258801.77 total 123.65 last 142.93) - open/close 21457 (time 1734258811.78 total 133.66 last 164.25) - open/close 23218 (time 1734258821.78 total 143.66 last 176.09) - open/close 25078 (time 1734258831.78 total 153.66 last 185.99) - open/close 26912 (time 1734258841.78 total 163.66 last 183.38) - open/close 28695 (time 1734258851.78 total 173.66 last 178.26) - open/close 30000 (time 1734258858.72 total 180.60 last 188.07) - open/close 31871 (time 1734258868.72 total 190.60 last 187.04) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 195.16 seconds: 167.18 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 2810 (time 1734258885.01 total 10.00 last 280.96) - /unlink 5644 (time 1734258895.01 total 20.00 last 283.36) - /unlink 8353 (time 1734258905.01 total 30.00 last 270.86) - /unlink 10000 (time 1734258911.10 total 36.09 last 270.52) - /unlink 12571 (time 1734258921.10 total 46.10 last 257.01) - /unlink 15268 (time 1734258931.10 total 56.10 last 269.68) - /unlink 18019 (time 1734258941.11 total 66.10 last 274.97) - /unlink 20000 (time 1734258947.92 total 72.92 last 290.67) - /unlink 22724 (time 1734258957.93 total 82.92 last 272.35) - /unlink 25749 (time 1734258967.93 total 92.92 last 302.49) - /unlink 28681 (time 1734258977.93 total 102.92 last 293.15) - /unlink 30000 (time 1734258982.39 total 107.38 last 295.82) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 115.52 seconds: 282.42 ops/second Waiting for MDT destroys to complete oleg352-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg352-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 - open/close 1997 (time 1734259008.63 total 10.00 last 199.63) - open/close 3800 (time 1734259018.64 total 20.01 last 180.22) - open/close 5484 (time 1734259028.64 total 30.01 last 168.32) - open/close 7058 (time 1734259038.65 total 40.02 last 157.35) - open/close 8613 (time 1734259048.65 total 50.02 last 155.41) - open/close 10000 (time 1734259057.18 total 58.55 last 162.58) - open/close 11577 (time 1734259067.19 total 68.56 last 157.64) - open/close 13075 (time 1734259077.19 total 78.56 last 149.73) - open/close 14546 (time 1734259087.19 total 88.56 last 147.07) - open/close 16052 (time 1734259097.20 total 98.57 last 150.53) - open/close 17561 (time 1734259107.20 total 108.57 last 150.85) - open/close 19280 (time 1734259117.20 total 118.57 last 171.85) - open/close 20000 (time 1734259121.16 total 122.53 last 181.92) - open/close 21644 (time 1734259131.17 total 132.54 last 164.33) - open/close 23178 (time 1734259141.18 total 142.55 last 153.26) - open/close 24757 (time 1734259151.18 total 152.55 last 157.89) - open/close 26433 (time 1734259161.18 total 162.55 last 167.59) - open/close 28207 (time 1734259171.18 total 172.55 last 177.35) - open/close 29958 (time 1734259181.18 total 182.55 last 175.02) - open/close 31489 (time 1734259191.19 total 192.56 last 152.98) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 199.57 seconds: 163.49 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 2953 (time 1734259209.68 total 10.00 last 295.28) - /unlink 5794 (time 1734259219.68 total 20.00 last 284.00) - /unlink 8916 (time 1734259229.68 total 30.01 last 312.17) - /unlink 10000 (time 1734259233.21 total 33.54 last 306.99) - /unlink 13335 (time 1734259243.21 total 43.54 last 333.42) - /unlink 17725 (time 1734259253.21 total 53.54 last 438.97) - /unlink 20000 (time 1734259258.89 total 59.21 last 401.18) - /unlink 23600 (time 1734259268.89 total 69.21 last 359.88) - /unlink 26827 (time 1734259278.89 total 79.22 last 322.60) - /unlink 30000 (time 1734259287.17 total 87.49 last 383.49) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 95.05 seconds: 343.26 ops/second Waiting for MDT destroys to complete oleg352-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg352-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 - open/close 1744 (time 1734259314.32 total 10.00 last 174.34) - open/close 3571 (time 1734259324.32 total 20.00 last 182.68) - open/close 5637 (time 1734259334.32 total 30.01 last 206.58) - open/close 7587 (time 1734259344.32 total 40.01 last 194.95) - open/close 9254 (time 1734259354.32 total 50.01 last 166.65) - open/close 10000 (time 1734259358.81 total 54.50 last 166.22) - open/close 11632 (time 1734259368.82 total 64.50 last 163.13) - open/close 13315 (time 1734259378.82 total 74.51 last 168.20) - open/close 14973 (time 1734259388.83 total 84.51 last 165.73) - open/close 16748 (time 1734259398.83 total 94.52 last 177.43) - open/close 18423 (time 1734259408.84 total 104.53 last 167.32) - open/close 20000 (time 1734259418.54 total 114.23 last 162.53) - open/close 21659 (time 1734259428.55 total 124.23 last 165.86) - open/close 23286 (time 1734259438.55 total 134.24 last 162.61) - open/close 24972 (time 1734259448.55 total 144.24 last 168.59) - open/close 26613 (time 1734259458.56 total 154.24 last 164.03) - open/close 28342 (time 1734259468.56 total 164.25 last 172.83) - open/close 30000 (time 1734259478.32 total 174.01 last 169.86) - open/close 31926 (time 1734259488.32 total 184.01 last 192.57) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 188.06 seconds: 173.49 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3096 (time 1734259503.70 total 10.00 last 309.48) - /unlink 6243 (time 1734259513.70 total 20.01 last 314.65) - /unlink 9813 (time 1734259523.70 total 30.01 last 356.97) - /unlink 14193 (time 1734259533.71 total 40.01 last 437.92) - /unlink 17785 (time 1734259543.71 total 50.01 last 359.20) - /unlink 20000 (time 1734259550.02 total 56.32 last 351.04) - /unlink 23722 (time 1734259560.02 total 66.32 last 372.12) - /unlink 27191 (time 1734259570.02 total 76.32 last 346.88) - /unlink 30000 (time 1734259578.53 total 84.83 last 330.16) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 91.16 seconds: 357.92 ops/second Waiting for MDT destroys to complete oleg352-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg352-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 - open/close 2423 (time 1734259604.22 total 10.00 last 242.30) - open/close 4908 (time 1734259614.22 total 20.00 last 248.45) - open/close 7308 (time 1734259624.23 total 30.01 last 239.85) - open/close 9662 (time 1734259634.23 total 40.01 last 235.30) - open/close 12045 (time 1734259644.24 total 50.01 last 238.25) - open/close 14481 (time 1734259654.24 total 60.02 last 243.54) - open/close 16707 (time 1734259664.24 total 70.02 last 222.60) - open/close 19003 (time 1734259674.24 total 80.02 last 229.58) - open/close 20000 (time 1734259678.67 total 84.45 last 224.87) - open/close 22254 (time 1734259688.67 total 94.45 last 225.40) - open/close 24584 (time 1734259698.68 total 104.46 last 232.92) - open/close 27085 (time 1734259708.68 total 114.46 last 250.04) - open/close 29333 (time 1734259718.68 total 124.46 last 224.75) - open/close 30000 (time 1734259721.97 total 127.75 last 202.81) - open/close 32016 (time 1734259731.97 total 137.75 last 201.53) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 141.05 seconds: 231.31 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3860 (time 1734259746.62 total 10.00 last 386.00) - /unlink 7327 (time 1734259756.63 total 20.00 last 346.66) - /unlink 10000 (time 1734259763.38 total 26.76 last 395.73) - /unlink 14133 (time 1734259773.38 total 36.76 last 413.29) - /unlink 18533 (time 1734259783.38 total 46.76 last 439.85) - /unlink 20000 (time 1734259786.70 total 50.07 last 443.02) - /unlink 24416 (time 1734259796.70 total 60.07 last 441.52) - /unlink 27970 (time 1734259806.70 total 70.08 last 355.30) - /unlink 30000 (time 1734259812.06 total 75.44 last 378.49) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 81.90 seconds: 398.38 ops/second Waiting for MDT destroys to complete oleg352-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg352-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 - open/close 2431 (time 1734259839.87 total 10.00 last 243.05) - open/close 4975 (time 1734259849.87 total 20.00 last 254.40) - open/close 7392 (time 1734259859.87 total 30.00 last 241.68) - open/close 9622 (time 1734259869.88 total 40.01 last 222.94) - open/close 11764 (time 1734259879.88 total 50.01 last 214.11) - open/close 13681 (time 1734259889.88 total 60.01 last 191.63) - open/close 15621 (time 1734259899.89 total 70.02 last 193.97) - open/close 17678 (time 1734259909.89 total 80.02 last 205.70) - open/close 19909 (time 1734259919.89 total 90.02 last 223.03) - open/close 22142 (time 1734259929.89 total 100.02 last 223.22) - open/close 24360 (time 1734259939.89 total 110.02 last 221.77) - open/close 26418 (time 1734259949.90 total 120.03 last 205.73) - open/close 28534 (time 1734259959.90 total 130.03 last 211.60) - open/close 30000 (time 1734259967.12 total 137.25 last 202.86) - open/close 32182 (time 1734259977.13 total 147.26 last 218.13) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 149.47 seconds: 218.27 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 4307 (time 1734259990.53 total 10.00 last 430.63) - /unlink 9321 (time 1734260000.53 total 20.00 last 501.37) - /unlink 15134 (time 1734260010.53 total 30.00 last 581.26) - /unlink 20000 (time 1734260020.04 total 39.51 last 511.87) - /unlink 25070 (time 1734260030.04 total 49.51 last 506.95) - /unlink 30000 (time 1734260039.50 total 58.96 last 521.47) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 64.64 seconds: 504.76 ops/second Waiting for MDT destroys to complete oleg352-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 oleg352-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg352-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server unloading modules via unload_modules_local on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg352-client: oleg352-client: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-client: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2