== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:29:52 (1734258592) Stopping clients: oleg321-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg321-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2 oleg321-server: oleg321-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg321-server' oleg321-server: oleg321-server.virtnet: executing load_modules_local oleg321-server: Loading modules from /home/green/git/lustre-release/lustre oleg321-server: detected 4 online CPUs by sysfs oleg321-server: Force libcfs to create 2 CPU partitions oleg321-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg321-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg321-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg321-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg321-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg321-client.virtnet: -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 272 79720 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 302 32594 1% /mnt/lustre[OST:0] filesystem_summary: 32866 272 32594 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1544 (time 1734258652.04 total 10.00 last 154.34) - open/close 3222 (time 1734258662.04 total 20.01 last 167.73) - open/close 4680 (time 1734258672.05 total 30.01 last 145.73) - open/close 6110 (time 1734258682.05 total 40.02 last 142.96) - open/close 7666 (time 1734258692.05 total 50.02 last 155.54) - open/close 9118 (time 1734258702.06 total 60.02 last 145.15) - open/close 10000 (time 1734258708.12 total 66.09 last 145.45) - open/close 11453 (time 1734258718.13 total 76.09 last 145.22) - open/close 13059 (time 1734258728.13 total 86.10 last 160.54) - open/close 14690 (time 1734258738.13 total 96.10 last 163.02) - open/close 16496 (time 1734258748.14 total 106.10 last 180.54) - open/close 18194 (time 1734258758.14 total 116.11 last 169.69) - open/close 19962 (time 1734258768.15 total 126.11 last 176.78) - open/close 21705 (time 1734258778.15 total 136.11 last 174.25) - open/close 23206 (time 1734258788.16 total 146.12 last 149.97) - open/close 24726 (time 1734258798.16 total 156.13 last 151.95) - open/close 26319 (time 1734258808.17 total 166.13 last 159.23) - open/close 28200 (time 1734258818.17 total 176.13 last 188.08) - open/close 29991 (time 1734258828.17 total 186.14 last 179.05) - open/close 31985 (time 1734258838.18 total 196.14 last 199.27) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 199.82 seconds: 163.27 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3663 (time 1734258853.32 total 10.00 last 366.30) - /unlink 7592 (time 1734258863.32 total 20.00 last 392.80) - /unlink 10000 (time 1734258869.50 total 26.18 last 389.67) - /unlink 13094 (time 1734258879.50 total 36.18 last 309.36) - /unlink 16249 (time 1734258889.50 total 46.19 last 315.44) - /unlink 19254 (time 1734258899.51 total 56.19 last 300.29) - /unlink 20000 (time 1734258902.05 total 58.74 last 293.32) - /unlink 22979 (time 1734258912.05 total 68.74 last 297.84) - /unlink 25862 (time 1734258922.06 total 78.74 last 288.23) - /unlink 28804 (time 1734258932.06 total 88.74 last 294.12) - /unlink 30000 (time 1734258936.03 total 92.71 last 301.19) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 101.04 seconds: 322.92 ops/second Waiting for MDT destroys to complete oleg321-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg321-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 - open/close 1779 (time 1734258962.86 total 10.00 last 177.89) - open/close 3511 (time 1734258972.87 total 20.00 last 173.16) - open/close 5230 (time 1734258982.87 total 30.01 last 171.83) - open/close 7232 (time 1734258992.87 total 40.01 last 200.16) - open/close 9255 (time 1734259002.88 total 50.01 last 202.20) - open/close 10000 (time 1734259006.02 total 53.15 last 237.31) - open/close 12123 (time 1734259016.02 total 63.16 last 212.21) - open/close 14033 (time 1734259026.02 total 73.16 last 190.97) - open/close 15797 (time 1734259036.02 total 83.16 last 176.40) - open/close 17483 (time 1734259046.02 total 93.16 last 168.57) - open/close 19279 (time 1734259056.03 total 103.17 last 179.45) - open/close 20000 (time 1734259059.74 total 106.87 last 194.61) - open/close 21680 (time 1734259069.74 total 116.88 last 167.93) - open/close 23279 (time 1734259079.74 total 126.88 last 159.87) - open/close 24954 (time 1734259089.75 total 136.88 last 167.47) - open/close 26609 (time 1734259099.75 total 146.89 last 165.40) - open/close 28250 (time 1734259109.75 total 156.89 last 164.07) - open/close 30000 (time 1734259118.44 total 165.58 last 201.41) - open/close 31934 (time 1734259128.44 total 175.58 last 193.39) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 179.42 seconds: 181.84 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3166 (time 1734259143.74 total 10.00 last 316.60) - /unlink 6546 (time 1734259153.74 total 20.00 last 337.98) - /unlink 10000 (time 1734259163.15 total 29.41 last 367.04) - /unlink 13874 (time 1734259173.16 total 39.41 last 387.29) - /unlink 17612 (time 1734259183.16 total 49.42 last 373.67) - /unlink 20000 (time 1734259190.60 total 56.85 last 321.12) - /unlink 23729 (time 1734259200.60 total 66.85 last 372.88) - /unlink 27245 (time 1734259210.60 total 76.86 last 351.53) - /unlink 30000 (time 1734259218.77 total 85.03 last 337.25) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 92.28 seconds: 353.57 ops/second Waiting for MDT destroys to complete oleg321-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 oleg321-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 - open/close 2375 (time 1734259247.48 total 10.00 last 237.38) - open/close 4947 (time 1734259257.48 total 20.01 last 257.16) - open/close 7207 (time 1734259267.49 total 30.01 last 225.95) - open/close 9125 (time 1734259277.49 total 40.01 last 191.72) - open/close 10000 (time 1734259281.78 total 44.30 last 203.93) - open/close 12275 (time 1734259291.78 total 54.30 last 227.48) - open/close 14208 (time 1734259301.78 total 64.30 last 193.30) - open/close 16289 (time 1734259311.78 total 74.31 last 208.07) - open/close 18223 (time 1734259321.79 total 84.31 last 193.35) - open/close 20000 (time 1734259329.02 total 91.55 last 245.45) - open/close 22308 (time 1734259339.03 total 101.55 last 230.75) - open/close 24413 (time 1734259349.03 total 111.55 last 210.45) - open/close 26339 (time 1734259359.03 total 121.56 last 192.55) - open/close 28245 (time 1734259369.03 total 131.56 last 190.54) - open/close 30000 (time 1734259377.83 total 140.35 last 199.59) - open/close 31776 (time 1734259387.83 total 150.35 last 177.54) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 154.53 seconds: 211.12 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3770 (time 1734259403.44 total 10.00 last 376.95) - /unlink 7468 (time 1734259413.44 total 20.00 last 369.70) - /unlink 10000 (time 1734259420.73 total 27.29 last 347.63) - /unlink 13662 (time 1734259430.73 total 37.29 last 366.14) - /unlink 17258 (time 1734259440.73 total 47.29 last 359.59) - /unlink 20000 (time 1734259448.06 total 54.62 last 374.06) - /unlink 23558 (time 1734259458.06 total 64.62 last 355.77) - /unlink 27376 (time 1734259468.07 total 74.63 last 381.61) - /unlink 30000 (time 1734259475.14 total 81.70 last 371.02) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 88.13 seconds: 370.20 ops/second Waiting for MDT destroys to complete oleg321-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 oleg321-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 - open/close 2064 (time 1734259499.32 total 10.01 last 206.25) - open/close 3944 (time 1734259509.32 total 20.01 last 187.94) - open/close 6150 (time 1734259519.32 total 30.01 last 220.53) - open/close 8545 (time 1734259529.32 total 40.01 last 239.46) - open/close 10000 (time 1734259534.89 total 45.58 last 261.47) - open/close 12161 (time 1734259544.89 total 55.58 last 216.07) - open/close 14259 (time 1734259554.89 total 65.58 last 209.78) - open/close 16550 (time 1734259564.89 total 75.58 last 229.05) - open/close 18507 (time 1734259574.91 total 85.60 last 195.42) - open/close 20000 (time 1734259581.71 total 92.40 last 219.41) - open/close 22353 (time 1734259591.71 total 102.40 last 235.29) - open/close 24890 (time 1734259601.72 total 112.41 last 253.63) - open/close 27693 (time 1734259611.72 total 122.41 last 280.25) - open/close 30000 (time 1734259620.09 total 130.78 last 275.40) - open/close 32590 (time 1734259630.09 total 140.78 last 258.99) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 140.93 seconds: 231.51 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 4986 (time 1734259641.42 total 10.00 last 498.55) - /unlink 10000 (time 1734259651.37 total 19.95 last 503.88) - /unlink 14940 (time 1734259661.37 total 29.95 last 493.97) - /unlink 19854 (time 1734259671.37 total 39.95 last 491.29) - /unlink 24788 (time 1734259681.37 total 49.96 last 493.28) - /unlink 29826 (time 1734259691.37 total 59.96 last 503.68) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 65.06 seconds: 501.44 ops/second Waiting for MDT destroys to complete oleg321-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg321-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 - open/close 2762 (time 1734259714.55 total 10.00 last 276.14) - open/close 5216 (time 1734259724.55 total 20.01 last 245.27) - open/close 7477 (time 1734259734.56 total 30.01 last 225.99) - open/close 9760 (time 1734259744.56 total 40.01 last 228.28) - open/close 11860 (time 1734259754.56 total 50.02 last 209.93) - open/close 14116 (time 1734259764.56 total 60.02 last 225.56) - open/close 16697 (time 1734259774.57 total 70.02 last 258.06) - open/close 19355 (time 1734259784.57 total 80.02 last 265.78) - open/close 20000 (time 1734259786.87 total 82.33 last 279.55) - open/close 22642 (time 1734259796.88 total 92.33 last 264.16) - open/close 24782 (time 1734259806.88 total 102.33 last 213.96) - open/close 27141 (time 1734259816.88 total 112.34 last 235.80) - open/close 29839 (time 1734259826.88 total 122.34 last 269.80) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 132.09 seconds: 247.01 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 5593 (time 1734259847.71 total 10.00 last 559.21) - /unlink 10000 (time 1734259855.62 total 17.91 last 557.51) - /unlink 15606 (time 1734259865.62 total 27.91 last 560.50) - /unlink 20000 (time 1734259875.19 total 37.48 last 459.13) - /unlink 24520 (time 1734259885.19 total 47.48 last 451.92) - /unlink 28799 (time 1734259895.19 total 57.48 last 427.84) - /unlink 30000 (time 1734259897.95 total 60.24 last 436.16) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 66.40 seconds: 491.39 ops/second Waiting for MDT destroys to complete oleg321-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg321-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg321-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg321-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server stop mds service on oleg321-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server unloading modules via unload_modules_local on: 'oleg321-server' oleg321-server: oleg321-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2