== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 00:29:28 (1708838968) Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:-f) oleg159-server: oleg159-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: detected 4 online CPUs by sysfs oleg159-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg159-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg159-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg159-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg159-client.virtnet: -o user_xattr,flock oleg159-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4555 (time 1708839005.79 total 10.00 last 455.48) - open/close 8245 (time 1708839015.79 total 20.00 last 368.98) - open/close 10000 (time 1708839020.91 total 25.13 last 342.25) - open/close 12974 (time 1708839030.92 total 35.13 last 297.36) - open/close 15501 (time 1708839040.92 total 45.13 last 252.67) - open/close 17788 (time 1708839050.92 total 55.13 last 228.62) - open/close 19679 (time 1708839060.92 total 65.14 last 189.03) - open/close 21530 (time 1708839070.93 total 75.14 last 185.06) - open/close 23210 (time 1708839080.93 total 85.15 last 167.90) - open/close 24727 (time 1708839090.93 total 95.15 last 151.66) - open/close 26070 (time 1708839100.94 total 105.15 last 134.23) - open/close 27364 (time 1708839110.94 total 115.16 last 129.34) - open/close 28504 (time 1708839120.95 total 125.16 last 113.97) - open/close 29619 (time 1708839130.96 total 135.17 last 111.38) - open/close 30000 (time 1708839134.13 total 138.35 last 120.07) - open/close 31371 (time 1708839144.13 total 148.35 last 137.08) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 157.95 seconds: 206.14 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 1238 (time 1708839164.57 total 10.00 last 123.77) - /unlink 2467 (time 1708839174.58 total 20.01 last 122.83) - /unlink 3909 (time 1708839184.58 total 30.01 last 144.17) - /unlink 5436 (time 1708839194.58 total 40.02 last 152.62) - /unlink 7219 (time 1708839204.59 total 50.02 last 178.25) - /unlink 8927 (time 1708839214.59 total 60.02 last 170.74) - /unlink 10000 (time 1708839220.39 total 65.83 last 184.83) - /unlink 12025 (time 1708839230.40 total 75.83 last 202.41) - /unlink 14181 (time 1708839240.40 total 85.83 last 215.53) - /unlink 16686 (time 1708839250.40 total 95.84 last 250.48) - /unlink 19635 (time 1708839260.41 total 105.84 last 294.83) - /unlink 23202 (time 1708839270.41 total 115.84 last 356.62) - /unlink 27944 (time 1708839280.41 total 125.84 last 474.17) - /unlink 30000 (time 1708839283.60 total 129.03 last 644.98) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 132.03 seconds: 246.61 ops/second Waiting for MDT destroys to complete - open/close 4649 (time 1708839307.48 total 10.00 last 464.84) - open/close 8019 (time 1708839317.48 total 20.00 last 337.00) - open/close 10000 (time 1708839323.90 total 26.42 last 308.77) - open/close 12762 (time 1708839333.90 total 36.42 last 276.11) - open/close 15309 (time 1708839343.90 total 46.42 last 254.66) - open/close 17537 (time 1708839353.90 total 56.42 last 222.77) - open/close 19466 (time 1708839363.91 total 66.43 last 192.80) - open/close 20000 (time 1708839366.52 total 69.04 last 204.11) - open/close 21771 (time 1708839376.53 total 79.05 last 177.05) - open/close 23612 (time 1708839386.53 total 89.05 last 184.09) - open/close 25134 (time 1708839396.53 total 99.05 last 152.12) - open/close 26551 (time 1708839406.54 total 109.06 last 141.63) - open/close 27879 (time 1708839416.54 total 119.06 last 132.76) - open/close 29094 (time 1708839426.54 total 129.06 last 121.47) - open/close 30000 (time 1708839434.61 total 137.13 last 112.27) - open/close 31201 (time 1708839444.62 total 147.14 last 120.06) - open/close 32517 (time 1708839454.62 total 157.14 last 131.60) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 157.43 seconds: 206.82 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 1340 (time 1708839465.78 total 10.00 last 133.94) - /unlink 2774 (time 1708839475.78 total 20.01 last 143.36) - /unlink 4151 (time 1708839485.78 total 30.01 last 137.70) - /unlink 5534 (time 1708839495.78 total 40.01 last 138.27) - /unlink 7152 (time 1708839505.79 total 50.01 last 161.72) - /unlink 8900 (time 1708839515.79 total 60.02 last 174.76) - /unlink 10000 (time 1708839522.48 total 66.71 last 164.44) - /unlink 11987 (time 1708839532.48 total 76.71 last 198.62) - /unlink 14136 (time 1708839542.48 total 86.71 last 214.89) - /unlink 16515 (time 1708839552.48 total 96.71 last 237.88) - /unlink 19128 (time 1708839562.49 total 106.71 last 261.21) - /unlink 20000 (time 1708839565.11 total 109.34 last 332.04) - /unlink 23631 (time 1708839575.11 total 119.34 last 363.09) - /unlink 28373 (time 1708839585.11 total 129.34 last 474.18) - /unlink 30000 (time 1708839587.75 total 131.98 last 617.01) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 135.91 seconds: 239.56 ops/second Waiting for MDT destroys to complete - open/close 3625 (time 1708839609.40 total 10.00 last 362.48) - open/close 7016 (time 1708839619.40 total 20.00 last 339.02) - open/close 10000 (time 1708839628.53 total 29.14 last 326.70) - open/close 12348 (time 1708839638.53 total 39.14 last 234.80) - open/close 14708 (time 1708839648.54 total 49.14 last 235.96) - open/close 17051 (time 1708839658.54 total 59.14 last 234.27) - open/close 18971 (time 1708839668.54 total 69.14 last 191.96) - open/close 20000 (time 1708839674.96 total 75.56 last 160.36) - open/close 21423 (time 1708839684.96 total 85.56 last 142.26) - open/close 22682 (time 1708839694.96 total 95.57 last 125.83) - open/close 24005 (time 1708839704.97 total 105.57 last 132.28) - open/close 25173 (time 1708839714.97 total 115.57 last 116.77) - open/close 26311 (time 1708839724.98 total 125.58 last 113.71) - open/close 27397 (time 1708839734.98 total 135.58 last 108.55) - open/close 28618 (time 1708839744.99 total 145.59 last 121.99) - open/close 29664 (time 1708839754.99 total 155.59 last 104.58) - open/close 30000 (time 1708839757.69 total 158.29 last 124.61) - open/close 31045 (time 1708839767.69 total 168.29 last 104.46) - open/close 32082 (time 1708839777.70 total 178.30 last 103.65) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 183.28 seconds: 177.65 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 1167 (time 1708839793.77 total 10.01 last 116.61) - /unlink 2194 (time 1708839803.78 total 20.01 last 102.64) - /unlink 3262 (time 1708839813.79 total 30.02 last 106.74) - /unlink 4477 (time 1708839823.79 total 40.02 last 121.48) - /unlink 5895 (time 1708839833.79 total 50.03 last 141.69) - /unlink 7451 (time 1708839843.80 total 60.03 last 155.53) - /unlink 8904 (time 1708839853.80 total 70.04 last 145.25) - /unlink 10000 (time 1708839861.90 total 78.13 last 135.42) - /unlink 11368 (time 1708839871.90 total 88.14 last 136.72) - /unlink 12875 (time 1708839881.91 total 98.14 last 150.59) - /unlink 14468 (time 1708839891.91 total 108.14 last 159.27) - /unlink 16334 (time 1708839901.91 total 118.15 last 186.57) - /unlink 18330 (time 1708839911.92 total 128.15 last 199.53) - /unlink 20000 (time 1708839918.86 total 135.10 last 240.34) - /unlink 22699 (time 1708839928.87 total 145.10 last 269.82) - /unlink 25042 (time 1708839938.87 total 155.10 last 234.28) - /unlink 28180 (time 1708839948.87 total 165.10 last 313.73) - /unlink 30000 (time 1708839954.31 total 170.54 last 334.73) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 176.03 seconds: 184.97 ops/second Waiting for MDT destroys to complete - open/close 2368 (time 1708839974.39 total 10.00 last 236.79) - open/close 4597 (time 1708839984.39 total 20.00 last 222.80) - open/close 6563 (time 1708839994.39 total 30.01 last 196.55) - open/close 8971 (time 1708840004.39 total 40.01 last 240.78) - open/close 10000 (time 1708840009.25 total 44.87 last 211.81) - open/close 11669 (time 1708840019.25 total 54.87 last 166.90) - open/close 13272 (time 1708840029.26 total 64.87 last 160.22) - open/close 14750 (time 1708840039.26 total 74.87 last 147.77) - open/close 16212 (time 1708840049.26 total 84.88 last 146.17) - open/close 17634 (time 1708840059.27 total 94.88 last 142.14) - open/close 18998 (time 1708840069.27 total 104.88 last 136.33) - open/close 20000 (time 1708840076.78 total 112.40 last 133.37) - open/close 21241 (time 1708840086.79 total 122.40 last 124.01) - open/close 22478 (time 1708840096.79 total 132.41 last 123.69) - open/close 23711 (time 1708840106.80 total 142.41 last 123.22) - open/close 24934 (time 1708840116.80 total 152.41 last 122.28) - open/close 26159 (time 1708840126.80 total 162.41 last 122.49) - open/close 27459 (time 1708840136.81 total 172.42 last 129.88) - open/close 28522 (time 1708840146.82 total 182.43 last 106.23) - open/close 29581 (time 1708840156.82 total 192.43 last 105.87) - open/close 30000 (time 1708840160.94 total 196.55 last 101.78) - open/close 31015 (time 1708840170.94 total 206.56 last 101.43) - open/close 32063 (time 1708840180.95 total 216.56 last 104.75) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 221.63 seconds: 146.91 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 999 (time 1708840197.33 total 10.00 last 99.87) - /unlink 2062 (time 1708840207.34 total 20.01 last 106.25) - /unlink 3185 (time 1708840217.34 total 30.01 last 112.24) - /unlink 4344 (time 1708840227.35 total 40.02 last 115.81) - /unlink 5553 (time 1708840237.35 total 50.03 last 120.84) - /unlink 6724 (time 1708840247.36 total 60.03 last 117.01) - /unlink 8077 (time 1708840257.37 total 70.04 last 135.25) - /unlink 9514 (time 1708840267.37 total 80.04 last 143.63) - /unlink 10000 (time 1708840271.00 total 83.67 last 133.79) - /unlink 11464 (time 1708840281.01 total 93.68 last 146.35) - /unlink 13695 (time 1708840291.01 total 103.68 last 223.06) - /unlink 16350 (time 1708840301.01 total 113.68 last 265.41) - /unlink 19196 (time 1708840311.01 total 123.69 last 284.55) - /unlink 20000 (time 1708840313.24 total 125.91 last 360.95) - /unlink 23688 (time 1708840323.24 total 135.91 last 368.80) - /unlink 28975 (time 1708840333.24 total 145.91 last 528.67) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 151.05 seconds: 215.56 ops/second Waiting for MDT destroys to complete - open/close 4449 (time 1708840354.38 total 10.00 last 444.90) - open/close 8357 (time 1708840364.38 total 20.00 last 390.76) - open/close 10000 (time 1708840371.76 total 27.38 last 222.76) - open/close 11811 (time 1708840381.76 total 37.38 last 181.09) - open/close 13493 (time 1708840391.76 total 47.38 last 168.20) - open/close 15052 (time 1708840401.76 total 57.38 last 155.86) - open/close 16567 (time 1708840411.76 total 67.38 last 151.49) - open/close 18017 (time 1708840421.77 total 77.38 last 144.96) - open/close 19604 (time 1708840431.77 total 87.39 last 158.63) - open/close 20000 (time 1708840434.56 total 90.18 last 141.82) - open/close 21813 (time 1708840444.57 total 100.18 last 181.25) - open/close 23479 (time 1708840454.57 total 110.19 last 166.56) - open/close 24693 (time 1708840464.58 total 120.19 last 121.30) - open/close 25878 (time 1708840474.58 total 130.20 last 118.48) - open/close 27383 (time 1708840484.58 total 140.20 last 150.43) - open/close 29051 (time 1708840494.59 total 150.21 last 166.67) - open/close 30000 (time 1708840503.39 total 159.01 last 107.81) - open/close 31410 (time 1708840513.40 total 169.02 last 140.93) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 176.64 seconds: 184.33 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 1757 (time 1708840531.97 total 10.00 last 175.63) - /unlink 3591 (time 1708840541.97 total 20.01 last 183.38) - /unlink 5227 (time 1708840551.98 total 30.01 last 163.55) - /unlink 7081 (time 1708840561.98 total 40.01 last 185.34) - /unlink 8800 (time 1708840571.99 total 50.02 last 171.81) - /unlink 10000 (time 1708840581.31 total 59.34 last 128.68) - /unlink 11634 (time 1708840591.31 total 69.34 last 163.39) - /unlink 13716 (time 1708840601.31 total 79.35 last 208.14) - /unlink 16568 (time 1708840611.32 total 89.35 last 285.17) - /unlink 19362 (time 1708840621.32 total 99.35 last 279.39) - /unlink 22448 (time 1708840631.32 total 109.35 last 308.56) - /unlink 26103 (time 1708840641.32 total 119.35 last 365.48) - /unlink 30000 (time 1708840649.86 total 127.89 last 456.14) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 133.03 seconds: 244.76 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg159-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg159-server unloading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2