== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:48:28 (1712357308) Stopping clients: oleg460-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg460-client.virtnet /mnt/lustre2 (opts:-f) oleg460-server: oleg460-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing load_modules_local oleg460-server: Loading modules from /home/green/git/lustre-release/lustre oleg460-server: detected 4 online CPUs by sysfs oleg460-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg460-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg460-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg460-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg460-client.virtnet: -o user_xattr,flock oleg460-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1646 (time 1712357362.20 total 10.00 last 164.60) - open/close 3648 (time 1712357372.21 total 20.01 last 200.09) - open/close 5635 (time 1712357382.21 total 30.01 last 198.68) - open/close 7636 (time 1712357392.21 total 40.01 last 200.07) - open/close 9332 (time 1712357402.21 total 50.01 last 169.53) - open/close 10000 (time 1712357406.76 total 54.56 last 147.00) - open/close 11276 (time 1712357416.76 total 64.56 last 127.52) - open/close 12380 (time 1712357426.77 total 74.57 last 110.37) - open/close 13983 (time 1712357436.77 total 84.57 last 160.29) - open/close 15594 (time 1712357446.77 total 94.57 last 161.05) - open/close 17453 (time 1712357456.77 total 104.57 last 185.88) - open/close 19392 (time 1712357466.77 total 114.57 last 193.89) - open/close 20000 (time 1712357471.10 total 118.90 last 140.41) - open/close 21261 (time 1712357481.10 total 128.90 last 126.10) - open/close 22553 (time 1712357491.11 total 138.91 last 129.15) - open/close 23574 (time 1712357501.11 total 148.91 last 102.08) - open/close 24581 (time 1712357511.11 total 158.91 last 100.67) - open/close 25622 (time 1712357521.12 total 168.92 last 104.06) - open/close 26801 (time 1712357531.13 total 178.93 last 117.77) - open/close 27636 (time 1712357541.14 total 188.94 last 83.40) - open/close 28606 (time 1712357551.14 total 198.94 last 96.96) - open/close 29552 (time 1712357561.15 total 208.95 last 94.58) - open/close 30000 (time 1712357565.02 total 212.82 last 115.53) - open/close 31200 (time 1712357575.03 total 222.83 last 119.96) - open/close 32406 (time 1712357585.04 total 232.84 last 120.50) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 234.20 seconds: 139.03 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 1093 (time 1712357597.25 total 10.00 last 109.27) - /unlink 2362 (time 1712357607.26 total 20.01 last 126.83) - /unlink 3810 (time 1712357617.26 total 30.01 last 144.79) - /unlink 5179 (time 1712357627.26 total 40.02 last 136.81) - /unlink 6740 (time 1712357637.27 total 50.02 last 156.06) - /unlink 8508 (time 1712357647.27 total 60.02 last 176.79) - /unlink 10000 (time 1712357654.48 total 67.23 last 206.86) - /unlink 12045 (time 1712357664.48 total 77.23 last 204.45) - /unlink 14225 (time 1712357674.48 total 87.24 last 217.95) - /unlink 16663 (time 1712357684.49 total 97.24 last 243.73) - /unlink 19550 (time 1712357694.49 total 107.24 last 288.61) - /unlink 22735 (time 1712357704.49 total 117.24 last 318.50) - /unlink 26178 (time 1712357714.49 total 127.25 last 344.18) - /unlink 30000 (time 1712357722.53 total 135.28 last 475.84) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 138.91 seconds: 234.39 ops/second Waiting for MDT destroys to complete - open/close 2796 (time 1712357743.16 total 10.00 last 279.59) - open/close 6056 (time 1712357753.17 total 20.00 last 325.96) - open/close 8635 (time 1712357763.17 total 30.00 last 257.85) - open/close 10000 (time 1712357767.87 total 34.71 last 290.23) - open/close 12546 (time 1712357777.87 total 44.71 last 254.58) - open/close 14264 (time 1712357787.88 total 54.71 last 171.71) - open/close 16047 (time 1712357797.88 total 64.71 last 178.30) - open/close 17555 (time 1712357807.88 total 74.72 last 150.76) - open/close 19122 (time 1712357817.88 total 84.72 last 156.70) - open/close 20000 (time 1712357824.63 total 91.46 last 130.11) - open/close 21245 (time 1712357834.63 total 101.46 last 124.49) - open/close 22594 (time 1712357844.63 total 111.47 last 134.87) - open/close 23955 (time 1712357854.63 total 121.47 last 136.09) - open/close 25413 (time 1712357864.64 total 131.48 last 145.69) - open/close 26641 (time 1712357874.65 total 141.48 last 122.70) - open/close 27939 (time 1712357884.65 total 151.49 last 129.77) - open/close 28732 (time 1712357894.66 total 161.49 last 79.25) - open/close 29369 (time 1712357904.67 total 171.50 last 63.63) - open/close 30000 (time 1712357910.60 total 177.43 last 106.38) - open/close 31026 (time 1712357920.60 total 187.44 last 102.57) - open/close 31890 (time 1712357930.60 total 197.44 last 86.38) - open/close 32504 (time 1712357940.60 total 207.44 last 61.40) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 208.15 seconds: 156.43 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 920 (time 1712357952.25 total 10.00 last 91.98) - /unlink 2007 (time 1712357962.25 total 20.00 last 108.69) - /unlink 3015 (time 1712357972.25 total 30.01 last 100.74) - /unlink 4173 (time 1712357982.26 total 40.01 last 115.76) - /unlink 5098 (time 1712357992.26 total 50.02 last 92.46) - /unlink 6044 (time 1712358002.27 total 60.02 last 94.56) - /unlink 6968 (time 1712358012.27 total 70.02 last 92.39) - /unlink 7908 (time 1712358022.27 total 80.03 last 93.97) - /unlink 8859 (time 1712358032.27 total 90.03 last 95.09) - /unlink 9860 (time 1712358042.28 total 100.03 last 100.03) - /unlink 10830 (time 1712358052.28 total 110.03 last 97.00) - /unlink 11906 (time 1712358062.28 total 120.04 last 107.54) - /unlink 13254 (time 1712358072.29 total 130.04 last 134.76) - /unlink 14889 (time 1712358082.29 total 140.05 last 163.41) - /unlink 16887 (time 1712358092.29 total 150.05 last 199.78) - /unlink 19181 (time 1712358102.30 total 160.05 last 229.35) - /unlink 20000 (time 1712358105.23 total 162.98 last 279.54) - /unlink 22971 (time 1712358115.23 total 172.98 last 297.05) - /unlink 26917 (time 1712358125.23 total 182.98 last 394.56) - /unlink 30000 (time 1712358130.80 total 188.56 last 552.98) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 191.82 seconds: 169.75 ops/second Waiting for MDT destroys to complete - open/close 4491 (time 1712358151.28 total 10.00 last 448.99) - open/close 7955 (time 1712358161.29 total 20.01 last 346.15) - open/close 10000 (time 1712358170.16 total 28.88 last 230.42) - open/close 11762 (time 1712358180.16 total 38.89 last 176.20) - open/close 13676 (time 1712358190.17 total 48.89 last 191.34) - open/close 15101 (time 1712358200.17 total 58.89 last 142.44) - open/close 16477 (time 1712358210.18 total 68.90 last 137.52) - open/close 18031 (time 1712358220.19 total 78.91 last 155.29) - open/close 19153 (time 1712358230.19 total 88.91 last 112.15) - open/close 20000 (time 1712358236.95 total 95.67 last 125.29) - open/close 21105 (time 1712358246.95 total 105.68 last 110.45) - open/close 22268 (time 1712358256.96 total 115.68 last 116.26) - open/close 23280 (time 1712358266.96 total 125.68 last 101.17) - open/close 24228 (time 1712358276.96 total 135.68 last 94.77) - open/close 25168 (time 1712358286.97 total 145.69 last 93.99) - open/close 26036 (time 1712358296.97 total 155.69 last 86.75) - open/close 27004 (time 1712358306.98 total 165.70 last 96.71) - open/close 27956 (time 1712358316.99 total 175.71 last 95.15) - open/close 28816 (time 1712358327.00 total 185.72 last 85.90) - open/close 29646 (time 1712358337.01 total 195.73 last 82.91) - open/close 30000 (time 1712358341.32 total 200.04 last 82.00) - open/close 30797 (time 1712358351.33 total 210.05 last 79.62) - open/close 31572 (time 1712358361.35 total 220.07 last 77.41) - open/close 32279 (time 1712358371.36 total 230.08 last 70.63) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 234.17 seconds: 139.05 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32880 16 100% /mnt/lustre[OST:0] filesystem_summary: 33106 33090 16 100% /mnt/lustre - /unlink 716 (time 1712358386.43 total 10.00 last 71.57) - /unlink 1571 (time 1712358396.45 total 20.02 last 85.35) - /unlink 2345 (time 1712358406.47 total 30.04 last 77.26) - /unlink 3183 (time 1712358416.48 total 40.05 last 83.66) - /unlink 3947 (time 1712358426.48 total 50.06 last 76.39) - /unlink 4753 (time 1712358436.49 total 60.06 last 80.56) - /unlink 5577 (time 1712358446.49 total 70.06 last 82.37) - /unlink 6458 (time 1712358456.50 total 80.07 last 88.02) - /unlink 7402 (time 1712358466.51 total 90.08 last 94.29) - /unlink 8491 (time 1712358476.51 total 100.09 last 108.87) - /unlink 9522 (time 1712358486.52 total 110.10 last 103.02) - /unlink 10000 (time 1712358490.97 total 114.54 last 107.58) - /unlink 11065 (time 1712358500.97 total 124.54 last 106.46) - /unlink 12331 (time 1712358510.97 total 134.54 last 126.58) - /unlink 13620 (time 1712358520.98 total 144.55 last 128.83) - /unlink 15160 (time 1712358530.98 total 154.55 last 153.95) - /unlink 16784 (time 1712358540.99 total 164.56 last 162.33) - /unlink 18708 (time 1712358550.99 total 174.56 last 192.32) - /unlink 20000 (time 1712358556.98 total 180.55 last 215.73) - /unlink 22397 (time 1712358566.98 total 190.55 last 239.64) - /unlink 24907 (time 1712358576.98 total 200.56 last 250.92) - /unlink 28150 (time 1712358586.99 total 210.56 last 324.22) - /unlink 30000 (time 1712358590.47 total 214.04 last 530.95) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 218.05 seconds: 149.32 ops/second Waiting for MDT destroys to complete - open/close 3436 (time 1712358609.49 total 10.00 last 343.57) - open/close 6422 (time 1712358619.50 total 20.00 last 298.51) - open/close 8811 (time 1712358629.50 total 30.01 last 238.86) - open/close 10000 (time 1712358634.81 total 35.32 last 223.94) - open/close 12236 (time 1712358644.81 total 45.32 last 223.59) - open/close 14036 (time 1712358654.81 total 55.32 last 179.97) - open/close 15722 (time 1712358664.81 total 65.32 last 168.51) - open/close 17397 (time 1712358674.82 total 75.33 last 167.44) - open/close 18819 (time 1712358684.82 total 85.33 last 142.12) - open/close 20000 (time 1712358693.22 total 93.73 last 140.69) - open/close 21335 (time 1712358703.22 total 103.73 last 133.44) - open/close 22540 (time 1712358713.23 total 113.74 last 120.44) - open/close 23890 (time 1712358723.23 total 123.74 last 134.96) - open/close 25025 (time 1712358733.24 total 133.75 last 113.41) - open/close 26083 (time 1712358743.24 total 143.75 last 105.75) - open/close 27079 (time 1712358753.24 total 153.75 last 99.58) - open/close 28122 (time 1712358763.25 total 163.76 last 104.20) - open/close 29149 (time 1712358773.26 total 173.77 last 102.63) - open/close 30000 (time 1712358781.87 total 182.38 last 98.86) - open/close 30923 (time 1712358791.87 total 192.38 last 92.24) - open/close 31830 (time 1712358801.88 total 202.39 last 90.67) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 210.08 seconds: 154.99 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32831 47161 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33086 33086 0 100% /mnt/lustre - /unlink 936 (time 1712358820.48 total 10.00 last 93.56) - /unlink 2019 (time 1712358830.48 total 20.01 last 108.28) - /unlink 3260 (time 1712358840.49 total 30.01 last 124.03) - /unlink 4568 (time 1712358850.49 total 40.02 last 130.75) - /unlink 6000 (time 1712358860.50 total 50.02 last 143.14) - /unlink 7478 (time 1712358870.50 total 60.02 last 147.73) - /unlink 9146 (time 1712358880.50 total 70.03 last 166.77) - /unlink 10000 (time 1712358885.52 total 75.05 last 170.14) - /unlink 11702 (time 1712358895.53 total 85.05 last 170.10) - /unlink 13742 (time 1712358905.53 total 95.05 last 203.93) - /unlink 16037 (time 1712358915.53 total 105.06 last 229.48) - /unlink 18658 (time 1712358925.53 total 115.06 last 262.09) - /unlink 20000 (time 1712358929.88 total 119.40 last 308.62) - /unlink 23454 (time 1712358939.88 total 129.41 last 345.34) - /unlink 28264 (time 1712358949.88 total 139.41 last 481.00) - /unlink 30000 (time 1712358952.47 total 142.00 last 670.02) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 145.03 seconds: 224.51 ops/second Waiting for MDT destroys to complete - open/close 3515 (time 1712358975.50 total 10.00 last 351.49) - open/close 7632 (time 1712358985.50 total 20.00 last 411.67) - open/close 10000 (time 1712358992.70 total 27.20 last 328.88) - open/close 13010 (time 1712359002.70 total 37.20 last 300.97) - open/close 15559 (time 1712359012.70 total 47.20 last 254.85) - open/close 17970 (time 1712359022.70 total 57.21 last 241.06) - open/close 20000 (time 1712359032.08 total 66.58 last 216.44) - open/close 21864 (time 1712359042.08 total 76.59 last 186.36) - open/close 23526 (time 1712359052.08 total 86.59 last 166.19) - open/close 25213 (time 1712359062.09 total 96.59 last 168.58) - open/close 26771 (time 1712359072.10 total 106.60 last 155.68) - open/close 28307 (time 1712359082.10 total 116.60 last 153.56) - open/close 29706 (time 1712359092.11 total 126.61 last 139.78) - open/close 30000 (time 1712359094.18 total 128.69 last 141.67) - open/close 31381 (time 1712359104.19 total 138.69 last 138.03) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 147.64 seconds: 220.54 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 1346 (time 1712359124.06 total 10.01 last 134.52) - /unlink 2732 (time 1712359134.07 total 20.01 last 138.59) - /unlink 4126 (time 1712359144.07 total 30.01 last 139.36) - /unlink 5633 (time 1712359154.07 total 40.01 last 150.64) - /unlink 7235 (time 1712359164.07 total 50.01 last 160.19) - /unlink 9045 (time 1712359174.08 total 60.02 last 180.96) - /unlink 10000 (time 1712359179.04 total 64.98 last 192.43) - /unlink 12111 (time 1712359189.04 total 74.98 last 211.06) - /unlink 14145 (time 1712359199.04 total 84.98 last 203.39) - /unlink 16667 (time 1712359209.04 total 94.99 last 252.11) - /unlink 19498 (time 1712359219.05 total 104.99 last 283.07) - /unlink 22746 (time 1712359229.05 total 114.99 last 324.80) - /unlink 27169 (time 1712359239.05 total 124.99 last 442.23) - /unlink 30000 (time 1712359244.48 total 130.42 last 521.51) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 134.33 seconds: 242.39 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg460-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg460-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server stop mds service on oleg460-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg460-server stop mds service on oleg460-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg460-server unloading modules on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2