== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 04:24:23 (1761121463) Stopping clients: oleg102-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg102-client.virtnet /mnt/lustre2 (opts:-f) oleg102-server: oleg102-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg102-server' oleg102-server: oleg102-server.virtnet: executing load_modules_local oleg102-server: Loading modules from /home/green/git/lustre-release/lustre oleg102-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg102-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg102-server: detected 4 online CPUs by sysfs oleg102-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg102-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg102-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg102-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg102-client.virtnet: -o user_xattr,flock oleg102-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 3414 (time 1761121521.77 total 10.00 last 341.33) - open/close 6919 (time 1761121531.77 total 20.00 last 350.44) - open/close 9901 (time 1761121541.77 total 30.01 last 298.00) - open/close 12624 (time 1761121551.78 total 40.01 last 272.25) - open/close 15355 (time 1761121561.78 total 50.01 last 273.10) - open/close 17840 (time 1761121571.78 total 60.02 last 248.44) - open/close 20000 (time 1761121581.48 total 69.71 last 222.76) - open/close 22715 (time 1761121591.48 total 79.72 last 271.40) - open/close 24237 (time 1761121601.48 total 89.72 last 152.19) - open/close 25601 (time 1761121611.48 total 99.72 last 136.39) - open/close 27088 (time 1761121621.48 total 109.72 last 148.68) - open/close 29276 (time 1761121631.48 total 119.72 last 218.76) - open/close 30000 (time 1761121634.46 total 122.70 last 243.29) - open/close 32112 (time 1761121644.46 total 132.70 last 211.14) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 135.23 seconds: 241.03 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 4196 (time 1761121659.09 total 10.00 last 419.40) - /unlink 9556 (time 1761121669.09 total 20.01 last 535.86) - /unlink 14730 (time 1761121679.09 total 30.01 last 517.36) - /unlink 18768 (time 1761121689.09 total 40.01 last 403.79) - /unlink 20000 (time 1761121691.72 total 42.64 last 467.71) - /unlink 25430 (time 1761121701.73 total 52.64 last 542.91) - /unlink 30000 (time 1761121709.97 total 60.89 last 554.03) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 66.21 seconds: 492.28 ops/second Waiting for MDT destroys to complete - open/close 2830 (time 1761121739.67 total 10.00 last 282.91) - open/close 5260 (time 1761121749.67 total 20.00 last 242.98) - open/close 7562 (time 1761121759.68 total 30.01 last 230.16) - open/close 10000 (time 1761121768.00 total 38.33 last 292.95) - open/close 13248 (time 1761121778.00 total 48.33 last 324.74) - open/close 16221 (time 1761121788.00 total 58.33 last 297.22) - open/close 18708 (time 1761121798.00 total 68.33 last 248.67) - open/close 20000 (time 1761121802.88 total 73.22 last 264.67) - open/close 22376 (time 1761121812.89 total 83.22 last 237.47) - open/close 24977 (time 1761121822.89 total 93.22 last 260.03) - open/close 27428 (time 1761121832.89 total 103.22 last 245.08) - open/close 30000 (time 1761121842.49 total 112.82 last 267.91) - open/close 32498 (time 1761121852.50 total 122.83 last 249.73) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 123.20 seconds: 264.56 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 4707 (time 1761121864.79 total 10.00 last 470.66) - /unlink 10000 (time 1761121874.78 total 19.99 last 530.00) - /unlink 15891 (time 1761121884.78 total 29.99 last 589.06) - /unlink 20000 (time 1761121891.25 total 36.46 last 634.66) - /unlink 25319 (time 1761121901.25 total 46.46 last 531.87) - /unlink 30000 (time 1761121910.85 total 56.06 last 487.52) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 60.75 seconds: 536.57 ops/second Waiting for MDT destroys to complete - open/close 2955 (time 1761121939.50 total 10.00 last 295.48) - open/close 5635 (time 1761121949.50 total 20.00 last 267.99) - open/close 8112 (time 1761121959.50 total 30.00 last 247.62) - open/close 10000 (time 1761121966.85 total 37.35 last 256.98) - open/close 12867 (time 1761121976.85 total 47.35 last 286.64) - open/close 15548 (time 1761121986.85 total 57.36 last 268.03) - open/close 17918 (time 1761121996.85 total 67.36 last 236.98) - open/close 20000 (time 1761122004.09 total 74.59 last 287.77) - open/close 23043 (time 1761122014.09 total 84.59 last 304.27) - open/close 25437 (time 1761122024.10 total 94.60 last 239.11) - open/close 26232 (time 1761122034.10 total 104.61 last 79.47) - open/close 27077 (time 1761122044.11 total 114.62 last 84.44) - open/close 27956 (time 1761122054.12 total 124.63 last 87.78) - open/close 28709 (time 1761122064.12 total 134.63 last 75.30) - open/close 29504 (time 1761122074.12 total 144.63 last 79.50) - open/close 30000 (time 1761122080.25 total 150.75 last 80.99) - open/close 30651 (time 1761122090.26 total 160.76 last 65.03) - open/close 31413 (time 1761122100.26 total 170.77 last 76.17) - open/close 32313 (time 1761122110.27 total 180.78 last 89.92) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 183.37 seconds: 177.75 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1924 (time 1761122125.63 total 10.01 last 192.30) - /unlink 4073 (time 1761122135.63 total 20.01 last 214.86) - /unlink 5945 (time 1761122145.64 total 30.01 last 187.11) - /unlink 7767 (time 1761122155.64 total 40.01 last 182.16) - /unlink 9587 (time 1761122165.64 total 50.02 last 181.96) - /unlink 10000 (time 1761122167.95 total 52.32 last 179.22) - /unlink 11776 (time 1761122177.95 total 62.32 last 177.59) - /unlink 13774 (time 1761122187.95 total 72.32 last 199.75) - /unlink 15582 (time 1761122197.95 total 82.33 last 180.77) - /unlink 17675 (time 1761122207.95 total 92.33 last 209.30) - /unlink 20000 (time 1761122216.70 total 101.07 last 265.86) - /unlink 21665 (time 1761122226.70 total 111.07 last 166.45) - /unlink 23531 (time 1761122236.70 total 121.07 last 186.58) - /unlink 25633 (time 1761122246.71 total 131.08 last 210.09) - /unlink 27480 (time 1761122256.71 total 141.08 last 184.62) - /unlink 29342 (time 1761122266.71 total 151.09 last 186.17) - /unlink 30000 (time 1761122270.21 total 154.58 last 188.12) - /unlink 32277 (time 1761122280.21 total 164.58 last 227.67) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 165.79 seconds: 196.59 ops/second Waiting for MDT destroys to complete - open/close 967 (time 1761122303.55 total 10.01 last 96.59) - open/close 1939 (time 1761122313.56 total 20.02 last 97.13) - open/close 2861 (time 1761122323.56 total 30.02 last 92.18) - open/close 3749 (time 1761122333.58 total 40.03 last 88.69) - open/close 4351 (time 1761122343.59 total 50.04 last 60.13) - open/close 4789 (time 1761122353.62 total 60.08 last 43.64) - open/close 5357 (time 1761122363.65 total 70.11 last 56.63) - open/close 5942 (time 1761122373.65 total 80.11 last 58.50) - open/close 6437 (time 1761122383.66 total 90.12 last 49.47) - open/close 7042 (time 1761122393.67 total 100.12 last 60.45) - open/close 7721 (time 1761122403.67 total 110.13 last 67.87) - open/close 8467 (time 1761122413.68 total 120.13 last 74.56) - open/close 9245 (time 1761122423.68 total 130.14 last 77.76) - open/close 10000 (time 1761122433.64 total 140.10 last 75.80) - open/close 10761 (time 1761122443.65 total 150.10 last 76.05) - open/close 11564 (time 1761122453.66 total 160.11 last 80.24) - open/close 12395 (time 1761122463.66 total 170.12 last 83.04) - open/close 13170 (time 1761122473.66 total 180.12 last 77.49) - open/close 13846 (time 1761122483.68 total 190.13 last 67.51) - open/close 14510 (time 1761122493.69 total 200.15 last 66.31) - open/close 15208 (time 1761122503.70 total 210.15 last 69.77) - open/close 15897 (time 1761122513.70 total 220.15 last 68.90) - open/close 16606 (time 1761122523.70 total 230.16 last 70.88) - open/close 17316 (time 1761122533.71 total 240.16 last 70.94) - open/close 17934 (time 1761122543.72 total 250.18 last 61.72) - open/close 18663 (time 1761122553.74 total 260.20 last 72.77) - open/close 19400 (time 1761122563.75 total 270.20 last 73.64) - open/close 20000 (time 1761122571.04 total 277.50 last 82.25) - open/close 20876 (time 1761122581.05 total 287.51 last 87.50) - open/close 21610 (time 1761122591.06 total 297.52 last 73.32) - open/close 22371 (time 1761122601.07 total 307.52 last 76.08) - open/close 23132 (time 1761122611.08 total 317.54 last 75.96) - open/close 23862 (time 1761122621.09 total 327.55 last 72.94) - open/close 24617 (time 1761122631.10 total 337.55 last 75.46) - open/close 25318 (time 1761122641.10 total 347.56 last 70.06) - open/close 26008 (time 1761122651.12 total 357.57 last 68.92) - open/close 26783 (time 1761122661.12 total 367.58 last 77.44) - open/close 27565 (time 1761122671.13 total 377.59 last 78.14) - open/close 28285 (time 1761122681.15 total 387.61 last 71.85) - open/close 28995 (time 1761122691.16 total 397.61 last 70.96) - open/close 29731 (time 1761122701.17 total 407.62 last 73.53) - open/close 30000 (time 1761122704.78 total 411.23 last 74.51) - open/close 30746 (time 1761122714.78 total 421.23 last 74.59) - open/close 31491 (time 1761122724.78 total 431.24 last 74.50) - open/close 32207 (time 1761122734.79 total 441.24 last 71.54) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 447.18 seconds: 72.89 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1319 (time 1761122754.94 total 10.00 last 131.85) - /unlink 2700 (time 1761122764.94 total 20.01 last 138.04) - /unlink 4033 (time 1761122774.95 total 30.01 last 133.26) - /unlink 5283 (time 1761122784.95 total 40.01 last 124.99) - /unlink 6709 (time 1761122794.95 total 50.02 last 142.53) - /unlink 8152 (time 1761122804.95 total 60.02 last 144.28) - /unlink 9614 (time 1761122814.96 total 70.02 last 146.14) - /unlink 10000 (time 1761122817.77 total 72.83 last 137.34) - /unlink 11389 (time 1761122827.77 total 82.84 last 138.86) - /unlink 12853 (time 1761122837.78 total 92.84 last 146.36) - /unlink 14171 (time 1761122847.78 total 102.84 last 131.74) - /unlink 15445 (time 1761122857.78 total 112.85 last 127.36) - /unlink 16695 (time 1761122867.79 total 122.86 last 124.87) - /unlink 17983 (time 1761122877.80 total 132.86 last 128.70) - /unlink 19320 (time 1761122887.80 total 142.87 last 133.67) - /unlink 20000 (time 1761122893.05 total 148.12 last 129.47) - /unlink 21298 (time 1761122903.06 total 158.12 last 129.76) - /unlink 22663 (time 1761122913.06 total 168.12 last 136.45) - /unlink 23971 (time 1761122923.07 total 178.13 last 130.74) - /unlink 25272 (time 1761122933.07 total 188.13 last 130.05) - /unlink 26874 (time 1761122943.07 total 198.13 last 160.17) - /unlink 28744 (time 1761122953.07 total 208.14 last 186.99) - /unlink 30000 (time 1761122960.74 total 215.80 last 163.89) - /unlink 31472 (time 1761122970.74 total 225.80 last 147.14) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 232.54 seconds: 140.17 ops/second Waiting for MDT destroys to complete - open/close 783 (time 1761123005.77 total 10.01 last 78.25) - open/close 1558 (time 1761123015.77 total 20.01 last 77.46) - open/close 2441 (time 1761123025.78 total 30.02 last 88.21) - open/close 3219 (time 1761123035.78 total 40.02 last 77.79) - open/close 4020 (time 1761123045.79 total 50.03 last 80.02) - open/close 4833 (time 1761123055.80 total 60.04 last 81.25) - open/close 5656 (time 1761123065.80 total 70.04 last 82.28) - open/close 6441 (time 1761123075.81 total 80.05 last 78.46) - open/close 7317 (time 1761123085.81 total 90.05 last 87.55) - open/close 8264 (time 1761123095.82 total 100.06 last 94.66) - open/close 9234 (time 1761123105.83 total 110.07 last 96.89) - open/close 10000 (time 1761123115.08 total 119.32 last 82.82) - open/close 10964 (time 1761123125.08 total 129.32 last 96.38) - open/close 11852 (time 1761123135.09 total 139.33 last 88.75) - open/close 12770 (time 1761123145.09 total 149.33 last 91.72) - open/close 13682 (time 1761123155.10 total 159.33 last 91.19) - open/close 14682 (time 1761123165.10 total 169.34 last 99.91) - open/close 15547 (time 1761123175.11 total 179.35 last 86.44) - open/close 16440 (time 1761123185.12 total 189.36 last 89.25) - open/close 17343 (time 1761123195.12 total 199.36 last 90.25) - open/close 18257 (time 1761123205.13 total 209.37 last 91.32) - open/close 19189 (time 1761123215.13 total 219.37 last 93.19) - open/close 20000 (time 1761123223.63 total 227.87 last 95.42) - open/close 20824 (time 1761123233.65 total 237.89 last 82.27) - open/close 21639 (time 1761123243.66 total 247.90 last 81.39) - open/close 22559 (time 1761123253.66 total 257.90 last 91.99) - open/close 23522 (time 1761123263.67 total 267.91 last 96.26) - open/close 24496 (time 1761123273.68 total 277.92 last 97.21) - open/close 25338 (time 1761123283.69 total 287.93 last 84.17) - open/close 26264 (time 1761123293.70 total 297.94 last 92.53) - open/close 27247 (time 1761123303.70 total 307.94 last 98.26) - open/close 28209 (time 1761123313.71 total 317.95 last 96.11) - open/close 29480 (time 1761123323.71 total 327.95 last 127.10) - open/close 30000 (time 1761123329.01 total 333.25 last 98.05) - open/close 31005 (time 1761123339.02 total 343.26 last 100.44) - open/close 32105 (time 1761123349.02 total 353.26 last 109.99) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 357.78 seconds: 91.10 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 2031 (time 1761123366.41 total 10.00 last 203.09) - /unlink 4101 (time 1761123376.41 total 20.00 last 206.99) - /unlink 6114 (time 1761123386.42 total 30.00 last 201.24) - /unlink 8131 (time 1761123396.42 total 40.01 last 201.66) - /unlink 10000 (time 1761123404.92 total 48.51 last 219.71) - /unlink 11934 (time 1761123414.93 total 58.52 last 193.33) - /unlink 13557 (time 1761123424.93 total 68.52 last 162.26) - /unlink 15315 (time 1761123434.93 total 78.52 last 175.78) - /unlink 17364 (time 1761123444.93 total 88.52 last 204.89) - /unlink 19567 (time 1761123454.93 total 98.52 last 220.29) - /unlink 20000 (time 1761123456.95 total 100.54 last 214.37) - /unlink 22227 (time 1761123466.95 total 110.54 last 222.68) - /unlink 24354 (time 1761123476.95 total 120.54 last 212.70) - /unlink 26044 (time 1761123486.96 total 130.54 last 168.95) - /unlink 28260 (time 1761123496.96 total 140.55 last 221.53) - /unlink 29930 (time 1761123506.96 total 150.55 last 166.98) - /unlink 31904 (time 1761123516.96 total 160.55 last 197.35) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 164.13 seconds: 198.59 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg102-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg102-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg102-server stop mds service on oleg102-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg102-server stop mds service on oleg102-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg102-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg102-server' oleg102-server: oleg102-server.virtnet: executing unload_modules_local oleg102-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg102-server: LNET ready to unload oleg102-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg102-client: oleg102-client: ssh exited with exit code 2 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 2 pdsh@oleg102-client: oleg102-client: ssh exited with exit code 2 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 2