== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 12:57:09 (1772906229) Stopping clients: oleg142-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg142-client.virtnet /mnt/lustre2 (opts:-f) oleg142-server: oleg142-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg142-server' oleg142-server: oleg142-server.virtnet: executing load_modules_local oleg142-server: Loading modules from /home/green/git/lustre-release/lustre oleg142-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg142-server: detected 4 online CPUs by sysfs oleg142-server: MODOPTS_LIBCFS= oleg142-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg142-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg142-server: oleg142-server.virtnet: executing set_default_debug -1 all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg142-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg142-server: oleg142-server.virtnet: executing set_default_debug -1 all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg142-client.virtnet: -o user_xattr,flock 192.168.201.142@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1348 (time 1772906289.91 total 10.00 last 134.77) - open/close 2676 (time 1772906299.92 total 20.01 last 132.72) - open/close 3809 (time 1772906309.92 total 30.01 last 113.28) - open/close 4909 (time 1772906319.93 total 40.02 last 109.92) - open/close 5931 (time 1772906329.94 total 50.03 last 102.12) - open/close 7148 (time 1772906339.94 total 60.03 last 121.67) - open/close 8201 (time 1772906349.95 total 70.03 last 105.22) - open/close 9302 (time 1772906359.95 total 80.04 last 110.05) - open/close 10000 (time 1772906367.36 total 87.45 last 94.20) - open/close 11043 (time 1772906377.37 total 97.46 last 104.24) - open/close 12235 (time 1772906387.38 total 107.47 last 119.04) - open/close 13193 (time 1772906397.39 total 117.48 last 95.71) - open/close 14808 (time 1772906407.39 total 127.48 last 161.47) - open/close 16984 (time 1772906417.40 total 137.49 last 217.47) - open/close 18568 (time 1772906427.40 total 147.49 last 158.32) - open/close 20000 (time 1772906437.05 total 157.14 last 148.38) - open/close 21770 (time 1772906447.06 total 167.15 last 176.84) - open/close 23566 (time 1772906457.06 total 177.15 last 179.58) - open/close 25677 (time 1772906467.07 total 187.15 last 211.06) - open/close 27442 (time 1772906477.07 total 197.16 last 176.36) - open/close 28931 (time 1772906487.07 total 207.16 last 148.90) - open/close 30000 (time 1772906494.79 total 214.88 last 138.50) - open/close 31608 (time 1772906504.80 total 224.89 last 160.71) - open/close 32775 (time 1772906514.80 total 234.89 last 116.69) - open/close 34190 (time 1772906524.80 total 244.89 last 141.49) - open/close 35691 (time 1772906534.80 total 254.89 last 150.07) - open/close 37139 (time 1772906544.80 total 264.89 last 144.79) - open/close 38798 (time 1772906554.80 total 274.89 last 165.88) - open/close 40000 (time 1772906563.01 total 283.09 last 146.54) - open/close 41236 (time 1772906573.01 total 293.10 last 123.54) - open/close 42519 (time 1772906583.01 total 303.10 last 128.27) - open/close 43771 (time 1772906593.02 total 313.11 last 125.10) - open/close 44698 (time 1772906603.02 total 323.11 last 92.67) - open/close 45823 (time 1772906613.03 total 333.11 last 112.48) - open/close 47035 (time 1772906623.03 total 343.11 last 121.20) - open/close 48231 (time 1772906633.03 total 353.12 last 119.52) - open/close 49379 (time 1772906643.03 total 363.12 last 114.79) - open/close 50000 (time 1772906648.74 total 368.83 last 108.81) - open/close 51093 (time 1772906658.74 total 378.83 last 109.28) - open/close 51980 (time 1772906668.74 total 388.83 last 88.69) - open/close 53170 (time 1772906678.75 total 398.84 last 118.89) - open/close 54538 (time 1772906688.76 total 408.84 last 136.77) - open/close 55858 (time 1772906698.76 total 418.85 last 131.93) - open/close 57121 (time 1772906708.76 total 428.85 last 126.25) - open/close 58420 (time 1772906718.77 total 438.86 last 129.83) - open/close 59618 (time 1772906728.78 total 448.87 last 119.69) - open/close 60000 (time 1772906731.90 total 451.99 last 122.51) - open/close 61221 (time 1772906741.90 total 461.99 last 122.03) - open/close 62472 (time 1772906751.91 total 472.00 last 124.95) - open/close 63829 (time 1772906761.92 total 482.01 last 135.67) total: 64768 open/close in 489.64 seconds: 132.28 ops/second - /unlink 2724 (time 1772906781.77 total 10.00 last 272.38) - /unlink 5459 (time 1772906791.77 total 20.00 last 273.50) - /unlink 7578 (time 1772906801.77 total 30.01 last 211.81) - /unlink 10000 (time 1772906810.06 total 38.29 last 292.30) - /unlink 13251 (time 1772906820.06 total 48.30 last 324.97) - /unlink 15960 (time 1772906830.06 total 58.30 last 270.82) - /unlink 18888 (time 1772906840.06 total 68.30 last 292.78) - /unlink 20000 (time 1772906843.54 total 71.78 last 319.65) - /unlink 22104 (time 1772906853.55 total 81.78 last 210.37) - /unlink 24783 (time 1772906863.55 total 91.78 last 267.83) - /unlink 27953 (time 1772906873.55 total 101.78 last 317.00) - /unlink 30000 (time 1772906879.66 total 107.89 last 335.16) - /unlink 33394 (time 1772906889.66 total 117.89 last 339.35) - /unlink 36614 (time 1772906899.66 total 127.89 last 321.91) - /unlink 39728 (time 1772906909.66 total 137.90 last 311.27) - /unlink 43268 (time 1772906919.66 total 147.90 last 353.96) - /unlink 46188 (time 1772906929.67 total 157.90 last 291.88) - /unlink 49361 (time 1772906939.67 total 167.91 last 317.19) - /unlink 50000 (time 1772906941.86 total 170.10 last 291.74) - /unlink 52604 (time 1772906951.87 total 180.10 last 260.30) - /unlink 55845 (time 1772906961.87 total 190.10 last 324.03) - /unlink 59057 (time 1772906971.87 total 200.10 last 321.19) - /unlink 60000 (time 1772906974.42 total 202.66 last 368.84) - /unlink 63293 (time 1772906984.43 total 212.66 last 329.24) total: 64768 /unlink in 216.51 seconds: 299.14 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2164 (time 1772907020.36 total 10.00 last 216.38) - open/close 4151 (time 1772907030.36 total 20.01 last 198.62) - open/close 6294 (time 1772907040.36 total 30.01 last 214.27) - open/close 7675 (time 1772907050.36 total 40.01 last 138.08) - open/close 9071 (time 1772907060.37 total 50.01 last 139.52) - open/close 10000 (time 1772907065.57 total 55.22 last 178.50) - open/close 11583 (time 1772907075.58 total 65.22 last 158.26) - open/close 12952 (time 1772907085.58 total 75.23 last 136.80) - open/close 14446 (time 1772907095.59 total 85.23 last 149.33) - open/close 15828 (time 1772907105.59 total 95.24 last 138.14) - open/close 17196 (time 1772907115.60 total 105.24 last 136.76) - open/close 19199 (time 1772907125.61 total 115.25 last 200.11) - open/close 20000 (time 1772907129.54 total 119.18 last 203.63) - open/close 21512 (time 1772907139.55 total 129.19 last 151.10) - open/close 22926 (time 1772907149.55 total 139.20 last 141.33) - open/close 24586 (time 1772907159.55 total 149.20 last 165.93) - open/close 26105 (time 1772907169.56 total 159.21 last 151.77) - open/close 27576 (time 1772907179.57 total 169.21 last 147.07) - open/close 29106 (time 1772907189.57 total 179.21 last 152.96) - open/close 30000 (time 1772907194.78 total 184.42 last 171.64) - open/close 31735 (time 1772907204.78 total 194.42 last 173.45) - open/close 33349 (time 1772907214.78 total 204.43 last 161.39) - open/close 35366 (time 1772907224.78 total 214.43 last 201.69) - open/close 37551 (time 1772907234.78 total 224.43 last 218.46) - open/close 39600 (time 1772907244.78 total 234.43 last 204.89) - open/close 40000 (time 1772907246.98 total 236.62 last 182.31) - open/close 42258 (time 1772907256.98 total 246.62 last 225.78) - open/close 44699 (time 1772907266.98 total 256.62 last 244.06) - open/close 46821 (time 1772907276.98 total 266.63 last 212.10) - open/close 49073 (time 1772907286.98 total 276.63 last 225.19) - open/close 50000 (time 1772907290.97 total 280.61 last 232.70) - open/close 51974 (time 1772907300.97 total 290.62 last 197.34) - open/close 54579 (time 1772907310.97 total 300.62 last 260.41) - open/close 57569 (time 1772907320.98 total 310.62 last 298.86) - open/close 60000 (time 1772907329.44 total 319.08 last 287.41) - open/close 63383 (time 1772907339.44 total 329.09 last 338.18) total: 64768 open/close in 332.84 seconds: 194.59 ops/second - /unlink 5468 (time 1772907354.86 total 10.00 last 546.78) - /unlink 10000 (time 1772907363.18 total 18.31 last 545.14) - /unlink 15026 (time 1772907373.18 total 28.32 last 502.53) - /unlink 19181 (time 1772907383.18 total 38.32 last 415.48) - /unlink 23452 (time 1772907393.18 total 48.32 last 426.98) - /unlink 27825 (time 1772907403.19 total 58.32 last 437.03) - /unlink 30000 (time 1772907407.17 total 62.31 last 545.84) - /unlink 34875 (time 1772907417.17 total 72.31 last 487.50) - /unlink 39498 (time 1772907427.17 total 82.31 last 462.24) - /unlink 43737 (time 1772907437.17 total 92.31 last 423.86) - /unlink 48788 (time 1772907447.17 total 102.31 last 505.10) - /unlink 50000 (time 1772907449.85 total 104.99 last 452.57) - /unlink 54575 (time 1772907459.85 total 114.99 last 457.47) - /unlink 59310 (time 1772907469.85 total 124.99 last 473.45) - /unlink 64274 (time 1772907479.87 total 135.01 last 495.46) total: 64768 /unlink in 136.36 seconds: 474.99 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2934 (time 1772907507.57 total 10.00 last 293.40) - open/close 6065 (time 1772907517.57 total 20.00 last 313.05) - open/close 9521 (time 1772907527.57 total 30.01 last 345.48) - open/close 13410 (time 1772907537.57 total 40.01 last 388.88) - open/close 16389 (time 1772907547.57 total 50.01 last 297.87) - open/close 19743 (time 1772907557.57 total 60.01 last 335.33) - open/close 22868 (time 1772907567.58 total 70.01 last 312.42) - open/close 25563 (time 1772907577.58 total 80.01 last 269.49) - open/close 28124 (time 1772907587.58 total 90.01 last 256.08) - open/close 30000 (time 1772907596.87 total 99.31 last 201.83) - open/close 33135 (time 1772907606.88 total 109.31 last 313.43) - open/close 35801 (time 1772907616.88 total 119.31 last 266.60) - open/close 38332 (time 1772907626.88 total 129.31 last 253.08) - open/close 40000 (time 1772907634.69 total 137.13 last 213.35) - open/close 42439 (time 1772907644.70 total 147.13 last 243.84) - open/close 44875 (time 1772907654.70 total 157.13 last 243.58) - open/close 46937 (time 1772907664.70 total 167.13 last 206.19) - open/close 49633 (time 1772907674.70 total 177.13 last 269.55) - open/close 51956 (time 1772907684.70 total 187.13 last 232.29) - open/close 54460 (time 1772907694.70 total 197.14 last 250.34) - open/close 56912 (time 1772907704.70 total 207.14 last 245.20) - open/close 59699 (time 1772907714.71 total 217.14 last 278.63) - open/close 61850 (time 1772907724.71 total 227.15 last 214.98) - open/close 64365 (time 1772907734.71 total 237.15 last 251.50) total: 64768 open/close in 238.84 seconds: 271.18 ops/second - /unlink 5408 (time 1772907748.29 total 10.00 last 540.79) - /unlink 9895 (time 1772907758.29 total 20.00 last 448.67) - /unlink 14163 (time 1772907768.29 total 30.00 last 426.79) - /unlink 19897 (time 1772907778.29 total 40.00 last 573.26) - /unlink 25221 (time 1772907788.29 total 50.00 last 532.35) - /unlink 29877 (time 1772907798.29 total 60.00 last 465.58) - /unlink 35256 (time 1772907808.29 total 70.01 last 537.84) - /unlink 40000 (time 1772907817.14 total 78.85 last 536.37) - /unlink 44674 (time 1772907827.14 total 88.85 last 467.40) - /unlink 48560 (time 1772907837.14 total 98.85 last 388.55) - /unlink 50000 (time 1772907839.98 total 101.69 last 507.91) - /unlink 55244 (time 1772907849.98 total 111.69 last 524.33) - /unlink 59995 (time 1772907859.98 total 121.69 last 475.04) total: 64768 /unlink in 130.41 seconds: 496.66 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3276 (time 1772907899.73 total 10.00 last 327.54) - open/close 6058 (time 1772907909.73 total 20.00 last 278.17) - open/close 8983 (time 1772907919.73 total 30.00 last 292.48) - open/close 10000 (time 1772907922.97 total 33.23 last 314.77) - open/close 12788 (time 1772907932.97 total 43.24 last 278.77) - open/close 16628 (time 1772907942.97 total 53.24 last 383.94) - open/close 20000 (time 1772907951.41 total 61.68 last 399.51) - open/close 23612 (time 1772907961.41 total 71.68 last 361.18) - open/close 26908 (time 1772907971.41 total 81.68 last 329.58) - open/close 29890 (time 1772907981.41 total 91.68 last 298.09) - open/close 33661 (time 1772907991.42 total 101.68 last 377.02) - open/close 37499 (time 1772908001.42 total 111.69 last 383.75) - open/close 40000 (time 1772908007.35 total 117.62 last 421.27) - open/close 44412 (time 1772908017.35 total 127.62 last 441.15) - open/close 47750 (time 1772908027.35 total 137.62 last 333.79) - open/close 50000 (time 1772908033.44 total 143.71 last 369.45) - open/close 52737 (time 1772908043.52 total 153.79 last 271.73) - open/close 55432 (time 1772908053.52 total 163.79 last 269.45) - open/close 58831 (time 1772908063.52 total 173.79 last 339.81) - open/close 60000 (time 1772908066.79 total 177.06 last 357.19) - open/close 63329 (time 1772908076.79 total 187.06 last 332.89) total: 64768 open/close in 191.17 seconds: 338.80 ops/second - /unlink 5901 (time 1772908092.49 total 10.00 last 590.03) - /unlink 10000 (time 1772908098.70 total 16.20 last 660.80) - /unlink 16745 (time 1772908108.70 total 26.21 last 674.43) - /unlink 20000 (time 1772908113.95 total 31.46 last 619.63) - /unlink 25757 (time 1772908123.95 total 41.46 last 575.66) - /unlink 30000 (time 1772908131.06 total 48.56 last 597.25) - /unlink 36215 (time 1772908141.06 total 58.56 last 621.45) - /unlink 40000 (time 1772908148.59 total 66.10 last 502.31) - /unlink 44239 (time 1772908158.59 total 76.10 last 423.90) - /unlink 49854 (time 1772908168.59 total 86.10 last 561.50) - /unlink 56323 (time 1772908178.59 total 96.10 last 646.85) - /unlink 60000 (time 1772908184.56 total 102.06 last 616.68) total: 64768 /unlink in 111.87 seconds: 578.96 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2412 (time 1772908229.27 total 10.01 last 241.07) - open/close 5147 (time 1772908239.27 total 20.01 last 273.48) - open/close 7269 (time 1772908249.27 total 30.01 last 212.12) - open/close 10000 (time 1772908258.46 total 39.20 last 297.17) - open/close 13252 (time 1772908268.46 total 49.20 last 325.17) - open/close 15251 (time 1772908278.46 total 59.20 last 199.89) - open/close 18158 (time 1772908288.46 total 69.20 last 290.69) - open/close 20000 (time 1772908293.98 total 74.72 last 333.94) - open/close 23000 (time 1772908303.98 total 84.72 last 299.95) - open/close 26386 (time 1772908313.98 total 94.72 last 338.57) - open/close 29024 (time 1772908323.98 total 104.72 last 263.75) - open/close 30000 (time 1772908327.45 total 108.19 last 281.81) - open/close 33224 (time 1772908337.45 total 118.19 last 322.35) - open/close 36873 (time 1772908347.45 total 128.19 last 364.89) - open/close 40000 (time 1772908356.80 total 137.54 last 334.52) - open/close 43867 (time 1772908366.80 total 147.54 last 386.60) - open/close 47554 (time 1772908376.80 total 157.54 last 368.61) - open/close 50000 (time 1772908384.13 total 164.87 last 333.75) - open/close 53738 (time 1772908394.13 total 174.87 last 373.77) - open/close 57288 (time 1772908404.13 total 184.87 last 354.99) - open/close 60000 (time 1772908412.75 total 193.49 last 314.61) - open/close 63485 (time 1772908422.78 total 203.52 last 347.56) total: 64768 open/close in 207.16 seconds: 312.65 ops/second - /unlink 6407 (time 1772908438.16 total 10.00 last 640.68) - /unlink 10000 (time 1772908443.84 total 15.68 last 632.33) - /unlink 16213 (time 1772908453.84 total 25.68 last 621.23) - /unlink 20000 (time 1772908459.34 total 31.18 last 688.39) - /unlink 26222 (time 1772908469.34 total 41.19 last 622.16) - /unlink 30000 (time 1772908475.05 total 46.89 last 662.65) - /unlink 36307 (time 1772908485.05 total 56.89 last 630.67) - /unlink 40000 (time 1772908490.91 total 62.75 last 629.96) - /unlink 45836 (time 1772908500.91 total 72.75 last 583.52) - /unlink 50000 (time 1772908506.90 total 78.74 last 695.02) - /unlink 56429 (time 1772908516.90 total 88.74 last 642.84) - /unlink 60000 (time 1772908521.75 total 93.59 last 736.69) total: 64768 /unlink in 100.23 seconds: 646.22 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg142-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg142-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg142-server stop mds service on oleg142-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg142-server unloading modules via unload_modules_local on: 'oleg142-server' oleg142-server: oleg142-server.virtnet: executing unload_modules_local oleg142-server: modules unloaded. pdsh@oleg142-client: oleg142-client: ssh exited with exit code 2 pdsh@oleg142-client: oleg142-server: ssh exited with exit code 2 pdsh@oleg142-client: oleg142-server: ssh exited with exit code 2 pdsh@oleg142-client: oleg142-client: ssh exited with exit code 2