== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:33:21 (1771968801) Stopping clients: oleg203-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg203-client.virtnet /mnt/lustre2 (opts:-f) oleg203-server: oleg203-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing load_modules_local oleg203-server: Loading modules from /home/green/git/lustre-release/lustre oleg203-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg203-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg203-server: detected 4 online CPUs by sysfs oleg203-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg203-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg203-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg203-client.virtnet: -o user_xattr,flock oleg203-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16166 358 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1114 (time 1771968883.42 total 10.00 last 111.35) - open/close 2092 (time 1771968893.42 total 20.01 last 97.78) - open/close 3090 (time 1771968903.43 total 30.01 last 99.73) - open/close 4146 (time 1771968913.44 total 40.03 last 105.44) - open/close 5339 (time 1771968923.45 total 50.03 last 119.24) - open/close 6490 (time 1771968933.45 total 60.04 last 115.09) - open/close 7738 (time 1771968943.45 total 70.04 last 124.77) - open/close 8945 (time 1771968953.46 total 80.04 last 120.62) - open/close 10000 (time 1771968962.45 total 89.04 last 117.32) - open/close 11110 (time 1771968972.45 total 99.04 last 111.00) - open/close 12203 (time 1771968982.46 total 109.05 last 109.19) - open/close 13349 (time 1771968992.46 total 119.05 last 114.58) - open/close 14617 (time 1771969002.47 total 129.05 last 126.71) - open/close 15693 (time 1771969012.47 total 139.06 last 107.57) - open/close 16735 (time 1771969022.47 total 149.06 last 104.18) - open/close 17806 (time 1771969032.47 total 159.06 last 107.09) - open/close 18899 (time 1771969042.47 total 169.06 last 109.29) - open/close 19875 (time 1771969052.48 total 179.06 last 97.58) - open/close 20854 (time 1771969062.48 total 189.07 last 97.85) - open/close 21916 (time 1771969072.49 total 199.07 last 106.14) - open/close 23056 (time 1771969082.50 total 209.09 last 113.83) - open/close 24098 (time 1771969092.51 total 219.10 last 104.13) - open/close 25221 (time 1771969102.51 total 229.10 last 112.24) - open/close 26423 (time 1771969112.52 total 239.10 last 120.17) - open/close 27628 (time 1771969122.52 total 249.10 last 120.49) - open/close 28712 (time 1771969132.52 total 259.11 last 108.33) - open/close 30000 (time 1771969142.26 total 268.85 last 132.32) - open/close 31284 (time 1771969152.26 total 278.85 last 128.38) - open/close 32746 (time 1771969162.26 total 288.85 last 146.16) - open/close 34126 (time 1771969172.27 total 298.86 last 137.93) - open/close 35396 (time 1771969182.27 total 308.86 last 126.98) - open/close 36646 (time 1771969192.27 total 318.86 last 124.98) - open/close 37823 (time 1771969202.28 total 328.87 last 117.60) - open/close 38994 (time 1771969212.28 total 338.87 last 117.06) - open/close 40000 (time 1771969220.09 total 346.68 last 128.88) - open/close 41230 (time 1771969230.09 total 356.68 last 122.93) - open/close 42446 (time 1771969240.09 total 366.68 last 121.59) - open/close 43684 (time 1771969250.11 total 376.70 last 123.62) - open/close 44983 (time 1771969260.11 total 386.70 last 129.88) - open/close 46396 (time 1771969270.12 total 396.70 last 141.21) - open/close 47704 (time 1771969280.12 total 406.71 last 130.74) - open/close 48892 (time 1771969290.13 total 416.72 last 118.69) - open/close 50000 (time 1771969299.85 total 426.44 last 113.97) - open/close 51114 (time 1771969309.86 total 436.44 last 111.38) - open/close 52401 (time 1771969319.86 total 446.45 last 128.63) - open/close 53620 (time 1771969329.87 total 456.46 last 121.73) - open/close 54768 (time 1771969339.88 total 466.47 last 114.76) - open/close 56095 (time 1771969349.88 total 476.47 last 132.68) - open/close 57315 (time 1771969359.89 total 486.47 last 121.90) - open/close 58709 (time 1771969369.90 total 496.48 last 139.30) - open/close 59987 (time 1771969379.90 total 506.48 last 127.77) - open/close 61225 (time 1771969389.91 total 516.49 last 123.70) - open/close 62698 (time 1771969399.91 total 526.49 last 147.29) - open/close 63987 (time 1771969409.91 total 536.49 last 128.88) total: 64768 open/close in 542.15 seconds: 119.47 ops/second - /unlink 3498 (time 1771969427.40 total 10.00 last 349.77) - /unlink 6235 (time 1771969437.40 total 20.00 last 273.67) - /unlink 8603 (time 1771969447.40 total 30.00 last 236.79) - /unlink 10000 (time 1771969452.58 total 35.18 last 269.92) - /unlink 12827 (time 1771969462.58 total 45.18 last 282.69) - /unlink 15764 (time 1771969472.58 total 55.18 last 293.64) - /unlink 18228 (time 1771969482.58 total 65.18 last 246.39) - /unlink 20000 (time 1771969490.36 total 72.96 last 227.81) - /unlink 22605 (time 1771969500.36 total 82.96 last 260.49) - /unlink 25343 (time 1771969510.37 total 92.96 last 273.72) - /unlink 28316 (time 1771969520.37 total 102.96 last 297.27) - /unlink 30000 (time 1771969526.49 total 109.09 last 275.03) - /unlink 32538 (time 1771969536.49 total 119.09 last 253.76) - /unlink 34921 (time 1771969546.49 total 129.09 last 238.28) - /unlink 37589 (time 1771969556.49 total 139.09 last 266.76) - /unlink 40000 (time 1771969565.59 total 148.19 last 264.88) - /unlink 42573 (time 1771969575.60 total 158.19 last 257.23) - /unlink 45223 (time 1771969585.60 total 168.20 last 264.97) - /unlink 47971 (time 1771969595.60 total 178.20 last 274.79) - /unlink 50000 (time 1771969601.59 total 184.19 last 338.44) - /unlink 53307 (time 1771969611.60 total 194.20 last 330.51) - /unlink 56176 (time 1771969621.60 total 204.20 last 286.83) - /unlink 58606 (time 1771969631.61 total 214.20 last 242.88) - /unlink 60000 (time 1771969637.19 total 219.79 last 249.59) - /unlink 63248 (time 1771969647.20 total 229.79 last 324.68) total: 64768 /unlink in 235.40 seconds: 275.14 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1798 (time 1771969684.99 total 10.01 last 179.69) - open/close 3732 (time 1771969694.99 total 20.01 last 193.33) - open/close 5605 (time 1771969705.00 total 30.01 last 187.26) - open/close 7643 (time 1771969715.00 total 40.01 last 203.73) - open/close 9641 (time 1771969725.00 total 50.02 last 199.76) - open/close 11303 (time 1771969735.01 total 60.02 last 166.10) - open/close 13036 (time 1771969745.01 total 70.02 last 173.27) - open/close 14632 (time 1771969755.01 total 80.03 last 159.59) - open/close 16787 (time 1771969765.01 total 90.03 last 215.39) - open/close 18716 (time 1771969775.02 total 100.03 last 192.85) - open/close 20000 (time 1771969780.03 total 105.05 last 255.93) - open/close 22000 (time 1771969790.04 total 115.05 last 199.96) - open/close 23547 (time 1771969800.04 total 125.06 last 154.64) - open/close 25395 (time 1771969810.04 total 135.06 last 184.72) - open/close 27255 (time 1771969820.04 total 145.06 last 185.98) - open/close 28954 (time 1771969830.05 total 155.06 last 169.85) - open/close 30000 (time 1771969835.63 total 160.64 last 187.40) - open/close 31646 (time 1771969845.63 total 170.65 last 164.58) - open/close 33241 (time 1771969855.64 total 180.65 last 159.39) - open/close 34961 (time 1771969865.65 total 190.66 last 171.87) - open/close 36497 (time 1771969875.65 total 200.67 last 153.50) - open/close 38147 (time 1771969885.66 total 210.67 last 164.94) - open/close 39871 (time 1771969895.66 total 220.67 last 172.37) - open/close 41571 (time 1771969905.66 total 230.68 last 169.93) - open/close 43516 (time 1771969915.66 total 240.68 last 194.49) - open/close 45464 (time 1771969925.67 total 250.68 last 194.70) - open/close 47206 (time 1771969935.67 total 260.69 last 174.07) - open/close 48871 (time 1771969945.68 total 270.69 last 166.48) - open/close 50000 (time 1771969952.37 total 277.38 last 168.69) - open/close 51887 (time 1771969962.37 total 287.39 last 188.68) - open/close 53812 (time 1771969972.37 total 297.39 last 192.46) - open/close 55861 (time 1771969982.37 total 307.39 last 204.89) - open/close 57850 (time 1771969992.38 total 317.39 last 198.85) - open/close 59795 (time 1771970002.38 total 327.39 last 194.45) - open/close 61910 (time 1771970012.38 total 337.40 last 211.40) - open/close 64092 (time 1771970022.39 total 347.41 last 218.02) total: 64768 open/close in 350.51 seconds: 184.78 ops/second - /unlink 3678 (time 1771970037.11 total 10.00 last 367.76) - /unlink 6805 (time 1771970047.11 total 20.00 last 312.67) - /unlink 9972 (time 1771970057.11 total 30.00 last 316.70) - /unlink 13648 (time 1771970067.11 total 40.01 last 367.48) - /unlink 17836 (time 1771970077.11 total 50.01 last 418.76) - /unlink 20000 (time 1771970082.28 total 55.17 last 418.70) - /unlink 24812 (time 1771970092.28 total 65.18 last 481.18) - /unlink 28951 (time 1771970102.28 total 75.18 last 413.88) - /unlink 30000 (time 1771970105.88 total 78.77 last 291.49) - /unlink 33083 (time 1771970115.88 total 88.77 last 308.29) - /unlink 36668 (time 1771970125.88 total 98.78 last 358.46) - /unlink 40000 (time 1771970134.30 total 107.19 last 395.85) - /unlink 44085 (time 1771970144.30 total 117.19 last 408.48) - /unlink 48603 (time 1771970154.30 total 127.20 last 451.73) - /unlink 50000 (time 1771970157.10 total 129.99 last 499.20) - /unlink 53878 (time 1771970167.10 total 140.00 last 387.75) - /unlink 57288 (time 1771970177.10 total 150.00 last 340.92) - /unlink 60000 (time 1771970184.53 total 157.43 last 365.08) - /unlink 63566 (time 1771970194.53 total 167.43 last 356.48) total: 64768 /unlink in 171.00 seconds: 378.77 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2369 (time 1771970227.84 total 10.01 last 236.77) - open/close 4190 (time 1771970237.84 total 20.01 last 182.05) - open/close 5959 (time 1771970247.85 total 30.01 last 176.85) - open/close 7852 (time 1771970257.85 total 40.01 last 189.30) - open/close 9660 (time 1771970267.85 total 50.01 last 180.76) - open/close 11622 (time 1771970277.85 total 60.01 last 196.17) - open/close 13411 (time 1771970287.85 total 70.02 last 178.83) - open/close 15337 (time 1771970297.85 total 80.02 last 192.56) - open/close 17266 (time 1771970307.86 total 90.02 last 192.89) - open/close 20000 (time 1771970317.79 total 99.96 last 275.14) - open/close 22947 (time 1771970327.79 total 109.96 last 294.66) - open/close 27420 (time 1771970337.80 total 119.96 last 447.20) - open/close 30000 (time 1771970344.60 total 126.77 last 379.05) - open/close 33744 (time 1771970354.61 total 136.77 last 374.22) - open/close 36689 (time 1771970364.61 total 146.77 last 294.46) - open/close 39510 (time 1771970374.61 total 156.78 last 282.04) - open/close 41985 (time 1771970384.61 total 166.78 last 247.38) - open/close 43973 (time 1771970394.62 total 176.78 last 198.78) - open/close 46280 (time 1771970404.62 total 186.78 last 230.66) - open/close 48731 (time 1771970414.70 total 196.86 last 243.13) - open/close 50000 (time 1771970419.80 total 201.97 last 248.68) - open/close 52259 (time 1771970429.80 total 211.97 last 225.90) - open/close 54997 (time 1771970439.80 total 221.97 last 273.77) - open/close 57362 (time 1771970449.81 total 231.97 last 236.40) - open/close 60000 (time 1771970459.48 total 241.65 last 272.70) - open/close 62148 (time 1771970469.48 total 251.65 last 214.75) - open/close 64702 (time 1771970479.48 total 261.65 last 255.36) total: 64768 open/close in 261.89 seconds: 247.31 ops/second - /unlink 4654 (time 1771970491.33 total 10.00 last 465.37) - /unlink 9696 (time 1771970501.33 total 20.00 last 504.16) - /unlink 15433 (time 1771970511.33 total 30.00 last 573.61) - /unlink 20000 (time 1771970518.55 total 37.22 last 632.52) - /unlink 25573 (time 1771970528.55 total 47.22 last 557.29) - /unlink 30000 (time 1771970536.59 total 55.26 last 550.79) - /unlink 36517 (time 1771970546.59 total 65.26 last 651.64) - /unlink 40000 (time 1771970553.55 total 72.22 last 500.43) - /unlink 45858 (time 1771970563.55 total 82.22 last 585.72) - /unlink 50000 (time 1771970571.91 total 90.58 last 495.59) - /unlink 55271 (time 1771970581.91 total 100.58 last 527.09) - /unlink 60000 (time 1771970589.86 total 108.53 last 595.05) - /unlink 64707 (time 1771970599.86 total 118.53 last 470.64) total: 64768 /unlink in 118.70 seconds: 545.65 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3331 (time 1771970631.22 total 10.00 last 333.04) - open/close 7094 (time 1771970641.23 total 20.00 last 376.23) - open/close 10000 (time 1771970648.10 total 26.88 last 422.53) - open/close 13576 (time 1771970658.10 total 36.88 last 357.59) - open/close 16954 (time 1771970668.11 total 46.88 last 337.76) - open/close 20000 (time 1771970676.28 total 55.06 last 372.49) - open/close 23274 (time 1771970686.29 total 65.06 last 327.25) - open/close 26578 (time 1771970696.29 total 75.06 last 330.40) - open/close 30000 (time 1771970705.93 total 84.71 last 354.91) - open/close 33671 (time 1771970715.93 total 94.71 last 367.08) - open/close 37707 (time 1771970725.93 total 104.71 last 403.55) - open/close 40000 (time 1771970732.76 total 111.54 last 335.75) - open/close 43461 (time 1771970742.76 total 121.54 last 346.04) - open/close 46973 (time 1771970752.76 total 131.54 last 351.16) - open/close 50000 (time 1771970761.55 total 140.33 last 344.36) - open/close 53221 (time 1771970771.56 total 150.33 last 322.00) - open/close 56868 (time 1771970781.56 total 160.34 last 364.64) - open/close 60000 (time 1771970789.69 total 168.47 last 385.21) - open/close 63610 (time 1771970799.69 total 178.47 last 360.98) total: 64768 open/close in 181.32 seconds: 357.20 ops/second - /unlink 7212 (time 1771970813.94 total 10.00 last 721.19) - /unlink 10000 (time 1771970817.88 total 13.94 last 707.40) - /unlink 16660 (time 1771970827.88 total 23.94 last 665.97) - /unlink 20000 (time 1771970832.83 total 28.88 last 675.79) - /unlink 26657 (time 1771970842.83 total 38.88 last 665.66) - /unlink 30000 (time 1771970847.20 total 43.25 last 764.99) - /unlink 36953 (time 1771970857.20 total 53.26 last 695.24) - /unlink 40000 (time 1771970862.03 total 58.09 last 630.88) - /unlink 45172 (time 1771970872.03 total 68.09 last 517.19) - /unlink 50000 (time 1771970881.25 total 77.30 last 523.70) - /unlink 56551 (time 1771970891.25 total 87.31 last 655.05) - /unlink 60000 (time 1771970895.55 total 91.61 last 802.17) total: 64768 /unlink in 97.76 seconds: 662.50 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3590 (time 1771970933.46 total 10.00 last 359.00) - open/close 7730 (time 1771970943.46 total 20.00 last 413.94) - open/close 10000 (time 1771970948.61 total 25.15 last 440.73) - open/close 14434 (time 1771970958.62 total 35.15 last 443.32) - open/close 18524 (time 1771970968.62 total 45.15 last 408.97) - open/close 20000 (time 1771970971.87 total 48.41 last 454.06) - open/close 24722 (time 1771970981.87 total 58.41 last 472.18) - open/close 29312 (time 1771970991.87 total 68.41 last 458.95) - open/close 33542 (time 1771971001.87 total 78.41 last 422.98) - open/close 38074 (time 1771971011.87 total 88.41 last 453.09) - open/close 40000 (time 1771971016.94 total 93.48 last 380.15) - open/close 44880 (time 1771971026.94 total 103.48 last 487.91) - open/close 49958 (time 1771971036.94 total 113.48 last 507.72) - open/close 55161 (time 1771971046.94 total 123.48 last 520.23) - open/close 60000 (time 1771971056.00 total 132.54 last 534.32) total: 64768 open/close in 141.96 seconds: 456.25 ops/second - /unlink 7635 (time 1771971076.72 total 10.00 last 763.42) - /unlink 10000 (time 1771971080.33 total 13.61 last 654.86) - /unlink 17260 (time 1771971090.33 total 23.61 last 725.94) - /unlink 20000 (time 1771971093.28 total 26.56 last 929.65) - /unlink 27952 (time 1771971103.28 total 36.56 last 795.19) - /unlink 30000 (time 1771971105.96 total 39.24 last 763.61) - /unlink 38369 (time 1771971115.96 total 49.24 last 836.88) - /unlink 46692 (time 1771971125.97 total 59.25 last 832.00) - /unlink 50000 (time 1771971129.86 total 63.14 last 849.38) - /unlink 58589 (time 1771971139.86 total 73.14 last 858.82) - /unlink 60000 (time 1771971141.97 total 75.24 last 671.09) total: 64768 /unlink in 80.96 seconds: 800.01 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg203-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server stop mds service on oleg203-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing unload_modules_local oleg203-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg203-server: LNET ready to unload oleg203-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2