== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:37:52 (1773668272) Stopping clients: oleg414-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg414-client.virtnet /mnt/lustre2 (opts:-f) oleg414-server: oleg414-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing load_modules_local oleg414-server: Loading modules from /home/green/git/lustre-release/lustre oleg414-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg414-server: detected 4 online CPUs by sysfs oleg414-server: MODOPTS_LIBCFS= oleg414-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg414-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg414-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2332 (time 1773668321.87 total 10.00 last 233.13) - open/close 4597 (time 1773668331.87 total 20.00 last 226.49) - open/close 7211 (time 1773668341.87 total 30.00 last 261.38) - open/close 10000 (time 1773668349.36 total 37.49 last 372.41) - open/close 14128 (time 1773668359.36 total 47.49 last 412.73) - open/close 18082 (time 1773668369.36 total 57.50 last 395.39) - open/close 20000 (time 1773668375.38 total 63.51 last 318.77) - open/close 22985 (time 1773668385.38 total 73.51 last 298.50) - open/close 25630 (time 1773668395.38 total 83.51 last 264.43) - open/close 28142 (time 1773668405.38 total 93.52 last 251.16) - open/close 30000 (time 1773668414.46 total 102.59 last 204.69) - open/close 32620 (time 1773668424.46 total 112.60 last 261.89) - open/close 35133 (time 1773668434.47 total 122.61 last 251.12) - open/close 37431 (time 1773668444.47 total 132.61 last 229.74) - open/close 39834 (time 1773668454.47 total 142.61 last 240.30) - open/close 42060 (time 1773668464.48 total 152.61 last 222.53) - open/close 44472 (time 1773668474.50 total 162.63 last 240.72) - open/close 46852 (time 1773668484.50 total 172.64 last 237.88) - open/close 49237 (time 1773668494.50 total 182.64 last 238.48) - open/close 50000 (time 1773668498.13 total 186.27 last 210.29) - open/close 52237 (time 1773668508.13 total 196.27 last 223.60) - open/close 55068 (time 1773668518.22 total 206.35 last 280.74) - open/close 57826 (time 1773668528.22 total 216.36 last 275.73) - open/close 60000 (time 1773668537.46 total 225.60 last 235.26) - open/close 61334 (time 1773668547.46 total 235.60 last 133.36) - open/close 62066 (time 1773668557.49 total 245.62 last 73.04) - open/close 62817 (time 1773668567.49 total 255.63 last 75.08) - open/close 63597 (time 1773668577.53 total 265.67 last 77.68) - open/close 64371 (time 1773668587.54 total 275.67 last 77.36) total: 64768 open/close in 280.67 seconds: 230.77 ops/second - /unlink 2141 (time 1773668605.33 total 10.00 last 214.06) - /unlink 4256 (time 1773668615.34 total 20.00 last 211.44) - /unlink 6642 (time 1773668625.34 total 30.01 last 238.53) - /unlink 8948 (time 1773668635.34 total 40.01 last 230.58) - /unlink 10000 (time 1773668640.93 total 45.60 last 188.05) - /unlink 11935 (time 1773668650.94 total 55.60 last 193.48) - /unlink 13932 (time 1773668660.94 total 65.60 last 199.69) - /unlink 15998 (time 1773668670.94 total 75.61 last 206.59) - /unlink 18525 (time 1773668680.94 total 85.61 last 252.68) - /unlink 20000 (time 1773668686.93 total 91.60 last 246.18) - /unlink 22159 (time 1773668696.93 total 101.60 last 215.81) - /unlink 24628 (time 1773668706.93 total 111.60 last 246.85) - /unlink 26746 (time 1773668716.94 total 121.61 last 211.73) - /unlink 28846 (time 1773668726.94 total 131.61 last 209.97) - /unlink 30000 (time 1773668730.63 total 135.30 last 312.60) - /unlink 32600 (time 1773668740.63 total 145.30 last 259.93) - /unlink 35227 (time 1773668750.64 total 155.31 last 262.60) - /unlink 37830 (time 1773668760.64 total 165.31 last 260.19) - /unlink 40000 (time 1773668769.63 total 174.30 last 241.40) - /unlink 42353 (time 1773668779.63 total 184.30 last 235.27) - /unlink 44415 (time 1773668789.63 total 194.30 last 206.17) - /unlink 46518 (time 1773668799.64 total 204.30 last 210.27) - /unlink 48655 (time 1773668809.64 total 214.30 last 213.70) - /unlink 50000 (time 1773668815.73 total 220.40 last 220.51) - /unlink 52100 (time 1773668825.74 total 230.41 last 209.90) - /unlink 54341 (time 1773668835.74 total 240.41 last 224.10) - /unlink 56570 (time 1773668845.74 total 250.41 last 222.84) - /unlink 58850 (time 1773668855.74 total 260.41 last 227.97) - /unlink 60000 (time 1773668860.17 total 264.84 last 259.71) - /unlink 62730 (time 1773668870.17 total 274.84 last 272.95) total: 64768 /unlink in 283.79 seconds: 228.22 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1305 (time 1773668909.20 total 10.00 last 130.46) - open/close 2706 (time 1773668919.20 total 20.00 last 140.08) - open/close 3997 (time 1773668929.21 total 30.01 last 129.05) - open/close 5439 (time 1773668939.21 total 40.01 last 144.17) - open/close 6779 (time 1773668949.22 total 50.02 last 133.91) - open/close 8038 (time 1773668959.23 total 60.03 last 125.78) - open/close 9488 (time 1773668969.23 total 70.03 last 144.90) - open/close 10000 (time 1773668973.23 total 74.04 last 127.96) - open/close 11339 (time 1773668983.24 total 84.04 last 133.82) - open/close 12554 (time 1773668993.24 total 94.04 last 121.48) - open/close 14279 (time 1773669003.24 total 104.05 last 172.43) - open/close 15947 (time 1773669013.25 total 114.05 last 166.78) - open/close 17354 (time 1773669023.25 total 124.05 last 140.69) - open/close 18748 (time 1773669033.25 total 134.05 last 139.38) - open/close 20000 (time 1773669041.42 total 142.22 last 153.15) - open/close 21588 (time 1773669051.42 total 152.23 last 158.78) - open/close 23083 (time 1773669061.43 total 162.23 last 149.47) - open/close 24533 (time 1773669071.43 total 172.23 last 144.97) - open/close 26238 (time 1773669081.43 total 182.24 last 170.41) - open/close 27934 (time 1773669091.46 total 192.26 last 169.10) - open/close 29479 (time 1773669101.46 total 202.27 last 154.48) - open/close 30000 (time 1773669104.87 total 205.67 last 152.93) - open/close 31549 (time 1773669114.87 total 215.67 last 154.89) - open/close 33103 (time 1773669124.88 total 225.68 last 155.31) - open/close 34594 (time 1773669134.88 total 235.68 last 149.08) - open/close 36198 (time 1773669144.88 total 245.68 last 160.40) - open/close 37693 (time 1773669154.88 total 255.68 last 149.48) - open/close 39341 (time 1773669164.88 total 265.69 last 164.75) - open/close 40000 (time 1773669168.67 total 269.47 last 173.92) - open/close 41836 (time 1773669178.67 total 279.48 last 183.56) - open/close 42519 (time 1773669188.69 total 289.49 last 68.23) - open/close 43125 (time 1773669198.77 total 299.57 last 60.12) - open/close 43613 (time 1773669208.78 total 309.58 last 48.73) - open/close 44273 (time 1773669218.79 total 319.59 last 65.96) - open/close 44756 (time 1773669228.80 total 329.60 last 48.23) - open/close 45143 (time 1773669238.82 total 339.62 last 38.64) - open/close 45509 (time 1773669248.82 total 349.63 last 36.57) - open/close 45931 (time 1773669258.84 total 359.64 last 42.14) - open/close 46590 (time 1773669268.85 total 369.65 last 65.84) - open/close 47312 (time 1773669278.86 total 379.66 last 72.12) - open/close 48011 (time 1773669288.86 total 389.66 last 69.88) - open/close 48771 (time 1773669298.88 total 399.68 last 75.89) - open/close 49544 (time 1773669308.88 total 409.69 last 77.24) - open/close 50000 (time 1773669314.46 total 415.26 last 81.79) - open/close 50799 (time 1773669324.47 total 425.28 last 79.78) - open/close 51586 (time 1773669334.48 total 435.28 last 78.65) - open/close 52322 (time 1773669344.49 total 445.29 last 73.54) - open/close 53083 (time 1773669354.49 total 455.29 last 76.07) - open/close 53889 (time 1773669364.50 total 465.30 last 80.56) - open/close 54681 (time 1773669374.50 total 475.30 last 79.17) - open/close 55505 (time 1773669384.50 total 485.31 last 82.37) - open/close 56273 (time 1773669394.51 total 495.31 last 76.75) - open/close 57075 (time 1773669404.52 total 505.32 last 80.16) - open/close 58025 (time 1773669414.52 total 515.32 last 94.98) - open/close 58930 (time 1773669424.52 total 525.32 last 90.48) - open/close 59862 (time 1773669434.55 total 535.36 last 92.89) - open/close 60812 (time 1773669444.56 total 545.37 last 94.90) - open/close 61711 (time 1773669454.57 total 555.38 last 89.81) - open/close 62644 (time 1773669464.58 total 565.38 last 93.27) - open/close 63528 (time 1773669474.58 total 575.38 last 88.37) - open/close 64346 (time 1773669484.58 total 585.38 last 81.80) total: 64768 open/close in 591.78 seconds: 109.45 ops/second - /unlink 1369 (time 1773669504.74 total 10.00 last 136.90) - /unlink 2595 (time 1773669514.74 total 20.00 last 122.57) - /unlink 3935 (time 1773669524.75 total 30.01 last 133.96) - /unlink 5327 (time 1773669534.75 total 40.01 last 139.14) - /unlink 6690 (time 1773669544.76 total 50.02 last 136.23) - /unlink 8212 (time 1773669554.78 total 60.03 last 151.91) - /unlink 9672 (time 1773669564.78 total 70.04 last 145.95) - /unlink 10000 (time 1773669566.81 total 72.07 last 161.39) - /unlink 11364 (time 1773669576.82 total 82.08 last 136.28) - /unlink 12661 (time 1773669586.82 total 92.08 last 129.64) - /unlink 14166 (time 1773669596.82 total 102.08 last 150.50) - /unlink 15584 (time 1773669606.82 total 112.08 last 141.79) - /unlink 17353 (time 1773669616.83 total 122.09 last 176.88) - /unlink 19063 (time 1773669626.83 total 132.09 last 171.00) - /unlink 20000 (time 1773669631.75 total 137.00 last 190.46) - /unlink 22099 (time 1773669641.75 total 147.01 last 209.87) - /unlink 24251 (time 1773669651.75 total 157.01 last 215.17) - /unlink 26256 (time 1773669661.75 total 167.01 last 200.48) - /unlink 28377 (time 1773669671.75 total 177.01 last 212.08) - /unlink 30000 (time 1773669676.69 total 181.95 last 328.39) - /unlink 32561 (time 1773669686.69 total 191.95 last 256.08) - /unlink 35167 (time 1773669696.70 total 201.95 last 260.55) - /unlink 37745 (time 1773669706.70 total 211.96 last 257.67) - /unlink 40000 (time 1773669716.52 total 221.78 last 229.64) - /unlink 42624 (time 1773669726.52 total 231.78 last 262.40) - /unlink 45140 (time 1773669736.52 total 241.78 last 251.58) - /unlink 47722 (time 1773669746.52 total 251.78 last 258.11) - /unlink 50000 (time 1773669754.46 total 259.72 last 286.90) - /unlink 53050 (time 1773669764.47 total 269.73 last 304.94) - /unlink 55724 (time 1773669774.47 total 279.73 last 267.29) - /unlink 59022 (time 1773669784.47 total 289.73 last 329.79) - /unlink 60000 (time 1773669787.34 total 292.60 last 341.10) - /unlink 62748 (time 1773669797.34 total 302.60 last 274.74) total: 64768 /unlink in 307.55 seconds: 210.59 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2735 (time 1773669828.69 total 10.00 last 273.47) - open/close 5673 (time 1773669838.70 total 20.00 last 293.73) - open/close 9208 (time 1773669848.70 total 30.01 last 353.43) - open/close 10000 (time 1773669851.35 total 32.65 last 298.93) - open/close 13010 (time 1773669861.35 total 42.66 last 300.92) - open/close 16201 (time 1773669871.35 total 52.66 last 319.04) - open/close 19368 (time 1773669881.35 total 62.66 last 316.63) - open/close 23335 (time 1773669891.35 total 72.66 last 396.69) - open/close 26566 (time 1773669901.36 total 82.67 last 322.94) - open/close 29935 (time 1773669911.36 total 92.67 last 336.87) - open/close 33748 (time 1773669921.36 total 102.67 last 381.21) - open/close 37048 (time 1773669931.36 total 112.67 last 329.94) - open/close 40000 (time 1773669940.53 total 121.84 last 322.10) - open/close 43167 (time 1773669950.53 total 131.84 last 316.61) - open/close 45914 (time 1773669960.53 total 141.84 last 274.69) - open/close 49158 (time 1773669970.53 total 151.84 last 324.34) - open/close 50000 (time 1773669972.83 total 154.14 last 366.74) - open/close 53525 (time 1773669982.83 total 164.14 last 352.44) - open/close 56788 (time 1773669992.83 total 174.14 last 326.26) - open/close 60000 (time 1773670001.61 total 182.92 last 365.88) - open/close 63126 (time 1773670011.61 total 192.92 last 312.54) total: 64768 open/close in 198.71 seconds: 325.94 ops/second - /unlink 4296 (time 1773670028.92 total 10.00 last 429.58) - /unlink 8371 (time 1773670038.92 total 20.00 last 407.35) - /unlink 10000 (time 1773670042.28 total 23.37 last 484.28) - /unlink 15307 (time 1773670052.29 total 33.37 last 530.63) - /unlink 20000 (time 1773670060.68 total 41.76 last 559.42) - /unlink 25490 (time 1773670070.68 total 51.76 last 548.92) - /unlink 30000 (time 1773670079.12 total 60.20 last 534.12) - /unlink 35535 (time 1773670089.12 total 70.20 last 553.45) - /unlink 40000 (time 1773670096.73 total 77.81 last 587.08) - /unlink 45699 (time 1773670106.73 total 87.81 last 569.81) - /unlink 50000 (time 1773670114.93 total 96.02 last 524.10) - /unlink 55454 (time 1773670124.94 total 106.02 last 545.35) - /unlink 60000 (time 1773670133.12 total 114.20 last 555.63) - /unlink 64705 (time 1773670143.12 total 124.20 last 470.47) total: 64768 /unlink in 124.32 seconds: 520.99 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3134 (time 1773670173.76 total 10.00 last 313.32) - open/close 6277 (time 1773670183.76 total 20.00 last 314.27) - open/close 8766 (time 1773670193.76 total 30.01 last 248.86) - open/close 10000 (time 1773670198.83 total 35.08 last 243.28) - open/close 12331 (time 1773670208.84 total 45.08 last 232.95) - open/close 14741 (time 1773670218.84 total 55.09 last 240.92) - open/close 18160 (time 1773670228.84 total 65.09 last 341.88) - open/close 20000 (time 1773670233.81 total 70.05 last 370.53) - open/close 23928 (time 1773670243.81 total 80.06 last 392.74) - open/close 27399 (time 1773670253.81 total 90.06 last 347.05) - open/close 30000 (time 1773670261.26 total 97.51 last 349.09) - open/close 33446 (time 1773670271.26 total 107.51 last 344.54) - open/close 36492 (time 1773670281.27 total 117.52 last 304.41) - open/close 38811 (time 1773670291.27 total 127.52 last 231.87) - open/close 40000 (time 1773670295.71 total 131.95 last 268.00) - open/close 43199 (time 1773670305.71 total 141.95 last 319.89) - open/close 46819 (time 1773670315.71 total 151.96 last 361.94) - open/close 50000 (time 1773670323.96 total 160.20 last 385.63) - open/close 53748 (time 1773670333.96 total 170.20 last 374.77) - open/close 57303 (time 1773670343.96 total 180.21 last 355.43) - open/close 60000 (time 1773670350.64 total 186.88 last 404.03) - open/close 63355 (time 1773670360.64 total 196.89 last 335.28) total: 64768 open/close in 201.63 seconds: 321.22 ops/second - /unlink 6526 (time 1773670376.85 total 10.00 last 652.53) - /unlink 10000 (time 1773670382.56 total 15.71 last 608.24) - /unlink 15718 (time 1773670392.56 total 25.71 last 571.79) - /unlink 20000 (time 1773670399.42 total 32.57 last 624.31) - /unlink 26445 (time 1773670409.42 total 42.57 last 644.44) - /unlink 30000 (time 1773670414.41 total 47.56 last 712.44) - /unlink 37128 (time 1773670424.41 total 57.56 last 712.78) - /unlink 40000 (time 1773670428.29 total 61.45 last 739.42) - /unlink 47563 (time 1773670438.29 total 71.45 last 756.28) - /unlink 50000 (time 1773670441.46 total 74.61 last 770.21) - /unlink 57068 (time 1773670451.46 total 84.61 last 706.77) - /unlink 60000 (time 1773670455.39 total 88.55 last 745.01) total: 64768 /unlink in 95.08 seconds: 681.18 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4555 (time 1773670494.18 total 10.00 last 455.42) - open/close 9204 (time 1773670504.18 total 20.00 last 464.89) - open/close 14179 (time 1773670514.18 total 30.00 last 497.43) - open/close 18576 (time 1773670524.18 total 40.00 last 439.70) - open/close 20000 (time 1773670527.32 total 43.14 last 453.92) - open/close 24748 (time 1773670537.35 total 53.17 last 473.40) - open/close 29507 (time 1773670547.35 total 63.17 last 475.84) - open/close 34399 (time 1773670557.35 total 73.17 last 489.11) - open/close 39309 (time 1773670567.35 total 83.17 last 490.95) - open/close 44299 (time 1773670577.35 total 93.17 last 498.98) - open/close 49283 (time 1773670587.35 total 103.18 last 498.32) - open/close 54072 (time 1773670597.36 total 113.18 last 478.83) - open/close 58883 (time 1773670607.36 total 123.18 last 481.08) - open/close 60000 (time 1773670609.66 total 125.48 last 485.22) total: 64768 open/close in 135.41 seconds: 478.32 ops/second - /unlink 8516 (time 1773670630.92 total 10.00 last 851.60) - /unlink 16604 (time 1773670640.92 total 20.00 last 808.75) - /unlink 20000 (time 1773670644.90 total 23.98 last 852.93) - /unlink 28050 (time 1773670654.90 total 33.98 last 804.93) - /unlink 30000 (time 1773670657.53 total 36.61 last 741.57) - /unlink 37992 (time 1773670667.53 total 46.61 last 799.12) - /unlink 40000 (time 1773670669.87 total 48.96 last 856.88) - /unlink 48162 (time 1773670679.87 total 58.96 last 816.16) - /unlink 50000 (time 1773670682.43 total 61.52 last 717.70) - /unlink 58085 (time 1773670692.43 total 71.52 last 808.47) - /unlink 60000 (time 1773670694.59 total 73.68 last 887.87) total: 64768 /unlink in 80.08 seconds: 808.77 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg414-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server unloading modules via unload_modules_local on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing unload_modules_local oleg414-server: modules unloaded. pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2