== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 00:58:48 (1759813128) Stopping clients: oleg424-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg424-client.virtnet /mnt/lustre2 (opts:-f) oleg424-server: oleg424-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing load_modules_local oleg424-server: Loading modules from /home/green/git/lustre-release/lustre oleg424-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg424-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg424-server: detected 4 online CPUs by sysfs oleg424-server: MODOPTS_LIBCFS= oleg424-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg424-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg424-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 940 (time 1759813248.21 total 10.01 last 93.93) - open/close 1764 (time 1759813258.21 total 20.01 last 82.37) - open/close 2602 (time 1759813268.22 total 30.02 last 83.74) - open/close 3439 (time 1759813278.22 total 40.02 last 83.67) - open/close 4324 (time 1759813288.23 total 50.03 last 88.45) - open/close 5190 (time 1759813298.23 total 60.03 last 86.59) - open/close 6100 (time 1759813308.23 total 70.03 last 90.99) - open/close 6996 (time 1759813318.23 total 80.03 last 89.57) - open/close 7922 (time 1759813328.24 total 90.04 last 92.58) - open/close 8991 (time 1759813338.25 total 100.05 last 106.78) - open/close 9807 (time 1759813348.26 total 110.06 last 81.46) - open/close 10000 (time 1759813350.75 total 112.55 last 77.75) - open/close 10936 (time 1759813360.76 total 122.56 last 93.51) - open/close 11751 (time 1759813370.76 total 132.56 last 81.44) - open/close 12560 (time 1759813380.77 total 142.58 last 80.81) - open/close 13451 (time 1759813390.78 total 152.58 last 89.03) - open/close 14289 (time 1759813400.79 total 162.59 last 83.72) - open/close 15152 (time 1759813410.80 total 172.60 last 86.21) - open/close 16081 (time 1759813420.80 total 182.60 last 92.90) - open/close 16951 (time 1759813430.80 total 192.60 last 86.99) - open/close 18062 (time 1759813440.81 total 202.61 last 111.06) - open/close 19391 (time 1759813450.81 total 212.61 last 132.90) - open/close 20000 (time 1759813455.42 total 217.22 last 132.10) - open/close 21464 (time 1759813465.42 total 227.22 last 146.34) - open/close 23397 (time 1759813475.42 total 237.22 last 193.27) - open/close 25242 (time 1759813485.42 total 247.22 last 184.49) - open/close 26624 (time 1759813495.42 total 257.22 last 138.19) - open/close 28080 (time 1759813505.43 total 267.23 last 145.57) - open/close 30000 (time 1759813514.48 total 276.28 last 212.18) - open/close 32010 (time 1759813524.48 total 286.28 last 200.93) - open/close 33979 (time 1759813534.48 total 296.29 last 196.79) - open/close 35794 (time 1759813544.49 total 306.29 last 181.46) - open/close 37630 (time 1759813554.49 total 316.29 last 183.58) - open/close 39558 (time 1759813564.49 total 326.29 last 192.80) - open/close 40000 (time 1759813566.87 total 328.67 last 185.27) - open/close 41536 (time 1759813576.87 total 338.68 last 153.59) - open/close 43296 (time 1759813586.87 total 348.68 last 176.00) - open/close 44988 (time 1759813596.88 total 358.68 last 169.12) - open/close 47233 (time 1759813606.88 total 368.68 last 224.48) - open/close 50000 (time 1759813616.64 total 378.44 last 283.64) - open/close 53446 (time 1759813626.64 total 388.44 last 344.57) - open/close 56461 (time 1759813636.64 total 398.44 last 301.48) - open/close 59506 (time 1759813646.64 total 408.44 last 304.47) - open/close 63055 (time 1759813656.64 total 418.44 last 354.81) total: 64768 open/close in 423.30 seconds: 153.01 ops/second - /unlink 6513 (time 1759813672.89 total 10.00 last 651.25) - /unlink 10000 (time 1759813678.61 total 15.72 last 609.61) - /unlink 16124 (time 1759813688.61 total 25.72 last 612.40) - /unlink 20000 (time 1759813694.41 total 31.52 last 668.11) - /unlink 26392 (time 1759813704.41 total 41.52 last 639.17) - /unlink 30000 (time 1759813709.98 total 47.09 last 647.97) - /unlink 35349 (time 1759813719.98 total 57.09 last 534.86) - /unlink 40000 (time 1759813728.28 total 65.39 last 560.23) - /unlink 46182 (time 1759813738.28 total 75.39 last 618.13) - /unlink 50000 (time 1759813745.59 total 82.71 last 522.27) - /unlink 56276 (time 1759813755.59 total 92.71 last 627.58) - /unlink 60000 (time 1759813761.90 total 99.02 last 590.03) total: 64768 /unlink in 107.97 seconds: 599.86 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3479 (time 1759813803.47 total 10.00 last 347.88) - open/close 6805 (time 1759813813.47 total 20.00 last 332.53) - open/close 9961 (time 1759813823.48 total 30.00 last 315.56) - open/close 13015 (time 1759813833.48 total 40.01 last 305.35) - open/close 15551 (time 1759813843.48 total 50.01 last 253.55) - open/close 17710 (time 1759813853.48 total 60.01 last 215.87) - open/close 20000 (time 1759813861.97 total 68.50 last 269.74) - open/close 23677 (time 1759813871.97 total 78.50 last 367.68) - open/close 27277 (time 1759813881.97 total 88.50 last 359.96) - open/close 30000 (time 1759813890.12 total 96.64 last 334.30) - open/close 33060 (time 1759813900.12 total 106.65 last 305.97) - open/close 36395 (time 1759813910.12 total 116.65 last 333.44) - open/close 40000 (time 1759813919.97 total 126.50 last 365.86) - open/close 42791 (time 1759813929.97 total 136.50 last 279.06) - open/close 46434 (time 1759813939.98 total 146.50 last 364.22) - open/close 49935 (time 1759813949.98 total 156.51 last 350.06) - open/close 53599 (time 1759813959.98 total 166.51 last 366.38) - open/close 56735 (time 1759813969.98 total 176.51 last 313.59) - open/close 60000 (time 1759813977.82 total 184.35 last 416.51) - open/close 63460 (time 1759813987.82 total 194.35 last 345.94) total: 64768 open/close in 197.64 seconds: 327.70 ops/second - /unlink 7966 (time 1759814002.43 total 10.00 last 796.55) - /unlink 10000 (time 1759814005.61 total 13.18 last 639.90) - /unlink 17595 (time 1759814015.61 total 23.18 last 759.44) - /unlink 20000 (time 1759814018.56 total 26.13 last 815.00) - /unlink 27951 (time 1759814028.56 total 36.13 last 795.04) - /unlink 30000 (time 1759814031.29 total 38.86 last 751.93) - /unlink 36895 (time 1759814041.29 total 48.86 last 689.47) - /unlink 40000 (time 1759814045.36 total 52.93 last 761.84) - /unlink 47635 (time 1759814055.38 total 62.95 last 762.35) - /unlink 50000 (time 1759814058.71 total 66.27 last 711.04) - /unlink 58020 (time 1759814068.71 total 76.27 last 801.93) - /unlink 60000 (time 1759814071.21 total 78.78 last 789.60) total: 64768 /unlink in 84.51 seconds: 766.43 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4638 (time 1759814108.90 total 10.00 last 463.75) - open/close 9975 (time 1759814118.90 total 20.00 last 533.63) - open/close 15166 (time 1759814128.90 total 30.00 last 519.02) - open/close 19642 (time 1759814138.91 total 40.01 last 447.51) - open/close 24117 (time 1759814148.91 total 50.01 last 447.47) - open/close 28680 (time 1759814158.91 total 60.01 last 456.29) - open/close 30000 (time 1759814161.82 total 62.92 last 452.81) - open/close 34467 (time 1759814171.82 total 72.92 last 446.66) - open/close 39013 (time 1759814181.83 total 82.92 last 454.52) - open/close 40000 (time 1759814183.96 total 85.06 last 462.09) - open/close 45007 (time 1759814193.96 total 95.06 last 500.70) - open/close 49884 (time 1759814203.96 total 105.06 last 487.66) - open/close 54671 (time 1759814213.96 total 115.06 last 478.67) - open/close 59312 (time 1759814223.96 total 125.06 last 464.03) - open/close 63861 (time 1759814233.97 total 135.06 last 454.85) total: 64768 open/close in 137.04 seconds: 472.61 ops/second - /unlink 8118 (time 1759814247.28 total 10.00 last 811.78) - /unlink 10000 (time 1759814249.57 total 12.29 last 822.96) - /unlink 18411 (time 1759814259.57 total 22.29 last 841.09) - /unlink 26999 (time 1759814269.57 total 32.29 last 858.77) - /unlink 30000 (time 1759814272.88 total 35.60 last 905.31) - /unlink 38079 (time 1759814282.88 total 45.60 last 807.89) - /unlink 40000 (time 1759814285.26 total 47.98 last 806.39) - /unlink 48265 (time 1759814295.26 total 57.99 last 826.39) - /unlink 56688 (time 1759814305.26 total 67.99 last 842.30) - /unlink 60000 (time 1759814308.87 total 71.60 last 917.70) total: 64768 /unlink in 77.78 seconds: 832.66 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4498 (time 1759814355.97 total 10.00 last 449.73) - open/close 8950 (time 1759814365.97 total 20.00 last 445.16) - open/close 10000 (time 1759814368.44 total 22.47 last 424.74) - open/close 14497 (time 1759814378.45 total 32.48 last 449.51) - open/close 19249 (time 1759814388.45 total 42.48 last 475.12) - open/close 23847 (time 1759814398.45 total 52.48 last 459.73) - open/close 28485 (time 1759814408.45 total 62.48 last 463.73) - open/close 30000 (time 1759814411.87 total 65.90 last 442.83) - open/close 34928 (time 1759814421.87 total 75.91 last 492.74) - open/close 39284 (time 1759814431.87 total 85.91 last 435.59) - open/close 43770 (time 1759814441.87 total 95.91 last 448.59) - open/close 48381 (time 1759814451.87 total 105.91 last 461.08) - open/close 50000 (time 1759814456.02 total 110.05 last 390.51) - open/close 53912 (time 1759814466.02 total 120.05 last 391.12) - open/close 57387 (time 1759814476.02 total 130.06 last 347.42) - open/close 60000 (time 1759814482.74 total 136.77 last 389.35) - open/close 62906 (time 1759814492.74 total 146.77 last 290.55) total: 64768 open/close in 152.89 seconds: 423.63 ops/second - /unlink 7431 (time 1759814510.22 total 10.00 last 742.99) - /unlink 10000 (time 1759814514.12 total 13.90 last 659.34) - /unlink 16631 (time 1759814524.12 total 23.90 last 663.07) - /unlink 20000 (time 1759814528.70 total 28.47 last 736.14) - /unlink 27808 (time 1759814538.70 total 38.48 last 780.78) - /unlink 30000 (time 1759814541.35 total 41.13 last 825.90) - /unlink 37740 (time 1759814551.35 total 51.13 last 773.99) - /unlink 40000 (time 1759814554.58 total 54.36 last 699.30) - /unlink 48197 (time 1759814564.58 total 64.36 last 819.61) - /unlink 50000 (time 1759814566.59 total 66.37 last 898.49) - /unlink 57159 (time 1759814576.59 total 76.37 last 715.87) - /unlink 60000 (time 1759814580.65 total 80.43 last 700.07) total: 64768 /unlink in 86.74 seconds: 746.70 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4251 (time 1759814615.52 total 10.00 last 425.07) - open/close 8471 (time 1759814625.53 total 20.00 last 421.89) - open/close 10000 (time 1759814629.43 total 23.91 last 391.42) - open/close 14151 (time 1759814639.43 total 33.91 last 415.08) - open/close 18622 (time 1759814649.44 total 43.91 last 447.01) - open/close 20000 (time 1759814652.47 total 46.95 last 453.84) - open/close 24438 (time 1759814662.47 total 56.95 last 443.70) - open/close 28959 (time 1759814672.47 total 66.95 last 452.10) - open/close 30000 (time 1759814674.67 total 69.14 last 475.04) - open/close 34685 (time 1759814684.67 total 79.14 last 468.48) - open/close 39558 (time 1759814694.67 total 89.14 last 487.22) - open/close 43748 (time 1759814704.67 total 99.15 last 418.97) - open/close 48181 (time 1759814714.67 total 109.15 last 443.26) - open/close 50000 (time 1759814718.63 total 113.10 last 459.49) - open/close 54749 (time 1759814728.63 total 123.11 last 474.84) - open/close 59497 (time 1759814738.63 total 133.11 last 474.79) - open/close 63715 (time 1759814748.63 total 143.11 last 421.79) total: 64768 open/close in 145.70 seconds: 444.53 ops/second - /unlink 7610 (time 1759814762.69 total 10.00 last 760.98) - /unlink 10000 (time 1759814765.69 total 13.00 last 797.12) - /unlink 17319 (time 1759814775.69 total 23.00 last 731.86) - /unlink 20000 (time 1759814779.22 total 26.53 last 760.18) - /unlink 28276 (time 1759814789.22 total 36.53 last 827.60) - /unlink 30000 (time 1759814791.49 total 38.80 last 758.49) - /unlink 37883 (time 1759814801.49 total 48.80 last 788.25) - /unlink 40000 (time 1759814803.99 total 51.29 last 848.49) - /unlink 47387 (time 1759814813.99 total 61.30 last 738.65) - /unlink 50000 (time 1759814818.10 total 65.40 last 636.27) - /unlink 56988 (time 1759814828.10 total 75.40 last 698.71) - /unlink 60000 (time 1759814831.46 total 78.77 last 894.50) total: 64768 /unlink in 85.12 seconds: 760.93 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg424-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server stop mds service on oleg424-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing unload_modules_local oleg424-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg424-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2