== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 12:29:20 (1774888160) Stopping clients: oleg620-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg620-client.virtnet /mnt/lustre2 (opts:-f) oleg620-server: oleg620-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg620-server' oleg620-server: oleg620-server.virtnet: executing load_modules_local oleg620-server: Loading modules from /home/green/git/lustre-release/lustre oleg620-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg620-server: detected 4 online CPUs by sysfs oleg620-server: MODOPTS_LIBCFS= oleg620-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg620-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg620-server: oleg620-server.virtnet: executing set_default_debug -1 all pdsh@oleg620-client: oleg620-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg620-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg620-server: oleg620-server.virtnet: executing set_default_debug -1 all pdsh@oleg620-client: oleg620-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg620-client.virtnet: -o user_xattr,flock 192.168.206.120@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 801 (time 1774888245.94 total 10.01 last 80.04) - open/close 1687 (time 1774888255.94 total 20.01 last 88.57) - open/close 2617 (time 1774888265.95 total 30.02 last 92.92) - open/close 3591 (time 1774888275.95 total 40.02 last 97.40) - open/close 4666 (time 1774888285.95 total 50.02 last 107.50) - open/close 5772 (time 1774888295.96 total 60.03 last 110.49) - open/close 6792 (time 1774888305.97 total 70.04 last 101.91) - open/close 7693 (time 1774888315.97 total 80.04 last 90.07) - open/close 8522 (time 1774888325.98 total 90.05 last 82.83) - open/close 9460 (time 1774888335.98 total 100.05 last 93.79) - open/close 10000 (time 1774888342.89 total 106.96 last 78.14) - open/close 10709 (time 1774888352.89 total 116.96 last 70.90) - open/close 11522 (time 1774888362.90 total 126.96 last 81.28) - open/close 12305 (time 1774888372.90 total 136.97 last 78.27) - open/close 13199 (time 1774888382.90 total 146.97 last 89.38) - open/close 14066 (time 1774888392.91 total 156.98 last 86.65) - open/close 15176 (time 1774888402.91 total 166.98 last 110.93) - open/close 16407 (time 1774888412.92 total 176.99 last 123.03) - open/close 17687 (time 1774888422.92 total 186.99 last 127.96) - open/close 18794 (time 1774888432.92 total 196.99 last 110.69) - open/close 19766 (time 1774888442.92 total 206.99 last 97.20) - open/close 20000 (time 1774888445.32 total 209.39 last 97.54) - open/close 20866 (time 1774888455.33 total 219.39 last 86.58) - open/close 21756 (time 1774888465.34 total 229.40 last 88.90) - open/close 22677 (time 1774888475.34 total 239.41 last 92.04) - open/close 23605 (time 1774888485.35 total 249.42 last 92.74) - open/close 24519 (time 1774888495.37 total 259.43 last 91.27) - open/close 25436 (time 1774888505.37 total 269.44 last 91.62) - open/close 26427 (time 1774888515.38 total 279.44 last 99.09) - open/close 27348 (time 1774888525.38 total 289.44 last 92.10) - open/close 28222 (time 1774888535.38 total 299.44 last 87.38) - open/close 29102 (time 1774888545.39 total 309.45 last 87.93) - open/close 29898 (time 1774888555.39 total 319.46 last 79.55) - open/close 30774 (time 1774888565.39 total 329.46 last 87.58) - open/close 31713 (time 1774888575.41 total 339.48 last 93.76) - open/close 32612 (time 1774888585.41 total 349.48 last 89.85) - open/close 33518 (time 1774888595.42 total 359.48 last 90.57) - open/close 34442 (time 1774888605.42 total 369.49 last 92.40) - open/close 35394 (time 1774888615.42 total 379.49 last 95.16) - open/close 36484 (time 1774888625.43 total 389.49 last 108.94) - open/close 37573 (time 1774888635.43 total 399.50 last 108.85) - open/close 38641 (time 1774888645.43 total 409.50 last 106.79) - open/close 39720 (time 1774888655.44 total 419.50 last 107.86) - open/close 40000 (time 1774888658.33 total 422.39 last 96.86) - open/close 40971 (time 1774888668.33 total 432.40 last 97.06) - open/close 41950 (time 1774888678.33 total 442.40 last 97.87) - open/close 42862 (time 1774888688.34 total 452.40 last 91.17) - open/close 43713 (time 1774888698.34 total 462.41 last 85.10) - open/close 44601 (time 1774888708.35 total 472.42 last 88.71) - open/close 45478 (time 1774888718.36 total 482.43 last 87.58) - open/close 46272 (time 1774888728.38 total 492.45 last 79.27) - open/close 47072 (time 1774888738.39 total 502.46 last 79.92) - open/close 47849 (time 1774888748.39 total 512.46 last 77.67) - open/close 48751 (time 1774888758.39 total 522.46 last 90.20) - open/close 49676 (time 1774888768.40 total 532.47 last 92.42) - open/close 50000 (time 1774888772.33 total 536.40 last 82.48) - open/close 50861 (time 1774888782.33 total 546.40 last 86.08) - open/close 51768 (time 1774888792.34 total 556.41 last 90.64) - open/close 52640 (time 1774888802.34 total 566.41 last 87.16) - open/close 53512 (time 1774888812.35 total 576.42 last 87.13) - open/close 54364 (time 1774888822.36 total 586.43 last 85.12) - open/close 55194 (time 1774888832.37 total 596.44 last 82.90) - open/close 56057 (time 1774888842.38 total 606.45 last 86.21) - open/close 56915 (time 1774888852.40 total 616.47 last 85.66) - open/close 57879 (time 1774888862.40 total 626.47 last 96.38) - open/close 58921 (time 1774888872.40 total 636.47 last 104.19) - open/close 59879 (time 1774888882.41 total 646.48 last 95.73) - open/close 60734 (time 1774888892.43 total 656.49 last 85.37) - open/close 61475 (time 1774888902.44 total 666.51 last 73.99) - open/close 62265 (time 1774888912.45 total 676.51 last 78.94) - open/close 63060 (time 1774888922.45 total 686.52 last 79.44) - open/close 63946 (time 1774888932.46 total 696.52 last 88.58) - open/close 64757 (time 1774888942.46 total 706.52 last 81.09) total: 64768 open/close in 706.66 seconds: 91.65 ops/second - /unlink 762 (time 1774888955.88 total 10.00 last 76.18) - /unlink 1567 (time 1774888965.89 total 20.01 last 80.45) - /unlink 2459 (time 1774888975.89 total 30.02 last 89.12) - /unlink 3240 (time 1774888985.90 total 40.02 last 78.06) - /unlink 4114 (time 1774888995.90 total 50.03 last 87.38) - /unlink 5001 (time 1774889005.90 total 60.03 last 88.69) - /unlink 5836 (time 1774889015.90 total 70.03 last 83.49) - /unlink 6688 (time 1774889025.91 total 80.04 last 85.11) - /unlink 7518 (time 1774889035.92 total 90.05 last 82.93) - /unlink 8425 (time 1774889045.93 total 100.06 last 90.60) - /unlink 9249 (time 1774889055.93 total 110.06 last 82.40) - /unlink 10000 (time 1774889064.85 total 118.97 last 84.25) - /unlink 10794 (time 1774889074.86 total 128.98 last 79.34) - /unlink 11562 (time 1774889084.86 total 138.99 last 76.74) - /unlink 12318 (time 1774889094.87 total 148.99 last 75.54) - /unlink 13114 (time 1774889104.87 total 159.00 last 79.58) - /unlink 14000 (time 1774889114.88 total 169.00 last 88.54) - /unlink 14906 (time 1774889124.89 total 179.01 last 90.52) - /unlink 15876 (time 1774889134.89 total 189.02 last 96.94) - /unlink 16794 (time 1774889144.90 total 199.02 last 91.76) - /unlink 17678 (time 1774889154.91 total 209.03 last 88.32) - /unlink 18589 (time 1774889164.92 total 219.04 last 91.01) - /unlink 19511 (time 1774889174.92 total 229.05 last 92.16) - /unlink 20000 (time 1774889180.69 total 234.81 last 84.81) - /unlink 20807 (time 1774889190.70 total 244.82 last 80.64) - /unlink 21544 (time 1774889200.70 total 254.82 last 73.70) - /unlink 22326 (time 1774889210.70 total 264.83 last 78.15) - /unlink 23145 (time 1774889220.71 total 274.84 last 81.83) - /unlink 23933 (time 1774889230.72 total 284.84 last 78.78) - /unlink 24855 (time 1774889240.72 total 294.84 last 92.18) - /unlink 25701 (time 1774889250.73 total 304.85 last 84.53) - /unlink 26583 (time 1774889260.73 total 314.86 last 88.14) - /unlink 27383 (time 1774889270.74 total 324.87 last 79.91) - /unlink 28175 (time 1774889280.75 total 334.87 last 79.15) - /unlink 28968 (time 1774889290.76 total 344.88 last 79.22) - /unlink 29733 (time 1774889300.77 total 354.89 last 76.46) - /unlink 30000 (time 1774889303.67 total 357.79 last 92.06) - /unlink 30861 (time 1774889313.67 total 367.80 last 86.05) - /unlink 31682 (time 1774889323.68 total 377.80 last 82.07) - /unlink 32560 (time 1774889333.68 total 387.81 last 87.75) - /unlink 33387 (time 1774889343.71 total 397.83 last 82.49) - /unlink 34321 (time 1774889353.71 total 407.84 last 93.34) - /unlink 35226 (time 1774889363.72 total 417.84 last 90.44) - /unlink 36326 (time 1774889373.72 total 427.85 last 109.98) - /unlink 37256 (time 1774889383.74 total 437.86 last 92.88) - /unlink 38157 (time 1774889393.74 total 447.86 last 90.08) - /unlink 39077 (time 1774889403.74 total 457.86 last 91.99) - /unlink 40000 (time 1774889413.36 total 467.49 last 95.91) - /unlink 40907 (time 1774889423.36 total 477.49 last 90.68) - /unlink 41883 (time 1774889433.37 total 487.49 last 97.58) - /unlink 42913 (time 1774889443.37 total 497.49 last 102.99) - /unlink 43891 (time 1774889453.37 total 507.50 last 97.74) - /unlink 44956 (time 1774889463.38 total 517.50 last 106.48) - /unlink 45920 (time 1774889473.38 total 527.50 last 96.39) - /unlink 47038 (time 1774889483.38 total 537.50 last 111.80) - /unlink 48179 (time 1774889493.41 total 547.53 last 113.77) - /unlink 49101 (time 1774889503.41 total 557.53 last 92.17) - /unlink 49884 (time 1774889513.42 total 567.55 last 78.20) - /unlink 50816 (time 1774889523.42 total 577.55 last 93.18) - /unlink 51732 (time 1774889533.43 total 587.55 last 91.54) - /unlink 52678 (time 1774889543.44 total 597.56 last 94.54) - /unlink 53618 (time 1774889553.44 total 607.56 last 93.99) - /unlink 54566 (time 1774889563.45 total 617.57 last 94.71) - /unlink 55435 (time 1774889573.45 total 627.57 last 86.87) - /unlink 56308 (time 1774889583.45 total 637.58 last 87.29) - /unlink 57262 (time 1774889593.46 total 647.58 last 95.36) - /unlink 58213 (time 1774889603.46 total 657.58 last 95.09) - /unlink 59122 (time 1774889613.46 total 667.59 last 90.85) - /unlink 60000 (time 1774889622.04 total 676.17 last 102.34) - /unlink 60967 (time 1774889632.05 total 686.17 last 96.62) - /unlink 61873 (time 1774889642.05 total 696.17 last 90.59) - /unlink 62778 (time 1774889652.07 total 706.19 last 90.33) - /unlink 63787 (time 1774889662.08 total 716.20 last 100.83) - /unlink 64658 (time 1774889672.08 total 726.21 last 87.04) total: 64768 /unlink in 727.37 seconds: 89.04 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1029 (time 1774889706.52 total 10.00 last 102.85) - open/close 2092 (time 1774889716.52 total 20.01 last 106.21) - open/close 3171 (time 1774889726.53 total 30.02 last 107.83) - open/close 4129 (time 1774889736.53 total 40.02 last 95.79) - open/close 5065 (time 1774889746.54 total 50.03 last 93.55) - open/close 6048 (time 1774889756.54 total 60.03 last 98.30) - open/close 6922 (time 1774889766.54 total 70.03 last 87.37) - open/close 7899 (time 1774889776.55 total 80.04 last 97.62) - open/close 8926 (time 1774889786.56 total 90.05 last 102.56) - open/close 9877 (time 1774889796.56 total 100.05 last 95.08) - open/close 10843 (time 1774889806.57 total 110.06 last 96.55) - open/close 11736 (time 1774889816.58 total 120.07 last 89.22) - open/close 12652 (time 1774889826.58 total 130.07 last 91.60) - open/close 13717 (time 1774889836.58 total 140.07 last 106.50) - open/close 14819 (time 1774889846.58 total 150.07 last 110.14) - open/close 15962 (time 1774889856.58 total 160.07 last 114.28) - open/close 16986 (time 1774889866.59 total 170.08 last 102.38) - open/close 18076 (time 1774889876.59 total 180.08 last 108.95) - open/close 19209 (time 1774889886.59 total 190.08 last 113.28) - open/close 20000 (time 1774889892.84 total 196.33 last 126.71) - open/close 21071 (time 1774889902.85 total 206.34 last 106.99) - open/close 22079 (time 1774889912.85 total 216.34 last 100.71) - open/close 23071 (time 1774889922.86 total 226.35 last 99.19) - open/close 24095 (time 1774889932.86 total 236.35 last 102.35) - open/close 25136 (time 1774889942.88 total 246.37 last 103.90) - open/close 26121 (time 1774889952.89 total 256.38 last 98.43) - open/close 27094 (time 1774889962.89 total 266.38 last 97.26) - open/close 28107 (time 1774889972.90 total 276.39 last 101.25) - open/close 29030 (time 1774889982.91 total 286.40 last 92.17) - open/close 30000 (time 1774889992.60 total 296.09 last 100.05) - open/close 30958 (time 1774890002.61 total 306.10 last 95.72) - open/close 31843 (time 1774890012.62 total 316.11 last 88.41) - open/close 32781 (time 1774890022.63 total 326.12 last 93.69) - open/close 33718 (time 1774890032.65 total 336.14 last 93.55) - open/close 34638 (time 1774890042.66 total 346.15 last 91.92) - open/close 35559 (time 1774890052.66 total 356.15 last 92.05) - open/close 36594 (time 1774890062.67 total 366.16 last 103.43) - open/close 37682 (time 1774890072.67 total 376.16 last 108.78) - open/close 38706 (time 1774890082.67 total 386.16 last 102.38) - open/close 39700 (time 1774890092.68 total 396.16 last 99.39) - open/close 40000 (time 1774890095.65 total 399.14 last 100.84) - open/close 40977 (time 1774890105.65 total 409.14 last 97.70) - open/close 41916 (time 1774890115.66 total 419.15 last 93.83) - open/close 42813 (time 1774890125.66 total 429.15 last 89.66) - open/close 43738 (time 1774890135.67 total 439.16 last 92.39) - open/close 44837 (time 1774890145.68 total 449.17 last 109.88) - open/close 45877 (time 1774890155.68 total 459.17 last 103.93) - open/close 46805 (time 1774890165.71 total 469.20 last 92.60) - open/close 47780 (time 1774890175.71 total 479.20 last 97.49) - open/close 49078 (time 1774890185.71 total 489.20 last 129.69) - open/close 50000 (time 1774890192.97 total 496.46 last 127.13) - open/close 51163 (time 1774890202.97 total 506.46 last 116.29) - open/close 52212 (time 1774890212.97 total 516.46 last 104.89) - open/close 53296 (time 1774890222.97 total 526.46 last 108.38) - open/close 54456 (time 1774890232.98 total 536.47 last 115.90) - open/close 55582 (time 1774890242.99 total 546.48 last 112.52) - open/close 56786 (time 1774890252.99 total 556.48 last 120.29) - open/close 57833 (time 1774890263.01 total 566.50 last 104.56) - open/close 59007 (time 1774890273.01 total 576.50 last 117.33) - open/close 60000 (time 1774890282.47 total 585.96 last 105.00) - open/close 61130 (time 1774890292.47 total 595.96 last 112.97) - open/close 62236 (time 1774890302.48 total 605.97 last 110.58) - open/close 63597 (time 1774890312.48 total 615.97 last 136.00) - open/close 64652 (time 1774890322.49 total 625.98 last 105.45) total: 64768 open/close in 626.93 seconds: 103.31 ops/second - /unlink 1315 (time 1774890335.58 total 10.00 last 131.46) - /unlink 2449 (time 1774890345.58 total 20.01 last 113.37) - /unlink 3475 (time 1774890355.58 total 30.01 last 102.56) - /unlink 4594 (time 1774890365.59 total 40.02 last 111.83) - /unlink 5664 (time 1774890375.60 total 50.03 last 106.86) - /unlink 6762 (time 1774890385.60 total 60.03 last 109.77) - /unlink 7896 (time 1774890395.61 total 70.03 last 113.36) - /unlink 9184 (time 1774890405.61 total 80.04 last 128.78) - /unlink 10000 (time 1774890412.51 total 86.94 last 118.20) - /unlink 11081 (time 1774890422.52 total 96.94 last 108.04) - /unlink 12209 (time 1774890432.52 total 106.95 last 112.77) - /unlink 13449 (time 1774890442.52 total 116.95 last 123.96) - /unlink 14630 (time 1774890452.53 total 126.95 last 118.06) - /unlink 15665 (time 1774890462.53 total 136.96 last 103.42) - /unlink 16754 (time 1774890472.54 total 146.97 last 108.86) - /unlink 17818 (time 1774890482.54 total 156.97 last 106.35) - /unlink 18925 (time 1774890492.55 total 166.97 last 110.66) - /unlink 20000 (time 1774890501.59 total 176.02 last 118.81) - /unlink 21527 (time 1774890511.60 total 186.02 last 152.67) - /unlink 22912 (time 1774890521.60 total 196.03 last 138.48) - /unlink 25329 (time 1774890531.60 total 206.03 last 241.63) - /unlink 27758 (time 1774890541.60 total 216.03 last 242.83) - /unlink 30000 (time 1774890549.52 total 223.94 last 283.41) - /unlink 33061 (time 1774890559.52 total 233.95 last 306.02) - /unlink 35933 (time 1774890569.52 total 243.95 last 287.17) - /unlink 38493 (time 1774890579.53 total 253.96 last 255.74) - /unlink 40000 (time 1774890585.88 total 260.31 last 237.13) - /unlink 42310 (time 1774890595.89 total 270.32 last 230.92) - /unlink 44596 (time 1774890605.89 total 280.32 last 228.55) - /unlink 46834 (time 1774890615.89 total 290.32 last 223.79) - /unlink 48983 (time 1774890625.89 total 300.32 last 214.84) - /unlink 50000 (time 1774890629.87 total 304.30 last 255.85) - /unlink 52709 (time 1774890639.87 total 314.30 last 270.83) - /unlink 55821 (time 1774890649.87 total 324.30 last 311.17) - /unlink 58886 (time 1774890659.87 total 334.30 last 306.44) - /unlink 60000 (time 1774890662.84 total 337.26 last 376.19) - /unlink 62934 (time 1774890672.84 total 347.27 last 293.31) total: 64768 /unlink in 353.69 seconds: 183.12 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1523 (time 1774890713.81 total 10.00 last 152.24) - open/close 3216 (time 1774890723.81 total 20.00 last 169.29) - open/close 5202 (time 1774890733.82 total 30.01 last 198.53) - open/close 7066 (time 1774890743.82 total 40.01 last 186.36) - open/close 9192 (time 1774890753.82 total 50.01 last 212.57) - open/close 10000 (time 1774890756.67 total 52.86 last 283.33) - open/close 12318 (time 1774890766.67 total 62.86 last 231.79) - open/close 14600 (time 1774890776.67 total 72.87 last 228.17) - open/close 16253 (time 1774890786.68 total 82.87 last 165.28) - open/close 18784 (time 1774890796.68 total 92.87 last 253.10) - open/close 20000 (time 1774890801.44 total 97.63 last 255.34) - open/close 21482 (time 1774890811.44 total 107.63 last 148.15) - open/close 23153 (time 1774890821.44 total 117.63 last 167.07) - open/close 25055 (time 1774890831.45 total 127.64 last 190.12) - open/close 27050 (time 1774890841.45 total 137.64 last 199.47) - open/close 29699 (time 1774890851.45 total 147.64 last 264.85) - open/close 32331 (time 1774890861.46 total 157.65 last 263.09) - open/close 34413 (time 1774890871.46 total 167.65 last 208.11) - open/close 36311 (time 1774890881.46 total 177.65 last 189.79) - open/close 38529 (time 1774890891.46 total 187.65 last 221.77) - open/close 40000 (time 1774890898.07 total 194.26 last 222.69) - open/close 42442 (time 1774890908.07 total 204.26 last 244.18) - open/close 44610 (time 1774890918.07 total 214.26 last 216.68) - open/close 46766 (time 1774890928.08 total 224.27 last 215.57) - open/close 49115 (time 1774890938.08 total 234.27 last 234.87) - open/close 50000 (time 1774890941.55 total 237.74 last 254.61) - open/close 52219 (time 1774890951.55 total 247.74 last 221.89) - open/close 54116 (time 1774890961.56 total 257.75 last 189.65) - open/close 56076 (time 1774890971.56 total 267.75 last 195.95) - open/close 58109 (time 1774890981.56 total 277.75 last 203.28) - open/close 60000 (time 1774890990.95 total 287.14 last 201.34) - open/close 62294 (time 1774891000.96 total 297.15 last 229.30) - open/close 64616 (time 1774891010.96 total 307.15 last 232.04) total: 64768 open/close in 308.02 seconds: 210.27 ops/second - /unlink 3920 (time 1774891023.42 total 10.00 last 391.96) - /unlink 7200 (time 1774891033.42 total 20.00 last 327.93) - /unlink 10000 (time 1774891041.80 total 28.38 last 334.06) - /unlink 14174 (time 1774891051.81 total 38.39 last 417.18) - /unlink 18038 (time 1774891061.81 total 48.39 last 386.38) - /unlink 20000 (time 1774891065.78 total 52.37 last 493.31) - /unlink 24705 (time 1774891075.78 total 62.37 last 470.49) - /unlink 29664 (time 1774891085.79 total 72.37 last 495.81) - /unlink 34168 (time 1774891095.79 total 82.37 last 450.35) - /unlink 39636 (time 1774891105.79 total 92.37 last 546.77) - /unlink 44456 (time 1774891115.79 total 102.37 last 481.93) - /unlink 49238 (time 1774891125.79 total 112.37 last 478.15) - /unlink 54160 (time 1774891135.79 total 122.38 last 492.15) - /unlink 58676 (time 1774891145.79 total 132.38 last 451.55) - /unlink 60000 (time 1774891148.97 total 135.56 last 416.14) - /unlink 64557 (time 1774891158.98 total 145.56 last 455.60) total: 64768 /unlink in 146.22 seconds: 442.95 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2788 (time 1774891186.58 total 10.00 last 278.73) - open/close 5451 (time 1774891196.58 total 20.00 last 266.27) - open/close 8159 (time 1774891206.58 total 30.01 last 270.72) - open/close 10000 (time 1774891211.79 total 35.21 last 353.79) - open/close 13325 (time 1774891221.79 total 45.21 last 332.47) - open/close 16310 (time 1774891231.79 total 55.21 last 298.49) - open/close 19671 (time 1774891241.79 total 65.21 last 336.03) - open/close 22829 (time 1774891251.79 total 75.22 last 315.75) - open/close 26769 (time 1774891261.79 total 85.22 last 393.99) - open/close 30000 (time 1774891269.10 total 92.52 last 442.37) - open/close 33741 (time 1774891279.10 total 102.52 last 374.09) - open/close 37366 (time 1774891289.10 total 112.52 last 362.44) - open/close 40000 (time 1774891296.39 total 119.81 last 361.16) - open/close 43889 (time 1774891306.39 total 129.82 last 388.84) - open/close 48012 (time 1774891316.39 total 139.82 last 412.29) - open/close 50000 (time 1774891320.29 total 143.71 last 510.16) - open/close 54217 (time 1774891330.29 total 153.71 last 421.62) - open/close 58344 (time 1774891340.29 total 163.72 last 412.62) - open/close 60000 (time 1774891344.91 total 168.33 last 358.91) - open/close 63573 (time 1774891354.91 total 178.33 last 357.29) total: 64768 open/close in 181.33 seconds: 357.19 ops/second - /unlink 7134 (time 1774891369.36 total 10.00 last 713.34) - /unlink 10000 (time 1774891373.37 total 14.01 last 714.70) - /unlink 14828 (time 1774891383.38 total 24.01 last 482.67) - /unlink 20000 (time 1774891392.10 total 32.74 last 592.93) - /unlink 26385 (time 1774891402.10 total 42.74 last 638.43) - /unlink 30000 (time 1774891407.78 total 48.42 last 636.20) - /unlink 35456 (time 1774891417.78 total 58.42 last 545.59) - /unlink 40000 (time 1774891424.50 total 65.14 last 676.14) - /unlink 46607 (time 1774891434.50 total 75.14 last 660.62) - /unlink 50000 (time 1774891440.31 total 80.95 last 584.64) - /unlink 56424 (time 1774891450.31 total 90.95 last 642.39) - /unlink 60000 (time 1774891454.85 total 95.49 last 786.98) total: 64768 /unlink in 103.20 seconds: 627.60 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4458 (time 1774891495.93 total 10.00 last 445.79) - open/close 8371 (time 1774891505.93 total 20.00 last 391.24) - open/close 10000 (time 1774891510.78 total 24.85 last 336.04) - open/close 14200 (time 1774891520.78 total 34.85 last 420.00) - open/close 18326 (time 1774891530.78 total 44.85 last 412.52) - open/close 20000 (time 1774891534.50 total 48.57 last 450.38) - open/close 23745 (time 1774891544.50 total 58.57 last 374.47) - open/close 27044 (time 1774891554.50 total 68.57 last 329.83) - open/close 30000 (time 1774891562.99 total 77.06 last 348.12) - open/close 33284 (time 1774891572.99 total 87.06 last 328.32) - open/close 36177 (time 1774891582.99 total 97.07 last 289.25) - open/close 39660 (time 1774891593.00 total 107.07 last 348.24) - open/close 42650 (time 1774891603.00 total 117.07 last 298.95) - open/close 46353 (time 1774891613.00 total 127.07 last 370.24) - open/close 50000 (time 1774891622.94 total 137.02 last 366.70) - open/close 53445 (time 1774891632.94 total 147.02 last 344.50) - open/close 57206 (time 1774891642.95 total 157.02 last 376.04) - open/close 60000 (time 1774891649.77 total 163.85 last 409.21) - open/close 64056 (time 1774891659.77 total 173.85 last 405.57) total: 64768 open/close in 175.36 seconds: 369.34 ops/second - /unlink 7482 (time 1774891672.67 total 10.00 last 748.11) - /unlink 10000 (time 1774891676.02 total 13.35 last 751.15) - /unlink 17287 (time 1774891686.03 total 23.35 last 728.59) - /unlink 20000 (time 1774891689.34 total 26.67 last 819.47) - /unlink 26681 (time 1774891699.34 total 36.67 last 668.09) - /unlink 30000 (time 1774891704.32 total 41.65 last 666.05) - /unlink 37113 (time 1774891714.32 total 51.65 last 711.22) - /unlink 40000 (time 1774891718.86 total 56.19 last 636.25) - /unlink 47023 (time 1774891728.86 total 66.19 last 702.25) - /unlink 50000 (time 1774891733.48 total 70.81 last 644.51) - /unlink 56228 (time 1774891743.48 total 80.81 last 622.73) - /unlink 60000 (time 1774891749.54 total 86.87 last 622.40) total: 64768 /unlink in 95.18 seconds: 680.51 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg620-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg620-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg620-server stop mds service on oleg620-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg620-server unloading modules via unload_modules_local on: 'oleg620-server' oleg620-server: oleg620-server.virtnet: executing unload_modules_local oleg620-server: modules unloaded. pdsh@oleg620-client: oleg620-client: ssh exited with exit code 2 pdsh@oleg620-client: oleg620-server: ssh exited with exit code 2 pdsh@oleg620-client: oleg620-client: ssh exited with exit code 2 pdsh@oleg620-client: oleg620-server: ssh exited with exit code 2