== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 22:37:20 (1763350640) Stopping clients: oleg120-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg120-client.virtnet /mnt/lustre2 (opts:-f) oleg120-server: oleg120-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg120-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg120-server: detected 4 online CPUs by sysfs oleg120-server: MODOPTS_LIBCFS= oleg120-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg120-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg120-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 582 (time 1763350744.91 total 10.01 last 58.17) - open/close 1195 (time 1763350754.91 total 20.01 last 61.29) - open/close 1794 (time 1763350764.94 total 30.04 last 59.72) - open/close 2444 (time 1763350774.94 total 40.04 last 65.00) - open/close 3066 (time 1763350784.96 total 50.05 last 62.11) - open/close 3715 (time 1763350794.96 total 60.06 last 64.87) - open/close 4351 (time 1763350804.97 total 70.06 last 63.57) - open/close 4952 (time 1763350814.97 total 80.06 last 60.09) - open/close 5584 (time 1763350824.98 total 90.07 last 63.14) - open/close 6190 (time 1763350834.99 total 100.08 last 60.54) - open/close 6794 (time 1763350845.00 total 110.09 last 60.37) - open/close 7408 (time 1763350855.01 total 120.10 last 61.33) - open/close 8076 (time 1763350865.01 total 130.10 last 66.80) - open/close 8688 (time 1763350875.02 total 140.11 last 61.13) - open/close 9326 (time 1763350885.05 total 150.14 last 63.60) - open/close 9907 (time 1763350895.06 total 160.16 last 58.02) - open/close 10504 (time 1763350905.07 total 170.16 last 59.66) - open/close 11123 (time 1763350915.08 total 180.17 last 61.85) - open/close 11760 (time 1763350925.10 total 190.19 last 63.58) - open/close 12390 (time 1763350935.12 total 200.21 last 62.89) - open/close 13039 (time 1763350945.15 total 210.25 last 64.65) - open/close 13671 (time 1763350955.16 total 220.26 last 63.13) - open/close 14303 (time 1763350965.18 total 230.27 last 63.11) - open/close 14936 (time 1763350975.19 total 240.28 last 63.26) - open/close 15599 (time 1763350985.20 total 250.29 last 66.23) - open/close 16232 (time 1763350995.21 total 260.30 last 63.22) - open/close 16862 (time 1763351005.22 total 270.32 last 62.91) - open/close 17504 (time 1763351015.24 total 280.33 last 64.08) - open/close 18152 (time 1763351025.24 total 290.33 last 64.80) - open/close 18811 (time 1763351035.26 total 300.35 last 65.79) - open/close 19439 (time 1763351045.27 total 310.36 last 62.73) - open/close 20000 (time 1763351054.25 total 319.34 last 62.49) - open/close 20608 (time 1763351064.25 total 329.34 last 60.78) - open/close 21267 (time 1763351074.26 total 339.35 last 65.84) - open/close 21963 (time 1763351084.28 total 349.37 last 69.46) - open/close 22588 (time 1763351094.29 total 359.38 last 62.41) - open/close 23207 (time 1763351104.30 total 369.40 last 61.82) - open/close 23835 (time 1763351114.32 total 379.41 last 62.70) - open/close 24478 (time 1763351124.32 total 389.42 last 64.28) - open/close 25134 (time 1763351134.34 total 399.43 last 65.51) - open/close 25761 (time 1763351144.35 total 409.45 last 62.59) - open/close 26384 (time 1763351154.36 total 419.45 last 62.25) - open/close 27007 (time 1763351164.36 total 429.46 last 62.29) - open/close 27578 (time 1763351174.37 total 439.46 last 57.08) - open/close 28197 (time 1763351184.38 total 449.47 last 61.82) - open/close 28801 (time 1763351194.39 total 459.48 last 60.34) - open/close 29433 (time 1763351204.40 total 469.50 last 63.13) - open/close 30000 (time 1763351213.29 total 478.38 last 63.80) - open/close 30644 (time 1763351223.29 total 488.38 last 64.39) - open/close 31239 (time 1763351233.29 total 498.38 last 59.50) - open/close 31831 (time 1763351243.31 total 508.41 last 59.08) - open/close 32400 (time 1763351253.32 total 518.42 last 56.84) - open/close 32867 (time 1763351263.33 total 528.42 last 46.68) - open/close 33445 (time 1763351273.35 total 538.44 last 57.67) - open/close 34056 (time 1763351283.36 total 548.45 last 61.04) - open/close 34645 (time 1763351293.37 total 558.46 last 58.84) - open/close 35251 (time 1763351303.40 total 568.49 last 60.44) - open/close 35869 (time 1763351313.40 total 578.49 last 61.77) - open/close 36483 (time 1763351323.40 total 588.50 last 61.38) - open/close 37127 (time 1763351333.41 total 598.50 last 64.35) - open/close 37749 (time 1763351343.42 total 608.51 last 62.15) - open/close 38450 (time 1763351353.42 total 618.51 last 70.10) - open/close 39158 (time 1763351363.44 total 628.53 last 70.67) - open/close 39869 (time 1763351373.44 total 638.53 last 71.10) - open/close 40000 (time 1763351375.57 total 640.66 last 61.44) - open/close 40669 (time 1763351385.57 total 650.67 last 66.88) - open/close 41355 (time 1763351395.57 total 660.67 last 68.60) - open/close 41959 (time 1763351405.58 total 670.67 last 60.39) - open/close 42578 (time 1763351415.58 total 680.68 last 61.86) - open/close 43229 (time 1763351425.60 total 690.69 last 65.00) - open/close 43890 (time 1763351435.60 total 700.69 last 66.09) - open/close 44576 (time 1763351445.60 total 710.70 last 68.58) - open/close 45241 (time 1763351455.62 total 720.71 last 66.42) - open/close 45877 (time 1763351465.63 total 730.72 last 63.53) - open/close 46503 (time 1763351475.63 total 740.73 last 62.56) - open/close 47060 (time 1763351485.63 total 750.73 last 55.69) - open/close 47730 (time 1763351495.65 total 760.74 last 66.93) - open/close 48442 (time 1763351505.66 total 770.75 last 71.13) - open/close 49073 (time 1763351515.67 total 780.76 last 63.02) - open/close 49706 (time 1763351525.67 total 790.76 last 63.30) - open/close 50000 (time 1763351530.24 total 795.33 last 64.30) - open/close 50655 (time 1763351540.25 total 805.34 last 65.44) - open/close 51316 (time 1763351550.26 total 815.36 last 66.00) - open/close 52007 (time 1763351560.26 total 825.36 last 69.09) - open/close 52648 (time 1763351570.27 total 835.36 last 64.09) - open/close 53305 (time 1763351580.27 total 845.36 last 65.69) - open/close 53991 (time 1763351590.28 total 855.37 last 68.51) - open/close 54583 (time 1763351600.29 total 865.38 last 59.15) - open/close 55172 (time 1763351610.30 total 875.39 last 58.86) - open/close 55841 (time 1763351620.31 total 885.40 last 66.83) - open/close 56495 (time 1763351630.33 total 895.42 last 65.27) - open/close 57158 (time 1763351640.34 total 905.43 last 66.25) - open/close 57857 (time 1763351650.34 total 915.44 last 69.85) - open/close 58485 (time 1763351660.35 total 925.44 last 62.76) - open/close 59152 (time 1763351670.35 total 935.45 last 66.67) - open/close 59829 (time 1763351680.36 total 945.45 last 67.68) - open/close 60000 (time 1763351682.84 total 947.93 last 68.89) - open/close 60685 (time 1763351692.85 total 957.94 last 68.44) - open/close 61351 (time 1763351702.86 total 967.95 last 66.54) - open/close 62028 (time 1763351712.86 total 977.95 last 67.68) - open/close 62711 (time 1763351722.86 total 987.95 last 68.30) - open/close 63406 (time 1763351732.87 total 997.96 last 69.43) - open/close 64108 (time 1763351742.88 total 1007.97 last 70.14) total: 64768 open/close in 1017.17 seconds: 63.67 ops/second - /unlink 676 (time 1763351764.81 total 10.01 last 67.52) - /unlink 1381 (time 1763351774.82 total 20.03 last 70.41) - /unlink 2077 (time 1763351784.85 total 30.05 last 69.40) - /unlink 2789 (time 1763351794.85 total 40.06 last 71.19) - /unlink 3500 (time 1763351804.86 total 50.06 last 71.04) - /unlink 4261 (time 1763351814.86 total 60.07 last 76.09) - /unlink 5148 (time 1763351824.86 total 70.07 last 88.69) - /unlink 6121 (time 1763351834.87 total 80.07 last 97.28) - /unlink 7039 (time 1763351844.87 total 90.08 last 91.73) - /unlink 7832 (time 1763351854.88 total 100.08 last 79.25) - /unlink 8574 (time 1763351864.89 total 110.09 last 74.16) - /unlink 9293 (time 1763351874.89 total 120.09 last 71.87) - /unlink 10000 (time 1763351884.18 total 129.39 last 76.08) - /unlink 10717 (time 1763351894.19 total 139.39 last 71.64) - /unlink 11486 (time 1763351904.20 total 149.40 last 76.87) - /unlink 12372 (time 1763351914.20 total 159.40 last 88.55) - /unlink 13190 (time 1763351924.21 total 169.41 last 81.75) - /unlink 14066 (time 1763351934.22 total 179.42 last 87.49) - /unlink 15056 (time 1763351944.23 total 189.43 last 98.89) - /unlink 15922 (time 1763351954.24 total 199.44 last 86.53) - /unlink 16954 (time 1763351964.24 total 209.45 last 103.15) - /unlink 17907 (time 1763351974.26 total 219.46 last 95.17) - /unlink 18871 (time 1763351984.27 total 229.47 last 96.31) - /unlink 20000 (time 1763351993.27 total 238.47 last 125.41) - /unlink 20943 (time 1763352003.27 total 248.48 last 94.26) - /unlink 21908 (time 1763352013.28 total 258.48 last 96.44) - /unlink 22822 (time 1763352023.29 total 268.49 last 91.33) - /unlink 23852 (time 1763352033.30 total 278.50 last 102.90) - /unlink 24791 (time 1763352043.30 total 288.50 last 93.86) - /unlink 25706 (time 1763352053.31 total 298.51 last 91.43) - /unlink 26577 (time 1763352063.31 total 308.51 last 87.08) - /unlink 27445 (time 1763352073.32 total 318.52 last 86.72) - /unlink 28449 (time 1763352083.33 total 328.53 last 100.35) - /unlink 29563 (time 1763352093.33 total 338.53 last 111.39) - /unlink 30000 (time 1763352097.55 total 342.75 last 103.47) - /unlink 30971 (time 1763352107.55 total 352.75 last 97.08) - /unlink 31935 (time 1763352117.55 total 362.76 last 96.39) - /unlink 32874 (time 1763352127.56 total 372.77 last 93.80) - /unlink 33765 (time 1763352137.57 total 382.77 last 89.08) - /unlink 34580 (time 1763352147.57 total 392.77 last 81.46) - /unlink 35457 (time 1763352157.58 total 402.79 last 87.58) - /unlink 36377 (time 1763352167.59 total 412.79 last 91.97) - /unlink 37282 (time 1763352177.60 total 422.80 last 90.42) - /unlink 38269 (time 1763352187.60 total 432.80 last 98.66) - /unlink 39385 (time 1763352197.60 total 442.81 last 111.56) - /unlink 40000 (time 1763352204.53 total 449.74 last 88.73) - /unlink 40949 (time 1763352214.54 total 459.74 last 94.89) - /unlink 41972 (time 1763352224.54 total 469.74 last 102.25) - /unlink 42896 (time 1763352234.56 total 479.76 last 92.23) - /unlink 43777 (time 1763352244.57 total 489.77 last 87.99) - /unlink 44724 (time 1763352254.58 total 499.78 last 94.65) - /unlink 45591 (time 1763352264.58 total 509.79 last 86.64) - /unlink 46569 (time 1763352274.59 total 519.79 last 97.75) - /unlink 47720 (time 1763352284.59 total 529.79 last 115.06) - /unlink 48720 (time 1763352294.60 total 539.80 last 99.96) - /unlink 49919 (time 1763352304.60 total 549.80 last 119.84) - /unlink 50980 (time 1763352314.61 total 559.81 last 106.05) - /unlink 52017 (time 1763352324.62 total 569.83 last 103.53) - /unlink 53046 (time 1763352334.63 total 579.83 last 102.87) - /unlink 53900 (time 1763352344.63 total 589.84 last 85.33) - /unlink 54990 (time 1763352354.64 total 599.84 last 108.97) - /unlink 55907 (time 1763352364.65 total 609.85 last 91.62) - /unlink 56934 (time 1763352374.66 total 619.86 last 102.58) - /unlink 57948 (time 1763352384.66 total 629.86 last 101.36) - /unlink 59019 (time 1763352394.67 total 639.87 last 107.01) - /unlink 60000 (time 1763352403.17 total 648.37 last 115.38) - /unlink 60944 (time 1763352413.18 total 658.38 last 94.35) - /unlink 61847 (time 1763352423.18 total 668.38 last 90.27) - /unlink 62805 (time 1763352433.18 total 678.39 last 95.76) - /unlink 63802 (time 1763352443.19 total 688.39 last 99.63) total: 64768 /unlink in 698.15 seconds: 92.77 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 972 (time 1763352487.53 total 10.00 last 97.16) - open/close 1977 (time 1763352497.53 total 20.01 last 100.49) - open/close 2990 (time 1763352507.53 total 30.01 last 101.27) - open/close 4051 (time 1763352517.54 total 40.01 last 106.09) - open/close 5052 (time 1763352527.54 total 50.02 last 100.03) - open/close 5910 (time 1763352537.55 total 60.03 last 85.70) - open/close 6753 (time 1763352547.56 total 70.03 last 84.29) - open/close 7578 (time 1763352557.57 total 80.04 last 82.42) - open/close 8568 (time 1763352567.57 total 90.04 last 98.96) - open/close 9721 (time 1763352577.57 total 100.05 last 115.25) - open/close 10000 (time 1763352580.25 total 102.73 last 104.13) - open/close 11061 (time 1763352590.26 total 112.73 last 106.04) - open/close 12246 (time 1763352600.26 total 122.73 last 118.48) - open/close 13327 (time 1763352610.27 total 132.74 last 108.03) - open/close 14343 (time 1763352620.27 total 142.75 last 101.54) - open/close 15510 (time 1763352630.28 total 152.75 last 116.61) - open/close 16648 (time 1763352640.28 total 162.76 last 113.75) - open/close 17578 (time 1763352650.29 total 172.76 last 92.99) - open/close 18520 (time 1763352660.29 total 182.76 last 94.14) - open/close 19642 (time 1763352670.29 total 192.77 last 112.19) - open/close 20000 (time 1763352673.94 total 196.41 last 98.13) - open/close 20915 (time 1763352683.94 total 206.42 last 91.47) - open/close 21744 (time 1763352693.95 total 216.42 last 82.88) - open/close 22591 (time 1763352703.95 total 226.42 last 84.69) - open/close 23471 (time 1763352713.96 total 236.44 last 87.87) - open/close 24374 (time 1763352723.97 total 246.44 last 90.23) - open/close 25445 (time 1763352733.98 total 256.45 last 107.01) - open/close 26651 (time 1763352743.98 total 266.46 last 120.54) - open/close 27647 (time 1763352753.98 total 276.46 last 99.59) - open/close 28545 (time 1763352763.99 total 286.46 last 89.79) - open/close 29533 (time 1763352773.99 total 296.46 last 98.74) - open/close 30000 (time 1763352779.15 total 301.63 last 90.45) - open/close 30992 (time 1763352789.17 total 311.64 last 99.06) - open/close 32054 (time 1763352799.17 total 321.65 last 106.15) - open/close 33208 (time 1763352809.19 total 331.66 last 115.27) - open/close 34486 (time 1763352819.19 total 341.66 last 127.80) - open/close 35564 (time 1763352829.19 total 351.66 last 107.74) - open/close 36474 (time 1763352839.20 total 361.67 last 90.95) - open/close 37428 (time 1763352849.20 total 371.67 last 95.36) - open/close 38666 (time 1763352859.21 total 381.68 last 123.74) - open/close 39920 (time 1763352869.21 total 391.69 last 125.31) - open/close 40968 (time 1763352879.22 total 401.69 last 104.78) - open/close 42152 (time 1763352889.22 total 411.69 last 118.35) - open/close 43383 (time 1763352899.22 total 421.69 last 123.08) - open/close 44495 (time 1763352909.23 total 431.70 last 111.13) - open/close 45442 (time 1763352919.23 total 441.71 last 94.65) - open/close 46535 (time 1763352929.24 total 451.71 last 109.24) - open/close 47693 (time 1763352939.24 total 461.72 last 115.74) - open/close 48913 (time 1763352949.25 total 471.72 last 121.91) - open/close 49997 (time 1763352959.25 total 481.73 last 108.38) - open/close 51227 (time 1763352969.26 total 491.73 last 122.91) - open/close 52428 (time 1763352979.26 total 501.73 last 120.10) - open/close 53893 (time 1763352989.27 total 511.74 last 146.44) - open/close 55047 (time 1763352999.27 total 521.74 last 115.40) - open/close 56186 (time 1763353009.27 total 531.74 last 113.84) - open/close 57581 (time 1763353019.29 total 541.76 last 139.23) - open/close 59233 (time 1763353029.29 total 551.77 last 165.14) - open/close 60000 (time 1763353033.60 total 556.07 last 178.25) - open/close 61746 (time 1763353043.60 total 566.08 last 174.49) - open/close 63372 (time 1763353053.60 total 576.08 last 162.59) total: 64768 open/close in 584.65 seconds: 110.78 ops/second - /unlink 2066 (time 1763353074.07 total 10.01 last 206.34) - /unlink 4227 (time 1763353084.07 total 20.01 last 216.05) - /unlink 6309 (time 1763353094.07 total 30.02 last 208.16) - /unlink 8601 (time 1763353104.07 total 40.02 last 229.17) - /unlink 10000 (time 1763353109.64 total 45.58 last 251.46) - /unlink 12665 (time 1763353119.64 total 55.58 last 266.49) - /unlink 15428 (time 1763353129.64 total 65.59 last 276.21) - /unlink 18248 (time 1763353139.64 total 75.59 last 281.94) - /unlink 20000 (time 1763353147.52 total 83.47 last 222.38) - /unlink 22817 (time 1763353157.56 total 93.50 last 280.70) - /unlink 27179 (time 1763353167.56 total 103.50 last 436.14) - /unlink 30000 (time 1763353177.28 total 113.22 last 290.18) - /unlink 32415 (time 1763353187.28 total 123.23 last 241.46) - /unlink 35149 (time 1763353197.28 total 133.23 last 273.35) - /unlink 38300 (time 1763353207.28 total 143.23 last 315.09) - /unlink 40000 (time 1763353213.07 total 149.02 last 293.70) - /unlink 42179 (time 1763353223.07 total 159.02 last 217.83) - /unlink 44717 (time 1763353233.08 total 169.02 last 253.75) - /unlink 47137 (time 1763353243.08 total 179.02 last 241.94) - /unlink 49583 (time 1763353253.08 total 189.03 last 244.51) - /unlink 52374 (time 1763353263.09 total 199.03 last 279.01) - /unlink 55152 (time 1763353273.09 total 209.04 last 277.66) - /unlink 57922 (time 1763353283.09 total 219.04 last 276.98) - /unlink 60000 (time 1763353290.57 total 226.52 last 277.71) - /unlink 62992 (time 1763353300.58 total 236.52 last 299.11) total: 64768 /unlink in 242.40 seconds: 267.20 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1637 (time 1763353335.49 total 10.00 last 163.63) - open/close 3276 (time 1763353345.49 total 20.01 last 163.86) - open/close 5009 (time 1763353355.49 total 30.01 last 173.22) - open/close 6318 (time 1763353365.50 total 40.02 last 130.83) - open/close 8051 (time 1763353375.51 total 50.02 last 173.21) - open/close 9679 (time 1763353385.51 total 60.03 last 162.69) - open/close 10000 (time 1763353387.80 total 62.31 last 140.45) - open/close 11413 (time 1763353397.80 total 72.32 last 141.25) - open/close 13095 (time 1763353407.81 total 82.33 last 168.03) - open/close 14828 (time 1763353417.81 total 92.33 last 173.28) - open/close 16154 (time 1763353427.82 total 102.33 last 132.54) - open/close 17523 (time 1763353437.82 total 112.34 last 136.78) - open/close 19118 (time 1763353447.83 total 122.35 last 159.44) - open/close 20000 (time 1763353453.14 total 127.66 last 165.94) - open/close 21548 (time 1763353463.16 total 137.68 last 154.56) - open/close 23226 (time 1763353473.16 total 147.68 last 167.71) - open/close 24892 (time 1763353483.17 total 157.69 last 166.51) - open/close 25989 (time 1763353493.18 total 167.69 last 109.65) - open/close 27344 (time 1763353503.18 total 177.70 last 135.41) - open/close 28933 (time 1763353513.19 total 187.70 last 158.83) - open/close 30000 (time 1763353520.48 total 195.00 last 146.23) - open/close 31360 (time 1763353530.49 total 205.00 last 135.96) - open/close 33040 (time 1763353540.49 total 215.00 last 167.97) - open/close 34768 (time 1763353550.49 total 225.01 last 172.79) - open/close 36477 (time 1763353560.49 total 235.01 last 170.85) - open/close 38139 (time 1763353570.50 total 245.01 last 166.14) - open/close 40000 (time 1763353580.05 total 254.57 last 194.80) - open/close 41801 (time 1763353590.05 total 264.57 last 180.04) - open/close 43257 (time 1763353600.06 total 274.57 last 145.55) - open/close 44742 (time 1763353610.06 total 284.58 last 148.44) - open/close 46137 (time 1763353620.06 total 294.58 last 139.43) - open/close 47676 (time 1763353630.07 total 304.59 last 153.80) - open/close 49127 (time 1763353640.08 total 314.60 last 144.95) - open/close 50000 (time 1763353646.68 total 321.20 last 132.28) - open/close 51612 (time 1763353656.71 total 331.23 last 160.66) - open/close 53512 (time 1763353666.72 total 341.23 last 190.00) - open/close 55076 (time 1763353676.72 total 351.23 last 156.39) - open/close 56992 (time 1763353686.72 total 361.23 last 191.55) - open/close 58508 (time 1763353696.72 total 371.24 last 151.58) - open/close 59631 (time 1763353706.72 total 381.24 last 112.29) - open/close 60000 (time 1763353709.55 total 384.07 last 130.41) - open/close 61436 (time 1763353719.56 total 394.08 last 143.43) - open/close 62765 (time 1763353729.57 total 404.08 last 132.82) - open/close 64247 (time 1763353739.57 total 414.09 last 148.12) total: 64768 open/close in 417.82 seconds: 155.01 ops/second - /unlink 2773 (time 1763353755.32 total 10.00 last 277.20) - /unlink 5346 (time 1763353765.32 total 20.01 last 257.21) - /unlink 7904 (time 1763353775.32 total 30.01 last 255.74) - /unlink 10000 (time 1763353781.96 total 36.65 last 315.64) - /unlink 12634 (time 1763353791.96 total 46.65 last 263.36) - /unlink 15189 (time 1763353801.96 total 56.65 last 255.50) - /unlink 18360 (time 1763353811.97 total 66.65 last 317.01) - /unlink 20000 (time 1763353817.65 total 72.33 last 288.78) - /unlink 23248 (time 1763353827.65 total 82.34 last 324.75) - /unlink 26525 (time 1763353837.65 total 92.34 last 327.64) - /unlink 29595 (time 1763353847.65 total 102.34 last 306.92) - /unlink 33062 (time 1763353857.66 total 112.34 last 346.59) - /unlink 35900 (time 1763353867.66 total 122.34 last 283.80) - /unlink 39090 (time 1763353877.66 total 132.34 last 318.99) - /unlink 40000 (time 1763353880.83 total 135.52 last 286.33) - /unlink 43071 (time 1763353890.83 total 145.52 last 307.10) - /unlink 45932 (time 1763353900.84 total 155.52 last 286.05) - /unlink 49170 (time 1763353910.84 total 165.52 last 323.78) - /unlink 50000 (time 1763353913.98 total 168.67 last 264.11) - /unlink 52607 (time 1763353923.98 total 178.67 last 260.67) - /unlink 55753 (time 1763353933.98 total 188.67 last 314.55) - /unlink 58525 (time 1763353943.98 total 198.67 last 277.14) - /unlink 60000 (time 1763353947.55 total 202.24 last 413.88) - /unlink 64135 (time 1763353957.55 total 212.24 last 413.45) total: 64768 /unlink in 213.41 seconds: 303.49 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2348 (time 1763353988.90 total 10.00 last 234.80) - open/close 4698 (time 1763353998.90 total 20.00 last 234.99) - open/close 6368 (time 1763354008.91 total 30.00 last 166.94) - open/close 8138 (time 1763354018.91 total 40.01 last 176.98) - open/close 9956 (time 1763354028.91 total 50.01 last 181.76) - open/close 11884 (time 1763354038.91 total 60.01 last 192.74) - open/close 13869 (time 1763354048.91 total 70.01 last 198.48) - open/close 15816 (time 1763354058.92 total 80.01 last 194.64) - open/close 17974 (time 1763354068.92 total 90.02 last 215.78) - open/close 19995 (time 1763354078.92 total 100.02 last 202.06) - open/close 21645 (time 1763354088.93 total 110.03 last 164.87) - open/close 23522 (time 1763354098.93 total 120.03 last 187.64) - open/close 25183 (time 1763354108.94 total 130.03 last 166.01) - open/close 27009 (time 1763354118.94 total 140.04 last 182.51) - open/close 28738 (time 1763354128.94 total 150.04 last 172.88) - open/close 30000 (time 1763354135.67 total 156.77 last 187.43) - open/close 32105 (time 1763354145.68 total 166.78 last 210.46) - open/close 33757 (time 1763354155.68 total 176.78 last 165.15) - open/close 35673 (time 1763354165.68 total 186.78 last 191.55) - open/close 37588 (time 1763354175.68 total 196.78 last 191.49) - open/close 39470 (time 1763354185.69 total 206.78 last 188.16) - open/close 40000 (time 1763354188.51 total 209.61 last 187.47) - open/close 41778 (time 1763354198.51 total 219.61 last 177.78) - open/close 43752 (time 1763354208.52 total 229.61 last 197.36) - open/close 45558 (time 1763354218.52 total 239.62 last 180.52) - open/close 47530 (time 1763354228.52 total 249.62 last 197.19) - open/close 49267 (time 1763354238.53 total 259.62 last 173.61) - open/close 50000 (time 1763354242.72 total 263.82 last 174.71) - open/close 51596 (time 1763354252.72 total 273.82 last 159.56) - open/close 53430 (time 1763354262.72 total 283.82 last 183.38) - open/close 55197 (time 1763354272.72 total 293.82 last 176.70) - open/close 56981 (time 1763354282.73 total 303.83 last 178.29) - open/close 58642 (time 1763354292.74 total 313.83 last 166.03) - open/close 60000 (time 1763354300.68 total 321.78 last 170.90) - open/close 61623 (time 1763354310.68 total 331.78 last 162.30) - open/close 63150 (time 1763354320.69 total 341.78 last 152.65) - open/close 64597 (time 1763354330.69 total 351.79 last 144.69) total: 64768 open/close in 352.89 seconds: 183.54 ops/second - /unlink 2568 (time 1763354343.73 total 10.00 last 256.70) - /unlink 5224 (time 1763354353.73 total 20.00 last 265.58) - /unlink 7767 (time 1763354363.74 total 30.01 last 254.26) - /unlink 10000 (time 1763354371.07 total 37.34 last 304.58) - /unlink 13590 (time 1763354381.07 total 47.34 last 358.91) - /unlink 16822 (time 1763354391.07 total 57.34 last 323.18) - /unlink 20000 (time 1763354399.35 total 65.62 last 384.07) - /unlink 24394 (time 1763354409.35 total 75.62 last 439.39) - /unlink 29210 (time 1763354419.35 total 85.62 last 481.55) - /unlink 30000 (time 1763354421.44 total 87.71 last 377.14) - /unlink 33473 (time 1763354431.44 total 97.71 last 347.27) - /unlink 37276 (time 1763354441.44 total 107.71 last 380.27) - /unlink 40000 (time 1763354447.96 total 114.23 last 418.23) - /unlink 43235 (time 1763354457.96 total 124.23 last 323.46) - /unlink 46611 (time 1763354467.96 total 134.23 last 337.44) - /unlink 50000 (time 1763354477.43 total 143.70 last 357.79) - /unlink 54075 (time 1763354487.44 total 153.71 last 407.43) - /unlink 58139 (time 1763354497.44 total 163.71 last 406.32) - /unlink 60000 (time 1763354501.82 total 168.09 last 424.34) - /unlink 63887 (time 1763354511.82 total 178.09 last 388.69) total: 64768 /unlink in 180.58 seconds: 358.66 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2005 (time 1763354545.81 total 10.00 last 200.48) - open/close 3758 (time 1763354555.81 total 20.01 last 175.20) - open/close 5413 (time 1763354565.82 total 30.01 last 165.43) - open/close 6944 (time 1763354575.82 total 40.01 last 153.10) - open/close 8522 (time 1763354585.82 total 50.02 last 157.72) - open/close 10000 (time 1763354594.12 total 58.31 last 178.09) - open/close 11239 (time 1763354604.13 total 68.32 last 123.86) - open/close 12508 (time 1763354614.13 total 78.32 last 126.85) - open/close 14401 (time 1763354624.13 total 88.32 last 189.24) - open/close 16202 (time 1763354634.14 total 98.33 last 179.95) - open/close 17909 (time 1763354644.15 total 108.34 last 170.61) - open/close 19425 (time 1763354654.15 total 118.34 last 151.54) - open/close 20000 (time 1763354657.93 total 122.12 last 152.33) - open/close 21941 (time 1763354667.93 total 132.12 last 194.06) - open/close 23776 (time 1763354677.94 total 142.13 last 183.33) - open/close 25410 (time 1763354687.94 total 152.13 last 163.33) - open/close 27576 (time 1763354697.94 total 162.14 last 216.52) - open/close 29503 (time 1763354707.95 total 172.14 last 192.66) - open/close 30000 (time 1763354710.45 total 174.64 last 198.72) - open/close 32180 (time 1763354720.45 total 184.64 last 217.89) - open/close 34077 (time 1763354730.46 total 194.65 last 189.62) - open/close 35800 (time 1763354740.46 total 204.65 last 172.21) - open/close 37787 (time 1763354750.46 total 214.66 last 198.65) - open/close 39656 (time 1763354760.46 total 224.66 last 186.89) - open/close 41499 (time 1763354770.47 total 234.66 last 184.29) - open/close 43100 (time 1763354780.47 total 244.66 last 160.08) - open/close 45015 (time 1763354790.47 total 254.66 last 191.39) - open/close 47495 (time 1763354800.47 total 264.67 last 247.94) - open/close 50000 (time 1763354809.01 total 273.20 last 293.52) - open/close 52658 (time 1763354819.01 total 283.20 last 265.78) - open/close 55181 (time 1763354829.01 total 293.20 last 252.24) - open/close 57987 (time 1763354839.02 total 303.21 last 280.53) - open/close 60000 (time 1763354845.51 total 309.71 last 309.71) - open/close 63493 (time 1763354855.53 total 319.72 last 348.94) total: 64768 open/close in 324.19 seconds: 199.79 ops/second - /unlink 6134 (time 1763354871.67 total 10.00 last 613.32) - /unlink 10000 (time 1763354878.96 total 17.30 last 530.04) - /unlink 15061 (time 1763354888.96 total 27.30 last 506.06) - /unlink 19908 (time 1763354898.96 total 37.30 last 484.62) - /unlink 23616 (time 1763354908.96 total 47.30 last 370.79) - /unlink 27667 (time 1763354918.97 total 57.30 last 405.00) - /unlink 30000 (time 1763354924.50 total 62.84 last 421.44) - /unlink 35160 (time 1763354934.50 total 72.84 last 515.85) - /unlink 40000 (time 1763354943.65 total 81.99 last 528.99) - /unlink 44397 (time 1763354953.65 total 91.99 last 439.66) - /unlink 48846 (time 1763354963.66 total 101.99 last 444.85) - /unlink 50000 (time 1763354966.53 total 104.86 last 401.48) - /unlink 53820 (time 1763354976.53 total 114.87 last 381.97) - /unlink 58006 (time 1763354986.53 total 124.87 last 418.51) - /unlink 60000 (time 1763354991.61 total 129.94 last 393.08) - /unlink 63917 (time 1763355001.61 total 139.94 last 391.65) total: 64768 /unlink in 142.13 seconds: 455.68 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg120-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing unload_modules_local oleg120-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg120-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2