== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 12:17:53 (1781281073) Stopping clients: oleg606-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg606-client.virtnet /mnt/lustre2 (opts:-f) oleg606-server: oleg606-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg606-server' oleg606-server: oleg606-server.virtnet: executing load_modules_local oleg606-server: Loading modules from /home/green/git/lustre-release/lustre oleg606-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg606-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg606-server: detected 4 online CPUs by sysfs oleg606-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg606-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg606-server: oleg606-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg606-client: oleg606-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg606-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg606-server: oleg606-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg606-client: oleg606-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg606-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg606-server: oleg606-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg606-client: oleg606-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg606-client.virtnet: -o user_xattr,flock oleg606-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16232 424 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16451 643 15808 4% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 731 (time 1781281214.29 total 10.03 last 72.90) - open/close 1476 (time 1781281224.29 total 20.03 last 74.49) - open/close 2080 (time 1781281234.29 total 30.03 last 60.38) - open/close 2789 (time 1781281244.30 total 40.04 last 70.82) - open/close 3497 (time 1781281254.32 total 50.06 last 70.68) - open/close 4247 (time 1781281264.33 total 60.07 last 74.91) - open/close 4987 (time 1781281274.33 total 70.08 last 73.97) - open/close 5656 (time 1781281284.34 total 80.08 last 66.86) - open/close 6417 (time 1781281294.34 total 90.09 last 76.08) - open/close 7207 (time 1781281304.35 total 100.09 last 78.98) - open/close 7959 (time 1781281314.35 total 110.09 last 75.20) - open/close 8609 (time 1781281324.36 total 120.11 last 64.88) - open/close 9368 (time 1781281334.37 total 130.11 last 75.89) - open/close 10000 (time 1781281342.89 total 138.63 last 74.16) - open/close 10708 (time 1781281352.89 total 148.63 last 70.79) - open/close 11435 (time 1781281362.90 total 158.64 last 72.63) - open/close 12169 (time 1781281372.90 total 168.64 last 73.40) - open/close 12964 (time 1781281382.90 total 178.65 last 79.46) - open/close 13732 (time 1781281392.91 total 188.65 last 76.74) - open/close 14474 (time 1781281402.92 total 198.66 last 74.15) - open/close 15208 (time 1781281412.93 total 208.67 last 73.30) - open/close 15976 (time 1781281422.93 total 218.68 last 76.79) - open/close 16690 (time 1781281432.95 total 228.69 last 71.30) - open/close 17413 (time 1781281442.95 total 238.69 last 72.27) - open/close 18157 (time 1781281452.95 total 248.70 last 74.39) - open/close 18864 (time 1781281462.96 total 258.70 last 70.66) - open/close 19606 (time 1781281472.98 total 268.72 last 74.04) - open/close 20000 (time 1781281477.62 total 273.37 last 84.88) - open/close 20771 (time 1781281487.62 total 283.37 last 77.10) - open/close 21532 (time 1781281497.64 total 293.38 last 76.00) - open/close 22285 (time 1781281507.64 total 303.39 last 75.25) - open/close 23100 (time 1781281517.65 total 313.39 last 81.43) - open/close 23882 (time 1781281527.66 total 323.40 last 78.17) - open/close 24766 (time 1781281537.66 total 333.40 last 88.34) - open/close 25554 (time 1781281547.67 total 343.41 last 78.74) - open/close 26325 (time 1781281557.68 total 353.42 last 77.01) - open/close 27141 (time 1781281567.70 total 363.44 last 81.49) - open/close 28012 (time 1781281577.70 total 373.44 last 87.08) - open/close 28847 (time 1781281587.70 total 383.45 last 83.44) - open/close 29724 (time 1781281597.71 total 393.45 last 87.68) - open/close 30000 (time 1781281601.21 total 396.95 last 78.79) - open/close 30776 (time 1781281611.22 total 406.96 last 77.54) - open/close 31592 (time 1781281621.23 total 416.97 last 81.53) - open/close 32336 (time 1781281631.23 total 426.97 last 74.35) - open/close 33142 (time 1781281641.24 total 436.98 last 80.55) - open/close 33892 (time 1781281651.24 total 446.98 last 74.98) - open/close 34680 (time 1781281661.24 total 456.99 last 78.77) - open/close 35438 (time 1781281671.25 total 466.99 last 75.78) - open/close 36218 (time 1781281681.26 total 477.00 last 77.90) - open/close 37051 (time 1781281691.26 total 487.00 last 83.29) - open/close 37848 (time 1781281701.27 total 497.01 last 79.64) - open/close 38621 (time 1781281711.29 total 507.03 last 77.11) - open/close 39368 (time 1781281721.30 total 517.04 last 74.66) - open/close 40000 (time 1781281728.44 total 524.18 last 88.55) - open/close 40829 (time 1781281738.44 total 534.18 last 82.84) - open/close 41616 (time 1781281748.45 total 544.19 last 78.63) - open/close 42421 (time 1781281758.46 total 554.20 last 80.46) - open/close 43213 (time 1781281768.46 total 564.20 last 79.18) - open/close 43991 (time 1781281778.47 total 574.21 last 77.70) - open/close 44805 (time 1781281788.48 total 584.22 last 81.35) - open/close 45583 (time 1781281798.49 total 594.23 last 77.72) - open/close 46393 (time 1781281808.50 total 604.24 last 80.93) - open/close 47241 (time 1781281818.50 total 614.24 last 84.77) - open/close 48142 (time 1781281828.51 total 624.25 last 90.00) - open/close 48990 (time 1781281838.52 total 634.26 last 84.72) - open/close 49803 (time 1781281848.52 total 644.26 last 81.29) - open/close 50000 (time 1781281851.28 total 647.02 last 71.46) - open/close 50812 (time 1781281861.28 total 657.02 last 81.18) - open/close 51709 (time 1781281871.29 total 667.03 last 89.67) - open/close 52476 (time 1781281881.29 total 677.03 last 76.69) - open/close 53330 (time 1781281891.30 total 687.04 last 85.31) - open/close 54227 (time 1781281901.31 total 697.05 last 89.62) - open/close 55207 (time 1781281911.31 total 707.05 last 97.97) - open/close 56199 (time 1781281921.31 total 717.06 last 99.15) - open/close 57017 (time 1781281931.32 total 727.06 last 81.78) - open/close 57912 (time 1781281941.32 total 737.07 last 89.43) - open/close 58680 (time 1781281951.33 total 747.07 last 76.75) - open/close 59602 (time 1781281961.34 total 757.09 last 92.09) - open/close 60000 (time 1781281965.80 total 761.54 last 89.29) - open/close 60813 (time 1781281975.81 total 771.55 last 81.23) - open/close 61665 (time 1781281985.81 total 781.55 last 85.19) - open/close 62614 (time 1781281995.81 total 791.55 last 94.90) - open/close 63497 (time 1781282005.82 total 801.56 last 88.25) - open/close 64412 (time 1781282015.82 total 811.56 last 91.50) total: 64768 open/close in 815.67 seconds: 79.40 ops/second - /unlink 1568 (time 1781282032.99 total 10.00 last 156.72) - /unlink 2916 (time 1781282043.01 total 20.02 last 134.62) - /unlink 4526 (time 1781282053.01 total 30.02 last 160.96) - /unlink 6003 (time 1781282063.02 total 40.03 last 147.61) - /unlink 7578 (time 1781282073.02 total 50.03 last 157.50) - /unlink 9043 (time 1781282083.02 total 60.04 last 146.38) - /unlink 10000 (time 1781282089.63 total 66.64 last 144.81) - /unlink 11437 (time 1781282099.64 total 76.65 last 143.64) - /unlink 13163 (time 1781282109.64 total 86.65 last 172.54) - /unlink 14906 (time 1781282119.64 total 96.65 last 174.27) - /unlink 16278 (time 1781282129.65 total 106.66 last 137.12) - /unlink 17604 (time 1781282139.65 total 116.66 last 132.57) - /unlink 19214 (time 1781282149.65 total 126.66 last 160.99) - /unlink 20000 (time 1781282153.96 total 130.97 last 182.31) - /unlink 21524 (time 1781282163.96 total 140.97 last 152.39) - /unlink 22892 (time 1781282173.97 total 150.98 last 136.70) - /unlink 24111 (time 1781282183.97 total 160.98 last 121.87) - /unlink 25382 (time 1781282193.98 total 170.99 last 127.03) - /unlink 26669 (time 1781282203.98 total 181.00 last 128.63) - /unlink 28288 (time 1781282213.98 total 191.00 last 161.90) - /unlink 29707 (time 1781282224.01 total 201.02 last 141.57) - /unlink 30000 (time 1781282226.08 total 203.10 last 141.09) - /unlink 31547 (time 1781282236.09 total 213.10 last 154.69) - /unlink 32903 (time 1781282246.09 total 223.10 last 135.53) - /unlink 34411 (time 1781282256.10 total 233.11 last 150.73) - /unlink 35917 (time 1781282266.10 total 243.11 last 150.56) - /unlink 37415 (time 1781282276.10 total 253.11 last 149.72) - /unlink 38965 (time 1781282286.10 total 263.12 last 154.98) - /unlink 40000 (time 1781282293.99 total 271.00 last 131.26) - /unlink 41374 (time 1781282304.00 total 281.01 last 137.33) - /unlink 42758 (time 1781282314.00 total 291.01 last 138.34) - /unlink 44062 (time 1781282324.00 total 301.01 last 130.37) - /unlink 45404 (time 1781282334.00 total 311.02 last 134.16) - /unlink 46668 (time 1781282344.01 total 321.02 last 126.36) - /unlink 47916 (time 1781282354.01 total 331.02 last 124.75) - /unlink 49345 (time 1781282364.01 total 341.02 last 142.90) - /unlink 50000 (time 1781282368.53 total 345.54 last 145.06) - /unlink 51556 (time 1781282378.53 total 355.54 last 155.57) - /unlink 53163 (time 1781282388.53 total 365.54 last 160.65) - /unlink 54653 (time 1781282398.54 total 375.55 last 148.92) - /unlink 56212 (time 1781282408.54 total 385.55 last 155.88) - /unlink 57733 (time 1781282418.54 total 395.55 last 152.03) - /unlink 59233 (time 1781282428.56 total 405.57 last 149.78) - /unlink 60000 (time 1781282433.98 total 411.00 last 141.33) - /unlink 61241 (time 1781282443.99 total 421.00 last 124.03) - /unlink 62576 (time 1781282454.00 total 431.01 last 133.42) - /unlink 64036 (time 1781282464.00 total 441.02 last 145.89) total: 64768 /unlink in 446.20 seconds: 145.16 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 816 (time 1781282503.63 total 10.01 last 81.49) - open/close 1631 (time 1781282513.64 total 20.02 last 81.46) - open/close 2472 (time 1781282523.65 total 30.03 last 84.03) - open/close 3290 (time 1781282533.65 total 40.03 last 81.75) - open/close 4216 (time 1781282543.66 total 50.04 last 92.57) - open/close 5033 (time 1781282553.66 total 60.04 last 81.67) - open/close 5838 (time 1781282563.67 total 70.05 last 80.42) - open/close 6619 (time 1781282573.68 total 80.06 last 78.04) - open/close 7390 (time 1781282583.70 total 90.08 last 76.93) - open/close 8169 (time 1781282593.70 total 100.08 last 77.90) - open/close 9030 (time 1781282603.71 total 110.08 last 86.07) - open/close 9926 (time 1781282613.71 total 120.09 last 89.57) - open/close 10889 (time 1781282623.71 total 130.09 last 96.29) - open/close 11797 (time 1781282633.71 total 140.09 last 90.76) - open/close 12513 (time 1781282643.72 total 150.10 last 71.56) - open/close 13325 (time 1781282653.73 total 160.11 last 81.15) - open/close 14085 (time 1781282663.73 total 170.11 last 75.98) - open/close 14910 (time 1781282673.73 total 180.11 last 82.50) - open/close 15740 (time 1781282683.73 total 190.11 last 82.96) - open/close 16595 (time 1781282693.74 total 200.12 last 85.41) - open/close 17381 (time 1781282703.76 total 210.14 last 78.50) - open/close 18178 (time 1781282713.76 total 220.14 last 79.67) - open/close 19042 (time 1781282723.77 total 230.15 last 86.34) - open/close 19876 (time 1781282733.78 total 240.16 last 83.31) - open/close 20722 (time 1781282743.78 total 250.16 last 84.56) - open/close 21556 (time 1781282753.79 total 260.17 last 83.38) - open/close 22360 (time 1781282763.79 total 270.17 last 80.34) - open/close 23161 (time 1781282773.80 total 280.18 last 80.05) - open/close 24063 (time 1781282783.80 total 290.18 last 90.20) - open/close 24970 (time 1781282793.80 total 300.18 last 90.67) - open/close 25830 (time 1781282803.81 total 310.19 last 85.98) - open/close 26575 (time 1781282813.81 total 320.19 last 74.44) - open/close 27331 (time 1781282823.82 total 330.20 last 75.53) - open/close 28058 (time 1781282833.83 total 340.21 last 72.67) - open/close 28833 (time 1781282843.84 total 350.21 last 77.43) - open/close 29571 (time 1781282853.84 total 360.22 last 73.76) - open/close 30000 (time 1781282859.91 total 366.29 last 70.65) - open/close 30810 (time 1781282869.94 total 376.32 last 80.80) - open/close 31575 (time 1781282879.94 total 386.32 last 76.47) - open/close 32354 (time 1781282889.95 total 396.33 last 77.86) - open/close 33101 (time 1781282899.95 total 406.33 last 74.70) - open/close 33880 (time 1781282909.95 total 416.33 last 77.86) - open/close 34638 (time 1781282919.96 total 426.34 last 75.76) - open/close 35392 (time 1781282929.96 total 436.34 last 75.38) - open/close 36203 (time 1781282939.96 total 446.34 last 81.08) - open/close 36996 (time 1781282949.97 total 456.35 last 79.20) - open/close 37858 (time 1781282959.97 total 466.35 last 86.20) - open/close 38725 (time 1781282969.98 total 476.36 last 86.63) - open/close 39612 (time 1781282980.00 total 486.38 last 88.55) - open/close 40000 (time 1781282984.79 total 491.17 last 80.96) - open/close 40861 (time 1781282994.80 total 501.18 last 86.01) - open/close 41734 (time 1781283004.80 total 511.18 last 87.29) - open/close 42534 (time 1781283014.81 total 521.19 last 79.94) - open/close 43328 (time 1781283024.81 total 531.19 last 79.37) - open/close 44115 (time 1781283034.82 total 541.20 last 78.67) - open/close 44907 (time 1781283044.82 total 551.20 last 79.15) - open/close 45732 (time 1781283054.82 total 561.20 last 82.50) - open/close 46534 (time 1781283064.84 total 571.22 last 80.08) - open/close 47313 (time 1781283074.85 total 581.23 last 77.80) - open/close 48190 (time 1781283084.87 total 591.25 last 87.54) - open/close 49003 (time 1781283094.88 total 601.26 last 81.18) - open/close 49930 (time 1781283104.89 total 611.27 last 92.61) - open/close 51005 (time 1781283114.89 total 621.27 last 107.48) - open/close 51823 (time 1781283124.90 total 631.28 last 81.79) - open/close 52711 (time 1781283134.90 total 641.28 last 88.76) - open/close 53626 (time 1781283144.90 total 651.28 last 91.47) - open/close 54645 (time 1781283154.91 total 661.29 last 101.87) - open/close 55524 (time 1781283164.92 total 671.30 last 87.81) - open/close 56348 (time 1781283174.93 total 681.31 last 82.32) - open/close 57163 (time 1781283184.93 total 691.31 last 81.43) - open/close 58001 (time 1781283194.94 total 701.32 last 83.76) - open/close 58742 (time 1781283204.94 total 711.32 last 74.09) - open/close 59520 (time 1781283214.95 total 721.33 last 77.75) - open/close 60000 (time 1781283220.76 total 727.14 last 82.52) - open/close 60797 (time 1781283230.78 total 737.16 last 79.59) - open/close 61594 (time 1781283240.78 total 747.16 last 79.64) - open/close 62402 (time 1781283250.78 total 757.16 last 80.79) - open/close 63258 (time 1781283260.79 total 767.17 last 85.55) - open/close 64144 (time 1781283270.80 total 777.18 last 88.53) total: 64768 open/close in 785.14 seconds: 82.49 ops/second - /unlink 1437 (time 1781283292.26 total 10.00 last 143.65) - /unlink 2828 (time 1781283302.27 total 20.01 last 139.03) - /unlink 4176 (time 1781283312.28 total 30.01 last 134.71) - /unlink 5470 (time 1781283322.28 total 40.02 last 129.33) - /unlink 6946 (time 1781283332.28 total 50.02 last 147.59) - /unlink 8255 (time 1781283342.28 total 60.02 last 130.90) - /unlink 9428 (time 1781283352.28 total 70.02 last 117.28) - /unlink 10000 (time 1781283356.62 total 74.36 last 131.98) - /unlink 11106 (time 1781283366.62 total 84.36 last 110.54) - /unlink 12384 (time 1781283376.63 total 94.37 last 127.70) - /unlink 13723 (time 1781283386.63 total 104.37 last 133.90) - /unlink 15059 (time 1781283396.63 total 114.37 last 133.58) - /unlink 16412 (time 1781283406.66 total 124.40 last 134.95) - /unlink 17738 (time 1781283416.66 total 134.40 last 132.59) - /unlink 19190 (time 1781283426.66 total 144.40 last 145.15) - /unlink 20000 (time 1781283432.42 total 150.16 last 140.57) - /unlink 21320 (time 1781283442.43 total 160.17 last 131.91) - /unlink 22458 (time 1781283452.44 total 170.18 last 113.70) - /unlink 23716 (time 1781283462.44 total 180.18 last 125.75) - /unlink 25124 (time 1781283472.47 total 190.21 last 140.48) - /unlink 26559 (time 1781283482.48 total 200.22 last 143.34) - /unlink 27891 (time 1781283492.49 total 210.22 last 133.11) - /unlink 29131 (time 1781283502.49 total 220.23 last 123.98) - /unlink 30000 (time 1781283509.32 total 227.06 last 127.20) - /unlink 31288 (time 1781283519.32 total 237.06 last 128.72) - /unlink 32491 (time 1781283529.33 total 247.07 last 120.26) - /unlink 33883 (time 1781283539.34 total 257.08 last 139.07) - /unlink 34967 (time 1781283549.34 total 267.08 last 108.36) - /unlink 36117 (time 1781283559.34 total 277.08 last 114.99) - /unlink 37278 (time 1781283569.34 total 287.08 last 116.10) - /unlink 38586 (time 1781283579.35 total 297.09 last 130.71) - /unlink 39979 (time 1781283589.35 total 307.09 last 139.25) - /unlink 41329 (time 1781283599.36 total 317.10 last 134.95) - /unlink 42506 (time 1781283609.36 total 327.10 last 117.64) - /unlink 43764 (time 1781283619.36 total 337.10 last 125.78) - /unlink 45119 (time 1781283629.37 total 347.11 last 135.43) - /unlink 46504 (time 1781283639.37 total 357.11 last 138.49) - /unlink 47921 (time 1781283649.37 total 367.11 last 141.62) - /unlink 49557 (time 1781283659.38 total 377.12 last 163.52) - /unlink 50000 (time 1781283663.09 total 380.83 last 119.23) - /unlink 51384 (time 1781283673.10 total 390.84 last 138.36) - /unlink 52787 (time 1781283683.12 total 400.86 last 140.01) - /unlink 54103 (time 1781283693.13 total 410.87 last 131.50) - /unlink 55409 (time 1781283703.13 total 420.87 last 130.59) - /unlink 56414 (time 1781283713.13 total 430.87 last 100.45) - /unlink 57654 (time 1781283723.13 total 440.87 last 123.96) - /unlink 58866 (time 1781283733.14 total 450.88 last 121.14) - /unlink 60000 (time 1781283742.15 total 459.89 last 125.91) - /unlink 61283 (time 1781283752.15 total 469.89 last 128.25) - /unlink 62769 (time 1781283762.15 total 479.89 last 148.55) - /unlink 64361 (time 1781283772.16 total 489.90 last 159.07) total: 64768 /unlink in 492.74 seconds: 131.45 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 712 (time 1781283808.29 total 10.01 last 71.13) - open/close 1531 (time 1781283818.30 total 20.01 last 81.86) - open/close 2271 (time 1781283828.30 total 30.02 last 74.00) - open/close 3034 (time 1781283838.30 total 40.02 last 76.28) - open/close 3815 (time 1781283848.33 total 50.04 last 77.89) - open/close 4656 (time 1781283858.34 total 60.06 last 84.00) - open/close 5478 (time 1781283868.34 total 70.06 last 82.20) - open/close 6299 (time 1781283878.34 total 80.06 last 82.10) - open/close 7129 (time 1781283888.34 total 90.06 last 83.00) - open/close 7944 (time 1781283898.35 total 100.07 last 81.42) - open/close 8831 (time 1781283908.35 total 110.07 last 88.67) - open/close 9654 (time 1781283918.36 total 120.08 last 82.25) - open/close 10000 (time 1781283922.52 total 124.24 last 83.17) - open/close 10790 (time 1781283932.52 total 134.24 last 79.00) - open/close 11569 (time 1781283942.53 total 144.25 last 77.81) - open/close 12333 (time 1781283952.53 total 154.25 last 76.40) - open/close 13163 (time 1781283962.54 total 164.26 last 82.92) - open/close 13915 (time 1781283972.54 total 174.26 last 75.18) - open/close 14713 (time 1781283982.56 total 184.28 last 79.66) - open/close 15515 (time 1781283992.56 total 194.28 last 80.17) - open/close 16275 (time 1781284002.57 total 204.29 last 75.96) - open/close 17000 (time 1781284012.57 total 214.29 last 72.46) - open/close 17770 (time 1781284022.58 total 224.30 last 76.98) - open/close 18519 (time 1781284032.58 total 234.30 last 74.88) - open/close 19326 (time 1781284042.59 total 244.30 last 80.65) - open/close 20000 (time 1781284051.65 total 253.37 last 74.32) - open/close 20768 (time 1781284061.66 total 263.38 last 76.73) - open/close 21544 (time 1781284071.67 total 273.39 last 77.56) - open/close 22228 (time 1781284081.68 total 283.40 last 68.31) - open/close 22987 (time 1781284091.69 total 293.41 last 75.85) - open/close 23789 (time 1781284101.69 total 303.41 last 80.16) - open/close 24604 (time 1781284111.71 total 313.43 last 81.33) - open/close 25447 (time 1781284121.72 total 323.43 last 84.28) - open/close 26032 (time 1781284131.74 total 333.46 last 58.34) - open/close 26726 (time 1781284141.75 total 343.46 last 69.38) - open/close 27538 (time 1781284151.75 total 353.47 last 81.14) - open/close 28279 (time 1781284161.76 total 363.48 last 74.06) - open/close 29095 (time 1781284171.77 total 373.49 last 81.54) - open/close 29779 (time 1781284181.77 total 383.49 last 68.35) - open/close 30000 (time 1781284185.28 total 387.00 last 63.05) - open/close 30557 (time 1781284195.29 total 397.01 last 55.61) - open/close 31185 (time 1781284205.30 total 407.02 last 62.77) - open/close 31919 (time 1781284215.31 total 417.03 last 73.31) - open/close 32712 (time 1781284225.32 total 427.04 last 79.22) - open/close 33451 (time 1781284235.32 total 437.04 last 73.88) - open/close 34286 (time 1781284245.33 total 447.05 last 83.46) - open/close 35044 (time 1781284255.33 total 457.05 last 75.80) - open/close 35819 (time 1781284265.33 total 467.05 last 77.50) - open/close 36630 (time 1781284275.33 total 477.05 last 81.06) - open/close 37411 (time 1781284285.34 total 487.06 last 78.06) - open/close 38208 (time 1781284295.36 total 497.07 last 79.58) - open/close 38954 (time 1781284305.37 total 507.09 last 74.48) - open/close 39740 (time 1781284315.37 total 517.09 last 78.58) - open/close 40000 (time 1781284318.53 total 520.25 last 82.27) - open/close 40776 (time 1781284328.55 total 530.26 last 77.52) - open/close 41597 (time 1781284338.56 total 540.27 last 82.01) - open/close 42382 (time 1781284348.57 total 550.29 last 78.40) - open/close 43122 (time 1781284358.58 total 560.30 last 73.93) - open/close 43950 (time 1781284368.58 total 570.30 last 82.79) - open/close 44790 (time 1781284378.59 total 580.30 last 83.94) - open/close 45564 (time 1781284388.59 total 590.31 last 77.39) - open/close 46378 (time 1781284398.59 total 600.31 last 81.39) - open/close 47055 (time 1781284408.60 total 610.31 last 67.65) - open/close 47569 (time 1781284418.62 total 620.33 last 51.30) - open/close 48242 (time 1781284428.62 total 630.34 last 67.27) - open/close 49052 (time 1781284438.62 total 640.34 last 80.98) - open/close 49743 (time 1781284448.63 total 650.35 last 69.02) - open/close 50000 (time 1781284451.96 total 653.68 last 77.24) - open/close 50806 (time 1781284461.97 total 663.69 last 80.53) - open/close 51627 (time 1781284471.99 total 673.71 last 81.96) - open/close 52392 (time 1781284481.99 total 683.71 last 76.47) - open/close 53255 (time 1781284492.00 total 693.72 last 86.24) - open/close 54019 (time 1781284502.01 total 703.72 last 76.34) - open/close 54777 (time 1781284512.01 total 713.72 last 75.80) - open/close 55617 (time 1781284522.01 total 723.73 last 83.96) - open/close 56393 (time 1781284532.02 total 733.74 last 77.53) - open/close 57168 (time 1781284542.03 total 743.75 last 77.43) - open/close 57933 (time 1781284552.05 total 753.77 last 76.31) - open/close 58665 (time 1781284562.06 total 763.78 last 73.15) - open/close 59439 (time 1781284572.07 total 773.79 last 77.33) - open/close 60000 (time 1781284579.24 total 780.96 last 78.23) - open/close 60816 (time 1781284589.25 total 790.97 last 81.53) - open/close 61601 (time 1781284599.25 total 800.97 last 78.49) - open/close 62369 (time 1781284609.27 total 810.98 last 76.69) - open/close 63029 (time 1781284619.29 total 821.00 last 65.87) - open/close 63824 (time 1781284629.29 total 831.01 last 79.47) - open/close 64596 (time 1781284639.30 total 841.02 last 77.11) total: 64768 open/close in 843.46 seconds: 76.79 ops/second - /unlink 1223 (time 1781284655.71 total 10.00 last 122.24) - /unlink 2419 (time 1781284665.72 total 20.01 last 119.54) - /unlink 3699 (time 1781284675.72 total 30.01 last 127.98) - /unlink 4963 (time 1781284685.72 total 40.01 last 126.36) - /unlink 6209 (time 1781284695.72 total 50.02 last 124.58) - /unlink 7454 (time 1781284705.73 total 60.02 last 124.40) - /unlink 8735 (time 1781284715.74 total 70.03 last 128.00) - /unlink 9940 (time 1781284725.74 total 80.03 last 120.48) - /unlink 11250 (time 1781284735.74 total 90.04 last 130.96) - /unlink 12565 (time 1781284745.76 total 100.05 last 131.30) - /unlink 13858 (time 1781284755.76 total 110.06 last 129.24) - /unlink 15198 (time 1781284765.77 total 120.06 last 133.98) - /unlink 16523 (time 1781284775.77 total 130.06 last 132.44) - /unlink 17808 (time 1781284785.78 total 140.07 last 128.42) - /unlink 19020 (time 1781284795.78 total 150.07 last 121.17) - /unlink 20000 (time 1781284804.59 total 158.88 last 111.24) - /unlink 21348 (time 1781284814.59 total 168.89 last 134.73) - /unlink 22639 (time 1781284824.60 total 178.89 last 129.05) - /unlink 23893 (time 1781284834.60 total 188.90 last 125.32) - /unlink 25164 (time 1781284844.61 total 198.90 last 127.04) - /unlink 26348 (time 1781284854.62 total 208.91 last 118.27) - /unlink 27551 (time 1781284864.62 total 218.92 last 120.23) - /unlink 28746 (time 1781284874.63 total 228.92 last 119.49) - /unlink 29909 (time 1781284884.64 total 238.93 last 116.13) - /unlink 31110 (time 1781284894.64 total 248.93 last 120.09) - /unlink 32340 (time 1781284904.64 total 258.94 last 122.98) - /unlink 33538 (time 1781284914.66 total 268.95 last 119.58) - /unlink 34802 (time 1781284924.66 total 278.95 last 126.40) - /unlink 36010 (time 1781284934.66 total 288.96 last 120.78) - /unlink 37306 (time 1781284944.67 total 298.96 last 129.52) - /unlink 38646 (time 1781284954.67 total 308.96 last 133.99) - /unlink 39937 (time 1781284964.67 total 318.96 last 129.06) - /unlink 41268 (time 1781284974.67 total 328.97 last 133.08) - /unlink 42348 (time 1781284984.68 total 338.97 last 107.99) - /unlink 43525 (time 1781284994.68 total 348.97 last 117.65) - /unlink 44905 (time 1781285004.68 total 358.98 last 137.94) - /unlink 46090 (time 1781285014.69 total 368.98 last 118.46) - /unlink 47169 (time 1781285024.69 total 378.98 last 107.89) - /unlink 48617 (time 1781285034.69 total 388.99 last 144.72) - /unlink 49864 (time 1781285044.70 total 398.99 last 124.67) - /unlink 51161 (time 1781285054.70 total 408.99 last 129.63) - /unlink 52419 (time 1781285064.71 total 419.00 last 125.75) - /unlink 53654 (time 1781285074.71 total 429.00 last 123.46) - /unlink 54968 (time 1781285084.71 total 439.00 last 131.37) - /unlink 56234 (time 1781285094.71 total 449.01 last 126.57) - /unlink 57426 (time 1781285104.74 total 459.03 last 118.85) - /unlink 58685 (time 1781285114.79 total 469.08 last 125.34) - /unlink 59881 (time 1781285124.79 total 479.08 last 119.53) - /unlink 61192 (time 1781285134.79 total 489.09 last 131.09) - /unlink 62523 (time 1781285144.80 total 499.09 last 133.06) - /unlink 63997 (time 1781285154.80 total 509.09 last 147.35) total: 64768 /unlink in 514.06 seconds: 125.99 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 891 (time 1781285197.46 total 10.00 last 89.09) - open/close 1762 (time 1781285207.47 total 20.01 last 87.00) - open/close 2565 (time 1781285217.48 total 30.01 last 80.28) - open/close 3234 (time 1781285227.49 total 40.03 last 66.82) - open/close 3991 (time 1781285237.49 total 50.03 last 75.67) - open/close 4790 (time 1781285247.50 total 60.03 last 79.88) - open/close 5536 (time 1781285257.51 total 70.04 last 74.52) - open/close 6362 (time 1781285267.51 total 80.05 last 82.58) - open/close 7179 (time 1781285277.51 total 90.05 last 81.69) - open/close 7999 (time 1781285287.51 total 100.05 last 81.98) - open/close 8834 (time 1781285297.51 total 110.05 last 83.47) - open/close 9636 (time 1781285307.52 total 120.06 last 80.17) - open/close 10000 (time 1781285311.81 total 124.35 last 84.87) - open/close 10838 (time 1781285321.81 total 134.35 last 83.76) - open/close 11661 (time 1781285331.82 total 144.36 last 82.24) - open/close 12486 (time 1781285341.83 total 154.36 last 82.46) - open/close 13220 (time 1781285351.83 total 164.37 last 73.38) - open/close 13998 (time 1781285361.83 total 174.37 last 77.77) - open/close 14811 (time 1781285371.83 total 184.37 last 81.29) - open/close 15588 (time 1781285381.83 total 194.37 last 77.69) - open/close 16411 (time 1781285391.85 total 204.38 last 82.20) - open/close 17185 (time 1781285401.85 total 214.39 last 77.35) - open/close 17909 (time 1781285411.87 total 224.41 last 72.28) - open/close 18723 (time 1781285421.88 total 234.41 last 81.36) - open/close 19530 (time 1781285431.89 total 244.43 last 80.54) - open/close 20000 (time 1781285437.73 total 250.27 last 80.56) - open/close 20774 (time 1781285447.73 total 260.27 last 77.40) - open/close 21515 (time 1781285457.73 total 270.27 last 74.06) - open/close 22217 (time 1781285467.75 total 280.29 last 70.10) - open/close 22969 (time 1781285477.76 total 290.29 last 75.14) - open/close 23701 (time 1781285487.76 total 300.30 last 73.17) - open/close 24490 (time 1781285497.76 total 310.30 last 78.87) - open/close 25257 (time 1781285507.77 total 320.31 last 76.66) - open/close 26038 (time 1781285517.78 total 330.32 last 78.01) - open/close 26741 (time 1781285527.79 total 340.32 last 70.25) - open/close 27523 (time 1781285537.79 total 350.33 last 78.16) - open/close 28313 (time 1781285547.80 total 360.34 last 78.95) - open/close 29099 (time 1781285557.81 total 370.35 last 78.52) - open/close 29839 (time 1781285567.85 total 380.39 last 73.70) - open/close 30000 (time 1781285569.85 total 382.39 last 80.47) - open/close 30690 (time 1781285579.85 total 392.39 last 68.98) - open/close 31473 (time 1781285589.87 total 402.41 last 78.18) - open/close 32174 (time 1781285599.87 total 412.41 last 70.09) - open/close 32901 (time 1781285609.87 total 422.41 last 72.68) - open/close 33591 (time 1781285619.88 total 432.42 last 68.92) - open/close 34323 (time 1781285629.89 total 442.43 last 73.15) - open/close 35075 (time 1781285639.90 total 452.44 last 75.14) - open/close 35810 (time 1781285649.91 total 462.45 last 73.41) - open/close 36559 (time 1781285659.93 total 472.46 last 74.79) - open/close 37312 (time 1781285669.94 total 482.48 last 75.21) - open/close 38094 (time 1781285679.95 total 492.49 last 78.12) - open/close 38799 (time 1781285689.95 total 502.49 last 70.46) - open/close 39409 (time 1781285700.01 total 512.55 last 60.66) - open/close 40000 (time 1781285707.57 total 520.11 last 78.19) - open/close 40682 (time 1781285717.57 total 530.11 last 68.19) - open/close 41427 (time 1781285727.58 total 540.11 last 74.45) - open/close 42214 (time 1781285737.58 total 550.12 last 78.67) - open/close 43000 (time 1781285747.58 total 560.12 last 78.57) - open/close 43815 (time 1781285757.59 total 570.13 last 81.44) - open/close 44676 (time 1781285767.62 total 580.15 last 85.89) - open/close 45451 (time 1781285777.62 total 590.16 last 77.47) - open/close 46221 (time 1781285787.62 total 600.16 last 76.98) - open/close 46981 (time 1781285797.63 total 610.17 last 75.93) - open/close 47697 (time 1781285807.65 total 620.19 last 71.47) - open/close 48407 (time 1781285817.66 total 630.20 last 70.90) - open/close 49187 (time 1781285827.67 total 640.21 last 77.97) - open/close 50000 (time 1781285837.51 total 650.05 last 82.59) - open/close 50816 (time 1781285847.51 total 660.05 last 81.57) - open/close 51589 (time 1781285857.52 total 670.06 last 77.25) - open/close 52359 (time 1781285867.53 total 680.07 last 76.93) - open/close 53234 (time 1781285877.53 total 690.07 last 87.46) - open/close 54055 (time 1781285887.54 total 700.07 last 82.09) - open/close 54857 (time 1781285897.54 total 710.07 last 80.19) - open/close 55587 (time 1781285907.55 total 720.09 last 72.91) - open/close 56410 (time 1781285917.55 total 730.09 last 82.29) - open/close 57158 (time 1781285927.56 total 740.09 last 74.77) - open/close 57987 (time 1781285937.58 total 750.12 last 82.72) - open/close 58766 (time 1781285947.58 total 760.12 last 77.86) - open/close 59531 (time 1781285957.59 total 770.13 last 76.44) - open/close 60000 (time 1781285963.87 total 776.40 last 74.74) - open/close 60808 (time 1781285973.87 total 786.40 last 80.78) - open/close 61628 (time 1781285983.87 total 796.41 last 81.97) - open/close 62426 (time 1781285993.88 total 806.41 last 79.75) - open/close 63219 (time 1781286003.88 total 816.42 last 79.27) - open/close 64079 (time 1781286013.88 total 826.42 last 85.98) total: 64768 open/close in 834.63 seconds: 77.60 ops/second - /unlink 1485 (time 1781286034.94 total 10.01 last 148.37) - /unlink 2971 (time 1781286044.94 total 20.01 last 148.60) - /unlink 4293 (time 1781286054.95 total 30.01 last 132.12) - /unlink 5638 (time 1781286064.96 total 40.02 last 134.38) - /unlink 6872 (time 1781286074.96 total 50.02 last 123.39) - /unlink 8167 (time 1781286084.97 total 60.04 last 129.34) - /unlink 9396 (time 1781286094.98 total 70.04 last 122.84) - /unlink 10000 (time 1781286100.14 total 75.20 last 117.08) - /unlink 11225 (time 1781286110.14 total 85.20 last 122.48) - /unlink 12519 (time 1781286120.14 total 95.21 last 129.32) - /unlink 13770 (time 1781286130.15 total 105.21 last 125.07) - /unlink 15013 (time 1781286140.15 total 115.22 last 124.22) - /unlink 16339 (time 1781286150.16 total 125.22 last 132.51) - /unlink 17550 (time 1781286160.16 total 135.23 last 121.07) - /unlink 18842 (time 1781286170.17 total 145.24 last 129.07) - /unlink 20000 (time 1781286179.35 total 154.42 last 126.16) - /unlink 21268 (time 1781286189.35 total 164.42 last 126.79) - /unlink 22550 (time 1781286199.35 total 174.42 last 128.17) - /unlink 23862 (time 1781286209.36 total 184.42 last 131.11) - /unlink 25134 (time 1781286219.36 total 194.43 last 127.16) - /unlink 26351 (time 1781286229.37 total 204.43 last 121.67) - /unlink 27519 (time 1781286239.37 total 214.44 last 116.72) - /unlink 28697 (time 1781286249.39 total 224.45 last 117.63) - /unlink 29833 (time 1781286259.40 total 234.46 last 113.48) - /unlink 31042 (time 1781286269.42 total 244.48 last 120.68) - /unlink 32316 (time 1781286279.42 total 254.49 last 127.31) - /unlink 33617 (time 1781286289.43 total 264.49 last 130.07) - /unlink 34941 (time 1781286299.43 total 274.50 last 132.29) - /unlink 36240 (time 1781286309.44 total 284.50 last 129.85) - /unlink 37446 (time 1781286319.44 total 294.50 last 120.57) - /unlink 38734 (time 1781286329.45 total 304.51 last 128.71) - /unlink 40000 (time 1781286338.96 total 314.03 last 133.02) - /unlink 41326 (time 1781286348.97 total 324.03 last 132.52) - /unlink 42681 (time 1781286358.97 total 334.04 last 135.48) - /unlink 44029 (time 1781286368.98 total 344.04 last 134.74) - /unlink 45349 (time 1781286378.98 total 354.04 last 131.99) - /unlink 46504 (time 1781286388.98 total 364.05 last 115.44) - /unlink 47686 (time 1781286398.98 total 374.05 last 118.20) - /unlink 48812 (time 1781286408.99 total 384.05 last 112.54) - /unlink 49914 (time 1781286418.99 total 394.06 last 110.14) - /unlink 51218 (time 1781286429.00 total 404.06 last 130.37) - /unlink 52555 (time 1781286439.00 total 414.06 last 133.66) - /unlink 53804 (time 1781286449.00 total 424.07 last 124.83) - /unlink 54974 (time 1781286459.02 total 434.08 last 116.84) - /unlink 56303 (time 1781286469.02 total 444.08 last 132.87) - /unlink 57754 (time 1781286479.02 total 454.09 last 145.07) - /unlink 58952 (time 1781286489.03 total 464.10 last 119.68) - /unlink 60000 (time 1781286497.59 total 472.65 last 122.48) - /unlink 61222 (time 1781286507.59 total 482.66 last 122.17) - /unlink 62346 (time 1781286517.59 total 492.66 last 112.38) - /unlink 63649 (time 1781286527.60 total 502.66 last 130.24) total: 64768 /unlink in 511.26 seconds: 126.68 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 764 (time 1781286574.50 total 10.01 last 76.34) - open/close 1603 (time 1781286584.50 total 20.01 last 83.87) - open/close 2426 (time 1781286594.50 total 30.02 last 82.26) - open/close 3197 (time 1781286604.51 total 40.02 last 77.08) - open/close 4019 (time 1781286614.53 total 50.04 last 82.03) - open/close 4846 (time 1781286624.54 total 60.05 last 82.59) - open/close 5632 (time 1781286634.55 total 70.06 last 78.53) - open/close 6411 (time 1781286644.55 total 80.06 last 77.88) - open/close 7105 (time 1781286654.55 total 90.07 last 69.39) - open/close 7726 (time 1781286664.56 total 100.07 last 62.06) - open/close 8226 (time 1781286674.61 total 110.12 last 49.75) - open/close 8888 (time 1781286684.63 total 120.14 last 66.09) - open/close 9646 (time 1781286694.63 total 130.14 last 75.77) - open/close 10000 (time 1781286699.18 total 134.69 last 77.86) - open/close 10783 (time 1781286709.18 total 144.70 last 78.25) - open/close 11562 (time 1781286719.19 total 154.70 last 77.86) - open/close 12412 (time 1781286729.19 total 164.70 last 84.97) - open/close 13181 (time 1781286739.20 total 174.71 last 76.88) - open/close 13935 (time 1781286749.20 total 184.71 last 75.39) - open/close 14714 (time 1781286759.20 total 194.71 last 77.86) - open/close 15515 (time 1781286769.21 total 204.72 last 80.02) - open/close 16267 (time 1781286779.21 total 214.72 last 75.20) - open/close 16944 (time 1781286789.24 total 224.75 last 67.52) - open/close 17639 (time 1781286799.25 total 234.76 last 69.41) - open/close 18441 (time 1781286809.26 total 244.77 last 80.16) - open/close 19220 (time 1781286819.26 total 254.78 last 77.84) - open/close 19973 (time 1781286829.27 total 264.78 last 75.25) - open/close 20720 (time 1781286839.28 total 274.79 last 74.64) - open/close 21482 (time 1781286849.28 total 284.79 last 76.18) - open/close 22228 (time 1781286859.29 total 294.80 last 74.56) - open/close 22991 (time 1781286869.31 total 304.82 last 76.16) - open/close 23755 (time 1781286879.33 total 314.84 last 76.24) - open/close 24501 (time 1781286889.34 total 324.85 last 74.53) - open/close 25276 (time 1781286899.36 total 334.87 last 77.34) - open/close 26097 (time 1781286909.36 total 344.87 last 82.08) - open/close 26862 (time 1781286919.37 total 354.88 last 76.45) - open/close 27565 (time 1781286929.37 total 364.88 last 70.26) - open/close 28283 (time 1781286939.37 total 374.89 last 71.78) - open/close 29063 (time 1781286949.38 total 384.89 last 77.99) - open/close 29838 (time 1781286959.39 total 394.90 last 77.40) - open/close 30000 (time 1781286961.44 total 396.96 last 78.78) - open/close 30791 (time 1781286971.46 total 406.97 last 78.97) - open/close 31495 (time 1781286981.47 total 416.99 last 70.31) - open/close 32257 (time 1781286991.48 total 426.99 last 76.13) - open/close 33054 (time 1781287001.49 total 437.00 last 79.64) - open/close 33844 (time 1781287011.50 total 447.01 last 78.92) - open/close 34662 (time 1781287021.51 total 457.03 last 81.70) - open/close 35497 (time 1781287031.52 total 467.03 last 83.45) - open/close 36191 (time 1781287041.53 total 477.04 last 69.31) - open/close 36932 (time 1781287051.54 total 487.05 last 74.06) - open/close 37756 (time 1781287061.55 total 497.06 last 82.33) - open/close 38525 (time 1781287071.55 total 507.06 last 76.89) - open/close 39283 (time 1781287081.56 total 517.07 last 75.71) - open/close 40000 (time 1781287091.44 total 526.96 last 72.53) - open/close 40815 (time 1781287101.45 total 536.96 last 81.44) - open/close 41642 (time 1781287111.45 total 546.97 last 82.67) - open/close 42406 (time 1781287121.47 total 556.98 last 76.32) - open/close 43211 (time 1781287131.47 total 566.99 last 80.44) - open/close 43987 (time 1781287141.49 total 577.00 last 77.50) - open/close 44723 (time 1781287151.49 total 587.00 last 73.58) - open/close 45482 (time 1781287161.49 total 597.00 last 75.88) - open/close 46114 (time 1781287171.50 total 607.01 last 63.14) - open/close 46757 (time 1781287181.51 total 617.03 last 64.23) - open/close 47393 (time 1781287191.51 total 627.03 last 63.59) - open/close 48090 (time 1781287201.53 total 637.04 last 69.61) - open/close 48779 (time 1781287211.53 total 647.05 last 68.86) - open/close 49520 (time 1781287221.53 total 657.05 last 74.10) - open/close 50000 (time 1781287228.68 total 664.19 last 67.19) - open/close 50738 (time 1781287238.69 total 674.20 last 73.74) - open/close 51477 (time 1781287248.69 total 684.20 last 73.88) - open/close 52211 (time 1781287258.72 total 694.23 last 73.17) - open/close 52874 (time 1781287268.74 total 704.25 last 66.16) - open/close 53543 (time 1781287278.77 total 714.28 last 66.71) - open/close 54215 (time 1781287288.78 total 724.29 last 67.17) - open/close 54855 (time 1781287298.78 total 734.29 last 63.97) - open/close 55543 (time 1781287308.79 total 744.30 last 68.73) - open/close 56265 (time 1781287318.82 total 754.33 last 71.97) - open/close 56929 (time 1781287328.82 total 764.34 last 66.38) - open/close 57587 (time 1781287338.88 total 774.39 last 65.46) - open/close 58300 (time 1781287348.88 total 784.39 last 71.28) - open/close 59003 (time 1781287358.89 total 794.40 last 70.23) - open/close 59699 (time 1781287368.91 total 804.42 last 69.48) - open/close 60000 (time 1781287373.51 total 809.02 last 65.38) - open/close 60662 (time 1781287383.51 total 819.03 last 66.18) - open/close 61391 (time 1781287393.52 total 829.04 last 72.82) - open/close 62067 (time 1781287403.52 total 839.04 last 67.59) - open/close 62800 (time 1781287413.54 total 849.05 last 73.21) - open/close 63546 (time 1781287423.55 total 859.06 last 74.54) - open/close 64252 (time 1781287433.56 total 869.07 last 70.48) total: 64768 open/close in 875.47 seconds: 73.98 ops/second - /unlink 1155 (time 1781287454.95 total 10.00 last 115.48) - /unlink 2112 (time 1781287464.95 total 20.00 last 95.69) - /unlink 3318 (time 1781287474.95 total 30.00 last 120.59) - /unlink 4537 (time 1781287484.96 total 40.00 last 121.89) - /unlink 5719 (time 1781287494.96 total 50.00 last 118.20) - /unlink 6835 (time 1781287504.96 total 60.01 last 111.58) - /unlink 8058 (time 1781287514.96 total 70.01 last 122.23) - /unlink 9322 (time 1781287524.98 total 80.03 last 126.21) - /unlink 10000 (time 1781287530.66 total 85.71 last 119.35) - /unlink 11197 (time 1781287540.67 total 95.72 last 119.62) - /unlink 12381 (time 1781287550.67 total 105.72 last 118.35) - /unlink 13552 (time 1781287560.67 total 115.72 last 117.06) - /unlink 14627 (time 1781287570.67 total 125.72 last 107.49) - /unlink 15739 (time 1781287580.68 total 135.73 last 111.12) - /unlink 16983 (time 1781287590.69 total 145.73 last 124.35) - /unlink 18228 (time 1781287600.69 total 155.74 last 124.43) - /unlink 19387 (time 1781287610.70 total 165.75 last 115.82) - /unlink 20000 (time 1781287616.30 total 171.35 last 109.42) - /unlink 21138 (time 1781287626.33 total 181.38 last 113.48) - /unlink 22205 (time 1781287636.37 total 191.42 last 106.29) - /unlink 23365 (time 1781287646.37 total 201.42 last 115.93) - /unlink 24520 (time 1781287656.38 total 211.43 last 115.42) - /unlink 25745 (time 1781287666.38 total 221.43 last 122.45) - /unlink 26901 (time 1781287676.39 total 231.44 last 115.57) - /unlink 28150 (time 1781287686.39 total 241.44 last 124.90) - /unlink 29334 (time 1781287696.39 total 251.44 last 118.35) - /unlink 30000 (time 1781287701.78 total 256.83 last 123.51) - /unlink 31187 (time 1781287711.79 total 266.84 last 118.63) - /unlink 32446 (time 1781287721.79 total 276.84 last 125.85) - /unlink 33677 (time 1781287731.80 total 286.85 last 123.04) - /unlink 34644 (time 1781287741.80 total 296.85 last 96.68) - /unlink 35507 (time 1781287751.81 total 306.86 last 86.20) - /unlink 36456 (time 1781287761.81 total 316.86 last 94.89) - /unlink 37461 (time 1781287771.82 total 326.86 last 100.48) - /unlink 38472 (time 1781287781.82 total 336.87 last 101.09) - /unlink 39456 (time 1781287791.82 total 346.87 last 98.32) - /unlink 40000 (time 1781287797.25 total 352.30 last 100.22) - /unlink 41108 (time 1781287807.25 total 362.30 last 110.78) - /unlink 42273 (time 1781287817.25 total 372.30 last 116.49) - /unlink 43397 (time 1781287827.26 total 382.31 last 112.33) - /unlink 44507 (time 1781287837.26 total 392.31 last 110.96) - /unlink 45591 (time 1781287847.26 total 402.31 last 108.40) - /unlink 46677 (time 1781287857.27 total 412.32 last 108.53) - /unlink 47646 (time 1781287867.29 total 422.34 last 96.73) - /unlink 48906 (time 1781287877.30 total 432.35 last 125.88) - /unlink 50000 (time 1781287887.07 total 442.11 last 112.01) - /unlink 51191 (time 1781287897.07 total 452.12 last 119.09) - /unlink 52400 (time 1781287907.07 total 462.12 last 120.85) - /unlink 53472 (time 1781287917.07 total 472.12 last 107.15) - /unlink 54730 (time 1781287927.08 total 482.13 last 125.72) - /unlink 56020 (time 1781287937.08 total 492.13 last 128.95) - /unlink 57372 (time 1781287947.08 total 502.13 last 135.20) - /unlink 58539 (time 1781287957.09 total 512.14 last 116.64) - /unlink 59612 (time 1781287967.09 total 522.14 last 107.25) - /unlink 60000 (time 1781287971.09 total 526.14 last 97.12) - /unlink 61131 (time 1781287981.09 total 536.14 last 113.08) - /unlink 62349 (time 1781287991.09 total 546.14 last 121.77) - /unlink 63619 (time 1781288001.10 total 556.15 last 126.92) total: 64768 /unlink in 564.07 seconds: 114.82 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg606-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg606-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg606-server stop mds service on oleg606-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg606-server stop mds service on oleg606-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg606-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg606-server' oleg606-server: oleg606-server.virtnet: executing unload_modules_local oleg606-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg606-server: LNET ready to unload oleg606-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg606-client: oleg606-client: ssh exited with exit code 2 pdsh@oleg606-client: oleg606-server: ssh exited with exit code 2 pdsh@oleg606-client: oleg606-client: ssh exited with exit code 2 pdsh@oleg606-client: oleg606-server: ssh exited with exit code 2