== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 12:17:58 (1781281078) Stopping clients: oleg630-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg630-client.virtnet /mnt/lustre2 (opts:-f) oleg630-server: oleg630-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg630-server' oleg630-server: oleg630-server.virtnet: executing load_modules_local oleg630-server: Loading modules from /home/green/git/lustre-release/lustre oleg630-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg630-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg630-server: detected 4 online CPUs by sysfs oleg630-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg630-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg630-server: oleg630-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg630-client: oleg630-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg630-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg630-server: oleg630-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg630-client: oleg630-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg630-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg630-server: oleg630-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg630-client: oleg630-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg630-client.virtnet: -o user_xattr,flock oleg630-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16232 424 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16451 643 15808 4% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 750 (time 1781281220.08 total 10.01 last 74.94) - open/close 1505 (time 1781281230.10 total 20.02 last 75.39) - open/close 2282 (time 1781281240.11 total 30.03 last 77.62) - open/close 3097 (time 1781281250.12 total 40.05 last 81.39) - open/close 3942 (time 1781281260.12 total 50.05 last 84.49) - open/close 4537 (time 1781281270.13 total 60.06 last 59.46) - open/close 5304 (time 1781281280.15 total 70.08 last 76.54) - open/close 6106 (time 1781281290.16 total 80.08 last 80.15) - open/close 6888 (time 1781281300.17 total 90.09 last 78.14) - open/close 7625 (time 1781281310.18 total 100.11 last 73.57) - open/close 8382 (time 1781281320.19 total 110.11 last 75.68) - open/close 9134 (time 1781281330.19 total 120.12 last 75.16) - open/close 9901 (time 1781281340.19 total 130.12 last 76.69) - open/close 10672 (time 1781281350.23 total 140.16 last 76.79) - open/close 11412 (time 1781281360.25 total 150.17 last 73.90) - open/close 12145 (time 1781281370.25 total 160.18 last 73.28) - open/close 12942 (time 1781281380.26 total 170.18 last 79.66) - open/close 13776 (time 1781281390.26 total 180.18 last 83.39) - open/close 14515 (time 1781281400.27 total 190.20 last 73.80) - open/close 15282 (time 1781281410.28 total 200.21 last 76.62) - open/close 16101 (time 1781281420.28 total 210.21 last 81.89) - open/close 16848 (time 1781281430.28 total 220.21 last 74.69) - open/close 17628 (time 1781281440.29 total 230.22 last 77.92) - open/close 18433 (time 1781281450.30 total 240.22 last 80.50) - open/close 19179 (time 1781281460.33 total 250.25 last 74.37) - open/close 19959 (time 1781281470.33 total 260.26 last 77.98) - open/close 20745 (time 1781281480.33 total 270.26 last 78.57) - open/close 21560 (time 1781281490.33 total 280.26 last 81.48) - open/close 22232 (time 1781281500.34 total 290.26 last 67.19) - open/close 23043 (time 1781281510.35 total 300.27 last 81.03) - open/close 23787 (time 1781281520.36 total 310.28 last 74.32) - open/close 24522 (time 1781281530.36 total 320.28 last 73.48) - open/close 25409 (time 1781281540.37 total 330.29 last 88.62) - open/close 26203 (time 1781281550.37 total 340.30 last 79.37) - open/close 26947 (time 1781281560.38 total 350.31 last 74.33) - open/close 27762 (time 1781281570.40 total 360.32 last 81.38) - open/close 28645 (time 1781281580.40 total 370.32 last 88.30) - open/close 29461 (time 1781281590.41 total 380.33 last 81.52) - open/close 30000 (time 1781281596.91 total 386.83 last 82.90) - open/close 30795 (time 1781281606.92 total 396.85 last 79.36) - open/close 31516 (time 1781281616.93 total 406.85 last 72.10) - open/close 32283 (time 1781281626.93 total 416.86 last 76.65) - open/close 33012 (time 1781281636.96 total 426.88 last 72.72) - open/close 33756 (time 1781281646.96 total 436.89 last 74.34) - open/close 34553 (time 1781281656.97 total 446.89 last 79.67) - open/close 35236 (time 1781281666.97 total 456.89 last 68.30) - open/close 36015 (time 1781281676.97 total 466.89 last 77.90) - open/close 36757 (time 1781281686.98 total 476.90 last 74.15) - open/close 37570 (time 1781281696.98 total 486.90 last 81.30) - open/close 38345 (time 1781281706.98 total 496.90 last 77.50) - open/close 39107 (time 1781281716.98 total 506.91 last 76.16) - open/close 39950 (time 1781281726.98 total 516.91 last 84.27) - open/close 40765 (time 1781281736.99 total 526.92 last 81.43) - open/close 41562 (time 1781281747.00 total 536.93 last 79.62) - open/close 42325 (time 1781281757.01 total 546.93 last 76.26) - open/close 43081 (time 1781281767.02 total 556.94 last 75.53) - open/close 43894 (time 1781281777.02 total 566.95 last 81.24) - open/close 44656 (time 1781281787.03 total 576.96 last 76.16) - open/close 45439 (time 1781281797.04 total 586.96 last 78.25) - open/close 46270 (time 1781281807.05 total 596.97 last 83.02) - open/close 47117 (time 1781281817.05 total 606.98 last 84.67) - open/close 48009 (time 1781281827.05 total 616.98 last 89.18) - open/close 48878 (time 1781281837.06 total 626.98 last 86.86) - open/close 49699 (time 1781281847.06 total 636.98 last 82.10) - open/close 50000 (time 1781281850.74 total 640.66 last 81.76) - open/close 50768 (time 1781281860.74 total 650.67 last 76.77) - open/close 51676 (time 1781281870.75 total 660.67 last 90.73) - open/close 52466 (time 1781281880.75 total 670.68 last 79.00) - open/close 53311 (time 1781281890.76 total 680.68 last 84.43) - open/close 54140 (time 1781281900.76 total 690.69 last 82.88) - open/close 55127 (time 1781281910.77 total 700.70 last 98.59) - open/close 56086 (time 1781281920.77 total 710.70 last 95.90) - open/close 56925 (time 1781281930.77 total 720.70 last 83.88) - open/close 57859 (time 1781281940.78 total 730.71 last 93.34) - open/close 58685 (time 1781281950.79 total 740.71 last 82.56) - open/close 59608 (time 1781281960.79 total 750.71 last 92.29) - open/close 60000 (time 1781281965.27 total 755.20 last 87.36) - open/close 60810 (time 1781281975.29 total 765.22 last 80.88) - open/close 61677 (time 1781281985.29 total 775.22 last 86.65) - open/close 62643 (time 1781281995.31 total 785.24 last 96.43) - open/close 63511 (time 1781282005.32 total 795.24 last 86.75) - open/close 64410 (time 1781282015.33 total 805.26 last 89.79) total: 64768 open/close in 809.36 seconds: 80.02 ops/second - /unlink 1530 (time 1781282032.66 total 10.00 last 152.93) - /unlink 2870 (time 1781282042.66 total 20.01 last 133.96) - /unlink 4513 (time 1781282052.67 total 30.01 last 164.20) - /unlink 5943 (time 1781282062.67 total 40.02 last 142.94) - /unlink 7546 (time 1781282072.68 total 50.02 last 160.27) - /unlink 9013 (time 1781282082.68 total 60.03 last 146.57) - /unlink 10000 (time 1781282089.77 total 67.12 last 139.21) - /unlink 11464 (time 1781282099.78 total 77.12 last 146.36) - /unlink 13236 (time 1781282109.78 total 87.12 last 177.18) - /unlink 14964 (time 1781282119.78 total 97.12 last 172.77) - /unlink 16308 (time 1781282129.78 total 107.13 last 134.35) - /unlink 17608 (time 1781282139.78 total 117.13 last 130.00) - /unlink 19199 (time 1781282149.79 total 127.13 last 159.08) - /unlink 20000 (time 1781282154.02 total 131.37 last 189.11) - /unlink 21476 (time 1781282164.03 total 141.37 last 147.48) - /unlink 22855 (time 1781282174.04 total 151.38 last 137.83) - /unlink 24153 (time 1781282184.04 total 161.38 last 129.75) - /unlink 25442 (time 1781282194.05 total 171.39 last 128.80) - /unlink 26689 (time 1781282204.05 total 181.40 last 124.62) - /unlink 28291 (time 1781282214.06 total 191.41 last 160.06) - /unlink 29691 (time 1781282224.07 total 201.41 last 139.92) - /unlink 30000 (time 1781282226.31 total 203.65 last 138.16) - /unlink 31511 (time 1781282236.34 total 213.68 last 150.58) - /unlink 32931 (time 1781282246.35 total 223.69 last 141.89) - /unlink 34464 (time 1781282256.35 total 233.70 last 153.25) - /unlink 35939 (time 1781282266.35 total 243.70 last 147.45) - /unlink 37394 (time 1781282276.36 total 253.71 last 145.39) - /unlink 38939 (time 1781282286.37 total 263.71 last 154.44) - /unlink 40000 (time 1781282294.15 total 271.49 last 136.30) - /unlink 41362 (time 1781282304.15 total 281.50 last 136.18) - /unlink 42749 (time 1781282314.15 total 291.50 last 138.68) - /unlink 44022 (time 1781282324.16 total 301.50 last 127.24) - /unlink 45370 (time 1781282334.16 total 311.50 last 134.77) - /unlink 46663 (time 1781282344.17 total 321.51 last 129.18) - /unlink 47945 (time 1781282354.18 total 331.52 last 128.12) - /unlink 49453 (time 1781282364.20 total 341.54 last 150.51) - /unlink 50000 (time 1781282367.64 total 344.98 last 158.85) - /unlink 51538 (time 1781282377.64 total 354.99 last 153.75) - /unlink 53112 (time 1781282387.64 total 364.99 last 157.38) - /unlink 54653 (time 1781282397.65 total 374.99 last 154.04) - /unlink 56194 (time 1781282407.65 total 384.99 last 154.06) - /unlink 57748 (time 1781282417.65 total 395.00 last 155.39) - /unlink 59221 (time 1781282427.65 total 405.00 last 147.28) - /unlink 60000 (time 1781282433.56 total 410.90 last 131.91) - /unlink 61269 (time 1781282443.56 total 420.91 last 126.85) - /unlink 62566 (time 1781282453.56 total 430.91 last 129.68) - /unlink 64010 (time 1781282463.56 total 440.91 last 144.38) total: 64768 /unlink in 446.65 seconds: 145.01 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 835 (time 1781282501.66 total 10.01 last 83.43) - open/close 1658 (time 1781282511.67 total 20.01 last 82.24) - open/close 2516 (time 1781282521.67 total 30.02 last 85.76) - open/close 3358 (time 1781282531.68 total 40.02 last 84.19) - open/close 4212 (time 1781282541.68 total 50.02 last 85.36) - open/close 5030 (time 1781282551.68 total 60.03 last 81.78) - open/close 5853 (time 1781282561.68 total 70.03 last 82.29) - open/close 6617 (time 1781282571.69 total 80.04 last 76.35) - open/close 7448 (time 1781282581.70 total 90.04 last 83.05) - open/close 8195 (time 1781282591.70 total 100.05 last 74.67) - open/close 9067 (time 1781282601.70 total 110.05 last 87.18) - open/close 9956 (time 1781282611.71 total 120.05 last 88.87) - open/close 10873 (time 1781282621.71 total 130.06 last 91.66) - open/close 11761 (time 1781282631.71 total 140.06 last 88.80) - open/close 12552 (time 1781282641.71 total 150.06 last 79.09) - open/close 13379 (time 1781282651.71 total 160.06 last 82.69) - open/close 14156 (time 1781282661.72 total 170.07 last 77.61) - open/close 14982 (time 1781282671.73 total 180.08 last 82.53) - open/close 15797 (time 1781282681.73 total 190.08 last 81.49) - open/close 16650 (time 1781282691.74 total 200.09 last 85.22) - open/close 17400 (time 1781282701.76 total 210.10 last 74.91) - open/close 18096 (time 1781282711.76 total 220.10 last 69.60) - open/close 18944 (time 1781282721.77 total 230.11 last 84.72) - open/close 19750 (time 1781282731.77 total 240.11 last 80.60) - open/close 20000 (time 1781282734.97 total 243.32 last 77.98) - open/close 20870 (time 1781282744.98 total 253.32 last 86.96) - open/close 21705 (time 1781282754.98 total 263.32 last 83.49) - open/close 22544 (time 1781282764.99 total 273.33 last 83.83) - open/close 23349 (time 1781282775.00 total 283.34 last 80.43) - open/close 24192 (time 1781282785.01 total 293.35 last 84.18) - open/close 25082 (time 1781282795.02 total 303.37 last 88.88) - open/close 25969 (time 1781282805.03 total 313.37 last 88.68) - open/close 26777 (time 1781282815.03 total 323.38 last 80.74) - open/close 27540 (time 1781282825.04 total 333.39 last 76.24) - open/close 28219 (time 1781282835.05 total 343.39 last 67.86) - open/close 28954 (time 1781282845.05 total 353.40 last 73.45) - open/close 29733 (time 1781282855.05 total 363.40 last 77.90) - open/close 30000 (time 1781282858.44 total 366.79 last 78.77) - open/close 30839 (time 1781282868.45 total 376.79 last 83.85) - open/close 31658 (time 1781282878.45 total 386.80 last 81.87) - open/close 32438 (time 1781282888.47 total 396.81 last 77.88) - open/close 33221 (time 1781282898.50 total 406.85 last 78.05) - open/close 33969 (time 1781282908.51 total 416.85 last 74.75) - open/close 34756 (time 1781282918.51 total 426.86 last 78.67) - open/close 35498 (time 1781282928.52 total 436.86 last 74.17) - open/close 36318 (time 1781282938.52 total 446.86 last 81.99) - open/close 37130 (time 1781282948.53 total 456.87 last 81.11) - open/close 37951 (time 1781282958.53 total 466.88 last 82.06) - open/close 38806 (time 1781282968.54 total 476.89 last 85.42) - open/close 39705 (time 1781282978.55 total 486.89 last 89.86) - open/close 40000 (time 1781282982.22 total 490.56 last 80.32) - open/close 40855 (time 1781282992.23 total 500.58 last 85.40) - open/close 41716 (time 1781283002.24 total 510.58 last 86.05) - open/close 42495 (time 1781283012.25 total 520.59 last 77.80) - open/close 43305 (time 1781283022.26 total 530.60 last 80.95) - open/close 44121 (time 1781283032.27 total 540.61 last 81.50) - open/close 44928 (time 1781283042.28 total 550.63 last 80.60) - open/close 45779 (time 1781283052.28 total 560.63 last 85.08) - open/close 46579 (time 1781283062.31 total 570.65 last 79.82) - open/close 47306 (time 1781283072.32 total 580.66 last 72.61) - open/close 48162 (time 1781283082.32 total 590.67 last 85.57) - open/close 48957 (time 1781283092.33 total 600.68 last 79.41) - open/close 49868 (time 1781283102.33 total 610.68 last 91.08) - open/close 50890 (time 1781283112.34 total 620.68 last 102.18) - open/close 51786 (time 1781283122.35 total 630.69 last 89.50) - open/close 52642 (time 1781283132.35 total 640.70 last 85.57) - open/close 53520 (time 1781283142.35 total 650.70 last 87.78) - open/close 54519 (time 1781283152.38 total 660.72 last 99.66) - open/close 55433 (time 1781283162.38 total 670.73 last 91.37) - open/close 56301 (time 1781283172.38 total 680.73 last 86.80) - open/close 57096 (time 1781283182.39 total 690.73 last 79.45) - open/close 57955 (time 1781283192.39 total 700.74 last 85.87) - open/close 58679 (time 1781283202.40 total 710.75 last 72.32) - open/close 59437 (time 1781283212.41 total 720.76 last 75.74) - open/close 60000 (time 1781283219.52 total 727.87 last 79.16) - open/close 60812 (time 1781283229.53 total 737.87 last 81.15) - open/close 61610 (time 1781283239.54 total 747.89 last 79.67) - open/close 62370 (time 1781283249.56 total 757.90 last 75.91) - open/close 63203 (time 1781283259.56 total 767.90 last 83.28) - open/close 64078 (time 1781283269.56 total 777.91 last 87.46) total: 64768 open/close in 786.12 seconds: 82.39 ops/second - /unlink 1411 (time 1781283291.22 total 10.01 last 141.02) - /unlink 2767 (time 1781283301.22 total 20.01 last 135.57) - /unlink 4134 (time 1781283311.23 total 30.01 last 136.64) - /unlink 5481 (time 1781283321.23 total 40.01 last 134.68) - /unlink 6924 (time 1781283331.23 total 50.02 last 144.23) - /unlink 8249 (time 1781283341.23 total 60.02 last 132.48) - /unlink 9441 (time 1781283351.24 total 70.03 last 119.14) - /unlink 10000 (time 1781283355.59 total 74.37 last 128.59) - /unlink 11049 (time 1781283365.60 total 84.38 last 104.77) - /unlink 12272 (time 1781283375.60 total 94.39 last 122.24) - /unlink 13637 (time 1781283385.60 total 104.39 last 136.50) - /unlink 15039 (time 1781283395.61 total 114.39 last 140.17) - /unlink 16361 (time 1781283405.61 total 124.40 last 132.13) - /unlink 17754 (time 1781283415.62 total 134.41 last 139.19) - /unlink 19252 (time 1781283425.62 total 144.41 last 149.77) - /unlink 20000 (time 1781283430.81 total 149.60 last 144.17) - /unlink 21343 (time 1781283440.82 total 159.60 last 134.18) - /unlink 22749 (time 1781283450.82 total 169.61 last 140.57) - /unlink 24048 (time 1781283460.82 total 179.61 last 129.88) - /unlink 25413 (time 1781283470.82 total 189.61 last 136.48) - /unlink 26844 (time 1781283480.82 total 199.61 last 143.08) - /unlink 28135 (time 1781283490.83 total 209.62 last 129.04) - /unlink 29409 (time 1781283500.83 total 219.62 last 127.35) - /unlink 30000 (time 1781283505.46 total 224.25 last 127.76) - /unlink 31248 (time 1781283515.47 total 234.25 last 124.72) - /unlink 32524 (time 1781283525.47 total 244.25 last 127.59) - /unlink 33895 (time 1781283535.47 total 254.26 last 137.02) - /unlink 35246 (time 1781283545.47 total 264.26 last 135.08) - /unlink 36305 (time 1781283555.48 total 274.27 last 105.79) - /unlink 37525 (time 1781283565.49 total 284.27 last 121.97) - /unlink 38734 (time 1781283575.50 total 294.29 last 120.68) - /unlink 40000 (time 1781283584.18 total 302.97 last 145.92) - /unlink 41534 (time 1781283594.18 total 312.97 last 153.36) - /unlink 42702 (time 1781283604.18 total 322.97 last 116.79) - /unlink 43934 (time 1781283614.19 total 332.97 last 123.17) - /unlink 45250 (time 1781283624.19 total 342.98 last 131.55) - /unlink 46570 (time 1781283634.20 total 352.98 last 131.93) - /unlink 47962 (time 1781283644.20 total 362.99 last 139.15) - /unlink 49600 (time 1781283654.21 total 373.00 last 163.66) - /unlink 50000 (time 1781283656.96 total 375.74 last 145.59) - /unlink 51374 (time 1781283666.96 total 385.75 last 137.30) - /unlink 52800 (time 1781283676.96 total 395.75 last 142.59) - /unlink 54159 (time 1781283686.97 total 405.75 last 135.85) - /unlink 55418 (time 1781283696.97 total 415.76 last 125.88) - /unlink 56631 (time 1781283706.97 total 425.76 last 121.23) - /unlink 57728 (time 1781283716.98 total 435.77 last 109.65) - /unlink 58980 (time 1781283726.98 total 445.77 last 125.14) - /unlink 60000 (time 1781283735.48 total 454.27 last 120.03) - /unlink 61175 (time 1781283745.48 total 464.27 last 117.49) - /unlink 62486 (time 1781283755.48 total 474.27 last 131.07) - /unlink 64032 (time 1781283765.49 total 484.27 last 154.57) total: 64768 /unlink in 489.21 seconds: 132.39 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 711 (time 1781283801.70 total 10.01 last 71.04) - open/close 1415 (time 1781283811.70 total 20.01 last 70.39) - open/close 2176 (time 1781283821.71 total 30.02 last 76.03) - open/close 2895 (time 1781283831.72 total 40.03 last 71.80) - open/close 3654 (time 1781283841.73 total 50.04 last 75.85) - open/close 4464 (time 1781283851.73 total 60.04 last 80.99) - open/close 5261 (time 1781283861.74 total 70.05 last 79.66) - open/close 6109 (time 1781283871.74 total 80.05 last 84.79) - open/close 6955 (time 1781283881.75 total 90.06 last 84.51) - open/close 7736 (time 1781283891.76 total 100.07 last 78.03) - open/close 8543 (time 1781283901.76 total 110.07 last 80.64) - open/close 9398 (time 1781283911.77 total 120.08 last 85.47) - open/close 10000 (time 1781283919.09 total 127.40 last 82.17) - open/close 10848 (time 1781283929.10 total 137.41 last 84.79) - open/close 11661 (time 1781283939.10 total 147.41 last 81.28) - open/close 12294 (time 1781283949.12 total 157.43 last 63.19) - open/close 13064 (time 1781283959.12 total 167.43 last 76.96) - open/close 13908 (time 1781283969.12 total 177.43 last 84.40) - open/close 14738 (time 1781283979.13 total 187.44 last 82.94) - open/close 15531 (time 1781283989.14 total 197.45 last 79.22) - open/close 16270 (time 1781283999.14 total 207.45 last 73.88) - open/close 16993 (time 1781284009.14 total 217.45 last 72.29) - open/close 17774 (time 1781284019.15 total 227.46 last 78.03) - open/close 18586 (time 1781284029.15 total 237.46 last 81.20) - open/close 19441 (time 1781284039.17 total 247.48 last 85.33) - open/close 20000 (time 1781284046.64 total 254.95 last 74.82) - open/close 20788 (time 1781284056.65 total 264.96 last 78.77) - open/close 21570 (time 1781284066.65 total 274.96 last 78.17) - open/close 22260 (time 1781284076.65 total 284.96 last 68.99) - open/close 23041 (time 1781284086.65 total 294.96 last 78.09) - open/close 23769 (time 1781284096.66 total 304.97 last 72.77) - open/close 24599 (time 1781284106.66 total 314.97 last 82.94) - open/close 25295 (time 1781284116.67 total 324.98 last 69.57) - open/close 25971 (time 1781284126.68 total 334.99 last 67.53) - open/close 26617 (time 1781284136.69 total 345.00 last 64.51) - open/close 27439 (time 1781284146.69 total 355.00 last 82.18) - open/close 28240 (time 1781284156.70 total 365.01 last 80.05) - open/close 29054 (time 1781284166.70 total 375.01 last 81.38) - open/close 29811 (time 1781284176.71 total 385.01 last 75.68) - open/close 30000 (time 1781284179.60 total 387.91 last 65.38) - open/close 30604 (time 1781284189.60 total 397.91 last 60.37) - open/close 31145 (time 1781284199.63 total 407.94 last 53.96) - open/close 31883 (time 1781284209.64 total 417.95 last 73.72) - open/close 32671 (time 1781284219.65 total 427.96 last 78.68) - open/close 33461 (time 1781284229.66 total 437.97 last 78.91) - open/close 34272 (time 1781284239.67 total 447.98 last 81.05) - open/close 35046 (time 1781284249.68 total 457.99 last 77.32) - open/close 35849 (time 1781284259.69 total 468.00 last 80.25) - open/close 36624 (time 1781284269.69 total 478.00 last 77.49) - open/close 37419 (time 1781284279.69 total 488.00 last 79.44) - open/close 38163 (time 1781284289.70 total 498.01 last 74.38) - open/close 38888 (time 1781284299.71 total 508.02 last 72.43) - open/close 39612 (time 1781284309.71 total 518.02 last 72.37) - open/close 40000 (time 1781284314.91 total 523.22 last 74.59) - open/close 40717 (time 1781284324.94 total 533.25 last 71.53) - open/close 41533 (time 1781284334.95 total 543.26 last 81.48) - open/close 42311 (time 1781284344.96 total 553.27 last 77.78) - open/close 43017 (time 1781284354.96 total 563.27 last 70.55) - open/close 43853 (time 1781284364.97 total 573.28 last 83.56) - open/close 44640 (time 1781284374.97 total 583.28 last 78.68) - open/close 45419 (time 1781284384.99 total 593.30 last 77.76) - open/close 46205 (time 1781284394.99 total 603.30 last 78.56) - open/close 46926 (time 1781284405.00 total 613.31 last 72.07) - open/close 47508 (time 1781284415.01 total 623.32 last 58.12) - open/close 48075 (time 1781284425.05 total 633.36 last 56.51) - open/close 48875 (time 1781284435.08 total 643.39 last 79.76) - open/close 49621 (time 1781284445.08 total 653.39 last 74.56) - open/close 50000 (time 1781284449.97 total 658.28 last 77.52) - open/close 50741 (time 1781284459.97 total 668.28 last 74.07) - open/close 51524 (time 1781284469.99 total 678.30 last 78.20) - open/close 52276 (time 1781284479.99 total 688.30 last 75.15) - open/close 53105 (time 1781284490.00 total 698.31 last 82.86) - open/close 53888 (time 1781284500.02 total 708.33 last 78.15) - open/close 54612 (time 1781284510.02 total 718.33 last 72.35) - open/close 55399 (time 1781284520.03 total 728.34 last 78.65) - open/close 56152 (time 1781284530.03 total 738.34 last 75.28) - open/close 56895 (time 1781284540.03 total 748.34 last 74.29) - open/close 57665 (time 1781284550.04 total 758.35 last 76.93) - open/close 58387 (time 1781284560.04 total 768.35 last 72.18) - open/close 59200 (time 1781284570.06 total 778.37 last 81.20) - open/close 59972 (time 1781284580.07 total 788.38 last 77.13) - open/close 60725 (time 1781284590.08 total 798.39 last 75.22) - open/close 61486 (time 1781284600.09 total 808.40 last 76.01) - open/close 62247 (time 1781284610.09 total 818.40 last 76.08) - open/close 62991 (time 1781284620.09 total 828.40 last 74.39) - open/close 63774 (time 1781284630.10 total 838.41 last 78.23) - open/close 64505 (time 1781284640.13 total 848.44 last 72.90) total: 64768 open/close in 851.84 seconds: 76.03 ops/second - /unlink 1211 (time 1781284657.46 total 10.01 last 120.96) - /unlink 2368 (time 1781284667.46 total 20.01 last 115.70) - /unlink 3586 (time 1781284677.47 total 30.01 last 121.79) - /unlink 4840 (time 1781284687.47 total 40.02 last 125.35) - /unlink 6214 (time 1781284697.47 total 50.02 last 137.32) - /unlink 7478 (time 1781284707.47 total 60.02 last 126.40) - /unlink 8792 (time 1781284717.50 total 70.05 last 131.03) - /unlink 10000 (time 1781284727.04 total 79.58 last 126.70) - /unlink 11320 (time 1781284737.05 total 89.59 last 131.89) - /unlink 12655 (time 1781284747.05 total 99.59 last 133.49) - /unlink 13922 (time 1781284757.05 total 109.60 last 126.64) - /unlink 15223 (time 1781284767.05 total 119.60 last 130.09) - /unlink 16588 (time 1781284777.06 total 129.60 last 136.45) - /unlink 17540 (time 1781284787.06 total 139.61 last 95.14) - /unlink 18571 (time 1781284797.06 total 149.61 last 103.10) - /unlink 19744 (time 1781284807.06 total 159.61 last 117.29) - /unlink 21077 (time 1781284817.07 total 169.61 last 133.24) - /unlink 22273 (time 1781284827.07 total 179.62 last 119.55) - /unlink 23567 (time 1781284837.07 total 189.62 last 129.38) - /unlink 24728 (time 1781284847.08 total 199.63 last 116.00) - /unlink 25871 (time 1781284857.08 total 209.63 last 114.27) - /unlink 27097 (time 1781284867.09 total 219.64 last 122.55) - /unlink 28241 (time 1781284877.10 total 229.64 last 114.30) - /unlink 29335 (time 1781284887.10 total 239.65 last 109.33) - /unlink 30000 (time 1781284893.19 total 245.74 last 109.22) - /unlink 31226 (time 1781284903.20 total 255.74 last 122.54) - /unlink 32385 (time 1781284913.20 total 265.75 last 115.86) - /unlink 33552 (time 1781284923.20 total 275.75 last 116.67) - /unlink 34787 (time 1781284933.21 total 285.75 last 123.46) - /unlink 36094 (time 1781284943.22 total 295.76 last 130.57) - /unlink 37434 (time 1781284953.22 total 305.76 last 133.99) - /unlink 38686 (time 1781284963.23 total 315.78 last 125.03) - /unlink 39992 (time 1781284973.23 total 325.78 last 130.56) - /unlink 41099 (time 1781284983.24 total 335.79 last 110.58) - /unlink 42271 (time 1781284993.29 total 345.83 last 116.72) - /unlink 43619 (time 1781285003.30 total 355.84 last 134.66) - /unlink 44834 (time 1781285013.30 total 365.84 last 121.49) - /unlink 45898 (time 1781285023.30 total 375.84 last 106.40) - /unlink 47193 (time 1781285033.30 total 385.85 last 129.46) - /unlink 48399 (time 1781285043.31 total 395.85 last 120.54) - /unlink 49614 (time 1781285053.32 total 405.86 last 121.39) - /unlink 50000 (time 1781285056.41 total 408.96 last 124.72) - /unlink 51226 (time 1781285066.42 total 418.96 last 122.54) - /unlink 52522 (time 1781285076.42 total 428.97 last 129.54) - /unlink 53816 (time 1781285086.42 total 438.97 last 129.35) - /unlink 54984 (time 1781285096.43 total 448.98 last 116.72) - /unlink 56187 (time 1781285106.43 total 458.98 last 120.26) - /unlink 57418 (time 1781285116.44 total 468.99 last 123.02) - /unlink 58658 (time 1781285126.44 total 478.99 last 123.95) - /unlink 59928 (time 1781285136.45 total 489.00 last 126.87) - /unlink 61282 (time 1781285146.46 total 499.00 last 135.39) - /unlink 62835 (time 1781285156.46 total 509.01 last 155.22) - /unlink 64432 (time 1781285166.46 total 519.01 last 159.64) total: 64768 /unlink in 521.62 seconds: 124.17 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 873 (time 1781285199.40 total 10.00 last 87.29) - open/close 1729 (time 1781285209.40 total 20.01 last 85.56) - open/close 2613 (time 1781285219.42 total 30.02 last 88.30) - open/close 3391 (time 1781285229.42 total 40.02 last 77.79) - open/close 4135 (time 1781285239.42 total 50.02 last 74.35) - open/close 4899 (time 1781285249.44 total 60.04 last 76.28) - open/close 5681 (time 1781285259.45 total 70.05 last 78.13) - open/close 6548 (time 1781285269.46 total 80.06 last 86.60) - open/close 7377 (time 1781285279.47 total 90.07 last 82.83) - open/close 8157 (time 1781285289.48 total 100.08 last 77.91) - open/close 8955 (time 1781285299.49 total 110.09 last 79.75) - open/close 9801 (time 1781285309.49 total 120.10 last 84.53) - open/close 10000 (time 1781285312.01 total 122.61 last 79.05) - open/close 10826 (time 1781285322.02 total 132.62 last 82.58) - open/close 11616 (time 1781285332.02 total 142.62 last 78.96) - open/close 12418 (time 1781285342.03 total 152.63 last 80.09) - open/close 13109 (time 1781285352.03 total 162.64 last 69.09) - open/close 13839 (time 1781285362.04 total 172.64 last 72.95) - open/close 14649 (time 1781285372.05 total 182.65 last 80.96) - open/close 15397 (time 1781285382.05 total 192.65 last 74.78) - open/close 16208 (time 1781285392.07 total 202.67 last 80.97) - open/close 17012 (time 1781285402.07 total 212.68 last 80.33) - open/close 17781 (time 1781285412.08 total 222.68 last 76.88) - open/close 18602 (time 1781285422.09 total 232.69 last 82.02) - open/close 19428 (time 1781285432.09 total 242.69 last 82.58) - open/close 20000 (time 1781285440.42 total 251.02 last 68.66) - open/close 20714 (time 1781285450.42 total 261.02 last 71.39) - open/close 21462 (time 1781285460.43 total 271.03 last 74.77) - open/close 22188 (time 1781285470.44 total 281.04 last 72.51) - open/close 22914 (time 1781285480.45 total 291.05 last 72.53) - open/close 23711 (time 1781285490.45 total 301.05 last 79.69) - open/close 24530 (time 1781285500.45 total 311.05 last 81.89) - open/close 25288 (time 1781285510.47 total 321.07 last 75.66) - open/close 26033 (time 1781285520.56 total 331.16 last 73.81) - open/close 26778 (time 1781285530.57 total 341.17 last 74.46) - open/close 27525 (time 1781285540.57 total 351.17 last 74.67) - open/close 28318 (time 1781285550.59 total 361.19 last 79.18) - open/close 29095 (time 1781285560.59 total 371.20 last 77.64) - open/close 29865 (time 1781285570.60 total 381.20 last 76.96) - open/close 30624 (time 1781285580.60 total 391.20 last 75.88) - open/close 31430 (time 1781285590.61 total 401.21 last 80.53) - open/close 32124 (time 1781285600.62 total 411.22 last 69.36) - open/close 32870 (time 1781285610.62 total 421.22 last 74.57) - open/close 33624 (time 1781285620.63 total 431.23 last 75.36) - open/close 34355 (time 1781285630.63 total 441.23 last 73.05) - open/close 35118 (time 1781285640.65 total 451.25 last 76.15) - open/close 35904 (time 1781285650.65 total 461.26 last 78.59) - open/close 36640 (time 1781285660.66 total 471.26 last 73.55) - open/close 37440 (time 1781285670.68 total 481.28 last 79.84) - open/close 38172 (time 1781285680.68 total 491.28 last 73.19) - open/close 38878 (time 1781285690.70 total 501.30 last 70.46) - open/close 39459 (time 1781285700.71 total 511.31 last 58.09) - open/close 40000 (time 1781285708.24 total 518.84 last 71.85) - open/close 40710 (time 1781285718.24 total 528.84 last 70.99) - open/close 41424 (time 1781285728.25 total 538.85 last 71.32) - open/close 42217 (time 1781285738.25 total 548.85 last 79.26) - open/close 43031 (time 1781285748.27 total 558.88 last 81.23) - open/close 43822 (time 1781285758.28 total 568.88 last 79.07) - open/close 44639 (time 1781285768.29 total 578.89 last 81.60) - open/close 45423 (time 1781285778.29 total 588.89 last 78.38) - open/close 46100 (time 1781285788.30 total 598.90 last 67.64) - open/close 46837 (time 1781285798.30 total 608.90 last 73.70) - open/close 47642 (time 1781285808.31 total 618.91 last 80.45) - open/close 48357 (time 1781285818.32 total 628.92 last 71.40) - open/close 49136 (time 1781285828.32 total 638.93 last 77.89) - open/close 49932 (time 1781285838.33 total 648.93 last 79.58) - open/close 50766 (time 1781285848.35 total 658.95 last 83.19) - open/close 51559 (time 1781285858.36 total 668.96 last 79.24) - open/close 52404 (time 1781285868.36 total 678.96 last 84.48) - open/close 53259 (time 1781285878.37 total 688.97 last 85.42) - open/close 54045 (time 1781285888.38 total 698.98 last 78.55) - open/close 54852 (time 1781285898.38 total 708.98 last 80.67) - open/close 55628 (time 1781285908.38 total 718.98 last 77.58) - open/close 56464 (time 1781285918.40 total 729.00 last 83.48) - open/close 57268 (time 1781285928.40 total 739.00 last 80.38) - open/close 58083 (time 1781285938.40 total 749.00 last 81.47) - open/close 58857 (time 1781285948.40 total 759.01 last 77.39) - open/close 59610 (time 1781285958.41 total 769.02 last 75.23) - open/close 60000 (time 1781285963.12 total 773.72 last 82.82) - open/close 60791 (time 1781285973.13 total 783.73 last 79.06) - open/close 61623 (time 1781285983.13 total 793.73 last 83.18) - open/close 62377 (time 1781285993.15 total 803.75 last 75.27) - open/close 63208 (time 1781286003.16 total 813.76 last 83.02) - open/close 64061 (time 1781286013.16 total 823.76 last 85.26) total: 64768 open/close in 831.94 seconds: 77.85 ops/second - /unlink 1476 (time 1781286034.32 total 10.00 last 147.59) - /unlink 2906 (time 1781286044.33 total 20.00 last 142.94) - /unlink 4238 (time 1781286054.33 total 30.01 last 133.20) - /unlink 5583 (time 1781286064.33 total 40.01 last 134.46) - /unlink 6877 (time 1781286074.34 total 50.02 last 129.29) - /unlink 8138 (time 1781286084.34 total 60.02 last 126.07) - /unlink 9331 (time 1781286094.35 total 70.03 last 119.23) - /unlink 10000 (time 1781286100.25 total 75.92 last 113.43) - /unlink 11245 (time 1781286110.25 total 85.93 last 124.47) - /unlink 12526 (time 1781286120.26 total 95.93 last 128.00) - /unlink 13776 (time 1781286130.27 total 105.95 last 124.84) - /unlink 15081 (time 1781286140.28 total 115.96 last 130.32) - /unlink 16372 (time 1781286150.30 total 125.98 last 128.84) - /unlink 17703 (time 1781286160.31 total 135.99 last 133.04) - /unlink 18938 (time 1781286170.31 total 145.99 last 123.49) - /unlink 20000 (time 1781286179.11 total 154.79 last 120.64) - /unlink 21258 (time 1781286189.14 total 164.82 last 125.44) - /unlink 22520 (time 1781286199.14 total 174.82 last 126.16) - /unlink 23780 (time 1781286209.14 total 184.82 last 126.00) - /unlink 25031 (time 1781286219.15 total 194.83 last 125.02) - /unlink 26292 (time 1781286229.15 total 204.83 last 126.08) - /unlink 27408 (time 1781286239.16 total 214.83 last 111.55) - /unlink 28536 (time 1781286249.17 total 224.85 last 112.66) - /unlink 29605 (time 1781286259.17 total 234.85 last 106.89) - /unlink 30000 (time 1781286262.43 total 238.10 last 121.26) - /unlink 31176 (time 1781286272.43 total 248.10 last 117.60) - /unlink 32406 (time 1781286282.43 total 258.11 last 122.97) - /unlink 33672 (time 1781286292.43 total 268.11 last 126.54) - /unlink 34915 (time 1781286302.44 total 278.11 last 124.29) - /unlink 36164 (time 1781286312.44 total 288.11 last 124.89) - /unlink 37528 (time 1781286322.44 total 298.12 last 136.33) - /unlink 38845 (time 1781286332.44 total 308.12 last 131.70) - /unlink 40000 (time 1781286340.57 total 316.25 last 142.00) - /unlink 41386 (time 1781286350.58 total 326.26 last 138.50) - /unlink 42836 (time 1781286360.58 total 336.26 last 144.97) - /unlink 44220 (time 1781286370.59 total 346.27 last 138.30) - /unlink 45498 (time 1781286380.60 total 356.28 last 127.63) - /unlink 46689 (time 1781286390.61 total 366.29 last 118.99) - /unlink 47857 (time 1781286400.64 total 376.32 last 116.48) - /unlink 48853 (time 1781286410.64 total 386.32 last 99.57) - /unlink 50000 (time 1781286419.73 total 395.41 last 126.17) - /unlink 51288 (time 1781286429.75 total 405.43 last 128.62) - /unlink 52544 (time 1781286439.75 total 415.43 last 125.56) - /unlink 53736 (time 1781286449.77 total 425.44 last 119.04) - /unlink 54924 (time 1781286459.78 total 435.46 last 118.65) - /unlink 56243 (time 1781286469.79 total 445.47 last 131.74) - /unlink 57704 (time 1781286479.79 total 455.47 last 146.05) - /unlink 58890 (time 1781286489.80 total 465.48 last 118.50) - /unlink 60000 (time 1781286498.56 total 474.23 last 126.78) - /unlink 61181 (time 1781286508.60 total 484.27 last 117.64) - /unlink 62236 (time 1781286518.60 total 494.28 last 105.43) - /unlink 63560 (time 1781286528.61 total 504.28 last 132.34) total: 64768 /unlink in 513.41 seconds: 126.15 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 787 (time 1781286570.11 total 10.00 last 78.67) - open/close 1555 (time 1781286580.12 total 20.01 last 76.74) - open/close 2356 (time 1781286590.13 total 30.02 last 80.01) - open/close 3153 (time 1781286600.15 total 40.04 last 79.53) - open/close 3951 (time 1781286610.15 total 50.05 last 79.78) - open/close 4763 (time 1781286620.15 total 60.05 last 81.19) - open/close 5613 (time 1781286630.16 total 70.06 last 84.92) - open/close 6412 (time 1781286640.18 total 80.07 last 79.80) - open/close 7183 (time 1781286650.18 total 90.07 last 77.10) - open/close 7849 (time 1781286660.18 total 100.08 last 66.56) - open/close 8399 (time 1781286670.19 total 110.08 last 54.97) - open/close 9070 (time 1781286680.19 total 120.09 last 67.08) - open/close 9796 (time 1781286690.20 total 130.09 last 72.55) - open/close 10000 (time 1781286692.71 total 132.61 last 81.11) - open/close 10673 (time 1781286702.72 total 142.62 last 67.24) - open/close 11349 (time 1781286712.73 total 152.63 last 67.54) - open/close 12115 (time 1781286722.74 total 162.63 last 76.56) - open/close 12880 (time 1781286732.75 total 172.64 last 76.42) - open/close 13671 (time 1781286742.76 total 182.65 last 79.03) - open/close 14383 (time 1781286752.77 total 192.66 last 71.11) - open/close 15140 (time 1781286762.78 total 202.67 last 75.65) - open/close 15912 (time 1781286772.78 total 212.67 last 77.18) - open/close 16606 (time 1781286782.79 total 222.68 last 69.34) - open/close 17303 (time 1781286792.79 total 232.69 last 69.66) - open/close 18130 (time 1781286802.80 total 242.70 last 82.63) - open/close 18860 (time 1781286812.81 total 252.71 last 72.92) - open/close 19545 (time 1781286822.82 total 262.71 last 68.44) - open/close 20000 (time 1781286828.84 total 268.73 last 75.63) - open/close 20734 (time 1781286838.85 total 278.74 last 73.33) - open/close 21539 (time 1781286848.85 total 288.74 last 80.50) - open/close 22250 (time 1781286858.87 total 298.76 last 70.95) - open/close 22989 (time 1781286868.87 total 308.77 last 73.87) - open/close 23777 (time 1781286878.88 total 318.77 last 78.75) - open/close 24516 (time 1781286888.89 total 328.78 last 73.84) - open/close 25226 (time 1781286898.89 total 338.78 last 70.97) - open/close 26004 (time 1781286908.89 total 348.79 last 77.79) - open/close 26807 (time 1781286918.90 total 358.79 last 80.25) - open/close 27547 (time 1781286928.90 total 368.79 last 73.98) - open/close 28256 (time 1781286938.91 total 378.80 last 70.86) - open/close 29030 (time 1781286948.96 total 388.85 last 77.00) - open/close 29785 (time 1781286958.97 total 398.86 last 75.44) - open/close 30000 (time 1781286961.69 total 401.58 last 78.93) - open/close 30774 (time 1781286971.70 total 411.59 last 77.35) - open/close 31523 (time 1781286981.71 total 421.60 last 74.84) - open/close 32346 (time 1781286991.71 total 431.60 last 82.26) - open/close 33120 (time 1781287001.72 total 441.61 last 77.36) - open/close 33874 (time 1781287011.72 total 451.61 last 75.39) - open/close 34731 (time 1781287021.72 total 461.62 last 85.65) - open/close 35569 (time 1781287031.73 total 471.62 last 83.78) - open/close 36166 (time 1781287041.73 total 481.62 last 59.69) - open/close 36924 (time 1781287051.73 total 491.63 last 75.77) - open/close 37669 (time 1781287061.73 total 501.63 last 74.49) - open/close 38504 (time 1781287071.75 total 511.64 last 83.36) - open/close 39233 (time 1781287081.76 total 521.65 last 72.85) - open/close 40000 (time 1781287091.71 total 531.60 last 77.09) - open/close 40782 (time 1781287101.72 total 541.61 last 78.08) - open/close 41540 (time 1781287111.73 total 551.63 last 75.71) - open/close 42274 (time 1781287121.74 total 561.63 last 73.37) - open/close 42965 (time 1781287131.74 total 571.63 last 69.09) - open/close 43690 (time 1781287141.78 total 581.67 last 72.20) - open/close 44413 (time 1781287151.78 total 591.68 last 72.27) - open/close 45195 (time 1781287161.79 total 601.69 last 78.13) - open/close 45791 (time 1781287171.81 total 611.70 last 59.52) - open/close 46381 (time 1781287181.81 total 621.71 last 58.96) - open/close 46986 (time 1781287191.83 total 631.72 last 60.42) - open/close 47725 (time 1781287201.83 total 641.73 last 73.85) - open/close 48420 (time 1781287211.83 total 651.73 last 69.49) - open/close 49160 (time 1781287221.84 total 661.73 last 73.98) - open/close 49875 (time 1781287231.85 total 671.75 last 71.39) - open/close 50584 (time 1781287241.86 total 681.75 last 70.87) - open/close 51317 (time 1781287251.87 total 691.77 last 73.18) - open/close 51990 (time 1781287261.88 total 701.78 last 67.23) - open/close 52705 (time 1781287271.89 total 711.78 last 71.47) - open/close 53385 (time 1781287281.90 total 721.79 last 67.94) - open/close 54090 (time 1781287291.91 total 731.80 last 70.43) - open/close 54718 (time 1781287301.91 total 741.81 last 62.75) - open/close 55341 (time 1781287311.93 total 751.82 last 62.23) - open/close 56061 (time 1781287321.94 total 761.83 last 71.90) - open/close 56651 (time 1781287331.98 total 771.87 last 58.77) - open/close 57252 (time 1781287342.00 total 781.89 last 59.99) - open/close 57952 (time 1781287352.00 total 791.90 last 69.96) - open/close 58582 (time 1781287362.01 total 801.90 last 62.96) - open/close 59243 (time 1781287372.02 total 811.91 last 66.02) - open/close 59954 (time 1781287382.02 total 821.91 last 71.10) - open/close 60704 (time 1781287392.03 total 831.92 last 74.95) - open/close 61419 (time 1781287402.04 total 841.93 last 71.42) - open/close 62131 (time 1781287412.04 total 851.93 last 71.20) - open/close 62861 (time 1781287422.05 total 861.94 last 72.96) - open/close 63615 (time 1781287432.06 total 871.95 last 75.32) - open/close 64303 (time 1781287442.08 total 881.98 last 68.63) total: 64768 open/close in 888.62 seconds: 72.89 ops/second - /unlink 1141 (time 1781287463.87 total 10.00 last 114.06) - /unlink 2285 (time 1781287473.88 total 20.01 last 114.35) - /unlink 3477 (time 1781287483.89 total 30.02 last 119.09) - /unlink 4652 (time 1781287493.89 total 40.02 last 117.49) - /unlink 5781 (time 1781287503.89 total 50.02 last 112.84) - /unlink 6887 (time 1781287513.90 total 60.03 last 110.58) - /unlink 8043 (time 1781287523.90 total 70.03 last 115.60) - /unlink 9177 (time 1781287533.90 total 80.03 last 113.35) - /unlink 10000 (time 1781287541.60 total 87.73 last 106.85) - /unlink 11115 (time 1781287551.62 total 97.75 last 111.35) - /unlink 12227 (time 1781287561.63 total 107.76 last 111.10) - /unlink 13304 (time 1781287571.63 total 117.76 last 107.67) - /unlink 14452 (time 1781287581.63 total 127.76 last 114.77) - /unlink 15684 (time 1781287591.63 total 137.76 last 123.20) - /unlink 16856 (time 1781287601.64 total 147.77 last 117.10) - /unlink 17912 (time 1781287611.64 total 157.77 last 105.55) - /unlink 19005 (time 1781287621.65 total 167.78 last 109.29) - /unlink 20000 (time 1781287630.25 total 176.38 last 115.67) - /unlink 21200 (time 1781287640.26 total 186.39 last 119.87) - /unlink 22317 (time 1781287650.26 total 196.39 last 111.67) - /unlink 23492 (time 1781287660.26 total 206.39 last 117.49) - /unlink 24751 (time 1781287670.27 total 216.40 last 125.83) - /unlink 25976 (time 1781287680.28 total 226.41 last 122.41) - /unlink 27170 (time 1781287690.28 total 236.41 last 119.39) - /unlink 28198 (time 1781287700.28 total 246.41 last 102.76) - /unlink 29364 (time 1781287710.28 total 256.41 last 116.57) - /unlink 30000 (time 1781287715.55 total 261.68 last 120.78) - /unlink 31253 (time 1781287725.55 total 271.68 last 125.27) - /unlink 32331 (time 1781287735.57 total 281.70 last 107.60) - /unlink 33267 (time 1781287745.58 total 291.71 last 93.54) - /unlink 34177 (time 1781287755.58 total 301.71 last 90.97) - /unlink 35124 (time 1781287765.58 total 311.71 last 94.66) - /unlink 36135 (time 1781287775.59 total 321.72 last 101.01) - /unlink 37128 (time 1781287785.60 total 331.73 last 99.22) - /unlink 38096 (time 1781287795.60 total 341.73 last 96.78) - /unlink 39189 (time 1781287805.60 total 351.73 last 109.28) - /unlink 40000 (time 1781287813.13 total 359.26 last 107.80) - /unlink 41121 (time 1781287823.13 total 369.26 last 112.06) - /unlink 42201 (time 1781287833.16 total 379.29 last 107.73) - /unlink 43344 (time 1781287843.16 total 389.29 last 114.29) - /unlink 44389 (time 1781287853.20 total 399.33 last 104.08) - /unlink 45571 (time 1781287863.20 total 409.33 last 118.14) - /unlink 46764 (time 1781287873.20 total 419.33 last 119.29) - /unlink 47873 (time 1781287883.20 total 429.33 last 110.89) - /unlink 49010 (time 1781287893.21 total 439.34 last 113.65) - /unlink 50000 (time 1781287901.19 total 447.32 last 124.04) - /unlink 51128 (time 1781287911.19 total 457.32 last 112.78) - /unlink 52199 (time 1781287921.20 total 467.33 last 107.04) - /unlink 53327 (time 1781287931.20 total 477.33 last 112.79) - /unlink 54628 (time 1781287941.21 total 487.34 last 129.93) - /unlink 55875 (time 1781287951.21 total 497.34 last 124.66) - /unlink 56963 (time 1781287961.24 total 507.37 last 108.55) - /unlink 58133 (time 1781287971.24 total 517.37 last 116.97) - /unlink 59303 (time 1781287981.24 total 527.37 last 116.93) - /unlink 60000 (time 1781287987.25 total 533.38 last 116.03) - /unlink 61211 (time 1781287997.25 total 543.38 last 121.09) - /unlink 62606 (time 1781288007.27 total 553.40 last 139.31) - /unlink 63880 (time 1781288017.27 total 563.40 last 127.40) total: 64768 /unlink in 570.95 seconds: 113.44 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg630-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg630-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg630-server stop mds service on oleg630-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg630-server stop mds service on oleg630-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg630-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg630-server' oleg630-server: oleg630-server.virtnet: executing unload_modules_local oleg630-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy