== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 12:16:57 (1781281017) Stopping clients: oleg647-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg647-client.virtnet /mnt/lustre2 (opts:-f) oleg647-server: oleg647-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg647-server' oleg647-server: oleg647-server.virtnet: executing load_modules_local oleg647-server: Loading modules from /home/green/git/lustre-release/lustre oleg647-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg647-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg647-server: detected 4 online CPUs by sysfs oleg647-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg647-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg647-server: oleg647-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg647-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg647-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg647-server: oleg647-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg647-client.virtnet: -o user_xattr,flock oleg647-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16166 358 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 777 (time 1781281128.47 total 10.02 last 77.55) - open/close 1519 (time 1781281138.47 total 20.02 last 74.18) - open/close 2328 (time 1781281148.47 total 30.02 last 80.89) - open/close 3163 (time 1781281158.48 total 40.03 last 83.47) - open/close 3979 (time 1781281168.48 total 50.03 last 81.58) - open/close 4688 (time 1781281178.48 total 60.03 last 70.86) - open/close 5492 (time 1781281188.49 total 70.04 last 80.35) - open/close 6267 (time 1781281198.50 total 80.05 last 77.42) - open/close 6997 (time 1781281208.51 total 90.06 last 72.94) - open/close 7777 (time 1781281218.51 total 100.06 last 77.97) - open/close 8510 (time 1781281228.52 total 110.07 last 73.28) - open/close 9213 (time 1781281238.52 total 120.07 last 70.25) - open/close 10000 (time 1781281247.97 total 129.52 last 83.28) - open/close 10741 (time 1781281257.97 total 139.52 last 74.09) - open/close 11458 (time 1781281267.98 total 149.53 last 71.68) - open/close 12267 (time 1781281277.98 total 159.53 last 80.88) - open/close 13023 (time 1781281287.99 total 169.54 last 75.53) - open/close 13787 (time 1781281298.00 total 179.55 last 76.35) - open/close 14525 (time 1781281308.00 total 189.55 last 73.75) - open/close 15279 (time 1781281318.01 total 199.56 last 75.34) - open/close 16017 (time 1781281328.01 total 209.56 last 73.77) - open/close 16798 (time 1781281338.02 total 219.57 last 78.07) - open/close 17594 (time 1781281348.02 total 229.57 last 79.59) - open/close 18362 (time 1781281358.03 total 239.58 last 76.74) - open/close 19145 (time 1781281368.03 total 249.58 last 78.30) - open/close 19955 (time 1781281378.03 total 259.58 last 81.00) - open/close 20678 (time 1781281388.03 total 269.58 last 72.27) - open/close 21462 (time 1781281398.03 total 279.58 last 78.39) - open/close 22232 (time 1781281408.03 total 289.58 last 76.98) - open/close 23015 (time 1781281418.06 total 299.61 last 78.11) - open/close 23795 (time 1781281428.06 total 309.61 last 77.99) - open/close 24582 (time 1781281438.06 total 319.61 last 78.69) - open/close 25375 (time 1781281448.07 total 329.62 last 79.23) - open/close 26172 (time 1781281458.08 total 339.63 last 79.61) - open/close 26933 (time 1781281468.09 total 349.64 last 76.03) - open/close 27718 (time 1781281478.10 total 359.65 last 78.44) - open/close 28521 (time 1781281488.10 total 369.65 last 80.24) - open/close 29238 (time 1781281498.11 total 379.66 last 71.67) - open/close 30000 (time 1781281507.72 total 389.27 last 79.23) - open/close 30784 (time 1781281517.73 total 399.28 last 78.37) - open/close 31567 (time 1781281527.74 total 409.29 last 78.24) - open/close 32452 (time 1781281537.75 total 419.30 last 88.40) - open/close 33272 (time 1781281547.75 total 429.30 last 81.97) - open/close 34105 (time 1781281557.75 total 439.30 last 83.27) - open/close 34931 (time 1781281567.77 total 449.32 last 82.49) - open/close 35816 (time 1781281577.77 total 459.32 last 88.50) - open/close 36625 (time 1781281587.77 total 469.32 last 80.84) - open/close 37533 (time 1781281597.78 total 479.33 last 90.71) - open/close 38381 (time 1781281607.79 total 489.34 last 84.77) - open/close 39184 (time 1781281617.79 total 499.34 last 80.24) - open/close 39888 (time 1781281627.81 total 509.36 last 70.27) - open/close 40688 (time 1781281637.82 total 519.37 last 79.95) - open/close 41442 (time 1781281647.84 total 529.39 last 75.22) - open/close 42284 (time 1781281657.85 total 539.40 last 84.18) - open/close 43047 (time 1781281667.85 total 549.40 last 76.27) - open/close 43836 (time 1781281677.87 total 559.42 last 78.78) - open/close 44659 (time 1781281687.87 total 569.42 last 82.28) - open/close 45484 (time 1781281697.87 total 579.42 last 82.45) - open/close 46256 (time 1781281707.88 total 589.43 last 77.15) - open/close 47020 (time 1781281717.88 total 599.43 last 76.40) - open/close 47828 (time 1781281727.88 total 609.43 last 80.80) - open/close 48669 (time 1781281737.89 total 619.44 last 84.05) - open/close 49493 (time 1781281747.89 total 629.44 last 82.37) - open/close 50000 (time 1781281754.52 total 636.07 last 76.51) - open/close 50754 (time 1781281764.53 total 646.08 last 75.33) - open/close 51544 (time 1781281774.54 total 656.09 last 78.87) - open/close 52342 (time 1781281784.55 total 666.10 last 79.78) - open/close 53125 (time 1781281794.55 total 676.11 last 78.23) - open/close 53870 (time 1781281804.56 total 686.11 last 74.49) - open/close 54725 (time 1781281814.56 total 696.11 last 85.47) - open/close 55608 (time 1781281824.57 total 706.12 last 88.22) - open/close 56519 (time 1781281834.58 total 716.13 last 91.04) - open/close 57366 (time 1781281844.58 total 726.13 last 84.67) - open/close 58162 (time 1781281854.59 total 736.14 last 79.54) - open/close 59032 (time 1781281864.59 total 746.14 last 86.95) - open/close 59874 (time 1781281874.59 total 756.14 last 84.19) - open/close 60727 (time 1781281884.60 total 766.15 last 85.27) - open/close 61629 (time 1781281894.60 total 776.15 last 90.17) - open/close 62605 (time 1781281904.61 total 786.16 last 97.54) - open/close 63673 (time 1781281914.61 total 796.16 last 106.80) - open/close 64547 (time 1781281924.62 total 806.17 last 87.31) total: 64768 open/close in 808.83 seconds: 80.08 ops/second - /unlink 1648 (time 1781281940.50 total 10.00 last 164.77) - /unlink 3080 (time 1781281950.50 total 20.00 last 143.16) - /unlink 4719 (time 1781281960.51 total 30.01 last 163.82) - /unlink 6282 (time 1781281970.52 total 40.02 last 156.06) - /unlink 7603 (time 1781281980.56 total 50.06 last 131.61) - /unlink 9318 (time 1781281990.56 total 60.07 last 171.45) - /unlink 10000 (time 1781281995.30 total 64.80 last 144.06) - /unlink 11584 (time 1781282005.30 total 74.80 last 158.39) - /unlink 13184 (time 1781282015.30 total 84.80 last 159.99) - /unlink 14718 (time 1781282025.30 total 94.80 last 153.38) - /unlink 16293 (time 1781282035.31 total 104.81 last 157.38) - /unlink 17775 (time 1781282045.32 total 114.82 last 148.09) - /unlink 19485 (time 1781282055.32 total 124.82 last 170.96) - /unlink 20000 (time 1781282058.76 total 128.26 last 149.66) - /unlink 21719 (time 1781282068.77 total 138.27 last 171.79) - /unlink 23290 (time 1781282078.77 total 148.27 last 157.09) - /unlink 24812 (time 1781282088.77 total 158.27 last 152.19) - /unlink 26394 (time 1781282098.77 total 168.27 last 158.17) - /unlink 28245 (time 1781282108.77 total 178.28 last 185.01) - /unlink 30000 (time 1781282117.98 total 187.48 last 190.61) - /unlink 31511 (time 1781282127.98 total 197.48 last 151.06) - /unlink 32822 (time 1781282137.99 total 207.49 last 131.05) - /unlink 34462 (time 1781282147.99 total 217.49 last 163.98) - /unlink 36291 (time 1781282157.99 total 227.49 last 182.88) - /unlink 37777 (time 1781282168.00 total 237.50 last 148.46) - /unlink 39164 (time 1781282178.00 total 247.50 last 138.67) - /unlink 40000 (time 1781282186.37 total 255.87 last 99.89) - /unlink 41256 (time 1781282196.38 total 265.88 last 125.55) - /unlink 42752 (time 1781282206.38 total 275.88 last 149.55) - /unlink 44347 (time 1781282216.38 total 285.88 last 159.46) - /unlink 45899 (time 1781282226.39 total 295.89 last 155.12) - /unlink 47566 (time 1781282236.39 total 305.89 last 166.69) - /unlink 49053 (time 1781282246.40 total 315.90 last 148.58) - /unlink 50000 (time 1781282252.05 total 321.55 last 167.56) - /unlink 51627 (time 1781282262.05 total 331.55 last 162.61) - /unlink 53139 (time 1781282272.06 total 341.56 last 151.04) - /unlink 54753 (time 1781282282.07 total 351.57 last 161.36) - /unlink 56350 (time 1781282292.07 total 361.57 last 159.62) - /unlink 57826 (time 1781282302.07 total 371.57 last 147.59) - /unlink 59308 (time 1781282312.08 total 381.58 last 148.10) - /unlink 60000 (time 1781282317.07 total 386.57 last 138.66) - /unlink 61459 (time 1781282327.07 total 396.57 last 145.90) - /unlink 62890 (time 1781282337.07 total 406.57 last 143.08) - /unlink 64257 (time 1781282347.07 total 416.57 last 136.66) total: 64768 /unlink in 420.24 seconds: 154.12 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1053 (time 1781282382.93 total 10.00 last 105.29) - open/close 2031 (time 1781282392.93 total 20.00 last 97.79) - open/close 2992 (time 1781282402.93 total 30.00 last 96.07) - open/close 3967 (time 1781282412.94 total 40.02 last 97.36) - open/close 4864 (time 1781282422.95 total 50.03 last 89.61) - open/close 5767 (time 1781282432.96 total 60.04 last 90.23) - open/close 6587 (time 1781282442.96 total 70.04 last 81.99) - open/close 7477 (time 1781282452.97 total 80.04 last 88.94) - open/close 8378 (time 1781282462.98 total 90.05 last 90.02) - open/close 9250 (time 1781282472.98 total 100.06 last 87.15) - open/close 10000 (time 1781282481.71 total 108.79 last 85.90) - open/close 10820 (time 1781282491.72 total 118.80 last 81.92) - open/close 11710 (time 1781282501.73 total 128.81 last 88.93) - open/close 12510 (time 1781282511.74 total 138.81 last 79.95) - open/close 13391 (time 1781282521.74 total 148.82 last 88.08) - open/close 14256 (time 1781282531.75 total 158.83 last 86.42) - open/close 15124 (time 1781282541.76 total 168.83 last 86.73) - open/close 16011 (time 1781282551.76 total 178.83 last 88.68) - open/close 16820 (time 1781282561.76 total 188.84 last 80.88) - open/close 17635 (time 1781282571.77 total 198.84 last 81.47) - open/close 18467 (time 1781282581.77 total 208.84 last 83.17) - open/close 19255 (time 1781282591.78 total 218.85 last 78.75) - open/close 20000 (time 1781282600.59 total 227.67 last 84.49) - open/close 20850 (time 1781282610.60 total 237.67 last 84.97) - open/close 21817 (time 1781282620.61 total 247.68 last 96.59) - open/close 22720 (time 1781282630.61 total 257.69 last 90.26) - open/close 23538 (time 1781282640.62 total 267.69 last 81.76) - open/close 24365 (time 1781282650.63 total 277.70 last 82.64) - open/close 25172 (time 1781282660.63 total 287.70 last 80.68) - open/close 25997 (time 1781282670.64 total 297.72 last 82.39) - open/close 26813 (time 1781282680.66 total 307.73 last 81.48) - open/close 27636 (time 1781282690.66 total 317.73 last 82.29) - open/close 28433 (time 1781282700.67 total 327.74 last 79.62) - open/close 29272 (time 1781282710.67 total 337.75 last 83.86) - open/close 30000 (time 1781282718.97 total 346.04 last 87.74) - open/close 30849 (time 1781282728.98 total 356.05 last 84.84) - open/close 31669 (time 1781282738.99 total 366.07 last 81.87) - open/close 32544 (time 1781282749.00 total 376.08 last 87.41) - open/close 33393 (time 1781282759.01 total 386.08 last 84.84) - open/close 34156 (time 1781282769.01 total 396.09 last 76.27) - open/close 35021 (time 1781282779.02 total 406.09 last 86.47) - open/close 35870 (time 1781282789.02 total 416.10 last 84.86) - open/close 36773 (time 1781282799.03 total 426.11 last 90.21) - open/close 37613 (time 1781282809.04 total 436.12 last 83.90) - open/close 38407 (time 1781282819.05 total 446.13 last 79.34) - open/close 39187 (time 1781282829.06 total 456.14 last 77.93) - open/close 39943 (time 1781282839.07 total 466.15 last 75.52) - open/close 40757 (time 1781282849.08 total 476.15 last 81.35) - open/close 41564 (time 1781282859.08 total 486.15 last 80.69) - open/close 42362 (time 1781282869.08 total 496.15 last 79.79) - open/close 43168 (time 1781282879.08 total 506.16 last 80.59) - open/close 43978 (time 1781282889.09 total 516.17 last 80.92) - open/close 44755 (time 1781282899.12 total 526.19 last 77.50) - open/close 45560 (time 1781282909.12 total 536.20 last 80.43) - open/close 46321 (time 1781282919.13 total 546.21 last 76.05) - open/close 47078 (time 1781282929.13 total 556.21 last 75.69) - open/close 47940 (time 1781282939.17 total 566.25 last 85.86) - open/close 48781 (time 1781282949.18 total 576.25 last 84.05) - open/close 49630 (time 1781282959.18 total 586.26 last 84.86) - open/close 50000 (time 1781282963.56 total 590.64 last 84.50) - open/close 50906 (time 1781282973.57 total 600.64 last 90.53) - open/close 51762 (time 1781282983.58 total 610.66 last 85.48) - open/close 52666 (time 1781282993.59 total 620.66 last 90.36) - open/close 53584 (time 1781283003.60 total 630.68 last 91.67) - open/close 54375 (time 1781283013.61 total 640.69 last 79.01) - open/close 55170 (time 1781283023.62 total 650.70 last 79.42) - open/close 55975 (time 1781283033.62 total 660.70 last 80.50) - open/close 56788 (time 1781283043.64 total 670.71 last 81.21) - open/close 57642 (time 1781283053.64 total 680.71 last 85.39) - open/close 58490 (time 1781283063.64 total 690.72 last 84.75) - open/close 59231 (time 1781283073.65 total 700.72 last 74.04) - open/close 60000 (time 1781283082.20 total 709.27 last 89.98) - open/close 60850 (time 1781283092.21 total 719.28 last 84.89) - open/close 61766 (time 1781283102.22 total 729.29 last 91.54) - open/close 62797 (time 1781283112.22 total 739.29 last 103.06) - open/close 63687 (time 1781283122.23 total 749.31 last 88.89) - open/close 64554 (time 1781283132.23 total 759.31 last 86.69) total: 64768 open/close in 761.85 seconds: 85.01 ops/second - /unlink 1780 (time 1781283148.31 total 10.01 last 177.90) - /unlink 3520 (time 1781283158.32 total 20.01 last 173.85) - /unlink 5018 (time 1781283168.32 total 30.02 last 149.74) - /unlink 6432 (time 1781283178.32 total 40.02 last 141.33) - /unlink 7926 (time 1781283188.33 total 50.03 last 149.34) - /unlink 9315 (time 1781283198.33 total 60.03 last 138.86) - /unlink 10000 (time 1781283203.67 total 65.37 last 128.24) - /unlink 11373 (time 1781283213.68 total 75.38 last 137.23) - /unlink 12809 (time 1781283223.68 total 85.38 last 143.51) - /unlink 14228 (time 1781283233.69 total 95.39 last 141.87) - /unlink 15572 (time 1781283243.69 total 105.39 last 134.39) - /unlink 17020 (time 1781283253.69 total 115.39 last 144.79) - /unlink 18603 (time 1781283263.69 total 125.39 last 158.21) - /unlink 19941 (time 1781283273.70 total 135.40 last 133.67) - /unlink 21348 (time 1781283283.70 total 145.40 last 140.68) - /unlink 22789 (time 1781283293.70 total 155.40 last 144.10) - /unlink 24188 (time 1781283303.71 total 165.41 last 139.85) - /unlink 25618 (time 1781283313.71 total 175.41 last 142.96) - /unlink 27020 (time 1781283323.71 total 185.41 last 140.19) - /unlink 28575 (time 1781283333.72 total 195.42 last 155.41) - /unlink 29895 (time 1781283343.72 total 205.42 last 131.98) - /unlink 31241 (time 1781283353.74 total 215.44 last 134.36) - /unlink 32416 (time 1781283363.74 total 225.44 last 117.41) - /unlink 33691 (time 1781283373.75 total 235.44 last 127.48) - /unlink 35084 (time 1781283383.75 total 245.45 last 139.25) - /unlink 36609 (time 1781283393.76 total 255.46 last 152.31) - /unlink 38011 (time 1781283403.77 total 265.46 last 140.16) - /unlink 39466 (time 1781283413.77 total 275.47 last 145.45) - /unlink 40000 (time 1781283417.62 total 279.32 last 138.55) - /unlink 41553 (time 1781283427.62 total 289.32 last 155.30) - /unlink 42964 (time 1781283437.62 total 299.32 last 141.09) - /unlink 44217 (time 1781283447.63 total 309.33 last 125.18) - /unlink 45600 (time 1781283457.63 total 319.33 last 138.30) - /unlink 47009 (time 1781283467.64 total 329.34 last 140.80) - /unlink 48471 (time 1781283477.66 total 339.36 last 145.95) - /unlink 49869 (time 1781283487.66 total 349.36 last 139.74) - /unlink 51182 (time 1781283497.67 total 359.36 last 131.23) - /unlink 52510 (time 1781283507.67 total 369.37 last 132.78) - /unlink 53903 (time 1781283517.67 total 379.37 last 139.24) - /unlink 55198 (time 1781283527.67 total 389.37 last 129.50) - /unlink 56646 (time 1781283537.68 total 399.38 last 144.74) - /unlink 57986 (time 1781283547.68 total 409.38 last 133.99) - /unlink 59287 (time 1781283557.68 total 419.38 last 130.04) - /unlink 60000 (time 1781283563.30 total 425.00 last 126.96) - /unlink 61318 (time 1781283573.30 total 435.00 last 131.79) - /unlink 62777 (time 1781283583.31 total 445.01 last 145.77) - /unlink 64376 (time 1781283593.31 total 455.01 last 159.82) total: 64768 /unlink in 458.28 seconds: 141.33 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 852 (time 1781283632.43 total 10.00 last 85.20) - open/close 1764 (time 1781283642.43 total 20.00 last 91.19) - open/close 2750 (time 1781283652.44 total 30.01 last 98.51) - open/close 3689 (time 1781283662.45 total 40.02 last 93.83) - open/close 4595 (time 1781283672.45 total 50.02 last 90.54) - open/close 5449 (time 1781283682.46 total 60.03 last 85.37) - open/close 6314 (time 1781283692.46 total 70.03 last 86.48) - open/close 7110 (time 1781283702.47 total 80.04 last 79.53) - open/close 7815 (time 1781283712.48 total 90.05 last 70.45) - open/close 8547 (time 1781283722.48 total 100.05 last 73.20) - open/close 9343 (time 1781283732.48 total 110.05 last 79.59) - open/close 10000 (time 1781283740.46 total 118.03 last 82.28) - open/close 10858 (time 1781283750.46 total 128.04 last 85.78) - open/close 11827 (time 1781283760.47 total 138.04 last 96.86) - open/close 12742 (time 1781283770.47 total 148.04 last 91.48) - open/close 13637 (time 1781283780.47 total 158.05 last 89.48) - open/close 14549 (time 1781283790.49 total 168.06 last 91.04) - open/close 15313 (time 1781283800.50 total 178.07 last 76.31) - open/close 16024 (time 1781283810.51 total 188.08 last 71.07) - open/close 16747 (time 1781283820.52 total 198.09 last 72.24) - open/close 17510 (time 1781283830.53 total 208.10 last 76.21) - open/close 18267 (time 1781283840.53 total 218.10 last 75.69) - open/close 18910 (time 1781283850.55 total 228.12 last 64.20) - open/close 19720 (time 1781283860.55 total 238.13 last 80.92) - open/close 20000 (time 1781283864.03 total 241.60 last 80.50) - open/close 20864 (time 1781283874.04 total 251.61 last 86.32) - open/close 21689 (time 1781283884.04 total 261.61 last 82.50) - open/close 22514 (time 1781283894.04 total 271.61 last 82.49) - open/close 23389 (time 1781283904.05 total 281.62 last 87.44) - open/close 24247 (time 1781283914.06 total 291.63 last 85.76) - open/close 25125 (time 1781283924.06 total 301.63 last 87.79) - open/close 25936 (time 1781283934.07 total 311.64 last 80.96) - open/close 26579 (time 1781283944.10 total 321.67 last 64.15) - open/close 27274 (time 1781283954.11 total 331.68 last 69.41) - open/close 28055 (time 1781283964.11 total 341.68 last 78.08) - open/close 28733 (time 1781283974.12 total 351.69 last 67.76) - open/close 29576 (time 1781283984.12 total 361.69 last 84.29) - open/close 30000 (time 1781283989.34 total 366.91 last 81.15) - open/close 30775 (time 1781283999.35 total 376.92 last 77.48) - open/close 31505 (time 1781284009.35 total 386.92 last 72.97) - open/close 32289 (time 1781284019.35 total 396.92 last 78.38) - open/close 33119 (time 1781284029.37 total 406.94 last 82.89) - open/close 33939 (time 1781284039.37 total 416.94 last 81.95) - open/close 34669 (time 1781284049.39 total 426.96 last 72.88) - open/close 35393 (time 1781284059.39 total 436.96 last 72.39) - open/close 36170 (time 1781284069.39 total 446.96 last 77.67) - open/close 36872 (time 1781284079.40 total 456.97 last 70.18) - open/close 37669 (time 1781284089.40 total 466.97 last 79.65) - open/close 38500 (time 1781284099.41 total 476.98 last 83.06) - open/close 39301 (time 1781284109.41 total 486.98 last 80.09) - open/close 40000 (time 1781284118.39 total 495.96 last 77.82) - open/close 40862 (time 1781284128.44 total 506.02 last 85.73) - open/close 41564 (time 1781284138.45 total 516.02 last 70.15) - open/close 42400 (time 1781284148.46 total 526.03 last 83.57) - open/close 43169 (time 1781284158.46 total 536.03 last 76.86) - open/close 44059 (time 1781284168.46 total 546.04 last 88.96) - open/close 44866 (time 1781284178.47 total 556.04 last 80.67) - open/close 45545 (time 1781284188.51 total 566.08 last 67.62) - open/close 46127 (time 1781284198.52 total 576.09 last 58.17) - open/close 46849 (time 1781284208.52 total 586.09 last 72.17) - open/close 47619 (time 1781284218.53 total 596.10 last 76.95) - open/close 48413 (time 1781284228.53 total 606.11 last 79.34) - open/close 49214 (time 1781284238.54 total 616.11 last 80.05) - open/close 49989 (time 1781284248.56 total 626.13 last 77.32) - open/close 50742 (time 1781284258.57 total 636.14 last 75.23) - open/close 51524 (time 1781284268.58 total 646.15 last 78.17) - open/close 52340 (time 1781284278.59 total 656.16 last 81.51) - open/close 53120 (time 1781284288.59 total 666.17 last 77.95) - open/close 53817 (time 1781284298.60 total 676.17 last 69.69) - open/close 54594 (time 1781284308.60 total 686.17 last 77.67) - open/close 55366 (time 1781284318.60 total 696.17 last 77.19) - open/close 56180 (time 1781284328.60 total 706.17 last 81.39) - open/close 57012 (time 1781284338.60 total 716.17 last 83.19) - open/close 57805 (time 1781284348.64 total 726.21 last 79.01) - open/close 58591 (time 1781284358.65 total 736.22 last 78.56) - open/close 59459 (time 1781284368.66 total 746.23 last 86.64) - open/close 60000 (time 1781284375.35 total 752.92 last 80.86) - open/close 60811 (time 1781284385.37 total 762.94 last 80.95) - open/close 61597 (time 1781284395.38 total 772.95 last 78.55) - open/close 62329 (time 1781284405.39 total 782.96 last 73.14) - open/close 62946 (time 1781284415.39 total 792.96 last 61.69) - open/close 63589 (time 1781284425.41 total 802.98 last 64.17) - open/close 64425 (time 1781284435.42 total 812.99 last 83.54) total: 64768 open/close in 816.97 seconds: 79.28 ops/second - /unlink 1321 (time 1781284453.52 total 10.01 last 132.02) - /unlink 2639 (time 1781284463.52 total 20.01 last 131.76) - /unlink 4003 (time 1781284473.52 total 30.01 last 136.36) - /unlink 5351 (time 1781284483.53 total 40.01 last 134.79) - /unlink 6779 (time 1781284493.53 total 50.01 last 142.78) - /unlink 7957 (time 1781284503.54 total 60.02 last 117.69) - /unlink 9312 (time 1781284513.54 total 70.03 last 135.40) - /unlink 10000 (time 1781284518.52 total 75.01 last 138.19) - /unlink 11241 (time 1781284528.52 total 85.01 last 124.07) - /unlink 12507 (time 1781284538.53 total 95.02 last 126.53) - /unlink 13813 (time 1781284548.55 total 105.04 last 130.33) - /unlink 15058 (time 1781284558.57 total 115.06 last 124.25) - /unlink 16411 (time 1781284568.57 total 125.06 last 135.29) - /unlink 17732 (time 1781284578.57 total 135.06 last 132.08) - /unlink 19000 (time 1781284588.58 total 145.06 last 126.78) - /unlink 20000 (time 1781284596.70 total 153.18 last 123.13) - /unlink 21283 (time 1781284606.70 total 163.19 last 128.26) - /unlink 22395 (time 1781284616.70 total 173.19 last 111.17) - /unlink 23721 (time 1781284626.71 total 183.19 last 132.55) - /unlink 25052 (time 1781284636.71 total 193.19 last 133.09) - /unlink 26319 (time 1781284646.71 total 203.20 last 126.69) - /unlink 27546 (time 1781284656.71 total 213.20 last 122.63) - /unlink 28852 (time 1781284666.72 total 223.21 last 130.53) - /unlink 30000 (time 1781284675.44 total 231.93 last 131.64) - /unlink 31359 (time 1781284685.45 total 241.94 last 135.76) - /unlink 32585 (time 1781284695.45 total 251.94 last 122.56) - /unlink 33723 (time 1781284705.45 total 261.94 last 113.78) - /unlink 35029 (time 1781284715.46 total 271.95 last 130.55) - /unlink 36303 (time 1781284725.47 total 281.95 last 127.31) - /unlink 37648 (time 1781284735.47 total 291.95 last 134.49) - /unlink 39131 (time 1781284745.47 total 301.96 last 148.17) - /unlink 40000 (time 1781284751.64 total 308.13 last 140.95) - /unlink 41390 (time 1781284761.65 total 318.13 last 138.91) - /unlink 42839 (time 1781284771.65 total 328.14 last 144.89) - /unlink 44266 (time 1781284781.65 total 338.14 last 142.62) - /unlink 45510 (time 1781284791.65 total 348.14 last 124.39) - /unlink 46725 (time 1781284801.66 total 358.15 last 121.43) - /unlink 48057 (time 1781284811.67 total 368.15 last 133.11) - /unlink 49408 (time 1781284821.67 total 378.16 last 135.06) - /unlink 50000 (time 1781284826.62 total 383.11 last 119.51) - /unlink 51346 (time 1781284836.64 total 393.13 last 134.40) - /unlink 52615 (time 1781284846.64 total 403.13 last 126.85) - /unlink 53895 (time 1781284856.64 total 413.13 last 128.00) - /unlink 55171 (time 1781284866.64 total 423.13 last 127.59) - /unlink 56457 (time 1781284876.65 total 433.13 last 128.55) - /unlink 57726 (time 1781284886.65 total 443.13 last 126.90) - /unlink 58915 (time 1781284896.65 total 453.13 last 118.90) - /unlink 60000 (time 1781284904.64 total 461.13 last 135.78) - /unlink 61269 (time 1781284914.64 total 471.13 last 126.88) - /unlink 62567 (time 1781284924.64 total 481.13 last 129.79) - /unlink 63891 (time 1781284934.64 total 491.13 last 132.38) total: 64768 /unlink in 497.26 seconds: 130.25 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 752 (time 1781284992.04 total 10.00 last 75.19) - open/close 1609 (time 1781285002.05 total 20.01 last 85.65) - open/close 2443 (time 1781285012.05 total 30.01 last 83.35) - open/close 3159 (time 1781285022.06 total 40.02 last 71.57) - open/close 4058 (time 1781285032.07 total 50.03 last 89.78) - open/close 4882 (time 1781285042.07 total 60.04 last 82.36) - open/close 5680 (time 1781285052.09 total 70.05 last 79.71) - open/close 6473 (time 1781285062.09 total 80.05 last 79.27) - open/close 7236 (time 1781285072.09 total 90.05 last 76.29) - open/close 8091 (time 1781285082.09 total 100.06 last 85.46) - open/close 8912 (time 1781285092.10 total 110.06 last 82.04) - open/close 9689 (time 1781285102.11 total 120.07 last 77.67) - open/close 10000 (time 1781285106.32 total 124.28 last 73.79) - open/close 10801 (time 1781285116.32 total 134.28 last 80.09) - open/close 11593 (time 1781285126.33 total 144.29 last 79.17) - open/close 12386 (time 1781285136.33 total 154.29 last 79.25) - open/close 13193 (time 1781285146.34 total 164.30 last 80.62) - open/close 14103 (time 1781285156.35 total 174.31 last 90.93) - open/close 15056 (time 1781285166.36 total 184.32 last 95.21) - open/close 16020 (time 1781285176.36 total 194.32 last 96.38) - open/close 16922 (time 1781285186.37 total 204.33 last 90.14) - open/close 17776 (time 1781285196.38 total 214.34 last 85.28) - open/close 18644 (time 1781285206.39 total 224.35 last 86.74) - open/close 19445 (time 1781285216.39 total 234.36 last 80.06) - open/close 20000 (time 1781285223.68 total 241.65 last 76.13) - open/close 20687 (time 1781285233.70 total 251.66 last 68.57) - open/close 21438 (time 1781285243.71 total 261.67 last 75.03) - open/close 22204 (time 1781285253.73 total 271.69 last 76.50) - open/close 23023 (time 1781285263.73 total 281.69 last 81.85) - open/close 23898 (time 1781285273.73 total 291.69 last 87.50) - open/close 24762 (time 1781285283.74 total 301.70 last 86.34) - open/close 25576 (time 1781285293.74 total 311.70 last 81.37) - open/close 26430 (time 1781285303.74 total 321.71 last 85.38) - open/close 27289 (time 1781285313.75 total 331.71 last 85.85) - open/close 28135 (time 1781285323.75 total 341.72 last 84.57) - open/close 28947 (time 1781285333.76 total 351.72 last 81.14) - open/close 29716 (time 1781285343.76 total 361.73 last 76.88) - open/close 30000 (time 1781285347.57 total 365.53 last 74.59) - open/close 30820 (time 1781285357.58 total 375.54 last 81.94) - open/close 31637 (time 1781285367.59 total 385.55 last 81.65) - open/close 32447 (time 1781285377.60 total 395.56 last 80.91) - open/close 33307 (time 1781285387.61 total 405.57 last 85.92) - open/close 34105 (time 1781285397.61 total 415.57 last 79.78) - open/close 34901 (time 1781285407.61 total 425.57 last 79.60) - open/close 35693 (time 1781285417.61 total 435.57 last 79.18) - open/close 36472 (time 1781285427.62 total 445.58 last 77.82) - open/close 37304 (time 1781285437.62 total 455.59 last 83.19) - open/close 38029 (time 1781285447.64 total 465.60 last 72.40) - open/close 38685 (time 1781285457.65 total 475.61 last 65.53) - open/close 39328 (time 1781285467.68 total 485.64 last 64.08) - open/close 40000 (time 1781285476.22 total 494.18 last 78.74) - open/close 40762 (time 1781285486.22 total 504.18 last 76.14) - open/close 41610 (time 1781285496.24 total 514.20 last 84.66) - open/close 42398 (time 1781285506.24 total 524.20 last 78.80) - open/close 43168 (time 1781285516.27 total 534.23 last 76.80) - open/close 43900 (time 1781285526.27 total 544.23 last 73.18) - open/close 44666 (time 1781285536.27 total 554.23 last 76.57) - open/close 45486 (time 1781285546.28 total 564.24 last 81.96) - open/close 46288 (time 1781285556.28 total 574.24 last 80.16) - open/close 47060 (time 1781285566.28 total 584.25 last 77.17) - open/close 47802 (time 1781285576.29 total 594.25 last 74.20) - open/close 48560 (time 1781285586.29 total 604.26 last 75.74) - open/close 49269 (time 1781285596.29 total 614.26 last 70.90) - open/close 49995 (time 1781285606.31 total 624.27 last 72.48) - open/close 50716 (time 1781285616.32 total 634.28 last 72.06) - open/close 51456 (time 1781285626.34 total 644.30 last 73.85) - open/close 52162 (time 1781285636.34 total 654.30 last 70.59) - open/close 52960 (time 1781285646.34 total 664.30 last 79.77) - open/close 53726 (time 1781285656.35 total 674.32 last 76.50) - open/close 54503 (time 1781285666.35 total 684.32 last 77.69) - open/close 55287 (time 1781285676.36 total 694.32 last 78.36) - open/close 56097 (time 1781285686.36 total 704.33 last 80.97) - open/close 56683 (time 1781285696.38 total 714.34 last 58.53) - open/close 57443 (time 1781285706.38 total 724.34 last 75.97) - open/close 58136 (time 1781285716.39 total 734.35 last 69.24) - open/close 58867 (time 1781285726.41 total 744.37 last 72.95) - open/close 59688 (time 1781285736.41 total 754.37 last 82.08) - open/close 60000 (time 1781285740.28 total 758.24 last 80.63) - open/close 60816 (time 1781285750.30 total 768.26 last 81.42) - open/close 61653 (time 1781285760.30 total 778.26 last 83.69) - open/close 62503 (time 1781285770.30 total 788.27 last 84.99) - open/close 63212 (time 1781285780.31 total 798.27 last 70.89) - open/close 63934 (time 1781285790.31 total 808.27 last 72.16) - open/close 64641 (time 1781285800.32 total 818.28 last 70.65) total: 64768 open/close in 820.02 seconds: 78.98 ops/second - /unlink 1267 (time 1781285815.72 total 10.00 last 126.65) - /unlink 2556 (time 1781285825.72 total 20.01 last 128.80) - /unlink 3967 (time 1781285835.73 total 30.02 last 141.04) - /unlink 5428 (time 1781285845.73 total 40.02 last 146.00) - /unlink 6745 (time 1781285855.74 total 50.03 last 131.67) - /unlink 8107 (time 1781285865.74 total 60.03 last 136.18) - /unlink 9597 (time 1781285875.74 total 70.03 last 148.93) - /unlink 10000 (time 1781285878.39 total 72.68 last 152.40) - /unlink 11418 (time 1781285888.39 total 82.68 last 141.77) - /unlink 12842 (time 1781285898.40 total 92.68 last 142.32) - /unlink 14118 (time 1781285908.41 total 102.69 last 127.47) - /unlink 15534 (time 1781285918.41 total 112.70 last 141.58) - /unlink 16869 (time 1781285928.41 total 122.70 last 133.45) - /unlink 18296 (time 1781285938.42 total 132.71 last 142.63) - /unlink 19540 (time 1781285948.42 total 142.71 last 124.34) - /unlink 20000 (time 1781285951.85 total 146.14 last 134.20) - /unlink 21321 (time 1781285961.85 total 156.14 last 132.07) - /unlink 22651 (time 1781285971.86 total 166.15 last 132.88) - /unlink 24098 (time 1781285981.86 total 176.15 last 144.63) - /unlink 25473 (time 1781285991.86 total 186.15 last 137.50) - /unlink 26805 (time 1781286001.87 total 196.16 last 133.18) - /unlink 28244 (time 1781286011.88 total 206.17 last 143.74) - /unlink 29626 (time 1781286021.88 total 216.17 last 138.13) - /unlink 30000 (time 1781286024.14 total 218.43 last 165.88) - /unlink 31480 (time 1781286034.14 total 228.43 last 147.95) - /unlink 32983 (time 1781286044.14 total 238.43 last 150.28) - /unlink 34288 (time 1781286054.14 total 248.43 last 130.49) - /unlink 35617 (time 1781286064.15 total 258.44 last 132.85) - /unlink 36941 (time 1781286074.15 total 268.44 last 132.40) - /unlink 38229 (time 1781286084.15 total 278.44 last 128.75) - /unlink 39430 (time 1781286094.16 total 288.44 last 120.04) - /unlink 40000 (time 1781286098.73 total 293.02 last 124.63) - /unlink 41284 (time 1781286108.74 total 303.02 last 128.32) - /unlink 42589 (time 1781286118.74 total 313.02 last 130.50) - /unlink 43843 (time 1781286128.75 total 323.04 last 125.26) - /unlink 45149 (time 1781286138.75 total 333.04 last 130.55) - /unlink 46524 (time 1781286148.76 total 343.05 last 137.37) - /unlink 47868 (time 1781286158.76 total 353.05 last 134.40) - /unlink 49166 (time 1781286168.76 total 363.05 last 129.75) - /unlink 50000 (time 1781286174.87 total 369.16 last 136.53) - /unlink 51293 (time 1781286184.88 total 379.17 last 129.23) - /unlink 52528 (time 1781286194.88 total 389.17 last 123.49) - /unlink 53875 (time 1781286204.89 total 399.17 last 134.60) - /unlink 55193 (time 1781286214.89 total 409.18 last 131.76) - /unlink 56528 (time 1781286224.91 total 419.20 last 133.23) - /unlink 57785 (time 1781286234.92 total 429.21 last 125.52) - /unlink 59040 (time 1781286244.92 total 439.21 last 125.49) - /unlink 60000 (time 1781286253.62 total 447.91 last 110.39) - /unlink 61250 (time 1781286263.63 total 457.92 last 124.86) - /unlink 62542 (time 1781286273.64 total 467.93 last 129.13) - /unlink 63905 (time 1781286283.64 total 477.93 last 136.29) total: 64768 /unlink in 484.94 seconds: 133.56 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 810 (time 1781286321.93 total 10.01 last 80.93) - open/close 1647 (time 1781286331.93 total 20.01 last 83.69) - open/close 2519 (time 1781286341.94 total 30.01 last 87.16) - open/close 3408 (time 1781286351.94 total 40.02 last 88.87) - open/close 4299 (time 1781286361.95 total 50.03 last 89.04) - open/close 5213 (time 1781286371.95 total 60.03 last 91.37) - open/close 6032 (time 1781286381.96 total 70.03 last 81.87) - open/close 6828 (time 1781286391.96 total 80.03 last 79.58) - open/close 7484 (time 1781286401.98 total 90.05 last 65.48) - open/close 8237 (time 1781286411.99 total 100.06 last 75.23) - open/close 8968 (time 1781286422.00 total 110.07 last 73.04) - open/close 9711 (time 1781286432.00 total 120.07 last 74.29) - open/close 10000 (time 1781286434.99 total 123.07 last 96.43) - open/close 10810 (time 1781286445.00 total 133.08 last 80.91) - open/close 11509 (time 1781286455.01 total 143.08 last 69.86) - open/close 12327 (time 1781286465.02 total 153.09 last 81.72) - open/close 13195 (time 1781286475.02 total 163.09 last 86.80) - open/close 14001 (time 1781286485.03 total 173.10 last 80.54) - open/close 14764 (time 1781286495.04 total 183.12 last 76.18) - open/close 15496 (time 1781286505.05 total 193.13 last 73.14) - open/close 16223 (time 1781286515.06 total 203.13 last 72.65) - open/close 17006 (time 1781286525.07 total 213.14 last 78.22) - open/close 17766 (time 1781286535.07 total 223.14 last 75.99) - open/close 18586 (time 1781286545.07 total 233.15 last 81.96) - open/close 19408 (time 1781286555.08 total 243.15 last 82.18) - open/close 20000 (time 1781286562.42 total 250.49 last 80.61) - open/close 20730 (time 1781286572.42 total 260.50 last 72.97) - open/close 21409 (time 1781286582.43 total 270.51 last 67.85) - open/close 22226 (time 1781286592.43 total 280.51 last 81.67) - open/close 23031 (time 1781286602.45 total 290.52 last 80.40) - open/close 23861 (time 1781286612.45 total 300.53 last 82.96) - open/close 24675 (time 1781286622.45 total 310.53 last 81.40) - open/close 25514 (time 1781286632.46 total 320.54 last 83.83) - open/close 26284 (time 1781286642.46 total 330.54 last 76.99) - open/close 26972 (time 1781286652.51 total 340.58 last 68.50) - open/close 27675 (time 1781286662.51 total 350.58 last 70.27) - open/close 28349 (time 1781286672.53 total 360.60 last 67.28) - open/close 29027 (time 1781286682.54 total 370.61 last 67.73) - open/close 29769 (time 1781286692.54 total 380.62 last 74.16) - open/close 30000 (time 1781286695.90 total 383.98 last 68.71) - open/close 30725 (time 1781286705.92 total 394.00 last 72.38) - open/close 31442 (time 1781286715.93 total 404.00 last 71.64) - open/close 32229 (time 1781286725.93 total 414.01 last 78.69) - open/close 33007 (time 1781286735.94 total 424.01 last 77.77) - open/close 33762 (time 1781286745.94 total 434.02 last 75.45) - open/close 34515 (time 1781286755.97 total 444.04 last 75.12) - open/close 35319 (time 1781286765.97 total 454.04 last 80.39) - open/close 36104 (time 1781286775.97 total 464.04 last 78.49) - open/close 36842 (time 1781286785.98 total 474.05 last 73.73) - open/close 37550 (time 1781286796.00 total 484.08 last 70.63) - open/close 38280 (time 1781286806.02 total 494.09 last 72.91) - open/close 39051 (time 1781286816.03 total 504.10 last 77.02) - open/close 39765 (time 1781286826.03 total 514.11 last 71.34) - open/close 40000 (time 1781286829.62 total 517.70 last 65.46) - open/close 40728 (time 1781286839.63 total 527.70 last 72.75) - open/close 41527 (time 1781286849.63 total 537.71 last 79.89) - open/close 42322 (time 1781286859.64 total 547.72 last 79.40) - open/close 43060 (time 1781286869.65 total 557.72 last 73.79) - open/close 43819 (time 1781286879.65 total 567.72 last 75.88) - open/close 44563 (time 1781286889.65 total 577.73 last 74.36) - open/close 45317 (time 1781286899.66 total 587.73 last 75.38) - open/close 46065 (time 1781286909.66 total 597.74 last 74.76) - open/close 46854 (time 1781286919.67 total 607.75 last 78.80) - open/close 47638 (time 1781286929.68 total 617.75 last 78.39) - open/close 48348 (time 1781286939.68 total 627.76 last 70.96) - open/close 49122 (time 1781286949.68 total 637.76 last 77.40) - open/close 49894 (time 1781286959.68 total 647.76 last 77.17) - open/close 50693 (time 1781286969.70 total 657.77 last 79.80) - open/close 51485 (time 1781286979.71 total 667.79 last 79.07) - open/close 52265 (time 1781286989.72 total 677.79 last 77.98) - open/close 53014 (time 1781286999.72 total 687.80 last 74.87) - open/close 53790 (time 1781287009.73 total 697.81 last 77.52) - open/close 54592 (time 1781287019.74 total 707.81 last 80.17) - open/close 55445 (time 1781287029.74 total 717.82 last 85.26) - open/close 56163 (time 1781287039.75 total 727.83 last 71.73) - open/close 56894 (time 1781287049.75 total 737.83 last 73.08) - open/close 57696 (time 1781287059.76 total 747.84 last 80.12) - open/close 58450 (time 1781287069.77 total 757.84 last 75.37) - open/close 59254 (time 1781287079.77 total 767.85 last 80.35) - open/close 60000 (time 1781287089.56 total 777.64 last 76.22) - open/close 60800 (time 1781287099.57 total 787.65 last 79.91) - open/close 61595 (time 1781287109.59 total 797.67 last 79.34) - open/close 62385 (time 1781287119.60 total 807.67 last 78.98) - open/close 63123 (time 1781287129.60 total 817.67 last 73.79) - open/close 63834 (time 1781287139.60 total 827.67 last 71.08) - open/close 64574 (time 1781287149.60 total 837.68 last 73.99) total: 64768 open/close in 840.42 seconds: 77.07 ops/second - /unlink 1138 (time 1781287167.65 total 10.00 last 113.76) - /unlink 2172 (time 1781287177.66 total 20.01 last 103.31) - /unlink 3305 (time 1781287187.66 total 30.02 last 113.26) - /unlink 4380 (time 1781287197.66 total 40.02 last 107.50) - /unlink 5602 (time 1781287207.66 total 50.02 last 122.14) - /unlink 6789 (time 1781287217.67 total 60.02 last 118.68) - /unlink 7990 (time 1781287227.67 total 70.02 last 120.10) - /unlink 9220 (time 1781287237.67 total 80.02 last 122.97) - /unlink 10000 (time 1781287244.42 total 86.78 last 115.44) - /unlink 11191 (time 1781287254.43 total 96.79 last 119.05) - /unlink 12350 (time 1781287264.43 total 106.79 last 115.88) - /unlink 13503 (time 1781287274.44 total 116.80 last 115.16) - /unlink 14577 (time 1781287284.45 total 126.80 last 107.36) - /unlink 15597 (time 1781287294.45 total 136.81 last 101.93) - /unlink 16652 (time 1781287304.47 total 146.82 last 105.34) - /unlink 17718 (time 1781287314.48 total 156.84 last 106.44) - /unlink 18827 (time 1781287324.48 total 166.84 last 110.88) - /unlink 19860 (time 1781287334.49 total 176.84 last 103.26) - /unlink 20995 (time 1781287344.50 total 186.85 last 113.39) - /unlink 22144 (time 1781287354.50 total 196.86 last 114.89) - /unlink 23274 (time 1781287364.52 total 206.88 last 112.74) - /unlink 24449 (time 1781287374.54 total 216.90 last 117.29) - /unlink 25561 (time 1781287384.54 total 226.90 last 111.16) - /unlink 26768 (time 1781287394.55 total 236.91 last 120.61) - /unlink 27892 (time 1781287404.56 total 246.91 last 112.34) - /unlink 29069 (time 1781287414.56 total 256.91 last 117.68) - /unlink 30000 (time 1781287422.46 total 264.82 last 117.80) - /unlink 31216 (time 1781287432.47 total 274.82 last 121.51) - /unlink 32365 (time 1781287442.47 total 284.83 last 114.86) - /unlink 33526 (time 1781287452.47 total 294.83 last 116.09) - /unlink 34575 (time 1781287462.48 total 304.83 last 104.85) - /unlink 35754 (time 1781287472.48 total 314.84 last 117.86) - /unlink 36998 (time 1781287482.49 total 324.84 last 124.31) - /unlink 38311 (time 1781287492.49 total 334.85 last 131.20) - /unlink 39571 (time 1781287502.53 total 344.88 last 125.59) - /unlink 40000 (time 1781287506.19 total 348.55 last 117.01) - /unlink 41167 (time 1781287516.19 total 358.55 last 116.69) - /unlink 42429 (time 1781287526.21 total 368.56 last 126.02) - /unlink 43640 (time 1781287536.21 total 378.57 last 121.09) - /unlink 44858 (time 1781287546.21 total 388.57 last 121.79) - /unlink 46159 (time 1781287556.22 total 398.57 last 130.00) - /unlink 47360 (time 1781287566.22 total 408.58 last 120.08) - /unlink 48598 (time 1781287576.23 total 418.59 last 123.68) - /unlink 49841 (time 1781287586.23 total 428.59 last 124.25) - /unlink 51154 (time 1781287596.24 total 438.60 last 131.24) - /unlink 52314 (time 1781287606.25 total 448.61 last 115.87) - /unlink 53457 (time 1781287616.25 total 458.61 last 114.27) - /unlink 54627 (time 1781287626.26 total 468.61 last 116.96) - /unlink 55887 (time 1781287636.26 total 478.62 last 125.90) - /unlink 57124 (time 1781287646.29 total 488.65 last 123.38) - /unlink 58361 (time 1781287656.31 total 498.67 last 123.40) - /unlink 59545 (time 1781287666.33 total 508.68 last 118.25) - /unlink 60000 (time 1781287669.82 total 512.18 last 130.09) - /unlink 61324 (time 1781287679.83 total 522.19 last 132.33) - /unlink 62527 (time 1781287689.83 total 532.19 last 120.24) - /unlink 63664 (time 1781287699.84 total 542.20 last 113.63) total: 64768 /unlink in 551.06 seconds: 117.53 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg647-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg647-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg647-server stop mds service on oleg647-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg647-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg647-server' oleg647-server: oleg647-server.virtnet: executing unload_modules_local oleg647-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg647-server: LNET ready to unload oleg647-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg647-client: oleg647-client: ssh exited with exit code 2 pdsh@oleg647-client: oleg647-server: ssh exited with exit code 2 pdsh@oleg647-client: oleg647-server: ssh exited with exit code 2 pdsh@oleg647-client: oleg647-client: ssh exited with exit code 2