== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:56:13 (1763153773) Stopping clients: oleg344-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg344-client.virtnet /mnt/lustre2 (opts:-f) oleg344-server: oleg344-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg344-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg344-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 732 (time 1763153872.61 total 10.01 last 73.12) - open/close 1509 (time 1763153882.61 total 20.02 last 77.66) - open/close 2324 (time 1763153892.62 total 30.02 last 81.47) - open/close 3131 (time 1763153902.62 total 40.03 last 80.65) - open/close 3910 (time 1763153912.62 total 50.03 last 77.90) - open/close 4674 (time 1763153922.63 total 60.03 last 76.35) - open/close 5467 (time 1763153932.64 total 70.04 last 79.24) - open/close 6101 (time 1763153942.64 total 80.05 last 63.36) - open/close 6775 (time 1763153952.64 total 90.05 last 67.40) - open/close 7463 (time 1763153962.65 total 100.05 last 68.79) - open/close 8182 (time 1763153972.65 total 110.05 last 71.89) - open/close 8920 (time 1763153982.66 total 120.06 last 73.70) - open/close 9668 (time 1763153992.66 total 130.07 last 74.79) - open/close 10000 (time 1763153997.12 total 134.53 last 74.44) - open/close 10753 (time 1763154007.13 total 144.53 last 75.26) - open/close 11525 (time 1763154017.14 total 154.54 last 77.15) - open/close 12241 (time 1763154027.14 total 164.55 last 71.54) - open/close 12982 (time 1763154037.15 total 174.55 last 74.05) - open/close 13721 (time 1763154047.15 total 184.56 last 73.88) - open/close 14432 (time 1763154057.16 total 194.56 last 71.09) - open/close 15165 (time 1763154067.16 total 204.56 last 73.30) - open/close 15950 (time 1763154077.17 total 214.57 last 78.41) - open/close 16665 (time 1763154087.17 total 224.58 last 71.47) - open/close 17449 (time 1763154097.18 total 234.58 last 78.37) - open/close 18229 (time 1763154107.18 total 244.59 last 77.96) - open/close 19010 (time 1763154117.18 total 254.59 last 78.08) - open/close 19722 (time 1763154127.19 total 264.60 last 71.14) - open/close 20000 (time 1763154130.89 total 268.29 last 75.19) - open/close 20815 (time 1763154140.90 total 278.30 last 81.45) - open/close 21630 (time 1763154150.90 total 288.31 last 81.44) - open/close 22313 (time 1763154160.91 total 298.31 last 68.27) - open/close 23074 (time 1763154170.92 total 308.32 last 76.05) - open/close 23801 (time 1763154180.93 total 318.33 last 72.59) - open/close 24501 (time 1763154190.94 total 328.34 last 69.96) - open/close 25226 (time 1763154200.94 total 338.35 last 72.44) - open/close 25911 (time 1763154210.95 total 348.35 last 68.49) - open/close 26608 (time 1763154220.96 total 358.36 last 69.63) - open/close 27366 (time 1763154230.96 total 368.36 last 75.80) - open/close 28156 (time 1763154240.96 total 378.36 last 78.99) - open/close 28944 (time 1763154250.97 total 388.38 last 78.68) - open/close 29706 (time 1763154260.98 total 398.38 last 76.15) - open/close 30000 (time 1763154265.23 total 402.63 last 69.16) - open/close 30686 (time 1763154275.24 total 412.64 last 68.55) - open/close 31406 (time 1763154285.26 total 422.66 last 71.85) - open/close 32214 (time 1763154295.27 total 432.67 last 80.71) - open/close 32987 (time 1763154305.27 total 442.67 last 77.29) - open/close 33691 (time 1763154315.28 total 452.69 last 70.31) - open/close 34401 (time 1763154325.29 total 462.69 last 70.95) - open/close 35197 (time 1763154335.29 total 472.70 last 79.59) - open/close 36070 (time 1763154345.30 total 482.70 last 87.27) - open/close 36870 (time 1763154355.33 total 492.73 last 79.74) - open/close 37644 (time 1763154365.34 total 502.75 last 77.29) - open/close 38448 (time 1763154375.35 total 512.76 last 80.30) - open/close 39256 (time 1763154385.36 total 522.76 last 80.76) - open/close 40000 (time 1763154394.85 total 532.26 last 78.36) - open/close 40769 (time 1763154404.86 total 542.26 last 76.89) - open/close 41488 (time 1763154414.86 total 552.26 last 71.88) - open/close 42194 (time 1763154424.87 total 562.27 last 70.53) - open/close 42873 (time 1763154434.87 total 572.27 last 67.89) - open/close 43648 (time 1763154444.88 total 582.28 last 77.44) - open/close 44403 (time 1763154454.89 total 592.29 last 75.43) - open/close 45153 (time 1763154464.89 total 602.29 last 74.97) - open/close 45839 (time 1763154474.90 total 612.31 last 68.51) - open/close 46530 (time 1763154484.91 total 622.31 last 69.07) - open/close 47246 (time 1763154494.91 total 632.31 last 71.59) - open/close 47954 (time 1763154504.91 total 642.32 last 70.77) - open/close 48637 (time 1763154514.92 total 652.33 last 68.24) - open/close 49239 (time 1763154524.94 total 662.34 last 60.13) - open/close 49897 (time 1763154534.94 total 672.34 last 65.79) - open/close 50520 (time 1763154544.96 total 682.37 last 62.13) - open/close 51083 (time 1763154554.98 total 692.38 last 56.23) - open/close 51738 (time 1763154564.98 total 702.38 last 65.48) - open/close 52454 (time 1763154574.98 total 712.39 last 71.58) - open/close 53161 (time 1763154584.99 total 722.39 last 70.66) - open/close 53913 (time 1763154594.99 total 732.39 last 75.18) - open/close 54657 (time 1763154605.00 total 742.40 last 74.35) - open/close 55353 (time 1763154615.00 total 752.40 last 69.57) - open/close 56113 (time 1763154625.01 total 762.42 last 75.90) - open/close 56839 (time 1763154635.02 total 772.42 last 72.59) - open/close 57591 (time 1763154645.03 total 782.43 last 75.13) - open/close 58309 (time 1763154655.04 total 792.44 last 71.72) - open/close 58995 (time 1763154665.06 total 802.46 last 68.44) - open/close 59692 (time 1763154675.06 total 812.47 last 69.67) - open/close 60000 (time 1763154679.77 total 817.17 last 65.45) - open/close 60660 (time 1763154689.79 total 827.19 last 65.89) - open/close 61263 (time 1763154699.80 total 837.20 last 60.21) - open/close 61831 (time 1763154709.81 total 847.21 last 56.78) - open/close 62528 (time 1763154719.83 total 857.23 last 69.53) - open/close 63227 (time 1763154729.83 total 867.24 last 69.86) - open/close 63938 (time 1763154739.84 total 877.24 last 71.06) - open/close 64669 (time 1763154749.85 total 887.26 last 73.01) total: 64768 open/close in 888.66 seconds: 72.88 ops/second - /unlink 746 (time 1763154765.51 total 10.01 last 74.52) - /unlink 1463 (time 1763154775.52 total 20.02 last 71.61) - /unlink 2166 (time 1763154785.53 total 30.03 last 70.24) - /unlink 2759 (time 1763154795.55 total 40.05 last 59.20) - /unlink 3460 (time 1763154805.56 total 50.06 last 70.04) - /unlink 4150 (time 1763154815.57 total 60.07 last 68.95) - /unlink 4874 (time 1763154825.58 total 70.08 last 72.31) - /unlink 5557 (time 1763154835.59 total 80.09 last 68.22) - /unlink 6259 (time 1763154845.59 total 90.09 last 70.17) - /unlink 6929 (time 1763154855.62 total 100.12 last 66.86) - /unlink 7610 (time 1763154865.63 total 110.13 last 68.01) - /unlink 8319 (time 1763154875.64 total 120.14 last 70.85) - /unlink 9037 (time 1763154885.65 total 130.15 last 71.68) - /unlink 9752 (time 1763154895.68 total 140.18 last 71.29) - /unlink 10000 (time 1763154898.76 total 143.26 last 80.70) - /unlink 10740 (time 1763154908.76 total 153.26 last 73.95) - /unlink 11464 (time 1763154918.76 total 163.26 last 72.38) - /unlink 12232 (time 1763154928.78 total 173.28 last 76.71) - /unlink 12941 (time 1763154938.79 total 183.29 last 70.83) - /unlink 13696 (time 1763154948.79 total 193.29 last 75.47) - /unlink 14426 (time 1763154958.82 total 203.32 last 72.80) - /unlink 15147 (time 1763154968.83 total 213.33 last 72.01) - /unlink 15841 (time 1763154978.83 total 223.33 last 69.39) - /unlink 16511 (time 1763154988.84 total 233.34 last 66.95) - /unlink 17143 (time 1763154998.85 total 243.35 last 63.15) - /unlink 17823 (time 1763155008.85 total 253.35 last 67.97) - /unlink 18527 (time 1763155018.88 total 263.38 last 70.21) - /unlink 19210 (time 1763155028.88 total 273.38 last 68.29) - /unlink 19959 (time 1763155038.89 total 283.39 last 74.81) - /unlink 20726 (time 1763155048.90 total 293.40 last 76.68) - /unlink 21498 (time 1763155058.90 total 303.40 last 77.14) - /unlink 22245 (time 1763155068.91 total 313.41 last 74.68) - /unlink 22998 (time 1763155078.91 total 323.41 last 75.27) - /unlink 23796 (time 1763155088.92 total 333.42 last 79.75) - /unlink 24543 (time 1763155098.92 total 343.42 last 74.70) - /unlink 25302 (time 1763155108.93 total 353.43 last 75.82) - /unlink 26073 (time 1763155118.95 total 363.45 last 76.95) - /unlink 26748 (time 1763155128.95 total 373.45 last 67.45) - /unlink 27415 (time 1763155138.96 total 383.46 last 66.66) - /unlink 28124 (time 1763155148.96 total 393.46 last 70.89) - /unlink 28835 (time 1763155158.97 total 403.47 last 71.03) - /unlink 29562 (time 1763155168.98 total 413.48 last 72.67) - /unlink 30000 (time 1763155174.99 total 419.49 last 72.77) - /unlink 30774 (time 1763155185.00 total 429.50 last 77.37) - /unlink 31551 (time 1763155195.01 total 439.51 last 77.64) - /unlink 32283 (time 1763155205.01 total 449.51 last 73.18) - /unlink 32961 (time 1763155215.02 total 459.52 last 67.75) - /unlink 33632 (time 1763155225.02 total 469.52 last 67.09) - /unlink 34345 (time 1763155235.03 total 479.53 last 71.25) - /unlink 35069 (time 1763155245.03 total 489.53 last 72.34) - /unlink 35821 (time 1763155255.04 total 499.54 last 75.15) - /unlink 36576 (time 1763155265.05 total 509.55 last 75.46) - /unlink 37233 (time 1763155275.06 total 519.56 last 65.63) - /unlink 37914 (time 1763155285.06 total 529.56 last 68.08) - /unlink 38599 (time 1763155295.06 total 539.56 last 68.49) - /unlink 39289 (time 1763155305.08 total 549.58 last 68.87) - /unlink 39980 (time 1763155315.09 total 559.59 last 69.05) - /unlink 40697 (time 1763155325.10 total 569.60 last 71.64) - /unlink 41375 (time 1763155335.10 total 579.60 last 67.78) - /unlink 42105 (time 1763155345.10 total 589.60 last 72.97) - /unlink 42817 (time 1763155355.11 total 599.61 last 71.17) - /unlink 43554 (time 1763155365.12 total 609.62 last 73.61) - /unlink 44310 (time 1763155375.13 total 619.63 last 75.50) - /unlink 45076 (time 1763155385.15 total 629.65 last 76.50) - /unlink 45811 (time 1763155395.15 total 639.65 last 73.46) - /unlink 46524 (time 1763155405.16 total 649.66 last 71.25) - /unlink 47267 (time 1763155415.20 total 659.70 last 74.01) - /unlink 47996 (time 1763155425.20 total 669.70 last 72.85) - /unlink 48683 (time 1763155435.21 total 679.71 last 68.65) - /unlink 49363 (time 1763155445.22 total 689.72 last 67.95) - /unlink 50000 (time 1763155454.42 total 698.92 last 69.25) - /unlink 50710 (time 1763155464.43 total 708.93 last 70.92) - /unlink 51382 (time 1763155474.44 total 718.94 last 67.15) - /unlink 52074 (time 1763155484.44 total 728.94 last 69.16) - /unlink 52717 (time 1763155494.46 total 738.96 last 64.20) - /unlink 53427 (time 1763155504.46 total 748.96 last 70.99) - /unlink 54159 (time 1763155514.47 total 758.96 last 73.16) - /unlink 54922 (time 1763155524.47 total 768.97 last 76.24) - /unlink 55660 (time 1763155534.47 total 778.97 last 73.79) - /unlink 56431 (time 1763155544.48 total 788.98 last 77.05) - /unlink 57127 (time 1763155554.49 total 798.99 last 69.55) - /unlink 57864 (time 1763155564.49 total 808.99 last 73.65) - /unlink 58598 (time 1763155574.50 total 819.00 last 73.38) - /unlink 59343 (time 1763155584.51 total 829.01 last 74.38) - /unlink 60000 (time 1763155593.55 total 838.05 last 72.69) - /unlink 60786 (time 1763155603.55 total 848.05 last 78.59) - /unlink 61571 (time 1763155613.56 total 858.06 last 78.43) - /unlink 62365 (time 1763155623.57 total 868.06 last 79.37) - /unlink 63080 (time 1763155633.57 total 878.07 last 71.48) - /unlink 63793 (time 1763155643.59 total 888.09 last 71.13) - /unlink 64544 (time 1763155653.60 total 898.10 last 75.08) total: 64768 /unlink in 901.35 seconds: 71.86 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 758 (time 1763155693.32 total 10.00 last 75.79) - open/close 1459 (time 1763155703.33 total 20.01 last 70.06) - open/close 2086 (time 1763155713.33 total 30.01 last 62.66) - open/close 2769 (time 1763155723.33 total 40.01 last 68.30) - open/close 3568 (time 1763155733.35 total 50.03 last 79.76) - open/close 4315 (time 1763155743.35 total 60.03 last 74.68) - open/close 5090 (time 1763155753.35 total 70.04 last 77.48) - open/close 5749 (time 1763155763.35 total 80.04 last 65.90) - open/close 6358 (time 1763155773.36 total 90.05 last 60.84) - open/close 7105 (time 1763155783.37 total 100.05 last 74.67) - open/close 7924 (time 1763155793.37 total 110.05 last 81.90) - open/close 8711 (time 1763155803.38 total 120.06 last 78.64) - open/close 9490 (time 1763155813.38 total 130.06 last 77.89) - open/close 10000 (time 1763155819.94 total 136.63 last 77.68) - open/close 10763 (time 1763155829.94 total 146.63 last 76.29) - open/close 11572 (time 1763155839.95 total 156.63 last 80.85) - open/close 12380 (time 1763155849.95 total 166.64 last 80.77) - open/close 13139 (time 1763155859.96 total 176.64 last 75.84) - open/close 13939 (time 1763155869.97 total 186.65 last 79.96) - open/close 14751 (time 1763155879.97 total 196.65 last 81.17) - open/close 15569 (time 1763155889.99 total 206.67 last 81.65) - open/close 16342 (time 1763155900.00 total 216.68 last 77.24) - open/close 17148 (time 1763155910.00 total 226.69 last 80.54) - open/close 17891 (time 1763155920.02 total 236.70 last 74.21) - open/close 18623 (time 1763155930.02 total 246.70 last 73.15) - open/close 19393 (time 1763155940.03 total 256.71 last 76.95) - open/close 20000 (time 1763155948.31 total 264.99 last 73.33) - open/close 20818 (time 1763155958.31 total 274.99 last 81.74) - open/close 21605 (time 1763155968.32 total 285.00 last 78.68) - open/close 22371 (time 1763155978.33 total 295.01 last 76.51) - open/close 23178 (time 1763155988.34 total 305.02 last 80.61) - open/close 23982 (time 1763155998.34 total 315.03 last 80.37) - open/close 24713 (time 1763156008.34 total 325.03 last 73.10) - open/close 25470 (time 1763156018.34 total 335.03 last 75.70) - open/close 26235 (time 1763156028.35 total 345.03 last 76.45) - open/close 26969 (time 1763156038.37 total 355.05 last 73.27) - open/close 27700 (time 1763156048.38 total 365.06 last 73.05) - open/close 28418 (time 1763156058.38 total 375.07 last 71.74) - open/close 29164 (time 1763156068.39 total 385.07 last 74.57) - open/close 29899 (time 1763156078.39 total 395.08 last 73.45) - open/close 30609 (time 1763156088.40 total 405.09 last 70.93) - open/close 31341 (time 1763156098.42 total 415.10 last 73.07) - open/close 32073 (time 1763156108.43 total 425.11 last 73.15) - open/close 32868 (time 1763156118.44 total 435.12 last 79.40) - open/close 33638 (time 1763156128.45 total 445.13 last 76.93) - open/close 34328 (time 1763156138.49 total 455.17 last 68.71) - open/close 35057 (time 1763156148.50 total 465.18 last 72.83) - open/close 35860 (time 1763156158.52 total 475.20 last 80.18) - open/close 36665 (time 1763156168.52 total 485.20 last 80.46) - open/close 37492 (time 1763156178.52 total 495.20 last 82.70) - open/close 38291 (time 1763156188.53 total 505.21 last 79.87) - open/close 39117 (time 1763156198.55 total 515.23 last 82.43) - open/close 39983 (time 1763156208.55 total 525.24 last 86.54) - open/close 40791 (time 1763156218.56 total 535.24 last 80.75) - open/close 41607 (time 1763156228.56 total 545.24 last 81.60) - open/close 42417 (time 1763156238.56 total 555.24 last 80.99) - open/close 43229 (time 1763156248.57 total 565.26 last 81.08) - open/close 44022 (time 1763156258.58 total 575.26 last 79.28) - open/close 44756 (time 1763156268.58 total 585.26 last 73.39) - open/close 45473 (time 1763156278.58 total 595.27 last 71.66) - open/close 46178 (time 1763156288.60 total 605.28 last 70.42) - open/close 46996 (time 1763156298.60 total 615.29 last 81.73) - open/close 47735 (time 1763156308.62 total 625.30 last 73.80) - open/close 48446 (time 1763156318.63 total 635.31 last 71.01) - open/close 49154 (time 1763156328.66 total 645.34 last 70.59) - open/close 49905 (time 1763156338.68 total 655.36 last 74.98) - open/close 50687 (time 1763156348.68 total 665.36 last 78.16) - open/close 51482 (time 1763156358.69 total 675.37 last 79.43) - open/close 52237 (time 1763156368.70 total 685.38 last 75.45) - open/close 53067 (time 1763156378.70 total 695.38 last 82.97) - open/close 53863 (time 1763156388.71 total 705.39 last 79.51) - open/close 54620 (time 1763156398.72 total 715.40 last 75.66) - open/close 55441 (time 1763156408.72 total 725.40 last 82.06) - open/close 56210 (time 1763156418.73 total 735.41 last 76.84) - open/close 56977 (time 1763156428.73 total 745.41 last 76.69) - open/close 57752 (time 1763156438.75 total 755.43 last 77.38) - open/close 58430 (time 1763156448.76 total 765.44 last 67.73) - open/close 59197 (time 1763156458.76 total 775.44 last 76.70) - open/close 59930 (time 1763156468.76 total 785.45 last 73.26) - open/close 60701 (time 1763156478.77 total 795.45 last 77.04) - open/close 61492 (time 1763156488.78 total 805.46 last 79.04) - open/close 62293 (time 1763156498.78 total 815.46 last 80.08) - open/close 63118 (time 1763156508.80 total 825.49 last 82.31) - open/close 63963 (time 1763156518.81 total 835.49 last 84.45) total: 64768 open/close in 844.89 seconds: 76.66 ops/second - /unlink 888 (time 1763156540.99 total 10.00 last 88.78) - /unlink 1894 (time 1763156550.99 total 20.00 last 100.60) - /unlink 2837 (time 1763156560.99 total 30.00 last 94.28) - /unlink 3735 (time 1763156571.00 total 40.01 last 89.72) - /unlink 4583 (time 1763156581.01 total 50.02 last 84.76) - /unlink 5457 (time 1763156591.02 total 60.03 last 87.33) - /unlink 6316 (time 1763156601.02 total 70.03 last 85.87) - /unlink 7228 (time 1763156611.02 total 80.03 last 91.17) - /unlink 8051 (time 1763156621.03 total 90.04 last 82.23) - /unlink 8918 (time 1763156631.04 total 100.05 last 86.67) - /unlink 9878 (time 1763156641.05 total 110.06 last 95.89) - /unlink 10767 (time 1763156651.05 total 120.06 last 88.88) - /unlink 11607 (time 1763156661.06 total 130.07 last 83.93) - /unlink 12463 (time 1763156671.06 total 140.07 last 85.57) - /unlink 13552 (time 1763156681.07 total 150.09 last 108.75) - /unlink 14582 (time 1763156691.08 total 160.09 last 102.99) - /unlink 15530 (time 1763156701.08 total 170.09 last 94.74) - /unlink 16701 (time 1763156711.08 total 180.09 last 117.08) - /unlink 17930 (time 1763156721.09 total 190.10 last 122.84) - /unlink 18988 (time 1763156731.09 total 200.10 last 105.78) - /unlink 19935 (time 1763156741.10 total 210.11 last 94.61) - /unlink 20930 (time 1763156751.10 total 220.11 last 99.45) - /unlink 21854 (time 1763156761.11 total 230.12 last 92.37) - /unlink 22841 (time 1763156771.11 total 240.12 last 98.69) - /unlink 23998 (time 1763156781.11 total 250.12 last 115.63) - /unlink 25135 (time 1763156791.11 total 260.13 last 113.69) - /unlink 26055 (time 1763156801.12 total 270.13 last 91.99) - /unlink 26983 (time 1763156811.12 total 280.13 last 92.76) - /unlink 28034 (time 1763156821.13 total 290.14 last 104.96) - /unlink 29155 (time 1763156831.13 total 300.14 last 112.10) - /unlink 30000 (time 1763156839.31 total 308.32 last 103.39) - /unlink 31138 (time 1763156849.31 total 318.32 last 113.73) - /unlink 32307 (time 1763156859.32 total 328.33 last 116.86) - /unlink 33675 (time 1763156869.32 total 338.33 last 136.76) - /unlink 35040 (time 1763156879.32 total 348.33 last 136.48) - /unlink 36453 (time 1763156889.32 total 358.34 last 141.24) - /unlink 37997 (time 1763156899.33 total 368.34 last 154.33) - /unlink 39365 (time 1763156909.33 total 378.34 last 136.78) - /unlink 40000 (time 1763156914.33 total 383.34 last 126.99) - /unlink 41123 (time 1763156924.33 total 393.34 last 112.28) - /unlink 42510 (time 1763156934.34 total 403.35 last 138.65) - /unlink 44156 (time 1763156944.34 total 413.35 last 164.52) - /unlink 45453 (time 1763156954.34 total 423.36 last 129.65) - /unlink 46846 (time 1763156964.35 total 433.36 last 139.26) - /unlink 48086 (time 1763156974.35 total 443.36 last 123.93) - /unlink 49570 (time 1763156984.36 total 453.37 last 148.30) - /unlink 50000 (time 1763156987.57 total 456.58 last 134.05) - /unlink 51349 (time 1763156997.58 total 466.59 last 134.78) - /unlink 52761 (time 1763157007.58 total 476.60 last 141.08) - /unlink 53973 (time 1763157017.59 total 486.60 last 121.14) - /unlink 55430 (time 1763157027.59 total 496.60 last 145.67) - /unlink 58479 (time 1763157037.59 total 506.60 last 304.89) - /unlink 60000 (time 1763157042.09 total 511.10 last 337.90) - /unlink 62728 (time 1763157052.09 total 521.11 last 272.75) total: 64768 /unlink in 529.44 seconds: 122.33 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2473 (time 1763157093.36 total 10.00 last 247.25) - open/close 4835 (time 1763157103.36 total 20.00 last 236.17) - open/close 6639 (time 1763157113.36 total 30.00 last 180.39) - open/close 8770 (time 1763157123.36 total 40.00 last 213.09) - open/close 10000 (time 1763157130.21 total 46.86 last 179.46) - open/close 11738 (time 1763157140.22 total 56.86 last 173.74) - open/close 13791 (time 1763157150.22 total 66.86 last 205.25) - open/close 15575 (time 1763157160.22 total 76.86 last 178.39) - open/close 18014 (time 1763157170.23 total 86.87 last 243.69) - open/close 20000 (time 1763157178.12 total 94.76 last 251.82) - open/close 23012 (time 1763157188.12 total 104.76 last 301.17) - open/close 25660 (time 1763157198.12 total 114.76 last 264.80) - open/close 28835 (time 1763157208.12 total 124.76 last 317.41) - open/close 30000 (time 1763157213.43 total 130.07 last 219.51) - open/close 33195 (time 1763157223.43 total 140.07 last 319.50) - open/close 36204 (time 1763157233.43 total 150.07 last 300.88) - open/close 39082 (time 1763157243.43 total 160.07 last 287.73) - open/close 40000 (time 1763157247.70 total 164.34 last 215.16) - open/close 42076 (time 1763157257.70 total 174.34 last 207.57) - open/close 43930 (time 1763157267.70 total 184.35 last 185.32) - open/close 46007 (time 1763157277.70 total 194.35 last 207.69) - open/close 48080 (time 1763157287.70 total 204.35 last 207.29) - open/close 50000 (time 1763157297.29 total 213.94 last 200.19) - open/close 51845 (time 1763157307.29 total 223.94 last 184.49) - open/close 53894 (time 1763157317.30 total 233.94 last 204.82) - open/close 56394 (time 1763157327.30 total 243.94 last 249.96) - open/close 59355 (time 1763157337.30 total 253.95 last 296.03) - open/close 63261 (time 1763157347.30 total 263.95 last 390.59) total: 64768 open/close in 268.35 seconds: 241.35 ops/second - /unlink 5780 (time 1763157363.20 total 10.00 last 577.98) - /unlink 10000 (time 1763157371.91 total 18.71 last 484.78) - /unlink 14904 (time 1763157381.91 total 28.71 last 490.37) - /unlink 19405 (time 1763157391.91 total 38.71 last 450.00) - /unlink 25689 (time 1763157401.91 total 48.71 last 628.27) - /unlink 30000 (time 1763157411.78 total 58.58 last 436.76) - /unlink 33319 (time 1763157421.79 total 68.58 last 331.76) - /unlink 36068 (time 1763157431.79 total 78.59 last 274.89) - /unlink 38582 (time 1763157441.79 total 88.59 last 251.37) - /unlink 40000 (time 1763157446.92 total 93.72 last 276.19) - /unlink 44111 (time 1763157456.92 total 103.72 last 411.09) - /unlink 47950 (time 1763157466.92 total 113.72 last 383.85) - /unlink 50000 (time 1763157472.33 total 119.13 last 379.39) - /unlink 54052 (time 1763157482.33 total 129.13 last 405.15) - /unlink 57523 (time 1763157492.33 total 139.13 last 347.08) - /unlink 60000 (time 1763157498.12 total 144.92 last 427.95) - /unlink 62651 (time 1763157508.12 total 154.92 last 265.08) total: 64768 /unlink in 162.73 seconds: 398.02 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2116 (time 1763157546.39 total 10.00 last 211.57) - open/close 4213 (time 1763157556.39 total 20.01 last 209.63) - open/close 5849 (time 1763157566.39 total 30.01 last 163.55) - open/close 8258 (time 1763157576.40 total 40.01 last 240.87) - open/close 10000 (time 1763157583.83 total 47.45 last 234.22) - open/close 11709 (time 1763157593.83 total 57.45 last 170.87) - open/close 13698 (time 1763157603.84 total 67.45 last 198.83) - open/close 16696 (time 1763157613.84 total 77.45 last 299.76) - open/close 19225 (time 1763157623.84 total 87.45 last 252.88) - open/close 20000 (time 1763157626.44 total 90.05 last 298.47) - open/close 22922 (time 1763157636.44 total 100.06 last 292.04) - open/close 25445 (time 1763157646.49 total 110.10 last 251.18) - open/close 28145 (time 1763157656.49 total 120.10 last 269.92) - open/close 30000 (time 1763157664.80 total 128.41 last 223.22) - open/close 31755 (time 1763157674.80 total 138.42 last 175.43) - open/close 34171 (time 1763157684.81 total 148.42 last 241.57) - open/close 37433 (time 1763157694.81 total 158.42 last 326.09) - open/close 40000 (time 1763157703.20 total 166.82 last 305.79) - open/close 42879 (time 1763157713.21 total 176.82 last 287.84) - open/close 47029 (time 1763157723.21 total 186.82 last 414.94) - open/close 50000 (time 1763157730.98 total 194.59 last 382.35) - open/close 53247 (time 1763157740.98 total 204.59 last 324.65) - open/close 56200 (time 1763157750.98 total 214.59 last 295.24) - open/close 59849 (time 1763157760.98 total 224.60 last 364.86) - open/close 62204 (time 1763157770.98 total 234.60 last 235.46) total: 64768 open/close in 242.30 seconds: 267.31 ops/second - /unlink 4474 (time 1763157790.24 total 10.00 last 447.35) - /unlink 10000 (time 1763157799.32 total 19.08 last 608.80) - /unlink 16085 (time 1763157809.32 total 29.08 last 608.42) - /unlink 20000 (time 1763157816.87 total 36.64 last 518.00) - /unlink 25189 (time 1763157826.88 total 46.64 last 518.85) - /unlink 30000 (time 1763157836.21 total 55.97 last 515.35) - /unlink 34944 (time 1763157846.21 total 65.97 last 494.40) - /unlink 39984 (time 1763157856.21 total 75.98 last 503.92) - /unlink 44951 (time 1763157866.21 total 85.98 last 496.65) - /unlink 50000 (time 1763157875.28 total 95.04 last 556.93) - /unlink 56252 (time 1763157885.28 total 105.04 last 625.15) - /unlink 60000 (time 1763157891.90 total 111.66 last 566.14) total: 64768 /unlink in 119.55 seconds: 541.76 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4958 (time 1763157932.65 total 10.00 last 495.70) - open/close 8413 (time 1763157942.66 total 20.00 last 345.43) - open/close 10000 (time 1763157947.99 total 25.34 last 297.59) - open/close 13282 (time 1763157957.99 total 35.34 last 328.17) - open/close 17021 (time 1763157967.99 total 45.34 last 373.85) - open/close 20000 (time 1763157977.00 total 54.34 last 330.85) - open/close 23105 (time 1763157987.00 total 64.34 last 310.47) - open/close 26477 (time 1763157997.00 total 74.34 last 337.19) - open/close 29766 (time 1763158007.00 total 84.35 last 328.76) - open/close 32835 (time 1763158017.00 total 94.35 last 306.86) - open/close 36407 (time 1763158027.01 total 104.35 last 357.11) - open/close 39303 (time 1763158037.01 total 114.35 last 289.58) - open/close 40000 (time 1763158039.99 total 117.34 last 233.49) - open/close 42994 (time 1763158049.99 total 127.34 last 299.34) - open/close 46371 (time 1763158059.99 total 137.34 last 337.67) - open/close 49694 (time 1763158070.00 total 147.34 last 332.23) - open/close 53953 (time 1763158080.00 total 157.35 last 425.82) - open/close 58010 (time 1763158090.07 total 167.42 last 402.61) - open/close 60000 (time 1763158095.19 total 172.53 last 389.27) - open/close 64324 (time 1763158105.19 total 182.54 last 432.31) total: 64768 open/close in 183.77 seconds: 352.45 ops/second - /unlink 6681 (time 1763158117.89 total 10.00 last 668.06) - /unlink 10000 (time 1763158122.50 total 14.60 last 721.17) - /unlink 15581 (time 1763158132.50 total 24.60 last 558.06) - /unlink 20000 (time 1763158140.25 total 32.36 last 570.06) - /unlink 26650 (time 1763158150.25 total 42.36 last 664.99) - /unlink 30000 (time 1763158155.24 total 47.35 last 670.84) - /unlink 36895 (time 1763158165.24 total 57.35 last 689.49) - /unlink 40000 (time 1763158169.18 total 61.29 last 788.49) - /unlink 48080 (time 1763158179.18 total 71.29 last 807.96) - /unlink 50000 (time 1763158181.27 total 73.37 last 921.41) - /unlink 58334 (time 1763158191.27 total 83.37 last 833.33) total: 64768 /unlink in 91.35 seconds: 708.99 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg344-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg344-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg344-server stop mds service on oleg344-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg344-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing unload_modules_local oleg344-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg344-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg344-client: oleg344-client: ssh exited with exit code 2 pdsh@oleg344-client: oleg344-server: ssh exited with exit code 2 pdsh@oleg344-client: oleg344-server: ssh exited with exit code 2 pdsh@oleg344-client: oleg344-client: ssh exited with exit code 2