== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 14:09:08 (1758737348) Stopping clients: oleg421-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg421-client.virtnet /mnt/lustre2 (opts:-f) oleg421-server: oleg421-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing load_modules_local oleg421-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg421-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg421-server: Loading modules from /home/green/git/lustre-release/lustre oleg421-server: detected 4 online CPUs by sysfs oleg421-server: MODOPTS_LIBCFS= oleg421-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg421-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg421-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 855 (time 1758737439.59 total 10.01 last 85.44) - open/close 1800 (time 1758737449.59 total 20.01 last 94.48) - open/close 2830 (time 1758737459.61 total 30.02 last 102.84) - open/close 3769 (time 1758737469.61 total 40.03 last 93.84) - open/close 4650 (time 1758737479.63 total 50.04 last 87.98) - open/close 5454 (time 1758737489.63 total 60.05 last 80.34) - open/close 6238 (time 1758737499.64 total 70.06 last 78.36) - open/close 7016 (time 1758737509.64 total 80.06 last 77.75) - open/close 7836 (time 1758737519.65 total 90.07 last 81.92) - open/close 8587 (time 1758737529.66 total 100.08 last 75.03) - open/close 9423 (time 1758737539.67 total 110.08 last 83.58) - open/close 10000 (time 1758737546.97 total 117.38 last 79.05) - open/close 10817 (time 1758737556.97 total 127.38 last 81.69) - open/close 11591 (time 1758737566.97 total 137.39 last 77.37) - open/close 12387 (time 1758737576.97 total 147.39 last 79.56) - open/close 13192 (time 1758737586.97 total 157.39 last 80.49) - open/close 13983 (time 1758737596.98 total 167.39 last 79.09) - open/close 14816 (time 1758737606.98 total 177.39 last 83.29) - open/close 15720 (time 1758737616.98 total 187.40 last 90.34) - open/close 16504 (time 1758737627.00 total 197.41 last 78.29) - open/close 17313 (time 1758737637.00 total 207.42 last 80.85) - open/close 18145 (time 1758737647.01 total 217.43 last 83.16) - open/close 18948 (time 1758737657.02 total 227.43 last 80.24) - open/close 19753 (time 1758737667.03 total 237.45 last 80.41) - open/close 20000 (time 1758737670.66 total 241.08 last 67.93) - open/close 20772 (time 1758737680.68 total 251.10 last 77.09) - open/close 21554 (time 1758737690.68 total 261.10 last 78.16) - open/close 22341 (time 1758737700.69 total 271.11 last 78.67) - open/close 23180 (time 1758737710.70 total 281.11 last 83.82) - open/close 24035 (time 1758737720.70 total 291.12 last 85.49) - open/close 24934 (time 1758737730.71 total 301.12 last 89.83) - open/close 25714 (time 1758737740.71 total 311.13 last 77.99) - open/close 26510 (time 1758737750.72 total 321.14 last 79.51) - open/close 27347 (time 1758737760.72 total 331.14 last 83.65) - open/close 28230 (time 1758737770.73 total 341.14 last 88.28) - open/close 29065 (time 1758737780.74 total 351.16 last 83.38) - open/close 29888 (time 1758737790.75 total 361.17 last 82.24) - open/close 30663 (time 1758737800.78 total 371.20 last 77.26) - open/close 31375 (time 1758737810.79 total 381.21 last 71.14) - open/close 32117 (time 1758737820.80 total 391.22 last 74.11) - open/close 32926 (time 1758737830.81 total 401.22 last 80.86) - open/close 33730 (time 1758737840.81 total 411.23 last 80.35) - open/close 34548 (time 1758737850.82 total 421.24 last 81.75) - open/close 35362 (time 1758737860.83 total 431.24 last 81.34) - open/close 36197 (time 1758737870.84 total 441.25 last 83.40) - open/close 37020 (time 1758737880.84 total 451.26 last 82.28) - open/close 37825 (time 1758737890.85 total 461.27 last 80.43) - open/close 38613 (time 1758737900.87 total 471.28 last 78.67) - open/close 39305 (time 1758737910.88 total 481.29 last 69.12) - open/close 40000 (time 1758737920.15 total 490.57 last 74.93) - open/close 40767 (time 1758737930.16 total 500.58 last 76.64) - open/close 41573 (time 1758737940.16 total 510.58 last 80.58) - open/close 42342 (time 1758737950.18 total 520.60 last 76.77) - open/close 43107 (time 1758737960.19 total 530.60 last 76.44) - open/close 43918 (time 1758737970.19 total 540.61 last 81.08) - open/close 44776 (time 1758737980.20 total 550.62 last 85.71) - open/close 45507 (time 1758737990.22 total 560.64 last 72.97) - open/close 46296 (time 1758738000.23 total 570.65 last 78.80) - open/close 47162 (time 1758738010.24 total 580.66 last 86.49) - open/close 47982 (time 1758738020.25 total 590.67 last 81.94) - open/close 48829 (time 1758738030.26 total 600.68 last 84.62) - open/close 49674 (time 1758738040.26 total 610.68 last 84.48) - open/close 50000 (time 1758738044.26 total 614.67 last 81.57) - open/close 50850 (time 1758738054.27 total 624.68 last 84.93) - open/close 51690 (time 1758738064.27 total 634.69 last 83.96) - open/close 52460 (time 1758738074.28 total 644.70 last 76.92) - open/close 53336 (time 1758738084.29 total 654.71 last 87.53) - open/close 54192 (time 1758738094.30 total 664.72 last 85.50) - open/close 55016 (time 1758738104.30 total 674.72 last 82.38) - open/close 55814 (time 1758738114.31 total 684.73 last 79.74) - open/close 56634 (time 1758738124.32 total 694.73 last 81.96) - open/close 57497 (time 1758738134.32 total 704.74 last 86.26) - open/close 58419 (time 1758738144.33 total 714.74 last 92.14) - open/close 59416 (time 1758738154.33 total 724.75 last 99.66) - open/close 60000 (time 1758738160.91 total 731.33 last 88.72) - open/close 60769 (time 1758738170.92 total 741.34 last 76.81) - open/close 61585 (time 1758738180.93 total 751.35 last 81.56) - open/close 62403 (time 1758738190.93 total 761.35 last 81.76) - open/close 63176 (time 1758738200.94 total 771.36 last 77.26) - open/close 63815 (time 1758738210.98 total 781.39 last 63.67) - open/close 64549 (time 1758738220.98 total 791.39 last 73.40) total: 64768 open/close in 794.41 seconds: 81.53 ops/second - /unlink 708 (time 1758738237.94 total 10.01 last 70.71) - /unlink 1430 (time 1758738247.95 total 20.02 last 72.15) - /unlink 2288 (time 1758738257.96 total 30.03 last 85.72) - /unlink 3165 (time 1758738267.98 total 40.05 last 87.57) - /unlink 4055 (time 1758738277.98 total 50.05 last 88.94) - /unlink 4884 (time 1758738287.99 total 60.06 last 82.83) - /unlink 5711 (time 1758738298.00 total 70.06 last 82.67) - /unlink 6526 (time 1758738308.00 total 80.07 last 81.46) - /unlink 7374 (time 1758738318.01 total 90.08 last 84.70) - /unlink 8161 (time 1758738328.02 total 100.09 last 78.63) - /unlink 8942 (time 1758738338.02 total 110.09 last 78.09) - /unlink 9790 (time 1758738348.03 total 120.10 last 84.75) - /unlink 10000 (time 1758738350.78 total 122.85 last 76.34) - /unlink 10801 (time 1758738360.79 total 132.86 last 79.97) - /unlink 11653 (time 1758738370.80 total 142.87 last 85.11) - /unlink 12596 (time 1758738380.81 total 152.88 last 94.26) - /unlink 13485 (time 1758738390.82 total 162.88 last 88.85) - /unlink 14356 (time 1758738400.82 total 172.89 last 87.03) - /unlink 15246 (time 1758738410.83 total 182.90 last 88.95) - /unlink 16133 (time 1758738420.83 total 192.90 last 88.68) - /unlink 16942 (time 1758738430.85 total 202.92 last 80.75) - /unlink 17696 (time 1758738440.86 total 212.93 last 75.35) - /unlink 18512 (time 1758738450.86 total 222.93 last 81.58) - /unlink 19318 (time 1758738460.86 total 232.93 last 80.57) - /unlink 20000 (time 1758738469.38 total 241.45 last 80.07) - /unlink 20751 (time 1758738479.38 total 251.45 last 75.08) - /unlink 21493 (time 1758738489.39 total 261.46 last 74.17) - /unlink 22298 (time 1758738499.39 total 271.46 last 80.48) - /unlink 23087 (time 1758738509.40 total 281.47 last 78.84) - /unlink 23886 (time 1758738519.40 total 291.47 last 79.85) - /unlink 24759 (time 1758738529.41 total 301.48 last 87.27) - /unlink 25592 (time 1758738539.41 total 311.48 last 83.23) - /unlink 26426 (time 1758738549.42 total 321.49 last 83.35) - /unlink 27274 (time 1758738559.43 total 331.50 last 84.73) - /unlink 28136 (time 1758738569.44 total 341.51 last 86.09) - /unlink 28986 (time 1758738579.45 total 351.52 last 84.97) - /unlink 29833 (time 1758738589.45 total 361.52 last 84.68) - /unlink 30725 (time 1758738599.49 total 371.55 last 88.87) - /unlink 31568 (time 1758738609.49 total 381.56 last 84.27) - /unlink 32392 (time 1758738619.49 total 391.56 last 82.39) - /unlink 33161 (time 1758738629.51 total 401.58 last 76.79) - /unlink 33904 (time 1758738639.51 total 411.58 last 74.24) - /unlink 34644 (time 1758738649.52 total 421.59 last 73.93) - /unlink 35382 (time 1758738659.54 total 431.61 last 73.68) - /unlink 36148 (time 1758738669.54 total 441.61 last 76.59) - /unlink 36991 (time 1758738679.54 total 451.61 last 84.26) - /unlink 37913 (time 1758738689.56 total 461.63 last 92.09) - /unlink 38698 (time 1758738699.56 total 471.63 last 78.48) - /unlink 39432 (time 1758738709.56 total 481.63 last 73.39) - /unlink 40000 (time 1758738716.79 total 488.86 last 78.54) - /unlink 40737 (time 1758738726.80 total 498.87 last 73.66) - /unlink 41410 (time 1758738736.81 total 508.88 last 67.25) - /unlink 42177 (time 1758738746.81 total 518.88 last 76.68) - /unlink 42973 (time 1758738756.81 total 528.88 last 79.59) - /unlink 43723 (time 1758738766.82 total 538.89 last 74.95) - /unlink 44551 (time 1758738776.82 total 548.89 last 82.80) - /unlink 45358 (time 1758738786.82 total 558.89 last 80.64) - /unlink 46152 (time 1758738796.83 total 568.90 last 79.32) - /unlink 47008 (time 1758738806.84 total 578.91 last 85.55) - /unlink 47860 (time 1758738816.85 total 588.92 last 85.13) - /unlink 48615 (time 1758738826.86 total 598.93 last 75.39) - /unlink 49388 (time 1758738836.87 total 608.94 last 77.23) - /unlink 50000 (time 1758738843.92 total 615.99 last 86.82) - /unlink 50859 (time 1758738853.93 total 626.00 last 85.80) - /unlink 51670 (time 1758738863.94 total 636.01 last 81.06) - /unlink 52455 (time 1758738873.96 total 646.03 last 78.37) - /unlink 53234 (time 1758738883.96 total 656.03 last 77.90) - /unlink 53988 (time 1758738893.96 total 666.03 last 75.35) - /unlink 54827 (time 1758738903.97 total 676.04 last 83.86) - /unlink 55602 (time 1758738913.97 total 686.04 last 77.48) - /unlink 56403 (time 1758738923.98 total 696.05 last 80.06) - /unlink 57246 (time 1758738933.98 total 706.05 last 84.26) - /unlink 58082 (time 1758738943.98 total 716.05 last 83.58) - /unlink 58938 (time 1758738953.99 total 726.06 last 85.53) - /unlink 59779 (time 1758738964.00 total 736.07 last 84.03) - /unlink 60000 (time 1758738966.47 total 738.54 last 89.47) - /unlink 60872 (time 1758738976.47 total 748.54 last 87.15) - /unlink 61781 (time 1758738986.48 total 758.55 last 90.82) - /unlink 62586 (time 1758738996.49 total 768.56 last 80.46) - /unlink 63389 (time 1758739006.51 total 778.58 last 80.09) - /unlink 64147 (time 1758739016.52 total 788.59 last 75.72) total: 64768 /unlink in 796.77 seconds: 81.29 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 833 (time 1758739060.02 total 10.01 last 83.26) - open/close 1657 (time 1758739070.02 total 20.01 last 82.37) - open/close 2497 (time 1758739080.04 total 30.02 last 83.86) - open/close 3340 (time 1758739090.04 total 40.03 last 84.29) - open/close 4127 (time 1758739100.05 total 50.03 last 78.63) - open/close 4940 (time 1758739110.05 total 60.04 last 81.29) - open/close 5777 (time 1758739120.06 total 70.04 last 83.63) - open/close 6602 (time 1758739130.07 total 80.05 last 82.43) - open/close 7414 (time 1758739140.07 total 90.06 last 81.18) - open/close 8231 (time 1758739150.08 total 100.07 last 81.60) - open/close 9070 (time 1758739160.09 total 110.07 last 83.85) - open/close 9904 (time 1758739170.10 total 120.08 last 83.30) - open/close 10735 (time 1758739180.11 total 130.09 last 83.03) - open/close 11527 (time 1758739190.12 total 140.10 last 79.13) - open/close 12378 (time 1758739200.12 total 150.11 last 85.07) - open/close 13168 (time 1758739210.13 total 160.12 last 78.92) - open/close 13949 (time 1758739220.14 total 170.12 last 78.04) - open/close 14728 (time 1758739230.16 total 180.14 last 77.78) - open/close 15464 (time 1758739240.17 total 190.15 last 73.51) - open/close 16276 (time 1758739250.19 total 200.17 last 81.04) - open/close 17101 (time 1758739260.19 total 210.18 last 82.46) - open/close 17953 (time 1758739270.21 total 220.20 last 85.02) - open/close 18744 (time 1758739280.22 total 230.20 last 79.05) - open/close 19596 (time 1758739290.23 total 240.21 last 85.16) - open/close 20000 (time 1758739294.74 total 244.72 last 89.58) - open/close 20838 (time 1758739304.74 total 254.73 last 83.73) - open/close 21681 (time 1758739314.75 total 264.73 last 84.28) - open/close 22490 (time 1758739324.75 total 274.73 last 80.90) - open/close 23274 (time 1758739334.75 total 284.74 last 78.37) - open/close 24101 (time 1758739344.75 total 294.74 last 82.68) - open/close 24991 (time 1758739354.76 total 304.75 last 88.93) - open/close 25784 (time 1758739364.77 total 314.75 last 79.23) - open/close 26570 (time 1758739374.78 total 324.76 last 78.55) - open/close 27413 (time 1758739384.78 total 334.76 last 84.29) - open/close 28210 (time 1758739394.78 total 344.77 last 79.67) - open/close 29009 (time 1758739404.78 total 354.77 last 79.89) - open/close 29828 (time 1758739414.78 total 364.77 last 81.89) - open/close 30000 (time 1758739416.88 total 366.86 last 82.26) - open/close 30836 (time 1758739426.88 total 376.86 last 83.59) - open/close 31593 (time 1758739436.88 total 386.87 last 75.65) - open/close 32292 (time 1758739446.89 total 396.87 last 69.85) - open/close 32928 (time 1758739456.90 total 406.89 last 63.52) - open/close 33690 (time 1758739466.91 total 416.90 last 76.13) - open/close 34435 (time 1758739476.91 total 426.90 last 74.49) - open/close 35148 (time 1758739486.92 total 436.90 last 71.28) - open/close 35975 (time 1758739496.93 total 446.91 last 82.59) - open/close 36771 (time 1758739506.93 total 456.91 last 79.60) - open/close 37571 (time 1758739516.94 total 466.93 last 79.89) - open/close 38415 (time 1758739526.95 total 476.93 last 84.37) - open/close 39221 (time 1758739536.97 total 486.95 last 80.46) - open/close 40000 (time 1758739546.57 total 496.56 last 81.08) - open/close 40820 (time 1758739556.57 total 506.56 last 82.00) - open/close 41515 (time 1758739566.60 total 516.59 last 69.29) - open/close 42217 (time 1758739576.61 total 526.59 last 70.16) - open/close 42963 (time 1758739586.64 total 536.62 last 74.39) - open/close 43676 (time 1758739596.64 total 546.63 last 71.25) - open/close 44401 (time 1758739606.65 total 556.63 last 72.48) - open/close 45166 (time 1758739616.65 total 566.63 last 76.48) - open/close 45943 (time 1758739626.67 total 576.66 last 77.52) - open/close 46704 (time 1758739636.68 total 586.67 last 76.01) - open/close 47535 (time 1758739646.69 total 596.67 last 83.06) - open/close 48290 (time 1758739656.71 total 606.69 last 75.35) - open/close 49103 (time 1758739666.71 total 616.70 last 81.27) - open/close 49893 (time 1758739676.72 total 626.70 last 78.97) - open/close 50652 (time 1758739686.72 total 636.70 last 75.88) - open/close 51319 (time 1758739696.72 total 646.70 last 66.69) - open/close 52064 (time 1758739706.75 total 656.73 last 74.29) - open/close 52809 (time 1758739716.75 total 666.73 last 74.50) - open/close 53590 (time 1758739726.76 total 676.75 last 77.97) - open/close 54384 (time 1758739736.77 total 686.76 last 79.31) - open/close 55151 (time 1758739746.78 total 696.76 last 76.68) - open/close 55883 (time 1758739756.79 total 706.77 last 73.12) - open/close 56629 (time 1758739766.80 total 716.78 last 74.50) - open/close 57400 (time 1758739776.81 total 726.79 last 77.06) - open/close 58189 (time 1758739786.81 total 736.79 last 78.87) - open/close 58975 (time 1758739796.81 total 746.80 last 78.57) - open/close 59840 (time 1758739806.82 total 756.80 last 86.49) - open/close 60000 (time 1758739809.04 total 759.02 last 72.03) - open/close 60794 (time 1758739819.05 total 769.03 last 79.32) - open/close 61618 (time 1758739829.05 total 779.04 last 82.34) - open/close 62463 (time 1758739839.07 total 789.05 last 84.40) - open/close 63259 (time 1758739849.08 total 799.07 last 79.47) - open/close 64041 (time 1758739859.09 total 809.07 last 78.17) total: 64768 open/close in 817.69 seconds: 79.21 ops/second - /unlink 775 (time 1758739881.08 total 10.00 last 77.48) - /unlink 1537 (time 1758739891.08 total 20.00 last 76.20) - /unlink 2286 (time 1758739901.09 total 30.01 last 74.84) - /unlink 3060 (time 1758739911.10 total 40.02 last 77.34) - /unlink 3720 (time 1758739921.11 total 50.03 last 65.92) - /unlink 4550 (time 1758739931.12 total 60.04 last 82.89) - /unlink 5363 (time 1758739941.14 total 70.06 last 81.16) - /unlink 6169 (time 1758739951.15 total 80.07 last 80.52) - /unlink 6990 (time 1758739961.16 total 90.08 last 82.05) - /unlink 7792 (time 1758739971.17 total 100.09 last 80.07) - /unlink 8576 (time 1758739981.18 total 110.10 last 78.37) - /unlink 9419 (time 1758739991.18 total 120.10 last 84.28) - /unlink 10000 (time 1758739998.56 total 127.48 last 78.65) - /unlink 10785 (time 1758740008.57 total 137.49 last 78.48) - /unlink 11579 (time 1758740018.57 total 147.49 last 79.36) - /unlink 12356 (time 1758740028.58 total 157.50 last 77.61) - /unlink 13179 (time 1758740038.59 total 167.51 last 82.29) - /unlink 13973 (time 1758740048.59 total 177.52 last 79.33) - /unlink 14751 (time 1758740058.60 total 187.52 last 77.77) - /unlink 15587 (time 1758740068.62 total 197.54 last 83.46) - /unlink 16466 (time 1758740078.65 total 207.57 last 87.61) - /unlink 17318 (time 1758740088.65 total 217.57 last 85.17) - /unlink 18226 (time 1758740098.66 total 227.58 last 90.73) - /unlink 19086 (time 1758740108.67 total 237.59 last 85.95) - /unlink 20000 (time 1758740118.37 total 247.29 last 94.16) - /unlink 21030 (time 1758740128.38 total 257.30 last 102.95) - /unlink 22017 (time 1758740138.38 total 267.30 last 98.66) - /unlink 22934 (time 1758740148.39 total 277.31 last 91.66) - /unlink 23903 (time 1758740158.40 total 287.32 last 96.80) - /unlink 24947 (time 1758740168.40 total 297.32 last 104.35) - /unlink 26007 (time 1758740178.41 total 307.33 last 105.90) - /unlink 27006 (time 1758740188.42 total 317.34 last 99.80) - /unlink 27994 (time 1758740198.43 total 327.35 last 98.72) - /unlink 29016 (time 1758740208.43 total 337.35 last 102.17) - /unlink 30000 (time 1758740218.36 total 347.28 last 99.07) - /unlink 31011 (time 1758740228.37 total 357.29 last 101.05) - /unlink 31978 (time 1758740238.38 total 367.30 last 96.59) - /unlink 33081 (time 1758740248.38 total 377.30 last 110.25) - /unlink 34014 (time 1758740258.39 total 387.31 last 93.27) - /unlink 34928 (time 1758740268.39 total 397.31 last 91.36) - /unlink 35814 (time 1758740278.40 total 407.32 last 88.49) - /unlink 36725 (time 1758740288.41 total 417.33 last 91.03) - /unlink 37586 (time 1758740298.41 total 427.33 last 86.10) - /unlink 38512 (time 1758740308.42 total 437.34 last 92.55) - /unlink 39444 (time 1758740318.42 total 447.34 last 93.15) - /unlink 40000 (time 1758740324.93 total 453.85 last 85.49) - /unlink 40907 (time 1758740334.94 total 463.86 last 90.58) - /unlink 41870 (time 1758740344.96 total 473.88 last 96.15) - /unlink 42767 (time 1758740354.96 total 483.88 last 89.69) - /unlink 43771 (time 1758740364.96 total 493.88 last 100.35) - /unlink 44705 (time 1758740374.98 total 503.90 last 93.26) - /unlink 45619 (time 1758740384.98 total 513.90 last 91.38) - /unlink 46554 (time 1758740394.98 total 523.90 last 93.48) - /unlink 47497 (time 1758740404.98 total 533.91 last 94.27) - /unlink 48409 (time 1758740414.99 total 543.91 last 91.18) - /unlink 49529 (time 1758740424.99 total 553.91 last 111.95) - /unlink 50000 (time 1758740429.01 total 557.93 last 117.29) - /unlink 51010 (time 1758740439.01 total 567.93 last 100.93) - /unlink 52338 (time 1758740449.02 total 577.94 last 132.72) - /unlink 53582 (time 1758740459.03 total 587.95 last 124.32) - /unlink 54592 (time 1758740469.03 total 597.95 last 100.98) - /unlink 55670 (time 1758740479.04 total 607.96 last 107.69) - /unlink 56812 (time 1758740489.04 total 617.96 last 114.14) - /unlink 57861 (time 1758740499.05 total 627.97 last 104.86) - /unlink 58908 (time 1758740509.05 total 637.97 last 104.69) - /unlink 60000 (time 1758740517.41 total 646.33 last 130.59) - /unlink 61155 (time 1758740527.42 total 656.34 last 115.40) - /unlink 62394 (time 1758740537.42 total 666.34 last 123.88) - /unlink 63584 (time 1758740547.42 total 676.34 last 118.96) total: 64768 /unlink in 685.62 seconds: 94.47 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1319 (time 1758740589.49 total 10.00 last 131.85) - open/close 2844 (time 1758740599.49 total 20.01 last 152.40) - open/close 4094 (time 1758740609.51 total 30.03 last 124.80) - open/close 5334 (time 1758740619.51 total 40.03 last 123.94) - open/close 6562 (time 1758740629.52 total 50.03 last 122.77) - open/close 8375 (time 1758740639.52 total 60.04 last 181.28) - open/close 9918 (time 1758740649.52 total 70.04 last 154.27) - open/close 11469 (time 1758740659.53 total 80.04 last 154.99) - open/close 12654 (time 1758740669.53 total 90.05 last 118.48) - open/close 13928 (time 1758740679.53 total 100.05 last 127.32) - open/close 15249 (time 1758740689.53 total 110.05 last 132.10) - open/close 16618 (time 1758740699.54 total 120.05 last 136.86) - open/close 18350 (time 1758740709.54 total 130.06 last 173.09) - open/close 19825 (time 1758740719.55 total 140.07 last 147.36) - open/close 21284 (time 1758740729.55 total 150.07 last 145.87) - open/close 22659 (time 1758740739.56 total 160.07 last 137.47) - open/close 24150 (time 1758740749.56 total 170.08 last 149.08) - open/close 25922 (time 1758740759.56 total 180.08 last 177.09) - open/close 27753 (time 1758740769.57 total 190.08 last 183.06) - open/close 29915 (time 1758740779.57 total 200.09 last 216.19) - open/close 31437 (time 1758740789.57 total 210.09 last 152.18) - open/close 33042 (time 1758740799.58 total 220.09 last 160.38) - open/close 34922 (time 1758740809.58 total 230.10 last 187.90) - open/close 36983 (time 1758740819.58 total 240.10 last 206.04) - open/close 39132 (time 1758740829.59 total 250.11 last 214.82) - open/close 40000 (time 1758740834.38 total 254.89 last 181.29) - open/close 42116 (time 1758740844.38 total 264.90 last 211.56) - open/close 44496 (time 1758740854.38 total 274.90 last 237.89) - open/close 46705 (time 1758740864.38 total 284.90 last 220.89) - open/close 48595 (time 1758740874.38 total 294.90 last 188.98) - open/close 50000 (time 1758740882.69 total 303.21 last 169.18) - open/close 51924 (time 1758740892.69 total 313.21 last 192.36) - open/close 53861 (time 1758740902.69 total 323.21 last 193.61) - open/close 55859 (time 1758740912.70 total 333.21 last 199.78) - open/close 58009 (time 1758740922.70 total 343.22 last 214.91) - open/close 60000 (time 1758740932.39 total 352.90 last 205.53) - open/close 63094 (time 1758740942.39 total 362.91 last 309.36) total: 64768 open/close in 369.49 seconds: 175.29 ops/second - /unlink 4348 (time 1758740960.72 total 10.00 last 434.72) - /unlink 8265 (time 1758740970.72 total 20.00 last 391.63) - /unlink 10000 (time 1758740974.88 total 24.16 last 417.20) - /unlink 14283 (time 1758740984.88 total 34.16 last 428.26) - /unlink 19227 (time 1758740994.88 total 44.16 last 494.35) - /unlink 20000 (time 1758740997.03 total 46.31 last 360.52) - /unlink 24819 (time 1758741007.03 total 56.31 last 481.85) - /unlink 29343 (time 1758741017.03 total 66.31 last 452.24) - /unlink 33425 (time 1758741027.03 total 76.32 last 408.11) - /unlink 39632 (time 1758741037.04 total 86.32 last 620.68) - /unlink 44428 (time 1758741047.04 total 96.32 last 479.59) - /unlink 49968 (time 1758741057.04 total 106.32 last 553.80) - /unlink 53679 (time 1758741067.04 total 116.32 last 371.06) - /unlink 58082 (time 1758741077.04 total 126.32 last 440.23) - /unlink 60000 (time 1758741080.87 total 130.15 last 501.08) - /unlink 64393 (time 1758741090.87 total 140.15 last 439.25) total: 64768 /unlink in 141.05 seconds: 459.19 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2445 (time 1758741124.36 total 10.00 last 244.48) - open/close 4422 (time 1758741134.36 total 20.01 last 197.59) - open/close 6768 (time 1758741144.37 total 30.01 last 234.56) - open/close 9650 (time 1758741154.37 total 40.01 last 288.14) - open/close 12067 (time 1758741164.37 total 50.01 last 241.66) - open/close 14566 (time 1758741174.37 total 60.01 last 249.87) - open/close 16685 (time 1758741184.37 total 70.01 last 211.89) - open/close 18925 (time 1758741194.37 total 80.02 last 223.96) - open/close 20000 (time 1758741199.67 total 85.31 last 203.00) - open/close 22090 (time 1758741209.67 total 95.31 last 208.98) - open/close 24310 (time 1758741219.67 total 105.31 last 221.96) - open/close 26921 (time 1758741229.67 total 115.31 last 261.08) - open/close 29696 (time 1758741239.68 total 125.32 last 277.38) - open/close 32244 (time 1758741249.68 total 135.32 last 254.75) - open/close 34527 (time 1758741259.68 total 145.32 last 228.27) - open/close 36709 (time 1758741269.69 total 155.33 last 218.05) - open/close 39111 (time 1758741279.69 total 165.33 last 240.18) - open/close 40000 (time 1758741282.74 total 168.38 last 291.74) - open/close 42726 (time 1758741292.74 total 178.38 last 272.59) - open/close 45122 (time 1758741302.74 total 188.38 last 239.53) - open/close 47571 (time 1758741312.74 total 198.38 last 244.89) - open/close 49433 (time 1758741322.74 total 208.39 last 186.09) - open/close 50000 (time 1758741325.08 total 210.72 last 243.06) - open/close 52119 (time 1758741335.08 total 220.73 last 211.78) - open/close 53683 (time 1758741345.09 total 230.73 last 156.25) - open/close 55396 (time 1758741355.10 total 240.74 last 171.23) - open/close 57056 (time 1758741365.10 total 250.74 last 165.98) - open/close 58634 (time 1758741375.10 total 260.74 last 157.75) - open/close 60000 (time 1758741381.93 total 267.57 last 200.11) - open/close 62114 (time 1758741391.93 total 277.57 last 211.40) - open/close 64607 (time 1758741401.93 total 287.57 last 249.30) total: 64768 open/close in 288.03 seconds: 224.86 ops/second - /unlink 4470 (time 1758741413.95 total 10.00 last 446.90) - /unlink 8313 (time 1758741423.95 total 20.00 last 384.30) - /unlink 10000 (time 1758741428.62 total 24.67 last 361.34) - /unlink 14459 (time 1758741438.62 total 34.67 last 445.86) - /unlink 18392 (time 1758741448.62 total 44.67 last 393.28) - /unlink 20000 (time 1758741451.30 total 47.35 last 600.26) - /unlink 25176 (time 1758741461.30 total 57.35 last 517.58) - /unlink 30000 (time 1758741469.71 total 65.76 last 573.60) - /unlink 35893 (time 1758741479.71 total 75.76 last 589.26) - /unlink 40000 (time 1758741487.97 total 84.02 last 497.63) - /unlink 44471 (time 1758741497.97 total 94.02 last 447.02) - /unlink 49745 (time 1758741507.97 total 104.02 last 527.38) - /unlink 54742 (time 1758741517.97 total 114.02 last 499.62) - /unlink 59045 (time 1758741527.97 total 124.02 last 430.13) - /unlink 60000 (time 1758741530.39 total 126.44 last 394.73) - /unlink 64487 (time 1758741540.39 total 136.44 last 448.66) total: 64768 /unlink in 136.94 seconds: 472.96 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2241 (time 1758741567.94 total 10.00 last 224.07) - open/close 5357 (time 1758741577.94 total 20.00 last 311.59) - open/close 8939 (time 1758741587.94 total 30.00 last 358.19) - open/close 10000 (time 1758741591.90 total 33.96 last 267.92) - open/close 13589 (time 1758741601.91 total 43.96 last 358.86) - open/close 17313 (time 1758741611.91 total 53.97 last 372.29) - open/close 20000 (time 1758741618.73 total 60.79 last 394.03) - open/close 23059 (time 1758741628.73 total 70.79 last 305.86) - open/close 26960 (time 1758741638.73 total 80.79 last 390.06) - open/close 30000 (time 1758741647.71 total 89.77 last 338.37) - open/close 33292 (time 1758741657.72 total 99.78 last 329.07) - open/close 36700 (time 1758741667.72 total 109.78 last 340.73) - open/close 39290 (time 1758741677.73 total 119.78 last 258.86) - open/close 40000 (time 1758741680.83 total 122.89 last 228.54) - open/close 43017 (time 1758741690.83 total 132.89 last 301.69) - open/close 45743 (time 1758741700.83 total 142.89 last 272.56) - open/close 48496 (time 1758741710.83 total 152.89 last 275.29) - open/close 50000 (time 1758741716.61 total 158.67 last 260.45) - open/close 52842 (time 1758741726.61 total 168.67 last 284.18) - open/close 55551 (time 1758741736.61 total 178.67 last 270.83) - open/close 58358 (time 1758741746.61 total 188.67 last 280.63) - open/close 60000 (time 1758741751.21 total 193.27 last 357.25) - open/close 63645 (time 1758741761.21 total 203.27 last 364.47) total: 64768 open/close in 205.69 seconds: 314.89 ops/second - /unlink 5447 (time 1758741774.95 total 10.00 last 544.64) - /unlink 10000 (time 1758741782.43 total 17.48 last 608.67) - /unlink 15369 (time 1758741792.43 total 27.48 last 536.86) - /unlink 20000 (time 1758741800.44 total 35.49 last 578.25) - /unlink 23895 (time 1758741810.44 total 45.49 last 389.48) - /unlink 29055 (time 1758741820.45 total 55.50 last 515.69) - /unlink 33757 (time 1758741830.45 total 65.50 last 470.16) - /unlink 37688 (time 1758741840.45 total 75.50 last 393.04) - /unlink 40000 (time 1758741846.50 total 81.55 last 382.43) - /unlink 44093 (time 1758741856.50 total 91.55 last 409.26) - /unlink 48330 (time 1758741866.51 total 101.55 last 423.32) - /unlink 50000 (time 1758741869.92 total 104.97 last 488.70) - /unlink 55321 (time 1758741879.93 total 114.98 last 531.89) - /unlink 59287 (time 1758741889.93 total 124.98 last 396.57) - /unlink 63583 (time 1758741899.93 total 134.98 last 429.56) total: 64768 /unlink in 138.36 seconds: 468.12 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg421-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg421-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg421-server stop mds service on oleg421-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg421-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing unload_modules_local oleg421-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg421-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg421-client: oleg421-client: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-client: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 2