== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 22:46:46 (1764733606) Stopping clients: oleg156-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg156-client.virtnet /mnt/lustre2 (opts:-f) oleg156-server: oleg156-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg156-server' oleg156-server: oleg156-server.virtnet: executing load_modules_local oleg156-server: Loading modules from /home/green/git/lustre-release/lustre oleg156-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg156-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg156-server: detected 4 online CPUs by sysfs oleg156-server: MODOPTS_LIBCFS= oleg156-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg156-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg156-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg156-client.virtnet: -o user_xattr,flock 192.168.201.156@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 582 (time 1764733720.22 total 10.03 last 58.05) - open/close 1220 (time 1764733730.22 total 20.04 last 63.74) - open/close 1834 (time 1764733740.23 total 30.04 last 61.39) - open/close 2494 (time 1764733750.24 total 40.06 last 65.88) - open/close 3169 (time 1764733760.25 total 50.06 last 67.48) - open/close 3872 (time 1764733770.25 total 60.06 last 70.26) - open/close 4590 (time 1764733780.26 total 70.08 last 71.72) - open/close 5348 (time 1764733790.27 total 80.08 last 75.77) - open/close 6052 (time 1764733800.27 total 90.08 last 70.40) - open/close 6711 (time 1764733810.27 total 100.08 last 65.88) - open/close 7395 (time 1764733820.28 total 110.09 last 68.38) - open/close 8067 (time 1764733830.28 total 120.09 last 67.19) - open/close 8788 (time 1764733840.28 total 130.10 last 72.05) - open/close 9449 (time 1764733850.29 total 140.10 last 66.09) - open/close 10000 (time 1764733858.70 total 148.51 last 65.48) - open/close 10725 (time 1764733868.71 total 158.52 last 72.45) - open/close 11379 (time 1764733878.72 total 168.53 last 65.34) - open/close 12056 (time 1764733888.73 total 178.54 last 67.65) - open/close 12724 (time 1764733898.73 total 188.54 last 66.77) - open/close 13425 (time 1764733908.74 total 198.55 last 70.06) - open/close 14137 (time 1764733918.74 total 208.55 last 71.20) - open/close 14852 (time 1764733928.78 total 218.59 last 71.18) - open/close 15568 (time 1764733938.78 total 228.59 last 71.58) - open/close 16276 (time 1764733948.78 total 238.59 last 70.80) - open/close 16964 (time 1764733958.80 total 248.62 last 68.66) - open/close 17692 (time 1764733968.81 total 258.62 last 72.77) - open/close 18535 (time 1764733978.82 total 268.63 last 84.24) - open/close 19307 (time 1764733988.83 total 278.64 last 77.11) - open/close 20000 (time 1764733996.85 total 286.66 last 86.42) - open/close 20774 (time 1764734006.85 total 296.66 last 77.39) - open/close 21505 (time 1764734016.86 total 306.67 last 73.01) - open/close 22252 (time 1764734026.87 total 316.68 last 74.60) - open/close 23077 (time 1764734036.88 total 326.69 last 82.43) - open/close 23867 (time 1764734046.90 total 336.71 last 78.86) - open/close 24702 (time 1764734056.91 total 346.72 last 83.43) - open/close 25560 (time 1764734066.92 total 356.73 last 85.70) - open/close 26399 (time 1764734076.93 total 366.74 last 83.84) - open/close 27212 (time 1764734086.94 total 376.75 last 81.17) - open/close 28021 (time 1764734096.95 total 386.76 last 80.87) - open/close 28788 (time 1764734106.96 total 396.77 last 76.56) - open/close 29605 (time 1764734116.97 total 406.78 last 81.69) - open/close 30000 (time 1764734121.06 total 410.87 last 96.40) - open/close 30742 (time 1764734131.07 total 420.88 last 74.18) - open/close 31508 (time 1764734141.07 total 430.88 last 76.55) - open/close 32258 (time 1764734151.08 total 440.89 last 74.97) - open/close 32783 (time 1764734161.08 total 450.89 last 52.47) - open/close 33374 (time 1764734171.08 total 460.89 last 59.09) - open/close 33936 (time 1764734181.09 total 470.90 last 56.18) - open/close 34572 (time 1764734191.10 total 480.91 last 63.52) - open/close 35289 (time 1764734201.10 total 490.91 last 71.67) - open/close 35975 (time 1764734211.11 total 500.92 last 68.57) - open/close 36645 (time 1764734221.12 total 510.93 last 66.90) - open/close 37322 (time 1764734231.13 total 520.94 last 67.66) - open/close 38072 (time 1764734241.14 total 530.95 last 74.92) - open/close 38777 (time 1764734251.14 total 540.95 last 70.48) - open/close 39503 (time 1764734261.14 total 550.96 last 72.58) - open/close 40000 (time 1764734268.65 total 558.46 last 66.23) - open/close 40688 (time 1764734278.67 total 568.48 last 68.66) - open/close 41371 (time 1764734288.71 total 578.52 last 68.02) - open/close 42078 (time 1764734298.72 total 588.53 last 70.63) - open/close 42696 (time 1764734308.73 total 598.55 last 61.70) - open/close 43371 (time 1764734318.74 total 608.55 last 67.47) - open/close 44071 (time 1764734328.74 total 618.55 last 69.98) - open/close 44772 (time 1764734338.74 total 628.55 last 70.09) - open/close 45494 (time 1764734348.75 total 638.57 last 72.12) - open/close 46188 (time 1764734358.76 total 648.57 last 69.38) - open/close 46855 (time 1764734368.76 total 658.57 last 66.68) - open/close 47534 (time 1764734378.78 total 668.59 last 67.78) - open/close 48248 (time 1764734388.78 total 678.59 last 71.39) - open/close 48955 (time 1764734398.79 total 688.60 last 70.63) - open/close 49636 (time 1764734408.79 total 698.60 last 68.07) - open/close 50000 (time 1764734413.95 total 703.76 last 70.55) - open/close 50689 (time 1764734423.96 total 713.77 last 68.87) - open/close 51433 (time 1764734433.96 total 723.78 last 74.35) - open/close 52110 (time 1764734443.97 total 733.78 last 67.64) - open/close 52779 (time 1764734453.97 total 743.79 last 66.89) - open/close 53382 (time 1764734463.99 total 753.80 last 60.20) - open/close 54044 (time 1764734474.00 total 763.82 last 66.10) - open/close 54742 (time 1764734484.01 total 773.83 last 69.73) - open/close 55382 (time 1764734494.02 total 783.83 last 63.96) - open/close 55989 (time 1764734504.03 total 793.84 last 60.62) - open/close 56600 (time 1764734514.05 total 803.86 last 61.01) - open/close 57298 (time 1764734524.07 total 813.88 last 69.67) - open/close 58010 (time 1764734534.08 total 823.89 last 71.09) - open/close 58733 (time 1764734544.09 total 833.90 last 72.28) - open/close 59493 (time 1764734554.09 total 843.90 last 75.95) - open/close 60000 (time 1764734560.99 total 850.80 last 73.54) - open/close 60681 (time 1764734571.00 total 860.81 last 68.04) - open/close 61334 (time 1764734581.01 total 870.82 last 65.24) - open/close 62093 (time 1764734591.01 total 880.82 last 75.90) - open/close 62806 (time 1764734601.01 total 890.83 last 71.24) - open/close 63594 (time 1764734611.02 total 900.83 last 78.78) - open/close 64316 (time 1764734621.02 total 910.83 last 72.16) total: 64768 open/close in 917.12 seconds: 70.62 ops/second - /unlink 797 (time 1764734640.46 total 10.01 last 79.65) - /unlink 1626 (time 1764734650.47 total 20.02 last 82.78) - /unlink 2386 (time 1764734660.48 total 30.03 last 75.96) - /unlink 3078 (time 1764734670.48 total 40.03 last 69.15) - /unlink 3832 (time 1764734680.49 total 50.03 last 75.39) - /unlink 4537 (time 1764734690.50 total 60.05 last 70.41) - /unlink 5228 (time 1764734700.51 total 70.05 last 69.05) - /unlink 5938 (time 1764734710.52 total 80.07 last 70.90) - /unlink 6743 (time 1764734720.53 total 90.08 last 80.43) - /unlink 7488 (time 1764734730.54 total 100.09 last 74.43) - /unlink 8227 (time 1764734740.55 total 110.10 last 73.80) - /unlink 8936 (time 1764734750.55 total 120.10 last 70.88) - /unlink 9748 (time 1764734760.57 total 130.12 last 81.05) - /unlink 10000 (time 1764734764.10 total 133.64 last 71.53) - /unlink 10701 (time 1764734774.11 total 143.65 last 70.03) - /unlink 11426 (time 1764734784.12 total 153.67 last 72.37) - /unlink 12146 (time 1764734794.13 total 163.67 last 71.98) - /unlink 12915 (time 1764734804.13 total 173.68 last 76.83) - /unlink 13742 (time 1764734814.14 total 183.69 last 82.62) - /unlink 14588 (time 1764734824.15 total 193.70 last 84.58) - /unlink 15639 (time 1764734834.16 total 203.71 last 105.00) - /unlink 16622 (time 1764734844.16 total 213.71 last 98.27) - /unlink 17573 (time 1764734854.16 total 223.71 last 95.09) - /unlink 18476 (time 1764734864.17 total 233.72 last 90.23) - /unlink 19501 (time 1764734874.17 total 243.72 last 102.45) - /unlink 20000 (time 1764734879.52 total 249.07 last 93.33) - /unlink 21038 (time 1764734889.53 total 259.08 last 103.72) - /unlink 21974 (time 1764734899.54 total 269.08 last 93.53) - /unlink 22852 (time 1764734909.54 total 279.09 last 87.73) - /unlink 23768 (time 1764734919.55 total 289.10 last 91.56) - /unlink 24690 (time 1764734929.56 total 299.11 last 92.10) - /unlink 25682 (time 1764734939.56 total 309.11 last 99.15) - /unlink 26591 (time 1764734949.56 total 319.11 last 90.90) - /unlink 27539 (time 1764734959.58 total 329.12 last 94.69) - /unlink 28408 (time 1764734969.58 total 339.13 last 86.83) - /unlink 29236 (time 1764734979.59 total 349.14 last 82.76) - /unlink 30000 (time 1764734988.67 total 358.21 last 84.16) - /unlink 30902 (time 1764734998.67 total 368.21 last 90.20) - /unlink 31734 (time 1764735008.68 total 378.22 last 83.12) - /unlink 32691 (time 1764735018.68 total 388.23 last 95.68) - /unlink 33497 (time 1764735028.68 total 398.23 last 80.58) - /unlink 34309 (time 1764735038.70 total 408.24 last 81.07) - /unlink 35298 (time 1764735048.70 total 418.24 last 98.90) - /unlink 36327 (time 1764735058.70 total 428.24 last 102.90) - /unlink 37210 (time 1764735068.70 total 438.25 last 88.23) - /unlink 38170 (time 1764735078.71 total 448.26 last 95.93) - /unlink 39223 (time 1764735088.72 total 458.26 last 105.26) - /unlink 40000 (time 1764735095.72 total 465.27 last 110.97) - /unlink 41215 (time 1764735105.72 total 475.27 last 121.50) - /unlink 42344 (time 1764735115.72 total 485.27 last 112.83) - /unlink 43528 (time 1764735125.73 total 495.28 last 118.35) - /unlink 44569 (time 1764735135.73 total 505.28 last 104.03) - /unlink 45777 (time 1764735145.74 total 515.29 last 120.73) - /unlink 46851 (time 1764735155.74 total 525.29 last 107.36) - /unlink 48138 (time 1764735165.75 total 535.29 last 128.67) - /unlink 49353 (time 1764735175.75 total 545.29 last 121.50) - /unlink 50000 (time 1764735181.86 total 551.41 last 105.81) - /unlink 51027 (time 1764735191.86 total 561.41 last 102.69) - /unlink 52329 (time 1764735201.87 total 571.41 last 130.15) - /unlink 53282 (time 1764735211.87 total 581.42 last 95.23) - /unlink 54300 (time 1764735221.88 total 591.43 last 101.73) - /unlink 55816 (time 1764735231.88 total 601.43 last 151.57) - /unlink 57105 (time 1764735241.89 total 611.44 last 128.84) - /unlink 58410 (time 1764735251.89 total 621.44 last 130.48) - /unlink 59729 (time 1764735261.89 total 631.44 last 131.89) - /unlink 60000 (time 1764735264.09 total 633.64 last 123.17) - /unlink 61223 (time 1764735274.09 total 643.64 last 122.29) - /unlink 62491 (time 1764735284.09 total 653.64 last 126.74) - /unlink 64189 (time 1764735294.10 total 663.64 last 169.78) total: 64768 /unlink in 667.86 seconds: 96.98 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1196 (time 1764735329.10 total 10.00 last 119.55) - open/close 1891 (time 1764735339.10 total 20.01 last 69.49) - open/close 2523 (time 1764735349.10 total 30.01 last 63.20) - open/close 3064 (time 1764735359.11 total 40.02 last 54.04) - open/close 3713 (time 1764735369.13 total 50.04 last 64.79) - open/close 4549 (time 1764735379.14 total 60.04 last 83.54) - open/close 5417 (time 1764735389.14 total 70.05 last 86.78) - open/close 6175 (time 1764735399.14 total 80.05 last 75.80) - open/close 6945 (time 1764735409.15 total 90.06 last 76.92) - open/close 7655 (time 1764735419.15 total 100.06 last 70.99) - open/close 8386 (time 1764735429.16 total 110.07 last 73.01) - open/close 9136 (time 1764735439.17 total 120.07 last 74.97) - open/close 9867 (time 1764735449.18 total 130.08 last 73.03) - open/close 10639 (time 1764735459.18 total 140.09 last 77.18) - open/close 11420 (time 1764735469.19 total 150.10 last 78.01) - open/close 12188 (time 1764735479.20 total 160.11 last 76.70) - open/close 12943 (time 1764735489.22 total 170.13 last 75.38) - open/close 13691 (time 1764735499.23 total 180.14 last 74.73) - open/close 14450 (time 1764735509.23 total 190.14 last 75.89) - open/close 15230 (time 1764735519.23 total 200.14 last 77.97) - open/close 16074 (time 1764735529.25 total 210.15 last 84.29) - open/close 16909 (time 1764735539.25 total 220.15 last 83.50) - open/close 17709 (time 1764735549.25 total 230.16 last 79.98) - open/close 18584 (time 1764735559.25 total 240.16 last 87.47) - open/close 19367 (time 1764735569.26 total 250.17 last 78.26) - open/close 20000 (time 1764735577.01 total 257.92 last 81.65) - open/close 20794 (time 1764735587.02 total 267.92 last 79.35) - open/close 21671 (time 1764735597.02 total 277.93 last 87.69) - open/close 22707 (time 1764735607.02 total 287.93 last 103.59) - open/close 23843 (time 1764735617.02 total 297.93 last 113.58) - open/close 25084 (time 1764735627.03 total 307.93 last 124.06) - open/close 26252 (time 1764735637.03 total 317.94 last 116.74) - open/close 27402 (time 1764735647.03 total 327.94 last 114.97) - open/close 28949 (time 1764735657.04 total 337.94 last 154.65) - open/close 30000 (time 1764735664.71 total 345.62 last 136.91) - open/close 31380 (time 1764735674.71 total 355.62 last 137.98) - open/close 32826 (time 1764735684.72 total 365.63 last 144.52) - open/close 34358 (time 1764735694.72 total 375.63 last 153.14) - open/close 35758 (time 1764735704.73 total 385.64 last 139.93) - open/close 37198 (time 1764735714.73 total 395.64 last 143.95) - open/close 38596 (time 1764735724.73 total 405.64 last 139.80) - open/close 40000 (time 1764735734.02 total 414.92 last 151.23) - open/close 41608 (time 1764735744.02 total 424.93 last 160.74) - open/close 42981 (time 1764735754.02 total 434.93 last 137.28) - open/close 44346 (time 1764735764.02 total 444.93 last 136.49) - open/close 45458 (time 1764735774.03 total 454.94 last 111.14) - open/close 46545 (time 1764735784.04 total 464.94 last 108.61) - open/close 47662 (time 1764735794.04 total 474.95 last 111.63) - open/close 48848 (time 1764735804.05 total 484.96 last 118.50) - open/close 50000 (time 1764735812.94 total 493.85 last 129.61) - open/close 51144 (time 1764735822.95 total 503.85 last 114.32) - open/close 52178 (time 1764735832.95 total 513.86 last 103.37) - open/close 53370 (time 1764735842.95 total 523.86 last 119.13) - open/close 54407 (time 1764735852.96 total 533.86 last 103.68) - open/close 55379 (time 1764735862.96 total 543.87 last 97.13) - open/close 56623 (time 1764735872.97 total 553.87 last 124.35) - open/close 57600 (time 1764735882.99 total 563.89 last 97.50) - open/close 58488 (time 1764735892.99 total 573.90 last 88.78) - open/close 59587 (time 1764735902.99 total 583.90 last 109.88) - open/close 60000 (time 1764735906.91 total 587.82 last 105.29) - open/close 61160 (time 1764735916.92 total 597.83 last 115.93) - open/close 62374 (time 1764735926.93 total 607.83 last 121.33) - open/close 63601 (time 1764735936.93 total 617.84 last 122.62) total: 64768 open/close in 627.15 seconds: 103.27 ops/second - /unlink 1312 (time 1764735958.45 total 10.01 last 131.13) - /unlink 2534 (time 1764735968.46 total 20.01 last 122.13) - /unlink 3685 (time 1764735978.46 total 30.02 last 115.04) - /unlink 4970 (time 1764735988.46 total 40.02 last 128.48) - /unlink 6056 (time 1764735998.47 total 50.02 last 108.55) - /unlink 7928 (time 1764736008.47 total 60.02 last 187.15) - /unlink 10000 (time 1764736016.63 total 68.19 last 253.83) - /unlink 12351 (time 1764736026.63 total 78.19 last 235.05) - /unlink 14907 (time 1764736036.64 total 88.19 last 255.58) - /unlink 17011 (time 1764736046.64 total 98.19 last 210.36) - /unlink 19181 (time 1764736056.64 total 108.20 last 216.93) - /unlink 20000 (time 1764736060.85 total 112.40 last 194.59) - /unlink 22287 (time 1764736070.85 total 122.41 last 228.64) - /unlink 24418 (time 1764736080.85 total 132.41 last 213.08) - /unlink 26467 (time 1764736090.85 total 142.41 last 204.89) - /unlink 28868 (time 1764736100.85 total 152.41 last 240.08) - /unlink 30000 (time 1764736106.43 total 157.99 last 202.97) - /unlink 32146 (time 1764736116.44 total 167.99 last 214.50) - /unlink 34150 (time 1764736126.44 total 177.99 last 200.34) - /unlink 36350 (time 1764736136.44 total 188.00 last 219.98) - /unlink 38820 (time 1764736146.45 total 198.00 last 246.85) - /unlink 40000 (time 1764736152.10 total 203.65 last 208.84) - /unlink 41435 (time 1764736162.10 total 213.65 last 143.50) - /unlink 42780 (time 1764736172.10 total 223.65 last 134.47) - /unlink 43963 (time 1764736182.11 total 233.66 last 118.22) - /unlink 45239 (time 1764736192.12 total 243.67 last 127.46) - /unlink 46256 (time 1764736202.13 total 253.68 last 101.58) - /unlink 47262 (time 1764736212.14 total 263.69 last 100.53) - /unlink 48563 (time 1764736222.15 total 273.71 last 129.88) - /unlink 49915 (time 1764736232.15 total 283.71 last 135.20) - /unlink 51266 (time 1764736242.15 total 293.71 last 135.09) - /unlink 52608 (time 1764736252.16 total 303.71 last 134.15) - /unlink 54059 (time 1764736262.16 total 313.72 last 144.99) - /unlink 55538 (time 1764736272.16 total 323.72 last 147.89) - /unlink 57074 (time 1764736282.17 total 333.72 last 153.58) - /unlink 58541 (time 1764736292.17 total 343.73 last 146.64) - /unlink 59893 (time 1764736302.18 total 353.73 last 135.13) - /unlink 61375 (time 1764736312.18 total 363.74 last 148.09) - /unlink 62972 (time 1764736322.19 total 373.75 last 159.55) - /unlink 64400 (time 1764736332.20 total 383.76 last 142.67) total: 64768 /unlink in 386.31 seconds: 167.66 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 781 (time 1764736369.92 total 10.00 last 78.08) - open/close 1604 (time 1764736379.92 total 20.00 last 82.29) - open/close 2391 (time 1764736389.92 total 30.01 last 78.68) - open/close 3161 (time 1764736399.93 total 40.01 last 76.95) - open/close 3966 (time 1764736409.93 total 50.02 last 80.46) - open/close 4990 (time 1764736419.93 total 60.02 last 102.39) - open/close 5796 (time 1764736429.94 total 70.03 last 80.53) - open/close 6577 (time 1764736439.95 total 80.04 last 78.01) - open/close 7397 (time 1764736449.96 total 90.04 last 81.97) - open/close 8154 (time 1764736459.96 total 100.04 last 75.69) - open/close 8992 (time 1764736469.98 total 110.06 last 83.67) - open/close 9889 (time 1764736479.98 total 120.07 last 89.64) - open/close 10711 (time 1764736489.99 total 130.07 last 82.14) - open/close 11545 (time 1764736500.01 total 140.09 last 83.23) - open/close 12443 (time 1764736510.01 total 150.10 last 89.76) - open/close 13356 (time 1764736520.02 total 160.10 last 91.24) - open/close 14306 (time 1764736530.03 total 170.11 last 94.93) - open/close 15215 (time 1764736540.03 total 180.11 last 90.90) - open/close 16080 (time 1764736550.04 total 190.12 last 86.41) - open/close 16991 (time 1764736560.04 total 200.13 last 91.07) - open/close 17930 (time 1764736570.05 total 210.13 last 93.85) - open/close 18782 (time 1764736580.05 total 220.13 last 85.17) - open/close 19607 (time 1764736590.06 total 230.14 last 82.43) - open/close 20000 (time 1764736593.83 total 233.91 last 104.33) - open/close 20971 (time 1764736603.83 total 243.91 last 97.05) - open/close 21885 (time 1764736613.84 total 253.92 last 91.31) - open/close 22739 (time 1764736623.85 total 263.93 last 85.36) - open/close 23703 (time 1764736633.86 total 273.94 last 96.31) - open/close 24729 (time 1764736643.86 total 283.94 last 102.58) - open/close 25781 (time 1764736653.87 total 293.95 last 105.08) - open/close 26740 (time 1764736663.88 total 303.96 last 95.84) - open/close 27625 (time 1764736673.88 total 313.97 last 88.42) - open/close 28472 (time 1764736683.89 total 323.98 last 84.63) - open/close 29336 (time 1764736693.90 total 333.98 last 86.37) - open/close 30000 (time 1764736701.24 total 341.33 last 90.37) - open/close 30936 (time 1764736711.24 total 351.33 last 93.59) - open/close 31623 (time 1764736721.26 total 361.34 last 68.62) - open/close 32204 (time 1764736731.26 total 371.34 last 58.08) - open/close 32754 (time 1764736741.28 total 381.37 last 54.89) - open/close 33238 (time 1764736751.28 total 391.37 last 48.39) - open/close 33950 (time 1764736761.29 total 401.38 last 71.14) - open/close 34680 (time 1764736771.31 total 411.39 last 72.90) - open/close 35441 (time 1764736781.31 total 421.40 last 76.05) - open/close 36183 (time 1764736791.32 total 431.40 last 74.13) - open/close 36997 (time 1764736801.33 total 441.42 last 81.29) - open/close 37779 (time 1764736811.35 total 451.43 last 78.11) - open/close 38619 (time 1764736821.35 total 461.43 last 84.00) - open/close 39402 (time 1764736831.35 total 471.43 last 78.27) - open/close 40000 (time 1764736838.46 total 478.54 last 84.13) - open/close 40822 (time 1764736848.48 total 488.56 last 82.03) - open/close 41642 (time 1764736858.48 total 498.56 last 82.00) - open/close 42526 (time 1764736868.48 total 508.56 last 88.40) - open/close 43406 (time 1764736878.48 total 518.56 last 87.99) - open/close 44267 (time 1764736888.49 total 528.57 last 86.05) - open/close 45111 (time 1764736898.49 total 538.57 last 84.37) - open/close 45877 (time 1764736908.50 total 548.58 last 76.53) - open/close 46692 (time 1764736918.50 total 558.58 last 81.49) - open/close 47495 (time 1764736928.51 total 568.60 last 80.20) - open/close 48279 (time 1764736938.53 total 578.61 last 78.30) - open/close 49069 (time 1764736948.55 total 588.64 last 78.78) - open/close 49920 (time 1764736958.56 total 598.64 last 85.08) - open/close 50788 (time 1764736968.56 total 608.65 last 86.74) - open/close 51750 (time 1764736978.57 total 618.65 last 96.15) - open/close 52656 (time 1764736988.58 total 628.66 last 90.53) - open/close 53578 (time 1764736998.58 total 638.66 last 92.15) - open/close 54439 (time 1764737008.58 total 648.67 last 86.09) - open/close 55416 (time 1764737018.59 total 658.67 last 97.66) - open/close 56269 (time 1764737028.60 total 668.68 last 85.22) - open/close 57048 (time 1764737038.60 total 678.69 last 77.85) - open/close 57903 (time 1764737048.60 total 688.69 last 85.50) - open/close 58780 (time 1764737058.62 total 698.70 last 87.56) - open/close 59774 (time 1764737068.62 total 708.70 last 99.38) - open/close 60922 (time 1764737078.63 total 718.71 last 114.71) - open/close 61946 (time 1764737088.63 total 728.72 last 102.37) - open/close 62751 (time 1764737098.64 total 738.72 last 80.47) - open/close 63650 (time 1764737108.64 total 748.72 last 89.88) - open/close 64671 (time 1764737118.64 total 758.73 last 102.06) total: 64768 open/close in 759.49 seconds: 85.28 ops/second - /unlink 1166 (time 1764737131.74 total 10.01 last 116.52) - /unlink 2268 (time 1764737141.74 total 20.01 last 110.17) - /unlink 3371 (time 1764737151.76 total 30.03 last 110.08) - /unlink 4614 (time 1764737161.76 total 40.03 last 124.28) - /unlink 6091 (time 1764737171.77 total 50.04 last 147.60) - /unlink 7274 (time 1764737181.77 total 60.04 last 118.28) - /unlink 8494 (time 1764737191.78 total 70.05 last 121.91) - /unlink 9701 (time 1764737201.78 total 80.05 last 120.67) - /unlink 10000 (time 1764737203.93 total 82.20 last 138.69) - /unlink 11318 (time 1764737213.94 total 92.21 last 131.74) - /unlink 12842 (time 1764737223.94 total 102.21 last 152.34) - /unlink 14478 (time 1764737233.95 total 112.22 last 163.43) - /unlink 15734 (time 1764737243.96 total 122.23 last 125.56) - /unlink 17596 (time 1764737253.96 total 132.23 last 186.18) - /unlink 19463 (time 1764737263.96 total 142.23 last 186.60) - /unlink 20000 (time 1764737268.19 total 146.46 last 126.91) - /unlink 21502 (time 1764737278.20 total 156.48 last 150.04) - /unlink 23039 (time 1764737288.21 total 166.48 last 153.64) - /unlink 25012 (time 1764737298.21 total 176.48 last 197.23) - /unlink 27317 (time 1764737308.21 total 186.48 last 230.46) - /unlink 29414 (time 1764737318.22 total 196.49 last 209.60) - /unlink 30000 (time 1764737321.80 total 200.07 last 163.48) - /unlink 31566 (time 1764737331.80 total 210.08 last 156.58) - /unlink 33872 (time 1764737341.81 total 220.08 last 230.54) - /unlink 36431 (time 1764737351.81 total 230.08 last 255.90) - /unlink 39932 (time 1764737361.82 total 240.09 last 349.81) - /unlink 43159 (time 1764737371.82 total 250.09 last 322.62) - /unlink 46245 (time 1764737381.82 total 260.09 last 308.57) - /unlink 49498 (time 1764737391.82 total 270.09 last 325.25) - /unlink 52435 (time 1764737401.82 total 280.09 last 293.69) - /unlink 56138 (time 1764737411.82 total 290.09 last 370.26) - /unlink 59508 (time 1764737421.82 total 300.09 last 337.00) - /unlink 60000 (time 1764737424.11 total 302.38 last 214.82) - /unlink 63338 (time 1764737434.11 total 312.38 last 333.74) total: 64768 /unlink in 317.30 seconds: 204.12 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1420 (time 1764737474.54 total 10.01 last 141.90) - open/close 2683 (time 1764737484.54 total 20.01 last 126.28) - open/close 4082 (time 1764737494.55 total 30.01 last 139.85) - open/close 5312 (time 1764737504.55 total 40.01 last 122.99) - open/close 6370 (time 1764737514.55 total 50.01 last 105.79) - open/close 7664 (time 1764737524.55 total 60.02 last 129.38) - open/close 9142 (time 1764737534.55 total 70.02 last 147.75) - open/close 10000 (time 1764737541.56 total 77.02 last 122.52) - open/close 11207 (time 1764737551.56 total 87.03 last 120.62) - open/close 12473 (time 1764737561.57 total 97.03 last 126.59) - open/close 13493 (time 1764737571.57 total 107.04 last 101.93) - open/close 14665 (time 1764737581.58 total 117.04 last 117.12) - open/close 16112 (time 1764737591.58 total 127.05 last 144.67) - open/close 17231 (time 1764737601.58 total 137.05 last 111.88) - open/close 18234 (time 1764737611.59 total 147.05 last 100.25) - open/close 19749 (time 1764737621.59 total 157.05 last 151.47) - open/close 21641 (time 1764737631.59 total 167.06 last 189.15) - open/close 23469 (time 1764737641.60 total 177.06 last 182.71) - open/close 25270 (time 1764737651.60 total 187.07 last 180.02) - open/close 26869 (time 1764737661.61 total 197.07 last 159.80) - open/close 28257 (time 1764737671.61 total 207.08 last 138.75) - open/close 29966 (time 1764737681.61 total 217.08 last 170.88) - open/close 31775 (time 1764737691.62 total 227.08 last 180.80) - open/close 33938 (time 1764737701.62 total 237.09 last 216.21) - open/close 36014 (time 1764737711.62 total 247.09 last 207.59) - open/close 37626 (time 1764737721.63 total 257.09 last 161.13) - open/close 38992 (time 1764737731.63 total 267.10 last 136.50) - open/close 40000 (time 1764737738.64 total 274.10 last 143.89) - open/close 41609 (time 1764737748.64 total 284.11 last 160.87) - open/close 43214 (time 1764737758.64 total 294.11 last 160.49) - open/close 45031 (time 1764737768.64 total 304.11 last 181.67) - open/close 46436 (time 1764737778.65 total 314.11 last 140.44) - open/close 48207 (time 1764737788.65 total 324.11 last 177.09) - open/close 50000 (time 1764737798.38 total 333.84 last 184.27) - open/close 51896 (time 1764737808.38 total 343.84 last 189.59) - open/close 53707 (time 1764737818.38 total 353.84 last 181.10) - open/close 55509 (time 1764737828.38 total 363.85 last 180.14) - open/close 57590 (time 1764737838.39 total 373.85 last 208.03) - open/close 59513 (time 1764737848.39 total 383.86 last 192.20) - open/close 60000 (time 1764737850.81 total 386.27 last 201.52) - open/close 61920 (time 1764737860.81 total 396.28 last 191.96) - open/close 64343 (time 1764737870.81 total 406.28 last 242.25) total: 64768 open/close in 407.94 seconds: 158.77 ops/second - /unlink 4230 (time 1764737884.33 total 10.00 last 422.93) - /unlink 9233 (time 1764737894.33 total 20.00 last 500.24) - /unlink 14536 (time 1764737904.33 total 30.00 last 530.27) - /unlink 19643 (time 1764737914.33 total 40.00 last 510.63) - /unlink 23705 (time 1764737924.34 total 50.01 last 406.12) - /unlink 27237 (time 1764737934.34 total 60.01 last 353.14) - /unlink 30000 (time 1764737942.03 total 67.70 last 359.25) - /unlink 33823 (time 1764737952.03 total 77.70 last 382.26) - /unlink 37087 (time 1764737962.04 total 87.71 last 326.25) - /unlink 40000 (time 1764737969.28 total 94.95 last 402.29) - /unlink 44099 (time 1764737979.28 total 104.95 last 409.83) - /unlink 47564 (time 1764737989.28 total 114.95 last 346.47) - /unlink 50000 (time 1764737995.47 total 121.14 last 393.43) - /unlink 55445 (time 1764738005.47 total 131.14 last 544.47) - /unlink 60000 (time 1764738014.09 total 139.76 last 528.71) total: 64768 /unlink in 148.49 seconds: 436.19 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2533 (time 1764738055.19 total 10.00 last 253.28) - open/close 4497 (time 1764738065.19 total 20.00 last 196.35) - open/close 6497 (time 1764738075.19 total 30.01 last 199.96) - open/close 8244 (time 1764738085.20 total 40.01 last 174.63) - open/close 10000 (time 1764738093.16 total 47.97 last 220.62) - open/close 12230 (time 1764738103.16 total 57.98 last 222.85) - open/close 13876 (time 1764738113.16 total 67.98 last 164.59) - open/close 15678 (time 1764738123.17 total 77.98 last 180.13) - open/close 18236 (time 1764738133.17 total 87.98 last 255.73) - open/close 20000 (time 1764738143.06 total 97.88 last 178.31) - open/close 21346 (time 1764738153.07 total 107.88 last 134.54) - open/close 22854 (time 1764738163.07 total 117.88 last 150.77) - open/close 24034 (time 1764738173.07 total 127.88 last 117.99) - open/close 25338 (time 1764738183.07 total 137.88 last 130.39) - open/close 26735 (time 1764738193.08 total 147.89 last 139.59) - open/close 28513 (time 1764738203.08 total 157.89 last 177.76) - open/close 30000 (time 1764738209.15 total 163.96 last 245.07) - open/close 31954 (time 1764738219.15 total 173.97 last 195.30) - open/close 33394 (time 1764738229.16 total 183.97 last 143.95) - open/close 35395 (time 1764738239.16 total 193.97 last 200.07) - open/close 37505 (time 1764738249.16 total 203.97 last 210.96) - open/close 39360 (time 1764738259.16 total 213.98 last 185.45) - open/close 40000 (time 1764738262.86 total 217.67 last 173.18) - open/close 42106 (time 1764738272.86 total 227.68 last 210.50) - open/close 44247 (time 1764738282.87 total 237.68 last 214.02) - open/close 46194 (time 1764738292.87 total 247.68 last 194.65) - open/close 48974 (time 1764738302.87 total 257.68 last 277.94) - open/close 50000 (time 1764738307.78 total 262.59 last 209.23) - open/close 52207 (time 1764738317.78 total 272.59 last 220.66) - open/close 54364 (time 1764738327.78 total 282.59 last 215.65) - open/close 56828 (time 1764738337.78 total 292.59 last 246.37) - open/close 59581 (time 1764738347.78 total 302.59 last 275.29) - open/close 62052 (time 1764738357.79 total 312.60 last 247.02) total: 64768 open/close in 321.05 seconds: 201.74 ops/second - /unlink 5469 (time 1764738377.71 total 10.00 last 546.87) - /unlink 10000 (time 1764738387.37 total 19.66 last 469.31) - /unlink 15028 (time 1764738397.37 total 29.66 last 502.77) - /unlink 19401 (time 1764738407.37 total 39.66 last 437.20) - /unlink 23042 (time 1764738417.37 total 49.66 last 364.02) - /unlink 26887 (time 1764738427.37 total 59.66 last 384.43) - /unlink 30000 (time 1764738436.39 total 68.68 last 345.26) - /unlink 33326 (time 1764738446.39 total 78.68 last 332.53) - /unlink 36880 (time 1764738456.39 total 88.68 last 355.36) - /unlink 40000 (time 1764738463.82 total 96.11 last 420.20) - /unlink 44472 (time 1764738473.82 total 106.11 last 447.16) - /unlink 48338 (time 1764738483.82 total 116.11 last 386.58) - /unlink 50000 (time 1764738487.41 total 119.70 last 462.43) - /unlink 55355 (time 1764738497.41 total 129.70 last 535.40) - /unlink 60000 (time 1764738506.27 total 138.56 last 524.34) total: 64768 /unlink in 146.57 seconds: 441.89 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg156-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg156-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg156-server stop mds service on oleg156-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg156-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg156-server' oleg156-server: oleg156-server.virtnet: executing unload_modules_local oleg156-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg156-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg156-client: oleg156-client: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-server: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-client: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-server: ssh exited with exit code 2