== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:22:26 (1764937346) Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f) oleg238-server: oleg238-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing load_modules_local oleg238-server: Loading modules from /home/green/git/lustre-release/lustre oleg238-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg238-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg238-server: detected 4 online CPUs by sysfs oleg238-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg238-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg238-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock oleg238-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16166 358 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 738 (time 1764937456.16 total 10.01 last 73.73) - open/close 1539 (time 1764937466.17 total 20.02 last 80.02) - open/close 2242 (time 1764937476.18 total 30.03 last 70.25) - open/close 2971 (time 1764937486.19 total 40.03 last 72.83) - open/close 3700 (time 1764937496.19 total 50.03 last 72.90) - open/close 4416 (time 1764937506.19 total 60.04 last 71.59) - open/close 5159 (time 1764937516.20 total 70.05 last 74.22) - open/close 5895 (time 1764937526.20 total 80.05 last 73.57) - open/close 6592 (time 1764937536.21 total 90.06 last 69.65) - open/close 7299 (time 1764937546.22 total 100.06 last 70.67) - open/close 8041 (time 1764937556.22 total 110.07 last 74.16) - open/close 8731 (time 1764937566.23 total 120.08 last 68.95) - open/close 9442 (time 1764937576.25 total 130.10 last 70.93) - open/close 10000 (time 1764937583.14 total 136.99 last 81.02) - open/close 10750 (time 1764937593.15 total 146.99 last 74.95) - open/close 11436 (time 1764937603.16 total 157.00 last 68.53) - open/close 12133 (time 1764937613.16 total 167.01 last 69.68) - open/close 12818 (time 1764937623.17 total 177.02 last 68.42) - open/close 13521 (time 1764937633.18 total 187.03 last 70.25) - open/close 14306 (time 1764937643.18 total 197.03 last 78.49) - open/close 15061 (time 1764937653.19 total 207.04 last 75.42) - open/close 15752 (time 1764937663.20 total 217.05 last 69.02) - open/close 16436 (time 1764937673.20 total 227.05 last 68.39) - open/close 17152 (time 1764937683.21 total 237.06 last 71.53) - open/close 17880 (time 1764937693.22 total 247.07 last 72.76) - open/close 18677 (time 1764937703.22 total 257.07 last 79.69) - open/close 19559 (time 1764937713.22 total 267.07 last 88.20) - open/close 20000 (time 1764937718.17 total 272.02 last 89.07) - open/close 20778 (time 1764937728.17 total 282.02 last 77.80) - open/close 21604 (time 1764937738.18 total 292.02 last 82.56) - open/close 22423 (time 1764937748.18 total 302.03 last 81.88) - open/close 23215 (time 1764937758.18 total 312.03 last 79.19) - open/close 23985 (time 1764937768.19 total 322.04 last 76.89) - open/close 24817 (time 1764937778.20 total 332.05 last 83.13) - open/close 25609 (time 1764937788.21 total 342.06 last 79.13) - open/close 26425 (time 1764937798.22 total 352.07 last 81.51) - open/close 27316 (time 1764937808.23 total 362.08 last 89.02) - open/close 28137 (time 1764937818.23 total 372.08 last 82.09) - open/close 28946 (time 1764937828.24 total 382.09 last 80.84) - open/close 29675 (time 1764937838.24 total 392.09 last 72.87) - open/close 30000 (time 1764937843.08 total 396.93 last 67.20) - open/close 30807 (time 1764937853.08 total 406.93 last 80.69) - open/close 31649 (time 1764937863.09 total 416.94 last 84.10) - open/close 32475 (time 1764937873.10 total 426.95 last 82.54) - open/close 33300 (time 1764937883.11 total 436.96 last 82.45) - open/close 34031 (time 1764937893.11 total 446.96 last 73.09) - open/close 34722 (time 1764937903.11 total 456.96 last 69.07) - open/close 35390 (time 1764937913.11 total 466.96 last 66.80) - open/close 36158 (time 1764937923.12 total 476.97 last 76.71) - open/close 36942 (time 1764937933.13 total 486.98 last 78.33) - open/close 37760 (time 1764937943.14 total 496.99 last 81.75) - open/close 38611 (time 1764937953.14 total 506.99 last 85.07) - open/close 39452 (time 1764937963.16 total 517.01 last 83.94) - open/close 40000 (time 1764937969.79 total 523.64 last 82.68) - open/close 40834 (time 1764937979.79 total 533.64 last 83.38) - open/close 41677 (time 1764937989.80 total 543.64 last 84.27) - open/close 42553 (time 1764937999.81 total 553.66 last 87.47) - open/close 43357 (time 1764938009.82 total 563.66 last 80.36) - open/close 44134 (time 1764938019.82 total 573.67 last 77.68) - open/close 44985 (time 1764938029.82 total 583.67 last 85.08) - open/close 45731 (time 1764938039.82 total 593.67 last 74.58) - open/close 46560 (time 1764938049.83 total 603.68 last 82.85) - open/close 47363 (time 1764938059.83 total 613.68 last 80.27) - open/close 48161 (time 1764938069.84 total 623.68 last 79.78) - open/close 48986 (time 1764938079.84 total 633.68 last 82.50) - open/close 49822 (time 1764938089.84 total 643.69 last 83.59) - open/close 50000 (time 1764938091.90 total 645.75 last 86.41) - open/close 50798 (time 1764938101.91 total 655.76 last 79.68) - open/close 51612 (time 1764938111.92 total 665.76 last 81.38) - open/close 52477 (time 1764938121.92 total 675.77 last 86.46) - open/close 53306 (time 1764938131.92 total 685.77 last 82.86) - open/close 54127 (time 1764938141.93 total 695.78 last 82.06) - open/close 54896 (time 1764938151.95 total 705.80 last 76.73) - open/close 55719 (time 1764938161.95 total 715.80 last 82.29) - open/close 56520 (time 1764938171.96 total 725.81 last 80.03) - open/close 57319 (time 1764938181.97 total 735.81 last 79.86) - open/close 58147 (time 1764938191.97 total 745.82 last 82.79) - open/close 59009 (time 1764938201.98 total 755.83 last 86.11) - open/close 59880 (time 1764938211.99 total 765.84 last 86.99) - open/close 60930 (time 1764938222.00 total 775.85 last 104.90) - open/close 61940 (time 1764938232.01 total 785.86 last 100.87) - open/close 62716 (time 1764938242.02 total 795.87 last 77.56) - open/close 63527 (time 1764938252.07 total 805.91 last 80.70) - open/close 64253 (time 1764938262.07 total 815.92 last 72.56) total: 64768 open/close in 822.46 seconds: 78.75 ops/second - /unlink 1215 (time 1764938281.98 total 10.00 last 121.46) - /unlink 2498 (time 1764938291.98 total 20.01 last 128.26) - /unlink 3803 (time 1764938301.99 total 30.02 last 130.36) - /unlink 5132 (time 1764938312.00 total 40.02 last 132.80) - /unlink 6509 (time 1764938322.00 total 50.02 last 137.69) - /unlink 8006 (time 1764938332.00 total 60.03 last 149.67) - /unlink 9452 (time 1764938342.00 total 70.03 last 144.56) - /unlink 10000 (time 1764938346.21 total 74.23 last 130.36) - /unlink 11471 (time 1764938356.22 total 84.24 last 146.97) - /unlink 12966 (time 1764938366.22 total 94.24 last 149.47) - /unlink 14288 (time 1764938376.22 total 104.25 last 132.18) - /unlink 15581 (time 1764938386.23 total 114.25 last 129.24) - /unlink 16836 (time 1764938396.23 total 124.26 last 125.39) - /unlink 18094 (time 1764938406.24 total 134.26 last 125.77) - /unlink 19426 (time 1764938416.24 total 144.26 last 133.16) - /unlink 20000 (time 1764938420.58 total 148.60 last 132.33) - /unlink 21288 (time 1764938430.58 total 158.61 last 128.76) - /unlink 22605 (time 1764938440.58 total 168.61 last 131.69) - /unlink 24114 (time 1764938450.59 total 178.62 last 150.76) - /unlink 25450 (time 1764938460.59 total 188.62 last 133.57) - /unlink 26652 (time 1764938470.60 total 198.62 last 120.16) - /unlink 27910 (time 1764938480.60 total 208.62 last 125.79) - /unlink 29233 (time 1764938490.60 total 218.63 last 132.25) - /unlink 30000 (time 1764938496.98 total 225.01 last 120.16) - /unlink 31228 (time 1764938506.99 total 235.01 last 122.73) - /unlink 32507 (time 1764938516.99 total 245.02 last 127.86) - /unlink 33717 (time 1764938527.00 total 255.02 last 120.95) - /unlink 34922 (time 1764938537.00 total 265.03 last 120.44) - /unlink 36138 (time 1764938547.01 total 275.03 last 121.53) - /unlink 37447 (time 1764938557.01 total 285.03 last 130.89) - /unlink 38761 (time 1764938567.01 total 295.04 last 131.34) - /unlink 40000 (time 1764938575.64 total 303.67 last 143.59) - /unlink 41309 (time 1764938585.65 total 313.67 last 130.82) - /unlink 42675 (time 1764938595.66 total 323.68 last 136.48) - /unlink 43916 (time 1764938605.68 total 333.70 last 123.82) - /unlink 45182 (time 1764938615.68 total 343.70 last 126.60) - /unlink 46489 (time 1764938625.68 total 353.71 last 130.63) - /unlink 47790 (time 1764938635.69 total 363.71 last 130.07) - /unlink 49108 (time 1764938645.69 total 373.72 last 131.74) - /unlink 50000 (time 1764938652.59 total 380.62 last 129.23) - /unlink 51284 (time 1764938662.60 total 390.63 last 128.30) - /unlink 52704 (time 1764938672.61 total 400.64 last 141.87) - /unlink 54017 (time 1764938682.61 total 410.64 last 131.26) - /unlink 55382 (time 1764938692.64 total 420.67 last 136.11) - /unlink 56694 (time 1764938702.65 total 430.68 last 131.08) - /unlink 58010 (time 1764938712.66 total 440.68 last 131.55) - /unlink 59267 (time 1764938722.67 total 450.69 last 125.53) - /unlink 60000 (time 1764938728.75 total 456.78 last 120.51) - /unlink 61319 (time 1764938738.77 total 466.79 last 131.66) - /unlink 62572 (time 1764938748.78 total 476.81 last 125.11) - /unlink 63893 (time 1764938758.79 total 486.81 last 132.06) total: 64768 /unlink in 494.16 seconds: 131.07 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 883 (time 1764938800.65 total 10.01 last 88.19) - open/close 1764 (time 1764938810.65 total 20.02 last 88.07) - open/close 2659 (time 1764938820.66 total 30.03 last 89.39) - open/close 3530 (time 1764938830.67 total 40.04 last 87.02) - open/close 4338 (time 1764938840.68 total 50.05 last 80.74) - open/close 5171 (time 1764938850.69 total 60.05 last 83.23) - open/close 6046 (time 1764938860.69 total 70.06 last 87.47) - open/close 6836 (time 1764938870.72 total 80.08 last 78.80) - open/close 7551 (time 1764938880.72 total 90.08 last 71.50) - open/close 8325 (time 1764938890.73 total 100.09 last 77.36) - open/close 9184 (time 1764938900.73 total 110.09 last 85.87) - open/close 10000 (time 1764938910.10 total 119.46 last 87.10) - open/close 10906 (time 1764938920.11 total 129.47 last 90.52) - open/close 11810 (time 1764938930.12 total 139.49 last 90.24) - open/close 12743 (time 1764938940.14 total 149.51 last 93.13) - open/close 13705 (time 1764938950.15 total 159.51 last 96.13) - open/close 14530 (time 1764938960.15 total 169.52 last 82.48) - open/close 15342 (time 1764938970.15 total 179.52 last 81.18) - open/close 16125 (time 1764938980.17 total 189.53 last 78.21) - open/close 16912 (time 1764938990.17 total 199.53 last 78.69) - open/close 17776 (time 1764939000.17 total 209.54 last 86.35) - open/close 18561 (time 1764939010.18 total 219.54 last 78.47) - open/close 19439 (time 1764939020.19 total 229.55 last 87.71) - open/close 20000 (time 1764939026.89 total 236.25 last 83.72) - open/close 20847 (time 1764939036.89 total 246.26 last 84.65) - open/close 21667 (time 1764939046.90 total 256.26 last 81.96) - open/close 22459 (time 1764939056.93 total 266.29 last 78.98) - open/close 23234 (time 1764939066.94 total 276.30 last 77.40) - open/close 24034 (time 1764939076.94 total 286.31 last 79.99) - open/close 24837 (time 1764939086.96 total 296.32 last 80.19) - open/close 25661 (time 1764939096.97 total 306.33 last 82.32) - open/close 26454 (time 1764939106.97 total 316.33 last 79.28) - open/close 27336 (time 1764939116.98 total 326.34 last 88.11) - open/close 28157 (time 1764939127.00 total 336.36 last 81.95) - open/close 28974 (time 1764939137.00 total 346.36 last 81.67) - open/close 29808 (time 1764939147.01 total 356.37 last 83.32) - open/close 30000 (time 1764939149.46 total 358.83 last 78.25) - open/close 30823 (time 1764939159.47 total 368.83 last 82.24) - open/close 31654 (time 1764939169.48 total 378.84 last 83.05) - open/close 32424 (time 1764939179.48 total 388.84 last 76.99) - open/close 33234 (time 1764939189.49 total 398.85 last 80.91) - open/close 34021 (time 1764939199.50 total 408.86 last 78.64) - open/close 34852 (time 1764939209.50 total 418.86 last 83.09) - open/close 35729 (time 1764939219.50 total 428.87 last 87.65) - open/close 36588 (time 1764939229.51 total 438.87 last 85.85) - open/close 37375 (time 1764939239.52 total 448.88 last 78.65) - open/close 38222 (time 1764939249.52 total 458.88 last 84.67) - open/close 39089 (time 1764939259.53 total 468.89 last 86.61) - open/close 39951 (time 1764939269.53 total 478.90 last 86.17) - open/close 40854 (time 1764939279.54 total 488.91 last 90.22) - open/close 41683 (time 1764939289.54 total 498.91 last 82.88) - open/close 42608 (time 1764939299.55 total 508.91 last 92.46) - open/close 43434 (time 1764939309.55 total 518.91 last 82.59) - open/close 44232 (time 1764939319.55 total 528.92 last 79.76) - open/close 45077 (time 1764939329.56 total 538.92 last 84.47) - open/close 45933 (time 1764939339.57 total 548.93 last 85.51) - open/close 46877 (time 1764939349.57 total 558.94 last 94.36) - open/close 47752 (time 1764939359.58 total 568.94 last 87.49) - open/close 48623 (time 1764939369.58 total 578.94 last 87.08) - open/close 49429 (time 1764939379.59 total 588.95 last 80.49) - open/close 50000 (time 1764939386.27 total 595.63 last 85.48) - open/close 50880 (time 1764939396.28 total 605.64 last 87.92) - open/close 51762 (time 1764939406.29 total 615.65 last 88.12) - open/close 52615 (time 1764939416.29 total 625.66 last 85.26) - open/close 53523 (time 1764939426.30 total 635.67 last 90.73) - open/close 54394 (time 1764939436.31 total 645.68 last 87.00) - open/close 55260 (time 1764939446.32 total 655.69 last 86.51) - open/close 56230 (time 1764939456.33 total 665.69 last 96.96) - open/close 57111 (time 1764939466.33 total 675.69 last 88.08) - open/close 57867 (time 1764939476.33 total 685.70 last 75.58) - open/close 58685 (time 1764939486.33 total 695.70 last 81.79) - open/close 59519 (time 1764939496.34 total 705.70 last 83.35) - open/close 60000 (time 1764939502.27 total 711.63 last 81.13) - open/close 60734 (time 1764939512.27 total 721.64 last 73.37) - open/close 61502 (time 1764939522.28 total 731.64 last 76.73) - open/close 62320 (time 1764939532.29 total 741.65 last 81.75) - open/close 63191 (time 1764939542.34 total 751.70 last 86.64) - open/close 63956 (time 1764939552.35 total 761.71 last 76.44) - open/close 64761 (time 1764939562.36 total 771.72 last 80.44) total: 64768 open/close in 771.79 seconds: 83.92 ops/second - /unlink 1142 (time 1764939575.64 total 10.00 last 114.16) - /unlink 2308 (time 1764939585.64 total 20.01 last 116.56) - /unlink 3494 (time 1764939595.65 total 30.01 last 118.53) - /unlink 4786 (time 1764939605.65 total 40.02 last 129.17) - /unlink 6204 (time 1764939615.65 total 50.02 last 141.75) - /unlink 7727 (time 1764939625.66 total 60.03 last 152.17) - /unlink 9305 (time 1764939635.66 total 70.03 last 157.71) - /unlink 10000 (time 1764939640.23 total 74.60 last 152.13) - /unlink 11423 (time 1764939650.24 total 84.61 last 142.24) - /unlink 12912 (time 1764939660.24 total 94.61 last 148.84) - /unlink 14213 (time 1764939670.24 total 104.61 last 130.07) - /unlink 15768 (time 1764939680.25 total 114.62 last 155.41) - /unlink 17412 (time 1764939690.26 total 124.63 last 164.28) - /unlink 18732 (time 1764939700.26 total 134.63 last 131.95) - /unlink 20000 (time 1764939708.56 total 142.93 last 152.75) - /unlink 21418 (time 1764939718.57 total 152.94 last 141.61) - /unlink 22917 (time 1764939728.58 total 162.94 last 149.88) - /unlink 24387 (time 1764939738.58 total 172.95 last 146.98) - /unlink 25903 (time 1764939748.58 total 182.95 last 151.56) - /unlink 27435 (time 1764939758.59 total 192.96 last 153.00) - /unlink 29185 (time 1764939768.60 total 202.97 last 174.87) - /unlink 30000 (time 1764939773.58 total 207.95 last 163.68) - /unlink 31565 (time 1764939783.59 total 217.96 last 156.34) - /unlink 32940 (time 1764939793.59 total 227.96 last 137.47) - /unlink 34250 (time 1764939803.60 total 237.97 last 130.92) - /unlink 35537 (time 1764939813.61 total 247.98 last 128.57) - /unlink 37213 (time 1764939823.61 total 257.98 last 167.54) - /unlink 38923 (time 1764939833.61 total 267.98 last 170.95) - /unlink 40000 (time 1764939840.14 total 274.51 last 164.99) - /unlink 41562 (time 1764939850.14 total 284.51 last 156.17) - /unlink 43265 (time 1764939860.15 total 294.52 last 170.21) - /unlink 45272 (time 1764939870.15 total 304.52 last 200.70) - /unlink 47147 (time 1764939880.15 total 314.52 last 187.47) - /unlink 48714 (time 1764939890.15 total 324.52 last 156.67) - /unlink 50000 (time 1764939897.65 total 332.02 last 171.51) - /unlink 51817 (time 1764939907.65 total 342.02 last 181.64) - /unlink 53845 (time 1764939917.66 total 352.02 last 202.78) - /unlink 55568 (time 1764939927.66 total 362.03 last 172.18) - /unlink 57046 (time 1764939937.67 total 372.04 last 147.73) - /unlink 58621 (time 1764939947.67 total 382.04 last 157.48) - /unlink 60000 (time 1764939955.77 total 390.14 last 170.16) - /unlink 61629 (time 1764939965.78 total 400.14 last 162.84) - /unlink 63106 (time 1764939975.79 total 410.16 last 147.51) - /unlink 64666 (time 1764939985.79 total 420.16 last 155.95) total: 64768 /unlink in 420.65 seconds: 153.97 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1065 (time 1764940017.76 total 10.00 last 106.47) - open/close 2058 (time 1764940027.76 total 20.01 last 99.26) - open/close 3087 (time 1764940037.78 total 30.02 last 102.76) - open/close 4040 (time 1764940047.78 total 40.02 last 95.26) - open/close 4983 (time 1764940057.78 total 50.03 last 94.27) - open/close 5871 (time 1764940067.79 total 60.04 last 88.72) - open/close 6855 (time 1764940077.79 total 70.04 last 98.38) - open/close 7652 (time 1764940087.80 total 80.04 last 79.66) - open/close 8547 (time 1764940097.82 total 90.06 last 89.33) - open/close 9448 (time 1764940107.82 total 100.06 last 90.10) - open/close 10000 (time 1764940113.30 total 105.55 last 100.65) - open/close 11080 (time 1764940123.31 total 115.56 last 107.89) - open/close 12262 (time 1764940133.31 total 125.56 last 118.19) - open/close 13508 (time 1764940143.32 total 135.56 last 124.51) - open/close 14668 (time 1764940153.32 total 145.57 last 115.98) - open/close 15896 (time 1764940163.35 total 155.59 last 122.47) - open/close 17279 (time 1764940173.35 total 165.59 last 138.28) - open/close 18700 (time 1764940183.35 total 175.60 last 142.06) - open/close 19981 (time 1764940193.36 total 185.60 last 128.04) - open/close 21507 (time 1764940203.36 total 195.60 last 152.58) - open/close 23197 (time 1764940213.36 total 205.60 last 168.97) - open/close 24392 (time 1764940223.37 total 215.62 last 119.36) - open/close 25607 (time 1764940233.37 total 225.62 last 121.48) - open/close 26685 (time 1764940243.37 total 235.62 last 107.80) - open/close 27822 (time 1764940253.39 total 245.63 last 113.57) - open/close 28880 (time 1764940263.39 total 255.63 last 105.73) - open/close 30000 (time 1764940272.28 total 264.52 last 126.04) - open/close 31251 (time 1764940282.29 total 274.53 last 124.98) - open/close 32405 (time 1764940292.29 total 284.53 last 115.36) - open/close 33397 (time 1764940302.30 total 294.54 last 99.09) - open/close 34603 (time 1764940312.31 total 304.55 last 120.52) - open/close 35874 (time 1764940322.32 total 314.56 last 127.01) - open/close 37027 (time 1764940332.32 total 324.56 last 115.23) - open/close 38118 (time 1764940342.32 total 334.57 last 109.09) - open/close 39152 (time 1764940352.32 total 344.57 last 103.40) - open/close 40000 (time 1764940359.64 total 351.88 last 115.93) - open/close 41216 (time 1764940369.64 total 361.88 last 121.55) - open/close 42254 (time 1764940379.64 total 371.89 last 103.79) - open/close 43265 (time 1764940389.65 total 381.89 last 101.02) - open/close 44537 (time 1764940399.65 total 391.90 last 127.14) - open/close 45740 (time 1764940409.66 total 401.90 last 120.24) - open/close 46988 (time 1764940419.66 total 411.91 last 124.75) - open/close 48460 (time 1764940429.67 total 421.91 last 147.18) - open/close 49662 (time 1764940439.67 total 431.91 last 120.16) - open/close 50000 (time 1764940442.17 total 434.42 last 134.90) - open/close 51141 (time 1764940452.17 total 444.42 last 114.09) - open/close 52206 (time 1764940462.18 total 454.43 last 106.40) - open/close 53404 (time 1764940472.19 total 464.43 last 119.74) - open/close 54730 (time 1764940482.20 total 474.44 last 132.46) - open/close 56063 (time 1764940492.20 total 484.45 last 133.26) - open/close 57381 (time 1764940502.21 total 494.45 last 131.76) - open/close 58859 (time 1764940512.21 total 504.46 last 147.68) - open/close 60000 (time 1764940521.77 total 514.02 last 119.35) - open/close 61288 (time 1764940531.78 total 524.02 last 128.77) - open/close 62645 (time 1764940541.78 total 534.02 last 135.68) - open/close 63735 (time 1764940551.78 total 544.02 last 108.98) total: 64768 open/close in 552.79 seconds: 117.17 ops/second - /unlink 2443 (time 1764940572.50 total 10.00 last 244.27) - /unlink 4963 (time 1764940582.50 total 20.00 last 251.92) - /unlink 7277 (time 1764940592.50 total 30.01 last 231.36) - /unlink 10000 (time 1764940602.39 total 39.89 last 275.40) - /unlink 12798 (time 1764940612.39 total 49.90 last 279.67) - /unlink 15829 (time 1764940622.39 total 59.90 last 303.07) - /unlink 18859 (time 1764940632.40 total 69.90 last 302.95) - /unlink 20000 (time 1764940635.54 total 73.05 last 362.71) - /unlink 23849 (time 1764940645.54 total 83.05 last 384.82) - /unlink 27048 (time 1764940655.55 total 93.05 last 319.84) - /unlink 30000 (time 1764940664.69 total 102.20 last 322.75) - /unlink 33189 (time 1764940674.69 total 112.20 last 318.87) - /unlink 36513 (time 1764940684.69 total 122.20 last 332.37) - /unlink 39873 (time 1764940694.69 total 132.20 last 335.98) - /unlink 44061 (time 1764940704.69 total 142.20 last 418.79) - /unlink 47358 (time 1764940714.70 total 152.20 last 329.59) - /unlink 50000 (time 1764940723.28 total 160.79 last 307.68) - /unlink 53753 (time 1764940733.29 total 170.79 last 375.22) - /unlink 57678 (time 1764940743.29 total 180.79 last 392.40) - /unlink 60000 (time 1764940750.74 total 188.24 last 311.65) - /unlink 62780 (time 1764940760.74 total 198.25 last 277.97) total: 64768 /unlink in 205.50 seconds: 315.18 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2767 (time 1764940794.72 total 10.00 last 276.66) - open/close 5127 (time 1764940804.72 total 20.00 last 235.96) - open/close 7608 (time 1764940814.72 total 30.00 last 248.09) - open/close 9829 (time 1764940824.72 total 40.01 last 221.95) - open/close 11844 (time 1764940834.73 total 50.02 last 201.40) - open/close 13814 (time 1764940844.73 total 60.02 last 196.92) - open/close 15960 (time 1764940854.74 total 70.02 last 214.56) - open/close 17608 (time 1764940864.74 total 80.02 last 164.74) - open/close 20000 (time 1764940874.49 total 89.77 last 245.41) - open/close 22431 (time 1764940884.49 total 99.78 last 243.01) - open/close 24281 (time 1764940894.49 total 109.78 last 184.98) - open/close 26696 (time 1764940904.49 total 119.78 last 241.48) - open/close 29601 (time 1764940914.49 total 129.78 last 290.48) - open/close 32546 (time 1764940924.49 total 139.78 last 294.47) - open/close 35078 (time 1764940934.49 total 149.78 last 253.16) - open/close 37748 (time 1764940944.50 total 159.78 last 266.98) - open/close 39868 (time 1764940954.50 total 169.78 last 211.96) - open/close 42114 (time 1764940964.50 total 179.79 last 224.51) - open/close 43991 (time 1764940974.51 total 189.79 last 187.60) - open/close 45636 (time 1764940984.51 total 199.79 last 164.47) - open/close 47667 (time 1764940994.51 total 209.80 last 203.05) - open/close 49193 (time 1764941004.51 total 219.80 last 152.59) - open/close 50000 (time 1764941010.25 total 225.54 last 140.59) - open/close 51691 (time 1764941020.25 total 235.54 last 169.10) - open/close 53681 (time 1764941030.26 total 245.54 last 198.93) - open/close 55580 (time 1764941040.26 total 255.54 last 189.85) - open/close 57097 (time 1764941050.26 total 265.54 last 151.69) - open/close 58782 (time 1764941060.26 total 275.55 last 168.42) - open/close 60000 (time 1764941066.40 total 281.68 last 198.62) - open/close 62037 (time 1764941076.40 total 291.69 last 203.60) - open/close 63742 (time 1764941086.41 total 301.69 last 170.43) total: 64768 open/close in 306.80 seconds: 211.11 ops/second - /unlink 3214 (time 1764941103.31 total 10.00 last 321.25) - /unlink 6814 (time 1764941113.32 total 20.01 last 359.68) - /unlink 10000 (time 1764941119.41 total 26.11 last 522.96) - /unlink 16152 (time 1764941129.41 total 36.11 last 615.16) - /unlink 20000 (time 1764941139.20 total 45.90 last 393.10) - /unlink 23893 (time 1764941149.20 total 55.90 last 389.24) - /unlink 28402 (time 1764941159.22 total 65.91 last 450.33) - /unlink 30000 (time 1764941162.66 total 69.36 last 463.55) - /unlink 34516 (time 1764941172.66 total 79.36 last 451.57) - /unlink 39357 (time 1764941182.66 total 89.36 last 484.08) - /unlink 43872 (time 1764941192.66 total 99.36 last 451.47) - /unlink 48854 (time 1764941202.66 total 109.36 last 498.19) - /unlink 54512 (time 1764941212.67 total 119.36 last 565.76) - /unlink 59250 (time 1764941222.67 total 129.36 last 473.73) - /unlink 64183 (time 1764941232.67 total 139.36 last 493.27) total: 64768 /unlink in 140.81 seconds: 459.98 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2285 (time 1764941261.05 total 10.00 last 228.46) - open/close 4501 (time 1764941271.05 total 20.01 last 221.52) - open/close 6930 (time 1764941281.05 total 30.01 last 242.83) - open/close 9234 (time 1764941291.06 total 40.01 last 230.35) - open/close 10000 (time 1764941294.33 total 43.29 last 233.77) - open/close 12064 (time 1764941304.33 total 53.29 last 206.37) - open/close 14266 (time 1764941314.33 total 63.29 last 220.20) - open/close 16838 (time 1764941324.34 total 73.29 last 257.11) - open/close 19406 (time 1764941334.34 total 83.29 last 256.74) - open/close 20000 (time 1764941336.40 total 85.36 last 287.76) - open/close 23325 (time 1764941346.41 total 95.36 last 332.42) - open/close 26640 (time 1764941356.41 total 105.37 last 331.29) - open/close 29546 (time 1764941366.41 total 115.37 last 290.56) - open/close 32066 (time 1764941376.42 total 125.37 last 251.96) - open/close 34139 (time 1764941386.42 total 135.37 last 207.30) - open/close 36286 (time 1764941396.42 total 145.37 last 214.68) - open/close 39126 (time 1764941406.42 total 155.37 last 283.92) - open/close 40000 (time 1764941409.36 total 158.32 last 297.06) - open/close 42738 (time 1764941419.36 total 168.32 last 273.80) - open/close 45335 (time 1764941429.36 total 178.32 last 259.69) - open/close 48297 (time 1764941439.37 total 188.32 last 296.11) - open/close 50000 (time 1764941445.48 total 194.43 last 278.62) - open/close 53313 (time 1764941455.48 total 204.43 last 331.30) - open/close 56244 (time 1764941465.48 total 214.43 last 293.04) - open/close 59069 (time 1764941475.48 total 224.44 last 282.43) - open/close 60000 (time 1764941478.51 total 227.46 last 307.39) - open/close 62621 (time 1764941488.51 total 237.47 last 262.01) total: 64768 open/close in 244.90 seconds: 264.47 ops/second - /unlink 5988 (time 1764941507.41 total 10.00 last 598.78) - /unlink 10000 (time 1764941515.53 total 18.12 last 493.89) - /unlink 15538 (time 1764941525.53 total 28.12 last 553.77) - /unlink 20000 (time 1764941533.92 total 36.51 last 531.83) - /unlink 25912 (time 1764941543.92 total 46.51 last 591.15) - /unlink 30000 (time 1764941549.86 total 52.46 last 687.78) - /unlink 33706 (time 1764941559.87 total 62.46 last 370.55) - /unlink 35049 (time 1764941569.88 total 72.47 last 134.16) - /unlink 36488 (time 1764941579.88 total 82.47 last 143.89) - /unlink 37635 (time 1764941589.88 total 92.48 last 114.63) - /unlink 38898 (time 1764941599.89 total 102.48 last 126.26) - /unlink 40000 (time 1764941609.09 total 111.68 last 119.76) - /unlink 41280 (time 1764941619.09 total 121.69 last 127.95) - /unlink 42766 (time 1764941629.10 total 131.69 last 148.51) - /unlink 44282 (time 1764941639.10 total 141.69 last 151.58) - /unlink 46520 (time 1764941649.10 total 151.70 last 223.76) - /unlink 48641 (time 1764941659.10 total 161.70 last 212.02) - /unlink 50000 (time 1764941667.13 total 169.72 last 169.41) - /unlink 51610 (time 1764941677.13 total 179.72 last 160.96) - /unlink 53211 (time 1764941687.13 total 189.73 last 160.01) - /unlink 54960 (time 1764941697.14 total 199.73 last 174.85) - /unlink 56762 (time 1764941707.14 total 209.73 last 180.16) - /unlink 58645 (time 1764941717.14 total 219.74 last 188.26) - /unlink 60000 (time 1764941723.63 total 226.22 last 208.91) - /unlink 62518 (time 1764941733.63 total 236.23 last 251.61) - /unlink 64094 (time 1764941743.64 total 246.24 last 157.52) total: 64768 /unlink in 250.27 seconds: 258.80 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server stop mds service on oleg238-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing unload_modules_local oleg238-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg238-server: LNET ready to unload oleg238-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2