== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 08:53:11 (1761310391) Stopping clients: oleg127-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg127-client.virtnet /mnt/lustre2 (opts:-f) oleg127-server: oleg127-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg127-server' oleg127-server: oleg127-server.virtnet: executing load_modules_local oleg127-server: Loading modules from /home/green/git/lustre-release/lustre oleg127-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg127-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg127-server: detected 4 online CPUs by sysfs oleg127-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg127-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg127-server: oleg127-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg127-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg127-server: oleg127-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg127-client.virtnet: -o user_xattr,flock oleg127-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16166 358 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1569 (time 1761310454.54 total 10.00 last 156.87) - open/close 3171 (time 1761310464.54 total 20.00 last 160.18) - open/close 5344 (time 1761310474.55 total 30.01 last 217.23) - open/close 8056 (time 1761310484.55 total 40.01 last 271.06) - open/close 9769 (time 1761310494.55 total 50.01 last 171.25) - open/close 11344 (time 1761310504.55 total 60.01 last 157.49) - open/close 12821 (time 1761310514.56 total 70.02 last 147.61) - open/close 14279 (time 1761310524.56 total 80.02 last 145.78) - open/close 16292 (time 1761310534.56 total 90.02 last 201.28) - open/close 18973 (time 1761310544.56 total 100.02 last 268.07) - open/close 20000 (time 1761310549.56 total 105.02 last 205.62) - open/close 22409 (time 1761310559.56 total 115.02 last 240.83) - open/close 24537 (time 1761310569.56 total 125.02 last 212.77) - open/close 26708 (time 1761310579.56 total 135.02 last 217.06) - open/close 28420 (time 1761310589.56 total 145.03 last 171.19) - open/close 30000 (time 1761310599.40 total 154.86 last 160.66) - open/close 31997 (time 1761310609.40 total 164.86 last 199.64) - open/close 34181 (time 1761310619.40 total 174.86 last 218.39) - open/close 35783 (time 1761310629.40 total 184.87 last 160.17) - open/close 37476 (time 1761310639.41 total 194.87 last 169.23) - open/close 39444 (time 1761310649.41 total 204.87 last 196.74) - open/close 40000 (time 1761310652.52 total 207.99 last 178.62) - open/close 41960 (time 1761310662.53 total 217.99 last 195.97) - open/close 43712 (time 1761310672.53 total 227.99 last 175.11) - open/close 45996 (time 1761310682.53 total 237.99 last 228.33) - open/close 48502 (time 1761310692.53 total 248.00 last 250.59) - open/close 50000 (time 1761310699.00 total 254.46 last 231.59) - open/close 52357 (time 1761310709.01 total 264.47 last 235.58) - open/close 54067 (time 1761310719.01 total 274.47 last 170.92) - open/close 55868 (time 1761310729.02 total 284.48 last 180.02) - open/close 58561 (time 1761310739.02 total 294.48 last 269.24) - open/close 60000 (time 1761310744.60 total 300.06 last 257.90) - open/close 61810 (time 1761310754.60 total 310.06 last 180.93) - open/close 63421 (time 1761310764.60 total 320.07 last 161.05) total: 64768 open/close in 327.22 seconds: 197.94 ops/second - /unlink 3064 (time 1761310783.64 total 10.00 last 306.29) - /unlink 5909 (time 1761310793.65 total 20.01 last 284.45) - /unlink 7440 (time 1761310803.65 total 30.01 last 153.04) - /unlink 8667 (time 1761310813.67 total 40.03 last 122.44) - /unlink 9826 (time 1761310823.68 total 50.04 last 115.83) - /unlink 11086 (time 1761310833.69 total 60.04 last 125.90) - /unlink 12388 (time 1761310843.69 total 70.05 last 130.17) - /unlink 13720 (time 1761310853.69 total 80.05 last 133.12) - /unlink 15186 (time 1761310863.70 total 90.06 last 146.54) - /unlink 16659 (time 1761310873.70 total 100.06 last 147.26) - /unlink 18117 (time 1761310883.71 total 110.07 last 145.72) - /unlink 19841 (time 1761310893.71 total 120.07 last 172.39) - /unlink 22212 (time 1761310903.71 total 130.07 last 237.07) - /unlink 24336 (time 1761310913.71 total 140.07 last 212.37) - /unlink 26266 (time 1761310923.72 total 150.07 last 192.88) - /unlink 28254 (time 1761310933.72 total 160.08 last 198.75) - /unlink 30000 (time 1761310940.86 total 167.21 last 244.64) - /unlink 32206 (time 1761310950.86 total 177.22 last 220.57) - /unlink 34166 (time 1761310960.86 total 187.22 last 195.93) - /unlink 35968 (time 1761310970.86 total 197.22 last 180.12) - /unlink 37715 (time 1761310980.87 total 207.23 last 174.61) - /unlink 39249 (time 1761310990.87 total 217.23 last 153.33) - /unlink 40000 (time 1761310995.70 total 222.06 last 155.44) - /unlink 41671 (time 1761311005.71 total 232.07 last 166.96) - /unlink 43930 (time 1761311015.72 total 242.08 last 225.82) - /unlink 45954 (time 1761311025.72 total 252.08 last 202.33) - /unlink 50000 (time 1761311035.70 total 262.06 last 405.44) - /unlink 52746 (time 1761311045.70 total 272.06 last 274.55) - /unlink 55368 (time 1761311055.70 total 282.06 last 262.19) - /unlink 59286 (time 1761311065.70 total 292.06 last 391.77) - /unlink 64314 (time 1761311075.70 total 302.06 last 502.72) total: 64768 /unlink in 303.17 seconds: 213.64 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2452 (time 1761311106.55 total 10.00 last 245.15) - open/close 4617 (time 1761311116.55 total 20.00 last 216.49) - open/close 6956 (time 1761311126.55 total 30.00 last 233.85) - open/close 9968 (time 1761311136.56 total 40.01 last 301.15) - open/close 13010 (time 1761311146.56 total 50.01 last 304.09) - open/close 15207 (time 1761311156.56 total 60.01 last 219.69) - open/close 17477 (time 1761311166.56 total 70.01 last 226.94) - open/close 20000 (time 1761311176.36 total 79.81 last 257.51) - open/close 22492 (time 1761311186.37 total 89.82 last 249.08) - open/close 24950 (time 1761311196.37 total 99.82 last 245.77) - open/close 27761 (time 1761311206.37 total 109.82 last 280.97) - open/close 30000 (time 1761311215.28 total 118.73 last 251.21) - open/close 32420 (time 1761311225.28 total 128.73 last 242.00) - open/close 35237 (time 1761311235.29 total 138.74 last 281.61) - open/close 37788 (time 1761311245.29 total 148.74 last 255.08) - open/close 39904 (time 1761311255.30 total 158.75 last 211.41) - open/close 42202 (time 1761311265.30 total 168.75 last 229.74) - open/close 44999 (time 1761311275.30 total 178.75 last 279.66) - open/close 48265 (time 1761311285.30 total 188.75 last 326.57) - open/close 49266 (time 1761311295.31 total 198.75 last 100.07) - open/close 50000 (time 1761311304.44 total 207.89 last 80.35) - open/close 50875 (time 1761311314.44 total 217.89 last 87.49) - open/close 51750 (time 1761311324.45 total 227.90 last 87.42) - open/close 52577 (time 1761311334.46 total 237.91 last 82.64) - open/close 53362 (time 1761311344.46 total 247.91 last 78.47) - open/close 54153 (time 1761311354.46 total 257.91 last 79.09) - open/close 55004 (time 1761311364.46 total 267.91 last 85.08) - open/close 55833 (time 1761311374.47 total 277.92 last 82.87) - open/close 56755 (time 1761311384.48 total 287.93 last 92.12) - open/close 57644 (time 1761311394.48 total 297.93 last 88.84) - open/close 58260 (time 1761311404.50 total 307.95 last 61.48) - open/close 59061 (time 1761311414.51 total 317.96 last 80.02) - open/close 59784 (time 1761311424.53 total 327.98 last 72.21) - open/close 60000 (time 1761311427.72 total 331.17 last 67.61) - open/close 60740 (time 1761311437.73 total 341.18 last 73.97) - open/close 61508 (time 1761311447.73 total 351.18 last 76.74) - open/close 62290 (time 1761311457.74 total 361.19 last 78.12) - open/close 63111 (time 1761311467.78 total 371.23 last 81.79) - open/close 63908 (time 1761311477.79 total 381.24 last 79.64) - open/close 64730 (time 1761311487.79 total 391.24 last 82.18) total: 64768 open/close in 391.62 seconds: 165.38 ops/second - /unlink 1263 (time 1761311501.17 total 10.02 last 126.10) - /unlink 2552 (time 1761311511.17 total 20.02 last 128.87) - /unlink 3814 (time 1761311521.17 total 30.02 last 126.17) - /unlink 5091 (time 1761311531.18 total 40.03 last 127.60) - /unlink 6476 (time 1761311541.19 total 50.03 last 138.42) - /unlink 7849 (time 1761311551.19 total 60.04 last 137.21) - /unlink 9174 (time 1761311561.20 total 70.04 last 132.44) - /unlink 10000 (time 1761311567.83 total 76.68 last 124.57) - /unlink 11282 (time 1761311577.83 total 86.68 last 128.13) - /unlink 12576 (time 1761311587.85 total 96.70 last 129.21) - /unlink 13843 (time 1761311597.85 total 106.70 last 126.64) - /unlink 15139 (time 1761311607.86 total 116.70 last 129.55) - /unlink 16425 (time 1761311617.86 total 126.71 last 128.56) - /unlink 17726 (time 1761311627.86 total 136.71 last 130.05) - /unlink 19134 (time 1761311637.87 total 146.71 last 140.77) - /unlink 20000 (time 1761311644.79 total 153.63 last 125.12) - /unlink 21266 (time 1761311654.79 total 163.64 last 126.52) - /unlink 22585 (time 1761311664.80 total 173.64 last 131.84) - /unlink 23994 (time 1761311674.80 total 183.65 last 140.86) - /unlink 25185 (time 1761311684.80 total 193.65 last 119.06) - /unlink 26464 (time 1761311694.81 total 203.66 last 127.79) - /unlink 27740 (time 1761311704.81 total 213.66 last 127.59) - /unlink 28913 (time 1761311714.82 total 223.67 last 117.17) - /unlink 30000 (time 1761311724.82 total 233.67 last 108.76) - /unlink 31135 (time 1761311734.83 total 243.67 last 113.42) - /unlink 32283 (time 1761311744.83 total 253.67 last 114.79) - /unlink 33431 (time 1761311754.83 total 263.68 last 114.71) - /unlink 34587 (time 1761311764.83 total 273.68 last 115.59) - /unlink 35791 (time 1761311774.84 total 283.68 last 120.37) - /unlink 36923 (time 1761311784.84 total 293.69 last 113.17) - /unlink 38197 (time 1761311794.84 total 303.69 last 127.39) - /unlink 39417 (time 1761311804.84 total 313.69 last 121.99) - /unlink 40000 (time 1761311809.68 total 318.53 last 120.50) - /unlink 41177 (time 1761311819.83 total 328.67 last 115.98) - /unlink 42331 (time 1761311829.83 total 338.68 last 115.33) - /unlink 43440 (time 1761311839.83 total 348.68 last 110.90) - /unlink 44611 (time 1761311849.83 total 358.68 last 117.10) - /unlink 45712 (time 1761311859.84 total 368.69 last 110.05) - /unlink 46902 (time 1761311869.84 total 378.69 last 119.00) - /unlink 48040 (time 1761311879.84 total 388.69 last 113.80) - /unlink 49213 (time 1761311889.84 total 398.69 last 117.25) - /unlink 50000 (time 1761311896.02 total 404.87 last 127.39) - /unlink 51192 (time 1761311906.02 total 414.87 last 119.15) - /unlink 52409 (time 1761311916.03 total 424.87 last 121.69) - /unlink 53597 (time 1761311926.03 total 434.87 last 118.78) - /unlink 54801 (time 1761311936.03 total 444.88 last 120.33) - /unlink 56051 (time 1761311946.04 total 454.89 last 124.94) - /unlink 57307 (time 1761311956.04 total 464.89 last 125.58) - /unlink 58562 (time 1761311966.05 total 474.90 last 125.35) - /unlink 59759 (time 1761311976.06 total 484.90 last 119.67) - /unlink 60974 (time 1761311986.06 total 494.90 last 121.48) - /unlink 62262 (time 1761311996.06 total 504.91 last 128.76) - /unlink 63498 (time 1761312006.07 total 514.91 last 123.53) - /unlink 64738 (time 1761312016.07 total 524.91 last 123.99) total: 64768 /unlink in 525.17 seconds: 123.33 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 805 (time 1761312050.24 total 10.00 last 80.48) - open/close 1552 (time 1761312060.25 total 20.01 last 74.62) - open/close 2357 (time 1761312070.26 total 30.02 last 80.41) - open/close 3051 (time 1761312080.26 total 40.03 last 69.39) - open/close 3785 (time 1761312090.27 total 50.03 last 73.33) - open/close 4505 (time 1761312100.28 total 60.04 last 71.95) - open/close 5232 (time 1761312110.31 total 70.08 last 72.43) - open/close 5952 (time 1761312120.32 total 80.08 last 71.97) - open/close 6677 (time 1761312130.33 total 90.10 last 72.41) - open/close 7424 (time 1761312140.33 total 100.10 last 74.69) - open/close 8163 (time 1761312150.33 total 110.10 last 73.89) - open/close 8855 (time 1761312160.34 total 120.10 last 69.18) - open/close 9570 (time 1761312170.34 total 130.11 last 71.46) - open/close 10000 (time 1761312176.48 total 136.24 last 70.06) - open/close 10722 (time 1761312186.49 total 146.25 last 72.15) - open/close 11423 (time 1761312196.49 total 156.25 last 70.08) - open/close 12158 (time 1761312206.50 total 166.26 last 73.44) - open/close 12948 (time 1761312216.51 total 176.27 last 78.91) - open/close 13626 (time 1761312226.52 total 186.28 last 67.76) - open/close 14428 (time 1761312236.53 total 196.30 last 80.07) - open/close 15125 (time 1761312246.54 total 206.30 last 69.68) - open/close 15901 (time 1761312256.54 total 216.30 last 77.59) - open/close 16608 (time 1761312266.54 total 226.30 last 70.68) - open/close 17292 (time 1761312276.55 total 236.32 last 68.32) - open/close 18036 (time 1761312286.57 total 246.33 last 74.29) - open/close 18807 (time 1761312296.58 total 256.34 last 77.04) - open/close 19554 (time 1761312306.59 total 266.36 last 74.56) - open/close 20000 (time 1761312312.01 total 271.77 last 82.41) - open/close 20795 (time 1761312322.01 total 281.77 last 79.50) - open/close 21581 (time 1761312332.01 total 291.78 last 78.56) - open/close 22406 (time 1761312342.02 total 301.78 last 82.47) - open/close 23245 (time 1761312352.02 total 311.78 last 83.88) - open/close 24040 (time 1761312362.02 total 321.79 last 79.46) - open/close 24834 (time 1761312372.03 total 331.79 last 79.35) - open/close 25574 (time 1761312382.05 total 341.81 last 73.89) - open/close 26368 (time 1761312392.05 total 351.81 last 79.37) - open/close 27125 (time 1761312402.05 total 361.81 last 75.69) - open/close 27879 (time 1761312412.06 total 371.82 last 75.33) - open/close 28571 (time 1761312422.07 total 381.83 last 69.14) - open/close 29293 (time 1761312432.08 total 391.84 last 72.14) - open/close 30000 (time 1761312441.57 total 401.34 last 74.47) - open/close 30790 (time 1761312451.58 total 411.35 last 78.91) - open/close 31558 (time 1761312461.58 total 421.35 last 76.79) - open/close 32351 (time 1761312471.59 total 431.35 last 79.25) - open/close 33141 (time 1761312481.61 total 441.37 last 78.85) - open/close 33923 (time 1761312491.61 total 451.38 last 78.17) - open/close 34706 (time 1761312501.62 total 461.39 last 78.24) - open/close 35426 (time 1761312511.62 total 471.39 last 71.99) - open/close 36232 (time 1761312521.63 total 481.39 last 80.57) - open/close 37006 (time 1761312531.64 total 491.40 last 77.31) - open/close 37790 (time 1761312541.64 total 501.41 last 78.35) - open/close 38671 (time 1761312551.65 total 511.42 last 88.04) - open/close 39535 (time 1761312561.66 total 521.42 last 86.34) - open/close 40000 (time 1761312567.43 total 527.20 last 80.52) - open/close 40776 (time 1761312577.43 total 537.20 last 77.60) - open/close 41711 (time 1761312587.44 total 547.20 last 93.46) - open/close 42593 (time 1761312597.45 total 557.21 last 88.12) - open/close 43522 (time 1761312607.45 total 567.21 last 92.86) - open/close 44433 (time 1761312617.46 total 577.22 last 91.04) - open/close 45290 (time 1761312627.46 total 587.22 last 85.68) - open/close 46145 (time 1761312637.47 total 597.23 last 85.45) - open/close 47108 (time 1761312647.47 total 607.23 last 96.26) - open/close 48002 (time 1761312657.48 total 617.24 last 89.34) - open/close 48807 (time 1761312667.48 total 627.25 last 80.46) - open/close 49631 (time 1761312677.48 total 637.25 last 82.38) - open/close 50000 (time 1761312682.33 total 642.09 last 76.15) - open/close 50894 (time 1761312692.34 total 652.10 last 89.36) - open/close 51782 (time 1761312702.34 total 662.11 last 88.72) - open/close 52668 (time 1761312712.36 total 672.12 last 88.49) - open/close 53584 (time 1761312722.36 total 682.13 last 91.55) - open/close 54468 (time 1761312732.37 total 692.14 last 88.32) - open/close 55323 (time 1761312742.37 total 702.14 last 85.48) - open/close 56305 (time 1761312752.38 total 712.14 last 98.15) - open/close 57104 (time 1761312762.38 total 722.15 last 79.85) - open/close 57907 (time 1761312772.39 total 732.15 last 80.27) - open/close 58782 (time 1761312782.40 total 742.16 last 87.43) - open/close 59638 (time 1761312792.40 total 752.17 last 85.56) - open/close 60000 (time 1761312796.32 total 756.08 last 92.46) - open/close 60848 (time 1761312806.33 total 766.10 last 84.68) - open/close 61676 (time 1761312816.34 total 776.11 last 82.69) - open/close 62467 (time 1761312826.35 total 786.12 last 79.04) - open/close 63300 (time 1761312836.36 total 796.12 last 83.24) - open/close 64090 (time 1761312846.36 total 806.13 last 78.97) total: 64768 open/close in 814.05 seconds: 79.56 ops/second - /unlink 1336 (time 1761312867.30 total 10.00 last 133.55) - /unlink 2665 (time 1761312877.31 total 20.01 last 132.82) - /unlink 3923 (time 1761312887.32 total 30.02 last 125.69) - /unlink 5373 (time 1761312897.32 total 40.02 last 144.99) - /unlink 6730 (time 1761312907.32 total 50.02 last 135.67) - /unlink 8072 (time 1761312917.32 total 60.02 last 134.18) - /unlink 9452 (time 1761312927.33 total 70.03 last 137.87) - /unlink 10000 (time 1761312931.32 total 74.02 last 137.44) - /unlink 11264 (time 1761312941.32 total 84.02 last 126.34) - /unlink 12628 (time 1761312951.33 total 94.03 last 136.30) - /unlink 14062 (time 1761312961.33 total 104.04 last 143.32) - /unlink 15732 (time 1761312971.34 total 114.04 last 166.92) - /unlink 17016 (time 1761312981.34 total 124.04 last 128.37) - /unlink 18340 (time 1761312991.34 total 134.04 last 132.39) - /unlink 19616 (time 1761313001.35 total 144.05 last 127.53) - /unlink 20000 (time 1761313004.45 total 147.16 last 123.59) - /unlink 21343 (time 1761313014.46 total 157.16 last 134.23) - /unlink 22800 (time 1761313024.47 total 167.17 last 145.59) - /unlink 24170 (time 1761313034.52 total 177.22 last 136.30) - /unlink 25510 (time 1761313044.52 total 187.22 last 133.98) - /unlink 26841 (time 1761313054.52 total 197.22 last 133.09) - /unlink 28223 (time 1761313064.52 total 207.22 last 138.19) - /unlink 29572 (time 1761313074.53 total 217.23 last 134.79) - /unlink 30000 (time 1761313077.05 total 219.76 last 169.45) - /unlink 31529 (time 1761313087.06 total 229.76 last 152.88) - /unlink 32968 (time 1761313097.06 total 239.76 last 143.86) - /unlink 34292 (time 1761313107.06 total 249.76 last 132.40) - /unlink 35634 (time 1761313117.06 total 259.76 last 134.19) - /unlink 36931 (time 1761313127.07 total 269.77 last 129.60) - /unlink 38280 (time 1761313137.07 total 279.78 last 134.79) - /unlink 39535 (time 1761313147.08 total 289.78 last 125.46) - /unlink 40000 (time 1761313150.99 total 293.69 last 118.83) - /unlink 41136 (time 1761313161.00 total 303.70 last 113.54) - /unlink 42494 (time 1761313171.00 total 313.71 last 135.70) - /unlink 43903 (time 1761313181.01 total 323.71 last 140.85) - /unlink 45266 (time 1761313191.01 total 333.71 last 136.25) - /unlink 46719 (time 1761313201.01 total 343.71 last 145.29) - /unlink 48235 (time 1761313211.01 total 353.72 last 151.59) - /unlink 49695 (time 1761313221.02 total 363.73 last 145.85) - /unlink 50000 (time 1761313223.13 total 365.83 last 144.99) - /unlink 51607 (time 1761313233.13 total 375.83 last 160.68) - /unlink 53056 (time 1761313243.13 total 385.83 last 144.86) - /unlink 54397 (time 1761313253.13 total 395.84 last 134.04) - /unlink 55756 (time 1761313263.14 total 405.84 last 135.86) - /unlink 57105 (time 1761313273.15 total 415.85 last 134.79) - /unlink 58545 (time 1761313283.15 total 425.85 last 143.93) - /unlink 60000 (time 1761313293.05 total 435.75 last 147.05) - /unlink 61484 (time 1761313303.05 total 445.75 last 148.35) - /unlink 62936 (time 1761313313.05 total 455.76 last 145.13) - /unlink 64332 (time 1761313323.06 total 465.76 last 139.52) total: 64768 /unlink in 469.04 seconds: 138.09 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 821 (time 1761313359.57 total 10.00 last 82.09) - open/close 1627 (time 1761313369.58 total 20.01 last 80.49) - open/close 2406 (time 1761313379.59 total 30.02 last 77.84) - open/close 3164 (time 1761313389.60 total 40.03 last 75.70) - open/close 3985 (time 1761313399.61 total 50.04 last 82.07) - open/close 4847 (time 1761313409.61 total 60.04 last 86.16) - open/close 5716 (time 1761313419.62 total 70.05 last 86.82) - open/close 6574 (time 1761313429.62 total 80.05 last 85.80) - open/close 7365 (time 1761313439.63 total 90.06 last 79.06) - open/close 8265 (time 1761313449.63 total 100.06 last 89.98) - open/close 9079 (time 1761313459.63 total 110.07 last 81.35) - open/close 9664 (time 1761313469.67 total 120.10 last 58.30) - open/close 10000 (time 1761313474.84 total 125.28 last 64.91) - open/close 10760 (time 1761313484.85 total 135.28 last 75.96) - open/close 11524 (time 1761313494.86 total 145.29 last 76.35) - open/close 12338 (time 1761313504.87 total 155.30 last 81.29) - open/close 13183 (time 1761313514.87 total 165.30 last 84.49) - open/close 14038 (time 1761313524.88 total 175.31 last 85.45) - open/close 14834 (time 1761313534.89 total 185.32 last 79.53) - open/close 15695 (time 1761313544.89 total 195.32 last 86.06) - open/close 16523 (time 1761313554.90 total 205.33 last 82.71) - open/close 17486 (time 1761313564.92 total 215.35 last 96.12) - open/close 18371 (time 1761313574.92 total 225.35 last 88.48) - open/close 19318 (time 1761313584.93 total 235.36 last 94.63) - open/close 20000 (time 1761313592.89 total 243.32 last 85.70) - open/close 20829 (time 1761313602.89 total 253.32 last 82.87) - open/close 21709 (time 1761313612.90 total 263.33 last 87.95) - open/close 22542 (time 1761313622.91 total 273.34 last 83.21) - open/close 23339 (time 1761313632.92 total 283.35 last 79.63) - open/close 24195 (time 1761313642.92 total 293.36 last 85.52) - open/close 25034 (time 1761313652.93 total 303.36 last 83.87) - open/close 25912 (time 1761313662.93 total 313.36 last 87.80) - open/close 26937 (time 1761313672.93 total 323.36 last 102.47) - open/close 27827 (time 1761313682.93 total 333.36 last 89.00) - open/close 28786 (time 1761313692.95 total 343.38 last 95.74) - open/close 29709 (time 1761313702.95 total 353.38 last 92.29) - open/close 30000 (time 1761313706.09 total 356.52 last 92.59) - open/close 31005 (time 1761313716.09 total 366.53 last 100.49) - open/close 31905 (time 1761313726.09 total 376.53 last 89.99) - open/close 32768 (time 1761313736.09 total 386.53 last 86.29) - open/close 33789 (time 1761313746.10 total 396.54 last 102.01) - open/close 34727 (time 1761313756.11 total 406.54 last 93.73) - open/close 35500 (time 1761313766.12 total 416.55 last 77.23) - open/close 36307 (time 1761313776.13 total 426.57 last 80.60) - open/close 37164 (time 1761313786.14 total 436.57 last 85.63) - open/close 38010 (time 1761313796.15 total 446.58 last 84.51) - open/close 38886 (time 1761313806.16 total 456.59 last 87.57) - open/close 39751 (time 1761313816.16 total 466.59 last 86.49) - open/close 40000 (time 1761313819.02 total 469.45 last 86.93) - open/close 40883 (time 1761313829.03 total 479.46 last 88.24) - open/close 41766 (time 1761313839.03 total 489.46 last 88.26) - open/close 42533 (time 1761313849.04 total 499.47 last 76.63) - open/close 43387 (time 1761313859.05 total 509.48 last 85.37) - open/close 44314 (time 1761313869.05 total 519.48 last 92.65) - open/close 45197 (time 1761313879.06 total 529.49 last 88.26) - open/close 46055 (time 1761313889.06 total 539.49 last 85.80) - open/close 46861 (time 1761313899.06 total 549.49 last 80.57) - open/close 47774 (time 1761313909.07 total 559.50 last 91.20) - open/close 48548 (time 1761313919.08 total 569.51 last 77.33) - open/close 49324 (time 1761313929.08 total 579.51 last 77.59) - open/close 50000 (time 1761313938.23 total 588.67 last 73.84) - open/close 50759 (time 1761313948.24 total 598.67 last 75.89) - open/close 51589 (time 1761313958.24 total 608.67 last 82.96) - open/close 52464 (time 1761313968.25 total 618.68 last 87.45) - open/close 53249 (time 1761313978.25 total 628.68 last 78.48) - open/close 54037 (time 1761313988.25 total 638.69 last 78.77) - open/close 54933 (time 1761313998.26 total 648.69 last 89.55) - open/close 55780 (time 1761314008.27 total 658.70 last 84.61) - open/close 56544 (time 1761314018.27 total 668.71 last 76.36) - open/close 57336 (time 1761314028.28 total 678.71 last 79.15) - open/close 58079 (time 1761314038.29 total 688.72 last 74.22) - open/close 58893 (time 1761314048.29 total 698.73 last 81.39) - open/close 59787 (time 1761314058.30 total 708.73 last 89.37) - open/close 60000 (time 1761314060.69 total 711.12 last 89.18) - open/close 60883 (time 1761314070.70 total 721.13 last 88.20) - open/close 61715 (time 1761314080.70 total 731.14 last 83.14) - open/close 62577 (time 1761314090.71 total 741.14 last 86.15) - open/close 63442 (time 1761314100.72 total 751.15 last 86.44) - open/close 64272 (time 1761314110.73 total 761.16 last 82.93) total: 64768 open/close in 768.67 seconds: 84.26 ops/second - /unlink 1393 (time 1761314132.42 total 10.00 last 139.30) - /unlink 2804 (time 1761314142.42 total 20.00 last 141.10) - /unlink 4285 (time 1761314152.42 total 30.00 last 148.04) - /unlink 5783 (time 1761314162.43 total 40.01 last 149.65) - /unlink 7303 (time 1761314172.43 total 50.02 last 151.92) - /unlink 8734 (time 1761314182.45 total 60.03 last 142.91) - /unlink 10000 (time 1761314192.02 total 69.60 last 132.27) - /unlink 11509 (time 1761314202.02 total 79.61 last 150.87) - /unlink 13019 (time 1761314212.02 total 89.61 last 150.95) - /unlink 14455 (time 1761314222.03 total 99.61 last 143.56) - /unlink 16098 (time 1761314232.03 total 109.61 last 164.26) - /unlink 17573 (time 1761314242.03 total 119.62 last 147.49) - /unlink 18935 (time 1761314252.03 total 129.62 last 136.16) - /unlink 20000 (time 1761314259.21 total 136.80 last 148.35) - /unlink 21721 (time 1761314269.22 total 146.80 last 172.00) - /unlink 23386 (time 1761314279.22 total 156.80 last 166.47) - /unlink 24830 (time 1761314289.22 total 166.81 last 144.34) - /unlink 26320 (time 1761314299.23 total 176.81 last 148.96) - /unlink 27927 (time 1761314309.24 total 186.83 last 160.48) - /unlink 29618 (time 1761314319.24 total 196.83 last 169.05) - /unlink 30000 (time 1761314321.38 total 198.96 last 179.01) - /unlink 31443 (time 1761314331.38 total 208.97 last 144.20) - /unlink 32739 (time 1761314341.39 total 218.97 last 129.55) - /unlink 34033 (time 1761314351.40 total 228.98 last 129.29) - /unlink 35454 (time 1761314361.40 total 238.98 last 142.06) - /unlink 37100 (time 1761314371.41 total 248.99 last 164.43) - /unlink 38641 (time 1761314381.42 total 259.00 last 153.97) - /unlink 40000 (time 1761314391.07 total 268.65 last 140.82) - /unlink 41441 (time 1761314401.07 total 278.66 last 144.03) - /unlink 43348 (time 1761314411.08 total 288.66 last 190.66) - /unlink 45260 (time 1761314421.08 total 298.66 last 191.20) - /unlink 47047 (time 1761314431.08 total 308.66 last 178.68) - /unlink 48178 (time 1761314441.11 total 318.70 last 112.73) - /unlink 49417 (time 1761314451.11 total 328.70 last 123.87) - /unlink 50000 (time 1761314455.66 total 333.24 last 128.34) - /unlink 51204 (time 1761314465.66 total 343.24 last 120.37) - /unlink 52521 (time 1761314475.66 total 353.24 last 131.68) - /unlink 53828 (time 1761314485.66 total 363.25 last 130.65) - /unlink 55169 (time 1761314495.67 total 373.25 last 134.05) - /unlink 56469 (time 1761314505.68 total 383.26 last 129.86) - /unlink 57626 (time 1761314515.68 total 393.26 last 115.70) - /unlink 58929 (time 1761314525.68 total 403.26 last 130.28) - /unlink 60000 (time 1761314534.90 total 412.49 last 116.13) - /unlink 61257 (time 1761314544.90 total 422.49 last 125.66) - /unlink 62570 (time 1761314554.91 total 432.50 last 131.21) - /unlink 63971 (time 1761314564.92 total 442.51 last 139.95) total: 64768 /unlink in 448.72 seconds: 144.34 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 835 (time 1761314604.53 total 10.00 last 83.47) - open/close 1615 (time 1761314614.53 total 20.01 last 77.96) - open/close 2442 (time 1761314624.54 total 30.02 last 82.64) - open/close 3262 (time 1761314634.54 total 40.02 last 81.97) - open/close 4151 (time 1761314644.55 total 50.03 last 88.82) - open/close 4969 (time 1761314654.56 total 60.03 last 81.77) - open/close 5751 (time 1761314664.56 total 70.04 last 78.16) - open/close 6527 (time 1761314674.56 total 80.04 last 77.58) - open/close 7295 (time 1761314684.57 total 90.04 last 76.76) - open/close 8067 (time 1761314694.59 total 100.07 last 77.00) - open/close 8843 (time 1761314704.61 total 110.08 last 77.50) - open/close 9643 (time 1761314714.61 total 120.09 last 79.98) - open/close 10000 (time 1761314719.03 total 124.50 last 80.82) - open/close 10809 (time 1761314729.03 total 134.51 last 80.87) - open/close 11754 (time 1761314739.04 total 144.52 last 94.42) - open/close 12680 (time 1761314749.04 total 154.52 last 92.56) - open/close 13706 (time 1761314759.04 total 164.52 last 102.60) - open/close 14766 (time 1761314769.05 total 174.52 last 105.97) - open/close 15768 (time 1761314779.05 total 184.53 last 100.18) - open/close 16621 (time 1761314789.05 total 194.53 last 85.27) - open/close 17522 (time 1761314799.06 total 204.54 last 90.00) - open/close 18440 (time 1761314809.07 total 214.55 last 91.75) - open/close 19443 (time 1761314819.08 total 224.55 last 100.24) - open/close 20000 (time 1761314825.63 total 231.10 last 85.04) - open/close 20918 (time 1761314835.63 total 241.11 last 91.72) - open/close 21841 (time 1761314845.64 total 251.12 last 92.25) - open/close 22813 (time 1761314855.64 total 261.12 last 97.19) - open/close 23894 (time 1761314865.64 total 271.12 last 108.06) - open/close 24889 (time 1761314875.65 total 281.12 last 99.48) - open/close 25801 (time 1761314885.65 total 291.13 last 91.18) - open/close 26749 (time 1761314895.65 total 301.13 last 94.79) - open/close 27839 (time 1761314905.66 total 311.14 last 108.89) - open/close 28834 (time 1761314915.66 total 321.14 last 99.49) - open/close 29714 (time 1761314925.67 total 331.14 last 87.96) - open/close 30000 (time 1761314928.64 total 334.12 last 96.01) - open/close 30912 (time 1761314938.65 total 344.13 last 91.12) - open/close 32070 (time 1761314948.66 total 354.13 last 115.77) - open/close 33076 (time 1761314958.66 total 364.14 last 100.51) - open/close 33948 (time 1761314968.67 total 374.14 last 87.19) - open/close 34823 (time 1761314978.67 total 384.14 last 87.49) - open/close 35722 (time 1761314988.67 total 394.15 last 89.85) - open/close 36625 (time 1761314998.67 total 404.15 last 90.28) - open/close 37628 (time 1761315008.68 total 414.16 last 100.23) - open/close 38472 (time 1761315018.70 total 424.18 last 84.23) - open/close 39300 (time 1761315028.71 total 434.19 last 82.70) - open/close 40000 (time 1761315036.91 total 442.38 last 85.44) - open/close 40859 (time 1761315046.91 total 452.39 last 85.84) - open/close 41905 (time 1761315056.92 total 462.39 last 104.58) - open/close 42899 (time 1761315066.92 total 472.39 last 99.40) - open/close 43763 (time 1761315076.92 total 482.40 last 86.34) - open/close 44647 (time 1761315086.92 total 492.40 last 88.39) - open/close 45543 (time 1761315096.94 total 502.41 last 89.48) - open/close 46442 (time 1761315106.94 total 512.42 last 89.85) - open/close 47317 (time 1761315116.94 total 522.42 last 87.50) - open/close 48119 (time 1761315126.95 total 532.42 last 80.16) - open/close 48979 (time 1761315136.95 total 542.42 last 85.99) - open/close 49879 (time 1761315146.95 total 552.43 last 89.96) - open/close 50761 (time 1761315156.96 total 562.43 last 88.16) - open/close 51716 (time 1761315166.96 total 572.44 last 95.46) - open/close 52607 (time 1761315176.97 total 582.45 last 89.00) - open/close 53505 (time 1761315186.98 total 592.46 last 89.74) - open/close 54293 (time 1761315196.99 total 602.47 last 78.72) - open/close 55130 (time 1761315206.99 total 612.47 last 83.68) - open/close 56028 (time 1761315217.00 total 622.48 last 89.71) - open/close 56920 (time 1761315227.02 total 632.49 last 89.05) - open/close 57748 (time 1761315237.02 total 642.49 last 82.80) - open/close 58722 (time 1761315247.02 total 652.50 last 97.35) - open/close 59580 (time 1761315257.03 total 662.50 last 85.75) - open/close 60000 (time 1761315260.87 total 666.35 last 109.25) - open/close 60911 (time 1761315270.88 total 676.35 last 91.06) - open/close 61822 (time 1761315280.89 total 686.36 last 91.03) - open/close 62740 (time 1761315290.89 total 696.36 last 91.77) - open/close 63600 (time 1761315300.89 total 706.36 last 86.00) - open/close 64422 (time 1761315310.89 total 716.37 last 82.18) total: 64768 open/close in 720.07 seconds: 89.95 ops/second - /unlink 1497 (time 1761315327.46 total 10.00 last 149.69) - /unlink 3146 (time 1761315337.46 total 20.00 last 164.88) - /unlink 4957 (time 1761315347.47 total 30.00 last 181.08) - /unlink 7326 (time 1761315357.47 total 40.01 last 236.86) - /unlink 9446 (time 1761315367.47 total 50.01 last 211.94) - /unlink 10000 (time 1761315370.59 total 53.13 last 177.38) - /unlink 11629 (time 1761315380.60 total 63.13 last 162.86) - /unlink 13373 (time 1761315390.60 total 73.14 last 174.36) - /unlink 14894 (time 1761315400.60 total 83.14 last 152.00) - /unlink 16471 (time 1761315410.61 total 93.15 last 157.65) - /unlink 18113 (time 1761315420.62 total 103.15 last 164.07) - /unlink 19395 (time 1761315430.62 total 113.16 last 128.15) - /unlink 20000 (time 1761315434.44 total 116.98 last 158.49) - /unlink 21456 (time 1761315444.44 total 126.98 last 145.54) - /unlink 22745 (time 1761315454.44 total 136.98 last 128.86) - /unlink 24475 (time 1761315464.45 total 146.98 last 172.97) - /unlink 25865 (time 1761315474.45 total 156.99 last 138.97) - /unlink 27281 (time 1761315484.47 total 167.01 last 141.24) - /unlink 28762 (time 1761315494.47 total 177.01 last 148.10) - /unlink 30000 (time 1761315503.52 total 186.05 last 136.92) - /unlink 31444 (time 1761315513.52 total 196.06 last 144.37) - /unlink 32863 (time 1761315523.52 total 206.06 last 141.85) - /unlink 34199 (time 1761315533.53 total 216.06 last 133.53) - /unlink 35609 (time 1761315543.53 total 226.07 last 140.98) - /unlink 36998 (time 1761315553.53 total 236.07 last 138.83) - /unlink 38308 (time 1761315563.54 total 246.07 last 130.95) - /unlink 39574 (time 1761315573.54 total 256.08 last 126.57) - /unlink 40000 (time 1761315576.40 total 258.94 last 148.68) - /unlink 41403 (time 1761315586.40 total 268.94 last 140.28) - /unlink 42880 (time 1761315596.41 total 278.95 last 147.65) - /unlink 44296 (time 1761315606.41 total 288.95 last 141.51) - /unlink 45557 (time 1761315616.41 total 298.95 last 126.10) - /unlink 46945 (time 1761315626.42 total 308.96 last 138.74) - /unlink 48343 (time 1761315636.42 total 318.96 last 139.73) - /unlink 49708 (time 1761315646.43 total 328.97 last 136.37) - /unlink 50000 (time 1761315648.50 total 331.03 last 141.56) - /unlink 51372 (time 1761315658.50 total 341.04 last 137.14) - /unlink 52853 (time 1761315668.51 total 351.04 last 148.03) - /unlink 54273 (time 1761315678.51 total 361.05 last 141.96) - /unlink 55626 (time 1761315688.51 total 371.05 last 135.23) - /unlink 56964 (time 1761315698.52 total 381.06 last 133.70) - /unlink 58547 (time 1761315708.53 total 391.06 last 158.24) - /unlink 60000 (time 1761315717.87 total 400.41 last 155.51) - /unlink 61767 (time 1761315727.87 total 410.41 last 176.59) - /unlink 63188 (time 1761315737.88 total 420.41 last 142.09) - /unlink 64601 (time 1761315747.88 total 430.42 last 141.26) total: 64768 /unlink in 431.74 seconds: 150.02 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg127-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg127-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg127-server stop mds service on oleg127-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg127-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg127-server' oleg127-server: oleg127-server.virtnet: executing unload_modules_local oleg127-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg127-server: LNET ready to unload oleg127-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg127-client: oleg127-server: ssh exited with exit code 2 pdsh@oleg127-client: oleg127-client: ssh exited with exit code 2 pdsh@oleg127-client: oleg127-server: ssh exited with exit code 2 pdsh@oleg127-client: oleg127-client: ssh exited with exit code 2