== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 22:28:42 (1763350122) Stopping clients: oleg303-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg303-client.virtnet /mnt/lustre2 (opts:-f) oleg303-server: oleg303-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg303-server' oleg303-server: oleg303-server.virtnet: executing load_modules_local oleg303-server: Loading modules from /home/green/git/lustre-release/lustre oleg303-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg303-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg303-server: detected 4 online CPUs by sysfs oleg303-server: MODOPTS_LIBCFS= oleg303-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg303-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg303-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 745 (time 1763350218.24 total 10.00 last 74.48) - open/close 1507 (time 1763350228.25 total 20.01 last 76.17) - open/close 2288 (time 1763350238.25 total 30.01 last 78.08) - open/close 3038 (time 1763350248.26 total 40.02 last 74.88) - open/close 3748 (time 1763350258.27 total 50.03 last 70.97) - open/close 4538 (time 1763350268.28 total 60.04 last 78.91) - open/close 5279 (time 1763350278.30 total 70.06 last 73.98) - open/close 6091 (time 1763350288.30 total 80.06 last 81.14) - open/close 6828 (time 1763350298.31 total 90.07 last 73.65) - open/close 7571 (time 1763350308.31 total 100.07 last 74.28) - open/close 8344 (time 1763350318.33 total 110.09 last 77.20) - open/close 9062 (time 1763350328.33 total 120.09 last 71.79) - open/close 9796 (time 1763350338.33 total 130.09 last 73.38) - open/close 10000 (time 1763350341.07 total 132.83 last 74.49) - open/close 10728 (time 1763350351.08 total 142.84 last 72.75) - open/close 11419 (time 1763350361.08 total 152.84 last 69.05) - open/close 12191 (time 1763350371.09 total 162.85 last 77.15) - open/close 12918 (time 1763350381.10 total 172.86 last 72.65) - open/close 13600 (time 1763350391.10 total 182.86 last 68.15) - open/close 14179 (time 1763350401.11 total 192.87 last 57.88) - open/close 14750 (time 1763350411.11 total 202.87 last 57.07) - open/close 15249 (time 1763350421.15 total 212.91 last 49.70) - open/close 15825 (time 1763350431.17 total 222.93 last 57.49) - open/close 16509 (time 1763350441.18 total 232.94 last 68.34) - open/close 17221 (time 1763350451.19 total 242.95 last 71.11) - open/close 17930 (time 1763350461.21 total 252.96 last 70.81) - open/close 18628 (time 1763350471.21 total 262.97 last 69.75) - open/close 19334 (time 1763350481.22 total 272.98 last 70.56) - open/close 20000 (time 1763350490.28 total 282.04 last 73.48) - open/close 20742 (time 1763350500.29 total 292.05 last 74.13) - open/close 21441 (time 1763350510.30 total 302.06 last 69.84) - open/close 22158 (time 1763350520.31 total 312.06 last 71.66) - open/close 22871 (time 1763350530.31 total 322.07 last 71.25) - open/close 23581 (time 1763350540.32 total 332.08 last 70.91) - open/close 24280 (time 1763350550.34 total 342.10 last 69.81) - open/close 24960 (time 1763350560.34 total 352.10 last 67.97) - open/close 25673 (time 1763350570.35 total 362.11 last 71.25) - open/close 26343 (time 1763350580.37 total 372.13 last 66.87) - open/close 27024 (time 1763350590.37 total 382.13 last 68.06) - open/close 27717 (time 1763350600.38 total 392.14 last 69.26) - open/close 28416 (time 1763350610.38 total 402.14 last 69.90) - open/close 29143 (time 1763350620.39 total 412.15 last 72.65) - open/close 29787 (time 1763350630.40 total 422.15 last 64.35) - open/close 30000 (time 1763350633.85 total 425.61 last 61.69) - open/close 30600 (time 1763350643.85 total 435.61 last 59.99) - open/close 31253 (time 1763350653.86 total 445.62 last 65.25) - open/close 31966 (time 1763350663.86 total 455.62 last 71.25) - open/close 32612 (time 1763350673.88 total 465.64 last 64.49) - open/close 33271 (time 1763350683.89 total 475.65 last 65.83) - open/close 33929 (time 1763350693.89 total 485.65 last 65.79) - open/close 34491 (time 1763350703.91 total 495.67 last 56.13) - open/close 35019 (time 1763350713.91 total 505.67 last 52.79) - open/close 35707 (time 1763350723.92 total 515.68 last 68.70) - open/close 36385 (time 1763350733.93 total 525.69 last 67.75) - open/close 37095 (time 1763350743.94 total 535.70 last 70.92) - open/close 37786 (time 1763350753.94 total 545.70 last 69.09) - open/close 38461 (time 1763350763.94 total 555.70 last 67.50) - open/close 39189 (time 1763350773.95 total 565.71 last 72.78) - open/close 39844 (time 1763350783.95 total 575.71 last 65.47) - open/close 40000 (time 1763350786.02 total 577.78 last 75.39) - open/close 40635 (time 1763350796.04 total 587.80 last 63.37) - open/close 41291 (time 1763350806.04 total 597.80 last 65.60) - open/close 41975 (time 1763350816.05 total 607.80 last 68.38) - open/close 42660 (time 1763350826.06 total 617.82 last 68.41) - open/close 43345 (time 1763350836.06 total 627.82 last 68.46) - open/close 44069 (time 1763350846.07 total 637.83 last 72.35) - open/close 44703 (time 1763350856.08 total 647.84 last 63.34) - open/close 45348 (time 1763350866.09 total 657.85 last 64.47) - open/close 45988 (time 1763350876.10 total 667.86 last 63.92) - open/close 46667 (time 1763350886.11 total 677.87 last 67.81) - open/close 47370 (time 1763350896.13 total 687.89 last 70.14) - open/close 48128 (time 1763350906.14 total 697.90 last 75.77) - open/close 48840 (time 1763350916.14 total 707.90 last 71.20) - open/close 49542 (time 1763350926.15 total 717.91 last 70.13) - open/close 50000 (time 1763350932.33 total 724.09 last 74.04) - open/close 50749 (time 1763350942.34 total 734.10 last 74.87) - open/close 51488 (time 1763350952.36 total 744.12 last 73.77) - open/close 52232 (time 1763350962.37 total 754.12 last 74.34) - open/close 52992 (time 1763350972.38 total 764.14 last 75.85) - open/close 53686 (time 1763350982.39 total 774.14 last 69.39) - open/close 54406 (time 1763350992.39 total 784.15 last 71.94) - open/close 55109 (time 1763351002.39 total 794.15 last 70.29) - open/close 55838 (time 1763351012.40 total 804.16 last 72.85) - open/close 56591 (time 1763351022.43 total 814.19 last 75.07) - open/close 57308 (time 1763351032.44 total 824.20 last 71.66) - open/close 58024 (time 1763351042.45 total 834.21 last 71.49) - open/close 58752 (time 1763351052.46 total 844.22 last 72.76) - open/close 59470 (time 1763351062.47 total 854.23 last 71.73) - open/close 60000 (time 1763351069.41 total 861.17 last 76.33) - open/close 60686 (time 1763351079.41 total 871.17 last 68.60) - open/close 61470 (time 1763351089.42 total 881.18 last 78.34) - open/close 62194 (time 1763351099.42 total 891.18 last 72.38) - open/close 62876 (time 1763351109.43 total 901.19 last 68.17) - open/close 63603 (time 1763351119.43 total 911.19 last 72.68) - open/close 64341 (time 1763351129.44 total 921.20 last 73.71) total: 64768 open/close in 927.30 seconds: 69.85 ops/second - /unlink 719 (time 1763351149.34 total 10.01 last 71.82) - /unlink 1400 (time 1763351159.34 total 20.01 last 68.08) - /unlink 2014 (time 1763351169.35 total 30.02 last 61.36) - /unlink 2679 (time 1763351179.36 total 40.03 last 66.42) - /unlink 3376 (time 1763351189.38 total 50.05 last 69.58) - /unlink 4041 (time 1763351199.38 total 60.06 last 66.46) - /unlink 4765 (time 1763351209.39 total 70.07 last 72.33) - /unlink 5456 (time 1763351219.40 total 80.08 last 69.01) - /unlink 6155 (time 1763351229.41 total 90.09 last 69.83) - /unlink 6848 (time 1763351239.42 total 100.09 last 69.25) - /unlink 7514 (time 1763351249.44 total 110.11 last 66.47) - /unlink 8200 (time 1763351259.44 total 120.12 last 68.58) - /unlink 8881 (time 1763351269.45 total 130.12 last 68.08) - /unlink 9544 (time 1763351279.46 total 140.13 last 66.22) - /unlink 10000 (time 1763351286.74 total 147.41 last 62.64) - /unlink 10635 (time 1763351296.75 total 157.42 last 63.44) - /unlink 11295 (time 1763351306.75 total 167.43 last 65.96) - /unlink 11987 (time 1763351316.76 total 177.43 last 69.19) - /unlink 12669 (time 1763351326.79 total 187.46 last 67.96) - /unlink 13342 (time 1763351336.80 total 197.47 last 67.25) - /unlink 14004 (time 1763351346.80 total 207.48 last 66.16) - /unlink 14712 (time 1763351356.80 total 217.48 last 70.79) - /unlink 15408 (time 1763351366.83 total 227.51 last 69.40) - /unlink 16028 (time 1763351376.84 total 237.51 last 61.97) - /unlink 16688 (time 1763351386.85 total 247.53 last 65.91) - /unlink 17319 (time 1763351396.86 total 257.53 last 63.07) - /unlink 18009 (time 1763351406.87 total 267.54 last 68.92) - /unlink 18755 (time 1763351416.88 total 277.55 last 74.54) - /unlink 19460 (time 1763351426.89 total 287.57 last 70.40) - /unlink 20000 (time 1763351434.55 total 295.22 last 70.52) - /unlink 20679 (time 1763351444.55 total 305.22 last 67.89) - /unlink 21350 (time 1763351454.56 total 315.24 last 67.02) - /unlink 21991 (time 1763351464.57 total 325.25 last 64.02) - /unlink 22683 (time 1763351474.58 total 335.26 last 69.13) - /unlink 23385 (time 1763351484.59 total 345.27 last 70.15) - /unlink 24104 (time 1763351494.60 total 355.28 last 71.81) - /unlink 24769 (time 1763351504.62 total 365.29 last 66.42) - /unlink 25452 (time 1763351514.62 total 375.29 last 68.28) - /unlink 26196 (time 1763351524.63 total 385.31 last 74.31) - /unlink 26900 (time 1763351534.64 total 395.31 last 70.37) - /unlink 27572 (time 1763351544.64 total 405.31 last 67.19) - /unlink 28286 (time 1763351554.64 total 415.32 last 71.35) - /unlink 28974 (time 1763351564.65 total 425.33 last 68.75) - /unlink 29670 (time 1763351574.65 total 435.33 last 69.59) - /unlink 30000 (time 1763351579.56 total 440.24 last 67.21) - /unlink 30630 (time 1763351589.57 total 450.24 last 62.96) - /unlink 31333 (time 1763351599.58 total 460.25 last 70.27) - /unlink 32008 (time 1763351609.59 total 470.27 last 67.37) - /unlink 32668 (time 1763351619.60 total 480.28 last 65.94) - /unlink 33355 (time 1763351629.61 total 490.29 last 68.62) - /unlink 34073 (time 1763351639.62 total 500.30 last 71.74) - /unlink 34750 (time 1763351649.63 total 510.30 last 67.65) - /unlink 35433 (time 1763351659.64 total 520.31 last 68.25) - /unlink 36193 (time 1763351669.64 total 530.31 last 75.98) - /unlink 36886 (time 1763351679.66 total 540.33 last 69.17) - /unlink 37612 (time 1763351689.67 total 550.34 last 72.55) - /unlink 38353 (time 1763351699.69 total 560.37 last 73.90) - /unlink 39077 (time 1763351709.70 total 570.37 last 72.34) - /unlink 39800 (time 1763351719.71 total 580.38 last 72.26) - /unlink 40000 (time 1763351722.60 total 583.28 last 69.06) - /unlink 40763 (time 1763351732.61 total 593.28 last 76.27) - /unlink 41512 (time 1763351742.61 total 603.29 last 74.84) - /unlink 42250 (time 1763351752.62 total 613.30 last 73.72) - /unlink 42954 (time 1763351762.63 total 623.30 last 70.37) - /unlink 43713 (time 1763351772.63 total 633.30 last 75.89) - /unlink 44450 (time 1763351782.63 total 643.31 last 73.69) - /unlink 45257 (time 1763351792.64 total 653.31 last 80.65) - /unlink 46088 (time 1763351802.64 total 663.31 last 83.08) - /unlink 47017 (time 1763351812.65 total 673.32 last 92.85) - /unlink 47774 (time 1763351822.65 total 683.33 last 75.65) - /unlink 48508 (time 1763351832.66 total 693.34 last 73.33) - /unlink 49301 (time 1763351842.67 total 703.34 last 79.24) - /unlink 50000 (time 1763351851.95 total 712.63 last 75.30) - /unlink 50710 (time 1763351861.96 total 722.63 last 70.98) - /unlink 51446 (time 1763351871.96 total 732.63 last 73.58) - /unlink 52220 (time 1763351881.97 total 742.65 last 77.31) - /unlink 53061 (time 1763351891.97 total 752.65 last 84.09) - /unlink 53831 (time 1763351901.98 total 762.65 last 76.96) - /unlink 54593 (time 1763351911.99 total 772.66 last 76.13) - /unlink 55495 (time 1763351921.99 total 782.66 last 90.19) - /unlink 56440 (time 1763351932.00 total 792.67 last 94.39) - /unlink 57289 (time 1763351942.01 total 802.68 last 84.81) - /unlink 58112 (time 1763351952.02 total 812.69 last 82.24) - /unlink 58932 (time 1763351962.02 total 822.70 last 81.94) - /unlink 59799 (time 1763351972.03 total 832.70 last 86.68) - /unlink 60000 (time 1763351974.37 total 835.05 last 85.71) - /unlink 60811 (time 1763351984.39 total 845.06 last 80.97) - /unlink 61589 (time 1763351994.39 total 855.07 last 77.75) - /unlink 62365 (time 1763352004.40 total 865.08 last 77.53) - /unlink 63095 (time 1763352014.41 total 875.08 last 72.98) - /unlink 63936 (time 1763352024.41 total 885.09 last 84.04) - /unlink 64754 (time 1763352034.42 total 895.10 last 81.73) total: 64768 /unlink in 895.28 seconds: 72.34 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 911 (time 1763352066.01 total 10.01 last 90.99) - open/close 1832 (time 1763352076.02 total 20.02 last 92.02) - open/close 2847 (time 1763352086.02 total 30.02 last 101.47) - open/close 3860 (time 1763352096.03 total 40.03 last 101.25) - open/close 4668 (time 1763352106.03 total 50.04 last 80.74) - open/close 5539 (time 1763352116.04 total 60.04 last 87.07) - open/close 6412 (time 1763352126.05 total 70.05 last 87.20) - open/close 7258 (time 1763352136.05 total 80.05 last 84.58) - open/close 8126 (time 1763352146.06 total 90.06 last 86.73) - open/close 9007 (time 1763352156.07 total 100.07 last 88.03) - open/close 10000 (time 1763352165.28 total 109.28 last 107.83) - open/close 10888 (time 1763352175.28 total 119.29 last 88.73) - open/close 11785 (time 1763352185.29 total 129.29 last 89.68) - open/close 12747 (time 1763352195.29 total 139.29 last 96.14) - open/close 13677 (time 1763352205.30 total 149.30 last 92.96) - open/close 14507 (time 1763352215.31 total 159.31 last 82.86) - open/close 15374 (time 1763352225.32 total 169.32 last 86.64) - open/close 16214 (time 1763352235.32 total 179.32 last 84.00) - open/close 17045 (time 1763352245.32 total 189.32 last 83.08) - open/close 17964 (time 1763352255.33 total 199.33 last 91.86) - open/close 18839 (time 1763352265.33 total 209.33 last 87.47) - open/close 19697 (time 1763352275.34 total 219.34 last 85.70) - open/close 20000 (time 1763352279.17 total 223.17 last 79.23) - open/close 20778 (time 1763352289.17 total 233.17 last 77.75) - open/close 21705 (time 1763352299.18 total 243.18 last 92.67) - open/close 22649 (time 1763352309.18 total 253.18 last 94.34) - open/close 23560 (time 1763352319.19 total 263.19 last 91.05) - open/close 24544 (time 1763352329.19 total 273.19 last 98.38) - open/close 25479 (time 1763352339.19 total 283.20 last 93.48) - open/close 26332 (time 1763352349.21 total 293.21 last 85.20) - open/close 27189 (time 1763352359.21 total 303.22 last 85.62) - open/close 28101 (time 1763352369.22 total 313.22 last 91.14) - open/close 28997 (time 1763352379.24 total 323.24 last 89.45) - open/close 29839 (time 1763352389.24 total 333.25 last 84.16) - open/close 30874 (time 1763352399.25 total 343.25 last 103.41) - open/close 31801 (time 1763352409.26 total 353.26 last 92.67) - open/close 32751 (time 1763352419.26 total 363.26 last 94.95) - open/close 33604 (time 1763352429.26 total 373.26 last 85.30) - open/close 34451 (time 1763352439.26 total 383.26 last 84.69) - open/close 35258 (time 1763352449.26 total 393.27 last 80.69) - open/close 36085 (time 1763352459.27 total 403.27 last 82.66) - open/close 36922 (time 1763352469.27 total 413.28 last 83.65) - open/close 37749 (time 1763352479.28 total 423.28 last 82.68) - open/close 38677 (time 1763352489.28 total 433.28 last 92.79) - open/close 39518 (time 1763352499.28 total 443.28 last 84.07) - open/close 40000 (time 1763352505.18 total 449.18 last 81.72) - open/close 40813 (time 1763352515.19 total 459.19 last 81.25) - open/close 41645 (time 1763352525.20 total 469.20 last 83.13) - open/close 42486 (time 1763352535.20 total 479.20 last 84.09) - open/close 43337 (time 1763352545.20 total 489.20 last 85.06) - open/close 44190 (time 1763352555.21 total 499.21 last 85.27) - open/close 44965 (time 1763352565.22 total 509.22 last 77.42) - open/close 45805 (time 1763352575.22 total 519.22 last 83.95) - open/close 46692 (time 1763352585.23 total 529.23 last 88.65) - open/close 47647 (time 1763352595.24 total 539.24 last 95.39) - open/close 48509 (time 1763352605.24 total 549.24 last 86.18) - open/close 49404 (time 1763352615.25 total 559.25 last 89.46) - open/close 50000 (time 1763352621.79 total 565.79 last 91.06) - open/close 50888 (time 1763352631.80 total 575.80 last 88.76) - open/close 51888 (time 1763352641.80 total 585.80 last 99.96) - open/close 52812 (time 1763352651.82 total 595.82 last 92.24) - open/close 53579 (time 1763352661.82 total 605.82 last 76.66) - open/close 54353 (time 1763352671.83 total 615.83 last 77.34) - open/close 55184 (time 1763352681.84 total 625.84 last 83.00) - open/close 56056 (time 1763352691.84 total 635.85 last 87.19) - open/close 56979 (time 1763352701.85 total 645.85 last 92.25) - open/close 57950 (time 1763352711.86 total 655.86 last 97.06) - open/close 59188 (time 1763352721.86 total 665.86 last 123.79) - open/close 60000 (time 1763352731.80 total 675.80 last 81.64) - open/close 60808 (time 1763352741.80 total 685.80 last 80.79) - open/close 61798 (time 1763352751.81 total 695.81 last 98.98) - open/close 62655 (time 1763352761.82 total 705.82 last 85.62) - open/close 63478 (time 1763352771.82 total 715.82 last 82.28) - open/close 64404 (time 1763352781.83 total 725.83 last 92.51) total: 64768 open/close in 729.61 seconds: 88.77 ops/second - /unlink 987 (time 1763352798.03 total 10.01 last 98.64) - /unlink 2038 (time 1763352808.04 total 20.01 last 105.06) - /unlink 3037 (time 1763352818.04 total 30.01 last 99.85) - /unlink 4126 (time 1763352828.05 total 40.02 last 108.82) - /unlink 5322 (time 1763352838.05 total 50.03 last 119.56) - /unlink 6304 (time 1763352848.06 total 60.03 last 98.16) - /unlink 7323 (time 1763352858.06 total 70.03 last 101.86) - /unlink 8338 (time 1763352868.07 total 80.04 last 101.43) - /unlink 9472 (time 1763352878.07 total 90.04 last 113.40) - /unlink 10000 (time 1763352883.19 total 95.17 last 102.97) - /unlink 11068 (time 1763352893.20 total 105.17 last 106.73) - /unlink 12333 (time 1763352903.20 total 115.18 last 126.46) - /unlink 13600 (time 1763352913.20 total 125.18 last 126.68) - /unlink 15180 (time 1763352923.21 total 135.18 last 157.95) - /unlink 16606 (time 1763352933.21 total 145.19 last 142.55) - /unlink 17863 (time 1763352943.22 total 155.19 last 125.64) - /unlink 19029 (time 1763352953.22 total 165.20 last 116.52) - /unlink 20000 (time 1763352960.88 total 172.85 last 126.90) - /unlink 21588 (time 1763352970.88 total 182.85 last 158.75) - /unlink 23022 (time 1763352980.89 total 192.86 last 143.30) - /unlink 24497 (time 1763352990.89 total 202.86 last 147.43) - /unlink 25802 (time 1763353000.89 total 212.87 last 130.45) - /unlink 27168 (time 1763353010.90 total 222.87 last 136.55) - /unlink 28361 (time 1763353020.91 total 232.88 last 119.16) - /unlink 29570 (time 1763353030.92 total 242.89 last 120.78) - /unlink 30000 (time 1763353034.01 total 245.99 last 138.99) - /unlink 31866 (time 1763353044.01 total 255.99 last 186.58) - /unlink 33289 (time 1763353054.01 total 265.99 last 142.30) - /unlink 35143 (time 1763353064.02 total 275.99 last 185.34) - /unlink 36820 (time 1763353074.02 total 285.99 last 167.65) - /unlink 38769 (time 1763353084.02 total 296.00 last 194.83) - /unlink 40000 (time 1763353088.17 total 300.14 last 296.92) - /unlink 43191 (time 1763353098.17 total 310.15 last 319.01) - /unlink 46275 (time 1763353108.18 total 320.15 last 308.32) - /unlink 49631 (time 1763353118.18 total 330.15 last 335.54) - /unlink 53218 (time 1763353128.18 total 340.15 last 358.60) - /unlink 56368 (time 1763353138.18 total 350.15 last 314.98) - /unlink 59016 (time 1763353148.18 total 360.16 last 264.73) - /unlink 60000 (time 1763353151.47 total 363.44 last 299.58) - /unlink 62799 (time 1763353161.47 total 373.45 last 279.81) total: 64768 /unlink in 379.72 seconds: 170.57 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1915 (time 1763353201.97 total 10.00 last 191.49) - open/close 4006 (time 1763353211.97 total 20.00 last 209.04) - open/close 5656 (time 1763353221.97 total 30.00 last 164.98) - open/close 7190 (time 1763353231.97 total 40.01 last 153.39) - open/close 8560 (time 1763353241.98 total 50.01 last 136.96) - open/close 9862 (time 1763353251.98 total 60.01 last 130.20) - open/close 11546 (time 1763353261.98 total 70.01 last 168.39) - open/close 13412 (time 1763353271.98 total 80.01 last 186.55) - open/close 15091 (time 1763353281.99 total 90.02 last 167.82) - open/close 17048 (time 1763353291.99 total 100.02 last 195.62) - open/close 19028 (time 1763353301.99 total 110.02 last 197.95) - open/close 20000 (time 1763353307.07 total 115.10 last 191.49) - open/close 22283 (time 1763353317.07 total 125.10 last 228.25) - open/close 24526 (time 1763353327.07 total 135.10 last 224.29) - open/close 26600 (time 1763353337.08 total 145.11 last 207.29) - open/close 28646 (time 1763353347.08 total 155.11 last 204.50) - open/close 30000 (time 1763353353.64 total 161.67 last 206.39) - open/close 31984 (time 1763353363.64 total 171.68 last 198.33) - open/close 33843 (time 1763353373.65 total 181.68 last 185.83) - open/close 35435 (time 1763353383.65 total 191.68 last 159.20) - open/close 37115 (time 1763353393.65 total 201.68 last 167.93) - open/close 38815 (time 1763353403.66 total 211.69 last 169.96) - open/close 40000 (time 1763353409.45 total 217.48 last 204.45) - open/close 42204 (time 1763353419.45 total 227.49 last 220.33) - open/close 44363 (time 1763353429.46 total 237.49 last 215.74) - open/close 45800 (time 1763353439.47 total 247.50 last 143.64) - open/close 47550 (time 1763353449.47 total 257.50 last 174.89) - open/close 49028 (time 1763353459.47 total 267.51 last 147.77) - open/close 50000 (time 1763353465.68 total 273.71 last 156.55) - open/close 51726 (time 1763353475.69 total 283.72 last 172.56) - open/close 53679 (time 1763353485.69 total 293.72 last 195.29) - open/close 55826 (time 1763353495.69 total 303.72 last 214.61) - open/close 57906 (time 1763353505.69 total 313.72 last 207.94) - open/close 59932 (time 1763353515.69 total 323.73 last 202.58) - open/close 61292 (time 1763353525.70 total 333.73 last 135.94) - open/close 62648 (time 1763353535.71 total 343.74 last 135.46) - open/close 63731 (time 1763353545.71 total 353.75 last 108.25) total: 64768 open/close in 361.90 seconds: 178.97 ops/second - /unlink 3647 (time 1763353565.84 total 10.00 last 364.70) - /unlink 6963 (time 1763353575.84 total 20.00 last 331.55) - /unlink 10000 (time 1763353583.86 total 28.02 last 378.73) - /unlink 13821 (time 1763353593.86 total 38.02 last 382.08) - /unlink 19294 (time 1763353603.86 total 48.02 last 547.27) - /unlink 24122 (time 1763353613.86 total 58.02 last 482.74) - /unlink 29736 (time 1763353623.86 total 68.02 last 561.39) - /unlink 34519 (time 1763353633.86 total 78.03 last 478.11) - /unlink 38606 (time 1763353643.86 total 88.03 last 408.66) - /unlink 40000 (time 1763353646.87 total 91.03 last 463.80) - /unlink 44207 (time 1763353656.87 total 101.03 last 420.65) - /unlink 48339 (time 1763353666.87 total 111.04 last 413.15) - /unlink 50000 (time 1763353671.44 total 115.61 last 363.33) - /unlink 55206 (time 1763353681.44 total 125.61 last 520.53) - /unlink 59649 (time 1763353691.44 total 135.61 last 444.27) - /unlink 64284 (time 1763353701.44 total 145.61 last 463.48) total: 64768 /unlink in 146.83 seconds: 441.12 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2810 (time 1763353731.01 total 10.00 last 280.89) - open/close 6018 (time 1763353741.01 total 20.01 last 320.71) - open/close 9552 (time 1763353751.01 total 30.01 last 353.33) - open/close 12913 (time 1763353761.01 total 40.01 last 336.06) - open/close 16139 (time 1763353771.01 total 50.01 last 322.58) - open/close 19468 (time 1763353781.01 total 60.01 last 332.90) - open/close 23784 (time 1763353791.01 total 70.01 last 431.56) - open/close 27755 (time 1763353801.02 total 80.01 last 397.00) - open/close 30000 (time 1763353808.76 total 87.76 last 289.83) - open/close 34503 (time 1763353818.76 total 97.76 last 450.24) - open/close 39230 (time 1763353828.76 total 107.76 last 472.69) - open/close 43694 (time 1763353838.76 total 117.76 last 446.31) - open/close 47432 (time 1763353848.77 total 127.76 last 373.75) - open/close 50000 (time 1763353855.20 total 134.20 last 398.93) - open/close 53865 (time 1763353865.20 total 144.20 last 386.48) - open/close 56934 (time 1763353875.20 total 154.20 last 306.90) - open/close 60000 (time 1763353883.01 total 162.01 last 392.54) - open/close 63646 (time 1763353893.02 total 172.01 last 364.56) total: 64768 open/close in 175.25 seconds: 369.57 ops/second - /unlink 4919 (time 1763353907.76 total 10.00 last 491.86) - /unlink 8230 (time 1763353917.76 total 20.00 last 331.06) - /unlink 10000 (time 1763353921.50 total 23.75 last 472.42) - /unlink 14855 (time 1763353931.50 total 33.75 last 485.47) - /unlink 20000 (time 1763353940.86 total 43.11 last 549.92) - /unlink 25762 (time 1763353950.86 total 53.11 last 576.17) - /unlink 30000 (time 1763353958.83 total 61.07 last 531.89) - /unlink 36747 (time 1763353968.83 total 71.08 last 674.56) - /unlink 40000 (time 1763353974.00 total 76.25 last 629.12) - /unlink 46678 (time 1763353984.00 total 86.25 last 667.77) - /unlink 50000 (time 1763353989.03 total 91.27 last 661.28) - /unlink 55977 (time 1763353999.03 total 101.28 last 597.24) - /unlink 60000 (time 1763354006.68 total 108.93 last 525.76) total: 64768 /unlink in 118.86 seconds: 544.90 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3900 (time 1763354050.75 total 10.00 last 389.92) - open/close 7924 (time 1763354060.76 total 20.00 last 402.34) - open/close 10000 (time 1763354065.55 total 24.80 last 432.63) - open/close 14764 (time 1763354075.56 total 34.80 last 476.39) - open/close 19162 (time 1763354085.56 total 44.80 last 439.77) - open/close 23302 (time 1763354095.56 total 54.80 last 413.96) - open/close 27353 (time 1763354105.56 total 64.80 last 405.08) - open/close 30000 (time 1763354112.54 total 71.79 last 378.99) - open/close 34236 (time 1763354122.54 total 81.79 last 423.56) - open/close 38435 (time 1763354132.54 total 91.79 last 419.88) - open/close 40000 (time 1763354136.71 total 95.96 last 375.65) - open/close 44300 (time 1763354146.71 total 105.96 last 429.97) - open/close 48069 (time 1763354156.71 total 115.96 last 376.82) - open/close 50000 (time 1763354162.41 total 121.66 last 338.66) - open/close 54381 (time 1763354172.42 total 131.66 last 438.04) - open/close 58617 (time 1763354182.42 total 141.66 last 423.57) - open/close 60000 (time 1763354185.58 total 144.83 last 437.41) - open/close 64624 (time 1763354195.58 total 154.83 last 462.40) total: 64768 open/close in 155.11 seconds: 417.55 ops/second - /unlink 8148 (time 1763354207.13 total 10.00 last 814.78) - /unlink 10000 (time 1763354209.81 total 12.68 last 692.27) - /unlink 17380 (time 1763354219.81 total 22.68 last 737.99) - /unlink 20000 (time 1763354222.87 total 25.74 last 854.60) - /unlink 28032 (time 1763354232.87 total 35.74 last 803.12) - /unlink 30000 (time 1763354235.36 total 38.23 last 790.22) - /unlink 37440 (time 1763354245.37 total 48.23 last 743.96) - /unlink 40000 (time 1763354248.67 total 51.53 last 775.75) - /unlink 46421 (time 1763354258.67 total 61.54 last 641.99) - /unlink 50000 (time 1763354264.35 total 67.22 last 629.41) - /unlink 56954 (time 1763354274.35 total 77.22 last 695.40) - /unlink 60000 (time 1763354278.54 total 81.41 last 727.24) total: 64768 /unlink in 87.31 seconds: 741.78 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg303-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg303-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg303-server stop mds service on oleg303-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg303-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg303-server' oleg303-server: oleg303-server.virtnet: executing unload_modules_local oleg303-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg303-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg303-client: oleg303-client: ssh exited with exit code 2 pdsh@oleg303-client: oleg303-server: ssh exited with exit code 2 pdsh@oleg303-client: oleg303-server: ssh exited with exit code 2 pdsh@oleg303-client: oleg303-client: ssh exited with exit code 2