== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 01:33:41 (1769754821) Stopping clients: oleg658-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg658-client.virtnet /mnt/lustre2 (opts:-f) oleg658-server: oleg658-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg658-server' oleg658-server: oleg658-server.virtnet: executing load_modules_local oleg658-server: Loading modules from /home/green/git/lustre-release/lustre oleg658-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg658-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg658-server: detected 4 online CPUs by sysfs oleg658-server: MODOPTS_LIBCFS= oleg658-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg658-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg658-server: oleg658-server.virtnet: executing set_default_debug -1 all pdsh@oleg658-client: oleg658-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg658-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg658-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg658-server: oleg658-server.virtnet: executing set_default_debug -1 all pdsh@oleg658-client: oleg658-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg658-client.virtnet: -o user_xattr,flock 192.168.206.158@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 708 (time 1769754947.62 total 10.01 last 70.73) - open/close 1253 (time 1769754957.63 total 20.01 last 54.48) - open/close 1836 (time 1769754967.63 total 30.02 last 58.27) - open/close 2514 (time 1769754977.65 total 40.04 last 67.70) - open/close 3206 (time 1769754987.65 total 50.04 last 69.17) - open/close 3845 (time 1769754997.66 total 60.05 last 63.87) - open/close 4507 (time 1769755007.66 total 70.05 last 66.20) - open/close 5214 (time 1769755017.66 total 80.05 last 70.70) - open/close 5908 (time 1769755027.66 total 90.05 last 69.40) - open/close 6559 (time 1769755037.67 total 100.06 last 65.03) - open/close 7099 (time 1769755047.67 total 110.06 last 53.98) - open/close 7690 (time 1769755057.68 total 120.07 last 59.06) - open/close 8326 (time 1769755067.69 total 130.08 last 63.55) - open/close 8968 (time 1769755077.69 total 140.08 last 64.19) - open/close 9762 (time 1769755087.69 total 150.08 last 79.35) - open/close 10000 (time 1769755090.91 total 153.30 last 73.98) - open/close 10757 (time 1769755100.92 total 163.31 last 75.66) - open/close 11543 (time 1769755110.92 total 173.31 last 78.54) - open/close 12386 (time 1769755120.93 total 183.32 last 84.26) - open/close 12993 (time 1769755130.94 total 193.33 last 60.62) - open/close 13569 (time 1769755140.94 total 203.33 last 57.60) - open/close 14217 (time 1769755150.95 total 213.34 last 64.76) - open/close 14906 (time 1769755160.95 total 223.34 last 68.87) - open/close 15647 (time 1769755170.95 total 233.34 last 74.09) - open/close 16232 (time 1769755180.96 total 243.35 last 58.45) - open/close 16702 (time 1769755190.96 total 253.35 last 46.99) - open/close 17307 (time 1769755200.97 total 263.36 last 60.44) - open/close 18323 (time 1769755210.99 total 273.38 last 101.46) - open/close 18944 (time 1769755221.00 total 283.39 last 62.02) - open/close 19620 (time 1769755231.00 total 293.39 last 67.60) - open/close 20000 (time 1769755236.67 total 299.06 last 67.02) - open/close 20679 (time 1769755246.68 total 309.07 last 67.85) - open/close 21226 (time 1769755256.69 total 319.08 last 54.65) - open/close 21808 (time 1769755266.70 total 329.08 last 58.15) - open/close 22438 (time 1769755276.71 total 339.10 last 62.92) - open/close 23197 (time 1769755286.72 total 349.11 last 75.82) - open/close 23972 (time 1769755296.72 total 359.11 last 77.49) - open/close 24688 (time 1769755306.72 total 369.11 last 71.57) - open/close 25456 (time 1769755316.73 total 379.12 last 76.75) - open/close 26116 (time 1769755326.75 total 389.14 last 65.88) - open/close 26731 (time 1769755336.75 total 399.14 last 61.49) - open/close 27430 (time 1769755346.76 total 409.15 last 69.87) - open/close 28166 (time 1769755356.82 total 419.21 last 73.14) - open/close 28827 (time 1769755366.83 total 429.22 last 66.00) - open/close 29517 (time 1769755376.84 total 439.23 last 68.97) - open/close 30000 (time 1769755383.14 total 445.53 last 76.60) - open/close 30741 (time 1769755393.15 total 455.54 last 74.08) - open/close 31328 (time 1769755403.16 total 465.55 last 58.65) - open/close 31871 (time 1769755413.20 total 475.59 last 54.06) - open/close 32577 (time 1769755423.21 total 485.60 last 70.55) - open/close 33216 (time 1769755433.22 total 495.61 last 63.84) - open/close 33892 (time 1769755443.22 total 505.61 last 67.59) - open/close 34618 (time 1769755453.22 total 515.61 last 72.56) - open/close 35363 (time 1769755463.23 total 525.61 last 74.50) - open/close 35972 (time 1769755473.23 total 535.62 last 60.87) - open/close 36676 (time 1769755483.24 total 545.63 last 70.31) - open/close 37441 (time 1769755493.24 total 555.63 last 76.49) - open/close 38329 (time 1769755503.25 total 565.64 last 88.72) - open/close 38869 (time 1769755513.26 total 575.64 last 53.99) - open/close 39716 (time 1769755523.26 total 585.65 last 84.65) - open/close 40000 (time 1769755526.81 total 589.20 last 80.10) - open/close 40746 (time 1769755536.81 total 599.20 last 74.54) - open/close 41499 (time 1769755546.83 total 609.21 last 75.22) - open/close 42181 (time 1769755556.83 total 619.22 last 68.17) - open/close 42934 (time 1769755566.83 total 629.22 last 75.30) - open/close 43695 (time 1769755576.83 total 639.22 last 76.07) - open/close 44434 (time 1769755586.85 total 649.24 last 73.78) - open/close 45191 (time 1769755596.85 total 659.24 last 75.68) - open/close 45968 (time 1769755606.87 total 669.26 last 77.61) - open/close 46732 (time 1769755616.87 total 679.26 last 76.35) - open/close 47473 (time 1769755626.88 total 689.27 last 74.05) - open/close 48153 (time 1769755636.88 total 699.27 last 67.97) - open/close 48832 (time 1769755646.91 total 709.30 last 67.74) - open/close 49367 (time 1769755656.92 total 719.31 last 53.45) - open/close 50000 (time 1769755665.85 total 728.24 last 70.86) - open/close 50767 (time 1769755675.86 total 738.25 last 76.63) - open/close 51550 (time 1769755685.88 total 748.27 last 78.12) - open/close 52338 (time 1769755695.88 total 758.27 last 78.79) - open/close 52999 (time 1769755705.90 total 768.29 last 65.99) - open/close 53774 (time 1769755715.90 total 778.29 last 77.47) - open/close 54594 (time 1769755725.93 total 788.32 last 81.79) - open/close 55367 (time 1769755735.94 total 798.33 last 77.23) - open/close 56161 (time 1769755745.94 total 808.33 last 79.37) - open/close 57000 (time 1769755755.95 total 818.34 last 83.84) - open/close 57846 (time 1769755765.95 total 828.34 last 84.59) - open/close 58612 (time 1769755775.96 total 838.35 last 76.54) - open/close 59432 (time 1769755785.96 total 848.35 last 81.98) - open/close 60000 (time 1769755793.34 total 855.73 last 76.96) - open/close 60871 (time 1769755803.35 total 865.74 last 87.03) - open/close 61788 (time 1769755813.35 total 875.74 last 91.66) - open/close 62562 (time 1769755823.35 total 885.74 last 77.40) - open/close 63411 (time 1769755833.37 total 895.76 last 84.77) - open/close 64276 (time 1769755843.38 total 905.77 last 86.40) total: 64768 open/close in 910.62 seconds: 71.13 ops/second - /unlink 836 (time 1769755862.08 total 10.00 last 83.58) - /unlink 1401 (time 1769755872.08 total 20.00 last 56.50) - /unlink 2014 (time 1769755882.09 total 30.02 last 61.21) - /unlink 2695 (time 1769755892.10 total 40.02 last 68.07) - /unlink 3456 (time 1769755902.11 total 50.03 last 76.03) - /unlink 4315 (time 1769755912.11 total 60.04 last 85.83) - /unlink 5166 (time 1769755922.13 total 70.06 last 84.97) - /unlink 5942 (time 1769755932.14 total 80.07 last 77.53) - /unlink 6755 (time 1769755942.15 total 90.08 last 81.21) - /unlink 7585 (time 1769755952.16 total 100.09 last 82.92) - /unlink 8443 (time 1769755962.16 total 110.09 last 85.78) - /unlink 9345 (time 1769755972.19 total 120.11 last 89.97) - /unlink 10000 (time 1769755980.48 total 128.40 last 79.01) - /unlink 10802 (time 1769755990.49 total 138.42 last 80.11) - /unlink 11603 (time 1769756000.49 total 148.42 last 80.08) - /unlink 12370 (time 1769756010.50 total 158.43 last 76.62) - /unlink 13133 (time 1769756020.52 total 168.45 last 76.17) - /unlink 13966 (time 1769756030.52 total 178.45 last 83.29) - /unlink 14818 (time 1769756040.53 total 188.45 last 85.15) - /unlink 15552 (time 1769756050.54 total 198.46 last 73.33) - /unlink 16209 (time 1769756060.55 total 208.48 last 65.57) - /unlink 16854 (time 1769756070.57 total 218.49 last 64.43) - /unlink 17760 (time 1769756080.57 total 228.50 last 90.55) - /unlink 18533 (time 1769756090.58 total 238.50 last 77.25) - /unlink 19225 (time 1769756100.60 total 248.53 last 69.04) - /unlink 19769 (time 1769756110.61 total 258.53 last 54.37) - /unlink 20000 (time 1769756115.22 total 263.14 last 50.10) - /unlink 20572 (time 1769756125.25 total 273.18 last 57.00) - /unlink 21179 (time 1769756135.30 total 283.23 last 60.41) - /unlink 22032 (time 1769756145.32 total 293.24 last 85.15) - /unlink 22847 (time 1769756155.33 total 303.25 last 81.44) - /unlink 23703 (time 1769756165.33 total 313.26 last 85.55) - /unlink 24559 (time 1769756175.33 total 323.26 last 85.58) - /unlink 25403 (time 1769756185.34 total 333.27 last 84.35) - /unlink 26107 (time 1769756195.34 total 343.27 last 70.40) - /unlink 26941 (time 1769756205.35 total 353.28 last 83.29) - /unlink 27692 (time 1769756215.36 total 363.29 last 75.06) - /unlink 28479 (time 1769756225.36 total 373.29 last 78.66) - /unlink 29254 (time 1769756235.37 total 383.30 last 77.42) - /unlink 29952 (time 1769756245.38 total 393.30 last 69.77) - /unlink 30756 (time 1769756255.38 total 403.31 last 80.37) - /unlink 31657 (time 1769756265.39 total 413.32 last 90.03) - /unlink 32590 (time 1769756275.40 total 423.33 last 93.19) - /unlink 33407 (time 1769756285.42 total 433.34 last 81.59) - /unlink 34022 (time 1769756295.42 total 443.35 last 61.45) - /unlink 34784 (time 1769756305.43 total 453.36 last 76.14) - /unlink 35572 (time 1769756315.44 total 463.37 last 78.72) - /unlink 36411 (time 1769756325.45 total 473.38 last 83.84) - /unlink 37298 (time 1769756335.46 total 483.39 last 88.59) - /unlink 38101 (time 1769756345.48 total 493.40 last 80.19) - /unlink 39034 (time 1769756355.48 total 503.41 last 93.26) - /unlink 39984 (time 1769756365.48 total 513.41 last 94.99) - /unlink 40816 (time 1769756375.49 total 523.42 last 83.13) - /unlink 41552 (time 1769756385.49 total 533.42 last 73.60) - /unlink 42389 (time 1769756395.51 total 543.44 last 83.54) - /unlink 43054 (time 1769756405.51 total 553.44 last 66.48) - /unlink 43878 (time 1769756415.54 total 563.46 last 82.22) - /unlink 44398 (time 1769756425.54 total 573.46 last 51.98) - /unlink 45027 (time 1769756435.56 total 583.49 last 62.77) - /unlink 45664 (time 1769756445.56 total 593.49 last 63.69) - /unlink 46644 (time 1769756455.56 total 603.49 last 97.96) - /unlink 47510 (time 1769756465.58 total 613.50 last 86.51) - /unlink 48373 (time 1769756475.58 total 623.50 last 86.27) - /unlink 49121 (time 1769756485.59 total 633.52 last 74.68) - /unlink 50000 (time 1769756495.27 total 643.20 last 90.81) - /unlink 50905 (time 1769756505.28 total 653.20 last 90.49) - /unlink 51780 (time 1769756515.29 total 663.21 last 87.41) - /unlink 52580 (time 1769756525.29 total 673.22 last 79.93) - /unlink 53369 (time 1769756535.30 total 683.23 last 78.86) - /unlink 54186 (time 1769756545.31 total 693.23 last 81.64) - /unlink 54966 (time 1769756555.31 total 703.24 last 77.97) - /unlink 55706 (time 1769756565.33 total 713.26 last 73.83) - /unlink 56547 (time 1769756575.35 total 723.27 last 84.01) - /unlink 57409 (time 1769756585.36 total 733.28 last 86.11) - /unlink 58297 (time 1769756595.37 total 743.30 last 88.67) - /unlink 59079 (time 1769756605.38 total 753.31 last 78.09) - /unlink 59797 (time 1769756615.39 total 763.31 last 71.77) - /unlink 60000 (time 1769756617.91 total 765.83 last 80.62) - /unlink 60923 (time 1769756627.91 total 775.83 last 92.30) - /unlink 61814 (time 1769756637.92 total 785.84 last 89.02) - /unlink 62642 (time 1769756647.92 total 795.84 last 82.78) - /unlink 63678 (time 1769756657.92 total 805.85 last 103.54) - /unlink 64612 (time 1769756667.93 total 815.85 last 93.37) total: 64768 /unlink in 817.82 seconds: 79.20 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1017 (time 1769756704.22 total 10.01 last 101.64) - open/close 1839 (time 1769756714.22 total 20.01 last 82.19) - open/close 2785 (time 1769756724.23 total 30.01 last 94.55) - open/close 3681 (time 1769756734.23 total 40.02 last 89.52) - open/close 4579 (time 1769756744.24 total 50.02 last 89.78) - open/close 5488 (time 1769756754.24 total 60.03 last 90.83) - open/close 6379 (time 1769756764.25 total 70.04 last 89.05) - open/close 7181 (time 1769756774.26 total 80.04 last 80.16) - open/close 8027 (time 1769756784.26 total 90.05 last 84.57) - open/close 8885 (time 1769756794.26 total 100.05 last 85.75) - open/close 9672 (time 1769756804.26 total 110.05 last 78.70) - open/close 10000 (time 1769756807.68 total 113.47 last 95.93) - open/close 10827 (time 1769756817.69 total 123.48 last 82.63) - open/close 11613 (time 1769756827.72 total 133.50 last 78.40) - open/close 12433 (time 1769756837.72 total 143.51 last 81.97) - open/close 13371 (time 1769756847.73 total 153.51 last 93.76) - open/close 14218 (time 1769756857.74 total 163.52 last 84.60) - open/close 15006 (time 1769756867.74 total 173.53 last 78.79) - open/close 15881 (time 1769756877.74 total 183.53 last 87.47) - open/close 16699 (time 1769756887.78 total 193.57 last 81.51) - open/close 17551 (time 1769756897.78 total 203.57 last 85.15) - open/close 18335 (time 1769756907.80 total 213.58 last 78.30) - open/close 19231 (time 1769756917.81 total 223.60 last 89.50) - open/close 20000 (time 1769756927.76 total 233.55 last 77.27) - open/close 20849 (time 1769756937.76 total 243.55 last 84.89) - open/close 21719 (time 1769756947.76 total 253.55 last 86.98) - open/close 22580 (time 1769756957.78 total 263.56 last 85.99) - open/close 23489 (time 1769756967.78 total 273.57 last 90.84) - open/close 24384 (time 1769756977.79 total 283.57 last 89.46) - open/close 25217 (time 1769756987.79 total 293.58 last 83.25) - open/close 26101 (time 1769756997.79 total 303.58 last 88.39) - open/close 26879 (time 1769757007.80 total 313.59 last 77.77) - open/close 27601 (time 1769757017.81 total 323.60 last 72.10) - open/close 28365 (time 1769757027.81 total 333.60 last 76.40) - open/close 29095 (time 1769757037.81 total 343.60 last 72.98) - open/close 29843 (time 1769757047.82 total 353.61 last 74.75) - open/close 30671 (time 1769757057.83 total 363.61 last 82.76) - open/close 31578 (time 1769757067.83 total 373.62 last 90.67) - open/close 32423 (time 1769757077.84 total 383.62 last 84.45) - open/close 33311 (time 1769757087.84 total 393.62 last 88.79) - open/close 34148 (time 1769757097.86 total 403.65 last 83.49) - open/close 34926 (time 1769757107.88 total 413.67 last 77.67) - open/close 35750 (time 1769757117.88 total 423.67 last 82.39) - open/close 36608 (time 1769757127.89 total 433.68 last 85.72) - open/close 37379 (time 1769757137.91 total 443.69 last 76.98) - open/close 38106 (time 1769757147.92 total 453.70 last 72.63) - open/close 38890 (time 1769757157.92 total 463.71 last 78.35) - open/close 39751 (time 1769757167.92 total 473.71 last 86.08) - open/close 40000 (time 1769757171.16 total 476.95 last 76.87) - open/close 40869 (time 1769757181.17 total 486.96 last 86.83) - open/close 41750 (time 1769757191.18 total 496.97 last 88.02) - open/close 42715 (time 1769757201.19 total 506.97 last 96.43) - open/close 43576 (time 1769757211.19 total 516.98 last 86.06) - open/close 44432 (time 1769757221.20 total 526.99 last 85.53) - open/close 45368 (time 1769757231.20 total 536.99 last 93.56) - open/close 46285 (time 1769757241.21 total 547.00 last 91.65) - open/close 47196 (time 1769757251.21 total 557.00 last 91.09) - open/close 48081 (time 1769757261.22 total 567.01 last 88.40) - open/close 48998 (time 1769757271.22 total 577.01 last 91.69) - open/close 49922 (time 1769757281.23 total 587.01 last 92.34) - open/close 50805 (time 1769757291.24 total 597.03 last 88.20) - open/close 51714 (time 1769757301.25 total 607.03 last 90.82) - open/close 52555 (time 1769757311.25 total 617.04 last 84.05) - open/close 53426 (time 1769757321.26 total 627.05 last 87.05) - open/close 54310 (time 1769757331.26 total 637.05 last 88.39) - open/close 55171 (time 1769757341.26 total 647.05 last 86.07) - open/close 56034 (time 1769757351.28 total 657.06 last 86.20) - open/close 56876 (time 1769757361.28 total 667.07 last 84.17) - open/close 57628 (time 1769757371.28 total 677.07 last 75.17) - open/close 58399 (time 1769757381.32 total 687.11 last 76.80) - open/close 59090 (time 1769757391.33 total 697.11 last 69.07) - open/close 59754 (time 1769757401.33 total 707.11 last 66.39) - open/close 60000 (time 1769757404.43 total 710.22 last 79.20) - open/close 60858 (time 1769757414.44 total 720.23 last 85.73) - open/close 61805 (time 1769757424.45 total 730.24 last 94.63) - open/close 62720 (time 1769757434.45 total 740.24 last 91.46) - open/close 63501 (time 1769757444.47 total 750.26 last 77.97) - open/close 64345 (time 1769757454.48 total 760.26 last 84.34) total: 64768 open/close in 765.97 seconds: 84.56 ops/second - /unlink 781 (time 1769757475.14 total 10.00 last 78.09) - /unlink 1534 (time 1769757485.16 total 20.02 last 75.20) - /unlink 2333 (time 1769757495.16 total 30.02 last 79.90) - /unlink 3158 (time 1769757505.17 total 40.03 last 82.42) - /unlink 4020 (time 1769757515.17 total 50.03 last 86.19) - /unlink 4851 (time 1769757525.17 total 60.03 last 83.05) - /unlink 5651 (time 1769757535.18 total 70.03 last 79.98) - /unlink 6349 (time 1769757545.20 total 80.06 last 69.60) - /unlink 7165 (time 1769757555.21 total 90.07 last 81.54) - /unlink 7989 (time 1769757565.22 total 100.08 last 82.31) - /unlink 8766 (time 1769757575.23 total 110.09 last 77.65) - /unlink 9509 (time 1769757585.24 total 120.09 last 74.25) - /unlink 10000 (time 1769757590.56 total 125.42 last 92.25) - /unlink 10864 (time 1769757600.57 total 135.43 last 86.29) - /unlink 11732 (time 1769757610.58 total 145.44 last 86.71) - /unlink 12603 (time 1769757620.59 total 155.45 last 87.04) - /unlink 13528 (time 1769757630.60 total 165.46 last 92.42) - /unlink 14213 (time 1769757640.60 total 175.46 last 68.47) - /unlink 15083 (time 1769757650.61 total 185.46 last 86.96) - /unlink 15934 (time 1769757660.61 total 195.47 last 85.08) - /unlink 16928 (time 1769757670.61 total 205.47 last 99.33) - /unlink 17768 (time 1769757680.62 total 215.48 last 83.93) - /unlink 18609 (time 1769757690.62 total 225.48 last 84.08) - /unlink 19472 (time 1769757700.63 total 235.49 last 86.28) - /unlink 20000 (time 1769757707.58 total 242.44 last 75.89) - /unlink 20829 (time 1769757717.59 total 252.45 last 82.89) - /unlink 21645 (time 1769757727.59 total 262.45 last 81.56) - /unlink 22688 (time 1769757737.60 total 272.46 last 104.23) - /unlink 23698 (time 1769757747.62 total 282.48 last 100.80) - /unlink 24661 (time 1769757757.62 total 292.48 last 96.28) - /unlink 25524 (time 1769757767.63 total 302.48 last 86.25) - /unlink 26456 (time 1769757777.63 total 312.49 last 93.16) - /unlink 27348 (time 1769757787.63 total 322.49 last 89.19) - /unlink 28307 (time 1769757797.64 total 332.50 last 95.85) - /unlink 29245 (time 1769757807.64 total 342.50 last 93.79) - /unlink 30000 (time 1769757815.96 total 350.82 last 90.68) - /unlink 30853 (time 1769757825.97 total 360.83 last 85.25) - /unlink 31791 (time 1769757835.98 total 370.84 last 93.74) - /unlink 32655 (time 1769757845.98 total 380.84 last 86.39) - /unlink 33575 (time 1769757855.98 total 390.84 last 91.95) - /unlink 34415 (time 1769757865.99 total 400.85 last 83.95) - /unlink 35263 (time 1769757875.99 total 410.85 last 84.80) - /unlink 36070 (time 1769757885.99 total 420.85 last 80.69) - /unlink 36910 (time 1769757895.99 total 430.85 last 83.97) - /unlink 37778 (time 1769757906.00 total 440.86 last 86.73) - /unlink 38628 (time 1769757916.01 total 450.87 last 84.92) - /unlink 39503 (time 1769757926.01 total 460.87 last 87.47) - /unlink 40000 (time 1769757930.72 total 465.58 last 105.62) - /unlink 40817 (time 1769757940.74 total 475.59 last 81.57) - /unlink 41678 (time 1769757950.74 total 485.60 last 86.05) - /unlink 42556 (time 1769757960.74 total 495.60 last 87.78) - /unlink 43430 (time 1769757970.75 total 505.61 last 87.36) - /unlink 44249 (time 1769757980.75 total 515.61 last 81.86) - /unlink 45052 (time 1769757990.75 total 525.61 last 80.30) - /unlink 45898 (time 1769758000.76 total 535.62 last 84.54) - /unlink 46848 (time 1769758010.77 total 545.63 last 94.91) - /unlink 47744 (time 1769758020.78 total 555.64 last 89.50) - /unlink 48692 (time 1769758030.79 total 565.65 last 94.71) - /unlink 49546 (time 1769758040.79 total 575.65 last 85.34) - /unlink 50000 (time 1769758045.71 total 580.57 last 92.41) - /unlink 50939 (time 1769758055.71 total 590.57 last 93.87) - /unlink 51615 (time 1769758065.72 total 600.58 last 67.54) - /unlink 52381 (time 1769758075.72 total 610.58 last 76.59) - /unlink 53307 (time 1769758085.72 total 620.58 last 92.59) - /unlink 54209 (time 1769758095.73 total 630.59 last 90.11) - /unlink 55077 (time 1769758105.74 total 640.60 last 86.77) - /unlink 56011 (time 1769758115.74 total 650.60 last 93.33) - /unlink 56913 (time 1769758125.75 total 660.61 last 90.11) - /unlink 57815 (time 1769758135.76 total 670.62 last 90.16) - /unlink 58726 (time 1769758145.76 total 680.62 last 91.08) - /unlink 59653 (time 1769758155.77 total 690.63 last 92.61) - /unlink 60000 (time 1769758160.09 total 694.95 last 80.30) - /unlink 60973 (time 1769758170.10 total 704.96 last 97.25) - /unlink 61923 (time 1769758180.10 total 714.96 last 94.98) - /unlink 62700 (time 1769758190.11 total 724.97 last 77.60) - /unlink 63713 (time 1769758200.12 total 734.98 last 101.22) - /unlink 64696 (time 1769758210.12 total 744.98 last 98.26) total: 64768 /unlink in 745.93 seconds: 86.83 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1001 (time 1769758244.90 total 10.01 last 100.04) - open/close 2066 (time 1769758254.90 total 20.01 last 106.43) - open/close 3115 (time 1769758264.91 total 30.02 last 104.87) - open/close 4178 (time 1769758274.91 total 40.02 last 106.30) - open/close 5154 (time 1769758284.91 total 50.02 last 97.58) - open/close 6243 (time 1769758294.92 total 60.02 last 108.83) - open/close 7360 (time 1769758304.93 total 70.03 last 111.60) - open/close 8628 (time 1769758314.93 total 80.04 last 126.71) - open/close 9818 (time 1769758324.94 total 90.05 last 118.87) - open/close 11009 (time 1769758334.95 total 100.05 last 119.08) - open/close 12214 (time 1769758344.95 total 110.06 last 120.45) - open/close 13462 (time 1769758354.95 total 120.06 last 124.74) - open/close 14682 (time 1769758364.97 total 130.07 last 121.87) - open/close 15713 (time 1769758374.97 total 140.08 last 103.05) - open/close 16923 (time 1769758384.98 total 150.09 last 120.84) - open/close 18007 (time 1769758395.00 total 160.11 last 108.25) - open/close 19195 (time 1769758405.00 total 170.11 last 118.75) - open/close 20000 (time 1769758412.01 total 177.12 last 114.86) - open/close 21055 (time 1769758422.02 total 187.12 last 105.45) - open/close 22117 (time 1769758432.02 total 197.13 last 106.14) - open/close 23164 (time 1769758442.02 total 207.13 last 104.68) - open/close 24285 (time 1769758452.03 total 217.13 last 112.06) - open/close 25318 (time 1769758462.03 total 227.14 last 103.22) - open/close 26479 (time 1769758472.04 total 237.14 last 116.05) - open/close 27609 (time 1769758482.04 total 247.15 last 112.97) - open/close 28668 (time 1769758492.05 total 257.16 last 105.82) - open/close 29789 (time 1769758502.05 total 267.16 last 112.04) - open/close 31006 (time 1769758512.06 total 277.17 last 121.62) - open/close 32164 (time 1769758522.07 total 287.18 last 115.71) - open/close 33229 (time 1769758532.08 total 297.19 last 106.38) - open/close 34371 (time 1769758542.08 total 307.19 last 114.17) - open/close 35396 (time 1769758552.09 total 317.20 last 102.42) - open/close 36489 (time 1769758562.09 total 327.20 last 109.26) - open/close 37639 (time 1769758572.10 total 337.21 last 114.91) - open/close 38808 (time 1769758582.11 total 347.22 last 116.80) - open/close 40000 (time 1769758591.09 total 356.20 last 132.71) - open/close 41167 (time 1769758601.10 total 366.21 last 116.55) - open/close 42278 (time 1769758611.11 total 376.21 last 111.09) - open/close 43361 (time 1769758621.11 total 386.22 last 108.24) - open/close 44299 (time 1769758631.12 total 396.22 last 93.75) - open/close 45402 (time 1769758641.12 total 406.23 last 110.27) - open/close 46628 (time 1769758651.12 total 416.23 last 122.53) - open/close 47955 (time 1769758661.13 total 426.24 last 132.60) - open/close 49264 (time 1769758671.14 total 436.25 last 130.78) - open/close 50000 (time 1769758676.13 total 441.23 last 147.66) - open/close 51354 (time 1769758686.13 total 451.24 last 135.29) - open/close 52594 (time 1769758696.15 total 461.26 last 123.80) - open/close 53870 (time 1769758706.16 total 471.26 last 127.53) - open/close 55149 (time 1769758716.16 total 481.26 last 127.89) - open/close 56514 (time 1769758726.16 total 491.27 last 136.38) - open/close 57891 (time 1769758736.17 total 501.28 last 137.64) - open/close 59254 (time 1769758746.17 total 511.28 last 136.22) - open/close 60000 (time 1769758752.27 total 517.38 last 122.37) - open/close 61179 (time 1769758762.27 total 527.38 last 117.88) - open/close 62399 (time 1769758772.27 total 537.38 last 121.98) - open/close 63707 (time 1769758782.28 total 547.39 last 130.74) total: 64768 open/close in 555.27 seconds: 116.64 ops/second - /unlink 2202 (time 1769758802.45 total 10.00 last 220.12) - /unlink 4201 (time 1769758812.45 total 20.00 last 199.89) - /unlink 6006 (time 1769758822.45 total 30.01 last 180.39) - /unlink 8195 (time 1769758832.46 total 40.02 last 218.79) - /unlink 10000 (time 1769758841.82 total 49.37 last 192.86) - /unlink 12010 (time 1769758851.82 total 59.38 last 200.98) - /unlink 14048 (time 1769758861.82 total 69.38 last 203.78) - /unlink 15971 (time 1769758871.82 total 79.38 last 192.27) - /unlink 17901 (time 1769758881.82 total 89.38 last 192.97) - /unlink 19820 (time 1769758891.82 total 99.38 last 191.90) - /unlink 22048 (time 1769758901.82 total 109.38 last 222.79) - /unlink 24188 (time 1769758911.83 total 119.38 last 213.93) - /unlink 26509 (time 1769758921.83 total 129.39 last 232.04) - /unlink 28659 (time 1769758931.83 total 139.39 last 214.97) - /unlink 30000 (time 1769758937.85 total 145.40 last 222.91) - /unlink 32161 (time 1769758947.86 total 155.42 last 215.85) - /unlink 34228 (time 1769758957.86 total 165.42 last 206.66) - /unlink 36326 (time 1769758967.86 total 175.42 last 209.68) - /unlink 38564 (time 1769758977.87 total 185.42 last 223.78) - /unlink 40000 (time 1769758985.51 total 193.06 last 187.96) - /unlink 41918 (time 1769758995.51 total 203.07 last 191.77) - /unlink 43783 (time 1769759005.51 total 213.07 last 186.50) - /unlink 45785 (time 1769759015.51 total 223.07 last 200.13) - /unlink 47693 (time 1769759025.51 total 233.07 last 190.79) - /unlink 49549 (time 1769759035.52 total 243.08 last 185.48) - /unlink 50000 (time 1769759037.53 total 245.09 last 224.48) - /unlink 51897 (time 1769759047.53 total 255.09 last 189.68) - /unlink 53871 (time 1769759057.53 total 265.09 last 197.35) - /unlink 55820 (time 1769759067.53 total 275.09 last 194.88) - /unlink 57909 (time 1769759077.53 total 285.09 last 208.84) - /unlink 59889 (time 1769759087.54 total 295.10 last 197.95) - /unlink 62054 (time 1769759097.54 total 305.10 last 216.47) - /unlink 64234 (time 1769759107.54 total 315.10 last 217.87) total: 64768 /unlink in 317.25 seconds: 204.16 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1142 (time 1769759144.66 total 10.02 last 114.02) - open/close 2375 (time 1769759154.66 total 20.02 last 123.29) - open/close 3601 (time 1769759164.66 total 30.02 last 122.59) - open/close 4980 (time 1769759174.66 total 40.02 last 137.83) - open/close 6318 (time 1769759184.68 total 50.04 last 133.62) - open/close 7616 (time 1769759194.68 total 60.04 last 129.78) - open/close 9077 (time 1769759204.68 total 70.04 last 146.08) - open/close 10000 (time 1769759211.95 total 77.31 last 126.93) - open/close 11217 (time 1769759221.96 total 87.32 last 121.56) - open/close 12482 (time 1769759231.97 total 97.33 last 126.46) - open/close 13884 (time 1769759241.97 total 107.33 last 140.20) - open/close 15309 (time 1769759251.97 total 117.33 last 142.42) - open/close 16658 (time 1769759261.98 total 127.34 last 134.81) - open/close 18260 (time 1769759271.98 total 137.34 last 160.15) - open/close 19912 (time 1769759281.98 total 147.34 last 165.20) - open/close 21324 (time 1769759291.98 total 157.34 last 141.18) - open/close 22835 (time 1769759301.99 total 167.35 last 151.01) - open/close 24266 (time 1769759311.99 total 177.35 last 143.02) - open/close 25703 (time 1769759322.00 total 187.36 last 143.62) - open/close 27208 (time 1769759332.01 total 197.37 last 150.34) - open/close 28804 (time 1769759342.01 total 207.37 last 159.55) - open/close 30000 (time 1769759349.27 total 214.63 last 164.79) - open/close 31627 (time 1769759359.27 total 224.63 last 162.66) - open/close 33420 (time 1769759369.28 total 234.64 last 179.26) - open/close 34910 (time 1769759379.28 total 244.64 last 149.00) - open/close 36559 (time 1769759389.29 total 254.65 last 164.65) - open/close 38306 (time 1769759399.29 total 264.65 last 174.65) - open/close 40000 (time 1769759407.76 total 273.12 last 200.04) - open/close 41613 (time 1769759417.77 total 283.13 last 161.25) - open/close 43251 (time 1769759427.77 total 293.13 last 163.79) - open/close 45055 (time 1769759437.77 total 303.13 last 180.38) - open/close 46332 (time 1769759447.77 total 313.13 last 127.67) - open/close 47641 (time 1769759457.78 total 323.15 last 130.70) - open/close 49357 (time 1769759467.79 total 333.15 last 171.44) - open/close 50000 (time 1769759472.01 total 337.37 last 152.57) - open/close 51564 (time 1769759482.01 total 347.37 last 156.35) - open/close 53415 (time 1769759492.02 total 357.38 last 184.99) - open/close 55503 (time 1769759502.06 total 367.42 last 207.91) - open/close 57155 (time 1769759512.06 total 377.42 last 165.18) - open/close 58508 (time 1769759522.07 total 387.43 last 135.25) - open/close 60000 (time 1769759530.21 total 395.57 last 183.28) - open/close 61719 (time 1769759540.21 total 405.57 last 171.86) - open/close 63288 (time 1769759550.21 total 415.57 last 156.89) - open/close 64759 (time 1769759560.21 total 425.57 last 147.06) total: 64768 open/close in 425.61 seconds: 152.18 ops/second - /unlink 2259 (time 1769759572.46 total 10.00 last 225.85) - /unlink 4352 (time 1769759582.46 total 20.00 last 209.25) - /unlink 6571 (time 1769759592.46 total 30.01 last 221.87) - /unlink 8533 (time 1769759602.47 total 40.01 last 196.05) - /unlink 10000 (time 1769759609.47 total 47.02 last 209.37) - /unlink 12345 (time 1769759619.48 total 57.02 last 234.48) - /unlink 14812 (time 1769759629.48 total 67.02 last 246.69) - /unlink 17431 (time 1769759639.48 total 77.03 last 261.80) - /unlink 19738 (time 1769759649.48 total 87.03 last 230.61) - /unlink 22117 (time 1769759659.48 total 97.03 last 237.88) - /unlink 24687 (time 1769759669.49 total 107.04 last 256.86) - /unlink 27001 (time 1769759679.49 total 117.04 last 231.36) - /unlink 29145 (time 1769759689.49 total 127.04 last 214.36) - /unlink 30000 (time 1769759693.13 total 130.67 last 235.38) - /unlink 32432 (time 1769759703.13 total 140.67 last 243.18) - /unlink 34836 (time 1769759713.13 total 150.67 last 240.40) - /unlink 37500 (time 1769759723.13 total 160.68 last 266.33) - /unlink 39850 (time 1769759733.13 total 170.68 last 234.97) - /unlink 42305 (time 1769759743.13 total 180.68 last 245.49) - /unlink 44863 (time 1769759753.13 total 190.68 last 255.76) - /unlink 47399 (time 1769759763.14 total 200.68 last 253.46) - /unlink 49761 (time 1769759773.14 total 210.69 last 236.14) - /unlink 52138 (time 1769759783.14 total 220.69 last 237.67) - /unlink 55017 (time 1769759793.14 total 230.69 last 287.84) - /unlink 57682 (time 1769759803.15 total 240.69 last 266.42) - /unlink 60000 (time 1769759812.28 total 249.82 last 253.86) - /unlink 62762 (time 1769759822.28 total 259.83 last 276.14) total: 64768 /unlink in 268.16 seconds: 241.53 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1774 (time 1769759858.20 total 10.00 last 177.38) - open/close 3445 (time 1769759868.20 total 20.01 last 167.03) - open/close 5236 (time 1769759878.20 total 30.01 last 179.08) - open/close 7154 (time 1769759888.20 total 40.01 last 191.73) - open/close 8842 (time 1769759898.21 total 50.01 last 168.79) - open/close 10000 (time 1769759905.23 total 57.03 last 164.86) - open/close 11707 (time 1769759915.23 total 67.03 last 170.70) - open/close 14065 (time 1769759925.23 total 77.04 last 235.77) - open/close 16150 (time 1769759935.23 total 87.04 last 208.43) - open/close 18781 (time 1769759945.24 total 97.04 last 263.04) - open/close 20000 (time 1769759950.95 total 102.76 last 213.18) - open/close 22243 (time 1769759960.96 total 112.76 last 224.26) - open/close 24462 (time 1769759970.96 total 122.76 last 221.85) - open/close 26709 (time 1769759980.96 total 132.76 last 224.70) - open/close 28760 (time 1769759990.96 total 142.77 last 205.01) - open/close 30000 (time 1769759997.95 total 149.76 last 177.48) - open/close 32267 (time 1769760007.95 total 159.76 last 226.68) - open/close 34318 (time 1769760017.95 total 169.76 last 205.06) - open/close 36178 (time 1769760027.95 total 179.76 last 185.98) - open/close 38624 (time 1769760037.95 total 189.76 last 244.57) - open/close 40000 (time 1769760045.79 total 197.59 last 175.68) - open/close 42088 (time 1769760055.79 total 207.59 last 208.80) - open/close 43766 (time 1769760065.79 total 217.59 last 167.79) - open/close 45682 (time 1769760075.79 total 227.60 last 191.48) - open/close 47311 (time 1769760085.80 total 237.60 last 162.87) - open/close 48783 (time 1769760095.80 total 247.60 last 147.18) - open/close 50000 (time 1769760103.12 total 254.93 last 166.15) - open/close 51895 (time 1769760113.13 total 264.93 last 189.43) - open/close 53858 (time 1769760123.13 total 274.94 last 196.16) - open/close 55888 (time 1769760133.13 total 284.94 last 202.97) - open/close 58654 (time 1769760143.14 total 294.94 last 276.57) - open/close 60000 (time 1769760150.44 total 302.25 last 184.20) - open/close 62017 (time 1769760160.45 total 312.25 last 201.65) - open/close 63666 (time 1769760170.45 total 322.25 last 164.84) total: 64768 open/close in 328.55 seconds: 197.13 ops/second - /unlink 3708 (time 1769760188.30 total 10.00 last 370.78) - /unlink 7309 (time 1769760198.30 total 20.00 last 360.09) - /unlink 10000 (time 1769760208.29 total 29.99 last 269.35) - /unlink 12922 (time 1769760218.30 total 39.99 last 292.14) - /unlink 17084 (time 1769760228.30 total 49.99 last 416.20) - /unlink 20000 (time 1769760234.84 total 56.53 last 445.93) - /unlink 24400 (time 1769760244.84 total 66.53 last 439.99) - /unlink 27948 (time 1769760254.84 total 76.53 last 354.73) - /unlink 30000 (time 1769760260.69 total 82.39 last 350.59) - /unlink 33735 (time 1769760270.69 total 92.39 last 373.50) - /unlink 36364 (time 1769760280.69 total 102.39 last 262.86) - /unlink 40000 (time 1769760288.16 total 109.86 last 486.76) - /unlink 44210 (time 1769760298.17 total 119.86 last 420.84) - /unlink 47564 (time 1769760308.17 total 129.86 last 335.37) - /unlink 50000 (time 1769760315.43 total 137.13 last 335.43) - /unlink 53788 (time 1769760325.43 total 147.13 last 378.65) - /unlink 57615 (time 1769760335.43 total 157.13 last 382.69) - /unlink 60000 (time 1769760341.42 total 163.12 last 398.15) - /unlink 64396 (time 1769760351.42 total 173.12 last 439.60) total: 64768 /unlink in 174.05 seconds: 372.12 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg658-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg658-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg658-server stop mds service on oleg658-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg658-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg658-server' oleg658-server: oleg658-server.virtnet: executing unload_modules_local oleg658-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg658-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg658-client: oleg658-client: ssh exited with exit code 2 pdsh@oleg658-client: oleg658-server: ssh exited with exit code 2 pdsh@oleg658-client: oleg658-server: ssh exited with exit code 2 pdsh@oleg658-client: oleg658-client: ssh exited with exit code 2