== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 19:35:48 (1763080548) Stopping clients: oleg220-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg220-client.virtnet /mnt/lustre2 (opts:-f) oleg220-server: oleg220-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg220-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg220-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 707 (time 1763080653.91 total 10.00 last 70.68) - open/close 1412 (time 1763080663.93 total 20.02 last 70.35) - open/close 2122 (time 1763080673.93 total 30.03 last 70.98) - open/close 2832 (time 1763080683.95 total 40.04 last 70.88) - open/close 3541 (time 1763080693.96 total 50.05 last 70.87) - open/close 4335 (time 1763080703.96 total 60.06 last 79.33) - open/close 5101 (time 1763080713.97 total 70.07 last 76.53) - open/close 5870 (time 1763080723.98 total 80.07 last 76.86) - open/close 6604 (time 1763080733.98 total 90.07 last 73.39) - open/close 7277 (time 1763080743.98 total 100.08 last 67.28) - open/close 7948 (time 1763080754.00 total 110.09 last 67.01) - open/close 8535 (time 1763080764.02 total 120.11 last 58.56) - open/close 9162 (time 1763080774.03 total 130.12 last 62.65) - open/close 9895 (time 1763080784.03 total 140.12 last 73.29) - open/close 10615 (time 1763080794.04 total 150.14 last 71.90) - open/close 11367 (time 1763080804.05 total 160.14 last 75.16) - open/close 12123 (time 1763080814.06 total 170.15 last 75.53) - open/close 12906 (time 1763080824.06 total 180.15 last 78.28) - open/close 13653 (time 1763080834.07 total 190.16 last 74.67) - open/close 14479 (time 1763080844.07 total 200.16 last 82.59) - open/close 15251 (time 1763080854.07 total 210.17 last 77.14) - open/close 16054 (time 1763080864.08 total 220.18 last 80.23) - open/close 16768 (time 1763080874.10 total 230.19 last 71.32) - open/close 17499 (time 1763080884.10 total 240.19 last 73.08) - open/close 18225 (time 1763080894.10 total 250.19 last 72.57) - open/close 18930 (time 1763080904.11 total 260.20 last 70.43) - open/close 19730 (time 1763080914.11 total 270.20 last 79.99) - open/close 20000 (time 1763080917.38 total 273.47 last 82.69) - open/close 20739 (time 1763080927.39 total 283.49 last 73.77) - open/close 21436 (time 1763080937.40 total 293.50 last 69.63) - open/close 22162 (time 1763080947.42 total 303.51 last 72.51) - open/close 22916 (time 1763080957.44 total 313.53 last 75.26) - open/close 23764 (time 1763080967.44 total 323.53 last 84.80) - open/close 24491 (time 1763080977.44 total 333.54 last 72.64) - open/close 25212 (time 1763080987.44 total 343.54 last 72.10) - open/close 25928 (time 1763080997.45 total 353.54 last 71.58) - open/close 26640 (time 1763081007.46 total 363.55 last 71.14) - open/close 27309 (time 1763081017.47 total 373.56 last 66.81) - open/close 28002 (time 1763081027.47 total 383.56 last 69.28) - open/close 28731 (time 1763081037.47 total 393.57 last 72.89) - open/close 29410 (time 1763081047.48 total 403.57 last 67.86) - open/close 30000 (time 1763081056.26 total 412.36 last 67.17) - open/close 30708 (time 1763081066.27 total 422.36 last 70.75) - open/close 31441 (time 1763081076.28 total 432.37 last 73.22) - open/close 32181 (time 1763081086.29 total 442.39 last 73.91) - open/close 32929 (time 1763081096.30 total 452.39 last 74.79) - open/close 33744 (time 1763081106.30 total 462.39 last 81.46) - open/close 34430 (time 1763081116.31 total 472.40 last 68.55) - open/close 35160 (time 1763081126.31 total 482.40 last 72.99) - open/close 35779 (time 1763081136.31 total 492.40 last 61.89) - open/close 36361 (time 1763081146.32 total 502.41 last 58.13) - open/close 36937 (time 1763081156.35 total 512.44 last 57.45) - open/close 37501 (time 1763081166.36 total 522.46 last 56.31) - open/close 38122 (time 1763081176.37 total 532.46 last 62.06) - open/close 38797 (time 1763081186.38 total 542.47 last 67.43) - open/close 39505 (time 1763081196.39 total 552.48 last 70.73) - open/close 40000 (time 1763081203.68 total 559.77 last 67.93) - open/close 40734 (time 1763081213.68 total 569.78 last 73.36) - open/close 41537 (time 1763081223.70 total 579.79 last 80.15) - open/close 42260 (time 1763081233.71 total 589.80 last 72.27) - open/close 42916 (time 1763081243.71 total 599.80 last 65.60) - open/close 43604 (time 1763081253.72 total 609.81 last 68.71) - open/close 44313 (time 1763081263.72 total 619.81 last 70.90) - open/close 44998 (time 1763081273.73 total 629.82 last 68.44) - open/close 45727 (time 1763081283.73 total 639.82 last 72.87) - open/close 46305 (time 1763081293.76 total 649.85 last 57.67) - open/close 47022 (time 1763081303.77 total 659.87 last 71.58) - open/close 47676 (time 1763081313.78 total 669.87 last 65.35) - open/close 48332 (time 1763081323.79 total 679.88 last 65.54) - open/close 49103 (time 1763081333.80 total 689.90 last 77.00) - open/close 49765 (time 1763081343.81 total 699.90 last 66.18) - open/close 50000 (time 1763081347.20 total 703.29 last 69.25) - open/close 50811 (time 1763081357.20 total 713.29 last 81.09) - open/close 51492 (time 1763081367.20 total 723.29 last 68.10) - open/close 52204 (time 1763081377.24 total 733.33 last 70.93) - open/close 52895 (time 1763081387.24 total 743.34 last 69.05) - open/close 53573 (time 1763081397.25 total 753.34 last 67.75) - open/close 54325 (time 1763081407.26 total 763.36 last 75.11) - open/close 54971 (time 1763081417.26 total 773.36 last 64.60) - open/close 55604 (time 1763081427.28 total 783.37 last 63.23) - open/close 56266 (time 1763081437.28 total 793.37 last 66.20) - open/close 57003 (time 1763081447.30 total 803.39 last 73.55) - open/close 57677 (time 1763081457.31 total 813.40 last 67.33) - open/close 58391 (time 1763081467.31 total 823.41 last 71.36) - open/close 59185 (time 1763081477.32 total 833.41 last 79.38) - open/close 60000 (time 1763081486.80 total 842.90 last 85.90) - open/close 60778 (time 1763081496.81 total 852.90 last 77.73) - open/close 61491 (time 1763081506.82 total 862.92 last 71.22) - open/close 62293 (time 1763081516.84 total 872.93 last 80.06) - open/close 62969 (time 1763081526.85 total 882.94 last 67.55) - open/close 63595 (time 1763081536.85 total 892.95 last 62.56) - open/close 64326 (time 1763081546.86 total 902.96 last 73.04) total: 64768 open/close in 908.37 seconds: 71.30 ops/second - /unlink 728 (time 1763081565.06 total 10.01 last 72.73) - /unlink 1397 (time 1763081575.06 total 20.02 last 66.87) - /unlink 2072 (time 1763081585.06 total 30.02 last 67.48) - /unlink 2770 (time 1763081595.06 total 40.02 last 69.80) - /unlink 3437 (time 1763081605.08 total 50.03 last 66.60) - /unlink 4122 (time 1763081615.08 total 60.04 last 68.46) - /unlink 4758 (time 1763081625.10 total 70.05 last 63.51) - /unlink 5440 (time 1763081635.11 total 80.06 last 68.14) - /unlink 6129 (time 1763081645.11 total 90.07 last 68.86) - /unlink 6845 (time 1763081655.11 total 100.07 last 71.58) - /unlink 7538 (time 1763081665.12 total 110.08 last 69.24) - /unlink 8259 (time 1763081675.13 total 120.08 last 72.09) - /unlink 9003 (time 1763081685.13 total 130.09 last 74.34) - /unlink 9694 (time 1763081695.14 total 140.10 last 69.04) - /unlink 10000 (time 1763081699.53 total 144.48 last 69.73) - /unlink 10702 (time 1763081709.54 total 154.50 last 70.11) - /unlink 11357 (time 1763081719.55 total 164.50 last 65.48) - /unlink 12094 (time 1763081729.55 total 174.50 last 73.68) - /unlink 12956 (time 1763081739.55 total 184.51 last 86.15) - /unlink 13824 (time 1763081749.56 total 194.51 last 86.75) - /unlink 14614 (time 1763081759.57 total 204.52 last 78.92) - /unlink 15340 (time 1763081769.58 total 214.54 last 72.50) - /unlink 16071 (time 1763081779.59 total 224.55 last 73.02) - /unlink 16751 (time 1763081789.60 total 234.56 last 67.93) - /unlink 17422 (time 1763081799.62 total 244.57 last 67.02) - /unlink 18107 (time 1763081809.62 total 254.58 last 68.46) - /unlink 18756 (time 1763081819.63 total 264.58 last 64.88) - /unlink 19390 (time 1763081829.64 total 274.60 last 63.30) - /unlink 19997 (time 1763081839.65 total 284.60 last 60.66) - /unlink 20682 (time 1763081849.67 total 294.62 last 68.38) - /unlink 21365 (time 1763081859.67 total 304.62 last 68.29) - /unlink 22066 (time 1763081869.68 total 314.63 last 70.02) - /unlink 22792 (time 1763081879.69 total 324.65 last 72.49) - /unlink 23480 (time 1763081889.70 total 334.65 last 68.77) - /unlink 24207 (time 1763081899.71 total 344.67 last 72.59) - /unlink 24876 (time 1763081909.73 total 354.68 last 66.82) - /unlink 25580 (time 1763081919.73 total 364.68 last 70.39) - /unlink 26247 (time 1763081929.73 total 374.69 last 66.66) - /unlink 27002 (time 1763081939.75 total 384.70 last 75.38) - /unlink 27713 (time 1763081949.76 total 394.71 last 71.03) - /unlink 28419 (time 1763081959.77 total 404.72 last 70.54) - /unlink 29060 (time 1763081969.78 total 414.73 last 64.05) - /unlink 29678 (time 1763081979.78 total 424.73 last 61.79) - /unlink 30000 (time 1763081984.60 total 429.55 last 66.79) - /unlink 30719 (time 1763081994.60 total 439.55 last 71.89) - /unlink 31370 (time 1763082004.61 total 449.57 last 65.02) - /unlink 32014 (time 1763082014.63 total 459.58 last 64.31) - /unlink 32746 (time 1763082024.63 total 469.59 last 73.16) - /unlink 33501 (time 1763082034.63 total 479.59 last 75.48) - /unlink 34164 (time 1763082044.64 total 489.59 last 66.26) - /unlink 34783 (time 1763082054.64 total 499.60 last 61.88) - /unlink 35453 (time 1763082064.65 total 509.61 last 66.94) - /unlink 36142 (time 1763082074.66 total 519.61 last 68.89) - /unlink 36800 (time 1763082084.68 total 529.64 last 65.61) - /unlink 37461 (time 1763082094.70 total 539.65 last 66.02) - /unlink 38133 (time 1763082104.71 total 549.66 last 67.11) - /unlink 38859 (time 1763082114.73 total 559.69 last 72.43) - /unlink 39531 (time 1763082124.73 total 569.69 last 67.19) - /unlink 40000 (time 1763082131.15 total 576.11 last 73.08) - /unlink 40694 (time 1763082141.16 total 586.12 last 69.32) - /unlink 41378 (time 1763082151.17 total 596.12 last 68.38) - /unlink 42095 (time 1763082161.17 total 606.12 last 71.68) - /unlink 42848 (time 1763082171.19 total 616.14 last 75.16) - /unlink 43494 (time 1763082181.19 total 626.14 last 64.59) - /unlink 44165 (time 1763082191.20 total 636.15 last 67.05) - /unlink 44830 (time 1763082201.21 total 646.16 last 66.41) - /unlink 45517 (time 1763082211.21 total 656.17 last 68.67) - /unlink 46228 (time 1763082221.22 total 666.18 last 71.05) - /unlink 46935 (time 1763082231.23 total 676.18 last 70.66) - /unlink 47618 (time 1763082241.23 total 686.18 last 68.28) - /unlink 48325 (time 1763082251.25 total 696.20 last 70.57) - /unlink 49062 (time 1763082261.25 total 706.21 last 73.65) - /unlink 49570 (time 1763082271.26 total 716.21 last 50.76) - /unlink 50000 (time 1763082278.96 total 723.92 last 55.82) - /unlink 50636 (time 1763082288.97 total 733.92 last 63.56) - /unlink 51247 (time 1763082298.99 total 743.95 last 60.96) - /unlink 51909 (time 1763082309.00 total 753.96 last 66.14) - /unlink 52554 (time 1763082319.01 total 763.96 last 64.47) - /unlink 53255 (time 1763082329.02 total 773.97 last 70.05) - /unlink 53957 (time 1763082339.02 total 783.98 last 70.13) - /unlink 54648 (time 1763082349.03 total 793.99 last 69.05) - /unlink 55330 (time 1763082359.04 total 803.99 last 68.17) - /unlink 56129 (time 1763082369.04 total 814.00 last 79.85) - /unlink 56835 (time 1763082379.05 total 824.00 last 70.55) - /unlink 57479 (time 1763082389.06 total 834.02 last 64.31) - /unlink 58154 (time 1763082399.07 total 844.02 last 67.47) - /unlink 58876 (time 1763082409.09 total 854.04 last 72.07) - /unlink 59580 (time 1763082419.10 total 864.06 last 70.27) - /unlink 60000 (time 1763082425.35 total 870.30 last 67.27) - /unlink 60752 (time 1763082435.36 total 880.31 last 75.14) - /unlink 61455 (time 1763082445.36 total 890.31 last 70.30) - /unlink 62090 (time 1763082455.37 total 900.32 last 63.41) - /unlink 62781 (time 1763082465.37 total 910.33 last 69.07) - /unlink 63500 (time 1763082475.37 total 920.33 last 71.90) - /unlink 64108 (time 1763082485.38 total 930.33 last 60.78) total: 64768 /unlink in 940.16 seconds: 68.89 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 744 (time 1763082533.59 total 10.01 last 74.31) - open/close 1401 (time 1763082543.60 total 20.03 last 65.61) - open/close 2068 (time 1763082553.61 total 30.03 last 66.67) - open/close 2751 (time 1763082563.61 total 40.04 last 68.26) - open/close 3601 (time 1763082573.62 total 50.04 last 84.95) - open/close 4353 (time 1763082583.63 total 60.06 last 75.10) - open/close 5052 (time 1763082593.64 total 70.06 last 69.84) - open/close 5738 (time 1763082603.66 total 80.08 last 68.51) - open/close 6485 (time 1763082613.66 total 90.09 last 74.63) - open/close 7192 (time 1763082623.69 total 100.11 last 70.55) - open/close 7882 (time 1763082633.70 total 110.12 last 68.90) - open/close 8641 (time 1763082643.72 total 120.14 last 75.75) - open/close 9371 (time 1763082653.73 total 130.15 last 72.95) - open/close 10000 (time 1763082660.67 total 137.09 last 90.60) - open/close 11104 (time 1763082670.67 total 147.10 last 110.35) - open/close 11987 (time 1763082680.68 total 157.10 last 88.27) - open/close 12773 (time 1763082690.69 total 167.11 last 78.50) - open/close 13511 (time 1763082700.70 total 177.12 last 73.73) - open/close 14279 (time 1763082710.70 total 187.12 last 76.78) - open/close 15072 (time 1763082720.71 total 197.13 last 79.27) - open/close 15749 (time 1763082730.71 total 207.13 last 67.66) - open/close 16471 (time 1763082740.74 total 217.16 last 71.98) - open/close 17257 (time 1763082750.75 total 227.17 last 78.58) - open/close 18147 (time 1763082760.75 total 237.17 last 88.95) - open/close 18809 (time 1763082770.76 total 247.18 last 66.13) - open/close 19443 (time 1763082780.77 total 257.19 last 63.37) - open/close 20000 (time 1763082787.98 total 264.40 last 77.20) - open/close 20699 (time 1763082798.00 total 274.42 last 69.77) - open/close 21480 (time 1763082808.00 total 284.43 last 78.08) - open/close 22159 (time 1763082818.01 total 294.43 last 67.89) - open/close 22900 (time 1763082828.02 total 304.44 last 74.01) - open/close 23602 (time 1763082838.02 total 314.44 last 70.18) - open/close 24302 (time 1763082848.02 total 324.45 last 69.96) - open/close 24951 (time 1763082858.03 total 334.45 last 64.90) - open/close 25698 (time 1763082868.03 total 344.45 last 74.65) - open/close 26327 (time 1763082878.04 total 354.46 last 62.88) - open/close 27041 (time 1763082888.05 total 364.47 last 71.31) - open/close 27704 (time 1763082898.06 total 374.48 last 66.21) - open/close 28394 (time 1763082908.07 total 384.49 last 68.93) - open/close 29051 (time 1763082918.08 total 394.50 last 65.68) - open/close 29733 (time 1763082928.09 total 404.51 last 68.10) - open/close 30000 (time 1763082932.29 total 408.71 last 63.56) - open/close 30649 (time 1763082942.30 total 418.72 last 64.83) - open/close 31333 (time 1763082952.30 total 428.73 last 68.38) - open/close 32043 (time 1763082962.31 total 438.73 last 70.94) - open/close 32649 (time 1763082972.32 total 448.74 last 60.55) - open/close 33168 (time 1763082982.33 total 458.76 last 51.83) - open/close 33761 (time 1763082992.34 total 468.76 last 59.27) - open/close 34449 (time 1763083002.35 total 478.77 last 68.74) - open/close 35074 (time 1763083012.35 total 488.77 last 62.50) - open/close 35744 (time 1763083022.36 total 498.79 last 66.90) - open/close 36422 (time 1763083032.37 total 508.79 last 67.79) - open/close 37050 (time 1763083042.37 total 518.80 last 62.75) - open/close 37747 (time 1763083052.38 total 528.80 last 69.66) - open/close 38468 (time 1763083062.40 total 538.82 last 71.99) - open/close 39212 (time 1763083072.40 total 548.82 last 74.38) - open/close 39958 (time 1763083082.40 total 558.82 last 74.58) - open/close 40621 (time 1763083092.42 total 568.85 last 66.14) - open/close 41268 (time 1763083102.43 total 578.85 last 64.64) - open/close 41969 (time 1763083112.45 total 588.87 last 70.01) - open/close 42683 (time 1763083122.46 total 598.88 last 71.33) - open/close 43395 (time 1763083132.47 total 608.89 last 71.13) - open/close 44065 (time 1763083142.47 total 618.89 last 66.96) - open/close 44715 (time 1763083152.47 total 628.90 last 64.99) - open/close 45343 (time 1763083162.49 total 638.91 last 62.68) - open/close 45992 (time 1763083172.50 total 648.92 last 64.84) - open/close 46611 (time 1763083182.52 total 658.94 last 61.81) - open/close 47317 (time 1763083192.52 total 668.94 last 70.58) - open/close 47922 (time 1763083202.53 total 678.96 last 60.41) - open/close 48536 (time 1763083212.53 total 688.96 last 61.39) - open/close 49203 (time 1763083222.54 total 698.96 last 66.69) - open/close 49845 (time 1763083232.54 total 708.96 last 64.16) - open/close 50000 (time 1763083234.65 total 711.08 last 73.38) - open/close 50767 (time 1763083244.66 total 721.08 last 76.65) - open/close 51493 (time 1763083254.66 total 731.09 last 72.58) - open/close 52235 (time 1763083264.68 total 741.10 last 74.09) - open/close 53038 (time 1763083274.69 total 751.11 last 80.19) - open/close 53896 (time 1763083284.70 total 761.12 last 85.73) - open/close 54691 (time 1763083294.71 total 771.13 last 79.47) - open/close 55437 (time 1763083304.71 total 781.14 last 74.54) - open/close 56143 (time 1763083314.72 total 791.14 last 70.59) - open/close 56868 (time 1763083324.72 total 801.14 last 72.46) - open/close 57667 (time 1763083334.73 total 811.15 last 79.82) - open/close 58444 (time 1763083344.73 total 821.16 last 77.67) - open/close 59171 (time 1763083354.74 total 831.16 last 72.67) - open/close 59943 (time 1763083364.75 total 841.17 last 77.12) - open/close 60784 (time 1763083374.75 total 851.17 last 84.08) - open/close 61514 (time 1763083384.75 total 861.18 last 72.98) - open/close 62357 (time 1763083394.77 total 871.19 last 84.19) - open/close 63098 (time 1763083404.77 total 881.20 last 74.05) - open/close 63815 (time 1763083414.78 total 891.20 last 71.65) - open/close 64529 (time 1763083424.78 total 901.21 last 71.38) total: 64768 open/close in 904.63 seconds: 71.60 ops/second - /unlink 783 (time 1763083441.40 total 10.01 last 78.23) - /unlink 1536 (time 1763083451.41 total 20.02 last 75.24) - /unlink 2289 (time 1763083461.41 total 30.02 last 75.30) - /unlink 3030 (time 1763083471.41 total 40.02 last 74.08) - /unlink 3799 (time 1763083481.42 total 50.03 last 76.82) - /unlink 4599 (time 1763083491.43 total 60.04 last 79.94) - /unlink 5358 (time 1763083501.43 total 70.04 last 75.89) - /unlink 6245 (time 1763083511.43 total 80.04 last 88.70) - /unlink 7193 (time 1763083521.43 total 90.04 last 94.78) - /unlink 8119 (time 1763083531.43 total 100.05 last 92.55) - /unlink 8935 (time 1763083541.45 total 110.06 last 81.49) - /unlink 9754 (time 1763083551.45 total 120.06 last 81.89) - /unlink 10000 (time 1763083554.52 total 123.13 last 80.12) - /unlink 10797 (time 1763083564.53 total 133.14 last 79.66) - /unlink 11557 (time 1763083574.53 total 143.15 last 75.93) - /unlink 12401 (time 1763083584.54 total 153.15 last 84.35) - /unlink 13296 (time 1763083594.55 total 163.16 last 89.44) - /unlink 14173 (time 1763083604.55 total 173.16 last 87.68) - /unlink 15069 (time 1763083614.56 total 183.17 last 89.55) - /unlink 15875 (time 1763083624.57 total 193.18 last 80.51) - /unlink 16632 (time 1763083634.59 total 203.20 last 75.56) - /unlink 17437 (time 1763083644.59 total 213.20 last 80.49) - /unlink 18393 (time 1763083654.59 total 223.21 last 95.53) - /unlink 19470 (time 1763083664.60 total 233.21 last 107.68) - /unlink 20000 (time 1763083670.26 total 238.87 last 93.53) - /unlink 20961 (time 1763083680.26 total 248.87 last 96.09) - /unlink 21834 (time 1763083690.27 total 258.88 last 87.22) - /unlink 22692 (time 1763083700.28 total 268.89 last 85.77) - /unlink 23484 (time 1763083710.28 total 278.89 last 79.20) - /unlink 24587 (time 1763083720.28 total 288.89 last 110.25) - /unlink 25524 (time 1763083730.29 total 298.90 last 93.60) - /unlink 26366 (time 1763083740.30 total 308.91 last 84.18) - /unlink 27241 (time 1763083750.30 total 318.91 last 87.48) - /unlink 28072 (time 1763083760.31 total 328.92 last 83.01) - /unlink 28896 (time 1763083770.31 total 338.92 last 82.39) - /unlink 29693 (time 1763083780.32 total 348.93 last 79.64) - /unlink 30000 (time 1763083783.61 total 352.22 last 93.32) - /unlink 30910 (time 1763083793.61 total 362.22 last 90.97) - /unlink 31710 (time 1763083803.61 total 372.22 last 79.99) - /unlink 32564 (time 1763083813.61 total 382.22 last 85.38) - /unlink 33652 (time 1763083823.62 total 392.23 last 108.75) - /unlink 34721 (time 1763083833.63 total 402.24 last 106.80) - /unlink 35644 (time 1763083843.63 total 412.24 last 92.29) - /unlink 36585 (time 1763083853.64 total 422.25 last 94.03) - /unlink 37477 (time 1763083863.64 total 432.25 last 89.13) - /unlink 38371 (time 1763083873.64 total 442.25 last 89.39) - /unlink 39221 (time 1763083883.65 total 452.26 last 84.95) - /unlink 40000 (time 1763083891.63 total 460.24 last 97.56) - /unlink 40911 (time 1763083901.64 total 470.25 last 91.07) - /unlink 41834 (time 1763083911.64 total 480.25 last 92.29) - /unlink 42688 (time 1763083921.65 total 490.26 last 85.33) - /unlink 43533 (time 1763083931.65 total 500.26 last 84.47) - /unlink 44394 (time 1763083941.65 total 510.26 last 86.08) - /unlink 45420 (time 1763083951.66 total 520.27 last 102.54) - /unlink 46512 (time 1763083961.66 total 530.27 last 109.15) - /unlink 47411 (time 1763083971.67 total 540.28 last 89.86) - /unlink 48387 (time 1763083981.67 total 550.28 last 97.59) - /unlink 49345 (time 1763083991.68 total 560.29 last 95.72) - /unlink 50000 (time 1763083999.27 total 567.88 last 86.33) - /unlink 50835 (time 1763084009.28 total 577.89 last 83.39) - /unlink 51713 (time 1763084019.28 total 587.89 last 87.76) - /unlink 52516 (time 1763084029.29 total 597.90 last 80.27) - /unlink 53378 (time 1763084039.30 total 607.91 last 86.12) - /unlink 54557 (time 1763084049.30 total 617.91 last 117.88) - /unlink 55680 (time 1763084059.30 total 627.91 last 112.26) - /unlink 56853 (time 1763084069.30 total 637.92 last 117.25) - /unlink 57824 (time 1763084079.31 total 647.92 last 97.08) - /unlink 58885 (time 1763084089.34 total 657.95 last 105.79) - /unlink 59982 (time 1763084099.34 total 667.95 last 109.67) - /unlink 61069 (time 1763084109.34 total 677.95 last 108.65) - /unlink 62253 (time 1763084119.35 total 687.96 last 118.34) - /unlink 63447 (time 1763084129.36 total 697.97 last 119.32) - /unlink 64742 (time 1763084139.36 total 707.97 last 129.43) total: 64768 /unlink in 708.17 seconds: 91.46 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1374 (time 1763084173.82 total 10.00 last 137.37) - open/close 2714 (time 1763084183.82 total 20.00 last 133.97) - open/close 4320 (time 1763084193.83 total 30.01 last 160.46) - open/close 6119 (time 1763084203.83 total 40.01 last 179.89) - open/close 7482 (time 1763084213.84 total 50.02 last 136.20) - open/close 8673 (time 1763084223.86 total 60.04 last 118.92) - open/close 9960 (time 1763084233.86 total 70.04 last 128.63) - open/close 11229 (time 1763084243.87 total 80.05 last 126.83) - open/close 12725 (time 1763084253.87 total 90.05 last 149.57) - open/close 14380 (time 1763084263.87 total 100.05 last 165.47) - open/close 15998 (time 1763084273.87 total 110.06 last 161.74) - open/close 17860 (time 1763084283.88 total 120.06 last 186.15) - open/close 19497 (time 1763084293.88 total 130.06 last 163.66) - open/close 20000 (time 1763084296.79 total 132.97 last 172.87) - open/close 21555 (time 1763084306.79 total 142.97 last 155.49) - open/close 23291 (time 1763084316.79 total 152.97 last 173.55) - open/close 24717 (time 1763084326.79 total 162.97 last 142.58) - open/close 26360 (time 1763084336.80 total 172.98 last 164.24) - open/close 27946 (time 1763084346.80 total 182.98 last 158.49) - open/close 29331 (time 1763084356.81 total 192.99 last 138.44) - open/close 30000 (time 1763084361.54 total 197.72 last 141.36) - open/close 31131 (time 1763084371.55 total 207.73 last 112.99) - open/close 32365 (time 1763084381.56 total 217.74 last 123.32) - open/close 33759 (time 1763084391.56 total 227.74 last 139.37) - open/close 35212 (time 1763084401.56 total 237.74 last 145.27) - open/close 36631 (time 1763084411.56 total 247.74 last 141.90) - open/close 38024 (time 1763084421.56 total 257.74 last 139.28) - open/close 39564 (time 1763084431.56 total 267.75 last 153.97) - open/close 40000 (time 1763084434.28 total 270.46 last 160.71) - open/close 41685 (time 1763084444.28 total 280.46 last 168.45) - open/close 43453 (time 1763084454.28 total 290.46 last 176.76) - open/close 45356 (time 1763084464.29 total 300.47 last 190.22) - open/close 46974 (time 1763084474.29 total 310.47 last 161.72) - open/close 48727 (time 1763084484.29 total 320.47 last 175.30) - open/close 50000 (time 1763084493.36 total 329.54 last 140.39) - open/close 51486 (time 1763084503.37 total 339.55 last 148.51) - open/close 52845 (time 1763084513.37 total 349.55 last 135.80) - open/close 54157 (time 1763084523.38 total 359.56 last 131.15) - open/close 55473 (time 1763084533.38 total 369.56 last 131.58) - open/close 57223 (time 1763084543.38 total 379.56 last 174.94) - open/close 58839 (time 1763084553.39 total 389.57 last 161.53) - open/close 60000 (time 1763084561.18 total 397.36 last 149.05) - open/close 61456 (time 1763084571.18 total 407.36 last 145.59) - open/close 63087 (time 1763084581.18 total 417.36 last 163.01) - open/close 64619 (time 1763084591.18 total 427.36 last 153.20) total: 64768 open/close in 428.50 seconds: 151.15 ops/second - /unlink 3139 (time 1763084604.30 total 10.00 last 313.77) - /unlink 6503 (time 1763084614.31 total 20.01 last 336.34) - /unlink 9767 (time 1763084624.31 total 30.01 last 326.36) - /unlink 13211 (time 1763084634.31 total 40.01 last 344.40) - /unlink 16415 (time 1763084644.31 total 50.01 last 320.34) - /unlink 19500 (time 1763084654.31 total 60.01 last 308.45) - /unlink 20000 (time 1763084656.31 total 62.01 last 249.83) - /unlink 22936 (time 1763084666.31 total 72.01 last 293.54) - /unlink 25643 (time 1763084676.32 total 82.02 last 270.62) - /unlink 28564 (time 1763084686.32 total 92.02 last 292.07) - /unlink 30000 (time 1763084691.64 total 97.33 last 270.08) - /unlink 32829 (time 1763084701.64 total 107.34 last 282.83) - /unlink 36218 (time 1763084711.64 total 117.34 last 338.88) - /unlink 39695 (time 1763084721.64 total 127.34 last 347.65) - /unlink 42767 (time 1763084731.64 total 137.34 last 307.15) - /unlink 45747 (time 1763084741.64 total 147.34 last 297.93) - /unlink 48962 (time 1763084751.65 total 157.35 last 321.40) - /unlink 50000 (time 1763084755.19 total 160.89 last 292.90) - /unlink 52916 (time 1763084765.19 total 170.89 last 291.50) - /unlink 55871 (time 1763084775.20 total 180.89 last 295.47) - /unlink 58991 (time 1763084785.20 total 190.90 last 311.92) - /unlink 60000 (time 1763084788.87 total 194.57 last 274.65) - /unlink 62472 (time 1763084798.87 total 204.57 last 247.19) total: 64768 /unlink in 212.80 seconds: 304.36 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2002 (time 1763084836.09 total 10.01 last 200.10) - open/close 3657 (time 1763084846.09 total 20.01 last 165.40) - open/close 5320 (time 1763084856.10 total 30.01 last 166.25) - open/close 6908 (time 1763084866.10 total 40.02 last 158.78) - open/close 8610 (time 1763084876.11 total 50.02 last 170.11) - open/close 10000 (time 1763084884.16 total 58.07 last 172.59) - open/close 11950 (time 1763084894.16 total 68.08 last 194.97) - open/close 14128 (time 1763084904.16 total 78.08 last 217.76) - open/close 16388 (time 1763084914.17 total 88.08 last 225.93) - open/close 18597 (time 1763084924.17 total 98.08 last 220.82) - open/close 20000 (time 1763084930.29 total 104.20 last 229.26) - open/close 22311 (time 1763084940.29 total 114.21 last 231.06) - open/close 25156 (time 1763084950.29 total 124.21 last 284.41) - open/close 27680 (time 1763084960.29 total 134.21 last 252.40) - open/close 30000 (time 1763084967.99 total 141.91 last 301.44) - open/close 32965 (time 1763084977.99 total 151.91 last 296.45) - open/close 35887 (time 1763084987.99 total 161.91 last 292.19) - open/close 39053 (time 1763084998.00 total 171.91 last 316.49) - open/close 40000 (time 1763085001.44 total 175.36 last 274.71) - open/close 43914 (time 1763085011.44 total 185.36 last 391.39) - open/close 47436 (time 1763085021.44 total 195.36 last 352.15) - open/close 50000 (time 1763085029.86 total 203.78 last 304.63) - open/close 52563 (time 1763085039.86 total 213.78 last 256.29) - open/close 55470 (time 1763085049.86 total 223.78 last 290.62) - open/close 58363 (time 1763085059.87 total 233.78 last 289.25) - open/close 60000 (time 1763085064.88 total 238.79 last 326.70) - open/close 63717 (time 1763085074.88 total 248.79 last 371.63) total: 64768 open/close in 252.43 seconds: 256.58 ops/second - /unlink 5118 (time 1763085090.14 total 10.00 last 511.74) - /unlink 10000 (time 1763085098.26 total 18.12 last 601.52) - /unlink 16185 (time 1763085108.26 total 28.12 last 618.49) - /unlink 20000 (time 1763085114.23 total 34.09 last 639.06) - /unlink 24801 (time 1763085124.23 total 44.09 last 480.05) - /unlink 30000 (time 1763085133.46 total 53.32 last 563.23) - /unlink 36541 (time 1763085143.46 total 63.32 last 654.04) - /unlink 40000 (time 1763085148.50 total 68.36 last 686.14) - /unlink 46820 (time 1763085158.50 total 78.36 last 681.99) - /unlink 50000 (time 1763085164.96 total 84.82 last 492.41) - /unlink 55061 (time 1763085174.96 total 94.82 last 506.05) - /unlink 60000 (time 1763085183.39 total 103.25 last 585.79) total: 64768 /unlink in 111.75 seconds: 579.59 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3539 (time 1763085221.61 total 10.00 last 353.87) - open/close 7655 (time 1763085231.62 total 20.00 last 411.50) - open/close 10000 (time 1763085238.32 total 26.71 last 349.77) - open/close 13911 (time 1763085248.32 total 36.71 last 391.08) - open/close 18352 (time 1763085258.32 total 46.71 last 444.03) - open/close 20000 (time 1763085262.83 total 51.22 last 365.36) - open/close 24045 (time 1763085272.84 total 61.22 last 404.39) - open/close 28256 (time 1763085282.84 total 71.22 last 421.03) - open/close 30000 (time 1763085287.32 total 75.71 last 388.98) - open/close 34751 (time 1763085297.32 total 85.71 last 475.03) - open/close 39447 (time 1763085307.32 total 95.71 last 469.54) - open/close 44120 (time 1763085317.32 total 105.71 last 467.25) - open/close 48760 (time 1763085327.32 total 115.71 last 463.99) - open/close 50000 (time 1763085329.89 total 118.28 last 483.01) - open/close 54853 (time 1763085339.89 total 128.28 last 485.23) - open/close 59345 (time 1763085349.89 total 138.28 last 449.14) - open/close 64049 (time 1763085359.89 total 148.28 last 470.40) total: 64768 open/close in 149.84 seconds: 432.25 ops/second - /unlink 7850 (time 1763085372.73 total 10.00 last 784.96) - /unlink 10000 (time 1763085375.10 total 12.37 last 906.54) - /unlink 17506 (time 1763085385.10 total 22.37 last 750.59) - /unlink 20000 (time 1763085389.01 total 26.28 last 637.90) - /unlink 27541 (time 1763085399.01 total 36.28 last 754.03) - /unlink 30000 (time 1763085402.35 total 39.63 last 735.72) - /unlink 37634 (time 1763085412.35 total 49.63 last 763.36) - /unlink 40000 (time 1763085415.06 total 52.33 last 874.56) - /unlink 47434 (time 1763085425.06 total 62.33 last 743.40) - /unlink 50000 (time 1763085428.46 total 65.73 last 753.99) - /unlink 57320 (time 1763085438.46 total 75.74 last 731.85) - /unlink 60000 (time 1763085441.83 total 79.11 last 795.38) total: 64768 /unlink in 84.81 seconds: 763.66 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg220-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg220-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg220-server stop mds service on oleg220-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg220-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing unload_modules_local oleg220-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg220-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg220-client: oleg220-client: ssh exited with exit code 2 pdsh@oleg220-client: oleg220-server: ssh exited with exit code 2 pdsh@oleg220-client: oleg220-client: ssh exited with exit code 2 pdsh@oleg220-client: oleg220-server: ssh exited with exit code 2