== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 01:41:06 (1769755266) Stopping clients: oleg110-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg110-client.virtnet /mnt/lustre2 (opts:-f) oleg110-server: oleg110-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg110-server' oleg110-server: oleg110-server.virtnet: executing load_modules_local oleg110-server: Loading modules from /home/green/git/lustre-release/lustre oleg110-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg110-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg110-server: detected 4 online CPUs by sysfs oleg110-server: MODOPTS_LIBCFS= oleg110-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg110-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg110-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 272 79720 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 65808 302 65506 1% /mnt/lustre[OST:0] filesystem_summary: 65778 272 65506 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 533 (time 1769755389.77 total 10.00 last 53.28) - open/close 1133 (time 1769755399.78 total 20.02 last 59.93) - open/close 1713 (time 1769755409.81 total 30.04 last 57.85) - open/close 2358 (time 1769755419.81 total 40.05 last 64.46) - open/close 3095 (time 1769755429.82 total 50.05 last 73.65) - open/close 3746 (time 1769755439.83 total 60.06 last 65.03) - open/close 4422 (time 1769755449.84 total 70.07 last 67.56) - open/close 5037 (time 1769755459.85 total 80.08 last 61.45) - open/close 5629 (time 1769755469.86 total 90.09 last 59.14) - open/close 6200 (time 1769755479.88 total 100.11 last 57.00) - open/close 6779 (time 1769755489.89 total 110.12 last 57.81) - open/close 7484 (time 1769755499.90 total 120.13 last 70.45) - open/close 8001 (time 1769755509.90 total 130.14 last 51.68) - open/close 8658 (time 1769755519.91 total 140.14 last 65.69) - open/close 9383 (time 1769755529.92 total 150.15 last 72.41) - open/close 10000 (time 1769755539.29 total 159.53 last 65.82) - open/close 10731 (time 1769755549.29 total 169.53 last 73.08) - open/close 11459 (time 1769755559.31 total 179.54 last 72.72) - open/close 12153 (time 1769755569.33 total 189.56 last 69.22) - open/close 12846 (time 1769755579.34 total 199.57 last 69.27) - open/close 13589 (time 1769755589.35 total 209.58 last 74.22) - open/close 14289 (time 1769755599.36 total 219.60 last 69.88) - open/close 15057 (time 1769755609.37 total 229.60 last 76.77) - open/close 15813 (time 1769755619.37 total 239.60 last 75.58) - open/close 16514 (time 1769755629.37 total 249.61 last 70.07) - open/close 17143 (time 1769755639.38 total 259.61 last 62.86) - open/close 17839 (time 1769755649.41 total 269.64 last 69.43) - open/close 18462 (time 1769755659.41 total 279.64 last 62.26) - open/close 19168 (time 1769755669.42 total 289.65 last 70.56) - open/close 19884 (time 1769755679.43 total 299.66 last 71.50) - open/close 20513 (time 1769755689.45 total 309.68 last 62.80) - open/close 21228 (time 1769755699.45 total 319.68 last 71.48) - open/close 22031 (time 1769755709.45 total 329.69 last 80.27) - open/close 22753 (time 1769755719.48 total 339.71 last 71.99) - open/close 23482 (time 1769755729.48 total 349.72 last 72.89) - open/close 24078 (time 1769755739.50 total 359.73 last 59.52) - open/close 24692 (time 1769755749.50 total 369.74 last 61.36) - open/close 25417 (time 1769755759.51 total 379.74 last 72.43) - open/close 26117 (time 1769755769.53 total 389.76 last 69.89) - open/close 26872 (time 1769755779.53 total 399.76 last 75.48) - open/close 27698 (time 1769755789.53 total 409.76 last 82.60) - open/close 28390 (time 1769755799.55 total 419.78 last 69.10) - open/close 29033 (time 1769755809.56 total 429.79 last 64.23) - open/close 29793 (time 1769755819.57 total 439.80 last 75.92) - open/close 30000 (time 1769755823.02 total 443.25 last 59.94) - open/close 30747 (time 1769755833.02 total 453.26 last 74.68) - open/close 31420 (time 1769755843.03 total 463.26 last 67.26) - open/close 32135 (time 1769755853.04 total 473.27 last 71.43) - open/close 32833 (time 1769755863.05 total 483.28 last 69.75) - open/close 33536 (time 1769755873.05 total 493.28 last 70.25) - open/close 34201 (time 1769755883.06 total 503.29 last 66.47) - open/close 34909 (time 1769755893.06 total 513.29 last 70.79) - open/close 35714 (time 1769755903.06 total 523.29 last 80.49) - open/close 36448 (time 1769755913.07 total 533.31 last 73.29) - open/close 37146 (time 1769755923.08 total 543.31 last 69.78) - open/close 37870 (time 1769755933.08 total 553.31 last 72.40) - open/close 38601 (time 1769755943.09 total 563.32 last 73.02) - open/close 39376 (time 1769755953.10 total 573.34 last 77.39) - open/close 40000 (time 1769755960.83 total 581.06 last 80.74) - open/close 40671 (time 1769755970.84 total 591.07 last 67.03) - open/close 41405 (time 1769755980.85 total 601.08 last 73.36) - open/close 42121 (time 1769755990.85 total 611.08 last 71.57) - open/close 42722 (time 1769756000.87 total 621.10 last 60.00) - open/close 43405 (time 1769756010.87 total 631.11 last 68.26) - open/close 44133 (time 1769756020.88 total 641.11 last 72.76) - open/close 44751 (time 1769756030.91 total 651.15 last 61.59) - open/close 45348 (time 1769756040.93 total 661.16 last 59.61) - open/close 45982 (time 1769756050.94 total 671.17 last 63.35) - open/close 46615 (time 1769756060.94 total 681.17 last 63.29) - open/close 47332 (time 1769756070.95 total 691.18 last 71.65) - open/close 47983 (time 1769756080.96 total 701.19 last 65.03) - open/close 48541 (time 1769756090.96 total 711.19 last 55.77) - open/close 49150 (time 1769756100.96 total 721.20 last 60.89) - open/close 49734 (time 1769756110.96 total 731.20 last 58.40) - open/close 50000 (time 1769756114.91 total 735.14 last 67.40) - open/close 50605 (time 1769756124.92 total 745.15 last 60.46) - open/close 51235 (time 1769756134.93 total 755.16 last 62.93) - open/close 51875 (time 1769756144.93 total 765.16 last 63.99) - open/close 52477 (time 1769756154.96 total 775.19 last 60.02) - open/close 53104 (time 1769756164.96 total 785.19 last 62.69) - open/close 53717 (time 1769756174.98 total 795.21 last 61.20) - open/close 54489 (time 1769756184.99 total 805.22 last 77.14) - open/close 55139 (time 1769756195.00 total 815.23 last 64.90) - open/close 55840 (time 1769756205.01 total 825.24 last 70.05) - open/close 56386 (time 1769756215.02 total 835.26 last 54.51) - open/close 56874 (time 1769756225.05 total 845.28 last 48.68) - open/close 57484 (time 1769756235.06 total 855.30 last 60.92) - open/close 58210 (time 1769756245.07 total 865.30 last 72.58) - open/close 59005 (time 1769756255.07 total 875.30 last 79.48) - open/close 59656 (time 1769756265.08 total 885.31 last 65.03) - open/close 60000 (time 1769756269.91 total 890.14 last 71.21) - open/close 60579 (time 1769756279.91 total 900.15 last 57.88) - open/close 61159 (time 1769756289.91 total 910.15 last 58.00) - open/close 61882 (time 1769756299.92 total 920.15 last 72.27) - open/close 62727 (time 1769756309.93 total 930.16 last 84.42) - open/close 63339 (time 1769756319.94 total 940.17 last 61.10) - open/close 64060 (time 1769756329.95 total 950.18 last 72.04) - open/close 64758 (time 1769756339.95 total 960.19 last 69.77) total: 64768 open/close in 960.29 seconds: 67.45 ops/second - /unlink 663 (time 1769756355.16 total 10.01 last 66.21) - /unlink 1299 (time 1769756365.16 total 20.02 last 63.59) - /unlink 1928 (time 1769756375.18 total 30.03 last 62.80) - /unlink 2442 (time 1769756385.19 total 40.05 last 51.32) - /unlink 2878 (time 1769756395.20 total 50.06 last 43.56) - /unlink 3425 (time 1769756405.21 total 60.06 last 54.66) - /unlink 3979 (time 1769756415.21 total 70.07 last 55.38) - /unlink 4748 (time 1769756425.23 total 80.09 last 76.74) - /unlink 5357 (time 1769756435.24 total 90.09 last 60.89) - /unlink 5885 (time 1769756445.24 total 100.09 last 52.79) - /unlink 6449 (time 1769756455.25 total 110.10 last 56.35) - /unlink 7059 (time 1769756465.25 total 120.10 last 61.00) - /unlink 7649 (time 1769756475.26 total 130.11 last 58.96) - /unlink 8309 (time 1769756485.26 total 140.11 last 65.97) - /unlink 8927 (time 1769756495.29 total 150.15 last 61.61) - /unlink 9496 (time 1769756505.31 total 160.16 last 56.82) - /unlink 10000 (time 1769756514.27 total 169.13 last 56.19) - /unlink 10687 (time 1769756524.29 total 179.14 last 68.63) - /unlink 11299 (time 1769756534.29 total 189.14 last 61.18) - /unlink 11948 (time 1769756544.30 total 199.16 last 64.81) - /unlink 12559 (time 1769756554.30 total 209.16 last 61.09) - /unlink 13150 (time 1769756564.32 total 219.17 last 59.01) - /unlink 13636 (time 1769756574.33 total 229.18 last 48.56) - /unlink 14205 (time 1769756584.35 total 239.20 last 56.79) - /unlink 14742 (time 1769756594.36 total 249.22 last 53.63) - /unlink 15394 (time 1769756604.37 total 259.22 last 65.17) - /unlink 15940 (time 1769756614.37 total 269.23 last 54.55) - /unlink 16545 (time 1769756624.38 total 279.24 last 60.45) - /unlink 17169 (time 1769756634.39 total 289.25 last 62.35) - /unlink 17833 (time 1769756644.40 total 299.25 last 66.35) - /unlink 18540 (time 1769756654.41 total 309.26 last 70.63) - /unlink 19176 (time 1769756664.41 total 319.27 last 63.57) - /unlink 19806 (time 1769756674.42 total 329.27 last 62.96) - /unlink 20000 (time 1769756677.38 total 332.23 last 65.54) - /unlink 20614 (time 1769756687.41 total 342.27 last 61.19) - /unlink 21243 (time 1769756697.41 total 352.27 last 62.89) - /unlink 21806 (time 1769756707.43 total 362.29 last 56.21) - /unlink 22397 (time 1769756717.45 total 372.30 last 59.02) - /unlink 22941 (time 1769756727.46 total 382.32 last 54.31) - /unlink 23493 (time 1769756737.47 total 392.33 last 55.14) - /unlink 24150 (time 1769756747.50 total 402.35 last 65.54) - /unlink 24706 (time 1769756757.51 total 412.36 last 55.56) - /unlink 25276 (time 1769756767.51 total 422.37 last 56.96) - /unlink 25773 (time 1769756777.52 total 432.37 last 49.68) - /unlink 26436 (time 1769756787.53 total 442.38 last 66.23) - /unlink 26946 (time 1769756797.54 total 452.40 last 50.91) - /unlink 27512 (time 1769756807.56 total 462.41 last 56.52) - /unlink 28199 (time 1769756817.56 total 472.42 last 68.67) - /unlink 28833 (time 1769756827.57 total 482.43 last 63.34) - /unlink 29437 (time 1769756837.58 total 492.43 last 60.38) - /unlink 29952 (time 1769756847.58 total 502.44 last 51.47) - /unlink 30527 (time 1769756857.60 total 512.45 last 57.40) - /unlink 31069 (time 1769756867.63 total 522.48 last 54.03) - /unlink 31605 (time 1769756877.65 total 532.51 last 53.48) - /unlink 32266 (time 1769756887.67 total 542.52 last 66.02) - /unlink 32899 (time 1769756897.67 total 552.52 last 63.28) - /unlink 33431 (time 1769756907.68 total 562.54 last 53.13) - /unlink 33990 (time 1769756917.69 total 572.55 last 55.83) - /unlink 34561 (time 1769756927.71 total 582.56 last 57.04) - /unlink 35124 (time 1769756937.71 total 592.56 last 56.29) - /unlink 35760 (time 1769756947.71 total 602.57 last 63.56) - /unlink 36322 (time 1769756957.73 total 612.59 last 56.08) - /unlink 36868 (time 1769756967.75 total 622.60 last 54.53) - /unlink 37523 (time 1769756977.76 total 632.62 last 65.41) - /unlink 38102 (time 1769756987.77 total 642.62 last 57.86) - /unlink 38681 (time 1769756997.79 total 652.64 last 57.80) - /unlink 39329 (time 1769757007.79 total 662.64 last 64.78) - /unlink 39882 (time 1769757017.80 total 672.65 last 55.26) - /unlink 40000 (time 1769757020.23 total 675.08 last 48.58) - /unlink 40613 (time 1769757030.23 total 685.08 last 61.28) - /unlink 41206 (time 1769757040.26 total 695.12 last 59.10) - /unlink 41781 (time 1769757050.29 total 705.15 last 57.33) - /unlink 42409 (time 1769757060.30 total 715.16 last 62.73) - /unlink 43127 (time 1769757070.31 total 725.16 last 71.75) - /unlink 43673 (time 1769757080.31 total 735.16 last 54.59) - /unlink 44354 (time 1769757090.31 total 745.17 last 68.09) - /unlink 45036 (time 1769757100.32 total 755.18 last 68.12) - /unlink 45646 (time 1769757110.32 total 765.18 last 61.00) - /unlink 46228 (time 1769757120.33 total 775.18 last 58.17) - /unlink 46777 (time 1769757130.34 total 785.19 last 54.86) - /unlink 47358 (time 1769757140.35 total 795.20 last 58.03) - /unlink 47936 (time 1769757150.35 total 805.20 last 57.79) - /unlink 48588 (time 1769757160.36 total 815.22 last 65.13) - /unlink 49189 (time 1769757170.39 total 825.24 last 59.96) - /unlink 49774 (time 1769757180.39 total 835.24 last 58.48) - /unlink 50000 (time 1769757184.49 total 839.35 last 55.07) - /unlink 50447 (time 1769757194.53 total 849.38 last 44.54) - /unlink 50999 (time 1769757204.53 total 859.39 last 55.19) - /unlink 51640 (time 1769757214.53 total 869.39 last 64.08) - /unlink 52384 (time 1769757224.55 total 879.40 last 74.29) - /unlink 53048 (time 1769757234.56 total 889.41 last 66.33) - /unlink 53752 (time 1769757244.57 total 899.42 last 70.33) - /unlink 54383 (time 1769757254.57 total 909.42 last 63.10) - /unlink 54969 (time 1769757264.57 total 919.42 last 58.60) - /unlink 55676 (time 1769757274.57 total 929.42 last 70.70) - /unlink 56342 (time 1769757284.57 total 939.43 last 66.56) - /unlink 57018 (time 1769757294.58 total 949.43 last 67.59) - /unlink 57556 (time 1769757304.58 total 959.43 last 53.77) - /unlink 58181 (time 1769757314.60 total 969.46 last 62.37) - /unlink 58836 (time 1769757324.61 total 979.47 last 65.44) - /unlink 59532 (time 1769757334.61 total 989.47 last 69.60) - /unlink 60000 (time 1769757342.12 total 996.98 last 62.30) - /unlink 60779 (time 1769757352.13 total 1006.99 last 77.83) - /unlink 61541 (time 1769757362.15 total 1017.00 last 76.09) - /unlink 62237 (time 1769757372.16 total 1027.01 last 69.54) - /unlink 62928 (time 1769757382.16 total 1037.01 last 69.09) - /unlink 63576 (time 1769757392.18 total 1047.03 last 64.68) - /unlink 64215 (time 1769757402.18 total 1057.03 last 63.87) total: 64768 /unlink in 1065.44 seconds: 60.79 ops/second Waiting for MDT destroys to complete oleg110-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 - open/close 607 (time 1769757438.04 total 10.00 last 60.69) - open/close 1307 (time 1769757448.04 total 20.00 last 69.98) - open/close 1885 (time 1769757458.06 total 30.02 last 57.68) - open/close 2572 (time 1769757468.07 total 40.03 last 68.68) - open/close 3258 (time 1769757478.09 total 50.05 last 68.46) - open/close 3877 (time 1769757488.09 total 60.05 last 61.88) - open/close 4540 (time 1769757498.10 total 70.06 last 66.22) - open/close 5233 (time 1769757508.13 total 80.09 last 69.10) - open/close 5825 (time 1769757518.14 total 90.10 last 59.16) - open/close 6467 (time 1769757528.15 total 100.11 last 64.15) - open/close 7200 (time 1769757538.15 total 110.11 last 73.30) - open/close 7911 (time 1769757548.15 total 120.11 last 71.09) - open/close 8489 (time 1769757558.15 total 130.11 last 57.79) - open/close 9111 (time 1769757568.16 total 140.12 last 62.14) - open/close 9707 (time 1769757578.16 total 150.13 last 59.57) - open/close 10000 (time 1769757582.68 total 154.64 last 64.88) - open/close 10734 (time 1769757592.70 total 164.66 last 73.29) - open/close 11380 (time 1769757602.71 total 174.67 last 64.53) - open/close 11948 (time 1769757612.71 total 184.67 last 56.77) - open/close 12552 (time 1769757622.73 total 194.69 last 60.31) - open/close 13241 (time 1769757632.73 total 204.69 last 68.88) - open/close 14040 (time 1769757642.74 total 214.70 last 79.80) - open/close 14830 (time 1769757652.75 total 224.71 last 78.96) - open/close 15666 (time 1769757662.76 total 234.72 last 83.48) - open/close 16341 (time 1769757672.77 total 244.73 last 67.44) - open/close 17143 (time 1769757682.78 total 254.75 last 80.08) - open/close 17840 (time 1769757692.79 total 264.75 last 69.67) - open/close 18593 (time 1769757702.80 total 274.76 last 75.26) - open/close 19263 (time 1769757712.81 total 284.77 last 66.90) - open/close 19900 (time 1769757722.83 total 294.79 last 63.60) - open/close 20538 (time 1769757732.84 total 304.80 last 63.71) - open/close 21219 (time 1769757742.86 total 314.82 last 67.96) - open/close 21870 (time 1769757752.87 total 324.83 last 65.07) - open/close 22606 (time 1769757762.87 total 334.83 last 73.59) - open/close 23202 (time 1769757772.88 total 344.84 last 59.52) - open/close 23799 (time 1769757782.90 total 354.86 last 59.56) - open/close 24526 (time 1769757792.91 total 364.87 last 72.67) - open/close 25250 (time 1769757802.91 total 374.87 last 72.38) - open/close 25916 (time 1769757812.92 total 384.88 last 66.52) - open/close 26560 (time 1769757822.93 total 394.89 last 64.36) - open/close 27281 (time 1769757832.94 total 404.90 last 72.04) - open/close 27901 (time 1769757842.95 total 414.91 last 61.91) - open/close 28484 (time 1769757852.95 total 424.91 last 58.28) - open/close 29150 (time 1769757862.97 total 434.93 last 66.52) - open/close 29873 (time 1769757872.97 total 444.93 last 72.25) - open/close 30000 (time 1769757875.08 total 447.04 last 60.37) - open/close 30723 (time 1769757885.08 total 457.04 last 72.27) - open/close 31434 (time 1769757895.08 total 467.04 last 71.09) - open/close 32117 (time 1769757905.09 total 477.05 last 68.25) - open/close 32797 (time 1769757915.11 total 487.07 last 67.85) - open/close 33353 (time 1769757925.13 total 497.09 last 55.52) - open/close 33991 (time 1769757935.14 total 507.10 last 63.70) - open/close 34636 (time 1769757945.15 total 517.11 last 64.46) - open/close 35329 (time 1769757955.16 total 527.12 last 69.25) - open/close 36042 (time 1769757965.16 total 537.12 last 71.24) - open/close 36725 (time 1769757975.17 total 547.13 last 68.29) - open/close 37388 (time 1769757985.17 total 557.13 last 66.29) - open/close 38067 (time 1769757995.18 total 567.14 last 67.81) - open/close 38779 (time 1769758005.19 total 577.15 last 71.11) - open/close 39388 (time 1769758015.22 total 587.18 last 60.74) - open/close 40000 (time 1769758023.79 total 595.75 last 71.37) - open/close 40683 (time 1769758033.80 total 605.76 last 68.27) - open/close 41351 (time 1769758043.80 total 615.77 last 66.76) - open/close 42182 (time 1769758053.81 total 625.77 last 83.06) - open/close 42888 (time 1769758063.83 total 635.79 last 70.46) - open/close 43531 (time 1769758073.83 total 645.79 last 64.27) - open/close 44174 (time 1769758083.84 total 655.80 last 64.28) - open/close 44861 (time 1769758093.84 total 665.80 last 68.69) - open/close 45523 (time 1769758103.84 total 675.80 last 66.19) - open/close 46304 (time 1769758113.86 total 685.82 last 77.97) - open/close 47032 (time 1769758123.86 total 695.82 last 72.80) - open/close 47697 (time 1769758133.86 total 705.82 last 66.45) - open/close 48362 (time 1769758143.87 total 715.83 last 66.47) - open/close 49016 (time 1769758153.88 total 725.84 last 65.32) - open/close 49677 (time 1769758163.89 total 735.85 last 66.03) - open/close 50000 (time 1769758167.93 total 739.89 last 80.03) - open/close 50800 (time 1769758177.94 total 749.90 last 79.93) - open/close 51509 (time 1769758187.94 total 759.90 last 70.85) - open/close 52244 (time 1769758197.96 total 769.92 last 73.36) - open/close 52933 (time 1769758207.97 total 779.93 last 68.85) - open/close 53676 (time 1769758217.98 total 789.94 last 74.23) - open/close 54543 (time 1769758227.98 total 799.94 last 86.67) - open/close 55296 (time 1769758237.99 total 809.95 last 75.25) - open/close 56037 (time 1769758248.00 total 819.96 last 74.04) - open/close 56782 (time 1769758258.01 total 829.97 last 74.43) - open/close 57498 (time 1769758268.01 total 839.98 last 71.54) - open/close 58281 (time 1769758278.02 total 849.98 last 78.28) - open/close 59071 (time 1769758288.02 total 859.98 last 78.99) - open/close 59879 (time 1769758298.02 total 869.99 last 80.76) - open/close 60715 (time 1769758308.03 total 879.99 last 83.59) - open/close 61521 (time 1769758318.03 total 889.99 last 80.59) - open/close 62261 (time 1769758328.03 total 899.99 last 74.00) - open/close 63115 (time 1769758338.03 total 909.99 last 85.38) - open/close 63915 (time 1769758348.04 total 920.00 last 79.94) - open/close 64611 (time 1769758358.04 total 930.00 last 69.59) total: 64768 open/close in 931.90 seconds: 69.50 ops/second - /unlink 640 (time 1769758373.97 total 10.01 last 63.95) - /unlink 1350 (time 1769758383.98 total 20.01 last 70.96) - /unlink 2088 (time 1769758394.00 total 30.03 last 73.64) - /unlink 2956 (time 1769758404.01 total 40.05 last 86.65) - /unlink 3557 (time 1769758414.02 total 50.05 last 60.10) - /unlink 4214 (time 1769758424.03 total 60.07 last 65.61) - /unlink 4931 (time 1769758434.04 total 70.07 last 71.66) - /unlink 5728 (time 1769758444.04 total 80.08 last 79.66) - /unlink 6568 (time 1769758454.05 total 90.09 last 83.93) - /unlink 7436 (time 1769758464.06 total 100.10 last 86.71) - /unlink 8177 (time 1769758474.07 total 110.11 last 74.04) - /unlink 8799 (time 1769758484.08 total 120.12 last 62.11) - /unlink 9536 (time 1769758494.09 total 130.13 last 73.61) - /unlink 10000 (time 1769758500.68 total 136.72 last 70.43) - /unlink 10846 (time 1769758510.70 total 146.73 last 84.49) - /unlink 11752 (time 1769758520.71 total 156.74 last 90.50) - /unlink 12652 (time 1769758530.71 total 166.75 last 89.97) - /unlink 13471 (time 1769758540.73 total 176.77 last 81.72) - /unlink 14271 (time 1769758550.74 total 186.78 last 79.94) - /unlink 15027 (time 1769758560.74 total 196.78 last 75.60) - /unlink 15851 (time 1769758570.74 total 206.78 last 82.37) - /unlink 16744 (time 1769758580.74 total 216.78 last 89.29) - /unlink 17551 (time 1769758590.75 total 226.78 last 80.68) - /unlink 18364 (time 1769758600.76 total 236.79 last 81.23) - /unlink 19231 (time 1769758610.76 total 246.80 last 86.66) - /unlink 20000 (time 1769758619.73 total 255.77 last 85.74) - /unlink 20816 (time 1769758629.74 total 265.78 last 81.51) - /unlink 21648 (time 1769758639.75 total 275.79 last 83.08) - /unlink 22409 (time 1769758649.75 total 285.79 last 76.10) - /unlink 23119 (time 1769758659.76 total 295.80 last 70.94) - /unlink 23901 (time 1769758669.78 total 305.81 last 78.09) - /unlink 24716 (time 1769758679.79 total 315.82 last 81.43) - /unlink 25670 (time 1769758689.80 total 325.84 last 95.26) - /unlink 26546 (time 1769758699.80 total 335.84 last 87.59) - /unlink 27231 (time 1769758709.81 total 345.85 last 68.44) - /unlink 27941 (time 1769758719.82 total 355.86 last 70.93) - /unlink 28756 (time 1769758729.82 total 365.86 last 81.50) - /unlink 29453 (time 1769758739.84 total 375.88 last 69.59) - /unlink 30000 (time 1769758746.60 total 382.63 last 80.93) - /unlink 30868 (time 1769758756.61 total 392.64 last 86.72) - /unlink 31757 (time 1769758766.62 total 402.65 last 88.83) - /unlink 32692 (time 1769758776.62 total 412.66 last 93.47) - /unlink 33489 (time 1769758786.63 total 422.67 last 79.61) - /unlink 34122 (time 1769758796.64 total 432.68 last 63.24) - /unlink 34910 (time 1769758806.64 total 442.68 last 78.78) - /unlink 35648 (time 1769758816.65 total 452.68 last 73.76) - /unlink 36362 (time 1769758826.65 total 462.69 last 71.36) - /unlink 37077 (time 1769758836.66 total 472.69 last 71.47) - /unlink 37789 (time 1769758846.66 total 482.70 last 71.16) - /unlink 38695 (time 1769758856.67 total 492.71 last 90.53) - /unlink 39653 (time 1769758866.67 total 502.71 last 95.77) - /unlink 40000 (time 1769758870.20 total 506.24 last 98.27) - /unlink 40904 (time 1769758880.21 total 516.25 last 90.35) - /unlink 41900 (time 1769758890.22 total 526.25 last 99.53) - /unlink 42815 (time 1769758900.23 total 536.27 last 91.37) - /unlink 43777 (time 1769758910.24 total 546.28 last 96.13) - /unlink 44745 (time 1769758920.24 total 556.28 last 96.75) - /unlink 45755 (time 1769758930.26 total 566.29 last 100.87) - /unlink 46631 (time 1769758940.26 total 576.30 last 87.56) - /unlink 47559 (time 1769758950.27 total 586.31 last 92.70) - /unlink 48447 (time 1769758960.28 total 596.31 last 88.74) - /unlink 49322 (time 1769758970.28 total 606.32 last 87.46) - /unlink 50000 (time 1769758977.60 total 613.63 last 92.69) - /unlink 51063 (time 1769758987.61 total 623.65 last 106.12) - /unlink 51825 (time 1769758997.62 total 633.65 last 76.18) - /unlink 52631 (time 1769759007.62 total 643.66 last 80.55) - /unlink 53544 (time 1769759017.63 total 653.67 last 91.23) - /unlink 54457 (time 1769759027.63 total 663.67 last 91.27) - /unlink 55322 (time 1769759037.64 total 673.68 last 86.44) - /unlink 56132 (time 1769759047.64 total 683.68 last 81.00) - /unlink 57014 (time 1769759057.65 total 693.68 last 88.14) - /unlink 57912 (time 1769759067.65 total 703.69 last 89.73) - /unlink 58878 (time 1769759077.66 total 713.70 last 96.53) - /unlink 59786 (time 1769759087.66 total 723.70 last 90.79) - /unlink 60000 (time 1769759089.81 total 725.84 last 99.90) - /unlink 60962 (time 1769759099.81 total 735.85 last 96.15) - /unlink 61849 (time 1769759109.82 total 745.86 last 88.63) - /unlink 62819 (time 1769759119.82 total 755.86 last 96.96) - /unlink 63814 (time 1769759129.83 total 765.86 last 99.46) - /unlink 64669 (time 1769759139.83 total 775.87 last 85.46) total: 64768 /unlink in 776.95 seconds: 83.36 ops/second Waiting for MDT destroys to complete oleg110-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 - open/close 1036 (time 1769759166.14 total 10.00 last 103.56) - open/close 2006 (time 1769759176.15 total 20.01 last 96.91) - open/close 2929 (time 1769759186.15 total 30.02 last 92.27) - open/close 3918 (time 1769759196.15 total 40.02 last 98.89) - open/close 4915 (time 1769759206.15 total 50.02 last 99.67) - open/close 5962 (time 1769759216.16 total 60.03 last 104.64) - open/close 6938 (time 1769759226.16 total 70.03 last 97.55) - open/close 7932 (time 1769759236.17 total 80.04 last 99.34) - open/close 8906 (time 1769759246.18 total 90.04 last 97.34) - open/close 9969 (time 1769759256.18 total 100.05 last 106.25) - open/close 11006 (time 1769759266.18 total 110.05 last 103.69) - open/close 12119 (time 1769759276.19 total 120.06 last 111.20) - open/close 13195 (time 1769759286.20 total 130.07 last 107.50) - open/close 14236 (time 1769759296.21 total 140.07 last 104.02) - open/close 15269 (time 1769759306.21 total 150.08 last 103.24) - open/close 16379 (time 1769759316.22 total 160.08 last 110.98) - open/close 17511 (time 1769759326.22 total 170.09 last 113.11) - open/close 19080 (time 1769759336.23 total 180.10 last 156.82) - open/close 20000 (time 1769759344.40 total 188.27 last 112.58) - open/close 21094 (time 1769759354.41 total 198.28 last 109.32) - open/close 22080 (time 1769759364.42 total 208.28 last 98.53) - open/close 23104 (time 1769759374.42 total 218.29 last 102.34) - open/close 24118 (time 1769759384.43 total 228.30 last 101.28) - open/close 25096 (time 1769759394.44 total 238.31 last 97.72) - open/close 26088 (time 1769759404.45 total 248.32 last 99.10) - open/close 27127 (time 1769759414.46 total 258.32 last 103.86) - open/close 28170 (time 1769759424.47 total 268.34 last 104.15) - open/close 29268 (time 1769759434.47 total 278.34 last 109.78) - open/close 30000 (time 1769759441.05 total 284.91 last 111.31) - open/close 31234 (time 1769759451.06 total 294.92 last 123.29) - open/close 32295 (time 1769759461.06 total 304.93 last 106.06) - open/close 33333 (time 1769759471.06 total 314.93 last 103.76) - open/close 34363 (time 1769759481.07 total 324.94 last 102.91) - open/close 35413 (time 1769759491.09 total 334.96 last 104.80) - open/close 36456 (time 1769759501.10 total 344.96 last 104.25) - open/close 37652 (time 1769759511.10 total 354.97 last 119.53) - open/close 38684 (time 1769759521.10 total 364.97 last 103.18) - open/close 39657 (time 1769759531.12 total 374.98 last 97.18) - open/close 40000 (time 1769759534.59 total 378.46 last 98.60) - open/close 41067 (time 1769759544.60 total 388.46 last 106.69) - open/close 42392 (time 1769759554.60 total 398.47 last 132.42) - open/close 43613 (time 1769759564.60 total 408.47 last 122.09) - open/close 44752 (time 1769759574.61 total 418.47 last 113.83) - open/close 45742 (time 1769759584.62 total 428.48 last 98.90) - open/close 46712 (time 1769759594.62 total 438.49 last 96.95) - open/close 47632 (time 1769759604.63 total 448.50 last 91.92) - open/close 48556 (time 1769759614.64 total 458.50 last 92.37) - open/close 49558 (time 1769759624.64 total 468.51 last 100.14) - open/close 50000 (time 1769759628.59 total 472.46 last 111.93) - open/close 50948 (time 1769759638.59 total 482.46 last 94.75) - open/close 51891 (time 1769759648.61 total 492.47 last 94.19) - open/close 52845 (time 1769759658.62 total 502.48 last 95.31) - open/close 53789 (time 1769759668.62 total 512.49 last 94.32) - open/close 54803 (time 1769759678.63 total 522.49 last 101.38) - open/close 55945 (time 1769759688.63 total 532.50 last 114.15) - open/close 57078 (time 1769759698.64 total 542.50 last 113.22) - open/close 58126 (time 1769759708.64 total 552.51 last 104.73) - open/close 59353 (time 1769759718.65 total 562.52 last 122.63) - open/close 60000 (time 1769759723.81 total 567.68 last 125.37) - open/close 61366 (time 1769759733.81 total 577.68 last 136.58) - open/close 62570 (time 1769759743.82 total 587.69 last 120.33) - open/close 63658 (time 1769759753.83 total 597.69 last 108.71) total: 64768 open/close in 606.95 seconds: 106.71 ops/second - /unlink 1272 (time 1769759775.19 total 10.00 last 127.15) - /unlink 2489 (time 1769759785.19 total 20.01 last 121.65) - /unlink 3795 (time 1769759795.20 total 30.02 last 130.49) - /unlink 6195 (time 1769759805.20 total 40.02 last 239.94) - /unlink 8957 (time 1769759815.20 total 50.02 last 276.19) - /unlink 10000 (time 1769759818.64 total 53.46 last 303.17) - /unlink 12565 (time 1769759828.64 total 63.46 last 256.42) - /unlink 15385 (time 1769759838.65 total 73.46 last 281.95) - /unlink 17875 (time 1769759848.65 total 83.47 last 248.96) - /unlink 20000 (time 1769759857.45 total 92.27 last 241.28) - /unlink 22356 (time 1769759867.46 total 102.27 last 235.56) - /unlink 24811 (time 1769759877.46 total 112.28 last 245.46) - /unlink 27029 (time 1769759887.46 total 122.28 last 221.74) - /unlink 29455 (time 1769759897.47 total 132.28 last 242.49) - /unlink 30000 (time 1769759899.60 total 134.41 last 255.81) - /unlink 32845 (time 1769759909.60 total 144.42 last 284.31) - /unlink 35407 (time 1769759919.61 total 154.43 last 256.08) - /unlink 37740 (time 1769759929.61 total 164.43 last 233.28) - /unlink 39998 (time 1769759939.61 total 174.43 last 225.74) - /unlink 42175 (time 1769759949.61 total 184.43 last 217.62) - /unlink 44262 (time 1769759959.62 total 194.44 last 208.56) - /unlink 46507 (time 1769759969.62 total 204.44 last 224.46) - /unlink 48989 (time 1769759979.62 total 214.44 last 248.15) - /unlink 50000 (time 1769759984.61 total 219.43 last 202.79) - /unlink 52166 (time 1769759994.62 total 229.43 last 216.47) - /unlink 54737 (time 1769760004.62 total 239.44 last 257.05) - /unlink 57329 (time 1769760014.62 total 249.44 last 259.19) - /unlink 59724 (time 1769760024.63 total 259.44 last 239.33) - /unlink 62434 (time 1769760034.63 total 269.45 last 270.83) total: 64768 /unlink in 279.16 seconds: 232.01 ops/second Waiting for MDT destroys to complete oleg110-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 - open/close 1242 (time 1769760067.17 total 10.01 last 124.06) - open/close 2577 (time 1769760077.17 total 20.01 last 133.50) - open/close 3878 (time 1769760087.17 total 30.01 last 130.09) - open/close 5221 (time 1769760097.17 total 40.01 last 134.26) - open/close 6716 (time 1769760107.18 total 50.02 last 149.43) - open/close 8112 (time 1769760117.18 total 60.02 last 139.59) - open/close 9629 (time 1769760127.18 total 70.02 last 151.64) - open/close 10000 (time 1769760129.22 total 72.06 last 182.04) - open/close 11660 (time 1769760139.22 total 82.06 last 165.96) - open/close 13232 (time 1769760149.23 total 92.07 last 157.10) - open/close 14805 (time 1769760159.23 total 102.07 last 157.30) - open/close 16810 (time 1769760169.23 total 112.08 last 200.37) - open/close 18961 (time 1769760179.25 total 122.09 last 214.80) - open/close 20000 (time 1769760184.00 total 126.84 last 218.78) - open/close 22476 (time 1769760194.00 total 136.84 last 247.56) - open/close 25002 (time 1769760204.00 total 146.84 last 252.53) - open/close 26928 (time 1769760214.00 total 156.85 last 192.59) - open/close 28676 (time 1769760224.00 total 166.85 last 174.76) - open/close 30000 (time 1769760231.77 total 174.62 last 170.41) - open/close 31585 (time 1769760241.77 total 184.62 last 158.50) - open/close 33200 (time 1769760251.78 total 194.62 last 161.40) - open/close 35100 (time 1769760261.78 total 204.63 last 189.91) - open/close 37025 (time 1769760271.79 total 214.63 last 192.46) - open/close 39151 (time 1769760281.79 total 224.64 last 212.45) - open/close 40000 (time 1769760286.64 total 229.49 last 175.02) - open/close 41755 (time 1769760296.64 total 239.49 last 175.49) - open/close 43496 (time 1769760306.65 total 249.49 last 174.06) - open/close 45700 (time 1769760316.65 total 259.49 last 220.39) - open/close 47467 (time 1769760326.65 total 269.49 last 176.67) - open/close 49164 (time 1769760336.65 total 279.50 last 169.66) - open/close 50000 (time 1769760341.83 total 284.67 last 161.48) - open/close 51047 (time 1769760351.84 total 294.68 last 104.61) - open/close 51774 (time 1769760361.84 total 304.69 last 72.65) - open/close 52576 (time 1769760371.86 total 314.70 last 80.10) - open/close 53679 (time 1769760381.86 total 324.70 last 110.27) - open/close 54998 (time 1769760391.86 total 334.70 last 131.88) - open/close 56390 (time 1769760401.87 total 344.71 last 139.12) - open/close 57674 (time 1769760411.88 total 354.72 last 128.25) - open/close 58932 (time 1769760421.88 total 364.72 last 125.77) - open/close 60000 (time 1769760430.09 total 372.93 last 130.16) - open/close 61178 (time 1769760440.09 total 382.94 last 117.72) - open/close 62448 (time 1769760450.09 total 392.94 last 126.99) - open/close 63693 (time 1769760460.10 total 402.94 last 124.46) total: 64768 open/close in 410.73 seconds: 157.69 ops/second - /unlink 2887 (time 1769760479.79 total 10.00 last 288.65) - /unlink 6470 (time 1769760489.79 total 20.00 last 358.26) - /unlink 9562 (time 1769760499.79 total 30.01 last 309.06) - /unlink 12420 (time 1769760509.80 total 40.01 last 285.78) - /unlink 15122 (time 1769760519.80 total 50.01 last 270.20) - /unlink 18066 (time 1769760529.80 total 60.01 last 294.34) - /unlink 20000 (time 1769760537.15 total 67.36 last 263.15) - /unlink 23636 (time 1769760547.15 total 77.36 last 363.60) - /unlink 27739 (time 1769760557.15 total 87.36 last 410.26) - /unlink 30000 (time 1769760562.53 total 92.75 last 419.75) - /unlink 33960 (time 1769760572.54 total 102.75 last 395.96) - /unlink 39279 (time 1769760582.54 total 112.75 last 531.84) - /unlink 44363 (time 1769760592.54 total 122.75 last 508.36) - /unlink 49170 (time 1769760602.54 total 132.75 last 480.67) - /unlink 54532 (time 1769760612.54 total 142.75 last 536.12) - /unlink 59040 (time 1769760622.54 total 152.75 last 450.78) - /unlink 60000 (time 1769760625.09 total 155.31 last 375.89) - /unlink 63841 (time 1769760635.10 total 165.31 last 384.05) total: 64768 /unlink in 167.68 seconds: 386.26 ops/second Waiting for MDT destroys to complete oleg110-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 - open/close 1600 (time 1769760659.33 total 10.00 last 159.94) - open/close 3433 (time 1769760669.33 total 20.00 last 183.30) - open/close 6130 (time 1769760679.33 total 30.01 last 269.67) - open/close 9140 (time 1769760689.33 total 40.01 last 300.97) - open/close 10000 (time 1769760692.46 total 43.13 last 274.89) - open/close 12589 (time 1769760702.46 total 53.13 last 258.90) - open/close 15070 (time 1769760712.46 total 63.14 last 248.07) - open/close 16811 (time 1769760722.46 total 73.14 last 174.08) - open/close 18522 (time 1769760732.46 total 83.14 last 171.07) - open/close 20000 (time 1769760740.45 total 91.12 last 185.10) - open/close 22387 (time 1769760750.45 total 101.12 last 238.68) - open/close 24702 (time 1769760760.45 total 111.13 last 231.42) - open/close 26860 (time 1769760770.45 total 121.13 last 215.77) - open/close 29227 (time 1769760780.46 total 131.13 last 236.60) - open/close 30000 (time 1769760783.81 total 134.49 last 230.34) - open/close 33196 (time 1769760793.81 total 144.49 last 319.50) - open/close 35357 (time 1769760803.82 total 154.49 last 216.06) - open/close 37832 (time 1769760813.82 total 164.50 last 247.40) - open/close 40000 (time 1769760820.50 total 171.18 last 324.61) - open/close 41906 (time 1769760830.51 total 181.18 last 190.48) - open/close 44034 (time 1769760840.51 total 191.19 last 212.71) - open/close 46107 (time 1769760850.51 total 201.19 last 207.24) - open/close 49331 (time 1769760860.51 total 211.19 last 322.37) - open/close 50000 (time 1769760862.93 total 213.61 last 277.10) - open/close 52668 (time 1769760872.93 total 223.61 last 266.74) - open/close 55664 (time 1769760882.93 total 233.61 last 299.54) - open/close 58065 (time 1769760892.94 total 243.62 last 239.93) - open/close 60000 (time 1769760900.13 total 250.81 last 269.08) - open/close 62427 (time 1769760910.13 total 260.81 last 242.63) total: 64768 open/close in 269.10 seconds: 240.68 ops/second - /unlink 5329 (time 1769760930.02 total 10.00 last 532.84) - /unlink 10000 (time 1769760939.73 total 19.71 last 480.96) - /unlink 14791 (time 1769760949.73 total 29.71 last 479.09) - /unlink 20000 (time 1769760958.86 total 38.84 last 570.71) - /unlink 25984 (time 1769760968.86 total 48.84 last 598.29) - /unlink 30000 (time 1769760977.90 total 57.89 last 443.98) - /unlink 34382 (time 1769760987.90 total 67.89 last 438.16) - /unlink 38680 (time 1769760997.90 total 77.89 last 429.77) - /unlink 40000 (time 1769761000.90 total 80.89 last 440.15) - /unlink 45250 (time 1769761010.90 total 90.89 last 524.98) - /unlink 50000 (time 1769761020.05 total 100.03 last 519.44) - /unlink 56109 (time 1769761030.05 total 110.03 last 610.88) - /unlink 60000 (time 1769761036.83 total 116.81 last 574.15) total: 64768 /unlink in 123.90 seconds: 522.75 ops/second Waiting for MDT destroys to complete oleg110-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg110-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg110-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg110-server stop mds service on oleg110-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg110-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg110-server' oleg110-server: oleg110-server.virtnet: executing unload_modules_local oleg110-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg110-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg110-client: oleg110-client: ssh exited with exit code 2 pdsh@oleg110-client: oleg110-server: ssh exited with exit code 2 pdsh@oleg110-client: oleg110-client: ssh exited with exit code 2 pdsh@oleg110-client: oleg110-server: ssh exited with exit code 2