== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 11:56:33 (1772470593) Stopping clients: oleg206-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg206-client.virtnet /mnt/lustre2 (opts:-f) oleg206-server: oleg206-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg206-server' oleg206-server: oleg206-server.virtnet: executing load_modules_local oleg206-server: Loading modules from /home/green/git/lustre-release/lustre oleg206-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg206-server: detected 4 online CPUs by sysfs oleg206-server: MODOPTS_LIBCFS= oleg206-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg206-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg206-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg206-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg206-client.virtnet: -o user_xattr,flock 192.168.202.106@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 917 (time 1772470743.03 total 10.00 last 91.68) - open/close 1676 (time 1772470753.03 total 20.01 last 75.88) - open/close 2547 (time 1772470763.05 total 30.02 last 87.01) - open/close 3299 (time 1772470773.05 total 40.02 last 75.14) - open/close 4143 (time 1772470783.06 total 50.03 last 84.32) - open/close 4955 (time 1772470793.06 total 60.04 last 81.18) - open/close 5806 (time 1772470803.08 total 70.05 last 84.99) - open/close 6739 (time 1772470813.08 total 80.05 last 93.25) - open/close 7723 (time 1772470823.09 total 90.06 last 98.36) - open/close 8521 (time 1772470833.10 total 100.08 last 79.65) - open/close 9241 (time 1772470843.11 total 110.09 last 71.93) - open/close 9930 (time 1772470853.13 total 120.10 last 68.82) - open/close 10691 (time 1772470863.14 total 130.11 last 76.03) - open/close 11620 (time 1772470873.14 total 140.11 last 92.83) - open/close 12409 (time 1772470883.14 total 150.11 last 78.90) - open/close 13310 (time 1772470893.14 total 160.12 last 90.08) - open/close 14113 (time 1772470903.15 total 170.12 last 80.27) - open/close 15111 (time 1772470913.15 total 180.13 last 99.74) - open/close 15972 (time 1772470923.17 total 190.14 last 85.99) - open/close 16852 (time 1772470933.17 total 200.14 last 88.00) - open/close 17742 (time 1772470943.17 total 210.14 last 88.99) - open/close 18708 (time 1772470953.20 total 220.17 last 96.28) - open/close 19611 (time 1772470963.20 total 230.17 last 90.29) - open/close 20000 (time 1772470967.20 total 234.17 last 97.26) - open/close 20957 (time 1772470977.21 total 244.18 last 95.64) - open/close 21786 (time 1772470987.21 total 254.19 last 82.85) - open/close 22562 (time 1772470997.22 total 264.19 last 77.58) - open/close 23401 (time 1772471007.22 total 274.19 last 83.87) - open/close 24104 (time 1772471017.23 total 284.20 last 70.22) - open/close 24793 (time 1772471027.25 total 294.22 last 68.80) - open/close 25559 (time 1772471037.26 total 304.23 last 76.50) - open/close 26303 (time 1772471047.27 total 314.24 last 74.34) - open/close 27260 (time 1772471057.27 total 324.24 last 95.66) - open/close 28043 (time 1772471067.28 total 334.25 last 78.25) - open/close 28896 (time 1772471077.29 total 344.26 last 85.22) - open/close 29816 (time 1772471087.29 total 354.26 last 91.96) - open/close 30000 (time 1772471089.41 total 356.38 last 86.67) - open/close 30913 (time 1772471099.42 total 366.39 last 91.24) - open/close 31853 (time 1772471109.42 total 376.40 last 93.96) - open/close 32747 (time 1772471119.44 total 386.41 last 89.26) - open/close 33457 (time 1772471129.44 total 396.41 last 70.98) - open/close 34305 (time 1772471139.46 total 406.43 last 84.70) - open/close 35169 (time 1772471149.46 total 416.43 last 86.39) - open/close 36129 (time 1772471159.46 total 426.44 last 95.92) - open/close 36847 (time 1772471169.49 total 436.46 last 71.60) - open/close 37699 (time 1772471179.49 total 446.46 last 85.19) - open/close 38484 (time 1772471189.51 total 456.48 last 78.39) - open/close 39176 (time 1772471199.51 total 466.48 last 69.16) - open/close 39789 (time 1772471209.52 total 476.49 last 61.28) - open/close 40000 (time 1772471212.72 total 479.69 last 65.94) - open/close 40601 (time 1772471222.72 total 489.69 last 60.09) - open/close 41372 (time 1772471232.72 total 499.69 last 77.06) - open/close 41955 (time 1772471242.73 total 509.70 last 58.27) - open/close 42669 (time 1772471252.74 total 519.71 last 71.35) - open/close 43421 (time 1772471262.75 total 529.72 last 75.09) - open/close 44242 (time 1772471272.76 total 539.73 last 82.05) - open/close 45044 (time 1772471282.76 total 549.73 last 80.16) - open/close 45863 (time 1772471292.77 total 559.74 last 81.80) - open/close 46586 (time 1772471302.78 total 569.75 last 72.24) - open/close 47382 (time 1772471312.79 total 579.76 last 79.50) - open/close 48239 (time 1772471322.81 total 589.78 last 85.58) - open/close 48859 (time 1772471332.83 total 599.80 last 61.87) - open/close 49480 (time 1772471342.83 total 609.80 last 62.06) - open/close 50000 (time 1772471350.80 total 617.77 last 65.30) - open/close 50799 (time 1772471360.82 total 627.79 last 79.73) - open/close 51495 (time 1772471370.82 total 637.79 last 69.59) - open/close 52201 (time 1772471380.83 total 647.81 last 70.49) - open/close 53035 (time 1772471390.84 total 657.81 last 83.36) - open/close 53796 (time 1772471400.85 total 667.82 last 76.06) - open/close 54733 (time 1772471410.85 total 677.82 last 93.63) - open/close 55558 (time 1772471420.86 total 687.83 last 82.46) - open/close 56430 (time 1772471430.86 total 697.83 last 87.19) - open/close 57252 (time 1772471440.86 total 707.83 last 82.19) - open/close 58079 (time 1772471450.86 total 717.84 last 82.67) - open/close 58745 (time 1772471460.87 total 727.84 last 66.57) - open/close 59531 (time 1772471470.87 total 737.84 last 78.57) - open/close 60000 (time 1772471476.99 total 743.96 last 76.73) - open/close 60762 (time 1772471487.02 total 753.99 last 75.95) - open/close 61525 (time 1772471497.02 total 763.99 last 76.27) - open/close 62337 (time 1772471507.04 total 774.01 last 81.04) - open/close 62932 (time 1772471517.05 total 784.02 last 59.46) - open/close 63705 (time 1772471527.06 total 794.03 last 77.25) - open/close 64638 (time 1772471537.09 total 804.06 last 93.01) total: 64768 open/close in 805.50 seconds: 80.41 ops/second - /unlink 874 (time 1772471553.61 total 10.01 last 87.28) - /unlink 1780 (time 1772471563.61 total 20.01 last 90.60) - /unlink 2502 (time 1772471573.62 total 30.02 last 72.15) - /unlink 3184 (time 1772471583.63 total 40.03 last 68.14) - /unlink 3827 (time 1772471593.64 total 50.04 last 64.26) - /unlink 4679 (time 1772471603.64 total 60.04 last 85.16) - /unlink 5442 (time 1772471613.64 total 70.04 last 76.28) - /unlink 6199 (time 1772471623.66 total 80.06 last 75.56) - /unlink 6996 (time 1772471633.67 total 90.07 last 79.67) - /unlink 7750 (time 1772471643.67 total 100.07 last 75.35) - /unlink 8440 (time 1772471653.68 total 110.08 last 68.93) - /unlink 9309 (time 1772471663.69 total 120.09 last 86.82) - /unlink 10000 (time 1772471672.89 total 129.29 last 75.09) - /unlink 10694 (time 1772471682.89 total 139.29 last 69.39) - /unlink 11520 (time 1772471692.90 total 149.30 last 82.56) - /unlink 12433 (time 1772471702.90 total 159.30 last 91.25) - /unlink 13230 (time 1772471712.92 total 169.32 last 79.59) - /unlink 14085 (time 1772471722.92 total 179.32 last 85.49) - /unlink 14941 (time 1772471732.93 total 189.33 last 85.54) - /unlink 15708 (time 1772471742.94 total 199.34 last 76.62) - /unlink 16531 (time 1772471752.94 total 209.34 last 82.24) - /unlink 17307 (time 1772471762.95 total 219.35 last 77.54) - /unlink 18078 (time 1772471772.96 total 229.36 last 77.05) - /unlink 18916 (time 1772471782.97 total 239.37 last 83.73) - /unlink 19723 (time 1772471792.97 total 249.37 last 80.63) - /unlink 20000 (time 1772471796.15 total 252.55 last 87.17) - /unlink 20718 (time 1772471806.16 total 262.56 last 71.74) - /unlink 21496 (time 1772471816.16 total 272.56 last 77.78) - /unlink 22328 (time 1772471826.17 total 282.57 last 83.14) - /unlink 23179 (time 1772471836.17 total 292.57 last 85.07) - /unlink 23994 (time 1772471846.18 total 302.58 last 81.48) - /unlink 24791 (time 1772471856.19 total 312.59 last 79.60) - /unlink 25631 (time 1772471866.21 total 322.61 last 83.84) - /unlink 26309 (time 1772471876.23 total 332.63 last 67.64) - /unlink 26969 (time 1772471886.24 total 342.64 last 65.93) - /unlink 27981 (time 1772471896.24 total 352.64 last 101.18) - /unlink 28755 (time 1772471906.26 total 362.66 last 77.30) - /unlink 29565 (time 1772471916.26 total 372.66 last 80.94) - /unlink 30000 (time 1772471921.20 total 377.60 last 88.05) - /unlink 30905 (time 1772471931.21 total 387.61 last 90.41) - /unlink 31785 (time 1772471941.22 total 397.62 last 87.94) - /unlink 32677 (time 1772471951.22 total 407.62 last 89.20) - /unlink 33634 (time 1772471961.23 total 417.63 last 95.66) - /unlink 34515 (time 1772471971.25 total 427.65 last 87.89) - /unlink 35223 (time 1772471981.27 total 437.67 last 70.68) - /unlink 35910 (time 1772471991.27 total 447.67 last 68.65) - /unlink 36825 (time 1772472001.28 total 457.68 last 91.42) - /unlink 37692 (time 1772472011.29 total 467.69 last 86.60) - /unlink 38531 (time 1772472021.32 total 477.72 last 83.64) - /unlink 39504 (time 1772472031.33 total 487.73 last 97.29) - /unlink 40000 (time 1772472036.14 total 492.54 last 103.10) - /unlink 40711 (time 1772472046.14 total 502.54 last 71.08) - /unlink 41474 (time 1772472056.15 total 512.55 last 76.23) - /unlink 42215 (time 1772472066.15 total 522.55 last 74.07) - /unlink 43037 (time 1772472076.16 total 532.56 last 82.16) - /unlink 43871 (time 1772472086.19 total 542.59 last 83.16) - /unlink 44555 (time 1772472096.19 total 552.59 last 68.37) - /unlink 45311 (time 1772472106.20 total 562.60 last 75.55) - /unlink 46223 (time 1772472116.20 total 572.60 last 91.15) - /unlink 47093 (time 1772472126.22 total 582.62 last 86.84) - /unlink 47871 (time 1772472136.23 total 592.63 last 77.76) - /unlink 48736 (time 1772472146.24 total 602.64 last 86.40) - /unlink 49562 (time 1772472156.24 total 612.64 last 82.57) - /unlink 50000 (time 1772472162.04 total 618.44 last 75.52) - /unlink 50828 (time 1772472172.04 total 628.44 last 82.80) - /unlink 51644 (time 1772472182.05 total 638.45 last 81.55) - /unlink 52398 (time 1772472192.05 total 648.45 last 75.35) - /unlink 53207 (time 1772472202.06 total 658.46 last 80.83) - /unlink 54079 (time 1772472212.06 total 668.46 last 87.20) - /unlink 54911 (time 1772472222.06 total 678.46 last 83.20) - /unlink 55677 (time 1772472232.07 total 688.47 last 76.52) - /unlink 56573 (time 1772472242.08 total 698.47 last 89.59) - /unlink 57400 (time 1772472252.08 total 708.47 last 82.70) - /unlink 58309 (time 1772472262.08 total 718.48 last 90.85) - /unlink 59175 (time 1772472272.09 total 728.49 last 86.52) - /unlink 59959 (time 1772472282.10 total 738.50 last 78.35) - /unlink 60807 (time 1772472292.10 total 748.50 last 84.75) - /unlink 61554 (time 1772472302.11 total 758.51 last 74.67) - /unlink 62345 (time 1772472312.11 total 768.51 last 79.05) - /unlink 63204 (time 1772472322.12 total 778.52 last 85.81) - /unlink 64079 (time 1772472332.14 total 788.54 last 87.35) total: 64768 /unlink in 797.41 seconds: 81.22 ops/second Waiting for MDT destroys to complete oleg206-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 - open/close 995 (time 1772472368.48 total 10.00 last 99.49) - open/close 1960 (time 1772472378.48 total 20.01 last 96.45) - open/close 2947 (time 1772472388.49 total 30.02 last 98.61) - open/close 3779 (time 1772472398.49 total 40.02 last 83.19) - open/close 4560 (time 1772472408.51 total 50.04 last 77.92) - open/close 5403 (time 1772472418.53 total 60.05 last 84.18) - open/close 6292 (time 1772472428.53 total 70.06 last 88.86) - open/close 7083 (time 1772472438.54 total 80.07 last 79.02) - open/close 7927 (time 1772472448.55 total 90.08 last 84.32) - open/close 8688 (time 1772472458.56 total 100.08 last 76.04) - open/close 9445 (time 1772472468.58 total 110.10 last 75.57) - open/close 10000 (time 1772472475.96 total 117.49 last 75.15) - open/close 10927 (time 1772472485.97 total 127.49 last 92.64) - open/close 11699 (time 1772472495.97 total 137.50 last 77.16) - open/close 12476 (time 1772472505.98 total 147.50 last 77.65) - open/close 13226 (time 1772472515.99 total 157.51 last 74.93) - open/close 14109 (time 1772472526.00 total 167.52 last 88.24) - open/close 15030 (time 1772472536.00 total 177.52 last 92.06) - open/close 15883 (time 1772472546.01 total 187.54 last 85.18) - open/close 16681 (time 1772472556.03 total 197.56 last 79.66) - open/close 17599 (time 1772472566.04 total 207.57 last 91.69) - open/close 18389 (time 1772472576.05 total 217.58 last 78.94) - open/close 19152 (time 1772472586.05 total 227.58 last 76.29) - open/close 20000 (time 1772472595.71 total 237.23 last 87.84) - open/close 20607 (time 1772472605.71 total 247.23 last 60.67) - open/close 21212 (time 1772472615.73 total 257.26 last 60.35) - open/close 21941 (time 1772472625.74 total 267.27 last 72.86) - open/close 22808 (time 1772472635.75 total 277.27 last 86.63) - open/close 23613 (time 1772472645.75 total 287.27 last 80.50) - open/close 24361 (time 1772472655.75 total 297.27 last 74.80) - open/close 25040 (time 1772472665.76 total 307.28 last 67.83) - open/close 25667 (time 1772472675.76 total 317.28 last 62.70) - open/close 26382 (time 1772472685.77 total 327.29 last 71.44) - open/close 27247 (time 1772472695.78 total 337.30 last 86.42) - open/close 28142 (time 1772472705.79 total 347.32 last 89.37) - open/close 29110 (time 1772472715.79 total 357.32 last 96.78) - open/close 30000 (time 1772472724.34 total 365.87 last 104.12) - open/close 30940 (time 1772472734.38 total 375.91 last 93.62) - open/close 31790 (time 1772472744.39 total 385.92 last 84.89) - open/close 32581 (time 1772472754.41 total 395.93 last 78.99) - open/close 33269 (time 1772472764.42 total 405.94 last 68.73) - open/close 34160 (time 1772472774.42 total 415.95 last 89.05) - open/close 35062 (time 1772472784.44 total 425.97 last 90.01) - open/close 36000 (time 1772472794.45 total 435.98 last 93.73) - open/close 36835 (time 1772472804.48 total 446.01 last 83.24) - open/close 37713 (time 1772472814.49 total 456.01 last 87.76) - open/close 38617 (time 1772472824.49 total 466.01 last 90.38) - open/close 39640 (time 1772472834.50 total 476.02 last 102.24) - open/close 40000 (time 1772472838.63 total 480.16 last 87.00) - open/close 40878 (time 1772472848.64 total 490.17 last 87.73) - open/close 41785 (time 1772472858.64 total 500.17 last 90.67) - open/close 42635 (time 1772472868.66 total 510.18 last 84.89) - open/close 43340 (time 1772472878.66 total 520.19 last 70.45) - open/close 44153 (time 1772472888.67 total 530.19 last 81.28) - open/close 44881 (time 1772472898.68 total 540.20 last 72.71) - open/close 45692 (time 1772472908.68 total 550.21 last 81.08) - open/close 46528 (time 1772472918.68 total 560.21 last 83.59) - open/close 47366 (time 1772472928.69 total 570.22 last 83.73) - open/close 47903 (time 1772472938.71 total 580.24 last 53.59) - open/close 48836 (time 1772472948.71 total 590.24 last 93.29) - open/close 49636 (time 1772472958.72 total 600.24 last 79.95) - open/close 50000 (time 1772472964.09 total 605.61 last 67.82) - open/close 50793 (time 1772472974.11 total 615.63 last 79.13) - open/close 51603 (time 1772472984.12 total 625.64 last 80.93) - open/close 52470 (time 1772472994.14 total 635.66 last 86.54) - open/close 53224 (time 1772473004.14 total 645.67 last 75.36) - open/close 54046 (time 1772473014.14 total 655.67 last 82.20) - open/close 54786 (time 1772473024.15 total 665.67 last 73.95) - open/close 55619 (time 1772473034.16 total 675.68 last 83.23) - open/close 56482 (time 1772473044.16 total 685.69 last 86.24) - open/close 57295 (time 1772473054.16 total 695.69 last 81.28) - open/close 58351 (time 1772473064.17 total 705.69 last 105.59) - open/close 59184 (time 1772473074.18 total 715.71 last 83.19) - open/close 60000 (time 1772473083.90 total 725.43 last 83.94) - open/close 60922 (time 1772473093.92 total 735.45 last 92.01) - open/close 61821 (time 1772473103.93 total 745.45 last 89.84) - open/close 62493 (time 1772473113.93 total 755.46 last 67.18) - open/close 63369 (time 1772473123.93 total 765.46 last 87.60) - open/close 64085 (time 1772473133.93 total 775.46 last 71.58) total: 64768 open/close in 784.79 seconds: 82.53 ops/second - /unlink 610 (time 1772473159.72 total 10.01 last 60.91) - /unlink 1196 (time 1772473169.73 total 20.02 last 58.55) - /unlink 1918 (time 1772473179.73 total 30.02 last 72.20) - /unlink 2615 (time 1772473189.73 total 40.03 last 69.68) - /unlink 3253 (time 1772473199.74 total 50.03 last 63.76) - /unlink 3929 (time 1772473209.75 total 60.04 last 67.54) - /unlink 4564 (time 1772473219.75 total 70.05 last 63.48) - /unlink 5264 (time 1772473229.76 total 80.05 last 69.94) - /unlink 6010 (time 1772473239.76 total 90.06 last 74.56) - /unlink 6758 (time 1772473249.76 total 100.06 last 74.79) - /unlink 7749 (time 1772473259.77 total 110.07 last 99.05) - /unlink 8707 (time 1772473269.77 total 120.07 last 95.79) - /unlink 9599 (time 1772473279.78 total 130.07 last 89.16) - /unlink 10000 (time 1772473284.57 total 134.87 last 83.62) - /unlink 10911 (time 1772473294.58 total 144.87 last 91.06) - /unlink 11776 (time 1772473304.61 total 154.90 last 86.21) - /unlink 12494 (time 1772473314.61 total 164.91 last 71.76) - /unlink 13207 (time 1772473324.63 total 174.92 last 71.22) - /unlink 13806 (time 1772473334.63 total 184.92 last 59.89) - /unlink 14450 (time 1772473344.63 total 194.92 last 64.40) - /unlink 15046 (time 1772473354.63 total 204.93 last 59.58) - /unlink 15735 (time 1772473364.64 total 214.93 last 68.86) - /unlink 16514 (time 1772473374.65 total 224.95 last 77.78) - /unlink 17465 (time 1772473384.65 total 234.95 last 95.08) - /unlink 18260 (time 1772473394.66 total 244.96 last 79.43) - /unlink 19068 (time 1772473404.67 total 254.96 last 80.76) - /unlink 20000 (time 1772473414.61 total 264.90 last 93.74) - /unlink 20955 (time 1772473424.61 total 274.91 last 95.44) - /unlink 21850 (time 1772473434.62 total 284.92 last 89.45) - /unlink 22703 (time 1772473444.62 total 294.92 last 85.27) - /unlink 23563 (time 1772473454.63 total 304.92 last 85.96) - /unlink 24364 (time 1772473464.63 total 314.93 last 80.07) - /unlink 25194 (time 1772473474.65 total 324.95 last 82.84) - /unlink 26007 (time 1772473484.66 total 334.96 last 81.21) - /unlink 26988 (time 1772473494.67 total 344.96 last 98.04) - /unlink 27941 (time 1772473504.67 total 354.97 last 95.26) - /unlink 28958 (time 1772473514.67 total 364.97 last 101.69) - /unlink 29880 (time 1772473524.69 total 374.98 last 92.09) - /unlink 30828 (time 1772473534.69 total 384.99 last 94.73) - /unlink 31809 (time 1772473544.70 total 395.00 last 97.99) - /unlink 32783 (time 1772473554.71 total 405.00 last 97.38) - /unlink 33736 (time 1772473564.71 total 415.01 last 95.24) - /unlink 34712 (time 1772473574.72 total 425.02 last 97.52) - /unlink 35692 (time 1772473584.73 total 435.02 last 97.97) - /unlink 36659 (time 1772473594.73 total 445.03 last 96.65) - /unlink 37621 (time 1772473604.74 total 455.04 last 96.10) - /unlink 38600 (time 1772473614.75 total 465.04 last 97.85) - /unlink 39596 (time 1772473624.75 total 475.04 last 99.59) - /unlink 40000 (time 1772473628.48 total 478.77 last 108.30) - /unlink 40989 (time 1772473638.48 total 488.78 last 98.85) - /unlink 42149 (time 1772473648.49 total 498.78 last 115.95) - /unlink 43184 (time 1772473658.49 total 508.78 last 103.49) - /unlink 44222 (time 1772473668.49 total 518.78 last 103.79) - /unlink 45317 (time 1772473678.49 total 528.79 last 109.45) - /unlink 46376 (time 1772473688.50 total 538.80 last 105.79) - /unlink 47369 (time 1772473698.50 total 548.80 last 99.29) - /unlink 48398 (time 1772473708.51 total 558.80 last 102.85) - /unlink 49465 (time 1772473718.51 total 568.81 last 106.68) - /unlink 50000 (time 1772473724.52 total 574.81 last 89.07) - /unlink 51007 (time 1772473734.53 total 584.82 last 100.63) - /unlink 52203 (time 1772473744.53 total 594.82 last 119.59) - /unlink 53353 (time 1772473754.53 total 604.82 last 114.98) - /unlink 54510 (time 1772473764.53 total 614.82 last 115.68) - /unlink 55659 (time 1772473774.53 total 624.82 last 114.90) - /unlink 56769 (time 1772473784.54 total 634.83 last 110.91) - /unlink 57964 (time 1772473794.54 total 644.83 last 119.48) - /unlink 59111 (time 1772473804.55 total 654.85 last 114.52) - /unlink 60000 (time 1772473812.46 total 662.76 last 112.41) - /unlink 61086 (time 1772473822.47 total 672.76 last 108.54) - /unlink 62257 (time 1772473832.47 total 682.77 last 117.06) - /unlink 63451 (time 1772473842.49 total 692.79 last 119.17) - /unlink 64620 (time 1772473852.50 total 702.80 last 116.78) total: 64768 /unlink in 704.16 seconds: 91.98 ops/second Waiting for MDT destroys to complete oleg206-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 - open/close 1280 (time 1772473874.25 total 10.00 last 127.99) - open/close 2507 (time 1772473884.26 total 20.01 last 122.62) - open/close 3759 (time 1772473894.26 total 30.01 last 125.17) - open/close 4990 (time 1772473904.27 total 40.02 last 123.00) - open/close 6215 (time 1772473914.27 total 50.02 last 122.44) - open/close 7635 (time 1772473924.28 total 60.03 last 141.88) - open/close 8919 (time 1772473934.28 total 70.03 last 128.38) - open/close 10000 (time 1772473942.47 total 78.22 last 132.08) - open/close 11264 (time 1772473952.47 total 88.22 last 126.38) - open/close 12711 (time 1772473962.47 total 98.22 last 144.66) - open/close 14089 (time 1772473972.47 total 108.22 last 137.77) - open/close 15480 (time 1772473982.48 total 118.23 last 139.02) - open/close 16939 (time 1772473992.49 total 128.24 last 145.80) - open/close 18224 (time 1772474002.49 total 138.24 last 128.48) - open/close 19432 (time 1772474012.50 total 148.25 last 120.65) - open/close 20000 (time 1772474017.20 total 152.95 last 120.86) - open/close 21276 (time 1772474027.20 total 162.96 last 127.53) - open/close 22556 (time 1772474037.21 total 172.96 last 127.96) - open/close 23837 (time 1772474047.21 total 182.96 last 128.07) - open/close 25160 (time 1772474057.22 total 192.97 last 132.22) - open/close 26410 (time 1772474067.22 total 202.98 last 124.90) - open/close 27684 (time 1772474077.23 total 212.98 last 127.35) - open/close 28916 (time 1772474087.23 total 222.98 last 123.16) - open/close 30000 (time 1772474095.14 total 230.89 last 137.06) - open/close 31376 (time 1772474105.15 total 240.90 last 137.52) - open/close 32795 (time 1772474115.15 total 250.90 last 141.88) - open/close 34038 (time 1772474125.16 total 260.91 last 124.22) - open/close 35341 (time 1772474135.16 total 270.91 last 130.23) - open/close 36718 (time 1772474145.17 total 280.92 last 137.62) - open/close 37994 (time 1772474155.18 total 290.93 last 127.47) - open/close 39346 (time 1772474165.19 total 300.94 last 135.04) - open/close 40000 (time 1772474170.66 total 306.41 last 119.47) - open/close 41356 (time 1772474180.66 total 316.41 last 135.57) - open/close 42601 (time 1772474190.66 total 326.41 last 124.50) - open/close 43861 (time 1772474200.68 total 336.43 last 125.81) - open/close 45107 (time 1772474210.69 total 346.44 last 124.51) - open/close 46457 (time 1772474220.69 total 356.44 last 134.99) - open/close 47888 (time 1772474230.69 total 366.44 last 143.04) - open/close 49110 (time 1772474240.69 total 376.44 last 122.20) - open/close 50000 (time 1772474247.67 total 383.42 last 127.54) - open/close 51204 (time 1772474257.68 total 393.43 last 120.34) - open/close 52476 (time 1772474267.69 total 403.44 last 127.04) - open/close 53768 (time 1772474277.69 total 413.44 last 129.17) - open/close 55092 (time 1772474287.69 total 423.44 last 132.36) - open/close 56327 (time 1772474297.70 total 433.45 last 123.42) - open/close 57662 (time 1772474307.70 total 443.45 last 133.48) - open/close 59024 (time 1772474317.70 total 453.45 last 136.16) - open/close 60000 (time 1772474324.98 total 460.73 last 134.16) - open/close 61097 (time 1772474334.98 total 470.73 last 109.64) - open/close 62383 (time 1772474344.99 total 480.74 last 128.48) - open/close 63551 (time 1772474355.00 total 490.75 last 116.76) - open/close 64709 (time 1772474365.00 total 500.75 last 115.77) total: 64768 open/close in 501.28 seconds: 129.20 ops/second - /unlink 2107 (time 1772474378.54 total 10.01 last 210.56) - /unlink 4299 (time 1772474388.54 total 20.01 last 219.11) - /unlink 6232 (time 1772474398.55 total 30.01 last 193.25) - /unlink 8515 (time 1772474408.55 total 40.02 last 228.25) - /unlink 10000 (time 1772474415.20 total 46.66 last 223.40) - /unlink 12203 (time 1772474425.20 total 56.66 last 220.26) - /unlink 14353 (time 1772474435.20 total 66.67 last 214.90) - /unlink 16621 (time 1772474445.20 total 76.67 last 226.74) - /unlink 18842 (time 1772474455.21 total 86.68 last 221.92) - /unlink 20000 (time 1772474460.33 total 91.79 last 226.49) - /unlink 22217 (time 1772474470.33 total 101.79 last 221.68) - /unlink 24402 (time 1772474480.33 total 111.79 last 218.49) - /unlink 26501 (time 1772474490.33 total 121.80 last 209.86) - /unlink 28803 (time 1772474500.33 total 131.80 last 230.15) - /unlink 30000 (time 1772474505.83 total 137.30 last 217.52) - /unlink 32226 (time 1772474515.84 total 147.30 last 222.55) - /unlink 34312 (time 1772474525.84 total 157.31 last 208.55) - /unlink 36459 (time 1772474535.84 total 167.31 last 214.68) - /unlink 38760 (time 1772474545.84 total 177.31 last 230.09) - /unlink 40000 (time 1772474551.14 total 182.61 last 233.80) - /unlink 42274 (time 1772474561.14 total 192.61 last 227.39) - /unlink 44629 (time 1772474571.14 total 202.61 last 235.50) - /unlink 46874 (time 1772474581.15 total 212.61 last 224.45) - /unlink 49199 (time 1772474591.15 total 222.62 last 232.45) - /unlink 50000 (time 1772474594.70 total 226.17 last 225.62) - /unlink 52119 (time 1772474604.70 total 236.17 last 211.82) - /unlink 54178 (time 1772474614.70 total 246.17 last 205.86) - /unlink 56328 (time 1772474624.71 total 256.17 last 214.94) - /unlink 58655 (time 1772474634.71 total 266.18 last 232.68) - /unlink 60000 (time 1772474641.08 total 272.55 last 211.15) - /unlink 62409 (time 1772474651.08 total 282.55 last 240.85) - /unlink 64718 (time 1772474661.08 total 292.55 last 230.87) total: 64768 /unlink in 292.76 seconds: 221.23 ops/second Waiting for MDT destroys to complete oleg206-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 - open/close 1464 (time 1772474687.79 total 10.00 last 146.38) - open/close 2775 (time 1772474697.79 total 20.00 last 131.09) - open/close 4079 (time 1772474707.80 total 30.01 last 130.32) - open/close 5265 (time 1772474717.81 total 40.02 last 118.49) - open/close 6544 (time 1772474727.81 total 50.02 last 127.87) - open/close 7851 (time 1772474737.82 total 60.03 last 130.55) - open/close 9144 (time 1772474747.82 total 70.03 last 129.28) - open/close 10000 (time 1772474754.54 total 76.75 last 127.42) - open/close 11421 (time 1772474764.55 total 86.76 last 142.00) - open/close 12799 (time 1772474774.55 total 96.76 last 137.74) - open/close 14164 (time 1772474784.56 total 106.77 last 136.45) - open/close 15482 (time 1772474794.56 total 116.77 last 131.76) - open/close 16909 (time 1772474804.56 total 126.77 last 142.64) - open/close 18337 (time 1772474814.57 total 136.78 last 142.71) - open/close 19674 (time 1772474824.58 total 146.78 last 133.63) - open/close 20000 (time 1772474827.25 total 149.46 last 121.80) - open/close 21349 (time 1772474837.26 total 159.47 last 134.80) - open/close 22691 (time 1772474847.26 total 169.47 last 134.14) - open/close 24027 (time 1772474857.27 total 179.48 last 133.52) - open/close 25382 (time 1772474867.28 total 189.49 last 135.42) - open/close 26711 (time 1772474877.28 total 199.49 last 132.87) - open/close 28069 (time 1772474887.28 total 209.49 last 135.74) - open/close 29432 (time 1772474897.29 total 219.50 last 136.24) - open/close 30000 (time 1772474901.47 total 223.68 last 135.73) - open/close 31374 (time 1772474911.48 total 233.68 last 137.36) - open/close 32802 (time 1772474921.48 total 243.69 last 142.78) - open/close 34116 (time 1772474931.48 total 253.69 last 131.32) - open/close 35417 (time 1772474941.48 total 263.69 last 130.07) - open/close 36716 (time 1772474951.49 total 273.70 last 129.81) - open/close 38077 (time 1772474961.50 total 283.71 last 135.99) - open/close 39463 (time 1772474971.50 total 293.71 last 138.54) - open/close 40000 (time 1772474975.41 total 297.62 last 137.53) - open/close 41350 (time 1772474985.41 total 307.62 last 134.92) - open/close 42664 (time 1772474995.41 total 317.62 last 131.39) - open/close 44024 (time 1772475005.42 total 327.63 last 135.92) - open/close 45541 (time 1772475015.43 total 337.64 last 151.57) - open/close 46926 (time 1772475025.44 total 347.64 last 138.41) - open/close 48327 (time 1772475035.44 total 357.65 last 140.06) - open/close 49640 (time 1772475045.44 total 367.65 last 131.25) - open/close 50000 (time 1772475048.21 total 370.41 last 130.28) - open/close 51305 (time 1772475058.21 total 380.42 last 130.44) - open/close 52615 (time 1772475068.21 total 390.42 last 130.94) - open/close 54001 (time 1772475078.22 total 400.42 last 138.59) - open/close 55342 (time 1772475088.22 total 410.43 last 133.99) - open/close 56696 (time 1772475098.23 total 420.44 last 135.31) - open/close 58023 (time 1772475108.23 total 430.44 last 132.66) - open/close 59382 (time 1772475118.24 total 440.44 last 135.87) - open/close 60000 (time 1772475122.83 total 445.04 last 134.42) - open/close 61397 (time 1772475132.83 total 455.04 last 139.70) - open/close 62838 (time 1772475142.84 total 465.05 last 144.04) - open/close 64670 (time 1772475152.84 total 475.05 last 183.19) total: 64768 open/close in 475.63 seconds: 136.17 ops/second - /unlink 2629 (time 1772475165.40 total 10.00 last 262.89) - /unlink 5241 (time 1772475175.40 total 20.00 last 261.16) - /unlink 8257 (time 1772475185.40 total 30.00 last 301.59) - /unlink 10000 (time 1772475190.32 total 34.93 last 353.95) - /unlink 13336 (time 1772475200.32 total 44.93 last 333.60) - /unlink 16614 (time 1772475210.32 total 54.93 last 327.77) - /unlink 19431 (time 1772475220.32 total 64.93 last 281.69) - /unlink 20000 (time 1772475222.47 total 67.08 last 264.64) - /unlink 22872 (time 1772475232.48 total 77.08 last 287.11) - /unlink 25530 (time 1772475242.48 total 87.08 last 265.77) - /unlink 28466 (time 1772475252.48 total 97.08 last 293.59) - /unlink 30000 (time 1772475257.39 total 102.00 last 312.10) - /unlink 33678 (time 1772475267.40 total 112.00 last 367.62) - /unlink 37797 (time 1772475277.40 total 122.00 last 411.89) - /unlink 40000 (time 1772475282.11 total 126.72 last 467.48) - /unlink 43646 (time 1772475292.11 total 136.72 last 364.58) - /unlink 46690 (time 1772475302.12 total 146.72 last 304.30) - /unlink 49537 (time 1772475312.12 total 156.72 last 284.62) - /unlink 52576 (time 1772475322.12 total 166.72 last 303.89) - /unlink 55556 (time 1772475332.12 total 176.73 last 297.87) - /unlink 58442 (time 1772475342.13 total 186.73 last 288.50) - /unlink 60000 (time 1772475347.74 total 192.34 last 277.67) - /unlink 62863 (time 1772475357.74 total 202.34 last 286.24) total: 64768 /unlink in 208.35 seconds: 310.86 ops/second Waiting for MDT destroys to complete oleg206-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 - open/close 1898 (time 1772475383.92 total 10.01 last 189.55) - open/close 3593 (time 1772475393.92 total 20.02 last 169.43) - open/close 5346 (time 1772475403.93 total 30.03 last 175.12) - open/close 6982 (time 1772475413.94 total 40.03 last 163.53) - open/close 9081 (time 1772475423.94 total 50.04 last 209.79) - open/close 10000 (time 1772475427.77 total 53.86 last 240.22) - open/close 12157 (time 1772475437.77 total 63.86 last 215.67) - open/close 14229 (time 1772475447.77 total 73.87 last 207.09) - open/close 16547 (time 1772475457.78 total 83.87 last 231.77) - open/close 18508 (time 1772475467.79 total 93.88 last 195.92) - open/close 20000 (time 1772475475.79 total 101.89 last 186.31) - open/close 22056 (time 1772475485.80 total 111.89 last 205.57) - open/close 23926 (time 1772475495.80 total 121.89 last 186.93) - open/close 26063 (time 1772475505.80 total 131.89 last 213.67) - open/close 27917 (time 1772475515.80 total 141.90 last 185.35) - open/close 29639 (time 1772475525.80 total 151.90 last 172.20) - open/close 30000 (time 1772475528.06 total 154.16 last 159.67) - open/close 31897 (time 1772475538.07 total 164.16 last 189.63) - open/close 33557 (time 1772475548.07 total 174.16 last 166.00) - open/close 35462 (time 1772475558.07 total 184.17 last 190.42) - open/close 37170 (time 1772475568.07 total 194.17 last 170.77) - open/close 38973 (time 1772475578.08 total 204.17 last 180.22) - open/close 40000 (time 1772475583.85 total 209.95 last 177.80) - open/close 41640 (time 1772475593.86 total 219.96 last 163.83) - open/close 43062 (time 1772475603.87 total 229.96 last 142.18) - open/close 44662 (time 1772475613.87 total 239.96 last 159.96) - open/close 46105 (time 1772475623.87 total 249.96 last 144.30) - open/close 47824 (time 1772475633.87 total 259.97 last 171.82) - open/close 49504 (time 1772475643.87 total 269.97 last 167.98) - open/close 50000 (time 1772475646.51 total 272.60 last 188.12) - open/close 52368 (time 1772475656.51 total 282.60 last 236.78) - open/close 54465 (time 1772475666.51 total 292.61 last 209.65) - open/close 56856 (time 1772475676.51 total 302.61 last 239.07) - open/close 59159 (time 1772475686.51 total 312.61 last 230.29) - open/close 60000 (time 1772475689.04 total 315.14 last 332.47) - open/close 62589 (time 1772475699.05 total 325.14 last 258.83) total: 64768 open/close in 333.07 seconds: 194.46 ops/second - /unlink 3836 (time 1772475718.50 total 10.00 last 383.51) - /unlink 6882 (time 1772475728.50 total 20.00 last 304.57) - /unlink 10000 (time 1772475737.99 total 29.49 last 328.61) - /unlink 13184 (time 1772475747.99 total 39.49 last 318.38) - /unlink 16250 (time 1772475757.99 total 49.50 last 306.48) - /unlink 19639 (time 1772475767.99 total 59.50 last 338.85) - /unlink 24604 (time 1772475777.99 total 69.50 last 496.35) - /unlink 28689 (time 1772475788.00 total 79.50 last 408.41) - /unlink 30000 (time 1772475792.47 total 83.97 last 293.33) - /unlink 33030 (time 1772475802.47 total 93.97 last 302.97) - /unlink 36468 (time 1772475812.47 total 103.97 last 343.79) - /unlink 39882 (time 1772475822.47 total 113.98 last 341.32) - /unlink 43239 (time 1772475832.47 total 123.98 last 335.66) - /unlink 47099 (time 1772475842.47 total 133.98 last 385.93) - /unlink 50000 (time 1772475848.17 total 139.68 last 509.24) - /unlink 54761 (time 1772475858.17 total 149.68 last 476.10) - /unlink 58297 (time 1772475868.17 total 159.68 last 353.59) - /unlink 60000 (time 1772475871.87 total 163.38 last 460.27) - /unlink 64164 (time 1772475881.87 total 173.38 last 416.33) total: 64768 /unlink in 174.76 seconds: 370.62 ops/second Waiting for MDT destroys to complete oleg206-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg206-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg206-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg206-server stop mds service on oleg206-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg206-server stop mds service on oleg206-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg206-server unloading modules via unload_modules_local on: 'oleg206-server' oleg206-server: oleg206-server.virtnet: executing unload_modules_local oleg206-server: modules unloaded. pdsh@oleg206-client: oleg206-client: ssh exited with exit code 2 pdsh@oleg206-client: oleg206-server: ssh exited with exit code 2 pdsh@oleg206-client: oleg206-client: ssh exited with exit code 2 pdsh@oleg206-client: oleg206-server: ssh exited with exit code 2