== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 01:26:13 (1769754373) Stopping clients: oleg342-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg342-client.virtnet /mnt/lustre2 (opts:-f) oleg342-server: oleg342-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg342-server' oleg342-server: oleg342-server.virtnet: executing load_modules_local oleg342-server: Loading modules from /home/green/git/lustre-release/lustre oleg342-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg342-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg342-server: detected 4 online CPUs by sysfs oleg342-server: MODOPTS_LIBCFS= oleg342-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg342-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg342-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg342-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 673 (time 1769754514.77 total 10.00 last 67.29) - open/close 1548 (time 1769754524.77 total 20.00 last 87.48) - open/close 2418 (time 1769754534.78 total 30.01 last 86.96) - open/close 3181 (time 1769754544.78 total 40.01 last 76.28) - open/close 4020 (time 1769754554.79 total 50.02 last 83.86) - open/close 4948 (time 1769754564.79 total 60.02 last 92.75) - open/close 5706 (time 1769754574.79 total 70.02 last 75.79) - open/close 6258 (time 1769754584.79 total 80.02 last 55.19) - open/close 6982 (time 1769754594.80 total 90.03 last 72.36) - open/close 7893 (time 1769754604.80 total 100.03 last 91.08) - open/close 8816 (time 1769754614.81 total 110.04 last 92.23) - open/close 9733 (time 1769754624.82 total 120.05 last 91.58) - open/close 10000 (time 1769754627.57 total 122.80 last 97.16) - open/close 10842 (time 1769754637.57 total 132.80 last 84.18) - open/close 11626 (time 1769754647.59 total 142.82 last 78.26) - open/close 12352 (time 1769754657.60 total 152.83 last 72.52) - open/close 13175 (time 1769754667.64 total 162.87 last 81.99) - open/close 13977 (time 1769754677.64 total 172.87 last 80.17) - open/close 14846 (time 1769754687.65 total 182.88 last 86.82) - open/close 15816 (time 1769754697.66 total 192.89 last 96.95) - open/close 16564 (time 1769754707.66 total 202.89 last 74.74) - open/close 17239 (time 1769754717.69 total 212.92 last 67.32) - open/close 18067 (time 1769754727.70 total 222.93 last 82.76) - open/close 18918 (time 1769754737.71 total 232.94 last 85.00) - open/close 19749 (time 1769754747.71 total 242.94 last 83.07) - open/close 20000 (time 1769754750.27 total 245.50 last 98.31) - open/close 20923 (time 1769754760.27 total 255.50 last 92.26) - open/close 21754 (time 1769754770.28 total 265.51 last 83.05) - open/close 22528 (time 1769754780.29 total 275.52 last 77.31) - open/close 23475 (time 1769754790.29 total 285.52 last 94.67) - open/close 24325 (time 1769754800.30 total 295.53 last 84.95) - open/close 25169 (time 1769754810.32 total 305.55 last 84.24) - open/close 26096 (time 1769754820.32 total 315.55 last 92.65) - open/close 26753 (time 1769754830.33 total 325.56 last 65.64) - open/close 27415 (time 1769754840.35 total 335.58 last 66.06) - open/close 28056 (time 1769754850.36 total 345.59 last 64.07) - open/close 28895 (time 1769754860.37 total 355.60 last 83.80) - open/close 29696 (time 1769754870.37 total 365.60 last 80.04) - open/close 30000 (time 1769754874.09 total 369.32 last 81.88) - open/close 30788 (time 1769754884.09 total 379.32 last 78.80) - open/close 31604 (time 1769754894.09 total 389.33 last 81.54) - open/close 32520 (time 1769754904.10 total 399.33 last 91.58) - open/close 33273 (time 1769754914.10 total 409.33 last 75.28) - open/close 34146 (time 1769754924.11 total 419.34 last 87.25) - open/close 35020 (time 1769754934.11 total 429.34 last 87.35) - open/close 35728 (time 1769754944.12 total 439.35 last 70.75) - open/close 36529 (time 1769754954.14 total 449.37 last 79.94) - open/close 37203 (time 1769754964.14 total 459.37 last 67.40) - open/close 38110 (time 1769754974.14 total 469.37 last 90.67) - open/close 39097 (time 1769754984.15 total 479.38 last 98.62) - open/close 39944 (time 1769754994.15 total 489.38 last 84.66) - open/close 40776 (time 1769755004.16 total 499.39 last 83.18) - open/close 41519 (time 1769755014.16 total 509.39 last 74.28) - open/close 42282 (time 1769755024.17 total 519.40 last 76.23) - open/close 43086 (time 1769755034.18 total 529.41 last 80.33) - open/close 43948 (time 1769755044.18 total 539.42 last 86.15) - open/close 44896 (time 1769755054.19 total 549.42 last 94.76) - open/close 45869 (time 1769755064.19 total 559.43 last 97.24) - open/close 46756 (time 1769755074.20 total 569.43 last 88.69) - open/close 47623 (time 1769755084.22 total 579.45 last 86.51) - open/close 48472 (time 1769755094.23 total 589.46 last 84.82) - open/close 49290 (time 1769755104.23 total 599.46 last 81.77) - open/close 50000 (time 1769755113.30 total 608.53 last 78.26) - open/close 50812 (time 1769755123.31 total 618.54 last 81.13) - open/close 51468 (time 1769755133.33 total 628.56 last 65.49) - open/close 52268 (time 1769755143.33 total 638.56 last 79.99) - open/close 53150 (time 1769755153.34 total 648.57 last 88.12) - open/close 54040 (time 1769755163.34 total 658.57 last 88.97) - open/close 54951 (time 1769755173.34 total 668.57 last 91.07) - open/close 55917 (time 1769755183.35 total 678.58 last 96.51) - open/close 56761 (time 1769755193.37 total 688.60 last 84.24) - open/close 57532 (time 1769755203.38 total 698.61 last 77.05) - open/close 58351 (time 1769755213.38 total 708.61 last 81.87) - open/close 59233 (time 1769755223.39 total 718.62 last 88.15) - open/close 60000 (time 1769755231.42 total 726.65 last 95.45) - open/close 60808 (time 1769755241.42 total 736.65 last 80.79) - open/close 61698 (time 1769755251.44 total 746.67 last 88.90) - open/close 62477 (time 1769755261.45 total 756.68 last 77.79) - open/close 63214 (time 1769755271.45 total 766.68 last 73.69) - open/close 63949 (time 1769755281.46 total 776.69 last 73.44) total: 64768 open/close in 786.32 seconds: 82.37 ops/second - /unlink 979 (time 1769755305.57 total 10.01 last 97.82) - /unlink 1831 (time 1769755315.57 total 20.01 last 85.19) - /unlink 2881 (time 1769755325.57 total 30.02 last 104.92) - /unlink 3796 (time 1769755335.58 total 40.03 last 91.42) - /unlink 4635 (time 1769755345.59 total 50.03 last 83.84) - /unlink 5425 (time 1769755355.60 total 60.04 last 78.93) - /unlink 6065 (time 1769755365.60 total 70.04 last 63.99) - /unlink 6727 (time 1769755375.61 total 80.05 last 66.14) - /unlink 7691 (time 1769755385.61 total 90.05 last 96.39) - /unlink 8509 (time 1769755395.61 total 100.06 last 81.78) - /unlink 9395 (time 1769755405.62 total 110.06 last 88.53) - /unlink 10000 (time 1769755412.26 total 116.70 last 91.13) - /unlink 10823 (time 1769755422.26 total 126.71 last 82.28) - /unlink 11633 (time 1769755432.27 total 136.71 last 80.94) - /unlink 12383 (time 1769755442.28 total 146.72 last 74.97) - /unlink 13154 (time 1769755452.28 total 156.72 last 77.04) - /unlink 13889 (time 1769755462.32 total 166.76 last 73.24) - /unlink 14746 (time 1769755472.32 total 176.77 last 85.66) - /unlink 15470 (time 1769755482.33 total 186.77 last 72.38) - /unlink 16215 (time 1769755492.33 total 196.77 last 74.47) - /unlink 16934 (time 1769755502.34 total 206.78 last 71.83) - /unlink 17682 (time 1769755512.35 total 216.79 last 74.70) - /unlink 18389 (time 1769755522.36 total 226.80 last 70.66) - /unlink 19123 (time 1769755532.36 total 236.80 last 73.39) - /unlink 19811 (time 1769755542.37 total 246.81 last 68.74) - /unlink 20000 (time 1769755546.01 total 250.45 last 51.90) - /unlink 20693 (time 1769755556.02 total 260.46 last 69.24) - /unlink 21591 (time 1769755566.03 total 270.48 last 89.66) - /unlink 22455 (time 1769755576.04 total 280.48 last 86.34) - /unlink 23330 (time 1769755586.05 total 290.49 last 87.42) - /unlink 24232 (time 1769755596.06 total 300.50 last 90.11) - /unlink 25115 (time 1769755606.07 total 310.51 last 88.24) - /unlink 25806 (time 1769755616.08 total 320.52 last 69.02) - /unlink 26692 (time 1769755626.08 total 330.52 last 88.60) - /unlink 27577 (time 1769755636.08 total 340.52 last 88.48) - /unlink 28371 (time 1769755646.08 total 350.53 last 79.36) - /unlink 29252 (time 1769755656.09 total 360.53 last 88.02) - /unlink 30000 (time 1769755665.12 total 369.56 last 82.87) - /unlink 30923 (time 1769755675.12 total 379.56 last 92.29) - /unlink 31805 (time 1769755685.12 total 389.56 last 88.20) - /unlink 32688 (time 1769755695.13 total 399.57 last 88.24) - /unlink 33629 (time 1769755705.13 total 409.57 last 94.06) - /unlink 34513 (time 1769755715.14 total 419.58 last 88.34) - /unlink 35280 (time 1769755725.15 total 429.59 last 76.64) - /unlink 36111 (time 1769755735.16 total 439.60 last 83.03) - /unlink 37000 (time 1769755745.16 total 449.60 last 88.90) - /unlink 37944 (time 1769755755.16 total 459.60 last 94.34) - /unlink 38877 (time 1769755765.16 total 469.60 last 93.30) - /unlink 39760 (time 1769755775.17 total 479.61 last 88.25) - /unlink 40000 (time 1769755778.17 total 482.61 last 79.95) - /unlink 40986 (time 1769755788.18 total 492.62 last 98.54) - /unlink 41843 (time 1769755798.20 total 502.64 last 85.54) - /unlink 42721 (time 1769755808.21 total 512.65 last 87.66) - /unlink 43642 (time 1769755818.22 total 522.66 last 92.05) - /unlink 44467 (time 1769755828.22 total 532.66 last 82.48) - /unlink 45309 (time 1769755838.22 total 542.66 last 84.19) - /unlink 46113 (time 1769755848.23 total 552.67 last 80.36) - /unlink 46955 (time 1769755858.26 total 562.70 last 83.93) - /unlink 47754 (time 1769755868.28 total 572.72 last 79.74) - /unlink 48637 (time 1769755878.28 total 582.73 last 88.24) - /unlink 49313 (time 1769755888.29 total 592.73 last 67.54) - /unlink 49993 (time 1769755898.30 total 602.74 last 67.96) - /unlink 50954 (time 1769755908.30 total 612.74 last 96.10) - /unlink 51909 (time 1769755918.30 total 622.75 last 95.45) - /unlink 52773 (time 1769755928.30 total 632.75 last 86.39) - /unlink 53680 (time 1769755938.31 total 642.75 last 90.68) - /unlink 54597 (time 1769755948.31 total 652.75 last 91.66) - /unlink 55559 (time 1769755958.32 total 662.76 last 96.13) - /unlink 56445 (time 1769755968.34 total 672.79 last 88.38) - /unlink 57321 (time 1769755978.35 total 682.79 last 87.56) - /unlink 58254 (time 1769755988.35 total 692.80 last 93.26) - /unlink 59007 (time 1769755998.36 total 702.81 last 75.22) - /unlink 59854 (time 1769756008.37 total 712.81 last 84.65) - /unlink 60000 (time 1769756010.39 total 714.83 last 72.21) - /unlink 60833 (time 1769756020.41 total 724.85 last 83.17) - /unlink 61700 (time 1769756030.41 total 734.86 last 86.65) - /unlink 62636 (time 1769756040.43 total 744.87 last 93.43) - /unlink 63612 (time 1769756050.44 total 754.88 last 97.58) - /unlink 64525 (time 1769756060.44 total 764.88 last 91.24) total: 64768 /unlink in 767.53 seconds: 84.39 ops/second Waiting for MDT destroys to complete oleg342-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 - open/close 898 (time 1769756090.46 total 10.01 last 89.75) - open/close 1876 (time 1769756100.46 total 20.01 last 97.78) - open/close 2769 (time 1769756110.47 total 30.02 last 89.23) - open/close 3687 (time 1769756120.47 total 40.02 last 91.75) - open/close 4694 (time 1769756130.48 total 50.03 last 100.64) - open/close 5573 (time 1769756140.48 total 60.03 last 87.87) - open/close 6554 (time 1769756150.49 total 70.03 last 98.06) - open/close 7424 (time 1769756160.49 total 80.04 last 86.96) - open/close 8304 (time 1769756170.49 total 90.04 last 87.97) - open/close 9240 (time 1769756180.49 total 100.04 last 93.59) - open/close 10000 (time 1769756187.62 total 107.17 last 106.67) - open/close 11024 (time 1769756197.62 total 117.17 last 102.38) - open/close 12008 (time 1769756207.64 total 127.19 last 98.22) - open/close 12912 (time 1769756217.65 total 137.20 last 90.33) - open/close 13791 (time 1769756227.65 total 147.20 last 87.86) - open/close 14739 (time 1769756237.66 total 157.21 last 94.73) - open/close 15754 (time 1769756247.66 total 167.21 last 101.47) - open/close 16674 (time 1769756257.68 total 177.23 last 91.86) - open/close 17408 (time 1769756267.69 total 187.24 last 73.31) - open/close 18355 (time 1769756277.70 total 197.25 last 94.62) - open/close 19314 (time 1769756287.71 total 207.25 last 95.83) - open/close 20000 (time 1769756294.46 total 214.01 last 101.52) - open/close 21039 (time 1769756304.47 total 224.02 last 103.84) - open/close 22108 (time 1769756314.47 total 234.02 last 106.84) - open/close 22924 (time 1769756324.48 total 244.03 last 81.55) - open/close 23940 (time 1769756334.48 total 254.03 last 101.57) - open/close 24985 (time 1769756344.49 total 264.04 last 104.46) - open/close 26088 (time 1769756354.50 total 274.05 last 110.20) - open/close 26956 (time 1769756364.50 total 284.05 last 86.75) - open/close 27956 (time 1769756374.51 total 294.06 last 99.95) - open/close 29045 (time 1769756384.51 total 304.06 last 108.87) - open/close 29911 (time 1769756394.51 total 314.06 last 86.60) - open/close 31024 (time 1769756404.52 total 324.07 last 111.20) - open/close 32067 (time 1769756414.53 total 334.07 last 104.22) - open/close 32994 (time 1769756424.53 total 344.08 last 92.66) - open/close 33797 (time 1769756434.54 total 354.09 last 80.25) - open/close 34596 (time 1769756444.55 total 364.09 last 79.83) - open/close 35526 (time 1769756454.55 total 374.10 last 92.95) - open/close 36513 (time 1769756464.55 total 384.10 last 98.70) - open/close 37421 (time 1769756474.55 total 394.10 last 90.79) - open/close 38285 (time 1769756484.56 total 404.10 last 86.36) - open/close 39149 (time 1769756494.56 total 414.11 last 86.35) - open/close 40000 (time 1769756504.13 total 423.68 last 88.93) - open/close 41028 (time 1769756514.14 total 433.69 last 102.72) - open/close 41846 (time 1769756524.14 total 443.69 last 81.76) - open/close 42556 (time 1769756534.15 total 453.70 last 70.97) - open/close 43382 (time 1769756544.15 total 463.70 last 82.59) - open/close 44350 (time 1769756554.15 total 473.70 last 96.74) - open/close 45342 (time 1769756564.16 total 483.71 last 99.13) - open/close 46235 (time 1769756574.16 total 493.71 last 89.29) - open/close 47177 (time 1769756584.17 total 503.72 last 94.15) - open/close 48060 (time 1769756594.17 total 513.72 last 88.28) - open/close 49040 (time 1769756604.18 total 523.73 last 97.88) - open/close 49766 (time 1769756614.20 total 533.75 last 72.49) - open/close 50000 (time 1769756616.86 total 536.41 last 87.87) - open/close 50596 (time 1769756626.89 total 546.43 last 59.45) - open/close 51306 (time 1769756636.90 total 556.45 last 70.92) - open/close 51933 (time 1769756646.90 total 566.45 last 62.67) - open/close 52694 (time 1769756656.91 total 576.46 last 76.03) - open/close 53476 (time 1769756666.92 total 586.47 last 78.11) - open/close 54185 (time 1769756676.93 total 596.48 last 70.83) - open/close 54888 (time 1769756686.97 total 606.52 last 70.03) - open/close 55667 (time 1769756696.98 total 616.53 last 77.84) - open/close 56620 (time 1769756706.98 total 626.53 last 95.29) - open/close 57433 (time 1769756716.99 total 636.54 last 81.22) - open/close 58262 (time 1769756727.00 total 646.55 last 82.82) - open/close 59246 (time 1769756737.01 total 656.55 last 98.33) - open/close 60000 (time 1769756744.74 total 664.28 last 97.53) - open/close 60797 (time 1769756754.74 total 674.29 last 79.69) - open/close 61641 (time 1769756764.74 total 684.29 last 84.37) - open/close 62548 (time 1769756774.75 total 694.30 last 90.65) - open/close 63396 (time 1769756784.75 total 704.30 last 84.74) - open/close 64237 (time 1769756794.76 total 714.31 last 84.07) total: 64768 open/close in 720.56 seconds: 89.89 ops/second - /unlink 830 (time 1769756815.71 total 10.00 last 82.96) - /unlink 1714 (time 1769756825.71 total 20.01 last 88.37) - /unlink 2574 (time 1769756835.72 total 30.02 last 85.93) - /unlink 3413 (time 1769756845.73 total 40.02 last 83.88) - /unlink 4299 (time 1769756855.73 total 50.02 last 88.59) - /unlink 5044 (time 1769756865.73 total 60.02 last 74.49) - /unlink 5664 (time 1769756875.73 total 70.03 last 61.96) - /unlink 6378 (time 1769756885.74 total 80.04 last 71.34) - /unlink 7208 (time 1769756895.74 total 90.04 last 82.98) - /unlink 8040 (time 1769756905.75 total 100.04 last 83.14) - /unlink 8913 (time 1769756915.75 total 110.05 last 87.28) - /unlink 9811 (time 1769756925.76 total 120.05 last 89.76) - /unlink 10000 (time 1769756928.79 total 123.08 last 62.31) - /unlink 10653 (time 1769756938.81 total 133.10 last 65.19) - /unlink 11386 (time 1769756948.82 total 143.11 last 73.23) - /unlink 12218 (time 1769756958.83 total 153.12 last 83.14) - /unlink 12961 (time 1769756968.84 total 163.13 last 74.21) - /unlink 13686 (time 1769756978.85 total 173.14 last 72.44) - /unlink 14381 (time 1769756988.85 total 183.14 last 69.47) - /unlink 15314 (time 1769756998.86 total 193.15 last 93.23) - /unlink 16188 (time 1769757008.87 total 203.16 last 87.33) - /unlink 17055 (time 1769757018.88 total 213.17 last 86.58) - /unlink 17927 (time 1769757028.89 total 223.19 last 87.10) - /unlink 18745 (time 1769757038.90 total 233.19 last 81.72) - /unlink 19599 (time 1769757048.90 total 243.19 last 85.39) - /unlink 20000 (time 1769757053.66 total 247.95 last 84.25) - /unlink 20968 (time 1769757063.67 total 257.96 last 96.74) - /unlink 21591 (time 1769757073.67 total 267.96 last 62.28) - /unlink 22229 (time 1769757083.68 total 277.97 last 63.75) - /unlink 23026 (time 1769757093.68 total 287.97 last 79.69) - /unlink 23851 (time 1769757103.69 total 297.98 last 82.44) - /unlink 24724 (time 1769757113.70 total 307.99 last 87.19) - /unlink 25537 (time 1769757123.71 total 318.00 last 81.21) - /unlink 26241 (time 1769757133.73 total 328.02 last 70.29) - /unlink 27031 (time 1769757143.73 total 338.02 last 78.98) - /unlink 27891 (time 1769757153.73 total 348.02 last 85.98) - /unlink 28662 (time 1769757163.73 total 358.02 last 77.10) - /unlink 29465 (time 1769757173.74 total 368.03 last 80.26) - /unlink 30000 (time 1769757179.31 total 373.60 last 95.97) - /unlink 30846 (time 1769757189.32 total 383.61 last 84.54) - /unlink 31724 (time 1769757199.33 total 393.62 last 87.74) - /unlink 32671 (time 1769757209.33 total 403.62 last 94.68) - /unlink 33658 (time 1769757219.34 total 413.63 last 98.61) - /unlink 34591 (time 1769757229.35 total 423.64 last 93.16) - /unlink 35421 (time 1769757239.35 total 433.65 last 82.99) - /unlink 36388 (time 1769757249.37 total 443.66 last 96.55) - /unlink 37324 (time 1769757259.37 total 453.67 last 93.56) - /unlink 38285 (time 1769757269.37 total 463.67 last 96.09) - /unlink 39209 (time 1769757279.38 total 473.67 last 92.39) - /unlink 40000 (time 1769757289.27 total 483.56 last 79.95) - /unlink 40935 (time 1769757299.28 total 493.57 last 93.43) - /unlink 41680 (time 1769757309.28 total 503.57 last 74.49) - /unlink 42449 (time 1769757319.28 total 513.57 last 76.88) - /unlink 43397 (time 1769757329.29 total 523.58 last 94.72) - /unlink 44336 (time 1769757339.30 total 533.59 last 93.78) - /unlink 45324 (time 1769757349.30 total 543.60 last 98.77) - /unlink 46332 (time 1769757359.31 total 553.60 last 100.76) - /unlink 47418 (time 1769757369.32 total 563.61 last 108.51) - /unlink 48266 (time 1769757379.32 total 573.61 last 84.78) - /unlink 49227 (time 1769757389.33 total 583.62 last 96.03) - /unlink 50000 (time 1769757397.90 total 592.20 last 90.13) - /unlink 50895 (time 1769757407.91 total 602.20 last 89.44) - /unlink 51627 (time 1769757417.93 total 612.22 last 73.05) - /unlink 52508 (time 1769757427.94 total 622.24 last 87.99) - /unlink 53276 (time 1769757437.95 total 632.24 last 76.76) - /unlink 54260 (time 1769757447.95 total 642.24 last 98.37) - /unlink 55076 (time 1769757457.97 total 652.26 last 81.46) - /unlink 56015 (time 1769757467.97 total 662.26 last 93.89) - /unlink 56882 (time 1769757477.97 total 672.27 last 86.64) - /unlink 57814 (time 1769757487.98 total 682.27 last 93.17) - /unlink 58780 (time 1769757497.99 total 692.28 last 96.51) - /unlink 59687 (time 1769757508.00 total 702.29 last 90.61) - /unlink 60000 (time 1769757511.33 total 705.62 last 94.06) - /unlink 60892 (time 1769757521.33 total 715.62 last 89.17) - /unlink 61884 (time 1769757531.33 total 725.63 last 99.16) - /unlink 62706 (time 1769757541.34 total 735.63 last 82.15) - /unlink 63616 (time 1769757551.36 total 745.66 last 90.79) - /unlink 64486 (time 1769757561.37 total 755.67 last 86.90) total: 64768 /unlink in 758.44 seconds: 85.40 ops/second Waiting for MDT destroys to complete oleg342-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 - open/close 930 (time 1769757589.08 total 10.01 last 92.94) - open/close 1924 (time 1769757599.09 total 20.01 last 99.32) - open/close 2899 (time 1769757609.10 total 30.02 last 97.46) - open/close 3968 (time 1769757619.10 total 40.03 last 106.81) - open/close 4919 (time 1769757629.11 total 50.03 last 95.09) - open/close 5835 (time 1769757639.11 total 60.03 last 91.60) - open/close 6801 (time 1769757649.11 total 70.03 last 96.54) - open/close 7807 (time 1769757659.12 total 80.04 last 100.56) - open/close 8820 (time 1769757669.12 total 90.04 last 101.27) - open/close 9827 (time 1769757679.13 total 100.05 last 100.59) - open/close 10795 (time 1769757689.13 total 110.05 last 96.79) - open/close 11776 (time 1769757699.14 total 120.06 last 98.05) - open/close 12746 (time 1769757709.15 total 130.07 last 96.90) - open/close 13705 (time 1769757719.15 total 140.07 last 95.88) - open/close 14646 (time 1769757729.15 total 150.07 last 94.10) - open/close 15618 (time 1769757739.15 total 160.07 last 97.18) - open/close 16372 (time 1769757749.15 total 170.08 last 75.37) - open/close 17283 (time 1769757759.15 total 180.08 last 91.09) - open/close 18327 (time 1769757769.16 total 190.08 last 104.34) - open/close 19272 (time 1769757779.16 total 200.09 last 94.46) - open/close 20000 (time 1769757787.47 total 208.39 last 87.66) - open/close 21055 (time 1769757797.47 total 218.40 last 105.46) - open/close 22092 (time 1769757807.48 total 228.40 last 103.67) - open/close 23000 (time 1769757817.48 total 238.41 last 90.74) - open/close 23889 (time 1769757827.50 total 248.42 last 88.79) - open/close 24969 (time 1769757837.50 total 258.42 last 107.93) - open/close 25985 (time 1769757847.51 total 268.43 last 101.57) - open/close 26953 (time 1769757857.52 total 278.44 last 96.70) - open/close 27897 (time 1769757867.52 total 288.44 last 94.37) - open/close 28939 (time 1769757877.52 total 298.45 last 104.14) - open/close 30000 (time 1769757887.29 total 308.21 last 108.67) - open/close 31045 (time 1769757897.29 total 318.21 last 104.49) - open/close 32034 (time 1769757907.30 total 328.23 last 98.76) - open/close 33003 (time 1769757917.31 total 338.23 last 96.86) - open/close 33971 (time 1769757927.32 total 348.24 last 96.67) - open/close 34881 (time 1769757937.32 total 358.24 last 90.99) - open/close 35787 (time 1769757947.33 total 368.25 last 90.54) - open/close 36768 (time 1769757957.34 total 378.27 last 97.96) - open/close 37768 (time 1769757967.35 total 388.27 last 99.96) - open/close 38787 (time 1769757977.35 total 398.28 last 101.82) - open/close 39772 (time 1769757987.35 total 408.28 last 98.50) - open/close 40000 (time 1769757989.36 total 410.28 last 113.77) - open/close 41061 (time 1769757999.36 total 420.28 last 106.07) - open/close 42163 (time 1769758009.37 total 430.29 last 110.15) - open/close 43225 (time 1769758019.37 total 440.29 last 106.18) - open/close 44309 (time 1769758029.37 total 450.29 last 108.38) - open/close 45349 (time 1769758039.38 total 460.30 last 103.94) - open/close 46442 (time 1769758049.38 total 470.30 last 109.29) - open/close 47446 (time 1769758059.38 total 480.31 last 100.33) - open/close 48576 (time 1769758069.39 total 490.31 last 112.92) - open/close 49599 (time 1769758079.40 total 500.32 last 102.19) - open/close 50000 (time 1769758082.95 total 503.88 last 112.88) - open/close 51111 (time 1769758092.96 total 513.89 last 111.00) - open/close 52205 (time 1769758102.97 total 523.89 last 109.35) - open/close 53330 (time 1769758112.98 total 533.90 last 112.36) - open/close 54349 (time 1769758122.99 total 543.91 last 101.84) - open/close 55432 (time 1769758133.00 total 553.92 last 108.18) - open/close 56520 (time 1769758143.01 total 563.94 last 108.63) - open/close 57614 (time 1769758153.01 total 573.94 last 109.38) - open/close 58678 (time 1769758163.02 total 583.94 last 106.33) - open/close 59825 (time 1769758173.03 total 593.95 last 114.62) - open/close 60885 (time 1769758183.03 total 603.95 last 105.96) - open/close 61962 (time 1769758193.04 total 613.96 last 107.63) - open/close 63019 (time 1769758203.04 total 623.97 last 105.63) - open/close 64126 (time 1769758213.05 total 633.97 last 110.66) total: 64768 open/close in 639.97 seconds: 101.20 ops/second - /unlink 1147 (time 1769758232.44 total 10.00 last 114.69) - /unlink 2198 (time 1769758242.45 total 20.01 last 104.99) - /unlink 3301 (time 1769758252.46 total 30.02 last 110.25) - /unlink 4333 (time 1769758262.47 total 40.02 last 103.10) - /unlink 5410 (time 1769758272.48 total 50.04 last 107.53) - /unlink 6481 (time 1769758282.50 total 60.06 last 106.89) - /unlink 7560 (time 1769758292.53 total 70.09 last 107.60) - /unlink 8654 (time 1769758302.53 total 80.09 last 109.34) - /unlink 9708 (time 1769758312.55 total 90.11 last 105.22) - /unlink 10000 (time 1769758315.49 total 93.04 last 99.48) - /unlink 10907 (time 1769758325.49 total 103.05 last 90.66) - /unlink 11953 (time 1769758335.50 total 113.05 last 104.56) - /unlink 12973 (time 1769758345.50 total 123.06 last 101.95) - /unlink 14013 (time 1769758355.50 total 133.06 last 103.99) - /unlink 15150 (time 1769758365.51 total 143.07 last 113.62) - /unlink 16226 (time 1769758375.52 total 153.07 last 107.52) - /unlink 17286 (time 1769758385.52 total 163.08 last 105.96) - /unlink 18366 (time 1769758395.52 total 173.08 last 107.95) - /unlink 19473 (time 1769758405.53 total 183.09 last 110.63) - /unlink 20000 (time 1769758410.62 total 188.18 last 103.50) - /unlink 21118 (time 1769758420.63 total 198.18 last 111.74) - /unlink 22247 (time 1769758430.64 total 208.19 last 112.81) - /unlink 23360 (time 1769758440.64 total 218.20 last 111.27) - /unlink 24543 (time 1769758450.64 total 228.20 last 118.25) - /unlink 25771 (time 1769758460.64 total 238.20 last 122.77) - /unlink 26844 (time 1769758470.66 total 248.21 last 107.18) - /unlink 27953 (time 1769758480.66 total 258.22 last 110.87) - /unlink 29105 (time 1769758490.67 total 268.23 last 115.08) - /unlink 30000 (time 1769758498.79 total 276.34 last 110.26) - /unlink 31083 (time 1769758508.79 total 286.35 last 108.23) - /unlink 32238 (time 1769758518.80 total 296.35 last 115.49) - /unlink 33412 (time 1769758528.80 total 306.36 last 117.31) - /unlink 34496 (time 1769758538.81 total 316.37 last 108.34) - /unlink 35553 (time 1769758548.81 total 326.37 last 105.64) - /unlink 36720 (time 1769758558.82 total 336.38 last 116.62) - /unlink 37876 (time 1769758568.83 total 346.38 last 115.55) - /unlink 39710 (time 1769758578.83 total 356.39 last 183.37) - /unlink 41734 (time 1769758588.83 total 366.39 last 202.29) - /unlink 43814 (time 1769758598.84 total 376.40 last 207.89) - /unlink 45940 (time 1769758608.84 total 386.40 last 212.53) - /unlink 48096 (time 1769758618.85 total 396.40 last 215.50) - /unlink 50000 (time 1769758627.31 total 404.87 last 225.04) - /unlink 52160 (time 1769758637.31 total 414.87 last 215.93) - /unlink 54396 (time 1769758647.31 total 424.87 last 223.56) - /unlink 56828 (time 1769758657.31 total 434.87 last 243.18) - /unlink 59350 (time 1769758667.32 total 444.87 last 252.14) - /unlink 60000 (time 1769758669.98 total 447.53 last 244.26) - /unlink 62357 (time 1769758679.98 total 457.54 last 235.64) total: 64768 /unlink in 467.19 seconds: 138.63 ops/second Waiting for MDT destroys to complete oleg342-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 - open/close 1456 (time 1769758712.53 total 10.01 last 145.45) - open/close 2822 (time 1769758722.54 total 20.02 last 136.45) - open/close 4161 (time 1769758732.54 total 30.02 last 133.89) - open/close 5435 (time 1769758742.55 total 40.03 last 127.33) - open/close 6722 (time 1769758752.55 total 50.03 last 128.68) - open/close 7902 (time 1769758762.55 total 60.03 last 117.93) - open/close 9078 (time 1769758772.56 total 70.04 last 117.53) - open/close 10000 (time 1769758780.02 total 77.50 last 123.63) - open/close 11383 (time 1769758790.03 total 87.51 last 138.21) - open/close 12884 (time 1769758800.03 total 97.51 last 150.01) - open/close 14262 (time 1769758810.03 total 107.51 last 137.77) - open/close 15663 (time 1769758820.04 total 117.52 last 140.03) - open/close 17205 (time 1769758830.04 total 127.52 last 154.16) - open/close 18503 (time 1769758840.04 total 137.52 last 129.77) - open/close 19918 (time 1769758850.05 total 147.53 last 141.46) - open/close 21181 (time 1769758860.05 total 157.53 last 126.23) - open/close 22409 (time 1769758870.06 total 167.54 last 122.76) - open/close 23751 (time 1769758880.06 total 177.54 last 134.10) - open/close 25147 (time 1769758890.07 total 187.55 last 139.56) - open/close 26682 (time 1769758900.07 total 197.55 last 153.43) - open/close 27991 (time 1769758910.07 total 207.55 last 130.90) - open/close 29719 (time 1769758920.07 total 217.55 last 172.78) - open/close 30000 (time 1769758922.32 total 219.80 last 125.04) - open/close 31431 (time 1769758932.33 total 229.81 last 142.99) - open/close 33013 (time 1769758942.34 total 239.82 last 158.00) - open/close 34459 (time 1769758952.35 total 249.83 last 144.49) - open/close 35779 (time 1769758962.35 total 259.83 last 131.96) - open/close 37426 (time 1769758972.36 total 269.84 last 164.58) - open/close 39031 (time 1769758982.36 total 279.84 last 160.41) - open/close 40000 (time 1769758989.54 total 287.02 last 135.00) - open/close 41474 (time 1769758999.55 total 297.03 last 147.30) - open/close 42794 (time 1769759009.55 total 307.03 last 131.99) - open/close 44099 (time 1769759019.55 total 317.03 last 130.47) - open/close 45557 (time 1769759029.56 total 327.04 last 145.73) - open/close 47151 (time 1769759039.56 total 337.04 last 159.38) - open/close 48751 (time 1769759049.56 total 347.04 last 159.98) - open/close 50000 (time 1769759056.67 total 354.15 last 175.71) - open/close 51726 (time 1769759066.67 total 364.15 last 172.51) - open/close 53315 (time 1769759076.67 total 374.15 last 158.85) - open/close 54790 (time 1769759086.68 total 384.16 last 147.41) - open/close 56403 (time 1769759096.69 total 394.17 last 161.20) - open/close 58245 (time 1769759106.69 total 404.17 last 184.19) - open/close 60000 (time 1769759116.58 total 414.06 last 177.35) - open/close 61668 (time 1769759126.59 total 424.07 last 166.75) - open/close 63366 (time 1769759136.59 total 434.07 last 169.75) total: 64768 open/close in 443.31 seconds: 146.10 ops/second - /unlink 2535 (time 1769759157.87 total 10.00 last 253.45) - /unlink 5048 (time 1769759167.87 total 20.00 last 251.28) - /unlink 7796 (time 1769759177.88 total 30.00 last 274.77) - /unlink 10000 (time 1769759186.16 total 38.28 last 266.16) - /unlink 12536 (time 1769759196.16 total 48.29 last 253.48) - /unlink 15387 (time 1769759206.16 total 58.29 last 285.05) - /unlink 17862 (time 1769759216.16 total 68.29 last 247.50) - /unlink 20000 (time 1769759224.62 total 76.75 last 252.74) - /unlink 23055 (time 1769759234.62 total 86.75 last 305.43) - /unlink 26310 (time 1769759244.63 total 96.75 last 325.46) - /unlink 29483 (time 1769759254.63 total 106.75 last 317.28) - /unlink 32551 (time 1769759264.63 total 116.76 last 306.61) - /unlink 36490 (time 1769759274.63 total 126.76 last 393.89) - /unlink 40000 (time 1769759283.46 total 135.59 last 397.67) - /unlink 44043 (time 1769759293.46 total 145.59 last 404.20) - /unlink 48142 (time 1769759303.46 total 155.59 last 409.89) - /unlink 50000 (time 1769759307.91 total 160.04 last 417.32) - /unlink 54158 (time 1769759317.92 total 170.04 last 415.69) - /unlink 58307 (time 1769759327.92 total 180.05 last 414.84) - /unlink 60000 (time 1769759332.61 total 184.73 last 361.08) - /unlink 64271 (time 1769759342.61 total 194.74 last 426.99) total: 64768 /unlink in 195.77 seconds: 330.84 ops/second Waiting for MDT destroys to complete oleg342-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 - open/close 2398 (time 1769759366.61 total 10.00 last 239.76) - open/close 4991 (time 1769759376.61 total 20.00 last 259.28) - open/close 7761 (time 1769759386.61 total 30.01 last 276.88) - open/close 10000 (time 1769759395.14 total 38.54 last 262.50) - open/close 12060 (time 1769759405.14 total 48.54 last 205.97) - open/close 15094 (time 1769759415.14 total 58.54 last 303.36) - open/close 17789 (time 1769759425.14 total 68.54 last 269.46) - open/close 20000 (time 1769759433.30 total 76.69 last 271.17) - open/close 22778 (time 1769759443.30 total 86.70 last 277.73) - open/close 25358 (time 1769759453.30 total 96.70 last 257.98) - open/close 28658 (time 1769759463.30 total 106.70 last 329.95) - open/close 30000 (time 1769759467.77 total 111.16 last 300.64) - open/close 32594 (time 1769759477.77 total 121.16 last 259.39) - open/close 35340 (time 1769759487.77 total 131.17 last 274.52) - open/close 38552 (time 1769759497.77 total 141.17 last 321.13) - open/close 40000 (time 1769759503.13 total 146.53 last 270.19) - open/close 42828 (time 1769759513.13 total 156.53 last 282.76) - open/close 45513 (time 1769759523.13 total 166.53 last 268.45) - open/close 48734 (time 1769759533.13 total 176.53 last 322.06) - open/close 50000 (time 1769759536.50 total 179.90 last 376.13) - open/close 53303 (time 1769759546.50 total 189.90 last 330.28) - open/close 56826 (time 1769759556.50 total 199.90 last 352.30) - open/close 59563 (time 1769759566.50 total 209.90 last 273.62) - open/close 62369 (time 1769759576.51 total 219.90 last 280.58) total: 64768 open/close in 227.70 seconds: 284.45 ops/second - /unlink 5744 (time 1769759595.77 total 10.00 last 574.18) - /unlink 10000 (time 1769759602.81 total 17.04 last 604.53) - /unlink 18061 (time 1769759612.81 total 27.04 last 806.03) - /unlink 20000 (time 1769759615.66 total 29.89 last 680.33) - /unlink 25786 (time 1769759625.66 total 39.90 last 578.60) - /unlink 30000 (time 1769759632.15 total 46.38 last 649.45) - /unlink 36754 (time 1769759642.15 total 56.38 last 675.33) - /unlink 40000 (time 1769759646.80 total 61.03 last 698.05) - /unlink 47596 (time 1769759656.80 total 71.04 last 759.57) - /unlink 50000 (time 1769759660.53 total 74.76 last 644.67) - /unlink 57201 (time 1769759670.53 total 84.76 last 720.06) - /unlink 60000 (time 1769759674.64 total 88.88 last 680.49) total: 64768 /unlink in 94.92 seconds: 682.32 ops/second Waiting for MDT destroys to complete oleg342-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg342-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg342-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg342-server stop mds service on oleg342-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg342-server stop mds service on oleg342-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg342-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg342-server' oleg342-server: oleg342-server.virtnet: executing unload_modules_local oleg342-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg342-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg342-client: oleg342-client: ssh exited with exit code 2 pdsh@oleg342-client: oleg342-server: ssh exited with exit code 2 pdsh@oleg342-client: oleg342-server: ssh exited with exit code 2 pdsh@oleg342-client: oleg342-client: ssh exited with exit code 2