== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 21:57:03 (1763348223) Stopping clients: oleg239-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg239-client.virtnet /mnt/lustre2 (opts:-f) oleg239-server: oleg239-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg239-server' oleg239-server: oleg239-server.virtnet: executing load_modules_local oleg239-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg239-server: Loading modules from /home/green/git/lustre-release/lustre oleg239-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg239-server: detected 4 online CPUs by sysfs oleg239-server: MODOPTS_LIBCFS= oleg239-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg239-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg239-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg239-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg239-client.virtnet: -o user_xattr,flock 192.168.202.139@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 796 (time 1763348344.49 total 10.02 last 79.42) - open/close 1592 (time 1763348354.49 total 20.02 last 79.59) - open/close 2296 (time 1763348364.50 total 30.03 last 70.37) - open/close 3027 (time 1763348374.50 total 40.03 last 73.10) - open/close 3843 (time 1763348384.51 total 50.04 last 81.53) - open/close 4610 (time 1763348394.51 total 60.04 last 76.70) - open/close 5392 (time 1763348404.51 total 70.04 last 78.17) - open/close 6151 (time 1763348414.52 total 80.05 last 75.86) - open/close 6984 (time 1763348424.53 total 90.06 last 83.22) - open/close 7751 (time 1763348434.53 total 100.06 last 76.65) - open/close 8453 (time 1763348444.54 total 110.07 last 70.16) - open/close 9231 (time 1763348454.54 total 120.07 last 77.77) - open/close 9947 (time 1763348464.55 total 130.08 last 71.55) - open/close 10664 (time 1763348474.56 total 140.09 last 71.60) - open/close 11448 (time 1763348484.57 total 150.10 last 78.35) - open/close 12229 (time 1763348494.57 total 160.10 last 78.09) - open/close 12990 (time 1763348504.59 total 170.12 last 75.99) - open/close 13752 (time 1763348514.61 total 180.14 last 76.02) - open/close 14530 (time 1763348524.62 total 190.15 last 77.75) - open/close 15456 (time 1763348534.62 total 200.15 last 92.59) - open/close 16239 (time 1763348544.63 total 210.16 last 78.21) - open/close 17010 (time 1763348554.63 total 220.16 last 77.05) - open/close 17768 (time 1763348564.64 total 230.17 last 75.76) - open/close 18506 (time 1763348574.65 total 240.18 last 73.76) - open/close 19249 (time 1763348584.65 total 250.18 last 74.29) - open/close 19999 (time 1763348594.65 total 260.18 last 74.96) - open/close 20609 (time 1763348604.66 total 270.19 last 60.93) - open/close 21376 (time 1763348614.67 total 280.20 last 76.69) - open/close 22198 (time 1763348624.67 total 290.20 last 82.13) - open/close 22930 (time 1763348634.68 total 300.21 last 73.17) - open/close 23672 (time 1763348644.69 total 310.22 last 74.15) - open/close 24412 (time 1763348654.70 total 320.23 last 73.92) - open/close 25099 (time 1763348664.70 total 330.23 last 68.68) - open/close 25844 (time 1763348674.70 total 340.23 last 74.50) - open/close 26656 (time 1763348684.70 total 350.23 last 81.18) - open/close 27456 (time 1763348694.72 total 360.25 last 79.90) - open/close 28325 (time 1763348704.72 total 370.25 last 86.83) - open/close 29087 (time 1763348714.73 total 380.26 last 76.19) - open/close 29864 (time 1763348724.74 total 390.27 last 77.60) - open/close 30646 (time 1763348734.75 total 400.28 last 78.15) - open/close 31443 (time 1763348744.75 total 410.28 last 79.69) - open/close 32230 (time 1763348754.75 total 420.28 last 78.67) - open/close 32985 (time 1763348764.76 total 430.29 last 75.45) - open/close 33770 (time 1763348774.76 total 440.29 last 78.47) - open/close 34569 (time 1763348784.77 total 450.30 last 79.80) - open/close 35346 (time 1763348794.78 total 460.31 last 77.66) - open/close 36065 (time 1763348804.79 total 470.32 last 71.85) - open/close 36694 (time 1763348814.81 total 480.34 last 62.76) - open/close 37461 (time 1763348824.81 total 490.34 last 76.68) - open/close 38244 (time 1763348834.82 total 500.35 last 78.20) - open/close 38990 (time 1763348844.83 total 510.36 last 74.55) - open/close 39643 (time 1763348854.83 total 520.36 last 65.30) - open/close 40000 (time 1763348860.28 total 525.81 last 65.45) - open/close 40705 (time 1763348870.29 total 535.82 last 70.44) - open/close 41345 (time 1763348880.30 total 545.83 last 63.96) - open/close 42001 (time 1763348890.30 total 555.83 last 65.58) - open/close 42657 (time 1763348900.31 total 565.84 last 65.56) - open/close 43448 (time 1763348910.31 total 575.84 last 79.09) - open/close 44153 (time 1763348920.32 total 585.85 last 70.42) - open/close 44763 (time 1763348930.32 total 595.85 last 60.98) - open/close 45473 (time 1763348940.33 total 605.86 last 70.96) - open/close 46193 (time 1763348950.34 total 615.87 last 71.94) - open/close 46960 (time 1763348960.34 total 625.87 last 76.68) - open/close 47674 (time 1763348970.34 total 635.87 last 71.37) - open/close 48438 (time 1763348980.35 total 645.88 last 76.35) - open/close 49147 (time 1763348990.35 total 655.88 last 70.88) - open/close 49898 (time 1763349000.35 total 665.88 last 75.09) - open/close 50608 (time 1763349010.36 total 675.89 last 70.95) - open/close 51327 (time 1763349020.37 total 685.90 last 71.81) - open/close 52054 (time 1763349030.39 total 695.92 last 72.61) - open/close 52710 (time 1763349040.39 total 705.92 last 65.55) - open/close 53407 (time 1763349050.40 total 715.93 last 69.63) - open/close 53988 (time 1763349060.41 total 725.94 last 58.06) - open/close 54705 (time 1763349070.41 total 735.94 last 71.69) - open/close 55443 (time 1763349080.42 total 745.95 last 73.77) - open/close 56123 (time 1763349090.42 total 755.95 last 67.99) - open/close 56801 (time 1763349100.43 total 765.96 last 67.72) - open/close 57436 (time 1763349110.44 total 775.97 last 63.44) - open/close 58015 (time 1763349120.45 total 785.98 last 57.87) - open/close 58674 (time 1763349130.45 total 795.98 last 65.84) - open/close 59376 (time 1763349140.46 total 805.99 last 70.13) - open/close 60000 (time 1763349150.18 total 815.71 last 64.21) - open/close 60641 (time 1763349160.18 total 825.71 last 64.08) - open/close 61356 (time 1763349170.19 total 835.72 last 71.44) - open/close 62084 (time 1763349180.20 total 845.73 last 72.72) - open/close 62809 (time 1763349190.22 total 855.75 last 72.36) - open/close 63507 (time 1763349200.23 total 865.76 last 69.78) - open/close 64250 (time 1763349210.23 total 875.76 last 74.29) total: 64768 open/close in 883.30 seconds: 73.32 ops/second - /unlink 696 (time 1763349232.27 total 10.02 last 69.48) - /unlink 1431 (time 1763349242.27 total 20.02 last 73.48) - /unlink 2128 (time 1763349252.29 total 30.04 last 69.60) - /unlink 2880 (time 1763349262.29 total 40.04 last 75.16) - /unlink 3569 (time 1763349272.31 total 50.06 last 68.79) - /unlink 4287 (time 1763349282.31 total 60.06 last 71.77) - /unlink 4931 (time 1763349292.33 total 70.08 last 64.28) - /unlink 5562 (time 1763349302.34 total 80.09 last 63.05) - /unlink 6319 (time 1763349312.34 total 90.09 last 75.66) - /unlink 7045 (time 1763349322.35 total 100.10 last 72.55) - /unlink 7756 (time 1763349332.36 total 110.11 last 71.01) - /unlink 8453 (time 1763349342.37 total 120.12 last 69.65) - /unlink 9179 (time 1763349352.37 total 130.12 last 72.58) - /unlink 9886 (time 1763349362.37 total 140.12 last 70.68) - /unlink 10577 (time 1763349372.38 total 150.12 last 69.10) - /unlink 11276 (time 1763349382.40 total 160.15 last 69.72) - /unlink 11900 (time 1763349392.40 total 170.15 last 62.39) - /unlink 12596 (time 1763349402.41 total 180.16 last 69.57) - /unlink 13290 (time 1763349412.42 total 190.17 last 69.30) - /unlink 13974 (time 1763349422.43 total 200.18 last 68.35) - /unlink 14596 (time 1763349432.43 total 210.18 last 62.18) - /unlink 15297 (time 1763349442.44 total 220.19 last 70.06) - /unlink 16003 (time 1763349452.45 total 230.20 last 70.50) - /unlink 16754 (time 1763349462.46 total 240.20 last 75.06) - /unlink 17514 (time 1763349472.46 total 250.21 last 75.96) - /unlink 18291 (time 1763349482.47 total 260.22 last 77.65) - /unlink 19020 (time 1763349492.48 total 270.23 last 72.82) - /unlink 19743 (time 1763349502.49 total 280.24 last 72.18) - /unlink 20000 (time 1763349506.06 total 283.81 last 72.05) - /unlink 20717 (time 1763349516.07 total 293.82 last 71.63) - /unlink 21399 (time 1763349526.08 total 303.83 last 68.14) - /unlink 22029 (time 1763349536.09 total 313.84 last 62.94) - /unlink 22654 (time 1763349546.10 total 323.85 last 62.45) - /unlink 23344 (time 1763349556.10 total 333.85 last 68.99) - /unlink 24020 (time 1763349566.11 total 343.86 last 67.52) - /unlink 24690 (time 1763349576.11 total 353.86 last 66.98) - /unlink 25409 (time 1763349586.12 total 363.87 last 71.84) - /unlink 26048 (time 1763349596.13 total 373.87 last 63.88) - /unlink 26727 (time 1763349606.13 total 383.88 last 67.88) - /unlink 27435 (time 1763349616.15 total 393.90 last 70.65) - /unlink 28097 (time 1763349626.16 total 403.91 last 66.13) - /unlink 28688 (time 1763349636.17 total 413.91 last 59.07) - /unlink 29343 (time 1763349646.17 total 423.92 last 65.49) - /unlink 30000 (time 1763349655.52 total 433.27 last 70.27) - /unlink 30671 (time 1763349665.52 total 443.27 last 67.05) - /unlink 31407 (time 1763349675.52 total 453.27 last 73.59) - /unlink 32099 (time 1763349685.53 total 463.28 last 69.15) - /unlink 32803 (time 1763349695.55 total 473.30 last 70.31) - /unlink 33530 (time 1763349705.55 total 483.30 last 72.67) - /unlink 34230 (time 1763349715.56 total 493.31 last 69.93) - /unlink 34929 (time 1763349725.56 total 503.31 last 69.88) - /unlink 35614 (time 1763349735.57 total 513.32 last 68.47) - /unlink 36288 (time 1763349745.57 total 523.32 last 67.37) - /unlink 36972 (time 1763349755.58 total 533.33 last 68.33) - /unlink 37571 (time 1763349765.58 total 543.33 last 59.89) - /unlink 38277 (time 1763349775.58 total 553.33 last 70.60) - /unlink 38950 (time 1763349785.59 total 563.34 last 67.25) - /unlink 39733 (time 1763349795.60 total 573.35 last 78.24) - /unlink 40000 (time 1763349799.19 total 576.94 last 74.38) - /unlink 40718 (time 1763349809.19 total 586.94 last 71.77) - /unlink 41415 (time 1763349819.20 total 596.94 last 69.67) - /unlink 42140 (time 1763349829.20 total 606.95 last 72.46) - /unlink 42883 (time 1763349839.20 total 616.95 last 74.29) - /unlink 43654 (time 1763349849.21 total 626.96 last 77.06) - /unlink 44461 (time 1763349859.21 total 636.96 last 80.67) - /unlink 45176 (time 1763349869.21 total 646.96 last 71.49) - /unlink 45896 (time 1763349879.22 total 656.97 last 71.97) - /unlink 46630 (time 1763349889.22 total 666.97 last 73.34) - /unlink 47381 (time 1763349899.23 total 676.98 last 75.04) - /unlink 48120 (time 1763349909.24 total 686.99 last 73.85) - /unlink 48908 (time 1763349919.24 total 696.99 last 78.78) - /unlink 49636 (time 1763349929.26 total 707.01 last 72.66) - /unlink 50000 (time 1763349935.41 total 713.16 last 59.21) - /unlink 50691 (time 1763349945.42 total 723.16 last 69.05) - /unlink 51386 (time 1763349955.42 total 733.17 last 69.48) - /unlink 52107 (time 1763349965.42 total 743.17 last 72.08) - /unlink 52813 (time 1763349975.43 total 753.18 last 70.56) - /unlink 53585 (time 1763349985.44 total 763.19 last 77.12) - /unlink 54336 (time 1763349995.44 total 773.19 last 75.10) - /unlink 55034 (time 1763350005.45 total 783.20 last 69.71) - /unlink 55646 (time 1763350015.47 total 793.22 last 61.10) - /unlink 56254 (time 1763350025.47 total 803.22 last 60.79) - /unlink 56947 (time 1763350035.47 total 813.22 last 69.29) - /unlink 57706 (time 1763350045.50 total 823.25 last 75.67) - /unlink 58313 (time 1763350055.52 total 833.27 last 60.57) - /unlink 58964 (time 1763350065.53 total 843.28 last 65.02) - /unlink 59650 (time 1763350075.54 total 853.29 last 68.54) - /unlink 60000 (time 1763350080.70 total 858.45 last 67.84) - /unlink 60651 (time 1763350090.71 total 868.46 last 65.05) - /unlink 61392 (time 1763350100.72 total 878.46 last 74.06) - /unlink 62138 (time 1763350110.72 total 888.46 last 74.60) - /unlink 62917 (time 1763350120.72 total 898.47 last 77.89) - /unlink 63710 (time 1763350130.73 total 908.48 last 79.20) - /unlink 64498 (time 1763350140.74 total 918.49 last 78.74) total: 64768 /unlink in 921.84 seconds: 70.26 ops/second Waiting for MDT destroys to complete oleg239-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 - open/close 730 (time 1763350174.80 total 10.00 last 72.97) - open/close 1463 (time 1763350184.80 total 20.00 last 73.29) - open/close 2158 (time 1763350194.81 total 30.01 last 69.44) - open/close 2824 (time 1763350204.82 total 40.02 last 66.53) - open/close 3327 (time 1763350214.83 total 50.04 last 50.23) - open/close 3888 (time 1763350224.85 total 60.05 last 56.00) - open/close 4588 (time 1763350234.85 total 70.06 last 69.99) - open/close 5170 (time 1763350244.87 total 80.08 last 58.07) - open/close 5784 (time 1763350254.90 total 90.10 last 61.26) - open/close 6384 (time 1763350264.90 total 100.11 last 59.98) - open/close 7034 (time 1763350274.90 total 110.11 last 64.99) - open/close 7760 (time 1763350284.92 total 120.13 last 72.44) - open/close 8469 (time 1763350294.93 total 130.13 last 70.86) - open/close 9237 (time 1763350304.94 total 140.15 last 76.69) - open/close 9936 (time 1763350314.94 total 150.15 last 69.89) - open/close 10642 (time 1763350324.95 total 160.16 last 70.56) - open/close 11387 (time 1763350334.95 total 170.16 last 74.47) - open/close 12094 (time 1763350344.96 total 180.17 last 70.62) - open/close 12644 (time 1763350354.99 total 190.19 last 54.87) - open/close 13164 (time 1763350365.02 total 200.22 last 51.84) - open/close 13752 (time 1763350375.03 total 210.23 last 58.74) - open/close 14481 (time 1763350385.04 total 220.25 last 72.80) - open/close 15271 (time 1763350395.05 total 230.25 last 78.97) - open/close 15980 (time 1763350405.05 total 240.26 last 70.86) - open/close 16750 (time 1763350415.05 total 250.26 last 76.98) - open/close 17491 (time 1763350425.06 total 260.27 last 74.03) - open/close 18244 (time 1763350435.06 total 270.27 last 75.30) - open/close 18960 (time 1763350445.07 total 280.27 last 71.60) - open/close 19664 (time 1763350455.08 total 290.29 last 70.28) - open/close 20000 (time 1763350459.82 total 295.02 last 70.96) - open/close 20672 (time 1763350469.84 total 305.04 last 67.06) - open/close 21375 (time 1763350479.84 total 315.05 last 70.27) - open/close 22132 (time 1763350489.86 total 325.06 last 75.59) - open/close 22827 (time 1763350499.86 total 335.06 last 69.49) - open/close 23549 (time 1763350509.87 total 345.07 last 72.12) - open/close 24252 (time 1763350519.88 total 355.09 last 70.21) - open/close 24999 (time 1763350529.88 total 365.09 last 74.68) - open/close 25741 (time 1763350539.89 total 375.09 last 74.18) - open/close 26436 (time 1763350549.89 total 385.09 last 69.49) - open/close 27075 (time 1763350559.89 total 395.10 last 63.87) - open/close 27764 (time 1763350569.89 total 405.10 last 68.90) - open/close 28511 (time 1763350579.90 total 415.11 last 74.62) - open/close 29270 (time 1763350589.91 total 425.11 last 75.86) - open/close 29973 (time 1763350599.92 total 435.12 last 70.23) - open/close 30607 (time 1763350609.92 total 445.13 last 63.37) - open/close 31375 (time 1763350619.94 total 455.15 last 76.67) - open/close 32056 (time 1763350629.95 total 465.16 last 68.02) - open/close 32804 (time 1763350639.96 total 475.17 last 74.74) - open/close 33476 (time 1763350649.97 total 485.18 last 67.11) - open/close 34170 (time 1763350659.98 total 495.19 last 69.34) - open/close 34880 (time 1763350669.98 total 505.19 last 70.99) - open/close 35658 (time 1763350679.99 total 515.19 last 77.78) - open/close 36395 (time 1763350689.99 total 525.20 last 73.68) - open/close 37131 (time 1763350700.00 total 535.20 last 73.55) - open/close 37811 (time 1763350710.01 total 545.21 last 67.94) - open/close 38602 (time 1763350720.02 total 555.22 last 79.03) - open/close 39378 (time 1763350730.02 total 565.22 last 77.60) - open/close 40000 (time 1763350737.83 total 573.03 last 79.64) - open/close 40781 (time 1763350747.84 total 583.04 last 78.01) - open/close 41552 (time 1763350757.84 total 593.04 last 77.09) - open/close 42324 (time 1763350767.84 total 603.05 last 77.18) - open/close 42950 (time 1763350777.85 total 613.05 last 62.56) - open/close 43687 (time 1763350787.85 total 623.06 last 73.64) - open/close 44378 (time 1763350797.86 total 633.07 last 69.06) - open/close 45129 (time 1763350807.88 total 643.08 last 74.96) - open/close 45848 (time 1763350817.88 total 653.09 last 71.88) - open/close 46588 (time 1763350827.89 total 663.10 last 73.91) - open/close 47407 (time 1763350837.90 total 673.10 last 81.86) - open/close 48180 (time 1763350847.91 total 683.12 last 77.21) - open/close 48914 (time 1763350857.92 total 693.12 last 73.33) - open/close 49628 (time 1763350867.92 total 703.12 last 71.40) - open/close 50000 (time 1763350872.74 total 707.95 last 77.10) - open/close 50697 (time 1763350882.75 total 717.95 last 69.69) - open/close 51454 (time 1763350892.76 total 727.97 last 75.59) - open/close 52147 (time 1763350902.79 total 737.99 last 69.12) - open/close 52941 (time 1763350912.79 total 748.00 last 79.37) - open/close 53717 (time 1763350922.81 total 758.01 last 77.49) - open/close 54471 (time 1763350932.81 total 768.02 last 75.34) - open/close 55203 (time 1763350942.82 total 778.02 last 73.16) - open/close 55950 (time 1763350952.83 total 788.03 last 74.65) - open/close 56680 (time 1763350962.83 total 798.03 last 72.98) - open/close 57442 (time 1763350972.83 total 808.04 last 76.17) - open/close 58169 (time 1763350982.83 total 818.04 last 72.69) - open/close 58859 (time 1763350992.83 total 828.04 last 68.99) - open/close 59576 (time 1763351002.86 total 838.06 last 71.53) - open/close 60000 (time 1763351009.16 total 844.37 last 67.26) - open/close 60781 (time 1763351019.17 total 854.38 last 78.05) - open/close 61573 (time 1763351029.17 total 864.38 last 79.18) - open/close 62317 (time 1763351039.19 total 874.39 last 74.29) - open/close 63072 (time 1763351049.20 total 884.41 last 75.38) - open/close 63859 (time 1763351059.20 total 894.41 last 78.69) - open/close 64648 (time 1763351069.21 total 904.42 last 78.86) total: 64768 open/close in 906.02 seconds: 71.49 ops/second - /unlink 799 (time 1763351084.96 total 10.01 last 79.84) - /unlink 1600 (time 1763351094.96 total 20.01 last 80.08) - /unlink 2368 (time 1763351104.97 total 30.01 last 76.76) - /unlink 3084 (time 1763351114.98 total 40.03 last 71.50) - /unlink 3846 (time 1763351124.99 total 50.04 last 76.09) - /unlink 4693 (time 1763351135.00 total 60.05 last 84.63) - /unlink 5526 (time 1763351145.02 total 70.07 last 83.17) - /unlink 6354 (time 1763351155.03 total 80.08 last 82.70) - /unlink 7076 (time 1763351165.04 total 90.09 last 72.15) - /unlink 7741 (time 1763351175.05 total 100.09 last 66.45) - /unlink 8414 (time 1763351185.06 total 110.10 last 67.22) - /unlink 9088 (time 1763351195.06 total 120.11 last 67.36) - /unlink 9771 (time 1763351205.07 total 130.12 last 68.26) - /unlink 10000 (time 1763351208.12 total 133.17 last 74.98) - /unlink 10748 (time 1763351218.13 total 143.18 last 74.74) - /unlink 11487 (time 1763351228.14 total 153.18 last 73.85) - /unlink 12178 (time 1763351238.14 total 163.19 last 69.05) - /unlink 12940 (time 1763351248.15 total 173.20 last 76.12) - /unlink 13672 (time 1763351258.17 total 183.21 last 73.12) - /unlink 14404 (time 1763351268.19 total 193.24 last 73.03) - /unlink 15179 (time 1763351278.20 total 203.24 last 77.44) - /unlink 15930 (time 1763351288.21 total 213.26 last 75.01) - /unlink 16685 (time 1763351298.21 total 223.26 last 75.48) - /unlink 17429 (time 1763351308.22 total 233.27 last 74.35) - /unlink 18172 (time 1763351318.23 total 243.28 last 74.20) - /unlink 18842 (time 1763351328.24 total 253.29 last 66.91) - /unlink 19556 (time 1763351338.25 total 263.30 last 71.34) - /unlink 20000 (time 1763351344.19 total 269.24 last 74.80) - /unlink 20709 (time 1763351354.20 total 279.25 last 70.82) - /unlink 21462 (time 1763351364.21 total 289.26 last 75.21) - /unlink 22243 (time 1763351374.22 total 299.26 last 78.06) - /unlink 22998 (time 1763351384.22 total 309.27 last 75.45) - /unlink 23621 (time 1763351394.23 total 319.28 last 62.25) - /unlink 24290 (time 1763351404.23 total 329.28 last 66.87) - /unlink 24981 (time 1763351414.26 total 339.30 last 68.95) - /unlink 25665 (time 1763351424.26 total 349.31 last 68.35) - /unlink 26368 (time 1763351434.27 total 359.32 last 70.28) - /unlink 27066 (time 1763351444.27 total 369.32 last 69.77) - /unlink 27764 (time 1763351454.28 total 379.33 last 69.74) - /unlink 28474 (time 1763351464.28 total 389.33 last 70.98) - /unlink 29204 (time 1763351474.29 total 399.33 last 72.98) - /unlink 29948 (time 1763351484.29 total 409.34 last 74.38) - /unlink 30709 (time 1763351494.29 total 419.34 last 76.08) - /unlink 31365 (time 1763351504.30 total 429.35 last 65.54) - /unlink 32102 (time 1763351514.31 total 439.36 last 73.59) - /unlink 32864 (time 1763351524.32 total 449.37 last 76.16) - /unlink 33608 (time 1763351534.32 total 459.37 last 74.37) - /unlink 34331 (time 1763351544.33 total 469.37 last 72.28) - /unlink 35067 (time 1763351554.33 total 479.38 last 73.56) - /unlink 35840 (time 1763351564.33 total 489.38 last 77.29) - /unlink 36603 (time 1763351574.34 total 499.39 last 76.23) - /unlink 37350 (time 1763351584.35 total 509.39 last 74.66) - /unlink 38136 (time 1763351594.35 total 519.40 last 78.57) - /unlink 38842 (time 1763351604.36 total 529.41 last 70.51) - /unlink 39645 (time 1763351614.37 total 539.41 last 80.27) - /unlink 40000 (time 1763351618.83 total 543.88 last 79.53) - /unlink 40732 (time 1763351628.83 total 553.88 last 73.19) - /unlink 41488 (time 1763351638.84 total 563.89 last 75.56) - /unlink 42317 (time 1763351648.84 total 573.89 last 82.85) - /unlink 43112 (time 1763351658.85 total 583.90 last 79.46) - /unlink 43915 (time 1763351668.86 total 593.91 last 80.22) - /unlink 44680 (time 1763351678.87 total 603.91 last 76.44) - /unlink 45422 (time 1763351688.88 total 613.93 last 74.10) - /unlink 46223 (time 1763351698.89 total 623.94 last 79.99) - /unlink 46972 (time 1763351708.90 total 633.95 last 74.86) - /unlink 47707 (time 1763351718.90 total 643.95 last 73.48) - /unlink 48476 (time 1763351728.90 total 653.95 last 76.88) - /unlink 49164 (time 1763351738.91 total 663.96 last 68.77) - /unlink 49825 (time 1763351748.92 total 673.96 last 66.04) - /unlink 50000 (time 1763351751.46 total 676.51 last 68.70) - /unlink 50713 (time 1763351761.47 total 686.52 last 71.25) - /unlink 51490 (time 1763351771.47 total 696.52 last 77.69) - /unlink 52263 (time 1763351781.49 total 706.53 last 77.20) - /unlink 53051 (time 1763351791.49 total 716.54 last 78.77) - /unlink 53803 (time 1763351801.51 total 726.55 last 75.07) - /unlink 54586 (time 1763351811.51 total 736.56 last 78.27) - /unlink 55335 (time 1763351821.51 total 746.56 last 74.90) - /unlink 56169 (time 1763351831.52 total 756.57 last 83.30) - /unlink 56983 (time 1763351841.53 total 766.58 last 81.34) - /unlink 57796 (time 1763351851.54 total 776.59 last 81.21) - /unlink 58596 (time 1763351861.55 total 786.59 last 79.97) - /unlink 59361 (time 1763351871.55 total 796.60 last 76.45) - /unlink 60000 (time 1763351879.50 total 804.55 last 80.41) - /unlink 60872 (time 1763351889.50 total 814.55 last 87.17) - /unlink 61657 (time 1763351899.50 total 824.55 last 78.49) - /unlink 62406 (time 1763351909.51 total 834.56 last 74.88) - /unlink 63216 (time 1763351919.51 total 844.56 last 80.96) - /unlink 63989 (time 1763351929.52 total 854.56 last 77.27) total: 64768 /unlink in 864.30 seconds: 74.94 ops/second Waiting for MDT destroys to complete oleg239-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 - open/close 900 (time 1763351960.05 total 10.00 last 89.96) - open/close 1731 (time 1763351970.05 total 20.00 last 83.10) - open/close 2642 (time 1763351980.05 total 30.01 last 91.02) - open/close 3478 (time 1763351990.07 total 40.03 last 83.49) - open/close 4305 (time 1763352000.07 total 50.03 last 82.70) - open/close 5098 (time 1763352010.07 total 60.03 last 79.29) - open/close 5934 (time 1763352020.09 total 70.05 last 83.39) - open/close 6803 (time 1763352030.10 total 80.05 last 86.89) - open/close 7668 (time 1763352040.10 total 90.06 last 86.46) - open/close 8538 (time 1763352050.11 total 100.07 last 86.88) - open/close 9581 (time 1763352060.12 total 110.08 last 104.21) - open/close 10000 (time 1763352064.93 total 114.89 last 87.13) - open/close 10931 (time 1763352074.93 total 124.89 last 93.09) - open/close 11878 (time 1763352084.93 total 134.89 last 94.69) - open/close 12890 (time 1763352094.94 total 144.90 last 101.12) - open/close 13931 (time 1763352104.94 total 154.90 last 104.06) - open/close 14907 (time 1763352114.95 total 164.91 last 97.57) - open/close 15790 (time 1763352124.95 total 174.91 last 88.26) - open/close 16759 (time 1763352134.96 total 184.92 last 96.85) - open/close 17783 (time 1763352144.96 total 194.92 last 102.38) - open/close 18749 (time 1763352154.96 total 204.92 last 96.55) - open/close 19815 (time 1763352164.97 total 214.92 last 106.57) - open/close 20744 (time 1763352174.97 total 224.93 last 92.87) - open/close 21737 (time 1763352184.97 total 234.93 last 99.24) - open/close 22699 (time 1763352194.98 total 244.93 last 96.19) - open/close 23543 (time 1763352204.98 total 254.93 last 84.40) - open/close 24402 (time 1763352214.98 total 264.94 last 85.90) - open/close 25475 (time 1763352224.98 total 274.94 last 107.27) - open/close 26499 (time 1763352234.98 total 284.94 last 102.38) - open/close 27613 (time 1763352244.98 total 294.94 last 111.36) - open/close 28863 (time 1763352254.99 total 304.95 last 124.92) - open/close 29880 (time 1763352264.99 total 314.95 last 101.70) - open/close 31025 (time 1763352274.99 total 324.95 last 114.47) - open/close 32318 (time 1763352285.00 total 334.96 last 129.28) - open/close 33653 (time 1763352295.00 total 344.96 last 133.49) - open/close 34876 (time 1763352305.01 total 354.97 last 122.18) - open/close 36270 (time 1763352315.01 total 364.97 last 139.33) - open/close 37439 (time 1763352325.02 total 374.98 last 116.84) - open/close 38622 (time 1763352335.02 total 384.98 last 118.29) - open/close 39877 (time 1763352345.02 total 394.98 last 125.48) - open/close 41144 (time 1763352355.02 total 404.98 last 126.68) - open/close 42385 (time 1763352365.03 total 414.99 last 123.97) - open/close 43604 (time 1763352375.04 total 424.99 last 121.86) - open/close 45051 (time 1763352385.04 total 435.00 last 144.67) - open/close 46566 (time 1763352395.04 total 445.00 last 151.40) - open/close 47975 (time 1763352405.05 total 455.01 last 140.84) - open/close 49316 (time 1763352415.05 total 465.01 last 134.09) - open/close 50000 (time 1763352420.56 total 470.52 last 124.18) - open/close 51176 (time 1763352430.56 total 480.52 last 117.55) - open/close 52375 (time 1763352440.57 total 490.53 last 119.79) - open/close 53919 (time 1763352450.57 total 500.53 last 154.40) - open/close 55761 (time 1763352460.58 total 510.54 last 184.08) - open/close 57596 (time 1763352470.58 total 520.54 last 183.43) - open/close 59266 (time 1763352480.59 total 530.54 last 166.92) - open/close 60000 (time 1763352483.42 total 533.38 last 259.03) - open/close 61739 (time 1763352493.42 total 543.38 last 173.84) - open/close 62987 (time 1763352503.42 total 553.38 last 124.80) - open/close 64356 (time 1763352513.42 total 563.38 last 136.90) total: 64768 open/close in 566.92 seconds: 114.25 ops/second - /unlink 2135 (time 1763352529.17 total 10.00 last 213.45) - /unlink 4701 (time 1763352539.17 total 20.00 last 256.59) - /unlink 7476 (time 1763352549.17 total 30.01 last 277.42) - /unlink 10000 (time 1763352557.40 total 38.24 last 306.62) - /unlink 12600 (time 1763352567.41 total 48.24 last 259.87) - /unlink 15058 (time 1763352577.41 total 58.24 last 245.79) - /unlink 17761 (time 1763352587.41 total 68.24 last 270.24) - /unlink 20000 (time 1763352594.90 total 75.74 last 298.90) - /unlink 22580 (time 1763352604.90 total 85.74 last 257.94) - /unlink 25101 (time 1763352614.90 total 95.74 last 252.10) - /unlink 27481 (time 1763352624.91 total 105.74 last 237.93) - /unlink 30000 (time 1763352634.89 total 115.72 last 252.39) - /unlink 32319 (time 1763352644.89 total 125.73 last 231.80) - /unlink 34602 (time 1763352654.90 total 135.73 last 228.21) - /unlink 36777 (time 1763352664.90 total 145.73 last 217.48) - /unlink 39028 (time 1763352674.90 total 155.73 last 225.03) - /unlink 40000 (time 1763352678.57 total 159.40 last 265.02) - /unlink 43000 (time 1763352688.57 total 169.40 last 299.97) - /unlink 45353 (time 1763352698.57 total 179.40 last 235.24) - /unlink 47671 (time 1763352708.57 total 189.41 last 231.72) - /unlink 50000 (time 1763352717.32 total 198.15 last 266.39) - /unlink 52494 (time 1763352727.32 total 208.15 last 249.31) - /unlink 54867 (time 1763352737.32 total 218.16 last 237.29) - /unlink 57488 (time 1763352747.32 total 228.16 last 262.08) - /unlink 60000 (time 1763352756.89 total 237.72 last 262.58) - /unlink 62636 (time 1763352766.89 total 247.72 last 263.53) total: 64768 /unlink in 257.15 seconds: 251.87 ops/second Waiting for MDT destroys to complete oleg239-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 - open/close 1455 (time 1763352797.52 total 10.01 last 145.42) - open/close 3049 (time 1763352807.52 total 20.01 last 159.37) - open/close 4533 (time 1763352817.53 total 30.01 last 148.35) - open/close 6371 (time 1763352827.53 total 40.01 last 183.80) - open/close 7977 (time 1763352837.53 total 50.02 last 160.45) - open/close 9737 (time 1763352847.54 total 60.02 last 175.95) - open/close 11303 (time 1763352857.54 total 70.03 last 156.50) - open/close 13018 (time 1763352867.55 total 80.03 last 171.42) - open/close 14697 (time 1763352877.55 total 90.04 last 167.84) - open/close 16416 (time 1763352887.55 total 100.04 last 171.90) - open/close 18217 (time 1763352897.56 total 110.04 last 180.03) - open/close 20000 (time 1763352906.27 total 118.75 last 204.63) - open/close 21458 (time 1763352916.27 total 128.75 last 145.80) - open/close 22888 (time 1763352926.27 total 138.76 last 142.96) - open/close 24282 (time 1763352936.28 total 148.76 last 139.32) - open/close 25628 (time 1763352946.28 total 158.77 last 134.55) - open/close 27581 (time 1763352956.28 total 168.77 last 195.27) - open/close 29285 (time 1763352966.28 total 178.77 last 170.40) - open/close 30000 (time 1763352970.62 total 183.11 last 164.79) - open/close 31734 (time 1763352980.63 total 193.11 last 173.30) - open/close 33206 (time 1763352990.63 total 203.11 last 147.18) - open/close 34954 (time 1763353000.63 total 213.12 last 174.76) - open/close 36557 (time 1763353010.66 total 223.15 last 159.78) - open/close 38247 (time 1763353020.67 total 233.15 last 168.99) - open/close 39889 (time 1763353030.67 total 243.15 last 164.15) - open/close 41587 (time 1763353040.67 total 253.15 last 169.79) - open/close 43175 (time 1763353050.67 total 263.16 last 158.73) - open/close 45087 (time 1763353060.68 total 273.16 last 191.15) - open/close 46790 (time 1763353070.68 total 283.16 last 170.30) - open/close 48529 (time 1763353080.68 total 293.16 last 173.87) - open/close 50000 (time 1763353086.09 total 298.58 last 271.58) - open/close 52460 (time 1763353096.10 total 308.58 last 245.93) - open/close 54511 (time 1763353106.10 total 318.59 last 204.94) - open/close 56299 (time 1763353116.11 total 328.59 last 178.73) - open/close 58071 (time 1763353126.11 total 338.60 last 177.19) - open/close 60000 (time 1763353134.95 total 347.43 last 218.22) - open/close 61717 (time 1763353144.95 total 357.44 last 171.64) - open/close 63782 (time 1763353154.96 total 367.44 last 206.42) total: 64768 open/close in 372.53 seconds: 173.86 ops/second - /unlink 3861 (time 1763353171.88 total 10.00 last 386.05) - /unlink 7766 (time 1763353181.89 total 20.00 last 390.42) - /unlink 10000 (time 1763353189.35 total 27.47 last 299.29) - /unlink 13307 (time 1763353199.35 total 37.47 last 330.61) - /unlink 16562 (time 1763353209.35 total 47.47 last 325.46) - /unlink 20000 (time 1763353219.02 total 57.14 last 355.62) - /unlink 23986 (time 1763353229.02 total 67.14 last 398.49) - /unlink 29360 (time 1763353239.03 total 77.14 last 537.31) - /unlink 33447 (time 1763353249.03 total 87.14 last 408.66) - /unlink 37619 (time 1763353259.03 total 97.14 last 417.19) - /unlink 40000 (time 1763353265.00 total 103.12 last 398.43) - /unlink 43967 (time 1763353275.01 total 113.12 last 396.58) - /unlink 47142 (time 1763353285.01 total 123.12 last 317.50) - /unlink 50000 (time 1763353293.29 total 131.41 last 345.08) - /unlink 53868 (time 1763353303.29 total 141.41 last 386.75) - /unlink 58033 (time 1763353313.29 total 151.41 last 416.41) - /unlink 60000 (time 1763353317.68 total 155.80 last 448.02) - /unlink 63104 (time 1763353327.68 total 165.80 last 310.37) total: 64768 /unlink in 169.94 seconds: 381.12 ops/second Waiting for MDT destroys to complete oleg239-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 - open/close 2286 (time 1763353352.98 total 10.00 last 228.56) - open/close 4452 (time 1763353362.99 total 20.01 last 216.46) - open/close 6654 (time 1763353372.99 total 30.01 last 220.15) - open/close 9634 (time 1763353382.99 total 40.01 last 297.98) - open/close 12122 (time 1763353392.99 total 50.01 last 248.78) - open/close 14763 (time 1763353403.00 total 60.02 last 263.96) - open/close 16778 (time 1763353413.00 total 70.02 last 201.47) - open/close 18780 (time 1763353423.00 total 80.02 last 200.16) - open/close 20000 (time 1763353429.65 total 86.67 last 183.49) - open/close 22536 (time 1763353439.65 total 96.67 last 253.60) - open/close 24771 (time 1763353449.65 total 106.67 last 223.44) - open/close 27320 (time 1763353459.66 total 116.67 last 254.88) - open/close 30000 (time 1763353468.61 total 125.63 last 299.17) - open/close 32090 (time 1763353478.62 total 135.64 last 208.90) - open/close 34165 (time 1763353488.62 total 145.64 last 207.49) - open/close 36589 (time 1763353498.62 total 155.64 last 242.35) - open/close 38663 (time 1763353508.62 total 165.64 last 207.38) - open/close 40000 (time 1763353513.75 total 170.77 last 260.60) - open/close 42745 (time 1763353523.75 total 180.77 last 274.45) - open/close 45576 (time 1763353533.75 total 190.77 last 283.08) - open/close 47925 (time 1763353543.76 total 200.78 last 234.82) - open/close 50000 (time 1763353552.63 total 209.65 last 233.86) - open/close 52855 (time 1763353562.63 total 219.65 last 285.44) - open/close 55322 (time 1763353572.64 total 229.65 last 246.62) - open/close 57906 (time 1763353582.64 total 239.66 last 258.29) - open/close 60000 (time 1763353588.38 total 245.40 last 364.95) - open/close 62279 (time 1763353598.38 total 255.40 last 227.87) - open/close 64334 (time 1763353608.39 total 265.40 last 205.36) total: 64768 open/close in 267.39 seconds: 242.22 ops/second - /unlink 5735 (time 1763353621.99 total 10.00 last 573.47) - /unlink 10000 (time 1763353627.71 total 15.73 last 744.56) - /unlink 17536 (time 1763353637.71 total 25.73 last 753.58) - /unlink 20000 (time 1763353640.77 total 28.78 last 806.38) - /unlink 27603 (time 1763353650.77 total 38.78 last 760.29) - /unlink 30000 (time 1763353654.06 total 42.08 last 727.93) - /unlink 37163 (time 1763353664.06 total 52.08 last 716.27) - /unlink 40000 (time 1763353667.84 total 55.85 last 751.30) - /unlink 47728 (time 1763353677.84 total 65.85 last 772.76) - /unlink 50000 (time 1763353680.53 total 68.55 last 843.50) - /unlink 58133 (time 1763353690.53 total 78.55 last 813.26) - /unlink 60000 (time 1763353692.95 total 80.97 last 771.28) total: 64768 /unlink in 87.17 seconds: 743.05 ops/second Waiting for MDT destroys to complete oleg239-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg239-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg239-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg239-server stop mds service on oleg239-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg239-server stop mds service on oleg239-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg239-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg239-server' oleg239-server: oleg239-server.virtnet: executing unload_modules_local oleg239-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg239-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg239-client: oleg239-client: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-server: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-client: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-server: ssh exited with exit code 2