== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:53:57 (1754405637) Stopping clients: oleg109-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg109-client.virtnet /mnt/lustre2 (opts:-f) oleg109-server: oleg109-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg109-server' oleg109-server: oleg109-server.virtnet: executing load_modules_local oleg109-server: Loading modules from /home/green/git/lustre-release/lustre oleg109-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg109-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg109-server: detected 4 online CPUs by sysfs oleg109-server: MODOPTS_LIBCFS= oleg109-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg109-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg109-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg109-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 761 (time 1754405746.17 total 10.01 last 76.05) - open/close 1530 (time 1754405756.17 total 20.01 last 76.88) - open/close 2296 (time 1754405766.18 total 30.01 last 76.57) - open/close 3087 (time 1754405776.18 total 40.02 last 79.09) - open/close 3925 (time 1754405786.18 total 50.02 last 83.80) - open/close 4684 (time 1754405796.18 total 60.02 last 75.85) - open/close 5540 (time 1754405806.19 total 70.03 last 85.57) - open/close 6488 (time 1754405816.20 total 80.04 last 94.70) - open/close 7345 (time 1754405826.20 total 90.04 last 85.69) - open/close 8269 (time 1754405836.21 total 100.05 last 92.28) - open/close 9087 (time 1754405846.23 total 110.06 last 81.69) - open/close 9867 (time 1754405856.23 total 120.07 last 77.98) - open/close 10634 (time 1754405866.24 total 130.08 last 76.60) - open/close 11417 (time 1754405876.25 total 140.09 last 78.22) - open/close 12259 (time 1754405886.26 total 150.09 last 84.18) - open/close 13166 (time 1754405896.26 total 160.10 last 90.65) - open/close 14005 (time 1754405906.27 total 170.11 last 83.82) - open/close 14958 (time 1754405916.28 total 180.12 last 95.24) - open/close 15951 (time 1754405926.28 total 190.12 last 99.26) - open/close 16949 (time 1754405936.28 total 200.12 last 99.78) - open/close 17873 (time 1754405946.29 total 210.13 last 92.33) - open/close 19121 (time 1754405956.30 total 220.14 last 124.70) - open/close 20000 (time 1754405962.65 total 226.49 last 138.33) - open/close 21328 (time 1754405972.65 total 236.49 last 132.79) - open/close 22607 (time 1754405982.66 total 246.50 last 127.87) - open/close 23630 (time 1754405992.66 total 256.50 last 102.27) - open/close 24567 (time 1754406002.67 total 266.51 last 93.60) - open/close 25440 (time 1754406012.68 total 276.51 last 87.24) - open/close 26430 (time 1754406022.68 total 286.52 last 98.95) - open/close 27389 (time 1754406032.69 total 296.53 last 95.81) - open/close 28334 (time 1754406042.70 total 306.54 last 94.45) - open/close 29414 (time 1754406052.71 total 316.54 last 107.90) - open/close 30000 (time 1754406058.03 total 321.87 last 110.10) - open/close 30982 (time 1754406068.04 total 331.87 last 98.14) - open/close 31925 (time 1754406078.04 total 341.87 last 94.28) - open/close 32926 (time 1754406088.04 total 351.88 last 100.03) - open/close 33894 (time 1754406098.05 total 361.88 last 96.78) - open/close 34891 (time 1754406108.05 total 371.89 last 99.68) - open/close 35879 (time 1754406118.05 total 381.89 last 98.77) - open/close 36852 (time 1754406128.05 total 391.89 last 97.27) - open/close 37863 (time 1754406138.06 total 401.89 last 101.09) - open/close 38789 (time 1754406148.06 total 411.90 last 92.56) - open/close 39778 (time 1754406158.07 total 421.91 last 98.82) - open/close 40000 (time 1754406160.26 total 424.10 last 101.32) - open/close 40934 (time 1754406170.26 total 434.10 last 93.35) - open/close 41971 (time 1754406180.27 total 444.11 last 103.64) - open/close 43015 (time 1754406190.27 total 454.11 last 104.37) - open/close 44095 (time 1754406200.28 total 464.12 last 107.87) - open/close 45136 (time 1754406210.29 total 474.13 last 104.01) - open/close 46496 (time 1754406220.29 total 484.13 last 135.98) - open/close 47583 (time 1754406230.29 total 494.13 last 108.69) - open/close 48597 (time 1754406240.30 total 504.14 last 101.37) - open/close 49713 (time 1754406250.30 total 514.14 last 111.53) - open/close 50000 (time 1754406253.46 total 517.30 last 90.98) - open/close 51029 (time 1754406263.46 total 527.30 last 102.90) - open/close 52076 (time 1754406273.47 total 537.30 last 104.63) - open/close 53184 (time 1754406283.47 total 547.31 last 110.71) - open/close 54330 (time 1754406293.48 total 557.32 last 114.56) - open/close 55643 (time 1754406303.48 total 567.32 last 131.20) - open/close 57122 (time 1754406313.49 total 577.32 last 147.87) - open/close 58673 (time 1754406323.49 total 587.33 last 155.07) - open/close 60000 (time 1754406332.03 total 595.86 last 155.45) - open/close 61668 (time 1754406342.03 total 605.87 last 166.74) - open/close 63102 (time 1754406352.03 total 615.87 last 143.37) - open/close 64376 (time 1754406362.03 total 625.87 last 127.39) total: 64768 open/close in 629.07 seconds: 102.96 ops/second - /unlink 1200 (time 1754406377.21 total 10.01 last 119.93) - /unlink 2445 (time 1754406387.22 total 20.01 last 124.48) - /unlink 3809 (time 1754406397.22 total 30.01 last 136.38) - /unlink 5311 (time 1754406407.22 total 40.01 last 150.15) - /unlink 7801 (time 1754406417.22 total 50.02 last 248.92) - /unlink 10000 (time 1754406423.88 total 56.67 last 330.57) - /unlink 13191 (time 1754406433.88 total 66.67 last 319.04) - /unlink 16022 (time 1754406443.88 total 76.67 last 283.01) - /unlink 19486 (time 1754406453.88 total 86.68 last 346.33) - /unlink 22657 (time 1754406463.88 total 96.68 last 317.06) - /unlink 26035 (time 1754406473.89 total 106.68 last 337.77) - /unlink 29536 (time 1754406483.89 total 116.68 last 350.02) - /unlink 32792 (time 1754406493.89 total 126.68 last 325.60) - /unlink 37034 (time 1754406503.89 total 136.68 last 424.15) - /unlink 40000 (time 1754406512.99 total 145.78 last 326.01) - /unlink 43590 (time 1754406522.99 total 155.78 last 358.97) - /unlink 46556 (time 1754406532.99 total 165.78 last 296.50) - /unlink 48870 (time 1754406542.99 total 175.79 last 231.31) - /unlink 50000 (time 1754406546.25 total 179.04 last 346.80) - /unlink 53918 (time 1754406556.26 total 189.05 last 391.72) - /unlink 58249 (time 1754406566.26 total 199.05 last 433.03) - /unlink 60000 (time 1754406570.30 total 203.09 last 432.89) - /unlink 64086 (time 1754406580.30 total 213.09 last 408.60) total: 64768 /unlink in 214.81 seconds: 301.52 ops/second Waiting for MDT destroys to complete oleg109-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 - open/close 1585 (time 1754406605.67 total 10.00 last 158.45) - open/close 3403 (time 1754406615.68 total 20.01 last 181.72) - open/close 5420 (time 1754406625.68 total 30.01 last 201.69) - open/close 8155 (time 1754406635.68 total 40.01 last 273.49) - open/close 10000 (time 1754406642.64 total 46.97 last 264.94) - open/close 12337 (time 1754406652.64 total 56.97 last 233.66) - open/close 15499 (time 1754406662.64 total 66.98 last 316.11) - open/close 17738 (time 1754406672.65 total 76.98 last 223.78) - open/close 20000 (time 1754406682.45 total 86.78 last 230.82) - open/close 23606 (time 1754406692.45 total 96.78 last 360.60) - open/close 27626 (time 1754406702.45 total 106.78 last 401.93) - open/close 30000 (time 1754406708.27 total 112.60 last 408.07) - open/close 33083 (time 1754406718.27 total 122.60 last 308.28) - open/close 36018 (time 1754406728.27 total 132.61 last 293.38) - open/close 38395 (time 1754406738.28 total 142.61 last 237.61) - open/close 40000 (time 1754406744.39 total 148.72 last 262.79) - open/close 42886 (time 1754406754.39 total 158.72 last 288.58) - open/close 44981 (time 1754406764.39 total 168.72 last 209.40) - open/close 47653 (time 1754406774.39 total 178.72 last 267.17) - open/close 50000 (time 1754406780.49 total 184.82 last 384.82) - open/close 53769 (time 1754406790.49 total 194.82 last 376.87) - open/close 57271 (time 1754406800.49 total 204.83 last 350.13) - open/close 60000 (time 1754406809.61 total 213.94 last 299.38) - open/close 64207 (time 1754406819.61 total 223.94 last 420.66) total: 64768 open/close in 225.24 seconds: 287.55 ops/second - /unlink 7971 (time 1754406832.39 total 10.00 last 797.05) - /unlink 10000 (time 1754406835.18 total 12.79 last 727.58) - /unlink 14912 (time 1754406845.18 total 22.79 last 491.18) - /unlink 19535 (time 1754406855.18 total 32.79 last 462.25) - /unlink 25461 (time 1754406865.19 total 42.79 last 592.51) - /unlink 30000 (time 1754406872.54 total 50.15 last 617.10) - /unlink 36498 (time 1754406882.54 total 60.15 last 649.74) - /unlink 40000 (time 1754406888.54 total 66.14 last 584.24) - /unlink 45586 (time 1754406898.54 total 76.14 last 558.58) - /unlink 50000 (time 1754406904.79 total 82.40 last 705.90) - /unlink 57070 (time 1754406914.79 total 92.40 last 706.90) - /unlink 60000 (time 1754406920.29 total 97.89 last 532.97) total: 64768 /unlink in 106.62 seconds: 607.47 ops/second Waiting for MDT destroys to complete oleg109-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 - open/close 3594 (time 1754406950.06 total 10.00 last 359.40) - open/close 7178 (time 1754406960.07 total 20.00 last 358.25) - open/close 10000 (time 1754406968.53 total 28.47 last 333.40) - open/close 13019 (time 1754406978.53 total 38.47 last 301.90) - open/close 16757 (time 1754406988.54 total 48.47 last 373.66) - open/close 20000 (time 1754406996.33 total 56.27 last 415.97) - open/close 24482 (time 1754407006.33 total 66.27 last 448.14) - open/close 27907 (time 1754407016.33 total 76.27 last 342.45) - open/close 30000 (time 1754407020.98 total 80.92 last 450.11) - open/close 33582 (time 1754407030.99 total 90.92 last 358.15) - open/close 37712 (time 1754407040.99 total 100.93 last 412.72) - open/close 40000 (time 1754407048.17 total 108.10 last 318.95) - open/close 43735 (time 1754407058.17 total 118.10 last 373.49) - open/close 47516 (time 1754407068.17 total 128.11 last 378.03) - open/close 50000 (time 1754407075.93 total 135.86 last 320.20) - open/close 54561 (time 1754407085.93 total 145.86 last 456.09) - open/close 58865 (time 1754407095.93 total 155.86 last 430.39) - open/close 60000 (time 1754407098.25 total 158.19 last 487.96) - open/close 64466 (time 1754407108.25 total 168.19 last 446.52) total: 64768 open/close in 168.86 seconds: 383.56 ops/second - /unlink 8399 (time 1754407120.28 total 10.00 last 839.89) - /unlink 10000 (time 1754407122.28 total 12.01 last 797.31) - /unlink 18341 (time 1754407132.28 total 22.01 last 834.06) - /unlink 26335 (time 1754407142.28 total 32.01 last 799.37) - /unlink 30000 (time 1754407146.82 total 36.54 last 808.28) - /unlink 38755 (time 1754407156.82 total 46.54 last 875.44) - /unlink 47094 (time 1754407166.82 total 56.54 last 833.81) - /unlink 50000 (time 1754407170.32 total 60.04 last 830.56) - /unlink 58508 (time 1754407180.32 total 70.04 last 850.70) - /unlink 60000 (time 1754407182.63 total 72.36 last 645.11) total: 64768 /unlink in 78.04 seconds: 829.98 ops/second Waiting for MDT destroys to complete oleg109-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 - open/close 3823 (time 1754407281.34 total 10.00 last 382.28) - open/close 7192 (time 1754407291.34 total 20.00 last 336.87) - open/close 10000 (time 1754407297.64 total 26.30 last 446.12) - open/close 13757 (time 1754407307.64 total 36.30 last 375.67) - open/close 18373 (time 1754407317.64 total 46.30 last 461.57) - open/close 20000 (time 1754407321.15 total 49.81 last 462.82) - open/close 24341 (time 1754407331.16 total 59.81 last 434.09) - open/close 28841 (time 1754407341.16 total 69.81 last 449.95) - open/close 30000 (time 1754407344.27 total 72.93 last 372.51) - open/close 34100 (time 1754407354.27 total 82.93 last 409.96) - open/close 38219 (time 1754407364.27 total 92.93 last 411.85) - open/close 40000 (time 1754407369.38 total 98.03 last 348.71) - open/close 44068 (time 1754407379.38 total 108.04 last 406.79) - open/close 48610 (time 1754407389.38 total 118.04 last 454.16) - open/close 50000 (time 1754407392.37 total 121.03 last 465.03) - open/close 54959 (time 1754407402.37 total 131.03 last 495.85) - open/close 59844 (time 1754407412.37 total 141.03 last 488.46) - open/close 64511 (time 1754407422.37 total 151.03 last 466.68) total: 64768 open/close in 151.57 seconds: 427.31 ops/second - /unlink 6614 (time 1754407434.34 total 10.00 last 661.36) - /unlink 10000 (time 1754407441.17 total 16.84 last 495.22) - /unlink 14827 (time 1754407451.17 total 26.84 last 482.68) - /unlink 20000 (time 1754407460.94 total 36.61 last 529.65) - /unlink 25145 (time 1754407470.94 total 46.61 last 514.49) - /unlink 30000 (time 1754407478.36 total 54.02 last 654.35) - /unlink 35148 (time 1754407488.36 total 64.02 last 514.79) - /unlink 39268 (time 1754407498.36 total 74.03 last 411.95) - /unlink 45390 (time 1754407508.36 total 84.03 last 612.19) - /unlink 50000 (time 1754407516.06 total 91.72 last 598.98) - /unlink 56775 (time 1754407526.06 total 101.72 last 677.43) - /unlink 60000 (time 1754407529.95 total 105.62 last 827.96) total: 64768 /unlink in 113.28 seconds: 571.74 ops/second Waiting for MDT destroys to complete oleg109-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 - open/close 4059 (time 1754407554.67 total 10.00 last 405.84) - open/close 8605 (time 1754407564.67 total 20.00 last 454.59) - open/close 10000 (time 1754407567.76 total 23.09 last 452.17) - open/close 13398 (time 1754407577.76 total 33.09 last 339.77) - open/close 18252 (time 1754407587.76 total 43.09 last 485.32) - open/close 20000 (time 1754407592.04 total 47.37 last 408.74) - open/close 22911 (time 1754407602.04 total 57.37 last 291.07) - open/close 25768 (time 1754407612.04 total 67.37 last 285.59) - open/close 29605 (time 1754407622.05 total 77.38 last 383.51) - open/close 32413 (time 1754407632.05 total 87.38 last 280.75) - open/close 35464 (time 1754407642.05 total 97.38 last 304.94) - open/close 38206 (time 1754407652.06 total 107.38 last 274.16) - open/close 40000 (time 1754407658.37 total 113.70 last 284.18) - open/close 44372 (time 1754407668.37 total 123.70 last 437.18) - open/close 49255 (time 1754407678.37 total 133.70 last 488.21) - open/close 53782 (time 1754407688.37 total 143.70 last 452.65) - open/close 58603 (time 1754407698.37 total 153.70 last 482.05) - open/close 60000 (time 1754407700.99 total 156.32 last 534.02) total: 64768 open/close in 165.41 seconds: 391.56 ops/second - /unlink 8445 (time 1754407721.37 total 10.00 last 844.43) - /unlink 10000 (time 1754407723.47 total 12.10 last 742.50) - /unlink 18884 (time 1754407733.47 total 22.10 last 888.34) - /unlink 26767 (time 1754407743.47 total 32.10 last 788.29) - /unlink 30000 (time 1754407747.46 total 36.09 last 810.36) - /unlink 37982 (time 1754407757.46 total 46.09 last 798.18) - /unlink 40000 (time 1754407759.71 total 48.34 last 894.17) - /unlink 49299 (time 1754407769.71 total 58.34 last 929.83) - /unlink 57234 (time 1754407779.72 total 68.34 last 793.41) - /unlink 60000 (time 1754407783.42 total 72.05 last 746.57) total: 64768 /unlink in 77.48 seconds: 835.88 ops/second Waiting for MDT destroys to complete oleg109-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg109-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg109-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg109-server stop mds service on oleg109-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg109-server stop mds service on oleg109-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg109-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg109-server' oleg109-server: oleg109-server.virtnet: executing unload_modules_local oleg109-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg109-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg109-client: oleg109-client: ssh exited with exit code 2 pdsh@oleg109-client: oleg109-server: ssh exited with exit code 2 pdsh@oleg109-client: oleg109-client: ssh exited with exit code 2 pdsh@oleg109-client: oleg109-server: ssh exited with exit code 2