== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:01:36 (1740484896) Stopping clients: oleg456-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg456-client.virtnet /mnt/lustre2 (opts:-f) oleg456-server: oleg456-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg456-server' oleg456-server: oleg456-server.virtnet: executing load_modules_local oleg456-server: Loading modules from /home/green/git/lustre-release/lustre oleg456-server: detected 4 online CPUs by sysfs oleg456-server: MODOPTS_LIBCFS= oleg456-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg456-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg456-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg456-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg456-client.virtnet: -o user_xattr,flock 192.168.204.156@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5476 (time 1740484936.10 total 10.00 last 547.57) - open/close 10000 (time 1740484944.20 total 18.10 last 558.61) - open/close 15563 (time 1740484954.20 total 28.10 last 556.23) - open/close 20000 (time 1740484962.09 total 35.99 last 562.58) - open/close 25499 (time 1740484972.09 total 45.99 last 549.81) - open/close 30000 (time 1740484980.48 total 54.38 last 536.36) - open/close 35399 (time 1740484990.48 total 64.38 last 539.84) - open/close 40000 (time 1740484999.40 total 73.30 last 516.02) - open/close 45364 (time 1740485009.40 total 83.30 last 536.36) - open/close 50000 (time 1740485018.77 total 92.66 last 495.01) - open/close 55216 (time 1740485028.77 total 102.67 last 521.55) - open/close 60000 (time 1740485038.19 total 112.09 last 507.62) total: 64768 open/close in 120.88 seconds: 535.80 ops/second - /unlink 9561 (time 1740485057.87 total 10.00 last 956.10) - /unlink 19171 (time 1740485067.87 total 20.00 last 960.91) - /unlink 29037 (time 1740485077.87 total 30.00 last 986.55) - /unlink 38961 (time 1740485087.87 total 40.00 last 992.35) - /unlink 48914 (time 1740485097.87 total 50.00 last 995.22) - /unlink 58479 (time 1740485107.87 total 60.00 last 956.48) total: 64768 /unlink in 66.64 seconds: 971.94 ops/second Waiting for MDT destroys to complete oleg456-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg456-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 - open/close 5728 (time 1740485220.61 total 10.00 last 572.76) - open/close 10000 (time 1740485228.14 total 17.52 last 568.05) - open/close 15657 (time 1740485238.14 total 27.52 last 565.60) - open/close 20000 (time 1740485245.72 total 35.10 last 572.99) - open/close 25533 (time 1740485255.72 total 45.10 last 553.30) - open/close 30000 (time 1740485263.79 total 53.17 last 553.41) - open/close 34805 (time 1740485273.79 total 63.18 last 480.46) - open/close 40000 (time 1740485283.37 total 72.76 last 542.03) - open/close 45280 (time 1740485293.37 total 82.76 last 527.97) - open/close 50000 (time 1740485302.36 total 91.74 last 525.43) - open/close 55292 (time 1740485312.36 total 101.75 last 529.10) - open/close 60000 (time 1740485321.38 total 110.77 last 521.67) total: 64768 open/close in 119.83 seconds: 540.48 ops/second - /unlink 9415 (time 1740485341.39 total 10.00 last 941.50) - /unlink 19083 (time 1740485351.39 total 20.00 last 966.80) - /unlink 28787 (time 1740485361.39 total 30.00 last 970.36) - /unlink 38655 (time 1740485371.39 total 40.00 last 986.75) - /unlink 48472 (time 1740485381.39 total 50.00 last 981.68) - /unlink 58277 (time 1740485391.39 total 60.00 last 980.49) total: 64768 /unlink in 66.61 seconds: 972.30 ops/second Waiting for MDT destroys to complete oleg456-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg456-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 - open/close 5796 (time 1740485501.34 total 10.00 last 579.52) - open/close 10000 (time 1740485508.79 total 17.46 last 563.81) - open/close 15769 (time 1740485518.79 total 27.46 last 576.88) - open/close 20000 (time 1740485526.16 total 34.83 last 573.99) - open/close 25780 (time 1740485536.16 total 44.83 last 577.99) - open/close 30000 (time 1740485545.01 total 53.68 last 476.95) - open/close 34847 (time 1740485555.01 total 63.68 last 484.65) - open/close 40000 (time 1740485564.85 total 73.51 last 524.00) - open/close 44831 (time 1740485574.85 total 83.51 last 483.10) - open/close 50000 (time 1740485584.79 total 93.45 last 519.97) - open/close 54771 (time 1740485594.79 total 103.45 last 477.06) - open/close 59350 (time 1740485604.79 total 113.46 last 457.84) - open/close 63382 (time 1740485614.79 total 123.46 last 403.20) total: 64768 open/close in 127.37 seconds: 508.51 ops/second - /unlink 5877 (time 1740485629.93 total 10.00 last 587.65) - /unlink 10000 (time 1740485637.00 total 17.07 last 583.28) - /unlink 14757 (time 1740485647.00 total 27.07 last 475.65) - /unlink 20000 (time 1740485656.70 total 36.76 last 540.98) - /unlink 25273 (time 1740485666.71 total 46.77 last 526.75) - /unlink 30000 (time 1740485674.31 total 54.38 last 621.75) - /unlink 36325 (time 1740485684.31 total 64.38 last 632.49) - /unlink 40000 (time 1740485690.03 total 70.10 last 642.06) - /unlink 47506 (time 1740485700.03 total 80.10 last 750.57) - /unlink 50000 (time 1740485702.72 total 82.79 last 926.70) - /unlink 57268 (time 1740485712.72 total 92.79 last 726.76) - /unlink 60000 (time 1740485717.08 total 97.15 last 626.57) total: 64768 /unlink in 103.33 seconds: 626.82 ops/second Waiting for MDT destroys to complete oleg456-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg456-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 - open/close 5229 (time 1740485770.69 total 10.00 last 522.85) - open/close 10000 (time 1740485779.96 total 19.28 last 514.22) - open/close 13607 (time 1740485789.97 total 29.28 last 360.60) - open/close 17018 (time 1740485799.97 total 39.28 last 341.04) - open/close 19696 (time 1740485809.97 total 49.29 last 267.75) - open/close 22487 (time 1740485819.97 total 59.29 last 279.09) - open/close 27252 (time 1740485829.97 total 69.29 last 476.50) - open/close 30000 (time 1740485835.44 total 74.76 last 502.28) - open/close 35206 (time 1740485845.44 total 84.76 last 520.52) - open/close 38901 (time 1740485855.45 total 94.76 last 369.46) - open/close 40000 (time 1740485859.62 total 98.94 last 263.03) - open/close 42906 (time 1740485869.62 total 108.94 last 290.59) - open/close 47391 (time 1740485879.62 total 118.94 last 448.49) - open/close 50000 (time 1740485884.41 total 123.72 last 545.72) - open/close 55169 (time 1740485894.41 total 133.72 last 516.83) - open/close 60000 (time 1740485903.74 total 143.06 last 517.45) total: 64768 open/close in 152.23 seconds: 425.46 ops/second - /unlink 6764 (time 1740485923.81 total 10.00 last 676.37) - /unlink 10000 (time 1740485930.19 total 16.37 last 507.76) - /unlink 15715 (time 1740485940.19 total 26.37 last 571.48) - /unlink 20000 (time 1740485948.40 total 34.59 last 521.65) - /unlink 25926 (time 1740485958.40 total 44.59 last 592.59) - /unlink 30000 (time 1740485966.73 total 52.92 last 489.24) - /unlink 35729 (time 1740485976.73 total 62.92 last 572.85) - /unlink 40000 (time 1740485982.48 total 68.67 last 742.95) - /unlink 45951 (time 1740485992.48 total 78.67 last 594.99) - /unlink 50000 (time 1740486000.77 total 86.95 last 488.59) - /unlink 55072 (time 1740486010.77 total 96.96 last 507.11) - /unlink 60000 (time 1740486019.91 total 106.10 last 539.19) total: 64768 /unlink in 115.53 seconds: 560.60 ops/second Waiting for MDT destroys to complete oleg456-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg456-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 - open/close 2671 (time 1740486046.83 total 10.00 last 267.04) - open/close 5442 (time 1740486056.83 total 20.00 last 277.09) - open/close 8050 (time 1740486066.83 total 30.01 last 260.74) - open/close 10000 (time 1740486074.50 total 37.68 last 254.24) - open/close 12658 (time 1740486084.51 total 47.68 last 265.77) - open/close 15211 (time 1740486094.51 total 57.68 last 255.30) - open/close 17804 (time 1740486104.51 total 67.68 last 259.24) - open/close 20000 (time 1740486113.29 total 76.46 last 250.16) - open/close 22574 (time 1740486123.29 total 86.46 last 257.32) - open/close 25187 (time 1740486133.29 total 96.46 last 261.28) - open/close 27823 (time 1740486143.29 total 106.46 last 263.52) - open/close 30000 (time 1740486151.98 total 115.15 last 250.76) - open/close 32547 (time 1740486161.98 total 125.15 last 254.65) - open/close 35584 (time 1740486171.98 total 135.15 last 303.69) - open/close 38127 (time 1740486181.98 total 145.15 last 254.20) - open/close 40000 (time 1740486189.51 total 152.68 last 248.86) - open/close 42525 (time 1740486199.51 total 162.68 last 252.45) - open/close 45132 (time 1740486209.51 total 172.68 last 260.65) - open/close 47662 (time 1740486219.52 total 182.69 last 252.91) - open/close 50000 (time 1740486228.31 total 191.48 last 265.85) - open/close 52563 (time 1740486238.31 total 201.48 last 256.24) - open/close 55354 (time 1740486248.32 total 211.49 last 279.01) - open/close 58247 (time 1740486258.32 total 221.49 last 289.25) - open/close 60000 (time 1740486263.88 total 227.05 last 315.05) - open/close 62773 (time 1740486273.88 total 237.05 last 277.21) total: 64768 open/close in 244.94 seconds: 264.43 ops/second - /unlink 4458 (time 1740486293.08 total 10.00 last 445.76) - /unlink 9236 (time 1740486303.08 total 20.00 last 477.72) - /unlink 13860 (time 1740486313.11 total 30.03 last 461.14) - /unlink 18666 (time 1740486323.11 total 40.03 last 480.55) - /unlink 20000 (time 1740486325.58 total 42.51 last 539.01) - /unlink 24854 (time 1740486335.58 total 52.51 last 485.35) - /unlink 30000 (time 1740486342.97 total 59.89 last 696.81) - /unlink 34442 (time 1740486352.97 total 69.89 last 444.13) - /unlink 39667 (time 1740486362.97 total 79.90 last 522.41) - /unlink 44425 (time 1740486372.97 total 89.90 last 475.75) - /unlink 50000 (time 1740486380.65 total 97.57 last 726.45) - /unlink 54797 (time 1740486390.65 total 107.57 last 479.65) - /unlink 59356 (time 1740486400.65 total 117.57 last 455.90) - /unlink 64102 (time 1740486410.65 total 127.57 last 474.52) total: 64768 /unlink in 129.00 seconds: 502.07 ops/second Waiting for MDT destroys to complete oleg456-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg456-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg456-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg456-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg456-server stop mds service on oleg456-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg456-server stop mds service on oleg456-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg456-server unloading modules via unload_modules_local on: 'oleg456-server' oleg456-server: oleg456-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg456-client: oleg456-client: ssh exited with exit code 2 pdsh@oleg456-client: oleg456-server: ssh exited with exit code 2 pdsh@oleg456-client: oleg456-client: ssh exited with exit code 2 pdsh@oleg456-client: oleg456-server: ssh exited with exit code 2