== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:33:26 (1741898006) Stopping clients: oleg319-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg319-client.virtnet /mnt/lustre2 (opts:-f) oleg319-server: oleg319-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing load_modules_local oleg319-server: Loading modules from /home/green/git/lustre-release/lustre oleg319-server: detected 4 online CPUs by sysfs oleg319-server: MODOPTS_LIBCFS= oleg319-server: Force libcfs to create 2 CPU partitions oleg319-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg319-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg319-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg319-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg319-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg319-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1151 (time 1741898072.84 total 10.00 last 115.06) - open/close 2445 (time 1741898082.84 total 20.01 last 129.34) - open/close 3667 (time 1741898092.84 total 30.01 last 122.17) - open/close 4804 (time 1741898102.85 total 40.01 last 113.65) - open/close 5867 (time 1741898112.85 total 50.02 last 106.26) - open/close 7106 (time 1741898122.85 total 60.02 last 123.88) - open/close 8493 (time 1741898132.85 total 70.02 last 138.68) - open/close 9972 (time 1741898142.86 total 80.02 last 147.85) - open/close 11281 (time 1741898152.86 total 90.03 last 130.83) - open/close 12626 (time 1741898162.86 total 100.03 last 134.49) - open/close 13952 (time 1741898172.87 total 110.04 last 132.50) - open/close 15444 (time 1741898182.87 total 120.04 last 149.15) - open/close 16772 (time 1741898192.88 total 130.04 last 132.77) - open/close 18140 (time 1741898202.88 total 140.04 last 136.78) - open/close 19635 (time 1741898212.88 total 150.05 last 149.44) - open/close 20000 (time 1741898215.10 total 152.26 last 164.83) - open/close 21653 (time 1741898225.10 total 162.26 last 165.27) - open/close 23387 (time 1741898235.10 total 172.27 last 173.35) - open/close 25013 (time 1741898245.11 total 182.28 last 162.47) - open/close 26422 (time 1741898255.11 total 192.28 last 140.88) - open/close 27820 (time 1741898265.11 total 202.28 last 139.75) - open/close 29167 (time 1741898275.11 total 212.28 last 134.70) - open/close 30000 (time 1741898281.43 total 218.60 last 131.89) - open/close 31212 (time 1741898291.43 total 228.60 last 121.20) - open/close 32547 (time 1741898301.44 total 238.60 last 133.40) - open/close 34178 (time 1741898311.44 total 248.60 last 163.10) - open/close 35821 (time 1741898321.44 total 258.61 last 164.27) - open/close 37166 (time 1741898331.44 total 268.61 last 134.46) - open/close 38655 (time 1741898341.44 total 278.61 last 148.89) - open/close 40000 (time 1741898350.76 total 287.93 last 144.38) - open/close 41463 (time 1741898360.76 total 297.93 last 146.26) - open/close 42886 (time 1741898370.77 total 307.93 last 142.24) - open/close 44392 (time 1741898380.77 total 317.93 last 150.58) - open/close 45693 (time 1741898390.78 total 327.94 last 129.97) - open/close 46992 (time 1741898400.78 total 337.95 last 129.89) - open/close 48296 (time 1741898410.78 total 347.95 last 130.32) - open/close 49570 (time 1741898420.78 total 357.95 last 127.39) - open/close 50000 (time 1741898424.22 total 361.39 last 125.23) - open/close 51387 (time 1741898434.22 total 371.39 last 138.65) - open/close 52646 (time 1741898444.22 total 381.39 last 125.89) - open/close 53989 (time 1741898454.22 total 391.39 last 134.30) - open/close 55385 (time 1741898464.23 total 401.40 last 139.50) - open/close 56637 (time 1741898474.23 total 411.40 last 125.18) - open/close 57798 (time 1741898484.24 total 421.40 last 116.05) - open/close 59008 (time 1741898494.24 total 431.41 last 120.93) - open/close 60000 (time 1741898502.73 total 439.90 last 116.85) - open/close 61320 (time 1741898512.74 total 449.90 last 131.96) - open/close 62758 (time 1741898522.74 total 459.91 last 143.69) - open/close 64080 (time 1741898532.75 total 469.92 last 132.08) total: 64768 open/close in 475.46 seconds: 136.22 ops/second - /unlink 2394 (time 1741898549.92 total 10.00 last 239.37) - /unlink 4774 (time 1741898559.92 total 20.00 last 238.00) - /unlink 7266 (time 1741898569.92 total 30.00 last 249.20) - /unlink 9904 (time 1741898579.92 total 40.00 last 263.75) - /unlink 12965 (time 1741898589.92 total 50.00 last 306.05) - /unlink 15589 (time 1741898599.93 total 60.01 last 262.29) - /unlink 18535 (time 1741898609.93 total 70.01 last 294.46) - /unlink 20000 (time 1741898615.33 total 75.41 last 271.41) - /unlink 22920 (time 1741898625.33 total 85.41 last 291.98) - /unlink 26084 (time 1741898635.33 total 95.41 last 316.36) - /unlink 29350 (time 1741898645.33 total 105.41 last 326.57) - /unlink 32505 (time 1741898655.33 total 115.42 last 315.48) - /unlink 35320 (time 1741898665.34 total 125.42 last 281.47) - /unlink 37957 (time 1741898675.34 total 135.42 last 263.63) - /unlink 40000 (time 1741898681.96 total 142.04 last 308.35) - /unlink 42910 (time 1741898691.97 total 152.05 last 290.84) - /unlink 45694 (time 1741898701.97 total 162.05 last 278.32) - /unlink 48469 (time 1741898711.97 total 172.05 last 277.44) - /unlink 50000 (time 1741898716.66 total 176.74 last 326.62) - /unlink 53235 (time 1741898726.66 total 186.74 last 323.45) - /unlink 57847 (time 1741898736.66 total 196.74 last 461.15) - /unlink 60000 (time 1741898741.59 total 201.67 last 436.76) total: 64768 /unlink in 211.07 seconds: 306.85 ops/second Waiting for MDT destroys to complete oleg319-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg319-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 - open/close 2821 (time 1741898769.04 total 10.00 last 282.04) - open/close 5673 (time 1741898779.04 total 20.00 last 285.17) - open/close 8365 (time 1741898789.04 total 30.01 last 269.09) - open/close 10000 (time 1741898794.95 total 35.92 last 276.61) - open/close 13372 (time 1741898804.95 total 45.92 last 337.16) - open/close 16122 (time 1741898814.96 total 55.92 last 274.91) - open/close 18341 (time 1741898824.96 total 65.92 last 221.86) - open/close 20000 (time 1741898833.18 total 74.14 last 201.86) - open/close 21922 (time 1741898843.18 total 84.15 last 192.14) - open/close 23818 (time 1741898853.18 total 94.15 last 189.60) - open/close 25660 (time 1741898863.19 total 104.15 last 184.09) - open/close 27768 (time 1741898873.19 total 114.16 last 210.69) - open/close 30000 (time 1741898882.30 total 123.27 last 244.93) - open/close 32028 (time 1741898892.31 total 133.27 last 202.73) - open/close 34079 (time 1741898902.31 total 143.28 last 205.05) - open/close 35956 (time 1741898912.31 total 153.28 last 187.68) - open/close 37948 (time 1741898922.32 total 163.28 last 199.14) - open/close 39905 (time 1741898932.32 total 173.28 last 195.63) - open/close 41963 (time 1741898942.32 total 183.29 last 205.75) - open/close 44114 (time 1741898952.33 total 193.29 last 215.00) - open/close 46808 (time 1741898962.33 total 203.29 last 269.37) - open/close 49308 (time 1741898972.33 total 213.29 last 249.94) - open/close 50000 (time 1741898975.67 total 216.64 last 207.01) - open/close 51967 (time 1741898985.68 total 226.64 last 196.60) - open/close 53939 (time 1741898995.68 total 236.65 last 197.12) - open/close 56023 (time 1741899005.69 total 246.65 last 208.30) - open/close 58246 (time 1741899015.69 total 256.66 last 222.21) - open/close 60000 (time 1741899023.15 total 264.11 last 235.25) - open/close 61810 (time 1741899033.15 total 274.11 last 180.99) - open/close 63609 (time 1741899043.15 total 284.11 last 179.88) total: 64768 open/close in 289.41 seconds: 223.79 ops/second - /unlink 3697 (time 1741899059.57 total 10.00 last 369.67) - /unlink 7814 (time 1741899069.57 total 20.00 last 411.67) - /unlink 10000 (time 1741899074.33 total 24.77 last 458.76) - /unlink 14825 (time 1741899084.34 total 34.77 last 482.43) - /unlink 20000 (time 1741899093.71 total 44.14 last 552.08) - /unlink 25492 (time 1741899103.71 total 54.14 last 549.19) - /unlink 30000 (time 1741899112.33 total 62.76 last 522.97) - /unlink 34862 (time 1741899122.33 total 72.76 last 486.19) - /unlink 39098 (time 1741899132.33 total 82.76 last 423.58) - /unlink 43813 (time 1741899142.33 total 92.76 last 471.49) - /unlink 48765 (time 1741899152.33 total 102.76 last 495.15) - /unlink 50000 (time 1741899154.93 total 105.36 last 475.50) - /unlink 54737 (time 1741899164.93 total 115.36 last 473.70) - /unlink 59436 (time 1741899174.93 total 125.36 last 469.86) total: 64768 /unlink in 135.35 seconds: 478.50 ops/second Waiting for MDT destroys to complete oleg319-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg319-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 - open/close 2984 (time 1741899204.52 total 10.00 last 298.33) - open/close 6420 (time 1741899214.52 total 20.00 last 343.54) - open/close 9562 (time 1741899224.53 total 30.01 last 314.16) - open/close 13120 (time 1741899234.53 total 40.01 last 355.77) - open/close 16463 (time 1741899244.53 total 50.01 last 334.30) - open/close 19820 (time 1741899254.53 total 60.01 last 335.67) - open/close 22043 (time 1741899264.53 total 70.01 last 222.23) - open/close 24393 (time 1741899274.53 total 80.01 last 234.96) - open/close 26735 (time 1741899284.54 total 90.02 last 234.09) - open/close 29283 (time 1741899294.54 total 100.02 last 254.80) - open/close 30000 (time 1741899297.11 total 102.59 last 278.69) - open/close 32766 (time 1741899307.11 total 112.59 last 276.58) - open/close 35957 (time 1741899317.11 total 122.59 last 319.07) - open/close 39482 (time 1741899327.11 total 132.59 last 352.45) - open/close 43197 (time 1741899337.12 total 142.60 last 371.40) - open/close 46494 (time 1741899347.12 total 152.60 last 329.69) - open/close 50000 (time 1741899353.96 total 159.44 last 512.24) - open/close 55047 (time 1741899363.96 total 169.44 last 504.66) - open/close 60000 (time 1741899373.43 total 178.91 last 523.12) total: 64768 open/close in 187.52 seconds: 345.40 ops/second - /unlink 10000 (time 1741899392.31 total 9.44 last 1059.74) - /unlink 20000 (time 1741899401.84 total 18.97 last 1049.13) - /unlink 30000 (time 1741899411.21 total 28.33 last 1067.62) - /unlink 38956 (time 1741899421.21 total 38.33 last 895.59) - /unlink 48161 (time 1741899431.21 total 48.34 last 920.45) - /unlink 57717 (time 1741899441.21 total 58.34 last 955.59) - /unlink 60000 (time 1741899443.67 total 60.80 last 927.02) total: 64768 /unlink in 65.19 seconds: 993.59 ops/second Waiting for MDT destroys to complete oleg319-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 oleg319-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 - open/close 3784 (time 1741899555.02 total 10.00 last 378.38) - open/close 7103 (time 1741899565.02 total 20.00 last 331.86) - open/close 10000 (time 1741899573.16 total 28.15 last 355.71) - open/close 14106 (time 1741899583.16 total 38.15 last 410.57) - open/close 16906 (time 1741899593.16 total 48.15 last 279.96) - open/close 20000 (time 1741899602.63 total 57.62 last 326.81) - open/close 23815 (time 1741899612.63 total 67.62 last 381.45) - open/close 26236 (time 1741899622.63 total 77.62 last 242.04) - open/close 29047 (time 1741899632.63 total 87.62 last 281.07) - open/close 30000 (time 1741899636.06 total 91.04 last 278.50) - open/close 33280 (time 1741899646.06 total 101.04 last 327.96) - open/close 36482 (time 1741899656.06 total 111.05 last 320.15) - open/close 40000 (time 1741899663.15 total 118.14 last 496.10) - open/close 44707 (time 1741899673.15 total 128.14 last 470.61) - open/close 49767 (time 1741899683.15 total 138.14 last 505.96) - open/close 54493 (time 1741899693.16 total 148.14 last 472.52) - open/close 59810 (time 1741899703.16 total 158.14 last 531.70) total: 64768 open/close in 167.59 seconds: 386.47 ops/second - /unlink 7792 (time 1741899723.48 total 10.00 last 779.14) - /unlink 10000 (time 1741899726.54 total 13.07 last 720.40) - /unlink 16882 (time 1741899736.54 total 23.07 last 688.17) - /unlink 20000 (time 1741899740.23 total 26.75 last 846.53) - /unlink 29441 (time 1741899750.23 total 36.75 last 944.07) - /unlink 36336 (time 1741899760.23 total 46.75 last 689.41) - /unlink 40000 (time 1741899767.75 total 54.27 last 487.03) - /unlink 46724 (time 1741899777.75 total 64.28 last 672.32) - /unlink 50000 (time 1741899781.55 total 68.07 last 862.20) - /unlink 58103 (time 1741899791.55 total 78.08 last 810.24) - /unlink 60000 (time 1741899793.85 total 80.38 last 823.88) total: 64768 /unlink in 86.35 seconds: 750.06 ops/second Waiting for MDT destroys to complete oleg319-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg319-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 - open/close 4981 (time 1741899875.33 total 10.00 last 498.05) - open/close 10000 (time 1741899883.70 total 18.37 last 599.64) - open/close 15988 (time 1741899893.70 total 28.37 last 598.75) - open/close 20000 (time 1741899900.67 total 35.34 last 575.80) - open/close 25620 (time 1741899910.67 total 45.34 last 561.99) - open/close 30000 (time 1741899919.84 total 54.51 last 477.69) - open/close 35291 (time 1741899929.84 total 64.51 last 529.06) - open/close 39062 (time 1741899939.84 total 74.51 last 377.04) - open/close 43845 (time 1741899949.84 total 84.51 last 478.26) - open/close 48277 (time 1741899959.84 total 94.51 last 443.20) - open/close 50000 (time 1741899963.35 total 98.02 last 491.71) - open/close 55765 (time 1741899973.35 total 108.02 last 576.45) - open/close 60000 (time 1741899980.91 total 115.58 last 559.90) - open/close 64635 (time 1741899990.91 total 125.58 last 463.39) total: 64768 open/close in 125.92 seconds: 514.36 ops/second - /unlink 8323 (time 1741900002.23 total 10.00 last 832.26) - /unlink 10000 (time 1741900004.30 total 12.07 last 811.31) - /unlink 18046 (time 1741900014.30 total 22.07 last 804.60) - /unlink 20000 (time 1741900016.75 total 24.52 last 796.94) - /unlink 29282 (time 1741900026.75 total 34.52 last 928.20) - /unlink 39856 (time 1741900036.75 total 44.52 last 1057.30) - /unlink 49489 (time 1741900046.75 total 54.52 last 963.24) - /unlink 60000 (time 1741900056.42 total 64.19 last 1087.13) total: 64768 /unlink in 68.74 seconds: 942.25 ops/second Waiting for MDT destroys to complete oleg319-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 oleg319-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg319-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg319-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server stop mds service on oleg319-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server stop mds service on oleg319-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg319-server unloading modules via unload_modules_local on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg319-client: oleg319-client: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-client: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2