== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:41:22 (1742902882) Stopping clients: oleg327-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg327-client.virtnet /mnt/lustre2 (opts:-f) oleg327-server: oleg327-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg327-server' oleg327-server: oleg327-server.virtnet: executing load_modules_local oleg327-server: Loading modules from /home/green/git/lustre-release/lustre oleg327-server: detected 4 online CPUs by sysfs oleg327-server: MODOPTS_LIBCFS= oleg327-server: Force libcfs to create 2 CPU partitions oleg327-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg327-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg327-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg327-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg327-client.virtnet: -o user_xattr,flock 192.168.203.127@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2613 (time 1742902930.79 total 10.00 last 261.24) - open/close 5392 (time 1742902940.79 total 20.01 last 277.79) - open/close 8623 (time 1742902950.80 total 30.01 last 322.94) - open/close 10000 (time 1742902954.71 total 33.93 last 351.72) - open/close 12609 (time 1742902964.72 total 43.93 last 260.81) - open/close 15035 (time 1742902974.72 total 53.93 last 242.59) - open/close 17293 (time 1742902984.72 total 63.93 last 225.80) - open/close 19048 (time 1742902994.72 total 73.93 last 175.47) - open/close 20000 (time 1742902999.50 total 78.71 last 199.20) - open/close 22114 (time 1742903009.50 total 88.71 last 211.36) - open/close 24529 (time 1742903019.50 total 98.71 last 241.49) - open/close 26956 (time 1742903029.50 total 108.71 last 242.69) - open/close 28958 (time 1742903039.50 total 118.72 last 200.18) - open/close 30000 (time 1742903044.49 total 123.70 last 208.98) - open/close 31679 (time 1742903054.49 total 133.71 last 167.84) - open/close 33861 (time 1742903064.49 total 143.71 last 218.14) - open/close 36540 (time 1742903074.50 total 153.71 last 267.88) - open/close 38636 (time 1742903084.50 total 163.71 last 209.55) - open/close 40000 (time 1742903092.66 total 171.87 last 167.12) - open/close 42210 (time 1742903102.66 total 181.87 last 220.99) - open/close 44185 (time 1742903112.67 total 191.88 last 197.40) - open/close 45696 (time 1742903122.67 total 201.88 last 151.06) - open/close 47303 (time 1742903132.67 total 211.89 last 160.62) - open/close 48996 (time 1742903142.68 total 221.89 last 169.21) - open/close 50000 (time 1742903147.81 total 227.02 last 195.72) - open/close 52159 (time 1742903157.81 total 237.02 last 215.84) - open/close 54171 (time 1742903167.82 total 247.03 last 201.10) - open/close 56112 (time 1742903177.82 total 257.03 last 194.06) - open/close 58340 (time 1742903187.82 total 267.04 last 222.67) - open/close 60000 (time 1742903197.23 total 276.44 last 176.45) - open/close 62110 (time 1742903207.23 total 286.45 last 210.94) - open/close 64214 (time 1742903217.23 total 296.45 last 210.39) total: 64768 open/close in 299.29 seconds: 216.40 ops/second - /unlink 3200 (time 1742903231.46 total 10.00 last 319.91) - /unlink 6485 (time 1742903241.47 total 20.01 last 328.43) - /unlink 9422 (time 1742903251.47 total 30.01 last 293.61) - /unlink 12905 (time 1742903261.47 total 40.01 last 348.28) - /unlink 16046 (time 1742903271.47 total 50.01 last 314.07) - /unlink 20000 (time 1742903280.49 total 59.03 last 438.23) - /unlink 24797 (time 1742903290.49 total 69.03 last 479.62) - /unlink 30000 (time 1742903300.20 total 78.74 last 536.29) - /unlink 34121 (time 1742903310.20 total 88.74 last 412.00) - /unlink 39657 (time 1742903320.20 total 98.74 last 553.44) - /unlink 44727 (time 1742903330.20 total 108.74 last 506.96) - /unlink 50000 (time 1742903337.47 total 116.01 last 725.36) - /unlink 56587 (time 1742903347.47 total 126.01 last 658.62) - /unlink 60000 (time 1742903351.37 total 129.91 last 875.21) total: 64768 /unlink in 135.72 seconds: 477.23 ops/second Waiting for MDT destroys to complete - open/close 4906 (time 1742903394.49 total 10.00 last 490.58) - open/close 10000 (time 1742903404.15 total 19.66 last 527.39) - open/close 15121 (time 1742903414.15 total 29.66 last 512.05) - open/close 19432 (time 1742903424.15 total 39.66 last 431.06) - open/close 23263 (time 1742903434.15 total 49.66 last 383.01) - open/close 26561 (time 1742903444.15 total 59.66 last 329.78) - open/close 29989 (time 1742903454.15 total 69.67 last 342.75) - open/close 33497 (time 1742903464.15 total 79.67 last 350.77) - open/close 37475 (time 1742903474.16 total 89.67 last 397.78) - open/close 40000 (time 1742903481.13 total 96.64 last 362.27) - open/close 43750 (time 1742903491.13 total 106.64 last 374.98) - open/close 47618 (time 1742903501.13 total 116.64 last 386.78) - open/close 50000 (time 1742903508.31 total 123.83 last 331.39) - open/close 53201 (time 1742903518.32 total 133.83 last 320.06) - open/close 56766 (time 1742903528.32 total 143.83 last 356.41) - open/close 59992 (time 1742903538.32 total 153.83 last 322.52) - open/close 63657 (time 1742903548.32 total 163.83 last 366.42) total: 64768 open/close in 166.81 seconds: 388.28 ops/second - /unlink 6103 (time 1742903562.31 total 10.00 last 610.29) - /unlink 10000 (time 1742903568.07 total 15.76 last 677.05) - /unlink 17836 (time 1742903578.07 total 25.76 last 783.47) - /unlink 20000 (time 1742903580.83 total 28.51 last 785.39) - /unlink 27511 (time 1742903590.83 total 38.51 last 751.07) - /unlink 30000 (time 1742903595.07 total 42.75 last 587.02) - /unlink 37414 (time 1742903605.07 total 52.75 last 741.36) - /unlink 40000 (time 1742903609.04 total 56.72 last 651.65) - /unlink 49048 (time 1742903619.04 total 66.72 last 904.78) - /unlink 57321 (time 1742903629.04 total 76.72 last 827.26) - /unlink 60000 (time 1742903632.48 total 80.17 last 777.89) total: 64768 /unlink in 85.00 seconds: 761.99 ops/second Waiting for MDT destroys to complete - open/close 4492 (time 1742903710.24 total 10.00 last 449.17) - open/close 9752 (time 1742903720.24 total 20.00 last 525.93) - open/close 15809 (time 1742903730.24 total 30.00 last 605.57) - open/close 20000 (time 1742903737.83 total 37.59 last 552.65) - open/close 25297 (time 1742903747.83 total 47.59 last 529.69) - open/close 30000 (time 1742903756.32 total 56.08 last 553.60) - open/close 36153 (time 1742903766.32 total 66.08 last 615.25) - open/close 40000 (time 1742903774.24 total 74.00 last 486.10) - open/close 44784 (time 1742903784.24 total 84.00 last 478.39) - open/close 50000 (time 1742903792.72 total 92.48 last 614.96) - open/close 55572 (time 1742903802.72 total 102.48 last 557.15) - open/close 60000 (time 1742903810.47 total 110.23 last 571.31) total: 64768 open/close in 118.73 seconds: 545.49 ops/second - /unlink 8794 (time 1742903829.84 total 10.01 last 878.67) - /unlink 17980 (time 1742903839.84 total 20.01 last 918.57) - /unlink 20000 (time 1742903841.91 total 22.07 last 978.39) - /unlink 29209 (time 1742903851.91 total 32.07 last 920.84) - /unlink 40000 (time 1742903861.78 total 41.94 last 1093.53) - /unlink 50000 (time 1742903871.34 total 51.51 last 1045.69) - /unlink 59976 (time 1742903881.34 total 61.51 last 997.54) total: 64768 /unlink in 65.68 seconds: 986.13 ops/second Waiting for MDT destroys to complete - open/close 5405 (time 1742903979.68 total 10.00 last 540.42) - open/close 10000 (time 1742903989.14 total 19.47 last 485.47) - open/close 14846 (time 1742903999.14 total 29.47 last 484.59) - open/close 20000 (time 1742904009.01 total 39.33 last 522.42) - open/close 25306 (time 1742904019.01 total 49.33 last 530.53) - open/close 30000 (time 1742904028.26 total 58.58 last 507.51) - open/close 35105 (time 1742904038.26 total 68.58 last 510.48) - open/close 39923 (time 1742904048.26 total 78.58 last 481.72) - open/close 44893 (time 1742904058.26 total 88.59 last 496.94) - open/close 49824 (time 1742904068.26 total 98.59 last 493.07) - open/close 54557 (time 1742904078.26 total 108.59 last 473.29) - open/close 59129 (time 1742904088.26 total 118.59 last 457.19) - open/close 63590 (time 1742904098.26 total 128.59 last 446.05) total: 64768 open/close in 130.83 seconds: 495.04 ops/second - /unlink 9010 (time 1742904111.40 total 10.00 last 900.99) - /unlink 18564 (time 1742904121.40 total 20.00 last 955.32) - /unlink 28094 (time 1742904131.40 total 30.00 last 952.99) - /unlink 37520 (time 1742904141.40 total 40.00 last 942.55) - /unlink 40000 (time 1742904144.05 total 42.65 last 938.14) - /unlink 49021 (time 1742904154.05 total 52.65 last 902.05) - /unlink 58577 (time 1742904164.05 total 62.65 last 955.59) total: 64768 /unlink in 69.58 seconds: 930.87 ops/second Waiting for MDT destroys to complete - open/close 5456 (time 1742904265.49 total 10.00 last 545.54) - open/close 10000 (time 1742904273.92 total 18.43 last 538.91) - open/close 15494 (time 1742904283.92 total 28.43 last 549.32) - open/close 20000 (time 1742904292.27 total 36.78 last 539.71) - open/close 25423 (time 1742904302.27 total 46.78 last 542.24) - open/close 30000 (time 1742904310.79 total 55.30 last 537.29) - open/close 35292 (time 1742904320.79 total 65.30 last 529.12) - open/close 40000 (time 1742904329.68 total 74.19 last 529.72) - open/close 45298 (time 1742904339.68 total 84.19 last 529.70) - open/close 50000 (time 1742904349.34 total 93.85 last 486.76) - open/close 55140 (time 1742904359.35 total 103.86 last 513.93) - open/close 60000 (time 1742904368.72 total 113.23 last 518.21) total: 64768 open/close in 122.49 seconds: 528.76 ops/second - /unlink 9249 (time 1742904388.85 total 10.00 last 924.81) - /unlink 18925 (time 1742904398.85 total 20.00 last 967.59) - /unlink 28565 (time 1742904408.85 total 30.00 last 963.96) - /unlink 37658 (time 1742904418.85 total 40.00 last 909.28) - /unlink 40000 (time 1742904421.52 total 42.67 last 879.01) - /unlink 49558 (time 1742904431.52 total 52.67 last 955.73) - /unlink 59402 (time 1742904441.52 total 62.67 last 984.38) total: 64768 /unlink in 68.03 seconds: 952.00 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg327-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg327-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg327-server stop mds service on oleg327-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg327-server stop mds service on oleg327-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg327-server unloading modules via unload_modules_local on: 'oleg327-server' oleg327-server: oleg327-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg327-client: oleg327-client: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-client: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 2