== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:21:18 (1747848078) Stopping clients: oleg430-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg430-client.virtnet /mnt/lustre2 (opts:-f) oleg430-server: oleg430-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing load_modules_local oleg430-server: Loading modules from /home/green/git/lustre-release/lustre oleg430-server: detected 4 online CPUs by sysfs oleg430-server: MODOPTS_LIBCFS= oleg430-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg430-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg430-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 272 79720 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 65808 302 65506 1% /mnt/lustre[OST:0] filesystem_summary: 65778 272 65506 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1992 (time 1747848116.72 total 10.01 last 199.06) - open/close 4066 (time 1747848126.73 total 20.01 last 207.25) - open/close 6070 (time 1747848136.74 total 30.02 last 200.27) - open/close 8051 (time 1747848146.74 total 40.02 last 198.03) - open/close 10000 (time 1747848155.00 total 48.28 last 235.94) - open/close 11963 (time 1747848165.00 total 58.29 last 196.23) - open/close 14098 (time 1747848175.01 total 68.29 last 213.44) - open/close 16560 (time 1747848185.01 total 78.29 last 246.19) - open/close 19596 (time 1747848195.01 total 88.29 last 303.56) - open/close 22073 (time 1747848205.01 total 98.29 last 247.69) - open/close 24292 (time 1747848215.01 total 108.29 last 221.86) - open/close 26434 (time 1747848225.01 total 118.30 last 214.14) - open/close 28858 (time 1747848235.01 total 128.30 last 242.34) - open/close 30000 (time 1747848239.61 total 132.89 last 248.71) - open/close 32739 (time 1747848249.61 total 142.89 last 273.88) - open/close 35176 (time 1747848259.61 total 152.89 last 243.66) - open/close 37109 (time 1747848269.61 total 162.90 last 193.26) - open/close 38615 (time 1747848279.61 total 172.90 last 150.57) - open/close 40000 (time 1747848288.72 total 182.00 last 152.10) - open/close 41480 (time 1747848298.72 total 192.01 last 147.94) - open/close 43410 (time 1747848308.73 total 202.01 last 192.96) - open/close 45087 (time 1747848318.73 total 212.01 last 167.68) - open/close 46825 (time 1747848328.73 total 222.01 last 173.77) - open/close 49221 (time 1747848338.73 total 232.02 last 239.52) - open/close 50000 (time 1747848342.20 total 235.49 last 224.41) - open/close 52361 (time 1747848352.20 total 245.49 last 236.07) - open/close 54541 (time 1747848362.21 total 255.49 last 217.95) - open/close 57051 (time 1747848372.21 total 265.49 last 250.99) - open/close 59537 (time 1747848382.21 total 275.50 last 248.51) - open/close 62509 (time 1747848392.21 total 285.50 last 297.14) total: 64768 open/close in 294.50 seconds: 219.93 ops/second - /unlink 4876 (time 1747848412.25 total 10.00 last 487.57) - /unlink 8934 (time 1747848422.25 total 20.00 last 405.80) - /unlink 10000 (time 1747848424.41 total 22.16 last 493.16) - /unlink 13881 (time 1747848434.41 total 32.16 last 388.08) - /unlink 19668 (time 1747848444.41 total 42.16 last 578.65) - /unlink 24836 (time 1747848454.41 total 52.16 last 516.80) - /unlink 29329 (time 1747848464.41 total 62.17 last 449.20) - /unlink 34700 (time 1747848474.41 total 72.17 last 537.05) - /unlink 40000 (time 1747848483.23 total 80.99 last 600.85) - /unlink 46479 (time 1747848493.23 total 90.99 last 647.84) - /unlink 50000 (time 1747848498.31 total 96.07 last 693.15) - /unlink 56741 (time 1747848508.31 total 106.07 last 674.07) - /unlink 60000 (time 1747848513.03 total 110.79 last 690.53) total: 64768 /unlink in 117.78 seconds: 549.90 ops/second Waiting for MDT destroys to complete - open/close 3578 (time 1747848549.41 total 10.00 last 357.71) - open/close 7243 (time 1747848559.41 total 20.00 last 366.48) - open/close 10000 (time 1747848566.67 total 27.26 last 380.10) - open/close 13823 (time 1747848576.67 total 37.26 last 382.27) - open/close 18681 (time 1747848586.67 total 47.26 last 485.72) - open/close 20000 (time 1747848589.65 total 50.24 last 442.08) - open/close 24204 (time 1747848599.65 total 60.24 last 420.31) - open/close 27748 (time 1747848609.66 total 70.25 last 354.32) - open/close 30000 (time 1747848616.34 total 76.93 last 337.19) - open/close 33668 (time 1747848626.34 total 86.93 last 366.74) - open/close 37776 (time 1747848636.34 total 96.93 last 410.72) - open/close 40000 (time 1747848642.35 total 102.94 last 370.16) - open/close 43489 (time 1747848652.35 total 112.94 last 348.85) - open/close 47598 (time 1747848662.35 total 122.94 last 410.84) - open/close 50000 (time 1747848667.87 total 128.46 last 434.85) - open/close 54968 (time 1747848677.87 total 138.46 last 496.76) - open/close 59540 (time 1747848687.88 total 148.47 last 457.02) - open/close 63730 (time 1747848697.88 total 158.47 last 418.94) total: 64768 open/close in 161.13 seconds: 401.97 ops/second - /unlink 7811 (time 1747848711.58 total 10.00 last 781.04) - /unlink 10000 (time 1747848715.66 total 14.08 last 536.12) - /unlink 16194 (time 1747848725.66 total 24.08 last 619.38) - /unlink 20000 (time 1747848731.86 total 30.29 last 613.36) - /unlink 25715 (time 1747848741.86 total 40.29 last 571.47) - /unlink 30000 (time 1747848748.98 total 47.40 last 602.57) - /unlink 36802 (time 1747848758.98 total 57.40 last 680.13) - /unlink 40000 (time 1747848763.05 total 61.47 last 785.95) - /unlink 49047 (time 1747848773.05 total 71.47 last 904.69) - /unlink 60000 (time 1747848782.39 total 80.82 last 1171.85) total: 64768 /unlink in 85.40 seconds: 758.40 ops/second Waiting for MDT destroys to complete - open/close 6266 (time 1747848870.42 total 10.00 last 626.52) - open/close 10000 (time 1747848878.57 total 18.15 last 458.29) - open/close 15161 (time 1747848888.57 total 28.15 last 516.09) - open/close 19680 (time 1747848898.57 total 38.15 last 451.77) - open/close 25049 (time 1747848908.57 total 48.15 last 536.90) - open/close 30000 (time 1747848917.98 total 57.56 last 526.14) - open/close 35406 (time 1747848927.98 total 67.56 last 540.54) - open/close 40000 (time 1747848936.28 total 75.86 last 553.91) - open/close 45857 (time 1747848946.28 total 85.86 last 585.61) - open/close 50000 (time 1747848952.75 total 92.33 last 639.78) - open/close 55971 (time 1747848962.75 total 102.34 last 597.04) - open/close 60000 (time 1747848970.17 total 109.75 last 543.46) total: 64768 open/close in 118.52 seconds: 546.49 ops/second - /unlink 8811 (time 1747848989.75 total 10.00 last 881.04) - /unlink 18905 (time 1747848999.75 total 20.00 last 1009.37) - /unlink 27848 (time 1747849009.75 total 30.00 last 894.23) - /unlink 30000 (time 1747849012.38 total 32.63 last 819.36) - /unlink 38770 (time 1747849022.38 total 42.63 last 876.95) - /unlink 46740 (time 1747849032.38 total 52.63 last 796.98) - /unlink 50000 (time 1747849036.18 total 56.43 last 858.38) - /unlink 58963 (time 1747849046.18 total 66.43 last 896.26) total: 64768 /unlink in 74.06 seconds: 874.48 ops/second Waiting for MDT destroys to complete - open/close 5543 (time 1747849150.90 total 10.00 last 554.24) - open/close 9759 (time 1747849160.90 total 20.00 last 421.57) - open/close 14175 (time 1747849170.90 total 30.00 last 441.48) - open/close 19592 (time 1747849180.90 total 40.00 last 541.67) - open/close 24705 (time 1747849190.90 total 50.01 last 511.24) - open/close 30000 (time 1747849200.30 total 59.41 last 563.09) - open/close 36287 (time 1747849210.30 total 69.41 last 628.62) - open/close 40000 (time 1747849217.13 total 76.23 last 544.32) - open/close 45765 (time 1747849227.14 total 86.25 last 575.58) - open/close 50000 (time 1747849233.95 total 93.06 last 621.69) - open/close 55647 (time 1747849243.96 total 103.06 last 564.63) - open/close 60000 (time 1747849251.66 total 110.77 last 564.70) total: 64768 open/close in 118.00 seconds: 548.87 ops/second - /unlink 10000 (time 1747849268.74 total 9.08 last 1101.27) - /unlink 19888 (time 1747849278.74 total 19.08 last 988.75) - /unlink 30000 (time 1747849287.40 total 27.74 last 1167.55) - /unlink 39779 (time 1747849297.41 total 37.74 last 977.83) - /unlink 50000 (time 1747849306.93 total 47.27 last 1072.67) - /unlink 60000 (time 1747849315.11 total 55.45 last 1222.79) total: 64768 /unlink in 60.68 seconds: 1067.40 ops/second Waiting for MDT destroys to complete - open/close 4533 (time 1747849430.95 total 10.00 last 453.27) - open/close 9260 (time 1747849440.95 total 20.00 last 472.64) - open/close 15171 (time 1747849450.95 total 30.00 last 591.02) - open/close 20000 (time 1747849460.49 total 39.54 last 506.11) - open/close 24723 (time 1747849470.50 total 49.55 last 472.23) - open/close 29650 (time 1747849480.50 total 59.55 last 492.64) - open/close 34766 (time 1747849490.50 total 69.55 last 511.56) - open/close 40000 (time 1747849499.51 total 78.56 last 580.52) - open/close 46007 (time 1747849509.51 total 88.56 last 600.69) - open/close 50000 (time 1747849517.38 total 96.43 last 507.39) - open/close 55436 (time 1747849527.38 total 106.44 last 543.54) - open/close 60000 (time 1747849535.51 total 114.56 last 561.92) total: 64768 open/close in 123.35 seconds: 525.07 ops/second - /unlink 10000 (time 1747849554.24 total 9.13 last 1094.97) - /unlink 18801 (time 1747849564.24 total 19.13 last 880.04) - /unlink 27723 (time 1747849574.24 total 29.13 last 892.16) - /unlink 30000 (time 1747849576.38 total 31.27 last 1064.14) - /unlink 40000 (time 1747849585.25 total 40.14 last 1127.61) - /unlink 50000 (time 1747849594.07 total 48.97 last 1133.37) - /unlink 59979 (time 1747849604.07 total 58.97 last 997.81) total: 64768 /unlink in 64.26 seconds: 1007.86 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg430-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server unloading modules via unload_modules_local on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg430-client: oleg430-client: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-client: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2