== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:09:14 (1749121754) Stopping clients: oleg109-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg109-client.virtnet /mnt/lustre2 (opts:-f) oleg109-server: oleg109-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg109-server' oleg109-server: oleg109-server.virtnet: executing load_modules_local oleg109-server: Loading modules from /home/green/git/lustre-release/lustre oleg109-server: detected 4 online CPUs by sysfs oleg109-server: MODOPTS_LIBCFS= oleg109-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg109-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg109-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg109-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1401 (time 1749121822.94 total 10.00 last 140.06) - open/close 2794 (time 1749121832.95 total 20.01 last 139.21) - open/close 4167 (time 1749121842.96 total 30.02 last 137.17) - open/close 5572 (time 1749121852.96 total 40.02 last 140.44) - open/close 7022 (time 1749121862.97 total 50.03 last 144.94) - open/close 8328 (time 1749121872.98 total 60.04 last 130.44) - open/close 9694 (time 1749121882.99 total 70.05 last 136.47) - open/close 10000 (time 1749121885.07 total 72.13 last 147.12) - open/close 11399 (time 1749121895.08 total 82.14 last 139.69) - open/close 12699 (time 1749121905.08 total 92.14 last 130.00) - open/close 14061 (time 1749121915.09 total 102.15 last 136.15) - open/close 15288 (time 1749121925.09 total 112.16 last 122.61) - open/close 16659 (time 1749121935.10 total 122.16 last 137.00) - open/close 18042 (time 1749121945.11 total 132.17 last 138.23) - open/close 19486 (time 1749121955.11 total 142.17 last 144.34) - open/close 20000 (time 1749121958.48 total 145.54 last 152.41) - open/close 21532 (time 1749121968.49 total 155.55 last 153.15) - open/close 23070 (time 1749121978.49 total 165.55 last 153.76) - open/close 24516 (time 1749121988.50 total 175.56 last 144.49) - open/close 25936 (time 1749121998.50 total 185.57 last 141.89) - open/close 27461 (time 1749122008.51 total 195.57 last 152.38) - open/close 29019 (time 1749122018.51 total 205.57 last 155.79) - open/close 30000 (time 1749122025.40 total 212.46 last 142.52) - open/close 31428 (time 1749122035.40 total 222.46 last 142.76) - open/close 32945 (time 1749122045.41 total 232.47 last 151.59) - open/close 34528 (time 1749122055.41 total 242.47 last 158.25) - open/close 36101 (time 1749122065.41 total 252.47 last 157.28) - open/close 37691 (time 1749122075.41 total 262.47 last 158.97) - open/close 39593 (time 1749122085.42 total 272.48 last 190.17) - open/close 40000 (time 1749122087.52 total 274.59 last 192.91) - open/close 41789 (time 1749122097.53 total 284.59 last 178.89) - open/close 43623 (time 1749122107.53 total 294.59 last 183.35) - open/close 45327 (time 1749122117.53 total 304.59 last 170.38) - open/close 47116 (time 1749122127.53 total 314.59 last 178.85) - open/close 48947 (time 1749122137.54 total 324.60 last 182.99) - open/close 50000 (time 1749122143.32 total 330.38 last 182.06) - open/close 51963 (time 1749122153.32 total 340.39 last 196.26) - open/close 53735 (time 1749122163.33 total 350.39 last 177.09) - open/close 55803 (time 1749122173.33 total 360.39 last 206.76) - open/close 57782 (time 1749122183.33 total 370.39 last 197.87) - open/close 59791 (time 1749122193.34 total 380.40 last 200.85) - open/close 61571 (time 1749122203.34 total 390.40 last 177.88) - open/close 63322 (time 1749122213.35 total 400.41 last 175.03) total: 64768 open/close in 408.31 seconds: 158.62 ops/second - /unlink 3679 (time 1749122232.60 total 10.00 last 367.80) - /unlink 7042 (time 1749122242.60 total 20.00 last 336.29) - /unlink 10000 (time 1749122250.52 total 27.92 last 373.48) - /unlink 13719 (time 1749122260.52 total 37.93 last 371.80) - /unlink 17216 (time 1749122270.53 total 47.93 last 349.57) - /unlink 20000 (time 1749122278.90 total 56.31 last 332.37) - /unlink 23725 (time 1749122288.91 total 66.31 last 372.43) - /unlink 27302 (time 1749122298.91 total 76.31 last 357.63) - /unlink 30000 (time 1749122306.20 total 83.60 last 369.93) - /unlink 33430 (time 1749122316.20 total 93.61 last 342.87) - /unlink 37181 (time 1749122326.21 total 103.61 last 374.98) - /unlink 40000 (time 1749122334.06 total 111.46 last 359.20) - /unlink 43622 (time 1749122344.06 total 121.46 last 362.12) - /unlink 47520 (time 1749122354.06 total 131.46 last 389.79) - /unlink 50000 (time 1749122359.02 total 136.42 last 500.13) - /unlink 54842 (time 1749122369.02 total 146.42 last 484.10) - /unlink 60000 (time 1749122378.96 total 156.36 last 518.92) - /unlink 64475 (time 1749122388.96 total 166.36 last 447.47) total: 64768 /unlink in 167.05 seconds: 387.71 ops/second Waiting for MDT destroys to complete - open/close 2506 (time 1749122407.85 total 10.00 last 250.52) - open/close 4819 (time 1749122417.85 total 20.00 last 231.29) - open/close 7453 (time 1749122427.85 total 30.00 last 263.39) - open/close 10000 (time 1749122436.39 total 38.54 last 298.53) - open/close 12935 (time 1749122446.39 total 48.54 last 293.42) - open/close 16192 (time 1749122456.39 total 58.54 last 325.64) - open/close 20000 (time 1749122465.95 total 68.09 last 398.57) - open/close 23428 (time 1749122475.95 total 78.09 last 342.78) - open/close 25818 (time 1749122485.95 total 88.10 last 238.93) - open/close 28219 (time 1749122495.95 total 98.10 last 240.08) - open/close 30000 (time 1749122501.71 total 103.85 last 309.41) - open/close 33548 (time 1749122511.71 total 113.86 last 354.78) - open/close 36944 (time 1749122521.71 total 123.86 last 339.56) - open/close 39988 (time 1749122531.71 total 133.86 last 304.32) - open/close 43587 (time 1749122541.71 total 143.86 last 359.87) - open/close 46594 (time 1749122551.71 total 153.86 last 300.68) - open/close 49560 (time 1749122561.71 total 163.86 last 296.54) - open/close 53261 (time 1749122571.71 total 173.86 last 370.05) - open/close 56694 (time 1749122581.72 total 183.86 last 343.26) - open/close 59874 (time 1749122591.72 total 193.87 last 317.91) - open/close 63240 (time 1749122601.72 total 203.87 last 336.59) total: 64768 open/close in 208.06 seconds: 311.29 ops/second - /unlink 7536 (time 1749122616.82 total 10.00 last 753.49) - /unlink 10000 (time 1749122620.10 total 13.29 last 749.89) - /unlink 18729 (time 1749122630.10 total 23.29 last 872.89) - /unlink 26407 (time 1749122640.10 total 33.29 last 767.75) - /unlink 30000 (time 1749122644.14 total 37.33 last 889.78) - /unlink 37947 (time 1749122654.14 total 47.33 last 794.68) - /unlink 40000 (time 1749122656.95 total 50.13 last 731.41) - /unlink 46774 (time 1749122666.95 total 60.13 last 677.33) - /unlink 50000 (time 1749122671.36 total 64.54 last 731.46) - /unlink 58194 (time 1749122681.37 total 74.55 last 818.71) - /unlink 60000 (time 1749122683.40 total 76.58 last 891.73) total: 64768 /unlink in 82.29 seconds: 787.03 ops/second Waiting for MDT destroys to complete - open/close 4397 (time 1749122778.60 total 10.00 last 439.69) - open/close 9000 (time 1749122788.60 total 20.00 last 460.26) - open/close 10000 (time 1749122790.92 total 22.32 last 430.86) - open/close 14021 (time 1749122800.93 total 32.32 last 402.03) - open/close 18409 (time 1749122810.93 total 42.33 last 438.74) - open/close 20000 (time 1749122815.14 total 46.53 last 377.96) - open/close 23700 (time 1749122825.14 total 56.54 last 369.95) - open/close 27165 (time 1749122835.14 total 66.54 last 346.43) - open/close 30000 (time 1749122843.67 total 75.07 last 332.39) - open/close 33460 (time 1749122853.67 total 85.07 last 345.97) - open/close 38214 (time 1749122863.67 total 95.07 last 475.39) - open/close 40000 (time 1749122867.10 total 98.50 last 520.48) - open/close 44794 (time 1749122877.10 total 108.50 last 479.34) - open/close 48985 (time 1749122887.10 total 118.50 last 419.04) - open/close 54442 (time 1749122897.10 total 128.50 last 545.68) - open/close 59683 (time 1749122907.11 total 138.50 last 524.04) - open/close 64525 (time 1749122917.11 total 148.51 last 484.10) total: 64768 open/close in 149.01 seconds: 434.66 ops/second - /unlink 8328 (time 1749122928.54 total 10.00 last 832.77) - /unlink 17581 (time 1749122938.54 total 20.00 last 925.29) - /unlink 20000 (time 1749122941.45 total 22.91 last 830.66) - /unlink 29213 (time 1749122951.46 total 32.91 last 921.23) - /unlink 38669 (time 1749122961.46 total 42.91 last 945.54) - /unlink 48237 (time 1749122971.46 total 52.91 last 956.80) - /unlink 57070 (time 1749122981.46 total 62.91 last 883.22) - /unlink 60000 (time 1749122985.08 total 66.54 last 807.96) total: 64768 /unlink in 71.33 seconds: 908.00 ops/second Waiting for MDT destroys to complete - open/close 5102 (time 1749123083.47 total 10.00 last 510.18) - open/close 9855 (time 1749123093.47 total 20.00 last 475.22) - open/close 13909 (time 1749123103.47 total 30.01 last 405.25) - open/close 18613 (time 1749123113.48 total 40.01 last 470.26) - open/close 20000 (time 1749123116.36 total 42.89 last 480.61) - open/close 24819 (time 1749123126.36 total 52.90 last 481.83) - open/close 30000 (time 1749123134.76 total 61.29 last 616.90) - open/close 35695 (time 1749123144.76 total 71.30 last 569.44) - open/close 40000 (time 1749123152.03 total 78.56 last 592.45) - open/close 45329 (time 1749123162.03 total 88.56 last 532.86) - open/close 50000 (time 1749123170.02 total 96.55 last 584.60) - open/close 55681 (time 1749123180.02 total 106.55 last 568.04) - open/close 60000 (time 1749123187.81 total 114.34 last 554.80) total: 64768 open/close in 123.33 seconds: 525.18 ops/second - /unlink 9204 (time 1749123207.65 total 10.00 last 920.34) - /unlink 18442 (time 1749123217.65 total 20.00 last 923.74) - /unlink 27630 (time 1749123227.65 total 30.00 last 918.77) - /unlink 30000 (time 1749123230.27 total 32.62 last 903.93) - /unlink 39394 (time 1749123240.28 total 42.63 last 938.45) - /unlink 48940 (time 1749123250.28 total 52.63 last 954.51) - /unlink 58392 (time 1749123260.28 total 62.64 last 945.14) total: 64768 /unlink in 69.40 seconds: 933.22 ops/second Waiting for MDT destroys to complete - open/close 5439 (time 1749123369.70 total 10.00 last 543.83) - open/close 10000 (time 1749123378.12 total 18.42 last 541.66) - open/close 15409 (time 1749123388.13 total 28.42 last 540.88) - open/close 20000 (time 1749123396.66 total 36.95 last 538.22) - open/close 25341 (time 1749123406.66 total 46.95 last 534.07) - open/close 30000 (time 1749123415.51 total 55.81 last 526.15) - open/close 35110 (time 1749123425.51 total 65.81 last 510.91) - open/close 40000 (time 1749123434.80 total 75.10 last 526.25) - open/close 45279 (time 1749123444.81 total 85.10 last 527.84) - open/close 50000 (time 1749123453.72 total 94.02 last 529.36) - open/close 55270 (time 1749123463.72 total 104.02 last 526.99) - open/close 60000 (time 1749123472.73 total 113.03 last 525.10) total: 64768 open/close in 122.10 seconds: 530.45 ops/second - /unlink 9574 (time 1749123492.65 total 10.00 last 957.34) - /unlink 19269 (time 1749123502.65 total 20.00 last 969.45) - /unlink 28856 (time 1749123512.65 total 30.00 last 958.65) - /unlink 38312 (time 1749123522.65 total 40.00 last 945.60) - /unlink 48218 (time 1749123532.65 total 50.00 last 990.59) - /unlink 57600 (time 1749123542.65 total 60.00 last 938.14) - /unlink 60000 (time 1749123545.12 total 62.47 last 971.00) total: 64768 /unlink in 67.36 seconds: 961.50 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg109-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg109-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg109-server stop mds service on oleg109-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg109-server stop mds service on oleg109-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg109-server unloading modules via unload_modules_local on: 'oleg109-server' oleg109-server: oleg109-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg109-client: oleg109-client: ssh exited with exit code 2 pdsh@oleg109-client: oleg109-server: ssh exited with exit code 2 pdsh@oleg109-client: oleg109-client: ssh exited with exit code 2 pdsh@oleg109-client: oleg109-server: ssh exited with exit code 2