== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 14:10:31 (1722795031) Stopping clients: oleg252-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg252-client.virtnet /mnt/lustre2 (opts:-f) oleg252-server: oleg252-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg252-server' oleg252-server: oleg252-server.virtnet: executing load_modules_local oleg252-server: Loading modules from /home/green/git/lustre-release/lustre oleg252-server: detected 4 online CPUs by sysfs oleg252-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg252-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg252-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg252-client.virtnet: -o user_xattr,flock oleg252-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4668 (time 1722795066.21 total 10.00 last 466.73) - open/close 10000 (time 1722795076.20 total 19.99 last 534.06) - open/close 15492 (time 1722795086.20 total 29.99 last 549.16) - open/close 20000 (time 1722795095.64 total 39.42 last 477.60) - open/close 24118 (time 1722795105.64 total 49.43 last 411.72) - open/close 27994 (time 1722795115.64 total 59.43 last 387.54) - open/close 30000 (time 1722795120.66 total 64.45 last 399.18) - open/close 34756 (time 1722795130.67 total 74.45 last 475.55) - open/close 38972 (time 1722795140.67 total 84.46 last 421.47) - open/close 40000 (time 1722795144.34 total 88.13 last 280.27) - open/close 44587 (time 1722795154.34 total 98.13 last 458.62) - open/close 49994 (time 1722795164.34 total 108.13 last 540.63) - open/close 55416 (time 1722795174.34 total 118.13 last 542.15) - open/close 60000 (time 1722795183.24 total 127.03 last 514.95) - open/close 63635 (time 1722795193.24 total 137.03 last 363.48) total: 64768 open/close in 139.86 seconds: 463.10 ops/second - /unlink 7361 (time 1722795207.07 total 10.00 last 736.02) - /unlink 10000 (time 1722795210.55 total 13.49 last 757.38) - /unlink 17868 (time 1722795220.55 total 23.49 last 786.74) - /unlink 20000 (time 1722795222.96 total 25.89 last 886.58) - /unlink 28392 (time 1722795232.96 total 35.89 last 839.17) - /unlink 37311 (time 1722795242.96 total 45.89 last 891.87) - /unlink 40000 (time 1722795246.19 total 49.12 last 832.33) - /unlink 48230 (time 1722795256.19 total 59.12 last 822.96) - /unlink 50000 (time 1722795258.30 total 61.24 last 837.56) - /unlink 58636 (time 1722795268.30 total 71.24 last 863.56) total: 64768 /unlink in 78.71 seconds: 822.86 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3818 (time 1722795306.09 total 10.00 last 381.70) - open/close 7472 (time 1722795316.10 total 20.00 last 365.31) - open/close 10000 (time 1722795322.29 total 26.20 last 408.02) - open/close 13115 (time 1722795332.29 total 36.20 last 311.47) - open/close 17010 (time 1722795342.29 total 46.20 last 389.47) - open/close 20000 (time 1722795351.50 total 55.41 last 324.76) - open/close 22889 (time 1722795361.50 total 65.41 last 288.84) - open/close 26652 (time 1722795371.50 total 75.41 last 376.29) - open/close 30000 (time 1722795380.45 total 84.35 last 374.39) - open/close 33631 (time 1722795390.45 total 94.36 last 363.06) - open/close 37330 (time 1722795400.45 total 104.36 last 369.86) - open/close 40000 (time 1722795408.21 total 112.12 last 344.04) - open/close 44214 (time 1722795418.21 total 122.12 last 421.35) - open/close 47725 (time 1722795428.25 total 132.16 last 349.75) - open/close 50000 (time 1722795432.99 total 136.90 last 479.91) - open/close 54954 (time 1722795442.99 total 146.90 last 495.36) - open/close 60000 (time 1722795452.53 total 156.44 last 528.95) total: 64768 open/close in 166.30 seconds: 389.47 ops/second - /unlink 6913 (time 1722795473.30 total 10.00 last 691.27) - /unlink 10000 (time 1722795477.38 total 14.08 last 755.94) - /unlink 18164 (time 1722795487.38 total 24.08 last 816.34) - /unlink 20000 (time 1722795489.41 total 26.11 last 906.50) - /unlink 27949 (time 1722795499.41 total 36.11 last 794.85) - /unlink 30000 (time 1722795503.48 total 40.18 last 503.80) - /unlink 37235 (time 1722795513.48 total 50.18 last 723.43) - /unlink 40000 (time 1722795517.67 total 54.37 last 660.07) - /unlink 47486 (time 1722795527.67 total 64.37 last 748.58) - /unlink 50000 (time 1722795531.18 total 67.88 last 716.50) - /unlink 58002 (time 1722795541.18 total 77.88 last 800.12) - /unlink 60000 (time 1722795543.67 total 80.38 last 800.34) total: 64768 /unlink in 85.63 seconds: 756.39 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5169 (time 1722795580.73 total 10.00 last 516.88) - open/close 10000 (time 1722795590.42 total 19.69 last 498.36) - open/close 13324 (time 1722795600.42 total 29.70 last 332.34) - open/close 16810 (time 1722795610.42 total 39.70 last 348.57) - open/close 20000 (time 1722795618.76 total 48.03 last 382.60) - open/close 24032 (time 1722795628.76 total 58.03 last 403.19) - open/close 28094 (time 1722795638.76 total 68.04 last 406.14) - open/close 30000 (time 1722795642.61 total 71.88 last 495.76) - open/close 33947 (time 1722795652.61 total 81.88 last 394.69) - open/close 37322 (time 1722795662.61 total 91.88 last 337.50) - open/close 40000 (time 1722795668.67 total 97.94 last 441.69) - open/close 44266 (time 1722795678.67 total 107.94 last 426.58) - open/close 47901 (time 1722795688.67 total 117.95 last 363.46) - open/close 50000 (time 1722795695.63 total 124.91 last 301.55) - open/close 52852 (time 1722795705.63 total 134.91 last 285.11) - open/close 55447 (time 1722795715.63 total 144.91 last 259.50) - open/close 58433 (time 1722795725.64 total 154.91 last 298.57) - open/close 60000 (time 1722795731.31 total 160.58 last 276.26) - open/close 62795 (time 1722795741.31 total 170.58 last 279.46) total: 64768 open/close in 177.46 seconds: 364.98 ops/second - /unlink 7151 (time 1722795759.36 total 10.00 last 715.04) - /unlink 10000 (time 1722795764.77 total 15.41 last 526.51) - /unlink 15874 (time 1722795774.77 total 25.41 last 587.39) - /unlink 20000 (time 1722795782.90 total 33.54 last 507.87) - /unlink 24602 (time 1722795792.90 total 43.54 last 460.08) - /unlink 29436 (time 1722795802.90 total 53.54 last 483.36) - /unlink 35377 (time 1722795812.90 total 63.54 last 594.04) - /unlink 40000 (time 1722795822.75 total 73.39 last 469.47) - /unlink 44964 (time 1722795832.75 total 83.39 last 496.35) - /unlink 50000 (time 1722795840.07 total 90.71 last 687.74) - /unlink 54882 (time 1722795850.07 total 100.71 last 488.09) - /unlink 59563 (time 1722795860.07 total 110.72 last 468.04) total: 64768 /unlink in 119.64 seconds: 541.35 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2704 (time 1722795897.35 total 10.00 last 270.32) - open/close 5618 (time 1722795907.35 total 20.01 last 291.33) - open/close 9118 (time 1722795917.35 total 30.01 last 349.92) - open/close 10000 (time 1722795920.85 total 33.51 last 252.06) - open/close 13449 (time 1722795930.85 total 43.51 last 344.87) - open/close 16384 (time 1722795940.86 total 53.51 last 293.41) - open/close 19404 (time 1722795950.86 total 63.51 last 302.00) - open/close 20000 (time 1722795953.16 total 65.81 last 258.73) - open/close 22844 (time 1722795963.16 total 75.82 last 284.37) - open/close 25452 (time 1722795973.16 total 85.82 last 260.76) - open/close 28604 (time 1722795983.16 total 95.82 last 315.10) - open/close 30000 (time 1722795988.45 total 101.10 last 264.18) - open/close 33115 (time 1722795998.45 total 111.11 last 311.47) - open/close 35669 (time 1722796008.47 total 121.12 last 254.99) - open/close 38283 (time 1722796018.47 total 131.12 last 261.37) - open/close 40000 (time 1722796024.93 total 137.58 last 265.81) - open/close 42698 (time 1722796034.93 total 147.58 last 269.76) - open/close 45280 (time 1722796044.93 total 157.58 last 258.17) - open/close 47957 (time 1722796054.93 total 167.59 last 267.67) - open/close 50000 (time 1722796062.39 total 175.04 last 273.92) - open/close 53117 (time 1722796072.39 total 185.05 last 311.65) - open/close 56061 (time 1722796082.39 total 195.05 last 294.29) - open/close 58856 (time 1722796092.39 total 205.05 last 279.49) - open/close 60000 (time 1722796096.94 total 209.59 last 251.87) - open/close 62548 (time 1722796106.94 total 219.59 last 254.73) total: 64768 open/close in 227.63 seconds: 284.54 ops/second - /unlink 4863 (time 1722796126.07 total 10.00 last 486.28) - /unlink 10000 (time 1722796135.79 total 19.71 last 528.87) - /unlink 14764 (time 1722796145.79 total 29.71 last 476.33) - /unlink 19916 (time 1722796155.79 total 39.72 last 515.17) - /unlink 25875 (time 1722796165.79 total 49.72 last 595.89) - /unlink 30000 (time 1722796171.53 total 55.46 last 718.74) - /unlink 35172 (time 1722796181.53 total 65.46 last 517.12) - /unlink 40000 (time 1722796190.70 total 74.63 last 526.23) - /unlink 45153 (time 1722796200.71 total 84.63 last 515.19) - /unlink 50000 (time 1722796210.24 total 94.17 last 508.18) - /unlink 54802 (time 1722796220.25 total 104.17 last 480.14) - /unlink 59638 (time 1722796230.25 total 114.17 last 483.57) total: 64768 /unlink in 123.53 seconds: 524.30 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2715 (time 1722796266.69 total 10.00 last 271.44) - open/close 5466 (time 1722796276.69 total 20.00 last 275.04) - open/close 7971 (time 1722796286.69 total 30.01 last 250.46) - open/close 10000 (time 1722796294.51 total 37.82 last 259.55) - open/close 13254 (time 1722796304.51 total 47.82 last 325.37) - open/close 15821 (time 1722796314.51 total 57.83 last 256.65) - open/close 18688 (time 1722796324.51 total 67.83 last 286.68) - open/close 20000 (time 1722796327.81 total 71.13 last 397.11) - open/close 24071 (time 1722796337.81 total 81.13 last 407.09) - open/close 28691 (time 1722796347.81 total 91.13 last 461.98) - open/close 30000 (time 1722796350.90 total 94.22 last 423.69) - open/close 34115 (time 1722796360.91 total 104.22 last 411.44) - open/close 37684 (time 1722796370.91 total 114.23 last 356.81) - open/close 40000 (time 1722796375.91 total 119.23 last 462.70) - open/close 44265 (time 1722796385.91 total 129.23 last 426.48) - open/close 47964 (time 1722796395.91 total 139.23 last 369.89) - open/close 50000 (time 1722796400.56 total 143.88 last 437.92) - open/close 54632 (time 1722796410.57 total 153.88 last 463.09) - open/close 57259 (time 1722796420.57 total 163.88 last 262.67) - open/close 59905 (time 1722796430.57 total 173.89 last 264.54) - open/close 62357 (time 1722796440.57 total 183.89 last 245.16) total: 64768 open/close in 193.51 seconds: 334.70 ops/second - /unlink 4425 (time 1722796461.46 total 10.00 last 442.41) - /unlink 9198 (time 1722796471.46 total 20.00 last 477.26) - /unlink 13953 (time 1722796481.46 total 30.00 last 475.47) - /unlink 18651 (time 1722796491.46 total 40.00 last 469.76) - /unlink 20000 (time 1722796494.13 total 42.67 last 505.89) - /unlink 27134 (time 1722796504.13 total 52.67 last 713.35) - /unlink 30000 (time 1722796510.24 total 58.78 last 469.27) - /unlink 35168 (time 1722796520.24 total 68.78 last 516.76) - /unlink 40000 (time 1722796528.57 total 77.11 last 580.07) - /unlink 47491 (time 1722796538.57 total 87.11 last 749.05) - /unlink 50000 (time 1722796541.93 total 90.48 last 745.38) - /unlink 57058 (time 1722796551.94 total 100.48 last 705.73) - /unlink 60000 (time 1722796556.01 total 104.56 last 721.33) total: 64768 /unlink in 111.42 seconds: 581.28 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg252-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg252-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg252-server stop mds service on oleg252-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg252-server unloading modules on: 'oleg252-server' oleg252-server: oleg252-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg252-client: oleg252-client: ssh exited with exit code 2 pdsh@oleg252-client: oleg252-server: ssh exited with exit code 2 pdsh@oleg252-client: oleg252-client: ssh exited with exit code 2 pdsh@oleg252-client: oleg252-server: ssh exited with exit code 2