== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 21:03:26 (1745456606) Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg339-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4303 (time 1745456640.71 total 10.00 last 430.28) - open/close 9315 (time 1745456650.71 total 20.00 last 501.16) - open/close 14647 (time 1745456660.71 total 30.00 last 533.14) - open/close 19173 (time 1745456670.71 total 40.00 last 452.59) - open/close 20000 (time 1745456673.59 total 42.88 last 287.78) - open/close 21186 (time 1745456683.60 total 52.88 last 118.51) - open/close 22464 (time 1745456693.60 total 62.89 last 127.78) - open/close 25402 (time 1745456703.60 total 72.89 last 293.78) - open/close 28003 (time 1745456713.60 total 82.89 last 260.02) - open/close 30000 (time 1745456722.89 total 92.18 last 214.98) - open/close 32192 (time 1745456732.89 total 102.18 last 219.12) - open/close 35401 (time 1745456742.89 total 112.18 last 320.87) - open/close 38636 (time 1745456752.90 total 122.19 last 323.42) - open/close 40000 (time 1745456755.90 total 125.19 last 453.51) - open/close 43304 (time 1745456765.91 total 135.19 last 330.36) - open/close 46142 (time 1745456775.91 total 145.20 last 283.77) - open/close 49262 (time 1745456785.91 total 155.20 last 311.97) - open/close 50000 (time 1745456789.47 total 158.76 last 206.90) - open/close 52468 (time 1745456799.48 total 168.77 last 246.66) - open/close 54781 (time 1745456809.48 total 178.77 last 231.26) - open/close 57346 (time 1745456819.49 total 188.77 last 256.41) - open/close 59086 (time 1745456829.49 total 198.78 last 173.93) - open/close 60000 (time 1745456834.69 total 203.98 last 175.83) - open/close 61777 (time 1745456844.69 total 213.98 last 177.64) - open/close 63807 (time 1745456854.69 total 223.98 last 202.94) total: 64768 open/close in 228.59 seconds: 283.33 ops/second - /unlink 3162 (time 1745456870.45 total 10.00 last 316.17) - /unlink 6309 (time 1745456880.46 total 20.00 last 314.68) - /unlink 9639 (time 1745456890.46 total 30.00 last 332.99) - /unlink 12824 (time 1745456900.46 total 40.01 last 318.26) - /unlink 16262 (time 1745456910.47 total 50.01 last 343.73) - /unlink 19910 (time 1745456920.47 total 60.02 last 364.67) - /unlink 24146 (time 1745456930.47 total 70.02 last 423.50) - /unlink 27947 (time 1745456940.47 total 80.02 last 380.00) - /unlink 30000 (time 1745456946.44 total 85.98 last 344.35) - /unlink 34022 (time 1745456956.44 total 95.98 last 402.15) - /unlink 37808 (time 1745456966.44 total 105.99 last 378.50) - /unlink 40000 (time 1745456972.32 total 111.86 last 372.90) - /unlink 42761 (time 1745456982.32 total 121.87 last 276.06) - /unlink 46194 (time 1745456992.32 total 131.87 last 343.28) - /unlink 49786 (time 1745457002.32 total 141.87 last 359.19) - /unlink 52590 (time 1745457012.32 total 151.87 last 280.40) - /unlink 56109 (time 1745457022.32 total 161.87 last 351.76) - /unlink 59676 (time 1745457032.32 total 171.87 last 356.69) - /unlink 64581 (time 1745457042.33 total 181.87 last 490.41) total: 64768 /unlink in 182.26 seconds: 355.36 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2449 (time 1745457070.70 total 10.00 last 244.88) - open/close 5385 (time 1745457080.70 total 20.00 last 293.51) - open/close 7849 (time 1745457090.70 total 30.01 last 246.37) - open/close 10000 (time 1745457098.08 total 37.38 last 291.63) - open/close 13287 (time 1745457108.08 total 47.39 last 328.54) - open/close 15976 (time 1745457118.08 total 57.39 last 268.88) - open/close 19038 (time 1745457128.08 total 67.39 last 306.15) - open/close 20000 (time 1745457131.00 total 70.30 last 329.84) - open/close 23040 (time 1745457141.00 total 80.31 last 303.96) - open/close 26392 (time 1745457151.00 total 90.31 last 335.16) - open/close 30000 (time 1745457160.83 total 100.14 last 366.98) - open/close 33513 (time 1745457170.84 total 110.14 last 351.23) - open/close 36422 (time 1745457180.84 total 120.14 last 290.90) - open/close 40000 (time 1745457190.28 total 129.59 last 378.75) - open/close 43636 (time 1745457200.28 total 139.59 last 363.59) - open/close 47391 (time 1745457210.28 total 149.59 last 375.50) - open/close 50000 (time 1745457218.73 total 158.04 last 308.76) - open/close 53891 (time 1745457228.74 total 168.04 last 389.02) - open/close 57468 (time 1745457238.74 total 178.04 last 357.63) - open/close 60000 (time 1745457246.77 total 186.08 last 315.16) - open/close 63664 (time 1745457256.77 total 196.08 last 366.39) total: 64768 open/close in 198.95 seconds: 325.55 ops/second - /unlink 8614 (time 1745457270.54 total 10.00 last 861.33) - /unlink 16184 (time 1745457280.55 total 20.00 last 756.96) - /unlink 20000 (time 1745457284.61 total 24.06 last 939.13) - /unlink 28858 (time 1745457294.61 total 34.07 last 885.77) - /unlink 37075 (time 1745457304.61 total 44.07 last 821.63) - /unlink 40000 (time 1745457308.31 total 47.76 last 791.34) - /unlink 47363 (time 1745457318.31 total 57.76 last 736.28) - /unlink 50000 (time 1745457321.67 total 61.13 last 782.89) - /unlink 58339 (time 1745457331.68 total 71.13 last 833.85) - /unlink 60000 (time 1745457333.75 total 73.20 last 802.10) total: 64768 /unlink in 79.77 seconds: 811.92 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5507 (time 1745457370.38 total 10.00 last 550.65) - open/close 10000 (time 1745457378.06 total 17.68 last 584.93) - open/close 15853 (time 1745457388.06 total 27.68 last 585.26) - open/close 20000 (time 1745457395.60 total 35.23 last 549.80) - open/close 25999 (time 1745457405.60 total 45.23 last 599.89) - open/close 30000 (time 1745457412.70 total 52.32 last 563.90) - open/close 35645 (time 1745457422.70 total 62.32 last 564.45) - open/close 40000 (time 1745457430.30 total 69.92 last 572.91) - open/close 45027 (time 1745457440.30 total 79.92 last 502.69) - open/close 50000 (time 1745457450.29 total 89.91 last 497.84) - open/close 55365 (time 1745457460.29 total 99.91 last 536.45) - open/close 60000 (time 1745457470.16 total 109.79 last 469.36) - open/close 64629 (time 1745457480.17 total 119.79 last 462.79) total: 64768 open/close in 120.19 seconds: 538.88 ops/second - /unlink 7282 (time 1745457491.43 total 10.00 last 728.19) - /unlink 10000 (time 1745457495.02 total 13.59 last 757.71) - /unlink 17017 (time 1745457505.02 total 23.59 last 701.67) - /unlink 20000 (time 1745457509.06 total 27.63 last 737.61) - /unlink 29062 (time 1745457519.07 total 37.63 last 906.10) - /unlink 37181 (time 1745457529.08 total 47.64 last 811.15) - /unlink 40000 (time 1745457532.02 total 50.58 last 958.62) - /unlink 49522 (time 1745457542.02 total 60.58 last 952.13) - /unlink 56822 (time 1745457552.02 total 70.58 last 729.98) - /unlink 60000 (time 1745457555.57 total 74.14 last 893.75) total: 64768 /unlink in 79.80 seconds: 811.63 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5077 (time 1745457595.66 total 10.00 last 507.69) - open/close 10000 (time 1745457604.56 total 18.90 last 552.93) - open/close 15263 (time 1745457614.56 total 28.90 last 526.27) - open/close 20000 (time 1745457623.61 total 37.95 last 523.61) - open/close 25508 (time 1745457633.61 total 47.95 last 550.79) - open/close 29588 (time 1745457643.61 total 57.95 last 407.94) - open/close 33953 (time 1745457653.61 total 67.95 last 436.45) - open/close 38360 (time 1745457663.61 total 77.95 last 440.70) - open/close 40000 (time 1745457667.51 total 81.85 last 420.84) - open/close 44649 (time 1745457677.51 total 91.85 last 464.83) - open/close 49658 (time 1745457687.51 total 101.85 last 500.84) - open/close 54731 (time 1745457697.55 total 111.89 last 505.21) - open/close 60000 (time 1745457707.24 total 121.58 last 544.09) total: 64768 open/close in 131.37 seconds: 493.04 ops/second - /unlink 7595 (time 1745457727.92 total 10.00 last 759.42) - /unlink 10000 (time 1745457731.10 total 13.18 last 757.43) - /unlink 18241 (time 1745457741.10 total 23.18 last 824.06) - /unlink 27901 (time 1745457751.10 total 33.18 last 965.92) - /unlink 30000 (time 1745457753.48 total 35.56 last 881.94) - /unlink 39370 (time 1745457763.48 total 45.56 last 936.94) - /unlink 48208 (time 1745457773.48 total 55.56 last 883.71) - /unlink 50000 (time 1745457775.83 total 57.90 last 764.26) - /unlink 58195 (time 1745457785.83 total 67.90 last 819.43) total: 64768 /unlink in 75.16 seconds: 861.74 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4585 (time 1745457825.79 total 10.00 last 458.50) - open/close 9274 (time 1745457835.79 total 20.00 last 468.83) - open/close 14611 (time 1745457845.80 total 30.00 last 533.63) - open/close 20000 (time 1745457855.15 total 39.36 last 575.87) - open/close 25710 (time 1745457865.15 total 49.36 last 570.94) - open/close 30000 (time 1745457873.50 total 57.70 last 514.29) - open/close 35385 (time 1745457883.50 total 67.70 last 538.47) - open/close 40000 (time 1745457891.81 total 76.02 last 555.13) - open/close 44789 (time 1745457901.81 total 86.02 last 478.83) - open/close 49965 (time 1745457911.81 total 96.02 last 517.56) - open/close 55540 (time 1745457921.81 total 106.02 last 557.43) - open/close 59978 (time 1745457931.81 total 116.02 last 443.76) total: 64768 open/close in 124.87 seconds: 518.70 ops/second - /unlink 8412 (time 1745457951.53 total 10.00 last 841.13) - /unlink 16166 (time 1745457961.53 total 20.00 last 775.39) - /unlink 20000 (time 1745457966.33 total 24.80 last 798.89) - /unlink 27920 (time 1745457976.33 total 34.80 last 791.99) - /unlink 30000 (time 1745457978.71 total 37.18 last 873.91) - /unlink 37976 (time 1745457988.71 total 47.18 last 797.59) - /unlink 40000 (time 1745457991.18 total 49.65 last 818.78) - /unlink 48676 (time 1745458001.19 total 59.65 last 867.55) - /unlink 57705 (time 1745458011.19 total 69.65 last 902.82) - /unlink 60000 (time 1745458013.77 total 72.24 last 888.11) total: 64768 /unlink in 77.51 seconds: 835.65 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2