== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:35:22 (1734258922) Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 oleg317-server: oleg317-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg317-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16131 291 15840 2% /mnt/lustre[OST:0] filesystem_summary: 16483 643 15840 4% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1682 (time 1734258996.69 total 10.01 last 168.07) - open/close 3457 (time 1734259006.69 total 20.01 last 177.49) - open/close 5237 (time 1734259016.69 total 30.01 last 177.98) - open/close 6801 (time 1734259026.69 total 40.01 last 156.35) - open/close 8263 (time 1734259036.69 total 50.01 last 146.17) - open/close 9713 (time 1734259046.70 total 60.02 last 144.92) - open/close 11211 (time 1734259056.70 total 70.02 last 149.72) - open/close 12786 (time 1734259066.71 total 80.03 last 157.40) - open/close 14224 (time 1734259076.72 total 90.04 last 143.69) - open/close 15645 (time 1734259086.72 total 100.04 last 142.10) - open/close 17060 (time 1734259096.72 total 110.04 last 141.49) - open/close 18426 (time 1734259106.73 total 120.05 last 136.46) - open/close 19984 (time 1734259116.73 total 130.05 last 155.78) - open/close 21662 (time 1734259126.73 total 140.05 last 167.77) - open/close 23078 (time 1734259136.73 total 150.05 last 141.59) - open/close 24476 (time 1734259146.74 total 160.06 last 139.76) - open/close 25945 (time 1734259156.74 total 170.06 last 146.82) - open/close 27553 (time 1734259166.74 total 180.06 last 160.76) - open/close 29172 (time 1734259176.75 total 190.07 last 161.85) - open/close 30000 (time 1734259181.91 total 195.23 last 160.46) - open/close 31437 (time 1734259191.91 total 205.23 last 143.70) - open/close 32995 (time 1734259201.91 total 215.23 last 155.75) - open/close 34470 (time 1734259211.92 total 225.24 last 147.42) - open/close 35898 (time 1734259221.92 total 235.24 last 142.80) - open/close 37468 (time 1734259231.92 total 245.24 last 156.96) - open/close 39068 (time 1734259241.92 total 255.24 last 159.98) - open/close 40000 (time 1734259246.84 total 260.16 last 189.28) - open/close 41981 (time 1734259256.85 total 270.17 last 198.09) - open/close 43813 (time 1734259266.85 total 280.17 last 183.14) - open/close 45312 (time 1734259276.85 total 290.17 last 149.86) - open/close 47041 (time 1734259286.85 total 300.17 last 172.87) - open/close 48641 (time 1734259296.86 total 310.18 last 159.96) - open/close 50000 (time 1734259305.56 total 318.88 last 156.07) - open/close 51585 (time 1734259315.56 total 328.88 last 158.49) - open/close 53264 (time 1734259325.57 total 338.89 last 167.78) - open/close 55142 (time 1734259335.58 total 348.90 last 187.70) - open/close 56945 (time 1734259345.58 total 358.90 last 180.29) - open/close 58484 (time 1734259355.58 total 368.90 last 153.85) - open/close 60000 (time 1734259365.22 total 378.54 last 157.28) - open/close 61584 (time 1734259375.22 total 388.54 last 158.32) - open/close 63126 (time 1734259385.23 total 398.55 last 154.14) total: 64768 open/close in 408.43 seconds: 158.58 ops/second - /unlink 2723 (time 1734259406.50 total 10.00 last 272.27) - /unlink 5387 (time 1734259416.50 total 20.00 last 266.35) - /unlink 8004 (time 1734259426.50 total 30.00 last 261.68) - /unlink 10000 (time 1734259434.01 total 37.51 last 265.80) - /unlink 12722 (time 1734259444.01 total 47.51 last 272.17) - /unlink 15447 (time 1734259454.02 total 57.52 last 272.37) - /unlink 18089 (time 1734259464.02 total 67.52 last 264.17) - /unlink 20000 (time 1734259471.16 total 74.66 last 267.71) - /unlink 22728 (time 1734259481.16 total 84.66 last 272.74) - /unlink 25800 (time 1734259491.16 total 94.66 last 307.10) - /unlink 28605 (time 1734259501.16 total 104.67 last 280.42) - /unlink 30000 (time 1734259506.81 total 110.31 last 246.95) - /unlink 32886 (time 1734259516.82 total 120.32 last 288.56) - /unlink 35998 (time 1734259526.82 total 130.32 last 311.16) - /unlink 39607 (time 1734259536.82 total 140.32 last 360.88) - /unlink 42543 (time 1734259546.82 total 150.32 last 293.54) - /unlink 45428 (time 1734259556.82 total 160.32 last 288.43) - /unlink 48417 (time 1734259566.82 total 170.33 last 298.80) - /unlink 50000 (time 1734259572.51 total 176.01 last 278.25) - /unlink 53052 (time 1734259582.52 total 186.02 last 305.13) - /unlink 56341 (time 1734259592.52 total 196.02 last 328.84) - /unlink 59834 (time 1734259602.52 total 206.02 last 349.29) - /unlink 63695 (time 1734259612.52 total 216.02 last 386.01) total: 64768 /unlink in 218.84 seconds: 295.96 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2075 (time 1734259641.10 total 10.00 last 207.42) - open/close 4114 (time 1734259651.10 total 20.01 last 203.83) - open/close 6100 (time 1734259661.10 total 30.01 last 198.60) - open/close 8097 (time 1734259671.10 total 40.01 last 199.66) - open/close 10000 (time 1734259680.97 total 49.88 last 192.81) - open/close 12014 (time 1734259690.97 total 59.88 last 201.38) - open/close 14186 (time 1734259700.98 total 69.88 last 217.16) - open/close 16383 (time 1734259710.98 total 79.88 last 219.65) - open/close 18344 (time 1734259720.98 total 89.89 last 196.05) - open/close 20000 (time 1734259729.42 total 98.33 last 196.26) - open/close 21803 (time 1734259739.42 total 108.33 last 180.28) - open/close 23664 (time 1734259749.43 total 118.33 last 186.00) - open/close 25445 (time 1734259759.43 total 128.33 last 178.06) - open/close 27382 (time 1734259769.43 total 138.34 last 193.61) - open/close 29370 (time 1734259779.43 total 148.34 last 198.78) - open/close 30000 (time 1734259782.52 total 151.43 last 204.11) - open/close 32074 (time 1734259792.52 total 161.43 last 207.30) - open/close 33864 (time 1734259802.53 total 171.43 last 178.93) - open/close 35569 (time 1734259812.53 total 181.44 last 170.42) - open/close 37486 (time 1734259822.54 total 191.44 last 191.66) - open/close 39686 (time 1734259832.54 total 201.44 last 219.92) - open/close 41853 (time 1734259842.54 total 211.45 last 216.66) - open/close 44063 (time 1734259852.54 total 221.45 last 220.96) - open/close 46158 (time 1734259862.55 total 231.45 last 209.40) - open/close 48037 (time 1734259872.55 total 241.46 last 187.81) - open/close 49898 (time 1734259882.56 total 251.46 last 186.05) - open/close 51649 (time 1734259892.56 total 261.46 last 175.04) - open/close 53355 (time 1734259902.56 total 271.47 last 170.53) - open/close 55249 (time 1734259912.56 total 281.47 last 189.38) - open/close 57245 (time 1734259922.56 total 291.47 last 199.60) - open/close 59230 (time 1734259932.57 total 301.47 last 198.42) - open/close 60000 (time 1734259936.45 total 305.36 last 198.17) - open/close 61834 (time 1734259946.45 total 315.36 last 183.39) - open/close 63739 (time 1734259956.46 total 325.36 last 190.41) total: 64768 open/close in 331.00 seconds: 195.67 ops/second - /unlink 3335 (time 1734259973.33 total 10.00 last 333.41) - /unlink 6741 (time 1734259983.33 total 20.00 last 340.58) - /unlink 10000 (time 1734259992.37 total 29.04 last 360.61) - /unlink 14193 (time 1734260002.37 total 39.04 last 419.23) - /unlink 18669 (time 1734260012.37 total 49.04 last 447.49) - /unlink 20000 (time 1734260015.72 total 52.40 last 397.25) - /unlink 23949 (time 1734260025.72 total 62.40 last 394.85) - /unlink 28067 (time 1734260035.72 total 72.40 last 411.79) - /unlink 30000 (time 1734260040.68 total 77.36 last 389.77) - /unlink 33972 (time 1734260050.68 total 87.36 last 397.12) - /unlink 38968 (time 1734260060.68 total 97.36 last 499.59) - /unlink 43850 (time 1734260070.69 total 107.36 last 488.12) - /unlink 48191 (time 1734260080.69 total 117.36 last 434.07) - /unlink 50000 (time 1734260084.78 total 121.45 last 442.17) - /unlink 54293 (time 1734260094.78 total 131.45 last 429.22) - /unlink 58746 (time 1734260104.78 total 141.46 last 445.22) - /unlink 60000 (time 1734260107.44 total 144.11 last 472.09) - /unlink 64525 (time 1734260117.44 total 154.11 last 452.44) total: 64768 /unlink in 154.67 seconds: 418.74 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2556 (time 1734260146.37 total 10.00 last 255.59) - open/close 5278 (time 1734260156.38 total 20.00 last 272.15) - open/close 8248 (time 1734260166.38 total 30.00 last 296.91) - open/close 10000 (time 1734260172.85 total 36.48 last 270.75) - open/close 12703 (time 1734260182.85 total 46.48 last 270.25) - open/close 15590 (time 1734260192.85 total 56.48 last 288.68) - open/close 18103 (time 1734260202.85 total 66.48 last 251.28) - open/close 20000 (time 1734260210.44 total 74.07 last 249.91) - open/close 22447 (time 1734260220.45 total 84.07 last 244.65) - open/close 24822 (time 1734260230.45 total 94.07 last 237.46) - open/close 27220 (time 1734260240.45 total 104.08 last 239.73) - open/close 29632 (time 1734260250.45 total 114.08 last 241.12) - open/close 32383 (time 1734260260.45 total 124.08 last 275.08) - open/close 34905 (time 1734260270.46 total 134.08 last 252.18) - open/close 37168 (time 1734260280.46 total 144.08 last 226.26) - open/close 39600 (time 1734260290.46 total 154.09 last 243.13) - open/close 41638 (time 1734260300.46 total 164.09 last 203.76) - open/close 43807 (time 1734260310.47 total 174.09 last 216.78) - open/close 45979 (time 1734260320.47 total 184.10 last 217.16) - open/close 48376 (time 1734260330.47 total 194.10 last 239.64) - open/close 50000 (time 1734260337.49 total 201.12 last 231.32) - open/close 52351 (time 1734260347.49 total 211.12 last 235.04) - open/close 54783 (time 1734260357.50 total 221.13 last 243.08) - open/close 57031 (time 1734260367.50 total 231.13 last 224.71) - open/close 59258 (time 1734260377.51 total 241.13 last 222.66) - open/close 60000 (time 1734260381.16 total 244.78 last 203.20) - open/close 62134 (time 1734260391.16 total 254.78 last 213.37) - open/close 64149 (time 1734260401.16 total 264.79 last 201.47) total: 64768 open/close in 268.08 seconds: 241.60 ops/second - /unlink 3523 (time 1734260415.72 total 10.00 last 352.24) - /unlink 7337 (time 1734260425.72 total 20.00 last 381.35) - /unlink 10000 (time 1734260432.75 total 27.03 last 379.00) - /unlink 13471 (time 1734260442.75 total 37.03 last 347.02) - /unlink 17080 (time 1734260452.75 total 47.03 last 360.85) - /unlink 20000 (time 1734260460.60 total 54.88 last 372.01) - /unlink 24096 (time 1734260470.60 total 64.88 last 409.57) - /unlink 28685 (time 1734260480.60 total 74.88 last 458.84) - /unlink 30000 (time 1734260483.43 total 77.71 last 464.87) - /unlink 34318 (time 1734260493.43 total 87.72 last 431.64) - /unlink 38937 (time 1734260503.43 total 97.72 last 461.83) - /unlink 40000 (time 1734260505.80 total 100.08 last 449.52) - /unlink 44479 (time 1734260515.80 total 110.08 last 447.88) - /unlink 49191 (time 1734260525.80 total 120.09 last 471.08) - /unlink 54551 (time 1734260535.80 total 130.09 last 535.93) - /unlink 58858 (time 1734260545.81 total 140.09 last 430.55) - /unlink 60000 (time 1734260549.16 total 143.45 last 340.39) - /unlink 64263 (time 1734260559.16 total 153.45 last 426.27) total: 64768 /unlink in 154.61 seconds: 418.90 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2720 (time 1734260587.09 total 10.00 last 271.94) - open/close 5497 (time 1734260597.10 total 20.01 last 277.61) - open/close 7918 (time 1734260607.10 total 30.01 last 242.01) - open/close 10000 (time 1734260615.48 total 38.39 last 248.57) - open/close 12689 (time 1734260625.48 total 48.39 last 268.81) - open/close 15091 (time 1734260635.48 total 58.39 last 240.18) - open/close 17643 (time 1734260645.48 total 68.39 last 255.20) - open/close 20000 (time 1734260654.15 total 77.05 last 272.03) - open/close 23016 (time 1734260664.15 total 87.06 last 301.56) - open/close 25555 (time 1734260674.15 total 97.06 last 253.82) - open/close 28086 (time 1734260684.15 total 107.06 last 253.03) - open/close 30000 (time 1734260691.10 total 114.01 last 275.46) - open/close 32725 (time 1734260701.10 total 124.01 last 272.50) - open/close 35448 (time 1734260711.10 total 134.01 last 272.28) - open/close 38510 (time 1734260721.10 total 144.01 last 306.20) - open/close 40000 (time 1734260725.51 total 148.41 last 338.34) - open/close 43504 (time 1734260735.51 total 158.42 last 350.37) - open/close 47194 (time 1734260745.51 total 168.42 last 368.98) - open/close 50000 (time 1734260753.40 total 176.30 last 355.72) - open/close 53611 (time 1734260763.40 total 186.31 last 361.02) - open/close 57796 (time 1734260773.40 total 196.31 last 418.44) - open/close 60000 (time 1734260777.58 total 200.49 last 526.68) - open/close 64468 (time 1734260787.59 total 210.50 last 446.68) total: 64768 open/close in 211.19 seconds: 306.68 ops/second - /unlink 9610 (time 1734260799.13 total 10.00 last 960.93) - /unlink 18343 (time 1734260809.13 total 20.00 last 873.16) - /unlink 20000 (time 1734260811.45 total 22.32 last 716.10) - /unlink 27781 (time 1734260821.45 total 32.32 last 778.02) - /unlink 30000 (time 1734260824.08 total 34.95 last 843.33) - /unlink 38189 (time 1734260834.08 total 44.95 last 818.86) - /unlink 40000 (time 1734260836.33 total 47.20 last 805.97) - /unlink 47955 (time 1734260846.33 total 57.20 last 795.48) - /unlink 50000 (time 1734260849.10 total 59.97 last 736.72) - /unlink 57552 (time 1734260859.11 total 69.97 last 755.13) - /unlink 60000 (time 1734260862.10 total 72.97 last 817.85) total: 64768 /unlink in 78.02 seconds: 830.19 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4784 (time 1734260897.94 total 10.00 last 478.39) - open/close 9757 (time 1734260907.94 total 20.00 last 497.29) - open/close 14533 (time 1734260917.94 total 30.00 last 477.56) - open/close 19320 (time 1734260927.94 total 40.00 last 478.69) - open/close 24770 (time 1734260937.94 total 50.00 last 544.91) - open/close 30000 (time 1734260947.30 total 59.36 last 558.97) - open/close 35419 (time 1734260957.30 total 69.36 last 541.87) - open/close 39928 (time 1734260967.30 total 79.36 last 450.89) - open/close 44692 (time 1734260977.30 total 89.36 last 476.35) - open/close 49553 (time 1734260987.30 total 99.36 last 486.05) - open/close 54170 (time 1734260997.30 total 109.36 last 461.69) - open/close 58875 (time 1734261007.30 total 119.36 last 470.42) - open/close 60000 (time 1734261009.59 total 121.65 last 491.22) total: 64768 open/close in 131.40 seconds: 492.89 ops/second - /unlink 7863 (time 1734261030.20 total 10.00 last 786.24) - /unlink 10000 (time 1734261032.77 total 12.57 last 833.36) - /unlink 18662 (time 1734261042.77 total 22.57 last 866.16) - /unlink 27394 (time 1734261052.77 total 32.57 last 873.13) - /unlink 30000 (time 1734261055.80 total 35.60 last 858.72) - /unlink 38513 (time 1734261065.80 total 45.60 last 851.29) - /unlink 47187 (time 1734261075.80 total 55.60 last 867.39) - /unlink 50000 (time 1734261079.14 total 58.94 last 842.73) - /unlink 58809 (time 1734261089.14 total 68.94 last 880.82) total: 64768 /unlink in 75.57 seconds: 857.03 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2