== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:34:19 (1734258859) Stopping clients: oleg310-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg310-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg310-client: oleg310-server: ssh exited with exit code 2 oleg310-server: oleg310-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing load_modules_local oleg310-server: Loading modules from /home/green/git/lustre-release/lustre oleg310-server: detected 4 online CPUs by sysfs oleg310-server: Force libcfs to create 2 CPU partitions oleg310-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg310-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg310-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg310-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg310-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg310-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16131 291 15840 2% /mnt/lustre[OST:0] filesystem_summary: 16483 643 15840 4% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1417 (time 1734258937.38 total 10.00 last 141.69) - open/close 2872 (time 1734258947.39 total 20.01 last 145.38) - open/close 4304 (time 1734258957.40 total 30.02 last 143.10) - open/close 5866 (time 1734258967.40 total 40.02 last 156.17) - open/close 7298 (time 1734258977.40 total 50.02 last 143.16) - open/close 8807 (time 1734258987.40 total 60.02 last 150.87) - open/close 10000 (time 1734258994.53 total 67.15 last 167.47) - open/close 11693 (time 1734259004.53 total 77.15 last 169.28) - open/close 13516 (time 1734259014.53 total 87.15 last 182.28) - open/close 15131 (time 1734259024.53 total 97.15 last 161.46) - open/close 16631 (time 1734259034.54 total 107.16 last 149.90) - open/close 18082 (time 1734259044.54 total 117.16 last 145.01) - open/close 19569 (time 1734259054.55 total 127.17 last 148.66) - open/close 20000 (time 1734259057.53 total 130.15 last 144.42) - open/close 21471 (time 1734259067.53 total 140.15 last 147.09) - open/close 22902 (time 1734259077.53 total 150.15 last 143.06) - open/close 24321 (time 1734259087.53 total 160.15 last 141.89) - open/close 25726 (time 1734259097.54 total 170.16 last 140.45) - open/close 27131 (time 1734259107.54 total 180.16 last 140.41) - open/close 28664 (time 1734259117.54 total 190.16 last 153.29) - open/close 30000 (time 1734259125.69 total 198.31 last 163.98) - open/close 31432 (time 1734259135.69 total 208.31 last 143.19) - open/close 32843 (time 1734259145.69 total 218.31 last 141.08) - open/close 34298 (time 1734259155.70 total 228.32 last 145.40) - open/close 35874 (time 1734259165.70 total 238.32 last 157.57) - open/close 37538 (time 1734259175.71 total 248.33 last 166.33) - open/close 39065 (time 1734259185.71 total 258.33 last 152.68) - open/close 40000 (time 1734259192.54 total 265.16 last 136.90) - open/close 41518 (time 1734259202.54 total 275.16 last 151.79) - open/close 42937 (time 1734259212.54 total 285.16 last 141.85) - open/close 44347 (time 1734259222.54 total 295.16 last 140.98) - open/close 45898 (time 1734259232.55 total 305.17 last 155.06) - open/close 47486 (time 1734259242.55 total 315.17 last 158.68) - open/close 49435 (time 1734259252.56 total 325.18 last 194.87) - open/close 50000 (time 1734259255.40 total 328.02 last 198.49) - open/close 51794 (time 1734259265.41 total 338.03 last 179.33) - open/close 53297 (time 1734259275.41 total 348.03 last 150.20) - open/close 54995 (time 1734259285.42 total 358.04 last 169.72) - open/close 56631 (time 1734259295.42 total 368.04 last 163.51) - open/close 58239 (time 1734259305.43 total 378.05 last 160.69) - open/close 59847 (time 1734259315.43 total 388.05 last 160.74) - open/close 61606 (time 1734259325.44 total 398.06 last 175.81) - open/close 63597 (time 1734259335.44 total 408.06 last 199.10) total: 64768 open/close in 414.23 seconds: 156.36 ops/second - /unlink 2773 (time 1734259352.83 total 10.00 last 277.29) - /unlink 5533 (time 1734259362.84 total 20.00 last 275.92) - /unlink 8316 (time 1734259372.84 total 30.01 last 278.22) - /unlink 10000 (time 1734259378.87 total 36.03 last 279.43) - /unlink 12645 (time 1734259388.87 total 46.04 last 264.44) - /unlink 15536 (time 1734259398.87 total 56.04 last 289.06) - /unlink 18406 (time 1734259408.87 total 66.04 last 286.97) - /unlink 20000 (time 1734259414.59 total 71.76 last 278.62) - /unlink 22665 (time 1734259424.59 total 81.76 last 266.44) - /unlink 25307 (time 1734259434.59 total 91.76 last 264.19) - /unlink 28095 (time 1734259444.60 total 101.76 last 278.77) - /unlink 30000 (time 1734259451.21 total 108.37 last 288.11) - /unlink 32682 (time 1734259461.21 total 118.38 last 268.11) - /unlink 35350 (time 1734259471.21 total 128.38 last 266.75) - /unlink 38246 (time 1734259481.21 total 138.38 last 289.60) - /unlink 40000 (time 1734259486.64 total 143.81 last 323.13) - /unlink 42975 (time 1734259496.64 total 153.81 last 297.48) - /unlink 45614 (time 1734259506.64 total 163.81 last 263.85) - /unlink 48517 (time 1734259516.65 total 173.81 last 290.22) - /unlink 50000 (time 1734259521.48 total 178.65 last 306.71) - /unlink 53514 (time 1734259531.48 total 188.65 last 351.32) - /unlink 56755 (time 1734259541.49 total 198.66 last 323.94) - /unlink 59697 (time 1734259551.49 total 208.66 last 294.16) - /unlink 62897 (time 1734259561.49 total 218.66 last 319.99) total: 64768 /unlink in 224.76 seconds: 288.17 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1987 (time 1734259595.31 total 10.01 last 198.59) - open/close 4174 (time 1734259605.31 total 20.01 last 218.64) - open/close 6473 (time 1734259615.31 total 30.01 last 229.81) - open/close 8673 (time 1734259625.31 total 40.01 last 219.98) - open/close 10000 (time 1734259631.61 total 46.31 last 210.79) - open/close 12128 (time 1734259641.61 total 56.31 last 212.80) - open/close 14285 (time 1734259651.61 total 66.31 last 215.67) - open/close 16298 (time 1734259661.62 total 76.32 last 201.20) - open/close 18295 (time 1734259671.62 total 86.32 last 199.68) - open/close 20000 (time 1734259679.80 total 94.50 last 208.23) - open/close 22101 (time 1734259689.81 total 104.51 last 210.05) - open/close 24348 (time 1734259699.81 total 114.51 last 224.70) - open/close 26727 (time 1734259709.81 total 124.51 last 237.80) - open/close 28821 (time 1734259719.81 total 134.51 last 209.34) - open/close 30000 (time 1734259725.72 total 140.42 last 199.62) - open/close 31808 (time 1734259735.72 total 150.42 last 180.76) - open/close 33721 (time 1734259745.72 total 160.42 last 191.25) - open/close 35458 (time 1734259755.73 total 170.43 last 173.69) - open/close 37404 (time 1734259765.73 total 180.43 last 194.55) - open/close 39522 (time 1734259775.73 total 190.43 last 211.78) - open/close 40000 (time 1734259777.73 total 192.43 last 238.40) - open/close 42177 (time 1734259787.74 total 202.44 last 217.63) - open/close 44270 (time 1734259797.74 total 212.44 last 209.24) - open/close 45999 (time 1734259807.74 total 222.44 last 172.89) - open/close 47948 (time 1734259817.74 total 232.44 last 194.90) - open/close 50000 (time 1734259827.23 total 241.93 last 216.24) - open/close 52215 (time 1734259837.23 total 251.93 last 221.43) - open/close 54523 (time 1734259847.23 total 261.93 last 230.79) - open/close 56755 (time 1734259857.24 total 271.94 last 223.15) - open/close 58932 (time 1734259867.24 total 281.94 last 217.66) - open/close 60000 (time 1734259872.89 total 287.59 last 188.96) - open/close 61955 (time 1734259882.89 total 297.59 last 195.49) - open/close 63808 (time 1734259892.89 total 307.59 last 185.27) total: 64768 open/close in 312.90 seconds: 206.99 ops/second - /unlink 3221 (time 1734259909.47 total 10.00 last 322.08) - /unlink 6872 (time 1734259919.48 total 20.00 last 365.07) - /unlink 10000 (time 1734259927.64 total 28.17 last 383.14) - /unlink 13673 (time 1734259937.64 total 38.17 last 367.24) - /unlink 17247 (time 1734259947.64 total 48.17 last 357.33) - /unlink 20000 (time 1734259954.81 total 55.34 last 384.12) - /unlink 23755 (time 1734259964.81 total 65.34 last 375.46) - /unlink 27765 (time 1734259974.81 total 75.34 last 400.91) - /unlink 30000 (time 1734259980.72 total 81.25 last 378.18) - /unlink 34098 (time 1734259990.72 total 91.25 last 409.77) - /unlink 38646 (time 1734260000.72 total 101.25 last 454.78) - /unlink 40000 (time 1734260003.16 total 103.69 last 555.64) - /unlink 45107 (time 1734260013.16 total 113.69 last 510.64) - /unlink 49630 (time 1734260023.16 total 123.69 last 452.22) - /unlink 54304 (time 1734260033.17 total 133.69 last 467.34) - /unlink 58796 (time 1734260043.17 total 143.69 last 449.11) - /unlink 60000 (time 1734260046.27 total 146.80 last 387.88) total: 64768 /unlink in 156.03 seconds: 415.11 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2962 (time 1734260080.71 total 10.00 last 296.16) - open/close 5988 (time 1734260090.71 total 20.00 last 302.58) - open/close 8798 (time 1734260100.71 total 30.00 last 280.93) - open/close 10000 (time 1734260104.96 total 34.25 last 282.90) - open/close 13039 (time 1734260114.96 total 44.26 last 303.84) - open/close 15804 (time 1734260124.96 total 54.26 last 276.48) - open/close 18458 (time 1734260134.97 total 64.26 last 265.31) - open/close 20000 (time 1734260141.02 total 70.31 last 254.69) - open/close 22914 (time 1734260151.02 total 80.32 last 291.36) - open/close 26142 (time 1734260161.02 total 90.32 last 322.72) - open/close 29190 (time 1734260171.03 total 100.32 last 304.76) - open/close 30000 (time 1734260173.83 total 103.12 last 289.14) - open/close 32948 (time 1734260183.83 total 113.12 last 294.75) - open/close 36109 (time 1734260193.83 total 123.13 last 316.01) - open/close 38873 (time 1734260203.83 total 133.13 last 276.35) - open/close 40000 (time 1734260207.96 total 137.25 last 273.16) - open/close 42771 (time 1734260217.96 total 147.26 last 277.01) - open/close 45454 (time 1734260227.97 total 157.26 last 268.21) - open/close 48141 (time 1734260237.97 total 167.26 last 268.63) - open/close 50000 (time 1734260245.13 total 174.42 last 259.54) - open/close 52836 (time 1734260255.13 total 184.43 last 283.54) - open/close 55869 (time 1734260265.13 total 194.43 last 303.26) - open/close 58553 (time 1734260275.14 total 204.43 last 268.34) - open/close 60000 (time 1734260281.07 total 210.36 last 243.89) - open/close 62804 (time 1734260291.07 total 220.37 last 280.30) total: 64768 open/close in 228.94 seconds: 282.91 ops/second - /unlink 4203 (time 1734260310.73 total 10.00 last 420.23) - /unlink 8469 (time 1734260320.73 total 20.00 last 426.58) - /unlink 10000 (time 1734260323.93 total 23.20 last 479.35) - /unlink 14509 (time 1734260333.93 total 33.20 last 450.83) - /unlink 18982 (time 1734260343.93 total 43.20 last 447.28) - /unlink 20000 (time 1734260346.19 total 45.46 last 449.77) - /unlink 24952 (time 1734260356.19 total 55.46 last 495.14) - /unlink 29402 (time 1734260366.20 total 65.46 last 444.94) - /unlink 33898 (time 1734260376.20 total 75.46 last 449.56) - /unlink 38092 (time 1734260386.20 total 85.47 last 419.34) - /unlink 40000 (time 1734260390.65 total 89.92 last 428.32) - /unlink 43979 (time 1734260400.65 total 99.92 last 397.81) - /unlink 47766 (time 1734260410.66 total 109.92 last 378.65) - /unlink 50000 (time 1734260415.64 total 114.91 last 448.08) - /unlink 54355 (time 1734260425.64 total 124.91 last 435.47) - /unlink 58628 (time 1734260435.64 total 134.91 last 427.23) - /unlink 60000 (time 1734260439.11 total 138.38 last 395.34) - /unlink 64205 (time 1734260449.12 total 148.38 last 420.45) total: 64768 /unlink in 149.65 seconds: 432.80 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2694 (time 1734260475.94 total 10.00 last 269.33) - open/close 5798 (time 1734260485.94 total 20.00 last 310.32) - open/close 8564 (time 1734260495.95 total 30.01 last 276.55) - open/close 10000 (time 1734260501.13 total 35.19 last 277.18) - open/close 12779 (time 1734260511.13 total 45.19 last 277.85) - open/close 15422 (time 1734260521.13 total 55.19 last 264.28) - open/close 18262 (time 1734260531.13 total 65.19 last 283.91) - open/close 20000 (time 1734260536.72 total 70.78 last 311.10) - open/close 22501 (time 1734260546.72 total 80.78 last 250.09) - open/close 25024 (time 1734260556.72 total 90.78 last 252.30) - open/close 27732 (time 1734260566.72 total 100.78 last 270.74) - open/close 30000 (time 1734260574.33 total 108.39 last 298.07) - open/close 33063 (time 1734260584.33 total 118.39 last 306.28) - open/close 36286 (time 1734260594.33 total 128.39 last 322.26) - open/close 39098 (time 1734260604.33 total 138.39 last 281.19) - open/close 40000 (time 1734260607.64 total 141.70 last 272.68) - open/close 43003 (time 1734260617.64 total 151.70 last 300.22) - open/close 46027 (time 1734260627.65 total 161.71 last 302.32) - open/close 48764 (time 1734260637.65 total 171.71 last 273.63) - open/close 50000 (time 1734260641.91 total 175.97 last 290.03) - open/close 53304 (time 1734260651.91 total 185.97 last 330.37) - open/close 56738 (time 1734260661.91 total 195.97 last 343.33) - open/close 59912 (time 1734260671.92 total 205.98 last 317.31) - open/close 63033 (time 1734260681.92 total 215.98 last 312.07) total: 64768 open/close in 221.48 seconds: 292.43 ops/second - /unlink 5514 (time 1734260698.38 total 10.00 last 551.39) - /unlink 10000 (time 1734260706.98 total 18.60 last 521.87) - /unlink 15715 (time 1734260716.98 total 28.60 last 571.41) - /unlink 20000 (time 1734260724.12 total 35.74 last 600.25) - /unlink 26071 (time 1734260734.12 total 45.74 last 607.05) - /unlink 30000 (time 1734260740.04 total 51.65 last 664.03) - /unlink 37016 (time 1734260750.04 total 61.66 last 701.46) - /unlink 40000 (time 1734260754.92 total 66.53 last 611.87) - /unlink 46987 (time 1734260764.92 total 76.53 last 698.66) - /unlink 50000 (time 1734260769.13 total 80.75 last 715.45) - /unlink 58885 (time 1734260779.13 total 90.75 last 888.47) total: 64768 /unlink in 97.62 seconds: 663.45 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4607 (time 1734260816.89 total 10.00 last 460.59) - open/close 9701 (time 1734260826.89 total 20.00 last 509.39) - open/close 14726 (time 1734260836.89 total 30.00 last 502.41) - open/close 19609 (time 1734260846.89 total 40.01 last 488.26) - open/close 24399 (time 1734260856.89 total 50.01 last 478.96) - open/close 29924 (time 1734260866.89 total 60.01 last 552.49) - open/close 35908 (time 1734260876.90 total 70.01 last 598.34) - open/close 40000 (time 1734260884.24 total 77.35 last 556.96) - open/close 44887 (time 1734260894.24 total 87.35 last 488.68) - open/close 49783 (time 1734260904.24 total 97.36 last 489.52) - open/close 54694 (time 1734260914.25 total 107.36 last 491.08) - open/close 59390 (time 1734260924.25 total 117.36 last 469.55) total: 64768 open/close in 127.29 seconds: 508.81 ops/second - /unlink 10000 (time 1734260944.92 total 9.97 last 1003.01) - /unlink 19685 (time 1734260954.92 total 19.97 last 968.45) - /unlink 27864 (time 1734260964.93 total 29.97 last 817.83) - /unlink 30000 (time 1734260967.52 total 32.56 last 823.62) - /unlink 38119 (time 1734260977.52 total 42.57 last 811.82) - /unlink 40000 (time 1734260979.84 total 44.89 last 809.08) - /unlink 48163 (time 1734260989.84 total 54.89 last 816.30) - /unlink 50000 (time 1734260992.20 total 57.25 last 779.10) - /unlink 58458 (time 1734261002.20 total 67.25 last 845.75) total: 64768 /unlink in 74.57 seconds: 868.52 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg310-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg310-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg310-server stop mds service on oleg310-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg310-server stop mds service on oleg310-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg310-server unloading modules via unload_modules_local on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg310-client: oleg310-client: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-server: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-client: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-server: ssh exited with exit code 2