== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:32:18 (1734258738) Stopping clients: oleg144-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg144-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg144-client: oleg144-server: ssh exited with exit code 2 oleg144-server: oleg144-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing load_modules_local oleg144-server: Loading modules from /home/green/git/lustre-release/lustre oleg144-server: detected 4 online CPUs by sysfs oleg144-server: Force libcfs to create 2 CPU partitions oleg144-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg144-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg144-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg144-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg144-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg144-client.virtnet: -o user_xattr,flock 192.168.201.144@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16266 458 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16451 643 15808 4% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1607 (time 1734258819.23 total 10.00 last 160.68) - open/close 3449 (time 1734258829.23 total 20.00 last 184.13) - open/close 5193 (time 1734258839.23 total 30.01 last 174.34) - open/close 6870 (time 1734258849.23 total 40.01 last 167.67) - open/close 8528 (time 1734258859.24 total 50.02 last 165.67) - open/close 10000 (time 1734258868.18 total 58.96 last 164.61) - open/close 11712 (time 1734258878.19 total 68.96 last 171.16) - open/close 13326 (time 1734258888.19 total 78.97 last 161.35) - open/close 14985 (time 1734258898.19 total 88.97 last 165.86) - open/close 16599 (time 1734258908.20 total 98.98 last 161.28) - open/close 18291 (time 1734258918.20 total 108.98 last 169.17) - open/close 20000 (time 1734258927.96 total 118.74 last 175.07) - open/close 21852 (time 1734258937.97 total 128.74 last 185.13) - open/close 23794 (time 1734258947.97 total 138.75 last 194.15) - open/close 25584 (time 1734258957.97 total 148.75 last 178.95) - open/close 27134 (time 1734258967.97 total 158.75 last 154.97) - open/close 28655 (time 1734258977.98 total 168.76 last 152.02) - open/close 30000 (time 1734258986.85 total 177.62 last 151.64) - open/close 31522 (time 1734258996.85 total 187.63 last 152.15) - open/close 33108 (time 1734259006.86 total 197.64 last 158.48) - open/close 34704 (time 1734259016.87 total 207.64 last 159.48) - open/close 36397 (time 1734259026.87 total 217.64 last 169.29) - open/close 37969 (time 1734259036.87 total 227.65 last 157.16) - open/close 39520 (time 1734259046.87 total 237.65 last 155.05) - open/close 40000 (time 1734259050.04 total 240.81 last 151.74) - open/close 41625 (time 1734259060.04 total 250.82 last 162.42) - open/close 43081 (time 1734259070.05 total 260.82 last 145.54) - open/close 44614 (time 1734259080.05 total 270.82 last 153.29) - open/close 46325 (time 1734259090.05 total 280.82 last 171.09) - open/close 47811 (time 1734259100.05 total 290.83 last 148.56) - open/close 49413 (time 1734259110.05 total 300.83 last 160.17) - open/close 50000 (time 1734259114.07 total 304.84 last 146.22) - open/close 51435 (time 1734259124.07 total 314.85 last 143.45) - open/close 52913 (time 1734259134.07 total 324.85 last 147.74) - open/close 54400 (time 1734259144.08 total 334.86 last 148.61) - open/close 55812 (time 1734259154.08 total 344.86 last 141.16) - open/close 57189 (time 1734259164.09 total 354.86 last 137.65) - open/close 58681 (time 1734259174.09 total 364.87 last 149.11) - open/close 60000 (time 1734259183.32 total 374.10 last 142.92) - open/close 61518 (time 1734259193.33 total 384.10 last 151.71) - open/close 62979 (time 1734259203.34 total 394.12 last 145.91) - open/close 64373 (time 1734259213.35 total 404.12 last 139.32) total: 64768 open/close in 406.73 seconds: 159.24 ops/second - /unlink 2390 (time 1734259227.61 total 10.00 last 238.98) - /unlink 4859 (time 1734259237.61 total 20.00 last 246.87) - /unlink 7277 (time 1734259247.62 total 30.01 last 241.66) - /unlink 9905 (time 1734259257.62 total 40.01 last 262.72) - /unlink 12544 (time 1734259267.63 total 50.02 last 263.78) - /unlink 15363 (time 1734259277.63 total 60.02 last 281.86) - /unlink 18290 (time 1734259287.63 total 70.02 last 292.63) - /unlink 20000 (time 1734259293.83 total 76.22 last 275.70) - /unlink 22717 (time 1734259303.83 total 86.22 last 271.62) - /unlink 25453 (time 1734259313.84 total 96.23 last 273.55) - /unlink 28031 (time 1734259323.84 total 106.23 last 257.76) - /unlink 30000 (time 1734259331.61 total 114.00 last 253.23) - /unlink 32279 (time 1734259341.62 total 124.01 last 227.84) - /unlink 34842 (time 1734259351.62 total 134.01 last 256.26) - /unlink 37416 (time 1734259361.62 total 144.01 last 257.34) - /unlink 39812 (time 1734259371.62 total 154.01 last 239.58) - /unlink 42536 (time 1734259381.62 total 164.01 last 272.40) - /unlink 45462 (time 1734259391.62 total 174.01 last 292.51) - /unlink 48346 (time 1734259401.63 total 184.02 last 288.19) - /unlink 50000 (time 1734259407.34 total 189.73 last 289.65) - /unlink 53029 (time 1734259417.34 total 199.73 last 302.83) - /unlink 56168 (time 1734259427.36 total 209.75 last 313.54) - /unlink 59220 (time 1734259437.36 total 219.75 last 305.01) - /unlink 60000 (time 1734259439.90 total 222.29 last 306.74) - /unlink 62857 (time 1734259449.91 total 232.30 last 285.62) total: 64768 /unlink in 238.55 seconds: 271.51 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1986 (time 1734259485.22 total 10.00 last 198.60) - open/close 4190 (time 1734259495.22 total 20.00 last 220.36) - open/close 6195 (time 1734259505.22 total 30.00 last 200.47) - open/close 8229 (time 1734259515.23 total 40.01 last 203.32) - open/close 10000 (time 1734259524.29 total 49.07 last 195.48) - open/close 11988 (time 1734259534.29 total 59.07 last 198.78) - open/close 13781 (time 1734259544.30 total 69.08 last 179.19) - open/close 15324 (time 1734259554.30 total 79.08 last 154.21) - open/close 16880 (time 1734259564.30 total 89.08 last 155.58) - open/close 18524 (time 1734259574.31 total 99.08 last 164.38) - open/close 20000 (time 1734259582.90 total 107.68 last 171.72) - open/close 21557 (time 1734259592.90 total 117.68 last 155.67) - open/close 23239 (time 1734259602.91 total 127.69 last 168.12) - open/close 25039 (time 1734259612.91 total 137.69 last 179.97) - open/close 26850 (time 1734259622.91 total 147.69 last 181.08) - open/close 28518 (time 1734259632.91 total 157.69 last 166.72) - open/close 30000 (time 1734259641.55 total 166.33 last 171.65) - open/close 31789 (time 1734259651.55 total 176.33 last 178.88) - open/close 33558 (time 1734259661.55 total 186.33 last 176.87) - open/close 35299 (time 1734259671.55 total 196.33 last 174.10) - open/close 36992 (time 1734259681.55 total 206.33 last 169.29) - open/close 38890 (time 1734259691.55 total 216.33 last 189.76) - open/close 40000 (time 1734259697.78 total 222.56 last 178.27) - open/close 41794 (time 1734259707.79 total 232.56 last 179.33) - open/close 43408 (time 1734259717.80 total 242.58 last 161.13) - open/close 45010 (time 1734259727.81 total 252.59 last 160.09) - open/close 46650 (time 1734259737.81 total 262.59 last 163.90) - open/close 48588 (time 1734259747.93 total 272.71 last 191.63) - open/close 50000 (time 1734259755.57 total 280.35 last 184.80) - open/close 51757 (time 1734259765.57 total 290.35 last 175.68) - open/close 53604 (time 1734259775.57 total 300.35 last 184.62) - open/close 55664 (time 1734259785.58 total 310.36 last 205.91) - open/close 57910 (time 1734259795.58 total 320.36 last 224.59) - open/close 59801 (time 1734259805.58 total 330.36 last 189.08) - open/close 61877 (time 1734259815.58 total 340.36 last 207.54) - open/close 63828 (time 1734259825.59 total 350.37 last 195.02) total: 64768 open/close in 355.56 seconds: 182.16 ops/second - /unlink 2911 (time 1734259842.37 total 10.00 last 291.08) - /unlink 5791 (time 1734259852.37 total 20.00 last 287.91) - /unlink 8933 (time 1734259862.37 total 30.00 last 314.17) - /unlink 10000 (time 1734259866.50 total 34.14 last 258.12) - /unlink 12831 (time 1734259876.51 total 44.14 last 283.05) - /unlink 15885 (time 1734259886.51 total 54.14 last 305.36) - /unlink 19054 (time 1734259896.51 total 64.14 last 316.86) - /unlink 20000 (time 1734259899.72 total 67.35 last 294.65) - /unlink 23338 (time 1734259909.72 total 77.35 last 333.80) - /unlink 26937 (time 1734259919.72 total 87.35 last 359.87) - /unlink 30000 (time 1734259928.14 total 95.77 last 363.81) - /unlink 33606 (time 1734259938.14 total 105.78 last 360.47) - /unlink 37287 (time 1734259948.15 total 115.78 last 368.03) - /unlink 40000 (time 1734259955.46 total 123.09 last 371.07) - /unlink 43941 (time 1734259965.46 total 133.09 last 394.03) - /unlink 47603 (time 1734259975.46 total 143.09 last 366.18) - /unlink 50000 (time 1734259981.73 total 149.36 last 382.30) - /unlink 53711 (time 1734259991.73 total 159.36 last 371.10) - /unlink 57531 (time 1734260001.73 total 169.36 last 381.96) - /unlink 60000 (time 1734260007.72 total 175.36 last 412.07) - /unlink 64027 (time 1734260017.72 total 185.36 last 402.64) total: 64768 /unlink in 187.16 seconds: 346.06 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2160 (time 1734260046.52 total 10.01 last 215.87) - open/close 4335 (time 1734260056.52 total 20.01 last 217.46) - open/close 6350 (time 1734260066.53 total 30.01 last 201.37) - open/close 8162 (time 1734260076.53 total 40.02 last 181.14) - open/close 10000 (time 1734260086.25 total 49.73 last 189.18) - open/close 12125 (time 1734260096.25 total 59.74 last 212.39) - open/close 14067 (time 1734260106.26 total 69.74 last 194.15) - open/close 16163 (time 1734260116.26 total 79.74 last 209.60) - open/close 18241 (time 1734260126.26 total 89.75 last 207.71) - open/close 20000 (time 1734260134.06 total 97.55 last 225.46) - open/close 22389 (time 1734260144.07 total 107.55 last 238.79) - open/close 24715 (time 1734260154.07 total 117.55 last 232.55) - open/close 27438 (time 1734260164.07 total 127.56 last 272.24) - open/close 29938 (time 1734260174.08 total 137.56 last 249.87) - open/close 32883 (time 1734260184.08 total 147.56 last 294.41) - open/close 36055 (time 1734260194.08 total 157.56 last 317.20) - open/close 38910 (time 1734260204.08 total 167.56 last 285.49) - open/close 40000 (time 1734260207.37 total 170.85 last 331.66) - open/close 43059 (time 1734260217.37 total 180.85 last 305.87) - open/close 46147 (time 1734260227.37 total 190.85 last 308.77) - open/close 49286 (time 1734260237.37 total 200.85 last 313.88) - open/close 50000 (time 1734260240.18 total 203.67 last 253.58) - open/close 52798 (time 1734260250.19 total 213.67 last 279.70) - open/close 55721 (time 1734260260.19 total 223.68 last 292.21) - open/close 58643 (time 1734260270.19 total 233.68 last 292.15) - open/close 60000 (time 1734260274.62 total 238.11 last 306.07) - open/close 63286 (time 1734260284.63 total 248.11 last 328.53) total: 64768 open/close in 252.70 seconds: 256.31 ops/second - /unlink 5412 (time 1734260300.29 total 10.00 last 541.12) - /unlink 9911 (time 1734260310.29 total 20.00 last 449.85) - /unlink 13989 (time 1734260320.30 total 30.00 last 407.71) - /unlink 18351 (time 1734260330.30 total 40.00 last 436.20) - /unlink 20000 (time 1734260333.99 total 43.70 last 446.30) - /unlink 24302 (time 1734260343.99 total 53.70 last 430.14) - /unlink 29118 (time 1734260353.99 total 63.70 last 481.56) - /unlink 35110 (time 1734260364.00 total 73.70 last 599.11) - /unlink 40000 (time 1734260373.23 total 82.94 last 529.38) - /unlink 44962 (time 1734260383.23 total 92.94 last 496.18) - /unlink 50000 (time 1734260392.76 total 102.47 last 528.76) - /unlink 55571 (time 1734260402.76 total 112.47 last 557.07) - /unlink 60000 (time 1734260411.04 total 120.75 last 535.12) total: 64768 /unlink in 130.48 seconds: 496.37 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2519 (time 1734260446.18 total 10.00 last 251.89) - open/close 5040 (time 1734260456.18 total 20.00 last 252.07) - open/close 7661 (time 1734260466.18 total 30.00 last 262.06) - open/close 10000 (time 1734260476.08 total 39.90 last 236.26) - open/close 12566 (time 1734260486.08 total 49.91 last 256.52) - open/close 14934 (time 1734260496.09 total 59.91 last 236.72) - open/close 17499 (time 1734260506.09 total 69.91 last 256.42) - open/close 20000 (time 1734260514.74 total 78.56 last 289.26) - open/close 23275 (time 1734260524.74 total 88.56 last 327.44) - open/close 26559 (time 1734260534.74 total 98.56 last 328.40) - open/close 29551 (time 1734260544.74 total 108.56 last 299.19) - open/close 32560 (time 1734260554.74 total 118.57 last 300.75) - open/close 35529 (time 1734260564.75 total 128.57 last 296.87) - open/close 39036 (time 1734260574.75 total 138.57 last 350.60) - open/close 40000 (time 1734260577.31 total 141.14 last 375.85) - open/close 43709 (time 1734260587.31 total 151.14 last 370.85) - open/close 46803 (time 1734260597.32 total 161.14 last 309.34) - open/close 50000 (time 1734260606.59 total 170.41 last 344.82) - open/close 53891 (time 1734260616.59 total 180.41 last 389.06) - open/close 57353 (time 1734260626.59 total 190.41 last 346.16) - open/close 60000 (time 1734260633.08 total 196.90 last 408.14) - open/close 64651 (time 1734260643.08 total 206.90 last 465.03) total: 64768 open/close in 207.28 seconds: 312.46 ops/second - /unlink 5075 (time 1734260654.43 total 10.00 last 507.49) - /unlink 10000 (time 1734260663.75 total 19.31 last 528.87) - /unlink 15928 (time 1734260673.75 total 29.31 last 592.75) - /unlink 20000 (time 1734260680.86 total 36.42 last 572.88) - /unlink 26390 (time 1734260690.86 total 46.42 last 638.95) - /unlink 30000 (time 1734260697.03 total 52.60 last 584.69) - /unlink 35935 (time 1734260707.03 total 62.60 last 593.48) - /unlink 40000 (time 1734260713.58 total 69.15 last 620.76) - /unlink 45714 (time 1734260723.58 total 79.15 last 571.39) - /unlink 50000 (time 1734260730.30 total 85.87 last 637.44) - /unlink 56118 (time 1734260740.30 total 95.87 last 611.77) - /unlink 60000 (time 1734260746.36 total 101.93 last 640.59) total: 64768 /unlink in 109.16 seconds: 593.34 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4269 (time 1734260782.33 total 10.00 last 426.79) - open/close 8785 (time 1734260792.34 total 20.00 last 451.52) - open/close 10000 (time 1734260795.04 total 22.71 last 449.25) - open/close 15703 (time 1734260805.04 total 32.71 last 570.26) - open/close 20000 (time 1734260812.34 total 40.01 last 588.50) - open/close 25432 (time 1734260822.34 total 50.01 last 543.20) - open/close 30000 (time 1734260831.04 total 58.71 last 524.90) - open/close 35109 (time 1734260841.05 total 68.71 last 510.88) - open/close 39909 (time 1734260851.05 total 78.71 last 479.97) - open/close 44734 (time 1734260861.05 total 88.72 last 482.24) - open/close 49062 (time 1734260871.05 total 98.72 last 432.76) - open/close 50000 (time 1734260873.18 total 100.85 last 439.93) - open/close 54310 (time 1734260883.18 total 110.85 last 431.00) - open/close 58860 (time 1734260893.18 total 120.85 last 455.00) - open/close 60000 (time 1734260895.45 total 123.12 last 502.52) total: 64768 open/close in 131.74 seconds: 491.65 ops/second - /unlink 9382 (time 1734260914.85 total 10.00 last 938.19) - /unlink 19278 (time 1734260924.85 total 20.00 last 989.48) - /unlink 27972 (time 1734260934.86 total 30.00 last 869.34) - /unlink 30000 (time 1734260937.20 total 32.35 last 863.55) - /unlink 37861 (time 1734260947.20 total 42.35 last 786.04) - /unlink 40000 (time 1734260949.67 total 44.82 last 867.52) - /unlink 49327 (time 1734260959.67 total 54.82 last 932.69) - /unlink 58905 (time 1734260969.67 total 64.82 last 957.72) total: 64768 /unlink in 70.76 seconds: 915.36 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg144-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg144-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg144-server stop mds service on oleg144-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg144-server stop mds service on oleg144-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg144-server unloading modules via unload_modules_local on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg144-client: oleg144-client: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-client: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 2