== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:32:59 (1734258779) Stopping clients: oleg105-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg105-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg105-client: oleg105-server: ssh exited with exit code 2 oleg105-server: oleg105-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg105-server' oleg105-server: oleg105-server.virtnet: executing load_modules_local oleg105-server: Loading modules from /home/green/git/lustre-release/lustre oleg105-server: detected 4 online CPUs by sysfs oleg105-server: Force libcfs to create 2 CPU partitions oleg105-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg105-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg105-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg105-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg105-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg105-client.virtnet: -o user_xattr,flock 192.168.201.105@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16266 458 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16451 643 15808 4% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1725 (time 1734258852.90 total 10.00 last 172.46) - open/close 3361 (time 1734258862.90 total 20.01 last 163.53) - open/close 5127 (time 1734258872.90 total 30.01 last 176.58) - open/close 6808 (time 1734258882.91 total 40.01 last 168.02) - open/close 8488 (time 1734258892.91 total 50.01 last 167.99) - open/close 10000 (time 1734258901.74 total 58.84 last 171.23) - open/close 11637 (time 1734258911.74 total 68.85 last 163.65) - open/close 13335 (time 1734258921.75 total 78.85 last 169.76) - open/close 15240 (time 1734258931.75 total 88.85 last 190.47) - open/close 17255 (time 1734258941.75 total 98.85 last 201.48) - open/close 19218 (time 1734258951.75 total 108.85 last 196.23) - open/close 20000 (time 1734258955.82 total 112.92 last 192.30) - open/close 21664 (time 1734258965.82 total 122.93 last 166.34) - open/close 23124 (time 1734258975.83 total 132.93 last 145.90) - open/close 24659 (time 1734258985.83 total 142.94 last 153.41) - open/close 26175 (time 1734258995.84 total 152.94 last 151.59) - open/close 27786 (time 1734259005.84 total 162.94 last 161.09) - open/close 29458 (time 1734259015.84 total 172.94 last 167.13) - open/close 30000 (time 1734259019.22 total 176.32 last 160.30) - open/close 31726 (time 1734259029.22 total 186.33 last 172.58) - open/close 33246 (time 1734259039.22 total 196.33 last 151.99) - open/close 34825 (time 1734259049.23 total 206.33 last 157.82) - open/close 36477 (time 1734259059.23 total 216.34 last 165.14) - open/close 37961 (time 1734259069.23 total 226.34 last 148.36) - open/close 39476 (time 1734259079.24 total 236.34 last 151.41) - open/close 40000 (time 1734259082.36 total 239.46 last 168.23) - open/close 41721 (time 1734259092.36 total 249.46 last 172.02) - open/close 43216 (time 1734259102.36 total 259.46 last 149.48) - open/close 44802 (time 1734259112.37 total 269.47 last 158.47) - open/close 46294 (time 1734259122.37 total 279.48 last 149.16) - open/close 47734 (time 1734259132.37 total 289.48 last 143.99) - open/close 49253 (time 1734259142.38 total 299.48 last 151.79) - open/close 50000 (time 1734259147.34 total 304.44 last 150.63) - open/close 51405 (time 1734259157.35 total 314.45 last 140.40) - open/close 52829 (time 1734259167.35 total 324.46 last 142.31) - open/close 54298 (time 1734259177.35 total 334.46 last 146.88) - open/close 55796 (time 1734259187.36 total 344.46 last 149.75) - open/close 57234 (time 1734259197.36 total 354.46 last 143.75) - open/close 58648 (time 1734259207.36 total 364.47 last 141.36) - open/close 60000 (time 1734259216.52 total 373.62 last 147.69) - open/close 61428 (time 1734259226.52 total 383.62 last 142.76) - open/close 62902 (time 1734259236.52 total 393.62 last 147.39) - open/close 64357 (time 1734259246.52 total 403.63 last 145.48) total: 64768 open/close in 406.49 seconds: 159.33 ops/second - /unlink 2727 (time 1734259260.77 total 10.00 last 272.69) - /unlink 5297 (time 1734259270.78 total 20.00 last 256.94) - /unlink 8337 (time 1734259280.78 total 30.01 last 303.93) - /unlink 10000 (time 1734259286.26 total 35.49 last 303.24) - /unlink 12713 (time 1734259296.27 total 45.49 last 271.14) - /unlink 15552 (time 1734259306.27 total 55.50 last 283.78) - /unlink 18221 (time 1734259316.27 total 65.50 last 266.84) - /unlink 20000 (time 1734259323.01 total 72.24 last 264.01) - /unlink 22566 (time 1734259333.02 total 82.24 last 256.51) - /unlink 24866 (time 1734259343.02 total 92.25 last 229.89) - /unlink 27672 (time 1734259353.02 total 102.25 last 280.58) - /unlink 30000 (time 1734259361.90 total 111.13 last 262.13) - /unlink 32525 (time 1734259371.91 total 121.13 last 252.41) - /unlink 35476 (time 1734259381.91 total 131.13 last 295.09) - /unlink 38531 (time 1734259391.91 total 141.13 last 305.49) - /unlink 40000 (time 1734259396.97 total 146.19 last 290.40) - /unlink 42937 (time 1734259406.97 total 156.19 last 293.66) - /unlink 45944 (time 1734259416.97 total 166.20 last 300.50) - /unlink 49197 (time 1734259426.98 total 176.20 last 325.23) - /unlink 50000 (time 1734259429.59 total 178.81 last 307.61) - /unlink 52985 (time 1734259439.59 total 188.82 last 298.43) - /unlink 55787 (time 1734259449.59 total 198.82 last 280.16) - /unlink 58940 (time 1734259459.59 total 208.82 last 315.30) - /unlink 60000 (time 1734259462.46 total 211.69 last 368.94) - /unlink 63698 (time 1734259472.46 total 221.69 last 369.76) total: 64768 /unlink in 224.82 seconds: 288.09 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2148 (time 1734259501.41 total 10.00 last 214.77) - open/close 4325 (time 1734259511.41 total 20.00 last 217.68) - open/close 6339 (time 1734259521.42 total 30.01 last 201.33) - open/close 8429 (time 1734259531.42 total 40.01 last 208.98) - open/close 10000 (time 1734259540.20 total 48.78 last 178.98) - open/close 11630 (time 1734259550.20 total 58.79 last 162.98) - open/close 13181 (time 1734259560.20 total 68.79 last 155.07) - open/close 14846 (time 1734259570.20 total 78.79 last 166.43) - open/close 16611 (time 1734259580.20 total 88.79 last 176.50) - open/close 18189 (time 1734259590.20 total 98.79 last 157.78) - open/close 19893 (time 1734259600.21 total 108.80 last 170.34) - open/close 21743 (time 1734259610.21 total 118.80 last 184.97) - open/close 23608 (time 1734259620.21 total 128.80 last 186.45) - open/close 25335 (time 1734259630.21 total 138.80 last 172.70) - open/close 27101 (time 1734259640.21 total 148.80 last 176.58) - open/close 28878 (time 1734259650.22 total 158.81 last 177.67) - open/close 30000 (time 1734259656.23 total 164.82 last 186.44) - open/close 31646 (time 1734259666.24 total 174.83 last 164.48) - open/close 33451 (time 1734259676.24 total 184.83 last 180.50) - open/close 35282 (time 1734259686.24 total 194.83 last 183.08) - open/close 37124 (time 1734259696.25 total 204.84 last 184.11) - open/close 38880 (time 1734259706.25 total 214.84 last 175.59) - open/close 40000 (time 1734259712.39 total 220.98 last 182.42) - open/close 41621 (time 1734259722.39 total 230.98 last 162.05) - open/close 43273 (time 1734259732.39 total 240.98 last 165.16) - open/close 45044 (time 1734259742.39 total 250.98 last 177.06) - open/close 46891 (time 1734259752.40 total 260.99 last 184.67) - open/close 48616 (time 1734259762.40 total 270.99 last 172.44) - open/close 50000 (time 1734259770.82 total 279.41 last 164.36) - open/close 52058 (time 1734259780.82 total 289.41 last 205.77) - open/close 54092 (time 1734259790.82 total 299.41 last 203.37) - open/close 56044 (time 1734259800.83 total 309.42 last 195.10) - open/close 57977 (time 1734259810.83 total 319.42 last 193.21) - open/close 59933 (time 1734259820.84 total 329.43 last 195.50) - open/close 61744 (time 1734259830.84 total 339.43 last 180.98) - open/close 63468 (time 1734259840.85 total 349.44 last 172.32) total: 64768 open/close in 357.26 seconds: 181.29 ops/second - /unlink 3335 (time 1734259859.98 total 10.00 last 333.46) - /unlink 6186 (time 1734259869.98 total 20.00 last 285.04) - /unlink 9012 (time 1734259879.99 total 30.01 last 282.52) - /unlink 10000 (time 1734259883.05 total 33.07 last 322.44) - /unlink 13277 (time 1734259893.05 total 43.07 last 327.67) - /unlink 16414 (time 1734259903.05 total 53.07 last 313.63) - /unlink 19926 (time 1734259913.05 total 63.07 last 351.16) - /unlink 23636 (time 1734259923.06 total 73.08 last 370.91) - /unlink 27166 (time 1734259933.06 total 83.08 last 352.94) - /unlink 30000 (time 1734259940.98 total 91.00 last 357.84) - /unlink 33615 (time 1734259950.98 total 101.00 last 361.46) - /unlink 37432 (time 1734259960.98 total 111.00 last 381.63) - /unlink 40000 (time 1734259967.81 total 117.83 last 376.04) - /unlink 43615 (time 1734259977.81 total 127.83 last 361.38) - /unlink 47236 (time 1734259987.82 total 137.84 last 362.01) - /unlink 50000 (time 1734259995.19 total 145.21 last 374.66) - /unlink 53901 (time 1734260005.20 total 155.22 last 390.00) - /unlink 57649 (time 1734260015.20 total 165.22 last 374.69) - /unlink 60000 (time 1734260021.51 total 171.53 last 372.44) - /unlink 63870 (time 1734260031.51 total 181.53 last 386.89) total: 64768 /unlink in 183.95 seconds: 352.10 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2028 (time 1734260062.50 total 10.00 last 202.72) - open/close 3845 (time 1734260072.50 total 20.00 last 181.70) - open/close 5719 (time 1734260082.51 total 30.01 last 187.32) - open/close 7762 (time 1734260092.51 total 40.01 last 204.20) - open/close 9736 (time 1734260102.51 total 50.01 last 197.38) - open/close 11778 (time 1734260112.51 total 60.02 last 204.17) - open/close 13845 (time 1734260122.52 total 70.02 last 206.66) - open/close 16087 (time 1734260132.52 total 80.02 last 224.13) - open/close 18422 (time 1734260142.52 total 90.02 last 233.49) - open/close 20000 (time 1734260149.79 total 97.29 last 217.02) - open/close 22605 (time 1734260159.79 total 107.29 last 260.44) - open/close 25014 (time 1734260169.79 total 117.29 last 240.90) - open/close 27604 (time 1734260179.79 total 127.30 last 258.97) - open/close 30000 (time 1734260188.17 total 135.67 last 286.15) - open/close 32828 (time 1734260198.17 total 145.67 last 282.78) - open/close 35780 (time 1734260208.17 total 155.67 last 295.15) - open/close 38906 (time 1734260218.17 total 165.67 last 312.52) - open/close 40000 (time 1734260221.85 total 169.35 last 297.79) - open/close 43346 (time 1734260231.85 total 179.35 last 334.56) - open/close 46383 (time 1734260241.85 total 189.35 last 303.68) - open/close 49060 (time 1734260251.85 total 199.35 last 267.70) - open/close 50000 (time 1734260255.21 total 202.71 last 279.47) - open/close 52923 (time 1734260265.21 total 212.72 last 292.21) - open/close 55866 (time 1734260275.22 total 222.72 last 294.27) - open/close 59079 (time 1734260285.22 total 232.72 last 321.27) - open/close 60000 (time 1734260288.09 total 235.59 last 320.59) - open/close 62985 (time 1734260298.09 total 245.59 last 298.43) total: 64768 open/close in 252.60 seconds: 256.41 ops/second - /unlink 4180 (time 1734260316.17 total 10.00 last 417.96) - /unlink 8204 (time 1734260326.17 total 20.00 last 402.32) - /unlink 10000 (time 1734260330.13 total 23.96 last 454.18) - /unlink 13946 (time 1734260340.13 total 33.96 last 394.56) - /unlink 18199 (time 1734260350.13 total 43.96 last 425.28) - /unlink 20000 (time 1734260354.27 total 48.10 last 434.69) - /unlink 25477 (time 1734260364.27 total 58.10 last 547.68) - /unlink 30000 (time 1734260373.78 total 67.61 last 475.75) - /unlink 34864 (time 1734260383.78 total 77.61 last 486.37) - /unlink 39866 (time 1734260393.78 total 87.61 last 500.16) - /unlink 45097 (time 1734260403.78 total 97.61 last 523.04) - /unlink 50000 (time 1734260413.70 total 107.53 last 494.27) - /unlink 54326 (time 1734260423.70 total 117.53 last 432.59) - /unlink 58933 (time 1734260433.70 total 127.53 last 460.64) - /unlink 60000 (time 1734260436.26 total 130.09 last 417.64) - /unlink 64178 (time 1734260446.26 total 140.09 last 417.79) total: 64768 /unlink in 141.51 seconds: 457.69 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2172 (time 1734260477.89 total 10.00 last 217.13) - open/close 4516 (time 1734260487.90 total 20.01 last 234.33) - open/close 6712 (time 1734260497.90 total 30.01 last 219.52) - open/close 9149 (time 1734260507.90 total 40.01 last 243.69) - open/close 10000 (time 1734260511.25 total 43.36 last 253.87) - open/close 12701 (time 1734260521.25 total 53.36 last 270.08) - open/close 15729 (time 1734260531.26 total 63.36 last 302.77) - open/close 18507 (time 1734260541.26 total 73.36 last 277.79) - open/close 20000 (time 1734260546.75 total 78.86 last 271.49) - open/close 22719 (time 1734260556.76 total 88.87 last 271.83) - open/close 25625 (time 1734260566.76 total 98.87 last 290.58) - open/close 28784 (time 1734260576.76 total 108.87 last 315.89) - open/close 30000 (time 1734260580.31 total 112.42 last 342.15) - open/close 33130 (time 1734260590.31 total 122.42 last 312.98) - open/close 35822 (time 1734260600.31 total 132.42 last 269.20) - open/close 39137 (time 1734260610.32 total 142.42 last 331.43) - open/close 40000 (time 1734260612.50 total 144.61 last 395.23) - open/close 43412 (time 1734260622.50 total 154.61 last 341.14) - open/close 47055 (time 1734260632.50 total 164.61 last 364.24) - open/close 50000 (time 1734260639.01 total 171.12 last 452.77) - open/close 52866 (time 1734260649.01 total 181.12 last 286.57) - open/close 55600 (time 1734260659.01 total 191.12 last 273.40) - open/close 58271 (time 1734260669.01 total 201.12 last 267.04) - open/close 60000 (time 1734260675.53 total 207.64 last 265.23) - open/close 62926 (time 1734260685.53 total 217.64 last 292.53) total: 64768 open/close in 223.45 seconds: 289.86 ops/second - /unlink 4921 (time 1734260702.32 total 10.00 last 492.06) - /unlink 10000 (time 1734260711.84 total 19.53 last 533.18) - /unlink 15172 (time 1734260721.84 total 29.53 last 517.19) - /unlink 20000 (time 1734260730.14 total 37.83 last 581.76) - /unlink 25609 (time 1734260740.14 total 47.83 last 560.89) - /unlink 30000 (time 1734260747.43 total 55.12 last 602.40) - /unlink 36726 (time 1734260757.43 total 65.12 last 672.50) - /unlink 40000 (time 1734260761.97 total 69.65 last 721.42) - /unlink 47762 (time 1734260771.97 total 79.66 last 776.18) - /unlink 50000 (time 1734260774.85 total 82.54 last 777.00) - /unlink 57553 (time 1734260784.85 total 92.54 last 755.22) - /unlink 60000 (time 1734260788.19 total 95.87 last 733.77) total: 64768 /unlink in 101.87 seconds: 635.80 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5444 (time 1734260823.31 total 10.00 last 544.31) - open/close 10000 (time 1734260832.33 total 19.02 last 505.28) - open/close 15238 (time 1734260842.33 total 29.02 last 523.74) - open/close 19996 (time 1734260852.33 total 39.02 last 475.73) - open/close 24453 (time 1734260862.33 total 49.02 last 445.63) - open/close 28712 (time 1734260872.33 total 59.02 last 425.85) - open/close 30000 (time 1734260875.89 total 62.59 last 361.36) - open/close 34311 (time 1734260885.90 total 72.59 last 431.04) - open/close 39006 (time 1734260895.90 total 82.59 last 469.43) - open/close 44692 (time 1734260905.90 total 92.59 last 568.53) - open/close 50000 (time 1734260915.66 total 102.35 last 543.75) - open/close 55605 (time 1734260925.66 total 112.35 last 560.49) - open/close 60000 (time 1734260934.45 total 121.14 last 500.23) total: 64768 open/close in 130.78 seconds: 495.25 ops/second - /unlink 8376 (time 1734260954.93 total 10.00 last 837.56) - /unlink 17791 (time 1734260964.93 total 20.00 last 941.45) - /unlink 20000 (time 1734260967.46 total 22.54 last 871.56) - /unlink 29513 (time 1734260977.46 total 32.54 last 951.21) - /unlink 38427 (time 1734260987.46 total 42.54 last 891.36) - /unlink 47001 (time 1734260997.46 total 52.54 last 857.40) - /unlink 50000 (time 1734261000.89 total 55.96 last 875.40) - /unlink 58535 (time 1734261010.89 total 65.96 last 853.44) total: 64768 /unlink in 73.08 seconds: 886.23 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg105-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg105-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg105-server stop mds service on oleg105-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg105-server stop mds service on oleg105-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg105-server unloading modules via unload_modules_local on: 'oleg105-server' oleg105-server: oleg105-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg105-client: oleg105-client: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-server: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-client: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-server: ssh exited with exit code 2