== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:35:57 (1734258957) Stopping clients: oleg216-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg216-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg216-client: oleg216-server: ssh exited with exit code 2 oleg216-server: oleg216-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing load_modules_local oleg216-server: Loading modules from /home/green/git/lustre-release/lustre oleg216-server: detected 4 online CPUs by sysfs oleg216-server: Force libcfs to create 2 CPU partitions oleg216-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg216-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg216-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg216-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg216-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16131 291 15840 2% /mnt/lustre[OST:0] filesystem_summary: 16483 643 15840 4% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2512 (time 1734259015.99 total 10.00 last 251.18) - open/close 4549 (time 1734259025.99 total 20.00 last 203.66) - open/close 6476 (time 1734259036.00 total 30.01 last 192.65) - open/close 8566 (time 1734259046.00 total 40.01 last 208.95) - open/close 10000 (time 1734259052.86 total 46.87 last 208.83) - open/close 12120 (time 1734259062.87 total 56.88 last 211.95) - open/close 14299 (time 1734259072.87 total 66.88 last 217.85) - open/close 16421 (time 1734259082.87 total 76.88 last 212.09) - open/close 18479 (time 1734259092.88 total 86.89 last 205.76) - open/close 20000 (time 1734259100.21 total 94.22 last 207.27) - open/close 21972 (time 1734259110.22 total 104.23 last 197.07) - open/close 24445 (time 1734259120.22 total 114.23 last 247.28) - open/close 26646 (time 1734259130.22 total 124.23 last 220.04) - open/close 28677 (time 1734259140.22 total 134.23 last 203.09) - open/close 30000 (time 1734259146.39 total 140.40 last 214.44) - open/close 32112 (time 1734259156.39 total 150.40 last 211.20) - open/close 34431 (time 1734259166.40 total 160.41 last 231.83) - open/close 36539 (time 1734259176.40 total 170.41 last 210.78) - open/close 38515 (time 1734259186.40 total 180.41 last 197.55) - open/close 40000 (time 1734259192.28 total 186.29 last 252.77) - open/close 42054 (time 1734259202.28 total 196.29 last 205.29) - open/close 44185 (time 1734259212.28 total 206.29 last 213.06) - open/close 46372 (time 1734259222.29 total 216.30 last 218.64) - open/close 48290 (time 1734259232.29 total 226.30 last 191.74) - open/close 50000 (time 1734259241.15 total 235.16 last 192.94) - open/close 52157 (time 1734259251.15 total 245.16 last 215.66) - open/close 54227 (time 1734259261.16 total 255.17 last 206.90) - open/close 56342 (time 1734259271.16 total 265.17 last 211.48) - open/close 58881 (time 1734259281.16 total 275.17 last 253.89) - open/close 60000 (time 1734259285.48 total 279.49 last 258.96) - open/close 62361 (time 1734259295.48 total 289.49 last 236.01) - open/close 64415 (time 1734259305.49 total 299.50 last 205.32) total: 64768 open/close in 301.19 seconds: 215.04 ops/second - /unlink 3027 (time 1734259318.35 total 10.00 last 302.69) - /unlink 6398 (time 1734259328.35 total 20.00 last 337.08) - /unlink 9921 (time 1734259338.35 total 30.00 last 352.28) - /unlink 14024 (time 1734259348.35 total 40.00 last 410.23) - /unlink 18010 (time 1734259358.36 total 50.01 last 398.53) - /unlink 20000 (time 1734259363.18 total 54.83 last 412.41) - /unlink 23692 (time 1734259373.18 total 64.83 last 369.14) - /unlink 27578 (time 1734259383.19 total 74.83 last 388.51) - /unlink 30000 (time 1734259389.48 total 81.13 last 384.94) - /unlink 33888 (time 1734259399.48 total 91.13 last 388.73) - /unlink 37598 (time 1734259409.48 total 101.13 last 370.92) - /unlink 40000 (time 1734259415.95 total 107.60 last 371.33) - /unlink 43527 (time 1734259425.95 total 117.60 last 352.61) - /unlink 47084 (time 1734259435.96 total 127.60 last 355.58) - /unlink 50000 (time 1734259444.68 total 136.33 last 334.08) - /unlink 53290 (time 1734259454.68 total 146.33 last 329.00) - /unlink 56834 (time 1734259464.69 total 156.33 last 354.39) - /unlink 60000 (time 1734259472.09 total 163.74 last 427.39) - /unlink 63879 (time 1734259482.09 total 173.74 last 387.84) total: 64768 /unlink in 176.17 seconds: 367.65 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2152 (time 1734259510.43 total 10.00 last 215.17) - open/close 4316 (time 1734259520.43 total 20.00 last 216.37) - open/close 6361 (time 1734259530.44 total 30.01 last 204.36) - open/close 8463 (time 1734259540.44 total 40.01 last 210.11) - open/close 10000 (time 1734259548.53 total 48.10 last 190.09) - open/close 12045 (time 1734259558.53 total 58.10 last 204.50) - open/close 14211 (time 1734259568.53 total 68.10 last 216.57) - open/close 16284 (time 1734259578.53 total 78.10 last 207.23) - open/close 18660 (time 1734259588.54 total 88.11 last 237.58) - open/close 20000 (time 1734259594.34 total 93.91 last 230.66) - open/close 22442 (time 1734259604.35 total 103.92 last 244.14) - open/close 24695 (time 1734259614.35 total 113.92 last 225.17) - open/close 27191 (time 1734259624.35 total 123.92 last 249.60) - open/close 30000 (time 1734259634.33 total 133.90 last 281.47) - open/close 32792 (time 1734259644.33 total 143.90 last 279.16) - open/close 35582 (time 1734259654.34 total 153.91 last 278.94) - open/close 37891 (time 1734259664.34 total 163.91 last 230.89) - open/close 40000 (time 1734259673.04 total 172.61 last 242.39) - open/close 42509 (time 1734259683.04 total 182.61 last 250.82) - open/close 45085 (time 1734259693.04 total 192.61 last 257.57) - open/close 48035 (time 1734259703.04 total 202.61 last 294.95) - open/close 50000 (time 1734259710.38 total 209.95 last 267.72) - open/close 52352 (time 1734259720.38 total 219.95 last 235.18) - open/close 54955 (time 1734259730.38 total 229.95 last 260.29) - open/close 57317 (time 1734259740.39 total 239.96 last 236.16) - open/close 59690 (time 1734259750.39 total 249.96 last 237.28) - open/close 62139 (time 1734259760.39 total 259.96 last 244.89) - open/close 64202 (time 1734259770.39 total 269.96 last 206.25) total: 64768 open/close in 272.77 seconds: 237.44 ops/second - /unlink 3738 (time 1734259784.34 total 10.00 last 373.77) - /unlink 7754 (time 1734259794.34 total 20.00 last 401.57) - /unlink 10000 (time 1734259799.87 total 25.53 last 405.97) - /unlink 15122 (time 1734259809.87 total 35.54 last 512.11) - /unlink 19831 (time 1734259819.87 total 45.54 last 470.79) - /unlink 24634 (time 1734259829.88 total 55.54 last 480.23) - /unlink 29211 (time 1734259839.88 total 65.54 last 457.60) - /unlink 33903 (time 1734259849.88 total 75.54 last 469.20) - /unlink 37634 (time 1734259859.88 total 85.54 last 373.08) - /unlink 40000 (time 1734259865.12 total 90.79 last 451.29) - /unlink 44157 (time 1734259875.12 total 100.79 last 415.63) - /unlink 48228 (time 1734259885.12 total 110.79 last 407.06) - /unlink 50000 (time 1734259888.92 total 114.58 last 467.18) - /unlink 54421 (time 1734259898.92 total 124.58 last 442.07) - /unlink 59117 (time 1734259908.92 total 134.58 last 469.58) - /unlink 63548 (time 1734259918.92 total 144.58 last 443.04) total: 64768 /unlink in 147.20 seconds: 440.00 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2744 (time 1734259949.75 total 10.00 last 274.35) - open/close 5548 (time 1734259959.75 total 20.00 last 280.37) - open/close 8523 (time 1734259969.75 total 30.00 last 297.48) - open/close 10000 (time 1734259974.43 total 34.68 last 316.14) - open/close 13358 (time 1734259984.43 total 44.68 last 335.78) - open/close 16314 (time 1734259994.43 total 54.68 last 295.51) - open/close 19268 (time 1734260004.43 total 64.68 last 295.34) - open/close 20000 (time 1734260006.73 total 66.98 last 318.72) - open/close 22845 (time 1734260016.73 total 76.98 last 284.40) - open/close 25769 (time 1734260026.73 total 86.98 last 292.33) - open/close 28805 (time 1734260036.73 total 96.98 last 303.59) - open/close 30000 (time 1734260040.99 total 101.24 last 280.80) - open/close 32369 (time 1734260050.99 total 111.24 last 236.83) - open/close 35220 (time 1734260060.99 total 121.24 last 285.09) - open/close 38394 (time 1734260070.99 total 131.24 last 317.38) - open/close 40000 (time 1734260076.52 total 136.77 last 290.65) - open/close 42600 (time 1734260086.52 total 146.77 last 259.98) - open/close 45238 (time 1734260096.52 total 156.77 last 263.80) - open/close 47803 (time 1734260106.52 total 166.77 last 256.47) - open/close 50000 (time 1734260114.08 total 174.33 last 290.66) - open/close 52619 (time 1734260124.08 total 184.33 last 261.84) - open/close 55229 (time 1734260134.08 total 194.33 last 260.99) - open/close 57909 (time 1734260144.08 total 204.33 last 267.98) - open/close 60000 (time 1734260151.49 total 211.74 last 282.48) - open/close 62927 (time 1734260161.49 total 221.74 last 292.64) total: 64768 open/close in 228.50 seconds: 283.45 ops/second - /unlink 4662 (time 1734260179.22 total 10.00 last 466.19) - /unlink 8667 (time 1734260189.22 total 20.00 last 400.48) - /unlink 10000 (time 1734260192.53 total 23.31 last 403.29) - /unlink 14552 (time 1734260202.53 total 33.31 last 455.14) - /unlink 19125 (time 1734260212.53 total 43.31 last 457.26) - /unlink 20000 (time 1734260214.70 total 45.48 last 403.75) - /unlink 24445 (time 1734260224.70 total 55.48 last 444.49) - /unlink 28282 (time 1734260234.70 total 65.48 last 383.68) - /unlink 30000 (time 1734260238.20 total 68.98 last 489.89) - /unlink 34627 (time 1734260248.21 total 78.98 last 462.68) - /unlink 39145 (time 1734260258.21 total 88.99 last 451.73) - /unlink 43893 (time 1734260268.21 total 98.99 last 474.80) - /unlink 49044 (time 1734260278.21 total 108.99 last 514.99) - /unlink 54013 (time 1734260288.21 total 118.99 last 496.82) - /unlink 59060 (time 1734260298.21 total 128.99 last 504.59) - /unlink 63384 (time 1734260308.21 total 138.99 last 432.40) total: 64768 /unlink in 142.37 seconds: 454.92 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2414 (time 1734260341.46 total 10.00 last 241.39) - open/close 4775 (time 1734260351.46 total 20.00 last 236.00) - open/close 7220 (time 1734260361.46 total 30.01 last 244.43) - open/close 9854 (time 1734260371.47 total 40.01 last 263.33) - open/close 12294 (time 1734260381.47 total 50.01 last 243.96) - open/close 14767 (time 1734260391.47 total 60.01 last 247.26) - open/close 17753 (time 1734260401.47 total 70.02 last 298.51) - open/close 20000 (time 1734260409.47 total 78.02 last 280.86) - open/close 22669 (time 1734260419.47 total 88.02 last 266.89) - open/close 25431 (time 1734260429.47 total 98.02 last 276.18) - open/close 27864 (time 1734260439.48 total 108.02 last 243.24) - open/close 30000 (time 1734260447.12 total 115.66 last 279.61) - open/close 33466 (time 1734260457.12 total 125.66 last 346.58) - open/close 36545 (time 1734260467.12 total 135.66 last 307.88) - open/close 39303 (time 1734260477.12 total 145.67 last 275.71) - open/close 40000 (time 1734260479.92 total 148.47 last 248.50) - open/close 42469 (time 1734260489.93 total 158.47 last 246.89) - open/close 45098 (time 1734260499.93 total 168.47 last 262.84) - open/close 48044 (time 1734260509.93 total 178.47 last 294.54) - open/close 50000 (time 1734260517.34 total 185.88 last 264.10) - open/close 52692 (time 1734260527.34 total 195.88 last 269.13) - open/close 55465 (time 1734260537.34 total 205.88 last 277.29) - open/close 58079 (time 1734260547.34 total 215.89 last 261.37) - open/close 60000 (time 1734260554.94 total 223.49 last 252.61) - open/close 62010 (time 1734260564.95 total 233.49 last 200.99) - open/close 64273 (time 1734260574.95 total 243.49 last 226.25) total: 64768 open/close in 245.65 seconds: 263.66 ops/second - /unlink 3154 (time 1734260588.15 total 10.00 last 315.38) - /unlink 7597 (time 1734260598.15 total 20.00 last 444.22) - /unlink 10000 (time 1734260603.59 total 25.44 last 441.77) - /unlink 13282 (time 1734260613.60 total 35.44 last 328.10) - /unlink 17455 (time 1734260623.60 total 45.45 last 417.27) - /unlink 20000 (time 1734260630.09 total 51.94 last 391.78) - /unlink 24508 (time 1734260640.09 total 61.94 last 450.75) - /unlink 28511 (time 1734260650.09 total 71.94 last 400.24) - /unlink 30000 (time 1734260653.81 total 75.66 last 401.25) - /unlink 34953 (time 1734260663.81 total 85.66 last 495.20) - /unlink 39211 (time 1734260673.81 total 95.66 last 425.76) - /unlink 44269 (time 1734260683.81 total 105.66 last 505.77) - /unlink 47972 (time 1734260693.81 total 115.66 last 370.26) - /unlink 50000 (time 1734260698.42 total 120.27 last 439.54) - /unlink 53934 (time 1734260708.43 total 130.28 last 393.32) - /unlink 58612 (time 1734260718.43 total 140.28 last 467.79) - /unlink 60000 (time 1734260721.20 total 143.05 last 500.39) - /unlink 64202 (time 1734260731.20 total 153.05 last 420.16) total: 64768 /unlink in 154.48 seconds: 419.27 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2587 (time 1734260761.28 total 10.01 last 258.57) - open/close 5098 (time 1734260771.28 total 20.01 last 251.08) - open/close 8168 (time 1734260781.28 total 30.01 last 306.97) - open/close 10000 (time 1734260787.73 total 36.45 last 284.13) - open/close 13059 (time 1734260797.73 total 46.46 last 305.86) - open/close 16229 (time 1734260807.73 total 56.46 last 316.91) - open/close 19561 (time 1734260817.73 total 66.46 last 333.19) - open/close 22581 (time 1734260827.73 total 76.46 last 302.00) - open/close 25637 (time 1734260837.73 total 86.46 last 305.58) - open/close 28749 (time 1734260847.74 total 96.46 last 311.15) - open/close 30000 (time 1734260851.90 total 100.62 last 300.53) - open/close 33225 (time 1734260861.90 total 110.63 last 322.41) - open/close 36045 (time 1734260871.90 total 120.63 last 282.00) - open/close 39428 (time 1734260881.90 total 130.63 last 338.23) - open/close 43421 (time 1734260891.90 total 140.63 last 399.25) - open/close 47182 (time 1734260901.91 total 150.63 last 376.03) - open/close 50000 (time 1734260910.23 total 158.95 last 338.75) - open/close 53106 (time 1734260920.23 total 168.95 last 310.54) - open/close 56018 (time 1734260930.23 total 178.96 last 291.12) - open/close 59111 (time 1734260940.23 total 188.96 last 309.21) - open/close 60000 (time 1734260942.94 total 191.66 last 328.89) - open/close 62933 (time 1734260952.94 total 201.66 last 293.19) total: 64768 open/close in 207.58 seconds: 312.01 ops/second - /unlink 4850 (time 1734260969.86 total 10.00 last 484.96) - /unlink 10000 (time 1734260978.47 total 18.61 last 598.49) - /unlink 16577 (time 1734260988.47 total 28.61 last 657.61) - /unlink 20000 (time 1734260994.23 total 34.37 last 594.29) - /unlink 24604 (time 1734261004.23 total 44.37 last 460.30) - /unlink 30000 (time 1734261012.72 total 52.86 last 635.41) - /unlink 36362 (time 1734261022.72 total 62.86 last 636.15) - /unlink 40000 (time 1734261028.66 total 68.80 last 612.41) - /unlink 46071 (time 1734261038.66 total 78.80 last 607.06) - /unlink 50000 (time 1734261044.41 total 84.55 last 684.05) - /unlink 57087 (time 1734261054.41 total 94.55 last 708.69) - /unlink 60000 (time 1734261057.82 total 97.96 last 853.23) total: 64768 /unlink in 103.48 seconds: 625.92 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg216-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg216-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg216-server stop mds service on oleg216-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg216-server stop mds service on oleg216-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg216-server unloading modules via unload_modules_local on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg216-client: oleg216-client: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-client: ssh exited with exit code 2