== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 06:53:50 (1734263630) Stopping clients: oleg210-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg210-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 oleg210-server: oleg210-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing load_modules_local oleg210-server: Loading modules from /home/green/git/lustre-release/lustre oleg210-server: detected 4 online CPUs by sysfs oleg210-server: Force libcfs to create 2 CPU partitions oleg210-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg210-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg210-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2810 (time 1734263679.15 total 10.00 last 280.92) - open/close 5380 (time 1734263689.15 total 20.01 last 256.93) - open/close 7865 (time 1734263699.15 total 30.01 last 248.40) - open/close 10000 (time 1734263707.49 total 38.35 last 256.00) - open/close 12514 (time 1734263717.50 total 48.35 last 251.32) - open/close 15018 (time 1734263727.50 total 58.35 last 250.37) - open/close 17546 (time 1734263737.50 total 68.35 last 252.79) - open/close 20000 (time 1734263746.41 total 77.27 last 275.40) - open/close 24263 (time 1734263756.41 total 87.27 last 426.23) - open/close 27337 (time 1734263766.41 total 97.27 last 307.33) - open/close 29841 (time 1734263776.41 total 107.27 last 250.36) - open/close 32416 (time 1734263786.42 total 117.27 last 257.45) - open/close 35000 (time 1734263796.42 total 127.27 last 258.36) - open/close 37464 (time 1734263806.42 total 137.28 last 246.38) - open/close 39970 (time 1734263816.42 total 147.28 last 250.60) - open/close 42495 (time 1734263826.42 total 157.28 last 252.47) - open/close 44918 (time 1734263836.46 total 167.31 last 241.39) - open/close 47375 (time 1734263846.46 total 177.32 last 245.60) - open/close 50000 (time 1734263856.41 total 187.27 last 263.72) - open/close 52645 (time 1734263866.41 total 197.27 last 264.49) - open/close 55205 (time 1734263876.42 total 207.27 last 255.99) - open/close 57781 (time 1734263886.42 total 217.27 last 257.57) - open/close 60000 (time 1734263895.45 total 226.31 last 245.67) - open/close 62491 (time 1734263905.45 total 236.31 last 249.07) total: 64768 open/close in 245.88 seconds: 263.42 ops/second - /unlink 4567 (time 1734263926.04 total 10.00 last 456.61) - /unlink 9482 (time 1734263936.05 total 20.00 last 491.45) - /unlink 14221 (time 1734263946.05 total 30.00 last 473.82) - /unlink 18730 (time 1734263956.05 total 40.00 last 450.90) - /unlink 20000 (time 1734263958.87 total 42.83 last 449.92) - /unlink 25450 (time 1734263968.87 total 52.83 last 544.88) - /unlink 30000 (time 1734263977.82 total 61.78 last 508.27) - /unlink 34558 (time 1734263987.83 total 71.78 last 455.70) - /unlink 39193 (time 1734263997.83 total 81.79 last 463.42) - /unlink 43730 (time 1734264007.83 total 91.79 last 453.62) - /unlink 48480 (time 1734264017.83 total 101.79 last 474.95) - /unlink 50000 (time 1734264021.00 total 104.96 last 479.09) - /unlink 54808 (time 1734264031.01 total 114.96 last 480.71) - /unlink 60000 (time 1734264040.84 total 124.80 last 527.80) total: 64768 /unlink in 133.83 seconds: 483.94 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2801 (time 1734264076.84 total 10.00 last 280.06) - open/close 5403 (time 1734264086.84 total 20.00 last 260.18) - open/close 7975 (time 1734264096.84 total 30.00 last 257.20) - open/close 10000 (time 1734264104.98 total 38.14 last 248.93) - open/close 12547 (time 1734264114.98 total 48.14 last 254.67) - open/close 15021 (time 1734264124.98 total 58.14 last 247.35) - open/close 17493 (time 1734264134.98 total 68.14 last 247.16) - open/close 19978 (time 1734264144.98 total 78.14 last 248.49) - open/close 22939 (time 1734264154.99 total 88.14 last 296.04) - open/close 25761 (time 1734264164.99 total 98.14 last 282.20) - open/close 28217 (time 1734264174.99 total 108.14 last 245.60) - open/close 30000 (time 1734264182.19 total 115.35 last 247.51) - open/close 32430 (time 1734264192.19 total 125.35 last 242.95) - open/close 34926 (time 1734264202.19 total 135.35 last 249.58) - open/close 37492 (time 1734264212.20 total 145.35 last 256.51) - open/close 39981 (time 1734264222.20 total 155.36 last 248.88) - open/close 42430 (time 1734264232.20 total 165.36 last 244.89) - open/close 44979 (time 1734264242.20 total 175.36 last 254.89) - open/close 47398 (time 1734264252.20 total 185.36 last 241.87) - open/close 49778 (time 1734264262.27 total 195.43 last 236.38) - open/close 52209 (time 1734264272.27 total 205.43 last 243.00) - open/close 56371 (time 1734264282.27 total 215.43 last 416.14) - open/close 58881 (time 1734264292.27 total 225.43 last 251.00) - open/close 60000 (time 1734264296.89 total 230.05 last 242.18) - open/close 62556 (time 1734264306.89 total 240.05 last 255.60) total: 64768 open/close in 249.33 seconds: 259.77 ops/second - /unlink 4460 (time 1734264327.39 total 10.00 last 445.98) - /unlink 8990 (time 1734264337.39 total 20.00 last 452.95) - /unlink 10000 (time 1734264339.79 total 22.39 last 422.09) - /unlink 14426 (time 1734264349.79 total 32.39 last 442.59) - /unlink 18973 (time 1734264359.79 total 42.40 last 454.59) - /unlink 20000 (time 1734264361.94 total 44.54 last 478.18) - /unlink 24455 (time 1734264371.94 total 54.54 last 445.50) - /unlink 28942 (time 1734264381.94 total 64.55 last 448.67) - /unlink 30000 (time 1734264384.35 total 66.96 last 438.64) - /unlink 34754 (time 1734264394.35 total 76.96 last 475.30) - /unlink 39406 (time 1734264404.35 total 86.96 last 465.13) - /unlink 43975 (time 1734264414.35 total 96.96 last 456.84) - /unlink 48816 (time 1734264424.36 total 106.96 last 484.02) - /unlink 50000 (time 1734264426.52 total 109.12 last 548.21) - /unlink 56223 (time 1734264436.52 total 119.12 last 622.28) - /unlink 60000 (time 1734264442.46 total 125.07 last 634.99) total: 64768 /unlink in 132.32 seconds: 489.49 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3726 (time 1734264477.85 total 10.00 last 372.54) - open/close 7467 (time 1734264487.86 total 20.00 last 374.02) - open/close 10000 (time 1734264495.07 total 27.22 last 351.19) - open/close 13549 (time 1734264505.07 total 37.22 last 354.80) - open/close 17328 (time 1734264515.07 total 47.22 last 377.84) - open/close 19895 (time 1734264525.07 total 57.22 last 256.70) - open/close 22411 (time 1734264535.08 total 67.22 last 251.58) - open/close 24921 (time 1734264545.08 total 77.22 last 250.99) - open/close 27463 (time 1734264555.08 total 87.22 last 254.20) - open/close 30000 (time 1734264565.05 total 97.20 last 254.30) - open/close 32586 (time 1734264575.05 total 107.20 last 258.55) - open/close 35163 (time 1734264585.06 total 117.20 last 257.65) - open/close 37670 (time 1734264595.06 total 127.20 last 250.64) - open/close 40000 (time 1734264604.21 total 136.35 last 254.63) - open/close 42628 (time 1734264614.21 total 146.35 last 262.80) - open/close 45386 (time 1734264624.21 total 156.36 last 275.79) - open/close 47899 (time 1734264634.21 total 166.36 last 251.28) - open/close 50000 (time 1734264642.43 total 174.57 last 255.70) - open/close 52906 (time 1734264652.43 total 184.57 last 290.57) - open/close 56337 (time 1734264662.43 total 194.58 last 343.05) - open/close 59044 (time 1734264672.43 total 204.58 last 270.64) - open/close 60000 (time 1734264675.82 total 207.97 last 281.78) - open/close 62758 (time 1734264685.82 total 217.97 last 275.80) total: 64768 open/close in 225.31 seconds: 287.47 ops/second - /unlink 4759 (time 1734264704.28 total 10.00 last 475.87) - /unlink 9369 (time 1734264714.28 total 20.00 last 460.92) - /unlink 14120 (time 1734264724.29 total 30.00 last 475.04) - /unlink 18804 (time 1734264734.29 total 40.00 last 468.36) - /unlink 20000 (time 1734264736.73 total 42.45 last 488.69) - /unlink 24583 (time 1734264746.74 total 52.45 last 458.21) - /unlink 30000 (time 1734264756.45 total 62.17 last 557.55) - /unlink 37780 (time 1734264766.45 total 72.17 last 777.96) - /unlink 40000 (time 1734264768.86 total 74.57 last 923.55) - /unlink 47934 (time 1734264778.86 total 84.58 last 793.30) - /unlink 50000 (time 1734264781.16 total 86.88 last 898.10) - /unlink 56357 (time 1734264791.16 total 96.88 last 635.68) - /unlink 60000 (time 1734264797.20 total 102.91 last 603.35) total: 64768 /unlink in 112.11 seconds: 577.74 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2729 (time 1734264833.09 total 10.00 last 272.79) - open/close 5342 (time 1734264843.10 total 20.01 last 261.18) - open/close 7712 (time 1734264853.10 total 30.01 last 236.98) - open/close 10000 (time 1734264861.48 total 38.39 last 273.11) - open/close 12780 (time 1734264871.48 total 48.39 last 277.99) - open/close 15493 (time 1734264881.48 total 58.39 last 271.20) - open/close 17929 (time 1734264891.48 total 68.39 last 243.55) - open/close 20000 (time 1734264900.15 total 77.06 last 239.00) - open/close 22409 (time 1734264910.15 total 87.06 last 240.86) - open/close 24800 (time 1734264920.15 total 97.06 last 239.08) - open/close 27245 (time 1734264930.15 total 107.06 last 244.42) - open/close 29730 (time 1734264940.16 total 117.07 last 248.41) - open/close 32118 (time 1734264950.16 total 127.07 last 238.72) - open/close 34605 (time 1734264960.16 total 137.07 last 248.67) - open/close 37067 (time 1734264970.17 total 147.08 last 246.11) - open/close 39474 (time 1734264980.17 total 157.08 last 240.65) - open/close 40000 (time 1734264982.42 total 159.33 last 233.70) - open/close 42416 (time 1734264992.42 total 169.33 last 241.51) - open/close 44873 (time 1734265002.42 total 179.34 last 245.64) - open/close 47324 (time 1734265012.42 total 189.34 last 245.09) - open/close 49753 (time 1734265022.43 total 199.34 last 242.81) - open/close 52184 (time 1734265032.43 total 209.34 last 243.06) - open/close 54625 (time 1734265042.43 total 219.34 last 244.04) - open/close 57010 (time 1734265052.44 total 229.35 last 238.40) - open/close 59612 (time 1734265062.44 total 239.35 last 260.12) - open/close 62084 (time 1734265072.44 total 249.35 last 247.14) - open/close 64540 (time 1734265082.44 total 259.36 last 245.55) total: 64768 open/close in 260.37 seconds: 248.75 ops/second - /unlink 4388 (time 1734265094.70 total 10.00 last 438.72) - /unlink 9529 (time 1734265104.70 total 20.00 last 514.03) - /unlink 14279 (time 1734265114.70 total 30.00 last 474.95) - /unlink 19539 (time 1734265124.70 total 40.01 last 525.93) - /unlink 24612 (time 1734265134.70 total 50.01 last 507.19) - /unlink 29132 (time 1734265144.70 total 60.01 last 451.91) - /unlink 34449 (time 1734265154.70 total 70.01 last 531.65) - /unlink 40000 (time 1734265160.85 total 76.16 last 903.19) - /unlink 47741 (time 1734265170.85 total 86.16 last 774.06) - /unlink 50000 (time 1734265173.41 total 88.72 last 881.09) - /unlink 55792 (time 1734265183.41 total 98.72 last 579.18) - /unlink 60000 (time 1734265191.10 total 106.41 last 547.29) total: 64768 /unlink in 115.16 seconds: 562.43 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2574 (time 1734265228.64 total 10.00 last 257.31) - open/close 5088 (time 1734265238.64 total 20.01 last 251.34) - open/close 7644 (time 1734265248.64 total 30.01 last 255.57) - open/close 10000 (time 1734265258.16 total 39.52 last 247.57) - open/close 12417 (time 1734265268.16 total 49.53 last 241.65) - open/close 14904 (time 1734265278.16 total 59.53 last 248.62) - open/close 17354 (time 1734265288.16 total 69.53 last 245.00) - open/close 19957 (time 1734265298.16 total 79.53 last 260.30) - open/close 22696 (time 1734265308.16 total 89.53 last 273.85) - open/close 25380 (time 1734265318.17 total 99.53 last 268.32) - open/close 29171 (time 1734265328.17 total 109.53 last 379.07) - open/close 30000 (time 1734265330.33 total 111.70 last 382.81) - open/close 32502 (time 1734265340.33 total 121.70 last 250.17) - open/close 35007 (time 1734265350.33 total 131.70 last 250.49) - open/close 37510 (time 1734265360.34 total 141.70 last 250.25) - open/close 40000 (time 1734265370.13 total 151.50 last 254.26) - open/close 42505 (time 1734265380.13 total 161.50 last 250.43) - open/close 44971 (time 1734265390.13 total 171.50 last 246.57) - open/close 47405 (time 1734265400.14 total 181.50 last 243.35) - open/close 49831 (time 1734265410.14 total 191.51 last 242.52) - open/close 52314 (time 1734265420.14 total 201.51 last 248.23) - open/close 54779 (time 1734265430.14 total 211.51 last 246.41) - open/close 57298 (time 1734265440.15 total 221.52 last 251.82) - open/close 59773 (time 1734265450.15 total 231.52 last 247.50) - open/close 62178 (time 1734265460.15 total 241.52 last 240.50) - open/close 64546 (time 1734265470.15 total 251.52 last 236.74) total: 64768 open/close in 252.43 seconds: 256.57 ops/second - /unlink 4414 (time 1734265482.38 total 10.00 last 441.36) - /unlink 8847 (time 1734265492.38 total 20.00 last 443.23) - /unlink 10000 (time 1734265494.90 total 22.52 last 457.13) - /unlink 14460 (time 1734265504.90 total 32.53 last 445.95) - /unlink 19057 (time 1734265514.90 total 42.53 last 459.64) - /unlink 20000 (time 1734265517.02 total 44.64 last 445.68) - /unlink 25034 (time 1734265527.02 total 54.64 last 503.39) - /unlink 30000 (time 1734265534.84 total 62.46 last 635.01) - /unlink 34628 (time 1734265544.84 total 72.46 last 462.79) - /unlink 39247 (time 1734265554.84 total 82.46 last 461.88) - /unlink 43706 (time 1734265564.84 total 92.46 last 445.90) - /unlink 48376 (time 1734265574.84 total 102.47 last 466.91) - /unlink 50000 (time 1734265578.24 total 105.87 last 477.15) - /unlink 54562 (time 1734265588.25 total 115.87 last 456.11) - /unlink 59281 (time 1734265598.25 total 125.87 last 471.89) - /unlink 64033 (time 1734265608.25 total 135.87 last 475.14) total: 64768 /unlink in 137.57 seconds: 470.81 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server unloading modules via unload_modules_local on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2