== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:28:30 (1718126910) Stopping clients: oleg256-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg256-client.virtnet /mnt/lustre2 (opts:-f) oleg256-server: oleg256-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing load_modules_local oleg256-server: Loading modules from /home/green/git/lustre-release/lustre oleg256-server: detected 4 online CPUs by sysfs oleg256-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg256-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg256-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg256-client.virtnet: -o user_xattr,flock oleg256-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2510 (time 1718126950.94 total 10.00 last 250.95) - open/close 5458 (time 1718126960.94 total 20.00 last 294.74) - open/close 8639 (time 1718126970.94 total 30.00 last 318.09) - open/close 10000 (time 1718126973.72 total 32.79 last 488.87) - open/close 12750 (time 1718126983.73 total 42.79 last 274.91) - open/close 15485 (time 1718126993.73 total 52.80 last 273.40) - open/close 18463 (time 1718127003.73 total 62.80 last 297.77) - open/close 20000 (time 1718127007.99 total 67.05 last 361.26) - open/close 24112 (time 1718127017.99 total 77.05 last 411.19) - open/close 28437 (time 1718127027.99 total 87.05 last 432.35) - open/close 30000 (time 1718127031.42 total 90.48 last 455.87) - open/close 34874 (time 1718127041.42 total 100.48 last 487.34) - open/close 38138 (time 1718127051.42 total 110.49 last 326.35) - open/close 40000 (time 1718127057.84 total 116.91 last 290.00) - open/close 42990 (time 1718127067.84 total 126.91 last 298.89) - open/close 46056 (time 1718127077.84 total 136.91 last 306.60) - open/close 48802 (time 1718127087.85 total 146.91 last 274.57) - open/close 50000 (time 1718127092.72 total 151.78 last 246.02) - open/close 52881 (time 1718127102.72 total 161.78 last 288.10) - open/close 55567 (time 1718127112.72 total 171.78 last 268.55) - open/close 59125 (time 1718127122.72 total 181.78 last 355.74) - open/close 60000 (time 1718127126.12 total 185.18 last 257.56) - open/close 62947 (time 1718127136.12 total 195.18 last 294.69) total: 64768 open/close in 202.43 seconds: 319.95 ops/second - /unlink 4896 (time 1718127154.39 total 10.00 last 489.59) - /unlink 9417 (time 1718127164.39 total 20.00 last 452.04) - /unlink 14708 (time 1718127174.39 total 30.00 last 529.09) - /unlink 19719 (time 1718127184.39 total 40.00 last 501.05) - /unlink 24861 (time 1718127194.40 total 50.00 last 514.19) - /unlink 29343 (time 1718127204.40 total 60.01 last 447.99) - /unlink 33865 (time 1718127214.40 total 70.01 last 452.19) - /unlink 38691 (time 1718127224.40 total 80.01 last 482.57) - /unlink 40000 (time 1718127227.14 total 82.75 last 478.31) - /unlink 44713 (time 1718127237.14 total 92.75 last 471.30) - /unlink 50000 (time 1718127246.79 total 102.40 last 547.83) - /unlink 54773 (time 1718127256.79 total 112.40 last 477.27) - /unlink 59714 (time 1718127266.79 total 122.40 last 494.06) - /unlink 64468 (time 1718127276.79 total 132.40 last 475.37) total: 64768 /unlink in 132.78 seconds: 487.79 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2460 (time 1718127304.14 total 10.00 last 245.99) - open/close 5088 (time 1718127314.14 total 20.00 last 262.76) - open/close 7738 (time 1718127324.14 total 30.00 last 264.97) - open/close 10000 (time 1718127333.35 total 39.22 last 245.48) - open/close 13598 (time 1718127343.35 total 49.22 last 359.71) - open/close 17018 (time 1718127353.36 total 59.22 last 341.93) - open/close 19293 (time 1718127363.36 total 69.23 last 227.43) - open/close 20000 (time 1718127365.94 total 71.80 last 274.44) - open/close 22983 (time 1718127375.94 total 81.80 last 298.23) - open/close 26176 (time 1718127385.94 total 91.81 last 319.20) - open/close 29208 (time 1718127395.94 total 101.81 last 303.15) - open/close 30000 (time 1718127398.89 total 104.75 last 269.09) - open/close 32665 (time 1718127408.89 total 114.75 last 266.45) - open/close 35023 (time 1718127418.89 total 124.76 last 235.76) - open/close 37493 (time 1718127428.89 total 134.76 last 246.93) - open/close 39891 (time 1718127438.90 total 144.76 last 239.71) - open/close 42383 (time 1718127448.90 total 154.76 last 249.19) - open/close 44931 (time 1718127458.90 total 164.77 last 254.72) - open/close 47579 (time 1718127468.90 total 174.77 last 264.78) - open/close 49973 (time 1718127478.90 total 184.77 last 239.33) - open/close 52571 (time 1718127488.90 total 194.77 last 259.80) - open/close 54962 (time 1718127498.91 total 204.77 last 239.02) - open/close 57351 (time 1718127508.91 total 214.77 last 238.85) - open/close 59713 (time 1718127518.91 total 224.78 last 236.13) - open/close 62112 (time 1718127528.92 total 234.78 last 239.80) - open/close 64413 (time 1718127538.92 total 244.79 last 230.00) total: 64768 open/close in 246.35 seconds: 262.91 ops/second - /unlink 4653 (time 1718127551.80 total 10.00 last 465.20) - /unlink 9221 (time 1718127561.80 total 20.00 last 456.77) - /unlink 13805 (time 1718127571.81 total 30.00 last 458.30) - /unlink 18525 (time 1718127581.81 total 40.00 last 472.00) - /unlink 20000 (time 1718127584.56 total 42.76 last 536.22) - /unlink 24779 (time 1718127594.56 total 52.76 last 477.85) - /unlink 29384 (time 1718127604.56 total 62.76 last 460.50) - /unlink 34388 (time 1718127614.56 total 72.76 last 500.36) - /unlink 39207 (time 1718127624.56 total 82.76 last 481.86) - /unlink 43857 (time 1718127634.56 total 92.76 last 464.91) - /unlink 48722 (time 1718127644.56 total 102.76 last 486.48) - /unlink 50000 (time 1718127647.22 total 105.42 last 481.07) - /unlink 54765 (time 1718127657.22 total 115.42 last 476.38) - /unlink 59438 (time 1718127667.22 total 125.42 last 467.25) - /unlink 64178 (time 1718127677.22 total 135.42 last 473.95) total: 64768 /unlink in 136.52 seconds: 474.41 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2579 (time 1718127704.23 total 10.00 last 257.83) - open/close 5197 (time 1718127714.23 total 20.01 last 261.72) - open/close 7768 (time 1718127724.23 total 30.01 last 257.07) - open/close 10000 (time 1718127732.76 total 38.54 last 261.65) - open/close 12471 (time 1718127742.76 total 48.54 last 247.06) - open/close 14979 (time 1718127752.76 total 58.54 last 250.78) - open/close 17649 (time 1718127762.77 total 68.54 last 266.94) - open/close 20000 (time 1718127771.03 total 76.81 last 284.42) - open/close 22754 (time 1718127781.03 total 86.81 last 275.39) - open/close 25371 (time 1718127791.03 total 96.81 last 261.64) - open/close 27970 (time 1718127801.04 total 106.81 last 259.86) - open/close 30000 (time 1718127808.97 total 114.75 last 255.86) - open/close 32486 (time 1718127818.97 total 124.75 last 248.49) - open/close 35221 (time 1718127828.98 total 134.75 last 273.41) - open/close 38113 (time 1718127838.98 total 144.76 last 289.10) - open/close 40000 (time 1718127845.28 total 151.06 last 299.46) - open/close 42741 (time 1718127855.28 total 161.06 last 274.06) - open/close 45597 (time 1718127865.29 total 171.06 last 285.53) - open/close 48211 (time 1718127875.29 total 181.06 last 261.36) - open/close 50000 (time 1718127882.11 total 187.89 last 262.01) - open/close 52804 (time 1718127892.12 total 197.89 last 280.39) - open/close 55235 (time 1718127902.12 total 207.89 last 243.07) - open/close 57678 (time 1718127912.12 total 217.89 last 244.29) - open/close 60000 (time 1718127921.71 total 227.49 last 241.94) - open/close 62434 (time 1718127931.72 total 237.49 last 243.36) total: 64768 open/close in 246.94 seconds: 262.29 ops/second - /unlink 4487 (time 1718127952.43 total 10.00 last 448.66) - /unlink 9569 (time 1718127962.43 total 20.00 last 508.16) - /unlink 14318 (time 1718127972.43 total 30.00 last 474.84) - /unlink 19198 (time 1718127982.43 total 40.00 last 487.92) - /unlink 23972 (time 1718127992.43 total 50.00 last 477.39) - /unlink 28768 (time 1718128002.43 total 60.01 last 479.51) - /unlink 30000 (time 1718128004.90 total 62.47 last 499.77) - /unlink 34661 (time 1718128014.90 total 72.47 last 466.03) - /unlink 39539 (time 1718128024.90 total 82.47 last 487.78) - /unlink 44303 (time 1718128034.90 total 92.48 last 476.32) - /unlink 48862 (time 1718128044.90 total 102.48 last 455.86) - /unlink 50000 (time 1718128047.55 total 105.13 last 428.88) - /unlink 54699 (time 1718128057.56 total 115.13 last 469.87) - /unlink 59356 (time 1718128067.56 total 125.13 last 465.67) - /unlink 64406 (time 1718128077.56 total 135.13 last 504.96) total: 64768 /unlink in 135.88 seconds: 476.66 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3243 (time 1718128105.33 total 10.00 last 324.26) - open/close 5850 (time 1718128115.34 total 20.00 last 260.68) - open/close 8532 (time 1718128125.34 total 30.00 last 268.17) - open/close 10000 (time 1718128129.50 total 34.17 last 352.46) - open/close 13024 (time 1718128139.50 total 44.17 last 302.34) - open/close 15658 (time 1718128149.51 total 54.17 last 263.36) - open/close 18816 (time 1718128159.51 total 64.17 last 315.75) - open/close 20000 (time 1718128162.69 total 67.36 last 371.84) - open/close 23924 (time 1718128172.69 total 77.36 last 392.36) - open/close 27140 (time 1718128182.69 total 87.36 last 321.54) - open/close 30000 (time 1718128192.04 total 96.71 last 305.87) - open/close 33423 (time 1718128202.05 total 106.71 last 342.24) - open/close 37133 (time 1718128212.05 total 116.71 last 370.93) - open/close 40000 (time 1718128217.82 total 122.49 last 496.63) - open/close 44368 (time 1718128227.82 total 132.49 last 436.70) - open/close 47691 (time 1718128237.82 total 142.49 last 332.25) - open/close 50000 (time 1718128245.88 total 150.54 last 286.78) - open/close 52950 (time 1718128255.88 total 160.54 last 294.95) - open/close 56360 (time 1718128265.88 total 170.55 last 340.88) - open/close 59478 (time 1718128275.88 total 180.55 last 311.74) - open/close 60000 (time 1718128278.04 total 182.71 last 241.59) - open/close 62430 (time 1718128288.05 total 192.71 last 242.93) total: 64768 open/close in 199.12 seconds: 325.27 ops/second - /unlink 5759 (time 1718128305.47 total 10.00 last 575.85) - /unlink 10000 (time 1718128312.82 total 17.34 last 577.50) - /unlink 15930 (time 1718128322.82 total 27.35 last 592.95) - /unlink 20000 (time 1718128329.94 total 34.47 last 571.26) - /unlink 25633 (time 1718128339.94 total 44.47 last 563.24) - /unlink 30000 (time 1718128347.35 total 51.88 last 589.81) - /unlink 35731 (time 1718128357.35 total 61.88 last 573.06) - /unlink 40000 (time 1718128365.04 total 69.57 last 554.90) - /unlink 45465 (time 1718128375.04 total 79.57 last 546.44) - /unlink 50000 (time 1718128383.01 total 87.54 last 569.17) - /unlink 55622 (time 1718128393.01 total 97.54 last 562.15) - /unlink 60000 (time 1718128400.71 total 105.24 last 568.71) total: 64768 /unlink in 112.91 seconds: 573.62 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3379 (time 1718128438.63 total 10.00 last 337.89) - open/close 6497 (time 1718128448.63 total 20.00 last 311.75) - open/close 9408 (time 1718128458.63 total 30.00 last 291.03) - open/close 10000 (time 1718128460.74 total 32.12 last 280.39) - open/close 14543 (time 1718128470.75 total 42.12 last 454.18) - open/close 19261 (time 1718128480.75 total 52.12 last 471.76) - open/close 24374 (time 1718128490.75 total 62.12 last 511.23) - open/close 29634 (time 1718128500.75 total 72.12 last 525.98) - open/close 34404 (time 1718128510.75 total 82.12 last 476.96) - open/close 39462 (time 1718128520.75 total 92.12 last 505.76) - open/close 44296 (time 1718128530.75 total 102.12 last 483.34) - open/close 48853 (time 1718128540.75 total 112.12 last 455.65) - open/close 50000 (time 1718128543.59 total 114.96 last 404.74) - open/close 54043 (time 1718128553.59 total 124.96 last 404.27) - open/close 58607 (time 1718128563.59 total 134.96 last 456.33) - open/close 60000 (time 1718128566.55 total 137.92 last 470.07) total: 64768 open/close in 147.54 seconds: 438.99 ops/second - /unlink 8482 (time 1718128587.01 total 10.00 last 848.19) - /unlink 17048 (time 1718128597.01 total 20.00 last 856.55) - /unlink 20000 (time 1718128600.59 total 23.57 last 826.11) - /unlink 27882 (time 1718128610.59 total 33.57 last 788.14) - /unlink 30000 (time 1718128613.21 total 36.19 last 808.42) - /unlink 36421 (time 1718128623.21 total 46.20 last 642.07) - /unlink 40000 (time 1718128630.13 total 53.12 last 516.61) - /unlink 45443 (time 1718128640.14 total 63.12 last 544.25) - /unlink 50000 (time 1718128648.72 total 71.71 last 530.92) - /unlink 54826 (time 1718128658.72 total 81.71 last 482.53) - /unlink 59740 (time 1718128668.72 total 91.71 last 491.39) - /unlink 64765 (time 1718128678.72 total 101.71 last 502.41) total: 64768 /unlink in 101.71 seconds: 636.76 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg256-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg256-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg256-server unloading modules on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2