== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:23:31 (1736263411) Stopping clients: oleg320-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg320-client.virtnet /mnt/lustre2 (opts:-f) oleg320-server: oleg320-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg320-server' oleg320-server: oleg320-server.virtnet: executing load_modules_local oleg320-server: Loading modules from /home/green/git/lustre-release/lustre oleg320-server: detected 4 online CPUs by sysfs oleg320-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg320-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg320-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg320-client.virtnet: -o user_xattr,flock oleg320-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16166 358 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1767 (time 1736263464.94 total 10.00 last 176.67) - open/close 3582 (time 1736263474.95 total 20.00 last 181.46) - open/close 5721 (time 1736263484.95 total 30.00 last 213.90) - open/close 8171 (time 1736263494.95 total 40.01 last 244.94) - open/close 10000 (time 1736263502.29 total 47.35 last 249.23) - open/close 11974 (time 1736263512.29 total 57.35 last 197.33) - open/close 13870 (time 1736263522.30 total 67.36 last 189.41) - open/close 15599 (time 1736263532.30 total 77.36 last 172.84) - open/close 17362 (time 1736263542.31 total 87.36 last 176.25) - open/close 18982 (time 1736263552.31 total 97.37 last 161.95) - open/close 20000 (time 1736263558.43 total 103.49 last 166.33) - open/close 21886 (time 1736263568.43 total 113.49 last 188.60) - open/close 23925 (time 1736263578.43 total 123.49 last 203.89) - open/close 25947 (time 1736263588.43 total 133.49 last 202.12) - open/close 28133 (time 1736263598.44 total 143.50 last 218.55) - open/close 30000 (time 1736263606.13 total 151.18 last 242.82) - open/close 32823 (time 1736263616.13 total 161.19 last 282.19) - open/close 35873 (time 1736263626.13 total 171.19 last 305.00) - open/close 38982 (time 1736263636.13 total 181.19 last 310.85) - open/close 40000 (time 1736263638.72 total 183.78 last 392.70) - open/close 43940 (time 1736263648.72 total 193.78 last 393.98) - open/close 47544 (time 1736263658.73 total 203.79 last 360.30) - open/close 50000 (time 1736263665.23 total 210.29 last 377.71) - open/close 54730 (time 1736263675.23 total 220.29 last 472.93) - open/close 59680 (time 1736263685.23 total 230.29 last 494.90) - open/close 64064 (time 1736263695.23 total 240.29 last 438.37) total: 64768 open/close in 242.07 seconds: 267.55 ops/second - /unlink 6367 (time 1736263707.91 total 10.00 last 636.62) - /unlink 10000 (time 1736263713.07 total 15.16 last 704.47) - /unlink 17844 (time 1736263723.07 total 25.16 last 784.35) - /unlink 20000 (time 1736263726.29 total 28.38 last 670.10) - /unlink 27113 (time 1736263736.29 total 38.38 last 711.28) - /unlink 30000 (time 1736263741.09 total 43.18 last 601.13) - /unlink 37553 (time 1736263751.09 total 53.18 last 755.28) - /unlink 40000 (time 1736263754.73 total 56.82 last 672.96) - /unlink 47075 (time 1736263764.73 total 66.82 last 707.46) - /unlink 50000 (time 1736263768.78 total 70.86 last 722.52) - /unlink 57562 (time 1736263778.78 total 80.87 last 756.17) - /unlink 60000 (time 1736263781.92 total 84.01 last 775.89) total: 64768 /unlink in 90.89 seconds: 712.57 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3617 (time 1736263817.97 total 10.00 last 361.62) - open/close 7342 (time 1736263827.98 total 20.00 last 372.47) - open/close 10000 (time 1736263834.20 total 26.23 last 427.06) - open/close 13669 (time 1736263844.20 total 36.23 last 366.89) - open/close 17031 (time 1736263854.20 total 46.23 last 336.12) - open/close 20000 (time 1736263863.42 total 55.44 last 322.20) - open/close 23249 (time 1736263873.42 total 65.45 last 324.83) - open/close 26268 (time 1736263883.42 total 75.45 last 301.86) - open/close 29946 (time 1736263893.42 total 85.45 last 367.80) - open/close 33458 (time 1736263903.42 total 95.45 last 351.18) - open/close 36721 (time 1736263913.42 total 105.45 last 326.27) - open/close 40000 (time 1736263922.96 total 114.99 last 343.66) - open/close 43360 (time 1736263932.96 total 124.99 last 336.00) - open/close 46637 (time 1736263942.96 total 134.99 last 327.68) - open/close 49799 (time 1736263952.97 total 144.99 last 316.11) - open/close 52836 (time 1736263962.97 total 155.00 last 303.64) - open/close 56404 (time 1736263972.97 total 165.00 last 356.74) - open/close 59972 (time 1736263982.97 total 175.00 last 356.74) - open/close 64251 (time 1736263992.97 total 185.00 last 427.83) total: 64768 open/close in 186.18 seconds: 347.89 ops/second - /unlink 7967 (time 1736264005.01 total 10.00 last 796.63) - /unlink 10000 (time 1736264007.03 total 12.02 last 1006.45) - /unlink 19486 (time 1736264017.03 total 22.02 last 948.51) - /unlink 28755 (time 1736264027.03 total 32.02 last 926.88) - /unlink 39109 (time 1736264037.03 total 42.02 last 1035.32) - /unlink 49628 (time 1736264047.03 total 52.02 last 1051.90) - /unlink 59724 (time 1736264057.03 total 62.02 last 1009.50) total: 64768 /unlink in 66.93 seconds: 967.66 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5011 (time 1736264108.70 total 10.00 last 501.08) - open/close 9804 (time 1736264118.71 total 20.00 last 479.26) - open/close 15102 (time 1736264128.71 total 30.00 last 529.75) - open/close 20000 (time 1736264137.48 total 38.77 last 558.55) - open/close 25180 (time 1736264147.48 total 48.77 last 517.93) - open/close 30000 (time 1736264156.03 total 57.32 last 563.68) - open/close 35645 (time 1736264166.03 total 67.32 last 564.47) - open/close 40000 (time 1736264173.62 total 74.91 last 573.84) - open/close 45310 (time 1736264183.62 total 84.91 last 530.97) - open/close 50000 (time 1736264191.77 total 93.06 last 575.60) - open/close 55625 (time 1736264201.77 total 103.06 last 562.48) - open/close 60000 (time 1736264210.25 total 111.54 last 515.79) total: 64768 open/close in 121.00 seconds: 535.26 ops/second - /unlink 8538 (time 1736264230.60 total 10.00 last 853.74) - /unlink 17037 (time 1736264240.60 total 20.00 last 849.80) - /unlink 20000 (time 1736264244.16 total 23.57 last 831.55) - /unlink 28324 (time 1736264254.16 total 33.57 last 832.36) - /unlink 37238 (time 1736264264.16 total 43.57 last 891.32) - /unlink 40000 (time 1736264267.23 total 46.64 last 899.08) - /unlink 48919 (time 1736264277.24 total 56.64 last 891.83) - /unlink 57579 (time 1736264287.24 total 66.64 last 865.98) - /unlink 60000 (time 1736264290.09 total 69.49 last 848.76) total: 64768 /unlink in 74.33 seconds: 871.36 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5339 (time 1736264333.96 total 10.00 last 533.81) - open/close 10000 (time 1736264343.30 total 19.35 last 498.83) - open/close 15109 (time 1736264353.31 total 29.35 last 510.84) - open/close 20000 (time 1736264363.22 total 39.27 last 493.09) - open/close 24848 (time 1736264373.23 total 49.27 last 484.73) - open/close 30000 (time 1736264383.17 total 59.21 last 518.23) - open/close 35170 (time 1736264393.17 total 69.21 last 516.96) - open/close 39884 (time 1736264403.17 total 79.21 last 471.25) - open/close 44622 (time 1736264413.23 total 89.27 last 470.97) - open/close 49244 (time 1736264423.23 total 99.27 last 462.16) - open/close 53900 (time 1736264433.23 total 109.27 last 465.57) - open/close 58914 (time 1736264443.23 total 119.27 last 501.38) - open/close 60000 (time 1736264445.37 total 121.41 last 508.90) total: 64768 open/close in 131.09 seconds: 494.07 ops/second - /unlink 9075 (time 1736264465.89 total 10.00 last 907.42) - /unlink 18171 (time 1736264475.89 total 20.00 last 909.50) - /unlink 20000 (time 1736264477.93 total 22.04 last 896.38) - /unlink 29344 (time 1736264487.93 total 32.04 last 934.36) - /unlink 38595 (time 1736264497.93 total 42.04 last 924.99) - /unlink 47822 (time 1736264507.93 total 52.04 last 922.65) - /unlink 50000 (time 1736264510.22 total 54.33 last 951.85) - /unlink 59192 (time 1736264520.22 total 64.33 last 919.15) total: 64768 /unlink in 70.17 seconds: 923.04 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5370 (time 1736264564.99 total 10.00 last 536.92) - open/close 10000 (time 1736264573.67 total 18.69 last 533.09) - open/close 15458 (time 1736264583.67 total 28.69 last 545.64) - open/close 20000 (time 1736264592.32 total 37.33 last 525.65) - open/close 25405 (time 1736264602.32 total 47.33 last 540.48) - open/close 30000 (time 1736264610.94 total 55.96 last 532.53) - open/close 35306 (time 1736264620.95 total 65.96 last 530.53) - open/close 40000 (time 1736264629.69 total 74.71 last 536.78) - open/close 45007 (time 1736264639.71 total 84.72 last 499.78) - open/close 50000 (time 1736264649.45 total 94.46 last 512.76) - open/close 55222 (time 1736264659.45 total 104.46 last 522.15) - open/close 60000 (time 1736264668.94 total 113.95 last 503.33) total: 64768 open/close in 123.27 seconds: 525.43 ops/second - /unlink 9197 (time 1736264689.11 total 10.00 last 919.67) - /unlink 18079 (time 1736264699.11 total 20.00 last 888.14) - /unlink 20000 (time 1736264701.23 total 22.12 last 907.56) - /unlink 29180 (time 1736264711.23 total 32.12 last 917.96) - /unlink 38437 (time 1736264721.23 total 42.12 last 925.63) - /unlink 47817 (time 1736264731.23 total 52.12 last 937.93) - /unlink 50000 (time 1736264733.56 total 54.45 last 937.18) - /unlink 59424 (time 1736264743.56 total 64.45 last 942.36) total: 64768 /unlink in 69.81 seconds: 927.74 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg320-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg320-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg320-server stop mds service on oleg320-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg320-server LNET ready to unload unloading modules on: 'oleg320-server' oleg320-server: oleg320-server.virtnet: executing unload_modules_local oleg320-server: LNET ready to unload modules unloaded. pdsh@oleg320-client: oleg320-client: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-server: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-client: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-server: ssh exited with exit code 2