== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 22:49:37 (1718851777) Stopping clients: oleg303-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg303-client.virtnet /mnt/lustre2 (opts:-f) oleg303-server: oleg303-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg303-server' oleg303-server: oleg303-server.virtnet: executing load_modules_local oleg303-server: Loading modules from /home/green/git/lustre-release/lustre oleg303-server: detected 4 online CPUs by sysfs oleg303-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg303-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg303-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg303-client.virtnet: -o user_xattr,flock oleg303-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2937 (time 1718851819.15 total 10.00 last 293.70) - open/close 5598 (time 1718851829.15 total 20.00 last 266.02) - open/close 8105 (time 1718851839.15 total 30.01 last 250.60) - open/close 10000 (time 1718851846.46 total 37.31 last 259.49) - open/close 12542 (time 1718851856.46 total 47.31 last 254.11) - open/close 15080 (time 1718851866.46 total 57.31 last 253.80) - open/close 17685 (time 1718851876.46 total 67.31 last 260.48) - open/close 20000 (time 1718851885.42 total 76.28 last 258.31) - open/close 22608 (time 1718851895.42 total 86.28 last 260.77) - open/close 25297 (time 1718851905.43 total 96.28 last 268.85) - open/close 27828 (time 1718851915.43 total 106.28 last 253.00) - open/close 30000 (time 1718851924.01 total 114.86 last 253.21) - open/close 32680 (time 1718851934.01 total 124.86 last 267.96) - open/close 35426 (time 1718851944.01 total 134.87 last 274.52) - open/close 37951 (time 1718851954.01 total 144.87 last 252.49) - open/close 40000 (time 1718851962.35 total 153.20 last 245.81) - open/close 42510 (time 1718851972.35 total 163.20 last 250.96) - open/close 45070 (time 1718851982.35 total 173.21 last 255.89) - open/close 47698 (time 1718851992.36 total 183.21 last 262.72) - open/close 50000 (time 1718852001.10 total 191.95 last 263.34) - open/close 52533 (time 1718852011.10 total 201.95 last 253.27) - open/close 55134 (time 1718852021.10 total 211.96 last 260.04) - open/close 57849 (time 1718852031.10 total 221.96 last 271.49) - open/close 60000 (time 1718852039.68 total 230.54 last 250.68) - open/close 62580 (time 1718852049.68 total 240.54 last 257.95) total: 64768 open/close in 249.46 seconds: 259.64 ops/second - /unlink 4578 (time 1718852069.91 total 10.00 last 457.77) - /unlink 9243 (time 1718852079.91 total 20.00 last 466.49) - /unlink 14174 (time 1718852089.91 total 30.00 last 493.07) - /unlink 18925 (time 1718852099.91 total 40.00 last 475.09) - /unlink 20000 (time 1718852102.27 total 42.37 last 454.81) - /unlink 24840 (time 1718852112.27 total 52.37 last 483.95) - /unlink 29665 (time 1718852122.27 total 62.37 last 482.48) - /unlink 34317 (time 1718852132.27 total 72.37 last 465.15) - /unlink 39203 (time 1718852142.28 total 82.37 last 488.52) - /unlink 44507 (time 1718852152.28 total 92.37 last 530.35) - /unlink 50000 (time 1718852161.63 total 101.72 last 587.33) - /unlink 54750 (time 1718852171.63 total 111.72 last 474.92) - /unlink 60000 (time 1718852180.78 total 120.87 last 573.90) total: 64768 /unlink in 128.65 seconds: 503.45 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2567 (time 1718852217.06 total 10.00 last 256.69) - open/close 5322 (time 1718852227.06 total 20.00 last 275.42) - open/close 8530 (time 1718852237.06 total 30.01 last 320.75) - open/close 10000 (time 1718852242.35 total 35.30 last 277.77) - open/close 12969 (time 1718852252.36 total 45.30 last 296.82) - open/close 15504 (time 1718852262.36 total 55.30 last 253.45) - open/close 18113 (time 1718852272.36 total 65.30 last 260.90) - open/close 20000 (time 1718852280.23 total 73.17 last 239.73) - open/close 22482 (time 1718852290.23 total 83.17 last 248.18) - open/close 25428 (time 1718852300.23 total 93.17 last 294.60) - open/close 28288 (time 1718852310.23 total 103.17 last 285.97) - open/close 30000 (time 1718852317.04 total 109.98 last 251.58) - open/close 34584 (time 1718852327.04 total 119.98 last 458.32) - open/close 38987 (time 1718852337.04 total 129.98 last 440.29) - open/close 44090 (time 1718852347.04 total 139.98 last 510.24) - open/close 49107 (time 1718852357.04 total 149.98 last 501.70) - open/close 53869 (time 1718852367.04 total 159.98 last 476.20) - open/close 58731 (time 1718852377.08 total 170.03 last 484.16) - open/close 60000 (time 1718852379.76 total 172.71 last 472.85) total: 64768 open/close in 181.81 seconds: 356.24 ops/second - /unlink 7142 (time 1718852399.67 total 10.00 last 714.13) - /unlink 10000 (time 1718852403.46 total 13.79 last 754.89) - /unlink 18363 (time 1718852413.46 total 23.79 last 836.25) - /unlink 26384 (time 1718852423.46 total 33.79 last 802.03) - /unlink 30000 (time 1718852428.77 total 39.10 last 681.24) - /unlink 37348 (time 1718852438.77 total 49.10 last 734.77) - /unlink 40000 (time 1718852443.05 total 53.37 last 620.10) - /unlink 46837 (time 1718852453.05 total 63.37 last 683.66) - /unlink 50000 (time 1718852459.72 total 70.05 last 473.84) - /unlink 55207 (time 1718852469.72 total 80.05 last 520.66) - /unlink 60000 (time 1718852479.12 total 89.45 last 509.98) total: 64768 /unlink in 99.06 seconds: 653.79 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3072 (time 1718852517.84 total 10.00 last 307.05) - open/close 5627 (time 1718852527.84 total 20.01 last 255.48) - open/close 8345 (time 1718852537.84 total 30.01 last 271.74) - open/close 10000 (time 1718852544.07 total 36.24 last 265.46) - open/close 12947 (time 1718852554.07 total 46.24 last 294.70) - open/close 15509 (time 1718852564.07 total 56.24 last 256.20) - open/close 18077 (time 1718852574.07 total 66.24 last 256.74) - open/close 20000 (time 1718852581.83 total 74.00 last 247.97) - open/close 22447 (time 1718852591.83 total 84.00 last 244.61) - open/close 25122 (time 1718852601.84 total 94.01 last 267.43) - open/close 27522 (time 1718852611.84 total 104.01 last 239.95) - open/close 30000 (time 1718852620.66 total 112.83 last 280.80) - open/close 32544 (time 1718852630.67 total 122.84 last 254.34) - open/close 35016 (time 1718852640.67 total 132.84 last 247.13) - open/close 37512 (time 1718852650.67 total 142.84 last 249.57) - open/close 40000 (time 1718852660.66 total 152.83 last 248.93) - open/close 42556 (time 1718852670.67 total 162.84 last 255.51) - open/close 45091 (time 1718852680.67 total 172.84 last 253.45) - open/close 47750 (time 1718852690.67 total 182.84 last 265.83) - open/close 50000 (time 1718852699.98 total 192.15 last 241.78) - open/close 52438 (time 1718852709.98 total 202.15 last 243.73) - open/close 55069 (time 1718852719.99 total 212.16 last 263.00) - open/close 57490 (time 1718852729.99 total 222.16 last 242.01) - open/close 59850 (time 1718852739.99 total 232.16 last 235.99) - open/close 62451 (time 1718852749.99 total 242.16 last 260.06) total: 64768 open/close in 251.91 seconds: 257.11 ops/second - /unlink 4230 (time 1718852771.07 total 10.00 last 422.90) - /unlink 8704 (time 1718852781.07 total 20.00 last 447.35) - /unlink 10000 (time 1718852783.75 total 22.68 last 484.46) - /unlink 14645 (time 1718852793.75 total 32.68 last 464.40) - /unlink 19109 (time 1718852803.75 total 42.68 last 446.40) - /unlink 20000 (time 1718852805.85 total 44.78 last 425.17) - /unlink 24414 (time 1718852815.85 total 54.78 last 441.35) - /unlink 28664 (time 1718852825.85 total 64.78 last 424.91) - /unlink 30000 (time 1718852828.90 total 67.83 last 437.92) - /unlink 34219 (time 1718852838.90 total 77.83 last 421.87) - /unlink 39245 (time 1718852848.90 total 87.83 last 502.58) - /unlink 43763 (time 1718852858.90 total 97.83 last 451.73) - /unlink 49168 (time 1718852868.91 total 107.84 last 540.37) - /unlink 53742 (time 1718852878.91 total 117.84 last 457.33) - /unlink 58420 (time 1718852888.91 total 127.84 last 467.78) - /unlink 60000 (time 1718852892.43 total 131.36 last 448.22) total: 64768 /unlink in 141.36 seconds: 458.19 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2829 (time 1718852928.42 total 10.00 last 282.80) - open/close 5757 (time 1718852938.42 total 20.01 last 292.74) - open/close 8348 (time 1718852948.43 total 30.01 last 259.01) - open/close 10000 (time 1718852954.76 total 36.34 last 261.05) - open/close 12631 (time 1718852964.76 total 46.34 last 263.05) - open/close 15362 (time 1718852974.76 total 56.34 last 273.03) - open/close 18018 (time 1718852984.76 total 66.34 last 265.59) - open/close 20000 (time 1718852991.75 total 73.33 last 283.66) - open/close 22841 (time 1718853001.75 total 83.33 last 284.00) - open/close 25377 (time 1718853011.75 total 93.33 last 253.55) - open/close 28274 (time 1718853021.75 total 103.34 last 289.67) - open/close 30000 (time 1718853027.71 total 109.29 last 289.67) - open/close 32651 (time 1718853037.71 total 119.30 last 265.06) - open/close 35140 (time 1718853047.72 total 129.30 last 248.86) - open/close 37542 (time 1718853057.72 total 139.30 last 240.11) - open/close 40000 (time 1718853067.46 total 149.04 last 252.40) - open/close 42644 (time 1718853077.46 total 159.04 last 264.36) - open/close 45089 (time 1718853087.46 total 169.04 last 244.40) - open/close 47743 (time 1718853097.46 total 179.05 last 265.40) - open/close 50000 (time 1718853106.96 total 188.54 last 237.80) - open/close 53023 (time 1718853116.96 total 198.54 last 302.26) - open/close 56635 (time 1718853126.96 total 208.54 last 361.15) - open/close 60000 (time 1718853135.90 total 217.48 last 376.22) - open/close 64518 (time 1718853145.90 total 227.48 last 451.74) total: 64768 open/close in 228.11 seconds: 283.93 ops/second - /unlink 6015 (time 1718853157.50 total 10.00 last 601.49) - /unlink 10000 (time 1718853163.34 total 15.84 last 682.66) - /unlink 15470 (time 1718853173.34 total 25.84 last 546.94) - /unlink 20000 (time 1718853181.34 total 33.84 last 566.47) - /unlink 25952 (time 1718853191.34 total 43.84 last 595.17) - /unlink 30000 (time 1718853198.20 total 50.69 last 590.25) - /unlink 34703 (time 1718853208.20 total 60.70 last 470.21) - /unlink 39671 (time 1718853218.20 total 70.70 last 496.68) - /unlink 44417 (time 1718853228.20 total 80.70 last 474.52) - /unlink 48711 (time 1718853238.21 total 90.70 last 429.31) - /unlink 50000 (time 1718853241.11 total 93.61 last 443.44) - /unlink 54615 (time 1718853251.11 total 103.61 last 461.45) - /unlink 59129 (time 1718853261.12 total 113.61 last 451.34) total: 64768 /unlink in 123.61 seconds: 523.98 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5191 (time 1718853298.39 total 10.00 last 519.09) - open/close 10000 (time 1718853308.06 total 19.67 last 497.13) - open/close 14836 (time 1718853318.07 total 29.67 last 483.55) - open/close 19871 (time 1718853328.07 total 39.68 last 503.41) - open/close 24805 (time 1718853338.07 total 49.68 last 493.30) - open/close 29388 (time 1718853348.07 total 59.68 last 458.19) - open/close 34068 (time 1718853358.07 total 69.68 last 467.96) - open/close 39223 (time 1718853368.07 total 79.68 last 515.43) - open/close 44240 (time 1718853378.07 total 89.68 last 501.68) - open/close 48736 (time 1718853388.10 total 99.71 last 448.54) - open/close 50000 (time 1718853390.79 total 102.40 last 469.32) - open/close 54785 (time 1718853400.79 total 112.40 last 478.45) - open/close 59777 (time 1718853410.79 total 122.40 last 499.20) total: 64768 open/close in 132.30 seconds: 489.55 ops/second - /unlink 8632 (time 1718853431.56 total 10.00 last 863.20) - /unlink 15976 (time 1718853441.56 total 20.00 last 734.40) - /unlink 20000 (time 1718853446.89 total 25.33 last 754.43) - /unlink 26643 (time 1718853456.89 total 35.33 last 664.26) - /unlink 30000 (time 1718853464.18 total 42.62 last 460.97) - /unlink 35287 (time 1718853474.18 total 52.62 last 528.68) - /unlink 40000 (time 1718853482.09 total 60.53 last 595.31) - /unlink 46255 (time 1718853492.10 total 70.54 last 625.40) - /unlink 50000 (time 1718853498.25 total 76.69 last 608.92) - /unlink 54595 (time 1718853508.25 total 86.69 last 459.43) - /unlink 59127 (time 1718853518.25 total 96.69 last 453.10) - /unlink 63828 (time 1718853528.25 total 106.69 last 470.04) total: 64768 /unlink in 108.67 seconds: 596.03 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg303-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg303-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg303-server stop mds service on oleg303-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg303-server unloading modules on: 'oleg303-server' oleg303-server: oleg303-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg303-client: oleg303-client: ssh exited with exit code 2 pdsh@oleg303-client: oleg303-server: ssh exited with exit code 2 pdsh@oleg303-client: oleg303-client: ssh exited with exit code 2 pdsh@oleg303-client: oleg303-server: ssh exited with exit code 2