== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 11:26:37 (1736785597) Stopping clients: oleg357-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg357-client.virtnet /mnt/lustre2 (opts:-f) oleg357-server: oleg357-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg357-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg357-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3107 (time 1736785635.95 total 10.00 last 310.63) - open/close 6010 (time 1736785645.95 total 20.01 last 290.20) - open/close 9063 (time 1736785655.96 total 30.01 last 305.20) - open/close 10000 (time 1736785658.79 total 32.85 last 330.07) - open/close 14564 (time 1736785668.79 total 42.85 last 456.39) - open/close 18498 (time 1736785678.79 total 52.85 last 393.40) - open/close 20000 (time 1736785682.56 total 56.61 last 399.15) - open/close 23708 (time 1736785692.56 total 66.61 last 370.80) - open/close 28364 (time 1736785702.56 total 76.61 last 465.49) - open/close 30000 (time 1736785706.59 total 80.65 last 405.47) - open/close 33719 (time 1736785716.60 total 90.65 last 371.85) - open/close 36919 (time 1736785726.60 total 100.65 last 319.91) - open/close 40000 (time 1736785734.93 total 108.99 last 369.64) - open/close 43268 (time 1736785744.93 total 118.99 last 326.79) - open/close 47870 (time 1736785754.94 total 128.99 last 460.14) - open/close 50000 (time 1736785759.63 total 133.68 last 453.95) - open/close 53170 (time 1736785769.63 total 143.68 last 316.95) - open/close 55972 (time 1736785779.63 total 153.68 last 280.19) - open/close 59120 (time 1736785789.63 total 163.68 last 314.76) - open/close 60000 (time 1736785792.64 total 166.69 last 292.88) - open/close 63222 (time 1736785802.64 total 176.69 last 322.19) total: 64768 open/close in 181.80 seconds: 356.26 ops/second - /unlink 7527 (time 1736785818.77 total 10.00 last 752.68) - /unlink 10000 (time 1736785822.06 total 13.30 last 750.09) - /unlink 18367 (time 1736785832.06 total 23.30 last 836.66) - /unlink 26318 (time 1736785842.07 total 33.30 last 795.02) - /unlink 30000 (time 1736785846.71 total 37.95 last 792.13) - /unlink 38935 (time 1736785856.71 total 47.95 last 893.42) - /unlink 46947 (time 1736785866.71 total 57.95 last 801.19) - /unlink 50000 (time 1736785869.97 total 61.20 last 938.28) - /unlink 58205 (time 1736785879.97 total 71.20 last 820.43) - /unlink 60000 (time 1736785882.65 total 73.88 last 669.42) total: 64768 /unlink in 82.96 seconds: 780.68 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3782 (time 1736785918.39 total 10.00 last 378.16) - open/close 7945 (time 1736785928.39 total 20.00 last 416.27) - open/close 10000 (time 1736785933.37 total 24.99 last 412.30) - open/close 13165 (time 1736785943.38 total 34.99 last 316.44) - open/close 16476 (time 1736785953.38 total 44.99 last 331.02) - open/close 19066 (time 1736785963.38 total 54.99 last 258.99) - open/close 20000 (time 1736785966.70 total 58.32 last 280.86) - open/close 22912 (time 1736785976.71 total 68.32 last 291.16) - open/close 27915 (time 1736785986.71 total 78.32 last 500.30) - open/close 30000 (time 1736785990.54 total 82.15 last 544.09) - open/close 35090 (time 1736786000.54 total 92.15 last 508.95) - open/close 39527 (time 1736786010.54 total 102.15 last 443.63) - open/close 43653 (time 1736786020.54 total 112.15 last 412.59) - open/close 48689 (time 1736786030.56 total 122.18 last 502.38) - open/close 50000 (time 1736786033.46 total 125.07 last 452.84) - open/close 54612 (time 1736786043.46 total 135.07 last 461.18) - open/close 58751 (time 1736786053.46 total 145.08 last 413.78) - open/close 60000 (time 1736786057.46 total 149.07 last 312.48) - open/close 63755 (time 1736786067.46 total 159.07 last 375.48) total: 64768 open/close in 162.92 seconds: 397.53 ops/second - /unlink 4727 (time 1736786082.62 total 10.00 last 472.60) - /unlink 9980 (time 1736786092.62 total 20.00 last 525.22) - /unlink 15129 (time 1736786102.62 total 30.00 last 514.88) - /unlink 20000 (time 1736786112.06 total 39.44 last 515.99) - /unlink 25070 (time 1736786122.06 total 49.45 last 506.91) - /unlink 30000 (time 1736786128.55 total 55.93 last 759.98) - /unlink 35885 (time 1736786138.55 total 65.93 last 588.44) - /unlink 40000 (time 1736786146.18 total 73.56 last 539.60) - /unlink 45302 (time 1736786156.18 total 83.56 last 530.16) - /unlink 50000 (time 1736786165.66 total 93.04 last 495.51) - /unlink 54807 (time 1736786175.66 total 103.04 last 480.68) - /unlink 60000 (time 1736786184.72 total 112.10 last 573.13) total: 64768 /unlink in 119.73 seconds: 540.96 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4363 (time 1736786219.01 total 10.00 last 436.27) - open/close 8572 (time 1736786229.01 total 20.00 last 420.86) - open/close 10000 (time 1736786232.27 total 23.26 last 438.23) - open/close 14437 (time 1736786242.27 total 33.26 last 443.58) - open/close 17545 (time 1736786252.27 total 43.26 last 310.78) - open/close 20000 (time 1736786258.71 total 49.71 last 380.93) - open/close 24583 (time 1736786268.71 total 59.71 last 458.25) - open/close 27776 (time 1736786278.72 total 69.71 last 319.26) - open/close 30000 (time 1736786285.13 total 76.13 last 346.62) - open/close 33061 (time 1736786295.13 total 86.13 last 306.10) - open/close 36704 (time 1736786305.14 total 96.13 last 364.17) - open/close 39590 (time 1736786315.14 total 106.13 last 288.54) - open/close 42370 (time 1736786325.14 total 116.13 last 277.93) - open/close 45789 (time 1736786335.14 total 126.14 last 341.84) - open/close 49401 (time 1736786345.14 total 136.14 last 361.13) - open/close 50000 (time 1736786347.36 total 138.35 last 270.28) - open/close 52798 (time 1736786357.36 total 148.36 last 279.79) - open/close 55828 (time 1736786367.36 total 158.36 last 302.93) - open/close 59224 (time 1736786377.36 total 168.36 last 339.57) - open/close 60000 (time 1736786379.63 total 170.62 last 342.78) - open/close 63402 (time 1736786389.63 total 180.63 last 340.09) total: 64768 open/close in 184.88 seconds: 350.33 ops/second - /unlink 5073 (time 1736786405.08 total 10.00 last 507.21) - /unlink 10000 (time 1736786411.81 total 16.73 last 732.25) - /unlink 17096 (time 1736786421.81 total 26.73 last 709.50) - /unlink 20000 (time 1736786426.65 total 31.57 last 599.97) - /unlink 25915 (time 1736786436.65 total 41.57 last 591.49) - /unlink 30000 (time 1736786442.05 total 46.97 last 756.40) - /unlink 36632 (time 1736786452.05 total 56.97 last 663.20) - /unlink 40000 (time 1736786456.90 total 61.83 last 693.76) - /unlink 47268 (time 1736786466.90 total 71.83 last 726.78) - /unlink 50000 (time 1736786470.88 total 75.80 last 687.31) - /unlink 55855 (time 1736786480.88 total 85.80 last 585.44) - /unlink 60000 (time 1736786487.22 total 92.14 last 654.09) total: 64768 /unlink in 100.70 seconds: 643.21 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2734 (time 1736786524.57 total 10.00 last 273.36) - open/close 5401 (time 1736786534.57 total 20.00 last 266.63) - open/close 8630 (time 1736786544.57 total 30.01 last 322.78) - open/close 10000 (time 1736786549.56 total 34.99 last 274.85) - open/close 13161 (time 1736786559.56 total 45.00 last 315.98) - open/close 16176 (time 1736786569.56 total 55.00 last 301.49) - open/close 19469 (time 1736786579.57 total 65.00 last 329.21) - open/close 23019 (time 1736786589.57 total 75.00 last 354.91) - open/close 25574 (time 1736786599.57 total 85.00 last 255.49) - open/close 28574 (time 1736786609.57 total 95.01 last 299.89) - open/close 30000 (time 1736786614.59 total 100.02 last 284.38) - open/close 32642 (time 1736786624.59 total 110.02 last 264.18) - open/close 35501 (time 1736786634.59 total 120.02 last 285.88) - open/close 39036 (time 1736786644.59 total 130.02 last 353.45) - open/close 40000 (time 1736786647.29 total 132.72 last 357.45) - open/close 43615 (time 1736786657.29 total 142.72 last 361.48) - open/close 47395 (time 1736786667.29 total 152.72 last 377.95) - open/close 50000 (time 1736786675.72 total 161.15 last 309.06) - open/close 52920 (time 1736786685.72 total 171.15 last 291.97) - open/close 55764 (time 1736786695.75 total 181.19 last 283.42) - open/close 58456 (time 1736786705.76 total 191.19 last 269.11) - open/close 60000 (time 1736786711.07 total 196.51 last 290.41) - open/close 63701 (time 1736786721.07 total 206.51 last 370.05) total: 64768 open/close in 209.73 seconds: 308.82 ops/second - /unlink 5272 (time 1736786735.48 total 10.00 last 527.14) - /unlink 10000 (time 1736786743.61 total 18.13 last 581.60) - /unlink 15184 (time 1736786753.61 total 28.13 last 518.36) - /unlink 20000 (time 1736786763.24 total 37.76 last 500.30) - /unlink 26051 (time 1736786773.24 total 47.76 last 605.07) - /unlink 30000 (time 1736786780.02 total 54.54 last 582.27) - /unlink 35228 (time 1736786790.03 total 64.54 last 522.71) - /unlink 40000 (time 1736786798.73 total 73.25 last 548.04) - /unlink 45285 (time 1736786808.73 total 83.25 last 528.47) - /unlink 50000 (time 1736786817.11 total 91.63 last 562.86) - /unlink 55153 (time 1736786827.11 total 101.63 last 515.27) - /unlink 60000 (time 1736786836.93 total 111.44 last 493.71) total: 64768 /unlink in 120.60 seconds: 537.05 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2716 (time 1736786874.78 total 10.00 last 271.52) - open/close 5460 (time 1736786884.79 total 20.01 last 274.30) - open/close 8500 (time 1736786894.79 total 30.01 last 304.00) - open/close 10000 (time 1736786898.99 total 34.21 last 357.03) - open/close 13068 (time 1736786908.99 total 44.21 last 306.71) - open/close 16260 (time 1736786918.99 total 54.21 last 319.19) - open/close 19009 (time 1736786928.99 total 64.21 last 274.86) - open/close 20000 (time 1736786932.79 total 68.01 last 261.10) - open/close 22600 (time 1736786942.79 total 78.01 last 259.96) - open/close 25164 (time 1736786952.79 total 88.01 last 256.40) - open/close 27901 (time 1736786962.79 total 98.01 last 273.67) - open/close 30000 (time 1736786971.08 total 106.30 last 253.30) - open/close 32552 (time 1736786981.08 total 116.30 last 255.18) - open/close 35085 (time 1736786991.08 total 126.30 last 253.22) - open/close 37663 (time 1736787001.08 total 136.30 last 257.72) - open/close 40000 (time 1736787010.38 total 145.60 last 251.54) - open/close 42551 (time 1736787020.38 total 155.60 last 255.10) - open/close 45204 (time 1736787030.38 total 165.60 last 265.25) - open/close 48112 (time 1736787040.38 total 175.60 last 290.74) - open/close 50000 (time 1736787047.41 total 182.63 last 268.39) - open/close 52578 (time 1736787057.42 total 192.63 last 257.77) - open/close 55161 (time 1736787067.42 total 202.64 last 258.22) - open/close 57615 (time 1736787077.42 total 212.64 last 245.33) - open/close 60000 (time 1736787087.22 total 222.44 last 243.29) - open/close 62488 (time 1736787097.23 total 232.45 last 248.74) total: 64768 open/close in 241.76 seconds: 267.90 ops/second - /unlink 4782 (time 1736787117.74 total 10.00 last 478.14) - /unlink 9674 (time 1736787127.74 total 20.00 last 489.14) - /unlink 14458 (time 1736787137.74 total 30.00 last 478.31) - /unlink 19624 (time 1736787147.74 total 40.01 last 516.56) - /unlink 24324 (time 1736787157.74 total 50.01 last 469.99) - /unlink 29032 (time 1736787167.74 total 60.01 last 470.78) - /unlink 34011 (time 1736787177.75 total 70.01 last 497.84) - /unlink 39327 (time 1736787187.75 total 80.01 last 531.49) - /unlink 44410 (time 1736787197.75 total 90.01 last 508.28) - /unlink 49335 (time 1736787207.75 total 100.01 last 492.47) - /unlink 54404 (time 1736787217.75 total 110.01 last 506.80) - /unlink 59331 (time 1736787227.75 total 120.01 last 492.62) total: 64768 /unlink in 129.35 seconds: 500.73 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg357-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server unloading modules via unload_modules_local on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2