== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:29:36 (1734258576) Stopping clients: oleg419-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg419-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 oleg419-server: oleg419-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: detected 4 online CPUs by sysfs oleg419-server: Force libcfs to create 2 CPU partitions oleg419-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg419-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg419-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg419-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg419-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16266 458 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16451 643 15808 4% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3483 (time 1734258637.18 total 10.00 last 348.22) - open/close 6242 (time 1734258647.18 total 20.00 last 275.84) - open/close 9202 (time 1734258657.19 total 30.01 last 295.87) - open/close 10000 (time 1734258659.57 total 32.39 last 335.24) - open/close 13358 (time 1734258669.57 total 42.39 last 335.74) - open/close 16939 (time 1734258679.57 total 52.39 last 358.09) - open/close 19872 (time 1734258689.57 total 62.39 last 293.24) - open/close 22849 (time 1734258699.57 total 72.39 last 297.68) - open/close 25354 (time 1734258709.58 total 82.40 last 250.43) - open/close 27748 (time 1734258719.58 total 92.40 last 239.36) - open/close 30000 (time 1734258727.16 total 99.98 last 297.12) - open/close 33287 (time 1734258737.16 total 109.98 last 328.62) - open/close 36811 (time 1734258747.16 total 119.98 last 352.40) - open/close 40000 (time 1734258756.77 total 129.59 last 331.79) - open/close 43602 (time 1734258766.77 total 139.59 last 360.16) - open/close 46709 (time 1734258776.77 total 149.59 last 310.68) - open/close 50000 (time 1734258786.55 total 159.37 last 336.54) - open/close 52730 (time 1734258796.56 total 169.38 last 272.91) - open/close 55402 (time 1734258806.56 total 179.38 last 267.13) - open/close 58007 (time 1734258816.56 total 189.38 last 260.44) - open/close 60000 (time 1734258823.90 total 196.72 last 271.46) - open/close 63258 (time 1734258833.90 total 206.72 last 325.75) total: 64768 open/close in 214.13 seconds: 302.48 ops/second - /unlink 3990 (time 1734258852.55 total 10.00 last 398.93) - /unlink 8412 (time 1734258862.56 total 20.00 last 442.08) - /unlink 10000 (time 1734258865.53 total 22.97 last 534.70) - /unlink 15083 (time 1734258875.53 total 32.98 last 508.22) - /unlink 20000 (time 1734258884.13 total 41.57 last 572.00) - /unlink 25118 (time 1734258894.13 total 51.57 last 511.78) - /unlink 29921 (time 1734258904.13 total 61.57 last 480.25) - /unlink 35129 (time 1734258914.13 total 71.58 last 520.66) - /unlink 39593 (time 1734258924.13 total 81.58 last 446.34) - /unlink 46590 (time 1734258934.13 total 91.58 last 699.67) - /unlink 50000 (time 1734258938.50 total 95.95 last 780.04) - /unlink 55725 (time 1734258948.50 total 105.95 last 572.47) - /unlink 60000 (time 1734258955.63 total 113.07 last 600.17) total: 64768 /unlink in 120.82 seconds: 536.07 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3137 (time 1734258990.05 total 10.00 last 313.69) - open/close 6217 (time 1734259000.05 total 20.00 last 307.98) - open/close 9610 (time 1734259010.05 total 30.00 last 339.24) - open/close 12863 (time 1734259020.06 total 40.01 last 325.14) - open/close 16035 (time 1734259030.06 total 50.01 last 317.15) - open/close 18955 (time 1734259040.06 total 60.01 last 291.98) - open/close 20000 (time 1734259043.53 total 63.48 last 301.39) - open/close 22873 (time 1734259053.53 total 73.48 last 287.23) - open/close 25618 (time 1734259063.53 total 83.48 last 274.45) - open/close 28981 (time 1734259073.53 total 93.48 last 336.26) - open/close 30000 (time 1734259076.22 total 96.17 last 379.37) - open/close 33482 (time 1734259086.22 total 106.17 last 348.15) - open/close 36160 (time 1734259096.22 total 116.17 last 267.75) - open/close 39173 (time 1734259106.22 total 126.17 last 301.27) - open/close 40000 (time 1734259108.43 total 128.38 last 374.10) - open/close 43467 (time 1734259118.43 total 138.39 last 346.63) - open/close 47128 (time 1734259128.44 total 148.39 last 366.04) - open/close 50000 (time 1734259135.57 total 155.52 last 402.64) - open/close 53854 (time 1734259145.57 total 165.52 last 385.39) - open/close 58316 (time 1734259155.57 total 175.52 last 446.14) - open/close 60000 (time 1734259159.65 total 179.60 last 413.04) - open/close 64565 (time 1734259169.65 total 189.60 last 456.46) total: 64768 open/close in 190.05 seconds: 340.80 ops/second - /unlink 7460 (time 1734259180.94 total 10.00 last 745.94) - /unlink 10000 (time 1734259184.33 total 13.38 last 750.89) - /unlink 18369 (time 1734259194.33 total 23.38 last 836.85) - /unlink 27391 (time 1734259204.33 total 33.38 last 902.12) - /unlink 30000 (time 1734259207.13 total 36.18 last 932.32) - /unlink 38408 (time 1734259217.13 total 46.18 last 840.75) - /unlink 47973 (time 1734259227.13 total 56.18 last 956.42) - /unlink 50000 (time 1734259229.16 total 58.22 last 997.67) - /unlink 58725 (time 1734259239.16 total 68.22 last 872.42) total: 64768 /unlink in 75.37 seconds: 859.37 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5327 (time 1734259281.04 total 10.00 last 532.69) - open/close 10000 (time 1734259290.34 total 19.29 last 502.98) - open/close 15280 (time 1734259300.34 total 29.29 last 527.96) - open/close 20000 (time 1734259308.35 total 37.31 last 588.70) - open/close 25544 (time 1734259318.35 total 47.31 last 554.34) - open/close 30000 (time 1734259326.27 total 55.22 last 563.06) - open/close 35714 (time 1734259336.27 total 65.23 last 571.31) - open/close 40000 (time 1734259344.69 total 73.65 last 508.76) - open/close 44510 (time 1734259354.69 total 83.65 last 450.99) - open/close 49404 (time 1734259364.70 total 93.65 last 489.32) - open/close 54515 (time 1734259374.70 total 103.65 last 511.06) - open/close 58686 (time 1734259384.70 total 113.65 last 417.03) - open/close 60000 (time 1734259388.06 total 117.01 last 391.32) - open/close 64178 (time 1734259398.06 total 127.01 last 417.74) total: 64768 open/close in 128.45 seconds: 504.24 ops/second - /unlink 7237 (time 1734259410.38 total 10.00 last 723.59) - /unlink 10000 (time 1734259413.95 total 13.57 last 773.60) - /unlink 17288 (time 1734259423.95 total 23.57 last 728.75) - /unlink 20000 (time 1734259427.27 total 26.90 last 816.26) - /unlink 27922 (time 1734259437.27 total 36.90 last 792.13) - /unlink 30000 (time 1734259439.80 total 39.42 last 822.30) - /unlink 37576 (time 1734259449.80 total 49.42 last 757.58) - /unlink 40000 (time 1734259452.52 total 52.15 last 889.60) - /unlink 48064 (time 1734259462.52 total 62.15 last 806.32) - /unlink 50000 (time 1734259465.06 total 64.69 last 763.41) - /unlink 57882 (time 1734259475.06 total 74.69 last 788.09) - /unlink 60000 (time 1734259477.57 total 77.20 last 843.16) total: 64768 /unlink in 82.84 seconds: 781.85 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5685 (time 1734259515.89 total 10.00 last 568.43) - open/close 10000 (time 1734259523.67 total 17.78 last 554.39) - open/close 15415 (time 1734259533.67 total 27.79 last 541.43) - open/close 20000 (time 1734259541.64 total 35.76 last 575.20) - open/close 25636 (time 1734259551.66 total 45.77 last 562.96) - open/close 30000 (time 1734259559.43 total 53.54 last 561.27) - open/close 35885 (time 1734259569.43 total 63.54 last 588.43) - open/close 40000 (time 1734259576.61 total 70.72 last 573.34) - open/close 45329 (time 1734259586.61 total 80.72 last 532.85) - open/close 50000 (time 1734259595.42 total 89.53 last 530.23) - open/close 55616 (time 1734259605.42 total 99.53 last 561.53) - open/close 60000 (time 1734259614.21 total 108.32 last 498.94) - open/close 64652 (time 1734259624.21 total 118.32 last 465.16) total: 64768 open/close in 118.55 seconds: 546.35 ops/second - /unlink 8088 (time 1734259635.28 total 10.00 last 808.73) - /unlink 17356 (time 1734259645.28 total 20.00 last 926.77) - /unlink 20000 (time 1734259648.25 total 22.96 last 892.66) - /unlink 28592 (time 1734259658.25 total 32.96 last 859.18) - /unlink 36850 (time 1734259668.25 total 42.96 last 825.75) - /unlink 40000 (time 1734259672.08 total 46.80 last 822.03) - /unlink 47724 (time 1734259682.08 total 56.80 last 772.31) - /unlink 50000 (time 1734259685.31 total 60.02 last 705.27) - /unlink 57885 (time 1734259695.31 total 70.02 last 788.44) - /unlink 60000 (time 1734259697.75 total 72.47 last 865.60) total: 64768 /unlink in 78.18 seconds: 828.43 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5003 (time 1734259736.44 total 10.00 last 500.23) - open/close 10000 (time 1734259745.09 total 18.65 last 577.95) - open/close 16122 (time 1734259755.09 total 28.65 last 612.09) - open/close 20000 (time 1734259762.53 total 36.09 last 521.52) - open/close 25009 (time 1734259772.53 total 46.09 last 500.80) - open/close 29751 (time 1734259782.53 total 56.09 last 474.15) - open/close 34780 (time 1734259792.53 total 66.09 last 502.86) - open/close 40000 (time 1734259802.43 total 75.99 last 527.35) - open/close 45276 (time 1734259812.43 total 85.99 last 527.49) - open/close 50000 (time 1734259822.04 total 95.60 last 491.74) - open/close 54975 (time 1734259832.04 total 105.60 last 497.41) - open/close 59841 (time 1734259842.04 total 115.60 last 486.59) total: 64768 open/close in 124.70 seconds: 519.41 ops/second - /unlink 8664 (time 1734259861.98 total 10.00 last 866.34) - /unlink 17234 (time 1734259871.98 total 20.00 last 856.93) - /unlink 20000 (time 1734259875.21 total 23.24 last 855.20) - /unlink 28743 (time 1734259885.21 total 33.24 last 874.28) - /unlink 37256 (time 1734259895.21 total 43.24 last 851.25) - /unlink 40000 (time 1734259898.39 total 46.41 last 865.04) - /unlink 48770 (time 1734259908.39 total 56.41 last 876.97) - /unlink 57450 (time 1734259918.39 total 66.41 last 867.99) - /unlink 60000 (time 1734259921.24 total 69.27 last 892.84) total: 64768 /unlink in 74.75 seconds: 866.42 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg419-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg419-server unloading modules via unload_modules_local on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2