== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:52:32 (1743508352) Stopping clients: oleg436-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg436-client.virtnet /mnt/lustre2 (opts:-f) oleg436-server: oleg436-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg436-server' oleg436-server: oleg436-server.virtnet: executing load_modules_local oleg436-server: Loading modules from /home/green/git/lustre-release/lustre oleg436-server: detected 4 online CPUs by sysfs oleg436-server: MODOPTS_LIBCFS= oleg436-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg436-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg436-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4924 (time 1743508386.58 total 10.00 last 492.40) - open/close 9786 (time 1743508396.59 total 20.00 last 486.17) - open/close 14532 (time 1743508406.59 total 30.00 last 474.53) - open/close 19114 (time 1743508416.59 total 40.00 last 458.18) - open/close 23792 (time 1743508426.59 total 50.00 last 467.76) - open/close 28462 (time 1743508436.59 total 60.00 last 466.96) - open/close 30000 (time 1743508439.86 total 63.28 last 469.68) - open/close 34115 (time 1743508449.87 total 73.28 last 411.44) - open/close 38975 (time 1743508459.87 total 83.28 last 485.93) - open/close 43905 (time 1743508469.87 total 93.28 last 492.92) - open/close 48756 (time 1743508479.87 total 103.29 last 485.01) - open/close 50000 (time 1743508482.58 total 106.00 last 458.35) - open/close 54654 (time 1743508492.59 total 116.00 last 465.34) - open/close 59633 (time 1743508502.59 total 126.00 last 497.88) total: 64768 open/close in 134.59 seconds: 481.21 ops/second - /unlink 9761 (time 1743508521.97 total 10.00 last 976.07) - /unlink 18955 (time 1743508531.97 total 20.00 last 919.35) - /unlink 27650 (time 1743508541.98 total 30.00 last 869.46) - /unlink 30000 (time 1743508544.57 total 32.59 last 906.11) - /unlink 38834 (time 1743508554.57 total 42.59 last 883.39) - /unlink 47206 (time 1743508564.57 total 52.60 last 837.16) - /unlink 50000 (time 1743508567.56 total 55.59 last 933.87) - /unlink 58412 (time 1743508577.56 total 65.59 last 841.16) - /unlink 60000 (time 1743508579.82 total 67.84 last 704.62) total: 64768 /unlink in 73.50 seconds: 881.22 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4600 (time 1743508619.73 total 10.00 last 459.99) - open/close 8980 (time 1743508629.73 total 20.00 last 437.96) - open/close 10000 (time 1743508631.97 total 22.24 last 455.09) - open/close 14953 (time 1743508641.97 total 32.24 last 495.29) - open/close 20000 (time 1743508651.89 total 42.16 last 508.93) - open/close 24975 (time 1743508661.89 total 52.16 last 497.43) - open/close 30000 (time 1743508670.66 total 60.94 last 572.69) - open/close 35475 (time 1743508680.66 total 70.94 last 547.45) - open/close 40000 (time 1743508687.82 total 78.09 last 632.39) - open/close 44724 (time 1743508697.82 total 88.09 last 472.38) - open/close 49581 (time 1743508707.82 total 98.09 last 485.68) - open/close 55453 (time 1743508717.82 total 108.09 last 587.16) - open/close 60000 (time 1743508726.52 total 116.79 last 522.94) total: 64768 open/close in 125.31 seconds: 516.86 ops/second - /unlink 10000 (time 1743508745.50 total 9.68 last 1032.62) - /unlink 19843 (time 1743508755.50 total 19.68 last 984.29) - /unlink 30000 (time 1743508765.33 total 29.51 last 1033.75) - /unlink 39696 (time 1743508775.33 total 39.51 last 969.47) - /unlink 49329 (time 1743508785.33 total 49.51 last 963.27) - /unlink 59306 (time 1743508795.33 total 59.51 last 997.63) total: 64768 /unlink in 65.26 seconds: 992.48 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5764 (time 1743508844.97 total 10.00 last 576.37) - open/close 10000 (time 1743508852.31 total 17.33 last 577.73) - open/close 15241 (time 1743508862.31 total 27.33 last 524.03) - open/close 20000 (time 1743508870.93 total 35.96 last 551.98) - open/close 26003 (time 1743508880.93 total 45.96 last 600.29) - open/close 30000 (time 1743508887.83 total 52.86 last 578.82) - open/close 35880 (time 1743508897.84 total 62.86 last 587.98) - open/close 40000 (time 1743508907.26 total 72.28 last 437.27) - open/close 44693 (time 1743508917.26 total 82.29 last 469.24) - open/close 48695 (time 1743508927.26 total 92.29 last 400.10) - open/close 50000 (time 1743508930.22 total 95.24 last 441.68) - open/close 54013 (time 1743508940.22 total 105.24 last 401.26) - open/close 57986 (time 1743508950.22 total 115.24 last 397.27) - open/close 60000 (time 1743508954.72 total 119.75 last 446.88) - open/close 64144 (time 1743508964.73 total 129.75 last 414.35) total: 64768 open/close in 131.16 seconds: 493.82 ops/second - /unlink 8173 (time 1743508976.96 total 10.00 last 817.15) - /unlink 10000 (time 1743508979.71 total 12.76 last 663.27) - /unlink 16058 (time 1743508989.71 total 22.76 last 605.72) - /unlink 20000 (time 1743508994.66 total 27.71 last 796.28) - /unlink 27126 (time 1743509004.67 total 37.71 last 712.44) - /unlink 30000 (time 1743509008.23 total 41.28 last 806.24) - /unlink 38497 (time 1743509018.23 total 51.28 last 849.65) - /unlink 40000 (time 1743509020.44 total 53.48 last 680.40) - /unlink 47117 (time 1743509030.44 total 63.49 last 711.65) - /unlink 50000 (time 1743509034.28 total 67.32 last 751.78) - /unlink 56407 (time 1743509044.28 total 77.32 last 640.66) - /unlink 60000 (time 1743509050.02 total 83.06 last 625.67) total: 64768 /unlink in 88.72 seconds: 730.01 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3556 (time 1743509086.12 total 10.00 last 355.58) - open/close 7682 (time 1743509096.12 total 20.00 last 412.50) - open/close 10000 (time 1743509101.43 total 25.31 last 436.57) - open/close 14787 (time 1743509111.43 total 35.32 last 478.57) - open/close 18863 (time 1743509121.43 total 45.32 last 407.59) - open/close 20000 (time 1743509124.48 total 48.37 last 372.75) - open/close 23595 (time 1743509134.49 total 58.37 last 359.45) - open/close 27607 (time 1743509144.49 total 68.37 last 401.19) - open/close 30000 (time 1743509150.47 total 74.35 last 400.01) - open/close 34073 (time 1743509160.47 total 84.35 last 407.25) - open/close 38045 (time 1743509170.47 total 94.35 last 397.16) - open/close 40000 (time 1743509175.61 total 99.49 last 380.38) - open/close 44468 (time 1743509185.61 total 109.49 last 446.75) - open/close 49059 (time 1743509195.61 total 119.49 last 459.06) - open/close 50000 (time 1743509197.81 total 121.70 last 427.34) - open/close 53879 (time 1743509207.82 total 131.70 last 387.85) - open/close 57842 (time 1743509217.82 total 141.70 last 396.21) - open/close 60000 (time 1743509224.53 total 148.41 last 321.61) - open/close 64099 (time 1743509234.53 total 158.41 last 409.84) total: 64768 open/close in 160.05 seconds: 404.66 ops/second - /unlink 6844 (time 1743509247.07 total 10.00 last 684.36) - /unlink 10000 (time 1743509251.03 total 13.96 last 797.05) - /unlink 17177 (time 1743509261.03 total 23.96 last 717.58) - /unlink 20000 (time 1743509265.01 total 27.94 last 708.95) - /unlink 27976 (time 1743509275.01 total 37.94 last 797.59) - /unlink 30000 (time 1743509277.77 total 40.70 last 733.31) - /unlink 37983 (time 1743509287.77 total 50.70 last 798.28) - /unlink 40000 (time 1743509290.26 total 53.19 last 810.90) - /unlink 48269 (time 1743509300.26 total 63.19 last 826.88) - /unlink 50000 (time 1743509302.26 total 65.19 last 865.01) - /unlink 57386 (time 1743509312.26 total 75.19 last 738.53) - /unlink 60000 (time 1743509315.67 total 78.60 last 768.53) total: 64768 /unlink in 84.21 seconds: 769.13 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4148 (time 1743509351.75 total 10.00 last 414.77) - open/close 8546 (time 1743509361.75 total 20.00 last 439.80) - open/close 10000 (time 1743509364.87 total 23.12 last 466.46) - open/close 14825 (time 1743509374.87 total 33.12 last 482.43) - open/close 19011 (time 1743509384.87 total 43.12 last 418.53) - open/close 20000 (time 1743509387.01 total 45.26 last 462.27) - open/close 24470 (time 1743509397.01 total 55.26 last 446.98) - open/close 29038 (time 1743509407.02 total 65.26 last 456.75) - open/close 30000 (time 1743509409.03 total 67.28 last 477.61) - open/close 35022 (time 1743509419.03 total 77.28 last 502.18) - open/close 39568 (time 1743509429.03 total 87.28 last 454.51) - open/close 44262 (time 1743509439.03 total 97.28 last 469.38) - open/close 48850 (time 1743509449.03 total 107.28 last 458.78) - open/close 50000 (time 1743509451.62 total 109.87 last 444.25) - open/close 54652 (time 1743509461.62 total 119.87 last 465.11) - open/close 59551 (time 1743509471.63 total 129.87 last 489.83) - open/close 63941 (time 1743509481.63 total 139.87 last 438.95) total: 64768 open/close in 141.68 seconds: 457.13 ops/second - /unlink 8303 (time 1743509494.35 total 10.00 last 830.27) - /unlink 10000 (time 1743509496.57 total 12.22 last 766.12) - /unlink 18061 (time 1743509506.57 total 22.22 last 806.06) - /unlink 20000 (time 1743509508.87 total 24.52 last 841.19) - /unlink 28048 (time 1743509518.87 total 34.52 last 804.74) - /unlink 30000 (time 1743509521.45 total 37.10 last 756.94) - /unlink 38573 (time 1743509531.45 total 47.10 last 857.17) - /unlink 46677 (time 1743509541.46 total 57.10 last 810.31) - /unlink 50000 (time 1743509545.64 total 61.28 last 794.72) - /unlink 57967 (time 1743509555.64 total 71.29 last 796.62) - /unlink 60000 (time 1743509558.04 total 73.69 last 846.54) total: 64768 /unlink in 79.78 seconds: 811.83 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg436-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg436-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg436-server stop mds service on oleg436-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg436-server unloading modules via unload_modules_local on: 'oleg436-server' oleg436-server: oleg436-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg436-client: oleg436-client: ssh exited with exit code 2 pdsh@oleg436-client: oleg436-server: ssh exited with exit code 2 pdsh@oleg436-client: oleg436-server: ssh exited with exit code 2 pdsh@oleg436-client: oleg436-client: ssh exited with exit code 2