== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 00:08:47 (1786939727) start mds service on oleg425-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg425-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-OST0000 oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92553 329 92224 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12329 425 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12233 329 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 810 (time 1786939762.27 total 10.00 last 80.99) - open/close/unlink 1557 (time 1786939772.28 total 20.01 last 74.64) - open/close/unlink 2193 (time 1786939782.28 total 30.01 last 63.59) - open/close/unlink 2873 (time 1786939792.29 total 40.02 last 67.94) - open/close/unlink 3711 (time 1786939802.30 total 50.03 last 83.74) - open/close/unlink 4608 (time 1786939812.30 total 60.03 last 89.69) - open/close/unlink 5509 (time 1786939822.30 total 70.03 last 90.08) - open/close/unlink 6607 (time 1786939832.31 total 80.04 last 109.74) - open/close/unlink 7741 (time 1786939842.32 total 90.04 last 113.32) - open/close/unlink 8775 (time 1786939852.32 total 100.05 last 103.37) - open/close/unlink 9892 (time 1786939862.32 total 110.05 last 111.69) - open/close/unlink 10886 (time 1786939872.33 total 120.06 last 99.30) - open/close/unlink 12056 (time 1786939882.33 total 130.06 last 116.98) - open/close/unlink 13191 (time 1786939892.34 total 140.06 last 113.44) - open/close/unlink 14520 (time 1786939902.34 total 150.07 last 132.86) - open/close/unlink 15902 (time 1786939912.34 total 160.07 last 138.14) - open/close/unlink 17150 (time 1786939922.35 total 170.07 last 124.76) - open/close/unlink 18373 (time 1786939932.36 total 180.08 last 122.17) - open/close/unlink 19825 (time 1786939942.36 total 190.09 last 145.16) - open/close/unlink 21268 (time 1786939952.36 total 200.09 last 144.23) - open/close/unlink 22875 (time 1786939962.37 total 210.09 last 160.65) - open/close/unlink 24310 (time 1786939972.37 total 220.10 last 143.47) - open/close/unlink 25922 (time 1786939982.37 total 230.10 last 161.13) - open/close/unlink 27477 (time 1786939992.38 total 240.10 last 155.47) - open/close/unlink 28932 (time 1786940002.38 total 250.11 last 145.41) - open/close/unlink 30000 (time 1786940009.44 total 257.17 last 151.27) - open/close/unlink 31714 (time 1786940019.44 total 267.17 last 171.39) - open/close/unlink 33436 (time 1786940029.45 total 277.18 last 172.08) - open/close/unlink 34935 (time 1786940039.46 total 287.19 last 149.79) - open/close/unlink 36662 (time 1786940049.46 total 297.19 last 172.67) - open/close/unlink 38264 (time 1786940059.46 total 307.19 last 160.20) - open/close/unlink 39934 (time 1786940069.46 total 317.19 last 166.92) - open/close/unlink 41563 (time 1786940079.47 total 327.19 last 162.86) - open/close/unlink 43360 (time 1786940089.47 total 337.20 last 179.69) - open/close/unlink 44884 (time 1786940099.48 total 347.20 last 152.28) - open/close/unlink 46104 (time 1786940109.48 total 357.21 last 121.93) - open/close/unlink 47707 (time 1786940119.48 total 367.21 last 160.26) - open/close/unlink 49337 (time 1786940129.48 total 377.21 last 162.99) - open/close/unlink 50000 (time 1786940134.10 total 381.82 last 143.73) - open/close/unlink 51384 (time 1786940144.10 total 391.83 last 138.36) - open/close/unlink 52911 (time 1786940154.10 total 401.83 last 152.70) - open/close/unlink 54248 (time 1786940164.10 total 411.83 last 133.65) - open/close/unlink 55618 (time 1786940174.11 total 421.83 last 136.95) - open/close/unlink 56952 (time 1786940184.11 total 431.84 last 133.31) - open/close/unlink 58209 (time 1786940194.13 total 441.85 last 125.54) - open/close/unlink 59754 (time 1786940204.14 total 451.87 last 154.30) - open/close/unlink 61612 (time 1786940214.15 total 461.88 last 185.64) - open/close/unlink 63427 (time 1786940224.16 total 471.89 last 181.24) total: 64768 open/close/unlink in 479.87 seconds: 134.97 ops/second - open/close/unlink 803 (time 1786939762.06 total 10.01 last 80.22) - open/close/unlink 1556 (time 1786939772.07 total 20.02 last 75.22) - open/close/unlink 2200 (time 1786939782.08 total 30.02 last 64.37) - open/close/unlink 2879 (time 1786939792.09 total 40.03 last 67.83) - open/close/unlink 3700 (time 1786939802.09 total 50.04 last 82.09) - open/close/unlink 4594 (time 1786939812.09 total 60.04 last 89.38) - open/close/unlink 5488 (time 1786939822.09 total 70.04 last 89.36) - open/close/unlink 6593 (time 1786939832.10 total 80.05 last 110.44) - open/close/unlink 7748 (time 1786939842.11 total 90.06 last 115.41) - open/close/unlink 8779 (time 1786939852.12 total 100.07 last 102.95) - open/close/unlink 9891 (time 1786939862.12 total 110.07 last 111.19) - open/close/unlink 10887 (time 1786939872.12 total 120.07 last 99.60) - open/close/unlink 12060 (time 1786939882.13 total 130.08 last 117.23) - open/close/unlink 13181 (time 1786939892.13 total 140.08 last 112.07) - open/close/unlink 14506 (time 1786939902.14 total 150.09 last 132.42) - open/close/unlink 15881 (time 1786939912.15 total 160.09 last 137.42) - open/close/unlink 17132 (time 1786939922.15 total 170.10 last 124.99) - open/close/unlink 18355 (time 1786939932.17 total 180.11 last 122.16) - open/close/unlink 19809 (time 1786939942.17 total 190.12 last 145.37) - open/close/unlink 21242 (time 1786939952.17 total 200.12 last 143.24) - open/close/unlink 22852 (time 1786939962.17 total 210.12 last 160.98) - open/close/unlink 24274 (time 1786939972.18 total 220.13 last 142.11) - open/close/unlink 25875 (time 1786939982.18 total 230.13 last 160.07) - open/close/unlink 27435 (time 1786939992.19 total 240.13 last 155.95) - open/close/unlink 28915 (time 1786940002.19 total 250.14 last 147.91) - open/close/unlink 30000 (time 1786940009.57 total 257.52 last 146.96) - open/close/unlink 31713 (time 1786940019.58 total 267.53 last 171.24) - open/close/unlink 33448 (time 1786940029.58 total 277.53 last 173.48) - open/close/unlink 34947 (time 1786940039.61 total 287.55 last 149.51) - open/close/unlink 36676 (time 1786940049.61 total 297.56 last 172.83) - open/close/unlink 38265 (time 1786940059.61 total 307.56 last 158.85) - open/close/unlink 39935 (time 1786940069.62 total 317.57 last 166.91) - open/close/unlink 41554 (time 1786940079.62 total 327.57 last 161.84) - open/close/unlink 43358 (time 1786940089.63 total 337.57 last 180.33) - open/close/unlink 44857 (time 1786940099.63 total 347.58 last 149.85) - open/close/unlink 46067 (time 1786940109.63 total 357.58 last 120.99) - open/close/unlink 47686 (time 1786940119.63 total 367.58 last 161.84) - open/close/unlink 49293 (time 1786940129.64 total 377.59 last 160.63) - open/close/unlink 50000 (time 1786940134.53 total 382.47 last 144.62) - open/close/unlink 51382 (time 1786940144.53 total 392.48 last 138.18) - open/close/unlink 52908 (time 1786940154.53 total 402.48 last 152.53) - open/close/unlink 54255 (time 1786940164.53 total 412.48 last 134.69) - open/close/unlink 55607 (time 1786940174.54 total 422.48 last 135.16) - open/close/unlink 56951 (time 1786940184.54 total 432.49 last 134.30) - open/close/unlink 58228 (time 1786940194.55 total 442.50 last 127.65) - open/close/unlink 59730 (time 1786940204.55 total 452.50 last 150.16) - open/close/unlink 61570 (time 1786940214.55 total 462.50 last 183.99) - open/close/unlink 63381 (time 1786940224.56 total 472.50 last 181.01) total: 64768 open/close/unlink in 481.02 seconds: 134.65 ops/second - open/close/unlink 795 (time 1786939762.06 total 10.01 last 79.46) - open/close/unlink 1552 (time 1786939772.07 total 20.02 last 75.59) - open/close/unlink 2201 (time 1786939782.08 total 30.03 last 64.86) - open/close/unlink 2881 (time 1786939792.09 total 40.04 last 67.91) - open/close/unlink 3709 (time 1786939802.10 total 50.05 last 82.74) - open/close/unlink 4610 (time 1786939812.10 total 60.05 last 90.06) - open/close/unlink 5507 (time 1786939822.11 total 70.05 last 89.69) - open/close/unlink 6609 (time 1786939832.11 total 80.06 last 110.17) - open/close/unlink 7743 (time 1786939842.11 total 90.06 last 113.35) - open/close/unlink 8769 (time 1786939852.13 total 100.07 last 102.47) - open/close/unlink 9877 (time 1786939862.13 total 110.08 last 110.76) - open/close/unlink 10875 (time 1786939872.13 total 120.08 last 99.77) - open/close/unlink 12030 (time 1786939882.14 total 130.08 last 115.44) - open/close/unlink 13151 (time 1786939892.14 total 140.09 last 112.04) - open/close/unlink 14471 (time 1786939902.14 total 150.09 last 131.99) - open/close/unlink 15850 (time 1786939912.15 total 160.10 last 137.83) - open/close/unlink 17106 (time 1786939922.15 total 170.10 last 125.53) - open/close/unlink 18334 (time 1786939932.16 total 180.11 last 122.71) - open/close/unlink 19787 (time 1786939942.16 total 190.11 last 145.29) - open/close/unlink 21222 (time 1786939952.17 total 200.11 last 143.45) - open/close/unlink 22826 (time 1786939962.17 total 210.12 last 160.31) - open/close/unlink 24264 (time 1786939972.18 total 220.12 last 143.72) - open/close/unlink 25865 (time 1786939982.18 total 230.13 last 160.05) - open/close/unlink 27421 (time 1786939992.18 total 240.13 last 155.55) - open/close/unlink 28889 (time 1786940002.19 total 250.13 last 146.73) - open/close/unlink 30000 (time 1786940009.76 total 257.70 last 146.80) - open/close/unlink 31710 (time 1786940019.76 total 267.70 last 170.98) - open/close/unlink 33445 (time 1786940029.76 total 277.71 last 173.38) - open/close/unlink 34940 (time 1786940039.76 total 287.71 last 149.49) - open/close/unlink 36681 (time 1786940049.77 total 297.71 last 174.05) - open/close/unlink 38274 (time 1786940059.77 total 307.72 last 159.27) - open/close/unlink 39942 (time 1786940069.77 total 317.72 last 166.71) - open/close/unlink 41571 (time 1786940079.77 total 327.72 last 162.90) - open/close/unlink 43352 (time 1786940089.78 total 337.72 last 178.09) - open/close/unlink 44858 (time 1786940099.78 total 347.72 last 150.58) - open/close/unlink 46073 (time 1786940109.78 total 357.73 last 121.47) - open/close/unlink 47694 (time 1786940119.78 total 367.73 last 162.08) - open/close/unlink 49296 (time 1786940129.78 total 377.73 last 160.18) - open/close/unlink 50000 (time 1786940134.70 total 382.65 last 143.04) - open/close/unlink 51383 (time 1786940144.71 total 392.65 last 138.26) - open/close/unlink 52900 (time 1786940154.71 total 402.66 last 151.63) - open/close/unlink 54234 (time 1786940164.72 total 412.66 last 133.35) - open/close/unlink 55592 (time 1786940174.72 total 422.67 last 135.70) - open/close/unlink 56931 (time 1786940184.72 total 432.67 last 133.89) - open/close/unlink 58197 (time 1786940194.73 total 442.67 last 126.57) - open/close/unlink 59726 (time 1786940204.73 total 452.67 last 152.86) - open/close/unlink 61571 (time 1786940214.73 total 462.68 last 184.42) - open/close/unlink 63376 (time 1786940224.73 total 472.68 last 180.49) total: 64768 open/close/unlink in 481.49 seconds: 134.52 ops/second - open/close/unlink 813 (time 1786939762.34 total 10.01 last 81.25) - open/close/unlink 1560 (time 1786939772.34 total 20.01 last 74.69) - open/close/unlink 2198 (time 1786939782.36 total 30.02 last 63.71) - open/close/unlink 2874 (time 1786939792.37 total 40.03 last 67.53) - open/close/unlink 3702 (time 1786939802.37 total 50.04 last 82.74) - open/close/unlink 4605 (time 1786939812.38 total 60.04 last 90.28) - open/close/unlink 5513 (time 1786939822.38 total 70.05 last 90.76) - open/close/unlink 6604 (time 1786939832.38 total 80.05 last 109.08) - open/close/unlink 7747 (time 1786939842.39 total 90.05 last 114.27) - open/close/unlink 8765 (time 1786939852.39 total 100.05 last 101.79) - open/close/unlink 9891 (time 1786939862.39 total 110.05 last 112.59) - open/close/unlink 10870 (time 1786939872.39 total 120.05 last 97.89) - open/close/unlink 12044 (time 1786939882.39 total 130.06 last 117.34) - open/close/unlink 13167 (time 1786939892.40 total 140.06 last 112.28) - open/close/unlink 14504 (time 1786939902.40 total 150.06 last 133.63) - open/close/unlink 15866 (time 1786939912.42 total 160.08 last 135.99) - open/close/unlink 17112 (time 1786939922.42 total 170.08 last 124.55) - open/close/unlink 18334 (time 1786939932.44 total 180.10 last 121.96) - open/close/unlink 19785 (time 1786939942.44 total 190.11 last 145.07) - open/close/unlink 21234 (time 1786939952.44 total 200.11 last 144.89) - open/close/unlink 22841 (time 1786939962.44 total 210.11 last 160.67) - open/close/unlink 24263 (time 1786939972.45 total 220.11 last 142.13) - open/close/unlink 25886 (time 1786939982.45 total 230.12 last 162.28) - open/close/unlink 27437 (time 1786939992.45 total 240.12 last 155.07) - open/close/unlink 28893 (time 1786940002.46 total 250.12 last 145.51) - open/close/unlink 30000 (time 1786940009.96 total 257.62 last 147.57) - open/close/unlink 31681 (time 1786940019.96 total 267.63 last 168.07) - open/close/unlink 33411 (time 1786940029.97 total 277.64 last 172.79) - open/close/unlink 34910 (time 1786940039.98 total 287.64 last 149.88) - open/close/unlink 36654 (time 1786940049.98 total 297.65 last 174.27) - open/close/unlink 38266 (time 1786940059.99 total 307.66 last 161.08) - open/close/unlink 39929 (time 1786940070.00 total 317.66 last 166.20) - open/close/unlink 41562 (time 1786940080.00 total 327.66 last 163.26) - open/close/unlink 43365 (time 1786940090.00 total 337.67 last 180.27) - open/close/unlink 44846 (time 1786940100.00 total 347.67 last 148.05) - open/close/unlink 46077 (time 1786940110.01 total 357.67 last 123.04) - open/close/unlink 47707 (time 1786940120.02 total 367.68 last 162.89) - open/close/unlink 49343 (time 1786940130.02 total 377.68 last 163.58) - open/close/unlink 50000 (time 1786940134.65 total 382.32 last 141.71) - open/close/unlink 51392 (time 1786940144.65 total 392.32 last 139.19) - open/close/unlink 52907 (time 1786940154.66 total 402.32 last 151.48) - open/close/unlink 54242 (time 1786940164.66 total 412.32 last 133.47) - open/close/unlink 55586 (time 1786940174.66 total 422.32 last 134.39) - open/close/unlink 56936 (time 1786940184.66 total 432.33 last 134.94) - open/close/unlink 58201 (time 1786940194.67 total 442.33 last 126.42) - open/close/unlink 59723 (time 1786940204.67 total 452.34 last 152.16) - open/close/unlink 61558 (time 1786940214.68 total 462.35 last 183.33) - open/close/unlink 63367 (time 1786940224.68 total 472.35 last 180.86) total: 64768 open/close/unlink in 481.21 seconds: 134.59 ops/second - open/close/unlink 808 (time 1786939762.16 total 10.00 last 80.78) - open/close/unlink 1552 (time 1786939772.16 total 20.00 last 74.38) - open/close/unlink 2195 (time 1786939782.16 total 30.01 last 64.27) - open/close/unlink 2870 (time 1786939792.17 total 40.01 last 67.47) - open/close/unlink 3703 (time 1786939802.18 total 50.02 last 83.21) - open/close/unlink 4605 (time 1786939812.18 total 60.03 last 90.18) - open/close/unlink 5500 (time 1786939822.18 total 70.03 last 89.49) - open/close/unlink 6598 (time 1786939832.18 total 80.03 last 109.78) - open/close/unlink 7745 (time 1786939842.19 total 90.03 last 114.68) - open/close/unlink 8771 (time 1786939852.19 total 100.03 last 102.57) - open/close/unlink 9885 (time 1786939862.19 total 110.04 last 111.38) - open/close/unlink 10889 (time 1786939872.19 total 120.04 last 100.35) - open/close/unlink 12049 (time 1786939882.20 total 130.05 last 115.94) - open/close/unlink 13168 (time 1786939892.21 total 140.05 last 111.82) - open/close/unlink 14488 (time 1786939902.21 total 150.06 last 131.95) - open/close/unlink 15868 (time 1786939912.21 total 160.06 last 137.98) - open/close/unlink 17120 (time 1786939922.22 total 170.06 last 125.14) - open/close/unlink 18352 (time 1786939932.22 total 180.07 last 123.13) - open/close/unlink 19799 (time 1786939942.23 total 190.07 last 144.67) - open/close/unlink 21227 (time 1786939952.23 total 200.07 last 142.76) - open/close/unlink 22826 (time 1786939962.23 total 210.08 last 159.83) - open/close/unlink 24247 (time 1786939972.23 total 220.08 last 142.08) - open/close/unlink 25842 (time 1786939982.24 total 230.08 last 159.47) - open/close/unlink 27401 (time 1786939992.24 total 240.08 last 155.87) - open/close/unlink 28870 (time 1786940002.25 total 250.09 last 146.74) - open/close/unlink 30000 (time 1786940009.94 total 257.78 last 146.96) - open/close/unlink 31691 (time 1786940019.94 total 267.78 last 169.09) - open/close/unlink 33418 (time 1786940029.95 total 277.79 last 172.57) - open/close/unlink 34925 (time 1786940039.95 total 287.80 last 150.61) - open/close/unlink 36675 (time 1786940049.96 total 297.80 last 174.92) - open/close/unlink 38291 (time 1786940059.96 total 307.81 last 161.49) - open/close/unlink 39959 (time 1786940069.97 total 317.81 last 166.69) - open/close/unlink 41602 (time 1786940079.97 total 327.81 last 164.30) - open/close/unlink 43388 (time 1786940089.97 total 337.82 last 178.58) - open/close/unlink 44860 (time 1786940099.98 total 347.82 last 147.13) - open/close/unlink 46078 (time 1786940109.98 total 357.82 last 121.79) - open/close/unlink 47692 (time 1786940119.98 total 367.83 last 161.32) - open/close/unlink 49297 (time 1786940129.98 total 377.83 last 160.47) - open/close/unlink 50000 (time 1786940134.88 total 382.73 last 143.49) - open/close/unlink 51392 (time 1786940144.89 total 392.73 last 139.16) - open/close/unlink 52904 (time 1786940154.89 total 402.73 last 151.16) - open/close/unlink 54245 (time 1786940164.89 total 412.74 last 134.07) - open/close/unlink 55581 (time 1786940174.90 total 422.74 last 133.54) - open/close/unlink 56918 (time 1786940184.90 total 432.74 last 133.65) - open/close/unlink 58187 (time 1786940194.90 total 442.75 last 126.85) - open/close/unlink 59696 (time 1786940204.91 total 452.75 last 150.86) - open/close/unlink 61534 (time 1786940214.91 total 462.75 last 183.74) - open/close/unlink 63325 (time 1786940224.92 total 472.76 last 178.96) - open/close/unlink 64749 (time 1786940234.93 total 482.77 last 142.22) total: 64768 open/close/unlink in 483.20 seconds: 134.04 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete oleg425-server: cannot open 'lustre-ost2': no such pool pdsh@oleg425-client: oleg425-server: ssh exited with exit code 255 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg425-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg425-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg425-server stop mds service on oleg425-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg425-server unloading modules via unload_modules_local on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing unload_modules_local oleg425-server: modules unloaded. pdsh@oleg425-client: oleg425-client: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-server: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-client: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-server: ssh exited with exit code 2