== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 20:43:31 (1777423411) start mds service on oleg338-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg338-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-OST0000 oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg338-client.virtnet: -o user_xattr,flock 192.168.203.138@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92553 329 92224 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12296 392 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12233 329 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 1199 (time 1777423443.54 total 10.01 last 119.81) - open/close/unlink 2561 (time 1777423453.54 total 20.01 last 136.17) - open/close/unlink 3996 (time 1777423463.54 total 30.01 last 143.43) - open/close/unlink 5230 (time 1777423473.55 total 40.02 last 123.34) - open/close/unlink 6365 (time 1777423483.55 total 50.02 last 113.48) - open/close/unlink 7676 (time 1777423493.56 total 60.03 last 131.00) - open/close/unlink 8965 (time 1777423503.56 total 70.03 last 128.85) - open/close/unlink 10000 (time 1777423512.94 total 79.41 last 110.33) - open/close/unlink 10994 (time 1777423522.94 total 89.41 last 99.38) - open/close/unlink 11966 (time 1777423532.95 total 99.42 last 97.18) - open/close/unlink 13086 (time 1777423542.95 total 109.43 last 111.91) - open/close/unlink 14265 (time 1777423552.96 total 119.43 last 117.85) - open/close/unlink 15448 (time 1777423562.96 total 129.43 last 118.24) - open/close/unlink 16676 (time 1777423572.97 total 139.44 last 122.77) - open/close/unlink 17984 (time 1777423582.97 total 149.44 last 130.76) - open/close/unlink 19512 (time 1777423592.97 total 159.44 last 152.73) - open/close/unlink 20000 (time 1777423596.29 total 162.76 last 147.38) - open/close/unlink 21537 (time 1777423606.29 total 172.76 last 153.65) - open/close/unlink 22922 (time 1777423616.29 total 182.76 last 138.45) - open/close/unlink 24018 (time 1777423626.30 total 192.77 last 109.51) - open/close/unlink 25117 (time 1777423636.30 total 202.77 last 109.86) - open/close/unlink 26261 (time 1777423646.31 total 212.78 last 114.37) - open/close/unlink 27621 (time 1777423656.31 total 222.78 last 135.97) - open/close/unlink 29055 (time 1777423666.31 total 232.79 last 143.30) - open/close/unlink 30000 (time 1777423672.81 total 239.28 last 145.46) - open/close/unlink 31318 (time 1777423682.82 total 249.29 last 131.72) - open/close/unlink 32524 (time 1777423692.82 total 259.29 last 120.58) - open/close/unlink 33955 (time 1777423702.82 total 269.29 last 143.03) - open/close/unlink 35476 (time 1777423712.83 total 279.30 last 152.04) - open/close/unlink 36980 (time 1777423722.83 total 289.30 last 150.37) - open/close/unlink 38376 (time 1777423732.83 total 299.30 last 139.57) - open/close/unlink 39596 (time 1777423742.83 total 309.30 last 121.98) - open/close/unlink 40000 (time 1777423745.38 total 311.85 last 158.52) - open/close/unlink 41195 (time 1777423755.39 total 321.86 last 119.43) - open/close/unlink 42453 (time 1777423765.39 total 331.86 last 125.77) - open/close/unlink 43606 (time 1777423775.39 total 341.87 last 115.25) - open/close/unlink 44945 (time 1777423785.40 total 351.87 last 133.88) - open/close/unlink 46449 (time 1777423795.40 total 361.87 last 150.36) - open/close/unlink 47800 (time 1777423805.40 total 371.87 last 135.08) - open/close/unlink 49087 (time 1777423815.40 total 381.87 last 128.68) - open/close/unlink 50000 (time 1777423822.28 total 388.75 last 132.73) - open/close/unlink 51363 (time 1777423832.28 total 398.75 last 136.26) - open/close/unlink 52586 (time 1777423842.29 total 408.76 last 122.27) - open/close/unlink 53672 (time 1777423852.29 total 418.76 last 108.57) - open/close/unlink 54645 (time 1777423862.29 total 428.76 last 97.27) - open/close/unlink 55665 (time 1777423872.29 total 438.76 last 101.98) - open/close/unlink 56862 (time 1777423882.30 total 448.77 last 119.64) - open/close/unlink 58124 (time 1777423892.30 total 458.77 last 126.20) - open/close/unlink 59443 (time 1777423902.31 total 468.78 last 131.80) - open/close/unlink 60000 (time 1777423907.15 total 473.62 last 115.07) - open/close/unlink 60937 (time 1777423917.15 total 483.62 last 93.66) - open/close/unlink 62164 (time 1777423927.15 total 493.62 last 122.69) - open/close/unlink 63563 (time 1777423937.16 total 503.63 last 139.85) total: 64768 open/close/unlink in 511.67 seconds: 126.58 ops/second - open/close/unlink 1194 (time 1777423443.64 total 10.00 last 119.39) - open/close/unlink 2551 (time 1777423453.65 total 20.01 last 135.59) - open/close/unlink 3979 (time 1777423463.66 total 30.01 last 142.75) - open/close/unlink 5212 (time 1777423473.66 total 40.02 last 123.23) - open/close/unlink 6355 (time 1777423483.67 total 50.02 last 114.25) - open/close/unlink 7654 (time 1777423493.67 total 60.02 last 129.87) - open/close/unlink 8947 (time 1777423503.67 total 70.03 last 129.27) - open/close/unlink 10000 (time 1777423513.23 total 79.59 last 110.14) - open/close/unlink 10993 (time 1777423523.24 total 89.60 last 99.21) - open/close/unlink 11965 (time 1777423533.24 total 99.60 last 97.14) - open/close/unlink 13066 (time 1777423543.25 total 109.61 last 110.04) - open/close/unlink 14264 (time 1777423553.26 total 119.62 last 119.67) - open/close/unlink 15450 (time 1777423563.26 total 129.62 last 118.58) - open/close/unlink 16668 (time 1777423573.27 total 139.63 last 121.72) - open/close/unlink 17970 (time 1777423583.27 total 149.63 last 130.15) - open/close/unlink 19527 (time 1777423593.28 total 159.64 last 155.59) - open/close/unlink 20000 (time 1777423596.42 total 162.78 last 150.73) - open/close/unlink 21532 (time 1777423606.42 total 172.78 last 153.13) - open/close/unlink 22899 (time 1777423616.42 total 182.78 last 136.69) - open/close/unlink 23984 (time 1777423626.43 total 192.79 last 108.39) - open/close/unlink 25088 (time 1777423636.44 total 202.79 last 110.38) - open/close/unlink 26235 (time 1777423646.44 total 212.80 last 114.65) - open/close/unlink 27592 (time 1777423656.45 total 222.80 last 135.63) - open/close/unlink 29019 (time 1777423666.45 total 232.81 last 142.60) - open/close/unlink 30000 (time 1777423673.20 total 239.56 last 145.34) - open/close/unlink 31300 (time 1777423683.20 total 249.56 last 129.99) - open/close/unlink 32531 (time 1777423693.20 total 259.56 last 123.08) - open/close/unlink 33960 (time 1777423703.21 total 269.56 last 142.87) - open/close/unlink 35507 (time 1777423713.21 total 279.57 last 154.61) - open/close/unlink 36991 (time 1777423723.21 total 289.57 last 148.38) - open/close/unlink 38365 (time 1777423733.22 total 299.57 last 137.34) - open/close/unlink 39604 (time 1777423743.22 total 309.58 last 123.87) - open/close/unlink 40000 (time 1777423745.63 total 311.99 last 164.20) - open/close/unlink 41192 (time 1777423755.63 total 321.99 last 119.18) - open/close/unlink 42451 (time 1777423765.63 total 331.99 last 125.89) - open/close/unlink 43590 (time 1777423775.64 total 341.99 last 113.85) - open/close/unlink 44931 (time 1777423785.64 total 352.00 last 134.06) - open/close/unlink 46461 (time 1777423795.65 total 362.00 last 152.91) - open/close/unlink 47811 (time 1777423805.65 total 372.01 last 134.94) - open/close/unlink 49085 (time 1777423815.65 total 382.01 last 127.36) - open/close/unlink 50000 (time 1777423822.49 total 388.85 last 133.84) - open/close/unlink 51362 (time 1777423832.49 total 398.85 last 136.15) - open/close/unlink 52580 (time 1777423842.50 total 408.85 last 121.76) - open/close/unlink 53662 (time 1777423852.50 total 418.86 last 108.14) - open/close/unlink 54652 (time 1777423862.51 total 428.86 last 98.97) - open/close/unlink 55659 (time 1777423872.51 total 438.87 last 100.67) - open/close/unlink 56865 (time 1777423882.51 total 448.87 last 120.59) - open/close/unlink 58107 (time 1777423892.52 total 458.87 last 124.13) - open/close/unlink 59428 (time 1777423902.52 total 468.88 last 132.04) - open/close/unlink 60000 (time 1777423907.55 total 473.90 last 113.78) - open/close/unlink 60958 (time 1777423917.56 total 483.92 last 95.69) - open/close/unlink 62205 (time 1777423927.56 total 493.92 last 124.64) - open/close/unlink 63597 (time 1777423937.57 total 503.93 last 139.10) total: 64768 open/close/unlink in 511.71 seconds: 126.57 ops/second - open/close/unlink 1201 (time 1777423443.68 total 10.00 last 120.09) - open/close/unlink 2559 (time 1777423453.69 total 20.01 last 135.70) - open/close/unlink 3986 (time 1777423463.69 total 30.01 last 142.61) - open/close/unlink 5223 (time 1777423473.70 total 40.02 last 123.64) - open/close/unlink 6377 (time 1777423483.70 total 50.02 last 115.39) - open/close/unlink 7665 (time 1777423493.70 total 60.02 last 128.78) - open/close/unlink 8946 (time 1777423503.70 total 70.02 last 128.09) - open/close/unlink 10000 (time 1777423513.24 total 79.56 last 110.49) - open/close/unlink 10993 (time 1777423523.24 total 89.56 last 99.29) - open/close/unlink 11956 (time 1777423533.24 total 99.56 last 96.28) - open/close/unlink 13063 (time 1777423543.24 total 109.57 last 110.67) - open/close/unlink 14253 (time 1777423553.25 total 119.57 last 118.92) - open/close/unlink 15426 (time 1777423563.26 total 129.58 last 117.26) - open/close/unlink 16640 (time 1777423573.26 total 139.59 last 121.30) - open/close/unlink 17947 (time 1777423583.27 total 149.59 last 130.62) - open/close/unlink 19501 (time 1777423593.27 total 159.59 last 155.39) - open/close/unlink 20000 (time 1777423596.57 total 162.89 last 151.36) - open/close/unlink 21514 (time 1777423606.57 total 172.89 last 151.36) - open/close/unlink 22896 (time 1777423616.58 total 182.90 last 138.10) - open/close/unlink 23975 (time 1777423626.58 total 192.90 last 107.90) - open/close/unlink 25075 (time 1777423636.58 total 202.90 last 109.95) - open/close/unlink 26230 (time 1777423646.58 total 212.90 last 115.49) - open/close/unlink 27582 (time 1777423656.58 total 222.90 last 135.18) - open/close/unlink 29008 (time 1777423666.58 total 232.91 last 142.59) - open/close/unlink 30000 (time 1777423673.41 total 239.73 last 145.32) - open/close/unlink 31296 (time 1777423683.41 total 249.74 last 129.56) - open/close/unlink 32534 (time 1777423693.42 total 259.74 last 123.76) - open/close/unlink 33969 (time 1777423703.42 total 269.74 last 143.47) - open/close/unlink 35505 (time 1777423713.42 total 279.75 last 153.53) - open/close/unlink 37015 (time 1777423723.43 total 289.75 last 150.94) - open/close/unlink 38381 (time 1777423733.43 total 299.75 last 136.55) - open/close/unlink 39620 (time 1777423743.43 total 309.75 last 123.89) - open/close/unlink 40000 (time 1777423745.73 total 312.05 last 165.65) - open/close/unlink 41204 (time 1777423755.73 total 322.05 last 120.35) - open/close/unlink 42468 (time 1777423765.73 total 332.06 last 126.35) - open/close/unlink 43600 (time 1777423775.74 total 342.06 last 113.19) - open/close/unlink 44946 (time 1777423785.74 total 352.06 last 134.56) - open/close/unlink 46464 (time 1777423795.74 total 362.06 last 151.80) - open/close/unlink 47816 (time 1777423805.74 total 372.06 last 135.16) - open/close/unlink 49097 (time 1777423815.74 total 382.07 last 128.09) - open/close/unlink 50000 (time 1777423822.54 total 388.86 last 132.89) - open/close/unlink 51371 (time 1777423832.54 total 398.86 last 137.04) - open/close/unlink 52583 (time 1777423842.55 total 408.87 last 121.17) - open/close/unlink 53667 (time 1777423852.55 total 418.87 last 108.36) - open/close/unlink 54649 (time 1777423862.55 total 428.87 last 98.20) - open/close/unlink 55661 (time 1777423872.56 total 438.88 last 101.11) - open/close/unlink 56867 (time 1777423882.57 total 448.89 last 120.52) - open/close/unlink 58120 (time 1777423892.57 total 458.89 last 125.25) - open/close/unlink 59442 (time 1777423902.57 total 468.89 last 132.18) - open/close/unlink 60000 (time 1777423907.50 total 473.82 last 113.30) - open/close/unlink 60956 (time 1777423917.50 total 483.83 last 95.52) - open/close/unlink 62176 (time 1777423927.51 total 493.83 last 121.96) - open/close/unlink 63570 (time 1777423937.51 total 503.83 last 139.38) total: 64768 open/close/unlink in 511.97 seconds: 126.51 ops/second - open/close/unlink 1204 (time 1777423443.44 total 10.01 last 120.34) - open/close/unlink 2567 (time 1777423453.44 total 20.01 last 136.19) - open/close/unlink 3987 (time 1777423463.45 total 30.02 last 141.93) - open/close/unlink 5216 (time 1777423473.45 total 40.02 last 122.89) - open/close/unlink 6348 (time 1777423483.45 total 50.02 last 113.19) - open/close/unlink 7660 (time 1777423493.46 total 60.03 last 131.12) - open/close/unlink 8941 (time 1777423503.46 total 70.03 last 128.09) - open/close/unlink 10000 (time 1777423513.00 total 79.57 last 110.96) - open/close/unlink 11003 (time 1777423523.01 total 89.58 last 100.23) - open/close/unlink 11964 (time 1777423533.01 total 99.58 last 96.09) - open/close/unlink 13072 (time 1777423543.03 total 109.60 last 110.57) - open/close/unlink 14263 (time 1777423553.03 total 119.60 last 119.06) - open/close/unlink 15434 (time 1777423563.03 total 129.60 last 117.09) - open/close/unlink 16667 (time 1777423573.05 total 139.61 last 123.17) - open/close/unlink 17981 (time 1777423583.12 total 149.69 last 130.36) - open/close/unlink 19547 (time 1777423593.14 total 159.70 last 156.43) - open/close/unlink 20000 (time 1777423596.20 total 162.77 last 147.72) - open/close/unlink 21527 (time 1777423606.21 total 172.78 last 152.61) - open/close/unlink 22905 (time 1777423616.21 total 182.78 last 137.79) - open/close/unlink 24012 (time 1777423626.22 total 192.78 last 110.63) - open/close/unlink 25114 (time 1777423636.22 total 202.78 last 110.20) - open/close/unlink 26253 (time 1777423646.22 total 212.79 last 113.87) - open/close/unlink 27608 (time 1777423656.22 total 222.79 last 135.46) - open/close/unlink 29036 (time 1777423666.22 total 232.79 last 142.79) - open/close/unlink 30000 (time 1777423672.86 total 239.43 last 145.20) - open/close/unlink 31313 (time 1777423682.86 total 249.43 last 131.25) - open/close/unlink 32533 (time 1777423692.87 total 259.44 last 121.96) - open/close/unlink 33943 (time 1777423702.87 total 269.44 last 140.97) - open/close/unlink 35478 (time 1777423712.87 total 279.44 last 153.46) - open/close/unlink 36978 (time 1777423722.88 total 289.44 last 149.96) - open/close/unlink 38377 (time 1777423732.88 total 299.45 last 139.88) - open/close/unlink 39603 (time 1777423742.88 total 309.45 last 122.58) - open/close/unlink 40000 (time 1777423745.38 total 311.95 last 158.71) - open/close/unlink 41194 (time 1777423755.38 total 321.95 last 119.38) - open/close/unlink 42437 (time 1777423765.39 total 331.95 last 124.25) - open/close/unlink 43574 (time 1777423775.39 total 341.96 last 113.67) - open/close/unlink 44896 (time 1777423785.39 total 351.96 last 132.17) - open/close/unlink 46411 (time 1777423795.39 total 361.96 last 151.49) - open/close/unlink 47768 (time 1777423805.39 total 371.96 last 135.68) - open/close/unlink 49045 (time 1777423815.39 total 381.96 last 127.68) - open/close/unlink 50000 (time 1777423822.54 total 389.11 last 133.67) - open/close/unlink 51360 (time 1777423832.54 total 399.11 last 135.95) - open/close/unlink 52569 (time 1777423842.54 total 409.11 last 120.87) - open/close/unlink 53646 (time 1777423852.55 total 419.12 last 107.63) - open/close/unlink 54631 (time 1777423862.56 total 429.12 last 98.45) - open/close/unlink 55636 (time 1777423872.56 total 439.13 last 100.49) - open/close/unlink 56840 (time 1777423882.56 total 449.13 last 120.38) - open/close/unlink 58071 (time 1777423892.56 total 459.13 last 123.09) - open/close/unlink 59397 (time 1777423902.56 total 469.13 last 132.59) - open/close/unlink 60000 (time 1777423907.85 total 474.42 last 113.96) - open/close/unlink 60951 (time 1777423917.86 total 484.43 last 95.04) - open/close/unlink 62196 (time 1777423927.86 total 494.43 last 124.47) - open/close/unlink 63609 (time 1777423937.87 total 504.43 last 141.22) total: 64768 open/close/unlink in 512.48 seconds: 126.38 ops/second - open/close/unlink 1199 (time 1777423443.52 total 10.01 last 119.82) - open/close/unlink 2556 (time 1777423453.53 total 20.01 last 135.68) - open/close/unlink 3989 (time 1777423463.53 total 30.01 last 143.28) - open/close/unlink 5224 (time 1777423473.53 total 40.01 last 123.47) - open/close/unlink 6356 (time 1777423483.53 total 50.02 last 113.14) - open/close/unlink 7670 (time 1777423493.54 total 60.02 last 131.30) - open/close/unlink 8963 (time 1777423503.55 total 70.03 last 129.21) - open/close/unlink 10000 (time 1777423512.97 total 79.45 last 110.10) - open/close/unlink 10995 (time 1777423522.97 total 89.46 last 99.45) - open/close/unlink 11951 (time 1777423532.98 total 99.46 last 95.53) - open/close/unlink 13065 (time 1777423542.99 total 109.47 last 111.29) - open/close/unlink 14247 (time 1777423552.99 total 119.48 last 118.14) - open/close/unlink 15424 (time 1777423563.00 total 129.48 last 117.68) - open/close/unlink 16652 (time 1777423573.01 total 139.49 last 122.68) - open/close/unlink 17968 (time 1777423583.01 total 149.50 last 131.50) - open/close/unlink 19509 (time 1777423593.02 total 159.50 last 154.07) - open/close/unlink 20000 (time 1777423596.35 total 162.83 last 147.41) - open/close/unlink 21517 (time 1777423606.35 total 172.84 last 151.61) - open/close/unlink 22902 (time 1777423616.36 total 182.84 last 138.46) - open/close/unlink 23987 (time 1777423626.36 total 192.84 last 108.49) - open/close/unlink 25091 (time 1777423636.36 total 202.84 last 110.34) - open/close/unlink 26230 (time 1777423646.37 total 212.85 last 113.83) - open/close/unlink 27581 (time 1777423656.37 total 222.86 last 135.04) - open/close/unlink 29000 (time 1777423666.38 total 232.86 last 141.85) - open/close/unlink 30000 (time 1777423673.21 total 239.70 last 146.26) - open/close/unlink 31295 (time 1777423683.21 total 249.70 last 129.48) - open/close/unlink 32532 (time 1777423693.22 total 259.70 last 123.68) - open/close/unlink 33946 (time 1777423703.22 total 269.70 last 141.36) - open/close/unlink 35482 (time 1777423713.22 total 279.71 last 153.55) - open/close/unlink 36985 (time 1777423723.23 total 289.71 last 150.24) - open/close/unlink 38351 (time 1777423733.23 total 299.71 last 136.58) - open/close/unlink 39592 (time 1777423743.23 total 309.72 last 124.03) - open/close/unlink 40000 (time 1777423745.72 total 312.20 last 164.25) - open/close/unlink 41189 (time 1777423755.73 total 322.21 last 118.80) - open/close/unlink 42450 (time 1777423765.73 total 332.21 last 126.07) - open/close/unlink 43583 (time 1777423775.73 total 342.21 last 113.28) - open/close/unlink 44929 (time 1777423785.73 total 352.21 last 134.59) - open/close/unlink 46448 (time 1777423795.73 total 362.22 last 151.84) - open/close/unlink 47796 (time 1777423805.74 total 372.22 last 134.77) - open/close/unlink 49068 (time 1777423815.74 total 382.22 last 127.15) - open/close/unlink 50000 (time 1777423822.71 total 389.19 last 133.75) - open/close/unlink 51350 (time 1777423832.71 total 399.20 last 134.92) - open/close/unlink 52558 (time 1777423842.72 total 409.20 last 120.73) - open/close/unlink 53632 (time 1777423852.73 total 419.21 last 107.33) - open/close/unlink 54618 (time 1777423862.73 total 429.22 last 98.53) - open/close/unlink 55642 (time 1777423872.74 total 439.23 last 102.30) - open/close/unlink 56841 (time 1777423882.75 total 449.23 last 119.86) - open/close/unlink 58076 (time 1777423892.75 total 459.23 last 123.47) - open/close/unlink 59399 (time 1777423902.75 total 469.23 last 132.26) - open/close/unlink 60000 (time 1777423908.06 total 474.55 last 113.14) - open/close/unlink 60951 (time 1777423918.07 total 484.55 last 95.05) - open/close/unlink 62199 (time 1777423928.07 total 494.56 last 124.73) - open/close/unlink 63607 (time 1777423938.08 total 504.56 last 140.78) total: 64768 open/close/unlink in 512.52 seconds: 126.37 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg338-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg338-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg338-server stop mds service on oleg338-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg338-server unloading modules via unload_modules_local on: 'oleg338-server' oleg338-server: oleg338-server.virtnet: executing unload_modules_local oleg338-server: modules unloaded. pdsh@oleg338-client: oleg338-client: ssh exited with exit code 2 pdsh@oleg338-client: oleg338-server: ssh exited with exit code 2 pdsh@oleg338-client: oleg338-client: ssh exited with exit code 2 pdsh@oleg338-client: oleg338-server: ssh exited with exit code 2