== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:36:55 (1786541815) start mds service on oleg122-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg122-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-OST0000 oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg122-client.virtnet: -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92553 329 92224 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12329 425 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12233 329 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 1298 (time 1786541842.06 total 10.01 last 129.72) - open/close/unlink 2868 (time 1786541852.06 total 20.01 last 157.00) - open/close/unlink 4606 (time 1786541862.07 total 30.01 last 173.76) - open/close/unlink 6437 (time 1786541872.07 total 40.01 last 183.10) - open/close/unlink 8226 (time 1786541882.07 total 50.01 last 178.87) - open/close/unlink 9486 (time 1786541892.08 total 60.02 last 125.89) - open/close/unlink 10000 (time 1786541894.53 total 62.48 last 209.17) - open/close/unlink 11828 (time 1786541904.54 total 72.48 last 182.73) - open/close/unlink 13596 (time 1786541914.54 total 82.48 last 176.80) - open/close/unlink 15355 (time 1786541924.54 total 92.49 last 175.82) - open/close/unlink 16742 (time 1786541934.54 total 102.49 last 138.69) - open/close/unlink 18349 (time 1786541944.55 total 112.49 last 160.63) - open/close/unlink 19918 (time 1786541954.55 total 122.49 last 156.88) - open/close/unlink 21520 (time 1786541964.55 total 132.49 last 160.15) - open/close/unlink 23126 (time 1786541974.55 total 142.50 last 160.58) - open/close/unlink 24680 (time 1786541984.55 total 152.50 last 155.39) - open/close/unlink 26261 (time 1786541994.55 total 162.50 last 158.08) - open/close/unlink 27818 (time 1786542004.56 total 172.50 last 155.62) - open/close/unlink 29454 (time 1786542014.56 total 182.51 last 163.54) - open/close/unlink 30000 (time 1786542017.47 total 185.42 last 187.58) - open/close/unlink 31595 (time 1786542027.48 total 195.42 last 159.45) - open/close/unlink 33194 (time 1786542037.48 total 205.42 last 159.86) - open/close/unlink 34792 (time 1786542047.48 total 215.43 last 159.75) - open/close/unlink 36397 (time 1786542057.48 total 225.43 last 160.48) - open/close/unlink 37953 (time 1786542067.49 total 235.43 last 155.56) - open/close/unlink 39505 (time 1786542077.49 total 245.43 last 155.14) - open/close/unlink 40000 (time 1786542080.68 total 248.62 last 155.43) - open/close/unlink 41662 (time 1786542090.68 total 258.62 last 166.16) - open/close/unlink 43306 (time 1786542100.70 total 268.64 last 164.02) - open/close/unlink 44949 (time 1786542110.71 total 278.65 last 164.22) - open/close/unlink 46485 (time 1786542120.71 total 288.66 last 153.49) - open/close/unlink 48110 (time 1786542130.72 total 298.66 last 162.46) - open/close/unlink 49751 (time 1786542140.72 total 308.66 last 164.05) - open/close/unlink 51229 (time 1786542150.72 total 318.66 last 147.79) - open/close/unlink 52830 (time 1786542160.72 total 328.66 last 160.07) - open/close/unlink 54388 (time 1786542170.72 total 338.67 last 155.75) - open/close/unlink 55975 (time 1786542180.73 total 348.67 last 158.65) - open/close/unlink 57540 (time 1786542190.73 total 358.67 last 156.47) - open/close/unlink 58868 (time 1786542200.73 total 368.67 last 132.78) - open/close/unlink 60000 (time 1786542209.91 total 377.85 last 123.37) - open/close/unlink 61394 (time 1786542219.91 total 387.86 last 139.32) - open/close/unlink 62679 (time 1786542229.91 total 397.86 last 128.47) - open/close/unlink 63970 (time 1786542239.92 total 407.86 last 129.06) total: 64768 open/close/unlink in 412.99 seconds: 156.83 ops/second - open/close/unlink 1282 (time 1786541842.02 total 10.00 last 128.20) - open/close/unlink 2841 (time 1786541852.02 total 20.00 last 155.87) - open/close/unlink 4581 (time 1786541862.03 total 30.01 last 173.93) - open/close/unlink 6402 (time 1786541872.03 total 40.01 last 182.08) - open/close/unlink 8191 (time 1786541882.03 total 50.01 last 178.82) - open/close/unlink 9443 (time 1786541892.03 total 60.01 last 125.17) - open/close/unlink 10000 (time 1786541894.74 total 62.72 last 206.14) - open/close/unlink 11800 (time 1786541904.74 total 72.72 last 179.98) - open/close/unlink 13568 (time 1786541914.74 total 82.72 last 176.71) - open/close/unlink 15314 (time 1786541924.75 total 92.73 last 174.55) - open/close/unlink 16707 (time 1786541934.75 total 102.73 last 139.24) - open/close/unlink 18297 (time 1786541944.75 total 112.73 last 158.98) - open/close/unlink 19872 (time 1786541954.75 total 122.73 last 157.47) - open/close/unlink 21483 (time 1786541964.76 total 132.74 last 161.03) - open/close/unlink 23099 (time 1786541974.76 total 142.74 last 161.58) - open/close/unlink 24650 (time 1786541984.76 total 152.74 last 155.06) - open/close/unlink 26235 (time 1786541994.77 total 162.74 last 158.45) - open/close/unlink 27791 (time 1786542004.77 total 172.75 last 155.56) - open/close/unlink 29416 (time 1786542014.77 total 182.75 last 162.47) - open/close/unlink 30000 (time 1786542017.96 total 185.94 last 183.16) - open/close/unlink 31600 (time 1786542027.96 total 195.94 last 159.92) - open/close/unlink 33204 (time 1786542037.97 total 205.95 last 160.37) - open/close/unlink 34778 (time 1786542047.97 total 215.95 last 157.38) - open/close/unlink 36398 (time 1786542057.97 total 225.95 last 161.97) - open/close/unlink 37974 (time 1786542067.97 total 235.95 last 157.54) - open/close/unlink 39512 (time 1786542077.98 total 245.96 last 153.75) - open/close/unlink 40000 (time 1786542081.32 total 249.30 last 145.81) - open/close/unlink 41636 (time 1786542091.32 total 259.30 last 163.57) - open/close/unlink 43254 (time 1786542101.33 total 269.31 last 161.68) - open/close/unlink 44910 (time 1786542111.34 total 279.32 last 165.44) - open/close/unlink 46444 (time 1786542121.34 total 289.32 last 153.37) - open/close/unlink 48100 (time 1786542131.35 total 299.33 last 165.53) - open/close/unlink 49770 (time 1786542141.36 total 309.34 last 166.72) - open/close/unlink 51285 (time 1786542151.36 total 319.34 last 151.49) - open/close/unlink 52884 (time 1786542161.37 total 329.35 last 159.87) - open/close/unlink 54418 (time 1786542171.37 total 339.35 last 153.33) - open/close/unlink 56006 (time 1786542181.38 total 349.36 last 158.73) - open/close/unlink 57551 (time 1786542191.38 total 359.36 last 154.47) - open/close/unlink 58870 (time 1786542201.38 total 369.36 last 131.83) - open/close/unlink 60000 (time 1786542210.54 total 378.52 last 123.47) - open/close/unlink 61417 (time 1786542220.54 total 388.52 last 141.69) - open/close/unlink 62710 (time 1786542230.54 total 398.52 last 129.22) - open/close/unlink 63996 (time 1786542240.55 total 408.53 last 128.53) total: 64768 open/close/unlink in 414.50 seconds: 156.25 ops/second - open/close/unlink 1309 (time 1786541842.24 total 10.00 last 130.87) - open/close/unlink 2871 (time 1786541852.24 total 20.01 last 156.15) - open/close/unlink 4608 (time 1786541862.24 total 30.01 last 173.67) - open/close/unlink 6434 (time 1786541872.24 total 40.01 last 182.53) - open/close/unlink 8209 (time 1786541882.24 total 50.01 last 177.49) - open/close/unlink 9463 (time 1786541892.24 total 60.01 last 125.40) - open/close/unlink 10000 (time 1786541894.79 total 62.56 last 210.61) - open/close/unlink 11803 (time 1786541904.80 total 72.56 last 180.23) - open/close/unlink 13549 (time 1786541914.81 total 82.57 last 174.47) - open/close/unlink 15304 (time 1786541924.81 total 92.58 last 175.40) - open/close/unlink 16705 (time 1786541934.81 total 102.58 last 140.05) - open/close/unlink 18302 (time 1786541944.82 total 112.58 last 159.66) - open/close/unlink 19865 (time 1786541954.82 total 122.59 last 156.22) - open/close/unlink 21479 (time 1786541964.82 total 132.59 last 161.39) - open/close/unlink 23088 (time 1786541974.83 total 142.59 last 160.85) - open/close/unlink 24628 (time 1786541984.83 total 152.60 last 153.94) - open/close/unlink 26205 (time 1786541994.83 total 162.60 last 157.62) - open/close/unlink 27755 (time 1786542004.84 total 172.61 last 154.91) - open/close/unlink 29374 (time 1786542014.84 total 182.61 last 161.88) - open/close/unlink 30000 (time 1786542018.30 total 186.07 last 180.88) - open/close/unlink 31609 (time 1786542028.30 total 196.07 last 160.87) - open/close/unlink 33210 (time 1786542038.31 total 206.08 last 160.02) - open/close/unlink 34801 (time 1786542048.31 total 216.08 last 159.03) - open/close/unlink 36418 (time 1786542058.32 total 226.08 last 161.64) - open/close/unlink 37996 (time 1786542068.32 total 236.09 last 157.79) - open/close/unlink 39559 (time 1786542078.32 total 246.09 last 156.29) - open/close/unlink 40000 (time 1786542081.43 total 249.20 last 141.79) - open/close/unlink 41637 (time 1786542091.43 total 259.20 last 163.68) - open/close/unlink 43227 (time 1786542101.43 total 269.20 last 158.96) - open/close/unlink 44873 (time 1786542111.45 total 279.22 last 164.33) - open/close/unlink 46433 (time 1786542121.45 total 289.22 last 155.92) - open/close/unlink 48093 (time 1786542131.46 total 299.22 last 165.97) - open/close/unlink 49774 (time 1786542141.46 total 309.22 last 168.10) - open/close/unlink 51278 (time 1786542151.46 total 319.22 last 150.40) - open/close/unlink 52889 (time 1786542161.46 total 329.22 last 161.10) - open/close/unlink 54453 (time 1786542171.46 total 339.22 last 156.38) - open/close/unlink 56058 (time 1786542181.46 total 349.23 last 160.47) - open/close/unlink 57628 (time 1786542191.46 total 359.23 last 156.96) - open/close/unlink 58927 (time 1786542201.47 total 369.23 last 129.85) - open/close/unlink 60000 (time 1786542210.25 total 378.02 last 122.12) - open/close/unlink 61386 (time 1786542220.26 total 388.03 last 138.52) - open/close/unlink 62669 (time 1786542230.26 total 398.03 last 128.28) - open/close/unlink 63961 (time 1786542240.26 total 408.03 last 129.15) total: 64768 open/close/unlink in 414.43 seconds: 156.28 ops/second - open/close/unlink 1297 (time 1786541842.12 total 10.00 last 129.64) - open/close/unlink 2866 (time 1786541852.12 total 20.00 last 156.90) - open/close/unlink 4607 (time 1786541862.12 total 30.01 last 174.07) - open/close/unlink 6419 (time 1786541872.12 total 40.01 last 181.19) - open/close/unlink 8204 (time 1786541882.12 total 50.01 last 178.49) - open/close/unlink 9440 (time 1786541892.12 total 60.01 last 123.56) - open/close/unlink 10000 (time 1786541894.81 total 62.70 last 208.62) - open/close/unlink 11809 (time 1786541904.81 total 72.70 last 180.87) - open/close/unlink 13550 (time 1786541914.81 total 82.70 last 174.09) - open/close/unlink 15303 (time 1786541924.81 total 92.70 last 175.25) - open/close/unlink 16710 (time 1786541934.82 total 102.71 last 140.63) - open/close/unlink 18300 (time 1786541944.82 total 112.71 last 158.96) - open/close/unlink 19866 (time 1786541954.82 total 122.71 last 156.54) - open/close/unlink 21464 (time 1786541964.83 total 132.71 last 159.75) - open/close/unlink 23073 (time 1786541974.83 total 142.72 last 160.83) - open/close/unlink 24615 (time 1786541984.84 total 152.72 last 154.11) - open/close/unlink 26214 (time 1786541994.84 total 162.73 last 159.78) - open/close/unlink 27787 (time 1786542004.85 total 172.74 last 157.21) - open/close/unlink 29408 (time 1786542014.85 total 182.74 last 162.10) - open/close/unlink 30000 (time 1786542018.08 total 185.97 last 183.42) - open/close/unlink 31612 (time 1786542028.08 total 195.97 last 161.17) - open/close/unlink 33209 (time 1786542038.08 total 205.97 last 159.67) - open/close/unlink 34806 (time 1786542048.08 total 215.97 last 159.62) - open/close/unlink 36416 (time 1786542058.09 total 225.98 last 160.95) - open/close/unlink 37977 (time 1786542068.09 total 235.98 last 156.07) - open/close/unlink 39529 (time 1786542078.10 total 245.98 last 155.12) - open/close/unlink 40000 (time 1786542081.43 total 249.32 last 141.07) - open/close/unlink 41641 (time 1786542091.44 total 259.32 last 164.07) - open/close/unlink 43234 (time 1786542101.45 total 269.34 last 159.04) - open/close/unlink 44864 (time 1786542111.46 total 279.35 last 162.80) - open/close/unlink 46405 (time 1786542121.47 total 289.35 last 154.08) - open/close/unlink 48055 (time 1786542131.47 total 299.36 last 164.88) - open/close/unlink 49726 (time 1786542141.48 total 309.37 last 167.00) - open/close/unlink 51249 (time 1786542151.48 total 319.37 last 152.23) - open/close/unlink 52864 (time 1786542161.48 total 329.37 last 161.48) - open/close/unlink 54412 (time 1786542171.48 total 339.37 last 154.79) - open/close/unlink 56017 (time 1786542181.49 total 349.37 last 160.49) - open/close/unlink 57576 (time 1786542191.49 total 359.38 last 155.82) - open/close/unlink 58887 (time 1786542201.49 total 369.38 last 131.08) - open/close/unlink 60000 (time 1786542210.64 total 378.53 last 121.66) - open/close/unlink 61391 (time 1786542220.65 total 388.53 last 139.02) - open/close/unlink 62669 (time 1786542230.65 total 398.54 last 127.76) - open/close/unlink 63955 (time 1786542240.66 total 408.55 last 128.49) total: 64768 open/close/unlink in 415.49 seconds: 155.88 ops/second - open/close/unlink 1312 (time 1786541842.24 total 10.00 last 131.15) - open/close/unlink 2881 (time 1786541852.24 total 20.01 last 156.83) - open/close/unlink 4614 (time 1786541862.25 total 30.01 last 173.20) - open/close/unlink 6436 (time 1786541872.25 total 40.02 last 182.15) - open/close/unlink 8201 (time 1786541882.25 total 50.02 last 176.47) - open/close/unlink 9454 (time 1786541892.26 total 60.02 last 125.25) - open/close/unlink 10000 (time 1786541894.84 total 62.60 last 211.56) - open/close/unlink 11812 (time 1786541904.84 total 72.61 last 181.11) - open/close/unlink 13562 (time 1786541914.84 total 82.61 last 174.97) - open/close/unlink 15308 (time 1786541924.85 total 92.61 last 174.53) - open/close/unlink 16705 (time 1786541934.85 total 102.62 last 139.65) - open/close/unlink 18299 (time 1786541944.85 total 112.62 last 159.32) - open/close/unlink 19869 (time 1786541954.86 total 122.63 last 156.94) - open/close/unlink 21494 (time 1786541964.86 total 132.63 last 162.44) - open/close/unlink 23093 (time 1786541974.87 total 142.63 last 159.85) - open/close/unlink 24628 (time 1786541984.87 total 152.64 last 153.47) - open/close/unlink 26227 (time 1786541994.87 total 162.64 last 159.86) - open/close/unlink 27785 (time 1786542004.87 total 172.64 last 155.77) - open/close/unlink 29385 (time 1786542014.88 total 182.64 last 159.93) - open/close/unlink 30000 (time 1786542018.38 total 186.15 last 175.64) - open/close/unlink 31608 (time 1786542028.38 total 196.15 last 160.74) - open/close/unlink 33201 (time 1786542038.38 total 206.15 last 159.28) - open/close/unlink 34811 (time 1786542048.39 total 216.15 last 160.96) - open/close/unlink 36424 (time 1786542058.39 total 226.16 last 161.25) - open/close/unlink 37973 (time 1786542068.39 total 236.16 last 154.89) - open/close/unlink 39518 (time 1786542078.39 total 246.16 last 154.43) - open/close/unlink 40000 (time 1786542081.79 total 249.56 last 142.10) - open/close/unlink 41640 (time 1786542091.79 total 259.56 last 163.99) - open/close/unlink 43249 (time 1786542101.79 total 269.56 last 160.90) - open/close/unlink 44840 (time 1786542111.79 total 279.56 last 159.04) - open/close/unlink 46399 (time 1786542121.80 total 289.56 last 155.82) - open/close/unlink 48044 (time 1786542131.80 total 299.57 last 164.38) - open/close/unlink 49746 (time 1786542141.80 total 309.57 last 170.18) - open/close/unlink 51247 (time 1786542151.83 total 319.59 last 149.78) - open/close/unlink 52859 (time 1786542161.83 total 329.60 last 161.19) - open/close/unlink 54418 (time 1786542171.83 total 339.60 last 155.83) - open/close/unlink 56017 (time 1786542181.84 total 349.60 last 159.84) - open/close/unlink 57567 (time 1786542191.84 total 359.60 last 154.98) - open/close/unlink 58849 (time 1786542201.84 total 369.61 last 128.20) - open/close/unlink 60000 (time 1786542211.16 total 378.92 last 123.51) - open/close/unlink 61384 (time 1786542221.16 total 388.93 last 138.35) - open/close/unlink 62649 (time 1786542231.16 total 398.93 last 126.46) - open/close/unlink 63930 (time 1786542241.17 total 408.94 last 128.03) total: 64768 open/close/unlink in 418.02 seconds: 154.94 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete oleg122-server: cannot open 'lustre-ost2': no such pool pdsh@oleg122-client: oleg122-server: ssh exited with exit code 255 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg122-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg122-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server stop mds service on oleg122-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server unloading modules via unload_modules_local on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing unload_modules_local oleg122-server: modules unloaded. pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2