== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 21:46:54 (1781660814) start mds service on oleg226-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg226-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg226-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0000 oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 50000 401 49599 1% /mnt/lustre[OST:0] filesystem_summary: 50128 529 49599 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 701 (time 1781660862.04 total 10.00 last 70.07) - open/close/unlink 1584 (time 1781660872.05 total 20.01 last 88.23) - open/close/unlink 2463 (time 1781660882.05 total 30.02 last 87.87) - open/close/unlink 3494 (time 1781660892.06 total 40.02 last 103.04) - open/close/unlink 4479 (time 1781660902.06 total 50.03 last 98.43) - open/close/unlink 5583 (time 1781660912.06 total 60.03 last 110.38) - open/close/unlink 6717 (time 1781660922.06 total 70.03 last 113.40) - open/close/unlink 7993 (time 1781660932.06 total 80.03 last 127.59) - open/close/unlink 9502 (time 1781660942.07 total 90.04 last 150.83) - open/close/unlink 10000 (time 1781660945.77 total 93.73 last 134.76) - open/close/unlink 11708 (time 1781660955.77 total 103.74 last 170.71) - open/close/unlink 13173 (time 1781660965.77 total 113.74 last 146.46) - open/close/unlink 14731 (time 1781660975.77 total 123.74 last 155.78) - open/close/unlink 16362 (time 1781660985.78 total 133.74 last 163.05) - open/close/unlink 17800 (time 1781660995.78 total 143.75 last 143.75) - open/close/unlink 19828 (time 1781661005.78 total 153.75 last 202.79) - open/close/unlink 21689 (time 1781661015.79 total 163.75 last 185.99) - open/close/unlink 23463 (time 1781661025.79 total 173.76 last 177.35) - open/close/unlink 25703 (time 1781661035.86 total 183.82 last 222.52) - open/close/unlink 28079 (time 1781661045.86 total 193.83 last 237.53) - open/close/unlink 30000 (time 1781661053.91 total 201.88 last 238.61) - open/close/unlink 32021 (time 1781661063.92 total 211.88 last 202.00) - open/close/unlink 33646 (time 1781661073.92 total 221.89 last 162.44) - open/close/unlink 35443 (time 1781661083.92 total 231.89 last 179.66) - open/close/unlink 37240 (time 1781661093.92 total 241.89 last 179.67) - open/close/unlink 39026 (time 1781661103.93 total 251.89 last 178.55) - open/close/unlink 40000 (time 1781661108.88 total 256.85 last 196.52) - open/close/unlink 41848 (time 1781661118.88 total 266.85 last 184.76) - open/close/unlink 43744 (time 1781661128.89 total 276.85 last 189.55) - open/close/unlink 45517 (time 1781661138.89 total 286.86 last 177.23) - open/close/unlink 47350 (time 1781661148.89 total 296.86 last 183.30) - open/close/unlink 49476 (time 1781661158.90 total 306.86 last 212.52) - open/close/unlink 50000 (time 1781661161.68 total 309.64 last 188.36) - open/close/unlink 52030 (time 1781661171.68 total 319.64 last 202.98) - open/close/unlink 53901 (time 1781661181.68 total 329.65 last 187.06) - open/close/unlink 55896 (time 1781661191.68 total 339.65 last 199.41) - open/close/unlink 57898 (time 1781661201.69 total 349.66 last 200.03) - open/close/unlink 59908 (time 1781661211.69 total 359.66 last 200.97) - open/close/unlink 61877 (time 1781661221.70 total 369.67 last 196.79) - open/close/unlink 63852 (time 1781661231.70 total 379.67 last 197.48) total: 64768 open/close/unlink in 385.18 seconds: 168.15 ops/second - open/close/unlink 710 (time 1781660862.24 total 10.00 last 70.98) - open/close/unlink 1589 (time 1781660872.25 total 20.01 last 87.88) - open/close/unlink 2465 (time 1781660882.28 total 30.04 last 87.29) - open/close/unlink 3512 (time 1781660892.29 total 40.05 last 104.60) - open/close/unlink 4488 (time 1781660902.29 total 50.05 last 97.56) - open/close/unlink 5589 (time 1781660912.30 total 60.06 last 110.03) - open/close/unlink 6748 (time 1781660922.30 total 70.06 last 115.87) - open/close/unlink 8027 (time 1781660932.32 total 80.08 last 127.75) - open/close/unlink 9521 (time 1781660942.32 total 90.08 last 149.30) - open/close/unlink 10000 (time 1781660945.80 total 93.56 last 137.61) - open/close/unlink 11686 (time 1781660955.80 total 103.56 last 168.58) - open/close/unlink 13155 (time 1781660965.81 total 113.57 last 146.82) - open/close/unlink 14701 (time 1781660975.81 total 123.57 last 154.57) - open/close/unlink 16343 (time 1781660985.81 total 133.57 last 164.20) - open/close/unlink 17777 (time 1781660995.81 total 143.57 last 143.36) - open/close/unlink 19803 (time 1781661005.82 total 153.58 last 202.52) - open/close/unlink 21678 (time 1781661015.82 total 163.58 last 187.45) - open/close/unlink 23471 (time 1781661025.82 total 173.58 last 179.26) - open/close/unlink 25618 (time 1781661035.86 total 183.62 last 213.99) - open/close/unlink 28019 (time 1781661045.86 total 193.62 last 240.04) - open/close/unlink 30000 (time 1781661054.04 total 201.80 last 242.03) - open/close/unlink 31974 (time 1781661064.05 total 211.81 last 197.31) - open/close/unlink 33585 (time 1781661074.05 total 221.81 last 161.08) - open/close/unlink 35397 (time 1781661084.05 total 231.81 last 181.15) - open/close/unlink 37192 (time 1781661094.06 total 241.82 last 179.45) - open/close/unlink 38983 (time 1781661104.06 total 251.82 last 178.99) - open/close/unlink 40000 (time 1781661109.23 total 256.99 last 196.80) - open/close/unlink 41868 (time 1781661119.23 total 266.99 last 186.75) - open/close/unlink 43800 (time 1781661129.24 total 277.00 last 193.09) - open/close/unlink 45574 (time 1781661139.24 total 287.00 last 177.31) - open/close/unlink 47474 (time 1781661149.25 total 297.01 last 189.95) - open/close/unlink 49559 (time 1781661159.25 total 307.01 last 208.44) - open/close/unlink 50000 (time 1781661161.58 total 309.34 last 189.34) - open/close/unlink 52001 (time 1781661171.58 total 319.34 last 200.01) - open/close/unlink 53896 (time 1781661181.59 total 329.35 last 189.40) - open/close/unlink 55881 (time 1781661191.59 total 339.35 last 198.48) - open/close/unlink 57839 (time 1781661201.60 total 349.36 last 195.66) - open/close/unlink 59871 (time 1781661211.60 total 359.36 last 203.14) - open/close/unlink 61843 (time 1781661221.62 total 369.38 last 196.87) - open/close/unlink 63770 (time 1781661231.62 total 379.38 last 192.67) total: 64768 open/close/unlink in 386.77 seconds: 167.46 ops/second - open/close/unlink 707 (time 1781660862.22 total 10.00 last 70.67) - open/close/unlink 1588 (time 1781660872.22 total 20.01 last 88.07) - open/close/unlink 2465 (time 1781660882.26 total 30.05 last 87.36) - open/close/unlink 3508 (time 1781660892.26 total 40.05 last 104.26) - open/close/unlink 4485 (time 1781660902.27 total 50.05 last 97.65) - open/close/unlink 5582 (time 1781660912.27 total 60.06 last 109.65) - open/close/unlink 6741 (time 1781660922.27 total 70.06 last 115.88) - open/close/unlink 8013 (time 1781660932.28 total 80.07 last 127.08) - open/close/unlink 9493 (time 1781660942.30 total 90.09 last 147.78) - open/close/unlink 10000 (time 1781660945.96 total 93.75 last 138.24) - open/close/unlink 11692 (time 1781660955.97 total 103.75 last 169.18) - open/close/unlink 13142 (time 1781660965.97 total 113.76 last 144.92) - open/close/unlink 14696 (time 1781660975.97 total 123.76 last 155.37) - open/close/unlink 16338 (time 1781660985.98 total 133.77 last 164.10) - open/close/unlink 17784 (time 1781660995.98 total 143.77 last 144.54) - open/close/unlink 19806 (time 1781661005.99 total 153.78 last 202.07) - open/close/unlink 21709 (time 1781661015.99 total 163.78 last 190.25) - open/close/unlink 23491 (time 1781661026.00 total 173.79 last 178.10) - open/close/unlink 25665 (time 1781661036.00 total 183.79 last 217.33) - open/close/unlink 28080 (time 1781661046.00 total 193.79 last 241.49) - open/close/unlink 30000 (time 1781661054.00 total 201.79 last 240.00) - open/close/unlink 31974 (time 1781661064.01 total 211.80 last 197.29) - open/close/unlink 33588 (time 1781661074.01 total 221.80 last 161.37) - open/close/unlink 35390 (time 1781661084.01 total 231.80 last 180.12) - open/close/unlink 37183 (time 1781661094.02 total 241.81 last 179.23) - open/close/unlink 38970 (time 1781661104.02 total 251.81 last 178.65) - open/close/unlink 40000 (time 1781661109.50 total 257.29 last 187.99) - open/close/unlink 41850 (time 1781661119.50 total 267.29 last 184.89) - open/close/unlink 43763 (time 1781661129.51 total 277.30 last 191.14) - open/close/unlink 45523 (time 1781661139.52 total 287.31 last 175.81) - open/close/unlink 47395 (time 1781661149.53 total 297.32 last 187.13) - open/close/unlink 49526 (time 1781661159.54 total 307.33 last 212.83) - open/close/unlink 50000 (time 1781661162.11 total 309.90 last 184.54) - open/close/unlink 52048 (time 1781661172.11 total 319.90 last 204.80) - open/close/unlink 53928 (time 1781661182.11 total 329.90 last 187.98) - open/close/unlink 55867 (time 1781661192.11 total 339.90 last 193.81) - open/close/unlink 57836 (time 1781661202.11 total 349.90 last 196.89) - open/close/unlink 59858 (time 1781661212.12 total 359.90 last 202.17) - open/close/unlink 61808 (time 1781661222.12 total 369.91 last 194.92) - open/close/unlink 63761 (time 1781661232.12 total 379.91 last 195.26) total: 64768 open/close/unlink in 388.96 seconds: 166.52 ops/second - open/close/unlink 714 (time 1781660862.19 total 10.01 last 71.36) - open/close/unlink 1599 (time 1781660872.19 total 20.01 last 88.47) - open/close/unlink 2487 (time 1781660882.24 total 30.06 last 88.34) - open/close/unlink 3521 (time 1781660892.25 total 40.07 last 103.31) - open/close/unlink 4500 (time 1781660902.26 total 50.08 last 97.83) - open/close/unlink 5595 (time 1781660912.26 total 60.08 last 109.43) - open/close/unlink 6758 (time 1781660922.27 total 70.09 last 116.27) - open/close/unlink 8030 (time 1781660932.27 total 80.09 last 127.10) - open/close/unlink 9514 (time 1781660942.28 total 90.11 last 148.24) - open/close/unlink 10000 (time 1781660945.83 total 93.65 last 137.26) - open/close/unlink 11671 (time 1781660955.84 total 103.66 last 166.88) - open/close/unlink 13138 (time 1781660965.84 total 113.66 last 146.66) - open/close/unlink 14659 (time 1781660975.84 total 123.67 last 152.05) - open/close/unlink 16284 (time 1781660985.85 total 133.67 last 162.48) - open/close/unlink 17723 (time 1781660995.85 total 143.67 last 143.89) - open/close/unlink 19723 (time 1781661005.85 total 153.67 last 200.00) - open/close/unlink 21620 (time 1781661015.85 total 163.67 last 189.58) - open/close/unlink 23410 (time 1781661025.85 total 173.67 last 178.98) - open/close/unlink 25592 (time 1781661035.86 total 183.68 last 218.16) - open/close/unlink 27961 (time 1781661045.86 total 193.68 last 236.82) - open/close/unlink 30000 (time 1781661054.15 total 201.97 last 245.91) - open/close/unlink 31926 (time 1781661064.16 total 211.98 last 192.52) - open/close/unlink 33560 (time 1781661074.16 total 221.98 last 163.35) - open/close/unlink 35379 (time 1781661084.16 total 231.98 last 181.88) - open/close/unlink 37175 (time 1781661094.16 total 241.98 last 179.54) - open/close/unlink 38966 (time 1781661104.16 total 251.98 last 179.08) - open/close/unlink 40000 (time 1781661109.79 total 257.61 last 183.92) - open/close/unlink 41861 (time 1781661119.79 total 267.61 last 186.07) - open/close/unlink 43790 (time 1781661129.79 total 277.61 last 192.85) - open/close/unlink 45548 (time 1781661139.79 total 287.61 last 175.79) - open/close/unlink 47422 (time 1781661149.79 total 297.61 last 187.35) - open/close/unlink 49475 (time 1781661159.80 total 307.62 last 205.24) - open/close/unlink 50000 (time 1781661162.37 total 310.20 last 203.63) - open/close/unlink 52040 (time 1781661172.38 total 320.20 last 203.93) - open/close/unlink 53907 (time 1781661182.38 total 330.20 last 186.64) - open/close/unlink 55873 (time 1781661192.38 total 340.21 last 196.54) - open/close/unlink 57843 (time 1781661202.39 total 350.21 last 196.98) - open/close/unlink 59894 (time 1781661212.39 total 360.21 last 205.07) - open/close/unlink 61854 (time 1781661222.39 total 370.21 last 195.94) - open/close/unlink 63784 (time 1781661232.40 total 380.22 last 192.86) total: 64768 open/close/unlink in 389.56 seconds: 166.26 ops/second - open/close/unlink 710 (time 1781660862.38 total 10.00 last 70.99) - open/close/unlink 1595 (time 1781660872.38 total 20.00 last 88.50) - open/close/unlink 2454 (time 1781660882.38 total 30.01 last 85.86) - open/close/unlink 3498 (time 1781660892.39 total 40.01 last 104.33) - open/close/unlink 4476 (time 1781660902.40 total 50.02 last 97.72) - open/close/unlink 5575 (time 1781660912.41 total 60.03 last 109.83) - open/close/unlink 6743 (time 1781660922.41 total 70.03 last 116.77) - open/close/unlink 8012 (time 1781660932.41 total 80.03 last 126.89) - open/close/unlink 9508 (time 1781660942.41 total 90.03 last 149.59) - open/close/unlink 10000 (time 1781660945.97 total 93.60 last 138.07) - open/close/unlink 11704 (time 1781660955.98 total 103.60 last 170.34) - open/close/unlink 13166 (time 1781660965.98 total 113.60 last 146.14) - open/close/unlink 14699 (time 1781660975.98 total 123.60 last 153.29) - open/close/unlink 16333 (time 1781660985.99 total 133.61 last 163.30) - open/close/unlink 17770 (time 1781660995.99 total 143.61 last 143.69) - open/close/unlink 19783 (time 1781661005.99 total 153.62 last 201.22) - open/close/unlink 21639 (time 1781661016.00 total 163.62 last 185.54) - open/close/unlink 23413 (time 1781661026.00 total 173.62 last 177.35) - open/close/unlink 25607 (time 1781661036.00 total 183.63 last 219.31) - open/close/unlink 27994 (time 1781661046.01 total 193.63 last 238.64) - open/close/unlink 30000 (time 1781661054.21 total 201.84 last 244.41) - open/close/unlink 31933 (time 1781661064.22 total 211.84 last 193.25) - open/close/unlink 33556 (time 1781661074.22 total 221.84 last 162.22) - open/close/unlink 35370 (time 1781661084.22 total 231.84 last 181.37) - open/close/unlink 37140 (time 1781661094.22 total 241.85 last 176.96) - open/close/unlink 38925 (time 1781661104.23 total 251.85 last 178.45) - open/close/unlink 40000 (time 1781661109.96 total 257.59 last 187.38) - open/close/unlink 41859 (time 1781661119.97 total 267.59 last 185.83) - open/close/unlink 43765 (time 1781661129.97 total 277.60 last 190.48) - open/close/unlink 45528 (time 1781661139.97 total 287.60 last 176.28) - open/close/unlink 47420 (time 1781661149.98 total 297.60 last 189.13) - open/close/unlink 49469 (time 1781661159.98 total 307.60 last 204.87) - open/close/unlink 50000 (time 1781661162.71 total 310.33 last 194.50) - open/close/unlink 52042 (time 1781661172.71 total 320.33 last 204.15) - open/close/unlink 53900 (time 1781661182.72 total 330.34 last 185.74) - open/close/unlink 55841 (time 1781661192.77 total 340.39 last 193.01) - open/close/unlink 57828 (time 1781661202.77 total 350.40 last 198.66) - open/close/unlink 59827 (time 1781661212.80 total 360.43 last 199.32) - open/close/unlink 61806 (time 1781661222.81 total 370.43 last 197.84) - open/close/unlink 63714 (time 1781661232.81 total 380.43 last 190.78) - open/close/unlink 64721 (time 1781661242.81 total 390.44 last 100.65) total: 64768 open/close/unlink in 391.06 seconds: 165.62 ops/second Waiting for MDT destroys to complete oleg226-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg226-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg226-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg226-server stop mds service on oleg226-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg226-server stop mds service on oleg226-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg226-server unloading modules via unload_modules_local on: 'oleg226-server' oleg226-server: oleg226-server.virtnet: executing unload_modules_local oleg226-server: modules unloaded. pdsh@oleg226-client: oleg226-client: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-client: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 2