== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:29:25 (1780086565) start mds service on oleg124-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg124-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg124-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-OST0000 oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 50000 433 49567 1% /mnt/lustre[OST:0] filesystem_summary: 50096 529 49567 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 1180 (time 1780086611.90 total 10.00 last 117.95) - open/close/unlink 2507 (time 1780086621.91 total 20.01 last 132.66) - open/close/unlink 3868 (time 1780086631.91 total 30.01 last 136.02) - open/close/unlink 5165 (time 1780086641.91 total 40.01 last 129.67) - open/close/unlink 6708 (time 1780086651.92 total 50.02 last 154.21) - open/close/unlink 8319 (time 1780086661.92 total 60.02 last 161.08) - open/close/unlink 9961 (time 1780086671.92 total 70.03 last 164.14) - open/close/unlink 11823 (time 1780086681.93 total 80.03 last 186.12) - open/close/unlink 13581 (time 1780086691.94 total 90.04 last 175.69) - open/close/unlink 14851 (time 1780086701.94 total 100.04 last 126.92) - open/close/unlink 16298 (time 1780086711.94 total 110.05 last 144.66) - open/close/unlink 17826 (time 1780086721.95 total 120.05 last 152.65) - open/close/unlink 18965 (time 1780086731.96 total 130.06 last 113.82) - open/close/unlink 20000 (time 1780086738.50 total 136.60 last 158.32) - open/close/unlink 21657 (time 1780086748.50 total 146.60 last 165.70) - open/close/unlink 23298 (time 1780086758.50 total 156.60 last 164.01) - open/close/unlink 24981 (time 1780086768.51 total 166.61 last 168.27) - open/close/unlink 26449 (time 1780086778.51 total 176.61 last 146.73) - open/close/unlink 27784 (time 1780086788.51 total 186.61 last 133.47) - open/close/unlink 28994 (time 1780086798.52 total 196.62 last 120.91) - open/close/unlink 30000 (time 1780086805.34 total 203.44 last 147.60) - open/close/unlink 31512 (time 1780086815.34 total 213.44 last 151.11) - open/close/unlink 32776 (time 1780086825.34 total 223.44 last 126.40) - open/close/unlink 34247 (time 1780086835.35 total 233.45 last 146.97) - open/close/unlink 35643 (time 1780086845.36 total 243.46 last 139.49) - open/close/unlink 37181 (time 1780086855.36 total 253.46 last 153.75) - open/close/unlink 38680 (time 1780086865.37 total 263.47 last 149.80) - open/close/unlink 39905 (time 1780086875.38 total 273.48 last 122.34) - open/close/unlink 41172 (time 1780086885.39 total 283.49 last 126.60) - open/close/unlink 42518 (time 1780086895.40 total 293.50 last 134.45) - open/close/unlink 43792 (time 1780086905.42 total 303.52 last 127.16) - open/close/unlink 45070 (time 1780086915.44 total 313.54 last 127.56) - open/close/unlink 46253 (time 1780086925.47 total 323.57 last 117.97) - open/close/unlink 47745 (time 1780086935.47 total 333.57 last 149.15) - open/close/unlink 49270 (time 1780086945.48 total 343.58 last 152.38) - open/close/unlink 50000 (time 1780086951.21 total 349.32 last 127.25) - open/close/unlink 51200 (time 1780086961.23 total 359.33 last 119.84) - open/close/unlink 52736 (time 1780086971.23 total 369.33 last 153.58) - open/close/unlink 54466 (time 1780086981.23 total 379.33 last 172.93) - open/close/unlink 56084 (time 1780086991.24 total 389.34 last 161.73) - open/close/unlink 57663 (time 1780087001.24 total 399.34 last 157.84) - open/close/unlink 59094 (time 1780087011.24 total 409.34 last 143.09) - open/close/unlink 60000 (time 1780087017.05 total 415.15 last 155.87) - open/close/unlink 61801 (time 1780087027.06 total 425.16 last 180.03) - open/close/unlink 63618 (time 1780087037.06 total 435.16 last 181.69) total: 64768 open/close/unlink in 441.92 seconds: 146.56 ops/second - open/close/unlink 1183 (time 1780086611.99 total 10.00 last 118.27) - open/close/unlink 2506 (time 1780086622.00 total 20.01 last 132.19) - open/close/unlink 3874 (time 1780086632.00 total 30.01 last 136.76) - open/close/unlink 5170 (time 1780086642.00 total 40.02 last 129.52) - open/close/unlink 6720 (time 1780086652.01 total 50.02 last 154.96) - open/close/unlink 8331 (time 1780086662.01 total 60.03 last 161.00) - open/close/unlink 9964 (time 1780086672.01 total 70.03 last 163.28) - open/close/unlink 11826 (time 1780086682.02 total 80.03 last 186.15) - open/close/unlink 13575 (time 1780086692.02 total 90.03 last 174.86) - open/close/unlink 14852 (time 1780086702.02 total 100.04 last 127.67) - open/close/unlink 16289 (time 1780086712.02 total 110.04 last 143.65) - open/close/unlink 17810 (time 1780086722.03 total 120.05 last 151.99) - open/close/unlink 18948 (time 1780086732.03 total 130.05 last 113.77) - open/close/unlink 20000 (time 1780086738.65 total 136.66 last 159.09) - open/close/unlink 21640 (time 1780086748.65 total 146.66 last 164.00) - open/close/unlink 23300 (time 1780086758.65 total 156.66 last 165.98) - open/close/unlink 24967 (time 1780086768.65 total 166.66 last 166.70) - open/close/unlink 26427 (time 1780086778.65 total 176.67 last 145.97) - open/close/unlink 27778 (time 1780086788.65 total 186.67 last 135.09) - open/close/unlink 28993 (time 1780086798.65 total 196.67 last 121.48) - open/close/unlink 30000 (time 1780086805.40 total 203.41 last 149.25) - open/close/unlink 31511 (time 1780086815.41 total 213.43 last 150.90) - open/close/unlink 32786 (time 1780086825.42 total 223.44 last 127.40) - open/close/unlink 34251 (time 1780086835.42 total 233.44 last 146.45) - open/close/unlink 35636 (time 1780086845.43 total 243.45 last 138.37) - open/close/unlink 37166 (time 1780086855.43 total 253.45 last 152.99) - open/close/unlink 38666 (time 1780086865.44 total 263.45 last 149.98) - open/close/unlink 39895 (time 1780086875.44 total 273.46 last 122.81) - open/close/unlink 41139 (time 1780086885.45 total 283.46 last 124.37) - open/close/unlink 42476 (time 1780086895.45 total 293.46 last 133.69) - open/close/unlink 43745 (time 1780086905.46 total 303.47 last 126.79) - open/close/unlink 45019 (time 1780086915.46 total 313.47 last 127.36) - open/close/unlink 46211 (time 1780086925.47 total 323.48 last 119.07) - open/close/unlink 47695 (time 1780086935.47 total 333.48 last 148.40) - open/close/unlink 49223 (time 1780086945.48 total 343.50 last 152.58) - open/close/unlink 50000 (time 1780086951.64 total 349.65 last 126.30) - open/close/unlink 51188 (time 1780086961.64 total 359.65 last 118.74) - open/close/unlink 52719 (time 1780086971.64 total 369.66 last 153.09) - open/close/unlink 54453 (time 1780086981.64 total 379.66 last 173.34) - open/close/unlink 56079 (time 1780086991.65 total 389.66 last 162.56) - open/close/unlink 57638 (time 1780087001.65 total 399.67 last 155.82) - open/close/unlink 59084 (time 1780087011.65 total 409.67 last 144.56) - open/close/unlink 60000 (time 1780087017.37 total 415.38 last 160.38) - open/close/unlink 61769 (time 1780087027.37 total 425.38 last 176.86) - open/close/unlink 63589 (time 1780087037.37 total 435.38 last 181.97) total: 64768 open/close/unlink in 443.89 seconds: 145.91 ops/second - open/close/unlink 1194 (time 1780086612.07 total 10.00 last 119.36) - open/close/unlink 2504 (time 1780086622.07 total 20.01 last 130.93) - open/close/unlink 3871 (time 1780086632.08 total 30.01 last 136.63) - open/close/unlink 5148 (time 1780086642.08 total 40.02 last 127.61) - open/close/unlink 6680 (time 1780086652.09 total 50.02 last 153.17) - open/close/unlink 8293 (time 1780086662.09 total 60.02 last 161.28) - open/close/unlink 9920 (time 1780086672.09 total 70.02 last 162.70) - open/close/unlink 11780 (time 1780086682.11 total 80.04 last 185.65) - open/close/unlink 13515 (time 1780086692.11 total 90.05 last 173.43) - open/close/unlink 14783 (time 1780086702.12 total 100.06 last 126.65) - open/close/unlink 16234 (time 1780086712.13 total 110.07 last 144.96) - open/close/unlink 17746 (time 1780086722.15 total 120.08 last 150.96) - open/close/unlink 18890 (time 1780086732.15 total 130.09 last 114.37) - open/close/unlink 20000 (time 1780086739.10 total 137.04 last 159.63) - open/close/unlink 21622 (time 1780086749.11 total 147.05 last 162.09) - open/close/unlink 23309 (time 1780086759.11 total 157.05 last 168.65) - open/close/unlink 24986 (time 1780086769.11 total 167.05 last 167.69) - open/close/unlink 26437 (time 1780086779.11 total 177.05 last 145.10) - open/close/unlink 27770 (time 1780086789.12 total 187.05 last 133.25) - open/close/unlink 28993 (time 1780086799.12 total 197.06 last 122.29) - open/close/unlink 30000 (time 1780086805.99 total 203.93 last 146.51) - open/close/unlink 31510 (time 1780086816.00 total 213.94 last 150.86) - open/close/unlink 32761 (time 1780086826.01 total 223.95 last 124.99) - open/close/unlink 34231 (time 1780086836.02 total 233.96 last 146.88) - open/close/unlink 35647 (time 1780086846.02 total 243.96 last 141.58) - open/close/unlink 37197 (time 1780086856.02 total 253.96 last 154.97) - open/close/unlink 38695 (time 1780086866.03 total 263.97 last 149.71) - open/close/unlink 39909 (time 1780086876.03 total 273.97 last 121.38) - open/close/unlink 41159 (time 1780086886.03 total 283.97 last 124.98) - open/close/unlink 42509 (time 1780086896.04 total 293.97 last 134.94) - open/close/unlink 43773 (time 1780086906.04 total 303.98 last 126.33) - open/close/unlink 45040 (time 1780086916.05 total 313.99 last 126.62) - open/close/unlink 46203 (time 1780086926.06 total 323.99 last 116.22) - open/close/unlink 47729 (time 1780086936.06 total 334.00 last 152.55) - open/close/unlink 49223 (time 1780086946.06 total 344.00 last 149.33) - open/close/unlink 50000 (time 1780086952.12 total 350.06 last 128.19) - open/close/unlink 51180 (time 1780086962.13 total 360.06 last 117.97) - open/close/unlink 52732 (time 1780086972.13 total 370.07 last 155.16) - open/close/unlink 54469 (time 1780086982.13 total 380.07 last 173.64) - open/close/unlink 56083 (time 1780086992.13 total 390.07 last 161.39) - open/close/unlink 57633 (time 1780087002.14 total 400.08 last 154.94) - open/close/unlink 59062 (time 1780087012.15 total 410.08 last 142.80) - open/close/unlink 60000 (time 1780087018.13 total 416.07 last 156.64) - open/close/unlink 61814 (time 1780087028.13 total 426.07 last 181.40) - open/close/unlink 63635 (time 1780087038.14 total 436.08 last 182.01) - open/close/unlink 64767 (time 1780087048.15 total 446.09 last 113.09) total: 64768 open/close/unlink in 446.09 seconds: 145.19 ops/second - open/close/unlink 1186 (time 1780086612.06 total 10.00 last 118.54) - open/close/unlink 2505 (time 1780086622.06 total 20.01 last 131.83) - open/close/unlink 3880 (time 1780086632.06 total 30.01 last 137.49) - open/close/unlink 5162 (time 1780086642.06 total 40.01 last 128.16) - open/close/unlink 6704 (time 1780086652.07 total 50.02 last 154.14) - open/close/unlink 8323 (time 1780086662.08 total 60.03 last 161.77) - open/close/unlink 9951 (time 1780086672.08 total 70.03 last 162.79) - open/close/unlink 11803 (time 1780086682.08 total 80.03 last 185.06) - open/close/unlink 13567 (time 1780086692.09 total 90.04 last 176.32) - open/close/unlink 14829 (time 1780086702.11 total 100.06 last 125.92) - open/close/unlink 16277 (time 1780086712.12 total 110.07 last 144.68) - open/close/unlink 17784 (time 1780086722.14 total 120.09 last 150.45) - open/close/unlink 18930 (time 1780086732.15 total 130.10 last 114.49) - open/close/unlink 20000 (time 1780086738.85 total 136.80 last 159.64) - open/close/unlink 21630 (time 1780086748.85 total 146.80 last 162.94) - open/close/unlink 23291 (time 1780086758.86 total 156.80 last 166.05) - open/close/unlink 24968 (time 1780086768.86 total 166.81 last 167.69) - open/close/unlink 26410 (time 1780086778.86 total 176.81 last 144.15) - open/close/unlink 27754 (time 1780086788.86 total 186.81 last 134.36) - open/close/unlink 28974 (time 1780086798.87 total 196.82 last 121.92) - open/close/unlink 30000 (time 1780086805.82 total 203.77 last 147.64) - open/close/unlink 31492 (time 1780086815.82 total 213.77 last 149.19) - open/close/unlink 32733 (time 1780086825.82 total 223.77 last 124.09) - open/close/unlink 34204 (time 1780086835.82 total 233.77 last 147.06) - open/close/unlink 35616 (time 1780086845.83 total 243.78 last 141.06) - open/close/unlink 37143 (time 1780086855.84 total 253.78 last 152.66) - open/close/unlink 38639 (time 1780086865.84 total 263.79 last 149.58) - open/close/unlink 39853 (time 1780086875.84 total 273.79 last 121.32) - open/close/unlink 41117 (time 1780086885.85 total 283.80 last 126.36) - open/close/unlink 42479 (time 1780086895.85 total 293.80 last 136.10) - open/close/unlink 43744 (time 1780086905.86 total 303.81 last 126.43) - open/close/unlink 45008 (time 1780086915.86 total 313.81 last 126.38) - open/close/unlink 46178 (time 1780086925.86 total 323.81 last 116.98) - open/close/unlink 47680 (time 1780086935.87 total 333.81 last 150.15) - open/close/unlink 49172 (time 1780086945.87 total 343.82 last 149.12) - open/close/unlink 50000 (time 1780086952.35 total 350.30 last 127.74) - open/close/unlink 51183 (time 1780086962.36 total 360.31 last 118.17) - open/close/unlink 52735 (time 1780086972.36 total 370.31 last 155.19) - open/close/unlink 54483 (time 1780086982.37 total 380.32 last 174.66) - open/close/unlink 56090 (time 1780086992.38 total 390.33 last 160.63) - open/close/unlink 57640 (time 1780087002.38 total 400.33 last 154.92) - open/close/unlink 59085 (time 1780087012.38 total 410.33 last 144.47) - open/close/unlink 60000 (time 1780087018.12 total 416.07 last 159.57) - open/close/unlink 61771 (time 1780087028.12 total 426.07 last 177.02) - open/close/unlink 63600 (time 1780087038.13 total 436.08 last 182.80) - open/close/unlink 64739 (time 1780087048.15 total 446.10 last 113.67) total: 64768 open/close/unlink in 446.65 seconds: 145.01 ops/second - open/close/unlink 1177 (time 1780086611.89 total 10.00 last 117.70) - open/close/unlink 2509 (time 1780086621.89 total 20.00 last 133.14) - open/close/unlink 3875 (time 1780086631.90 total 30.01 last 136.50) - open/close/unlink 5172 (time 1780086641.90 total 40.01 last 129.67) - open/close/unlink 6711 (time 1780086651.90 total 50.02 last 153.89) - open/close/unlink 8301 (time 1780086661.90 total 60.02 last 158.96) - open/close/unlink 9934 (time 1780086671.91 total 70.02 last 163.27) - open/close/unlink 11786 (time 1780086681.91 total 80.02 last 185.13) - open/close/unlink 13528 (time 1780086691.92 total 90.03 last 174.11) - open/close/unlink 14798 (time 1780086701.92 total 100.03 last 126.93) - open/close/unlink 16242 (time 1780086711.93 total 110.04 last 144.34) - open/close/unlink 17775 (time 1780086721.93 total 120.04 last 153.28) - open/close/unlink 18904 (time 1780086731.93 total 130.04 last 112.87) - open/close/unlink 20000 (time 1780086738.79 total 136.91 last 159.69) - open/close/unlink 21617 (time 1780086748.80 total 146.91 last 161.62) - open/close/unlink 23278 (time 1780086758.80 total 156.91 last 166.03) - open/close/unlink 24942 (time 1780086768.80 total 166.92 last 166.38) - open/close/unlink 26402 (time 1780086778.80 total 176.92 last 145.99) - open/close/unlink 27750 (time 1780086788.81 total 186.92 last 134.72) - open/close/unlink 28955 (time 1780086798.81 total 196.92 last 120.46) - open/close/unlink 30000 (time 1780086805.86 total 203.97 last 148.27) - open/close/unlink 31487 (time 1780086815.86 total 213.98 last 148.63) - open/close/unlink 32734 (time 1780086825.87 total 223.98 last 124.63) - open/close/unlink 34202 (time 1780086835.87 total 233.99 last 146.76) - open/close/unlink 35610 (time 1780086845.88 total 243.99 last 140.73) - open/close/unlink 37139 (time 1780086855.88 total 253.99 last 152.85) - open/close/unlink 38629 (time 1780086865.88 total 263.99 last 148.98) - open/close/unlink 39832 (time 1780086875.88 total 274.00 last 120.25) - open/close/unlink 41088 (time 1780086885.89 total 284.00 last 125.55) - open/close/unlink 42454 (time 1780086895.89 total 294.01 last 136.54) - open/close/unlink 43705 (time 1780086905.90 total 304.01 last 125.05) - open/close/unlink 44981 (time 1780086915.90 total 314.02 last 127.55) - open/close/unlink 46144 (time 1780086925.90 total 324.02 last 116.26) - open/close/unlink 47640 (time 1780086935.91 total 334.02 last 149.56) - open/close/unlink 49132 (time 1780086945.91 total 344.02 last 149.15) - open/close/unlink 50000 (time 1780086952.72 total 350.83 last 127.49) - open/close/unlink 51193 (time 1780086962.72 total 360.84 last 119.25) - open/close/unlink 52743 (time 1780086972.73 total 370.84 last 154.89) - open/close/unlink 54480 (time 1780086982.73 total 380.85 last 173.63) - open/close/unlink 56086 (time 1780086992.74 total 390.85 last 160.56) - open/close/unlink 57634 (time 1780087002.74 total 400.85 last 154.76) - open/close/unlink 59090 (time 1780087012.74 total 410.85 last 145.58) - open/close/unlink 60000 (time 1780087018.46 total 416.58 last 159.04) - open/close/unlink 61791 (time 1780087028.46 total 426.58 last 179.04) - open/close/unlink 63590 (time 1780087038.47 total 436.58 last 179.79) - open/close/unlink 64672 (time 1780087048.47 total 446.59 last 108.19) total: 64768 open/close/unlink in 447.93 seconds: 144.59 ops/second Waiting for MDT destroys to complete oleg124-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg124-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg124-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg124-server stop mds service on oleg124-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg124-server stop mds service on oleg124-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg124-server unloading modules via unload_modules_local on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing unload_modules_local oleg124-server: modules unloaded. pdsh@oleg124-client: oleg124-client: ssh exited with exit code 2 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 2 pdsh@oleg124-client: oleg124-client: ssh exited with exit code 2 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 2