== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:46:41 (1779126401) start mds service on oleg455-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg455-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Started lustre-OST0000 oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg455-client.virtnet: -o user_xattr,flock 192.168.204.155@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92553 329 92224 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12329 425 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12233 329 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 1425 (time 1779126428.46 total 10.01 last 142.43) - open/close/unlink 2800 (time 1779126438.46 total 20.01 last 137.43) - open/close/unlink 4226 (time 1779126448.46 total 30.01 last 142.59) - open/close/unlink 5740 (time 1779126458.46 total 40.01 last 151.38) - open/close/unlink 7162 (time 1779126468.46 total 50.01 last 142.17) - open/close/unlink 8525 (time 1779126478.47 total 60.02 last 136.26) - open/close/unlink 9836 (time 1779126488.47 total 70.02 last 131.09) - open/close/unlink 10942 (time 1779126498.47 total 80.02 last 110.58) - open/close/unlink 12275 (time 1779126508.48 total 90.03 last 133.22) - open/close/unlink 13769 (time 1779126518.48 total 100.03 last 149.32) - open/close/unlink 15135 (time 1779126528.48 total 110.03 last 136.59) - open/close/unlink 16592 (time 1779126538.49 total 120.04 last 145.62) - open/close/unlink 18032 (time 1779126548.50 total 130.05 last 143.84) - open/close/unlink 19397 (time 1779126558.51 total 140.06 last 136.34) - open/close/unlink 20000 (time 1779126562.92 total 144.47 last 136.88) - open/close/unlink 21222 (time 1779126572.92 total 154.47 last 122.15) - open/close/unlink 22482 (time 1779126582.93 total 164.47 last 125.93) - open/close/unlink 23773 (time 1779126592.93 total 174.48 last 129.07) - open/close/unlink 25202 (time 1779126602.93 total 184.48 last 142.90) - open/close/unlink 26634 (time 1779126612.93 total 194.48 last 143.15) - open/close/unlink 28012 (time 1779126622.94 total 204.48 last 137.74) - open/close/unlink 29133 (time 1779126632.94 total 214.49 last 112.04) - open/close/unlink 30000 (time 1779126640.16 total 221.71 last 120.13) - open/close/unlink 31208 (time 1779126650.16 total 231.71 last 120.75) - open/close/unlink 32423 (time 1779126660.17 total 241.72 last 121.45) - open/close/unlink 33988 (time 1779126670.17 total 251.72 last 156.43) - open/close/unlink 35259 (time 1779126680.17 total 261.72 last 127.07) - open/close/unlink 36449 (time 1779126690.17 total 271.72 last 118.98) - open/close/unlink 37516 (time 1779126700.18 total 281.73 last 106.65) - open/close/unlink 38764 (time 1779126710.18 total 291.73 last 124.76) - open/close/unlink 40000 (time 1779126718.57 total 300.12 last 147.29) - open/close/unlink 41408 (time 1779126728.58 total 310.13 last 140.74) - open/close/unlink 42976 (time 1779126738.59 total 320.14 last 156.66) - open/close/unlink 44612 (time 1779126748.59 total 330.14 last 163.60) - open/close/unlink 46016 (time 1779126758.59 total 340.14 last 140.34) - open/close/unlink 47165 (time 1779126768.60 total 350.15 last 114.80) - open/close/unlink 48363 (time 1779126778.64 total 360.19 last 119.30) - open/close/unlink 49413 (time 1779126788.65 total 370.20 last 104.95) - open/close/unlink 50000 (time 1779126794.88 total 376.43 last 94.25) - open/close/unlink 51275 (time 1779126804.88 total 386.43 last 127.46) - open/close/unlink 52270 (time 1779126814.88 total 396.43 last 99.47) - open/close/unlink 53544 (time 1779126824.89 total 406.44 last 127.24) - open/close/unlink 54810 (time 1779126834.90 total 416.45 last 126.50) - open/close/unlink 55903 (time 1779126844.91 total 426.46 last 109.26) - open/close/unlink 57193 (time 1779126854.91 total 436.46 last 128.94) - open/close/unlink 58473 (time 1779126864.91 total 446.46 last 127.95) - open/close/unlink 59806 (time 1779126874.91 total 456.46 last 133.30) - open/close/unlink 61109 (time 1779126884.92 total 466.47 last 130.28) - open/close/unlink 62747 (time 1779126894.92 total 476.47 last 163.75) - open/close/unlink 64433 (time 1779126904.92 total 486.47 last 168.55) total: 64768 open/close/unlink in 488.48 seconds: 132.59 ops/second - open/close/unlink 1430 (time 1779126428.77 total 10.01 last 142.92) - open/close/unlink 2799 (time 1779126438.78 total 20.01 last 136.87) - open/close/unlink 4223 (time 1779126448.78 total 30.01 last 142.35) - open/close/unlink 5749 (time 1779126458.78 total 40.02 last 152.52) - open/close/unlink 7181 (time 1779126468.79 total 50.02 last 143.11) - open/close/unlink 8545 (time 1779126478.80 total 60.03 last 136.28) - open/close/unlink 9849 (time 1779126488.80 total 70.03 last 130.37) - open/close/unlink 10956 (time 1779126498.81 total 80.04 last 110.66) - open/close/unlink 12292 (time 1779126508.81 total 90.04 last 133.56) - open/close/unlink 13773 (time 1779126518.81 total 100.04 last 148.06) - open/close/unlink 15130 (time 1779126528.81 total 110.05 last 135.68) - open/close/unlink 16591 (time 1779126538.82 total 120.06 last 145.95) - open/close/unlink 18042 (time 1779126548.83 total 130.06 last 145.03) - open/close/unlink 19397 (time 1779126558.83 total 140.06 last 135.46) - open/close/unlink 20000 (time 1779126563.20 total 144.44 last 137.93) - open/close/unlink 21215 (time 1779126573.21 total 154.44 last 121.45) - open/close/unlink 22496 (time 1779126583.21 total 164.45 last 128.01) - open/close/unlink 23797 (time 1779126593.22 total 174.45 last 130.00) - open/close/unlink 25236 (time 1779126603.23 total 184.46 last 143.86) - open/close/unlink 26674 (time 1779126613.23 total 194.46 last 143.77) - open/close/unlink 28048 (time 1779126623.23 total 204.46 last 137.38) - open/close/unlink 29163 (time 1779126633.23 total 214.46 last 111.47) - open/close/unlink 30000 (time 1779126640.29 total 221.52 last 118.61) - open/close/unlink 31218 (time 1779126650.29 total 231.52 last 121.79) - open/close/unlink 32437 (time 1779126660.29 total 241.52 last 121.86) - open/close/unlink 34024 (time 1779126670.29 total 251.52 last 158.68) - open/close/unlink 35284 (time 1779126680.30 total 261.53 last 125.95) - open/close/unlink 36467 (time 1779126690.30 total 271.53 last 118.26) - open/close/unlink 37545 (time 1779126700.31 total 281.54 last 107.74) - open/close/unlink 38805 (time 1779126710.31 total 291.54 last 126.00) - open/close/unlink 40000 (time 1779126718.47 total 299.70 last 146.38) - open/close/unlink 41414 (time 1779126728.47 total 309.70 last 141.37) - open/close/unlink 42985 (time 1779126738.48 total 319.71 last 157.02) - open/close/unlink 44627 (time 1779126748.49 total 329.72 last 164.04) - open/close/unlink 46040 (time 1779126758.49 total 339.72 last 141.28) - open/close/unlink 47195 (time 1779126768.49 total 349.72 last 115.50) - open/close/unlink 48391 (time 1779126778.49 total 359.72 last 119.59) - open/close/unlink 49431 (time 1779126788.49 total 369.72 last 103.99) - open/close/unlink 50000 (time 1779126794.47 total 375.71 last 95.06) - open/close/unlink 51292 (time 1779126804.48 total 385.71 last 129.11) - open/close/unlink 52292 (time 1779126814.49 total 395.72 last 99.91) - open/close/unlink 53542 (time 1779126824.50 total 405.73 last 124.93) - open/close/unlink 54814 (time 1779126834.50 total 415.73 last 127.19) - open/close/unlink 55880 (time 1779126844.50 total 425.73 last 106.56) - open/close/unlink 57153 (time 1779126854.51 total 435.74 last 127.20) - open/close/unlink 58428 (time 1779126864.51 total 445.75 last 127.44) - open/close/unlink 59769 (time 1779126874.52 total 455.75 last 134.08) - open/close/unlink 61061 (time 1779126884.52 total 465.75 last 129.14) - open/close/unlink 62677 (time 1779126894.52 total 475.75 last 161.58) - open/close/unlink 64359 (time 1779126904.52 total 485.75 last 168.17) total: 64768 open/close/unlink in 488.28 seconds: 132.64 ops/second - open/close/unlink 1423 (time 1779126428.73 total 10.00 last 142.23) - open/close/unlink 2796 (time 1779126438.73 total 20.01 last 137.27) - open/close/unlink 4220 (time 1779126448.74 total 30.01 last 142.31) - open/close/unlink 5734 (time 1779126458.74 total 40.01 last 151.39) - open/close/unlink 7157 (time 1779126468.74 total 50.02 last 142.24) - open/close/unlink 8533 (time 1779126478.75 total 60.02 last 137.56) - open/close/unlink 9840 (time 1779126488.75 total 70.02 last 130.65) - open/close/unlink 10934 (time 1779126498.76 total 80.03 last 109.32) - open/close/unlink 12251 (time 1779126508.76 total 90.03 last 131.69) - open/close/unlink 13735 (time 1779126518.76 total 100.03 last 148.39) - open/close/unlink 15103 (time 1779126528.78 total 110.05 last 136.55) - open/close/unlink 16556 (time 1779126538.79 total 120.06 last 145.17) - open/close/unlink 18008 (time 1779126548.80 total 130.07 last 145.02) - open/close/unlink 19363 (time 1779126558.81 total 140.08 last 135.34) - open/close/unlink 20000 (time 1779126563.47 total 144.75 last 136.57) - open/close/unlink 21222 (time 1779126573.48 total 154.75 last 122.17) - open/close/unlink 22491 (time 1779126583.48 total 164.75 last 126.88) - open/close/unlink 23793 (time 1779126593.48 total 174.76 last 130.15) - open/close/unlink 25242 (time 1779126603.48 total 184.76 last 144.87) - open/close/unlink 26670 (time 1779126613.49 total 194.76 last 142.73) - open/close/unlink 28042 (time 1779126623.49 total 204.77 last 137.16) - open/close/unlink 29170 (time 1779126633.50 total 214.77 last 112.72) - open/close/unlink 30000 (time 1779126640.63 total 221.90 last 116.45) - open/close/unlink 31249 (time 1779126650.63 total 231.90 last 124.85) - open/close/unlink 32456 (time 1779126660.63 total 241.91 last 120.69) - open/close/unlink 34053 (time 1779126670.63 total 251.91 last 159.67) - open/close/unlink 35312 (time 1779126680.63 total 261.91 last 125.90) - open/close/unlink 36491 (time 1779126690.64 total 271.91 last 117.87) - open/close/unlink 37558 (time 1779126700.64 total 281.91 last 106.67) - open/close/unlink 38818 (time 1779126710.64 total 291.92 last 125.95) - open/close/unlink 40000 (time 1779126718.63 total 299.90 last 148.00) - open/close/unlink 41418 (time 1779126728.63 total 309.91 last 141.75) - open/close/unlink 42985 (time 1779126738.64 total 319.91 last 156.61) - open/close/unlink 44624 (time 1779126748.64 total 329.92 last 163.85) - open/close/unlink 46043 (time 1779126758.65 total 339.93 last 141.76) - open/close/unlink 47183 (time 1779126768.67 total 349.94 last 113.79) - open/close/unlink 48385 (time 1779126778.68 total 359.95 last 120.11) - open/close/unlink 49434 (time 1779126788.68 total 369.95 last 104.89) - open/close/unlink 50000 (time 1779126794.78 total 376.05 last 92.78) - open/close/unlink 51274 (time 1779126804.78 total 386.05 last 127.40) - open/close/unlink 52270 (time 1779126814.78 total 396.06 last 99.57) - open/close/unlink 53535 (time 1779126824.79 total 406.06 last 126.43) - open/close/unlink 54798 (time 1779126834.80 total 416.07 last 126.18) - open/close/unlink 55879 (time 1779126844.80 total 426.08 last 108.03) - open/close/unlink 57159 (time 1779126854.81 total 436.08 last 127.96) - open/close/unlink 58442 (time 1779126864.81 total 446.09 last 128.25) - open/close/unlink 59777 (time 1779126874.81 total 456.09 last 133.50) - open/close/unlink 60000 (time 1779126876.85 total 458.12 last 109.53) - open/close/unlink 61379 (time 1779126886.86 total 468.13 last 137.78) - open/close/unlink 63068 (time 1779126896.86 total 478.13 last 168.84) - open/close/unlink 64725 (time 1779126906.87 total 488.14 last 165.59) total: 64768 open/close/unlink in 488.81 seconds: 132.50 ops/second - open/close/unlink 1430 (time 1779126428.75 total 10.00 last 142.99) - open/close/unlink 2804 (time 1779126438.75 total 20.00 last 137.40) - open/close/unlink 4238 (time 1779126448.76 total 30.01 last 143.30) - open/close/unlink 5764 (time 1779126458.77 total 40.02 last 152.47) - open/close/unlink 7195 (time 1779126468.77 total 50.02 last 143.05) - open/close/unlink 8565 (time 1779126478.78 total 60.03 last 136.91) - open/close/unlink 9876 (time 1779126488.79 total 70.03 last 131.00) - open/close/unlink 10974 (time 1779126498.79 total 80.04 last 109.76) - open/close/unlink 12303 (time 1779126508.79 total 90.04 last 132.87) - open/close/unlink 13772 (time 1779126518.79 total 100.04 last 146.87) - open/close/unlink 15140 (time 1779126528.80 total 110.05 last 136.74) - open/close/unlink 16577 (time 1779126538.80 total 120.05 last 143.59) - open/close/unlink 18026 (time 1779126548.81 total 130.06 last 144.80) - open/close/unlink 19402 (time 1779126558.81 total 140.06 last 137.57) - open/close/unlink 20000 (time 1779126563.12 total 144.37 last 138.87) - open/close/unlink 21215 (time 1779126573.12 total 154.37 last 121.48) - open/close/unlink 22468 (time 1779126583.13 total 164.38 last 125.22) - open/close/unlink 23756 (time 1779126593.13 total 174.38 last 128.73) - open/close/unlink 25209 (time 1779126603.14 total 184.38 last 145.28) - open/close/unlink 26638 (time 1779126613.14 total 194.39 last 142.86) - open/close/unlink 28026 (time 1779126623.14 total 204.39 last 138.75) - open/close/unlink 29148 (time 1779126633.14 total 214.39 last 112.19) - open/close/unlink 30000 (time 1779126640.28 total 221.53 last 119.31) - open/close/unlink 31217 (time 1779126650.28 total 231.53 last 121.69) - open/close/unlink 32420 (time 1779126660.29 total 241.54 last 120.24) - open/close/unlink 34007 (time 1779126670.29 total 251.54 last 158.70) - open/close/unlink 35271 (time 1779126680.29 total 261.54 last 126.38) - open/close/unlink 36438 (time 1779126690.29 total 271.54 last 116.67) - open/close/unlink 37510 (time 1779126700.29 total 281.54 last 107.19) - open/close/unlink 38770 (time 1779126710.30 total 291.54 last 125.99) - open/close/unlink 40000 (time 1779126718.60 total 299.85 last 148.13) - open/close/unlink 41413 (time 1779126728.60 total 309.85 last 141.24) - open/close/unlink 42978 (time 1779126738.61 total 319.86 last 156.40) - open/close/unlink 44615 (time 1779126748.61 total 329.86 last 163.70) - open/close/unlink 46028 (time 1779126758.61 total 339.86 last 141.26) - open/close/unlink 47169 (time 1779126768.62 total 349.87 last 114.03) - open/close/unlink 48360 (time 1779126778.64 total 359.89 last 118.88) - open/close/unlink 49407 (time 1779126788.64 total 369.89 last 104.65) - open/close/unlink 50000 (time 1779126794.90 total 376.15 last 94.77) - open/close/unlink 51271 (time 1779126804.90 total 386.15 last 127.07) - open/close/unlink 52277 (time 1779126814.91 total 396.16 last 100.51) - open/close/unlink 53565 (time 1779126824.93 total 406.18 last 128.59) - open/close/unlink 54839 (time 1779126834.95 total 416.20 last 127.07) - open/close/unlink 55935 (time 1779126844.96 total 426.20 last 109.57) - open/close/unlink 57213 (time 1779126854.96 total 436.21 last 127.72) - open/close/unlink 58488 (time 1779126864.97 total 446.22 last 127.43) - open/close/unlink 59796 (time 1779126874.97 total 456.22 last 130.78) - open/close/unlink 61103 (time 1779126884.97 total 466.22 last 130.66) - open/close/unlink 62743 (time 1779126894.97 total 476.22 last 164.00) - open/close/unlink 64410 (time 1779126904.98 total 486.23 last 166.65) total: 64768 open/close/unlink in 488.79 seconds: 132.51 ops/second - open/close/unlink 1424 (time 1779126428.80 total 10.00 last 142.36) - open/close/unlink 2797 (time 1779126438.80 total 20.00 last 137.30) - open/close/unlink 4230 (time 1779126448.80 total 30.00 last 143.28) - open/close/unlink 5736 (time 1779126458.80 total 40.01 last 150.58) - open/close/unlink 7156 (time 1779126468.80 total 50.01 last 141.98) - open/close/unlink 8516 (time 1779126478.81 total 60.01 last 135.94) - open/close/unlink 9817 (time 1779126488.81 total 70.02 last 130.01) - open/close/unlink 10918 (time 1779126498.82 total 80.02 last 110.09) - open/close/unlink 12261 (time 1779126508.82 total 90.02 last 134.26) - open/close/unlink 13734 (time 1779126518.83 total 100.03 last 147.19) - open/close/unlink 15094 (time 1779126528.83 total 110.04 last 135.90) - open/close/unlink 16531 (time 1779126538.83 total 120.04 last 143.69) - open/close/unlink 17977 (time 1779126548.83 total 130.04 last 144.60) - open/close/unlink 19331 (time 1779126558.84 total 140.05 last 135.27) - open/close/unlink 20000 (time 1779126563.80 total 145.01 last 134.91) - open/close/unlink 21242 (time 1779126573.80 total 155.01 last 124.19) - open/close/unlink 22522 (time 1779126583.81 total 165.01 last 127.98) - open/close/unlink 23793 (time 1779126593.81 total 175.01 last 127.09) - open/close/unlink 25245 (time 1779126603.81 total 185.02 last 145.12) - open/close/unlink 26675 (time 1779126613.82 total 195.02 last 142.93) - open/close/unlink 28047 (time 1779126623.82 total 205.02 last 137.19) - open/close/unlink 29163 (time 1779126633.83 total 215.03 last 111.50) - open/close/unlink 30000 (time 1779126641.39 total 222.60 last 110.65) - open/close/unlink 31251 (time 1779126651.39 total 232.60 last 125.06) - open/close/unlink 32482 (time 1779126661.40 total 242.60 last 123.05) - open/close/unlink 34077 (time 1779126671.41 total 252.61 last 159.39) - open/close/unlink 35332 (time 1779126681.41 total 262.61 last 125.44) - open/close/unlink 36500 (time 1779126691.42 total 272.62 last 116.73) - open/close/unlink 37568 (time 1779126701.42 total 282.62 last 106.76) - open/close/unlink 38838 (time 1779126711.42 total 292.62 last 126.99) - open/close/unlink 40000 (time 1779126719.30 total 300.51 last 147.45) - open/close/unlink 41410 (time 1779126729.30 total 310.51 last 141.00) - open/close/unlink 42972 (time 1779126739.30 total 320.51 last 156.17) - open/close/unlink 44593 (time 1779126749.31 total 330.51 last 162.06) - open/close/unlink 45967 (time 1779126759.31 total 340.51 last 137.39) - open/close/unlink 47121 (time 1779126769.31 total 350.51 last 115.36) - open/close/unlink 48304 (time 1779126779.34 total 360.54 last 117.95) - open/close/unlink 49355 (time 1779126789.35 total 370.55 last 104.98) - open/close/unlink 50000 (time 1779126795.53 total 376.74 last 104.32) - open/close/unlink 51231 (time 1779126805.54 total 386.74 last 123.03) - open/close/unlink 52222 (time 1779126815.54 total 396.75 last 99.05) - open/close/unlink 53480 (time 1779126825.55 total 406.75 last 125.72) - open/close/unlink 54728 (time 1779126835.55 total 416.76 last 124.76) - open/close/unlink 55838 (time 1779126845.56 total 426.76 last 110.91) - open/close/unlink 57127 (time 1779126855.56 total 436.77 last 128.86) - open/close/unlink 58410 (time 1779126865.57 total 446.77 last 128.23) - open/close/unlink 59738 (time 1779126875.57 total 456.78 last 132.74) - open/close/unlink 60000 (time 1779126877.95 total 459.15 last 110.44) - open/close/unlink 61382 (time 1779126887.95 total 469.15 last 138.14) - open/close/unlink 63066 (time 1779126897.95 total 479.16 last 168.35) - open/close/unlink 64632 (time 1779126907.96 total 489.16 last 156.51) total: 64768 open/close/unlink in 490.98 seconds: 131.92 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete oleg455-server: cannot open 'lustre-ost2': no such pool pdsh@oleg455-client: oleg455-server: ssh exited with exit code 255 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg455-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg455-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg455-server stop mds service on oleg455-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg455-server unloading modules via unload_modules_local on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing unload_modules_local oleg455-server: modules unloaded. pdsh@oleg455-client: oleg455-client: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-client: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2