== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 00:52:18 (1780807938) start mds service on oleg159-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg159-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-OST0000 oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92553 329 92224 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12329 425 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12233 329 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 1403 (time 1780807969.21 total 10.00 last 140.26) - open/close/unlink 2764 (time 1780807979.21 total 20.01 last 136.06) - open/close/unlink 4230 (time 1780807989.21 total 30.01 last 146.54) - open/close/unlink 5709 (time 1780807999.21 total 40.01 last 147.88) - open/close/unlink 7297 (time 1780808009.22 total 50.01 last 158.77) - open/close/unlink 8854 (time 1780808019.22 total 60.02 last 155.65) - open/close/unlink 10000 (time 1780808026.62 total 67.41 last 154.90) - open/close/unlink 11472 (time 1780808036.62 total 77.42 last 147.16) - open/close/unlink 13018 (time 1780808046.62 total 87.42 last 154.56) - open/close/unlink 14518 (time 1780808056.62 total 97.42 last 149.98) - open/close/unlink 16093 (time 1780808066.63 total 107.42 last 157.48) - open/close/unlink 17732 (time 1780808076.63 total 117.43 last 163.87) - open/close/unlink 19362 (time 1780808086.63 total 127.43 last 162.94) - open/close/unlink 20000 (time 1780808090.63 total 131.42 last 159.72) - open/close/unlink 21671 (time 1780808100.63 total 141.43 last 167.07) - open/close/unlink 23167 (time 1780808110.64 total 151.43 last 149.47) - open/close/unlink 24531 (time 1780808120.65 total 161.44 last 136.28) - open/close/unlink 25967 (time 1780808130.65 total 171.45 last 143.47) - open/close/unlink 27529 (time 1780808140.66 total 181.46 last 156.08) - open/close/unlink 29109 (time 1780808150.66 total 191.46 last 157.96) - open/close/unlink 30000 (time 1780808156.84 total 197.64 last 144.16) - open/close/unlink 31584 (time 1780808166.85 total 207.64 last 158.37) - open/close/unlink 33166 (time 1780808176.85 total 217.65 last 158.16) - open/close/unlink 34656 (time 1780808186.87 total 227.67 last 148.72) - open/close/unlink 36184 (time 1780808196.87 total 237.67 last 152.71) - open/close/unlink 37611 (time 1780808206.88 total 247.67 last 142.68) - open/close/unlink 38958 (time 1780808216.88 total 257.67 last 134.70) - open/close/unlink 40000 (time 1780808225.33 total 266.13 last 123.21) - open/close/unlink 41218 (time 1780808235.34 total 276.14 last 121.74) - open/close/unlink 42654 (time 1780808245.34 total 286.14 last 143.54) - open/close/unlink 44128 (time 1780808255.35 total 296.15 last 147.30) - open/close/unlink 45607 (time 1780808265.35 total 306.15 last 147.86) - open/close/unlink 47110 (time 1780808275.35 total 316.15 last 150.26) - open/close/unlink 48637 (time 1780808285.36 total 326.15 last 152.65) - open/close/unlink 50000 (time 1780808294.13 total 334.93 last 155.32) - open/close/unlink 51535 (time 1780808304.13 total 344.93 last 153.49) - open/close/unlink 53037 (time 1780808314.13 total 354.93 last 150.20) - open/close/unlink 54460 (time 1780808324.14 total 364.94 last 142.18) - open/close/unlink 56027 (time 1780808334.14 total 374.94 last 156.65) - open/close/unlink 57382 (time 1780808344.15 total 384.94 last 135.47) - open/close/unlink 58845 (time 1780808354.15 total 394.95 last 146.23) - open/close/unlink 60000 (time 1780808361.90 total 402.70 last 149.03) - open/close/unlink 61510 (time 1780808371.90 total 412.70 last 150.98) - open/close/unlink 63136 (time 1780808381.90 total 422.70 last 162.59) - open/close/unlink 64600 (time 1780808391.91 total 432.71 last 146.27) total: 64768 open/close/unlink in 433.92 seconds: 149.26 ops/second - open/close/unlink 1419 (time 1780807969.54 total 10.00 last 141.86) - open/close/unlink 2779 (time 1780807979.55 total 20.01 last 135.88) - open/close/unlink 4242 (time 1780807989.56 total 30.01 last 146.27) - open/close/unlink 5720 (time 1780807999.56 total 40.02 last 147.72) - open/close/unlink 7308 (time 1780808009.56 total 50.02 last 158.76) - open/close/unlink 8847 (time 1780808019.56 total 60.02 last 153.89) - open/close/unlink 10000 (time 1780808027.01 total 67.47 last 154.90) - open/close/unlink 11477 (time 1780808037.01 total 77.47 last 147.66) - open/close/unlink 13056 (time 1780808047.02 total 87.47 last 157.82) - open/close/unlink 14570 (time 1780808057.02 total 97.48 last 151.34) - open/close/unlink 16143 (time 1780808067.02 total 107.48 last 157.22) - open/close/unlink 17778 (time 1780808077.03 total 117.49 last 163.40) - open/close/unlink 19431 (time 1780808087.03 total 127.49 last 165.21) - open/close/unlink 20000 (time 1780808090.66 total 131.12 last 157.07) - open/close/unlink 21675 (time 1780808100.66 total 141.12 last 167.42) - open/close/unlink 23167 (time 1780808110.67 total 151.12 last 149.15) - open/close/unlink 24541 (time 1780808120.67 total 161.13 last 137.34) - open/close/unlink 25987 (time 1780808130.67 total 171.13 last 144.59) - open/close/unlink 27545 (time 1780808140.67 total 181.13 last 155.79) - open/close/unlink 29112 (time 1780808150.68 total 191.14 last 156.56) - open/close/unlink 30000 (time 1780808156.85 total 197.31 last 143.88) - open/close/unlink 31585 (time 1780808166.86 total 207.32 last 158.42) - open/close/unlink 33170 (time 1780808176.87 total 217.33 last 158.32) - open/close/unlink 34640 (time 1780808186.89 total 227.35 last 146.69) - open/close/unlink 36155 (time 1780808196.89 total 237.35 last 151.49) - open/close/unlink 37597 (time 1780808206.92 total 247.38 last 143.75) - open/close/unlink 38942 (time 1780808216.93 total 257.39 last 134.41) - open/close/unlink 40000 (time 1780808225.52 total 265.98 last 123.14) - open/close/unlink 41225 (time 1780808235.52 total 275.98 last 122.45) - open/close/unlink 42671 (time 1780808245.53 total 285.98 last 144.57) - open/close/unlink 44125 (time 1780808255.53 total 295.99 last 145.36) - open/close/unlink 45598 (time 1780808265.54 total 305.99 last 147.21) - open/close/unlink 47101 (time 1780808275.54 total 316.00 last 150.24) - open/close/unlink 48638 (time 1780808285.54 total 326.00 last 153.64) - open/close/unlink 50000 (time 1780808294.59 total 335.05 last 150.57) - open/close/unlink 51540 (time 1780808304.59 total 345.05 last 153.93) - open/close/unlink 53077 (time 1780808314.61 total 355.06 last 153.51) - open/close/unlink 54518 (time 1780808324.61 total 365.07 last 143.98) - open/close/unlink 56099 (time 1780808334.62 total 375.08 last 158.03) - open/close/unlink 57445 (time 1780808344.62 total 385.08 last 134.59) - open/close/unlink 58937 (time 1780808354.62 total 395.08 last 149.16) - open/close/unlink 60000 (time 1780808361.94 total 402.40 last 145.31) - open/close/unlink 61516 (time 1780808371.94 total 412.40 last 151.52) - open/close/unlink 63135 (time 1780808381.95 total 422.41 last 161.77) - open/close/unlink 64594 (time 1780808391.96 total 432.42 last 145.77) total: 64768 open/close/unlink in 433.73 seconds: 149.33 ops/second - open/close/unlink 1402 (time 1780807969.48 total 10.00 last 140.16) - open/close/unlink 2769 (time 1780807979.48 total 20.01 last 136.67) - open/close/unlink 4230 (time 1780807989.49 total 30.01 last 146.03) - open/close/unlink 5708 (time 1780807999.49 total 40.01 last 147.74) - open/close/unlink 7295 (time 1780808009.49 total 50.02 last 158.64) - open/close/unlink 8834 (time 1780808019.50 total 60.02 last 153.80) - open/close/unlink 10000 (time 1780808026.98 total 67.50 last 155.90) - open/close/unlink 11480 (time 1780808036.98 total 77.51 last 147.94) - open/close/unlink 13045 (time 1780808046.99 total 87.51 last 156.46) - open/close/unlink 14567 (time 1780808056.99 total 97.51 last 152.18) - open/close/unlink 16165 (time 1780808066.99 total 107.51 last 159.76) - open/close/unlink 17792 (time 1780808076.99 total 117.52 last 162.69) - open/close/unlink 19418 (time 1780808087.00 total 127.52 last 162.54) - open/close/unlink 20000 (time 1780808090.71 total 131.24 last 156.52) - open/close/unlink 21657 (time 1780808100.72 total 141.24 last 165.65) - open/close/unlink 23133 (time 1780808110.73 total 151.25 last 147.42) - open/close/unlink 24498 (time 1780808120.73 total 161.26 last 136.43) - open/close/unlink 25944 (time 1780808130.75 total 171.27 last 144.38) - open/close/unlink 27486 (time 1780808140.75 total 181.28 last 154.14) - open/close/unlink 29066 (time 1780808150.75 total 191.28 last 157.99) - open/close/unlink 30000 (time 1780808157.17 total 197.69 last 145.71) - open/close/unlink 31596 (time 1780808167.17 total 207.69 last 159.58) - open/close/unlink 33165 (time 1780808177.17 total 217.69 last 156.89) - open/close/unlink 34634 (time 1780808187.17 total 227.70 last 146.80) - open/close/unlink 36168 (time 1780808197.18 total 237.70 last 153.31) - open/close/unlink 37597 (time 1780808207.18 total 247.71 last 142.86) - open/close/unlink 38952 (time 1780808217.19 total 257.71 last 135.42) - open/close/unlink 40000 (time 1780808225.64 total 266.16 last 124.04) - open/close/unlink 41241 (time 1780808235.64 total 276.16 last 124.09) - open/close/unlink 42701 (time 1780808245.64 total 286.16 last 146.00) - open/close/unlink 44155 (time 1780808255.64 total 296.17 last 145.32) - open/close/unlink 45621 (time 1780808265.65 total 306.17 last 146.50) - open/close/unlink 47138 (time 1780808275.66 total 316.18 last 151.64) - open/close/unlink 48682 (time 1780808285.66 total 326.18 last 154.35) - open/close/unlink 50000 (time 1780808294.26 total 334.78 last 153.28) - open/close/unlink 51546 (time 1780808304.26 total 344.79 last 154.50) - open/close/unlink 53058 (time 1780808314.27 total 354.79 last 151.15) - open/close/unlink 54497 (time 1780808324.27 total 364.79 last 143.85) - open/close/unlink 56061 (time 1780808334.28 total 374.80 last 156.32) - open/close/unlink 57404 (time 1780808344.28 total 384.81 last 134.23) - open/close/unlink 58872 (time 1780808354.29 total 394.81 last 146.74) - open/close/unlink 60000 (time 1780808361.96 total 402.48 last 147.07) - open/close/unlink 61520 (time 1780808371.96 total 412.48 last 151.92) - open/close/unlink 63138 (time 1780808381.97 total 422.49 last 161.62) - open/close/unlink 64590 (time 1780808391.98 total 432.50 last 145.07) total: 64768 open/close/unlink in 433.90 seconds: 149.27 ops/second - open/close/unlink 1407 (time 1780807969.49 total 10.00 last 140.67) - open/close/unlink 2776 (time 1780807979.49 total 20.01 last 136.82) - open/close/unlink 4236 (time 1780807989.50 total 30.01 last 145.93) - open/close/unlink 5727 (time 1780807999.50 total 40.02 last 149.05) - open/close/unlink 7312 (time 1780808009.51 total 50.02 last 158.44) - open/close/unlink 8854 (time 1780808019.51 total 60.03 last 154.10) - open/close/unlink 10000 (time 1780808026.85 total 67.37 last 156.16) - open/close/unlink 11453 (time 1780808036.85 total 77.37 last 145.28) - open/close/unlink 13023 (time 1780808046.86 total 87.37 last 156.90) - open/close/unlink 14517 (time 1780808056.86 total 97.38 last 149.34) - open/close/unlink 16110 (time 1780808066.87 total 107.38 last 159.25) - open/close/unlink 17746 (time 1780808076.87 total 117.38 last 163.60) - open/close/unlink 19397 (time 1780808086.87 total 127.38 last 165.04) - open/close/unlink 20000 (time 1780808090.68 total 131.20 last 158.13) - open/close/unlink 21671 (time 1780808100.69 total 141.20 last 167.02) - open/close/unlink 23156 (time 1780808110.69 total 151.20 last 148.49) - open/close/unlink 24531 (time 1780808120.70 total 161.21 last 137.42) - open/close/unlink 25965 (time 1780808130.70 total 171.21 last 143.36) - open/close/unlink 27522 (time 1780808140.70 total 181.22 last 155.64) - open/close/unlink 29092 (time 1780808150.71 total 191.22 last 156.93) - open/close/unlink 30000 (time 1780808157.09 total 197.60 last 142.27) - open/close/unlink 31584 (time 1780808167.09 total 207.60 last 158.37) - open/close/unlink 33150 (time 1780808177.10 total 217.61 last 156.51) - open/close/unlink 34628 (time 1780808187.10 total 227.61 last 147.78) - open/close/unlink 36168 (time 1780808197.11 total 237.62 last 153.88) - open/close/unlink 37581 (time 1780808207.11 total 247.62 last 141.22) - open/close/unlink 38933 (time 1780808217.12 total 257.63 last 135.11) - open/close/unlink 40000 (time 1780808225.85 total 266.37 last 122.14) - open/close/unlink 41254 (time 1780808235.85 total 276.37 last 125.39) - open/close/unlink 42691 (time 1780808245.85 total 286.37 last 143.70) - open/close/unlink 44139 (time 1780808255.86 total 296.37 last 144.78) - open/close/unlink 45617 (time 1780808265.86 total 306.37 last 147.77) - open/close/unlink 47136 (time 1780808275.86 total 316.37 last 151.89) - open/close/unlink 48676 (time 1780808285.86 total 326.38 last 153.91) - open/close/unlink 50000 (time 1780808294.72 total 335.24 last 149.47) - open/close/unlink 51537 (time 1780808304.73 total 345.24 last 153.62) - open/close/unlink 53064 (time 1780808314.73 total 355.24 last 152.65) - open/close/unlink 54517 (time 1780808324.74 total 365.25 last 145.22) - open/close/unlink 56085 (time 1780808334.74 total 375.25 last 156.73) - open/close/unlink 57415 (time 1780808344.74 total 385.26 last 132.97) - open/close/unlink 58904 (time 1780808354.74 total 395.26 last 148.87) - open/close/unlink 60000 (time 1780808362.32 total 402.83 last 144.72) - open/close/unlink 61492 (time 1780808372.32 total 412.84 last 149.13) - open/close/unlink 63136 (time 1780808382.33 total 422.84 last 164.32) - open/close/unlink 64573 (time 1780808392.33 total 432.85 last 143.62) total: 64768 open/close/unlink in 434.64 seconds: 149.01 ops/second - open/close/unlink 1407 (time 1780807969.51 total 10.00 last 140.70) - open/close/unlink 2778 (time 1780807979.52 total 20.01 last 136.97) - open/close/unlink 4240 (time 1780807989.52 total 30.01 last 146.14) - open/close/unlink 5719 (time 1780807999.53 total 40.02 last 147.80) - open/close/unlink 7304 (time 1780808009.53 total 50.03 last 158.42) - open/close/unlink 8837 (time 1780808019.54 total 60.03 last 153.23) - open/close/unlink 10000 (time 1780808027.09 total 67.58 last 154.06) - open/close/unlink 11476 (time 1780808037.09 total 77.58 last 147.60) - open/close/unlink 13044 (time 1780808047.09 total 87.58 last 156.74) - open/close/unlink 14555 (time 1780808057.09 total 97.59 last 151.07) - open/close/unlink 16131 (time 1780808067.10 total 107.59 last 157.53) - open/close/unlink 17780 (time 1780808077.11 total 117.60 last 164.79) - open/close/unlink 19432 (time 1780808087.11 total 127.60 last 165.12) - open/close/unlink 20000 (time 1780808090.75 total 131.24 last 156.05) - open/close/unlink 21657 (time 1780808100.75 total 141.25 last 165.63) - open/close/unlink 23129 (time 1780808110.76 total 151.25 last 147.16) - open/close/unlink 24497 (time 1780808120.76 total 161.25 last 136.73) - open/close/unlink 25939 (time 1780808130.77 total 171.26 last 144.06) - open/close/unlink 27492 (time 1780808140.78 total 181.27 last 155.24) - open/close/unlink 29074 (time 1780808150.78 total 191.27 last 158.13) - open/close/unlink 30000 (time 1780808157.16 total 197.65 last 145.12) - open/close/unlink 31599 (time 1780808167.17 total 207.66 last 159.83) - open/close/unlink 33148 (time 1780808177.18 total 217.67 last 154.73) - open/close/unlink 34625 (time 1780808187.19 total 227.68 last 147.51) - open/close/unlink 36143 (time 1780808197.20 total 237.69 last 151.68) - open/close/unlink 37578 (time 1780808207.21 total 247.70 last 143.37) - open/close/unlink 38934 (time 1780808217.21 total 257.70 last 135.54) - open/close/unlink 40000 (time 1780808225.89 total 266.38 last 122.86) - open/close/unlink 41255 (time 1780808235.89 total 276.38 last 125.43) - open/close/unlink 42688 (time 1780808245.89 total 286.39 last 143.27) - open/close/unlink 44162 (time 1780808255.89 total 296.39 last 147.39) - open/close/unlink 45630 (time 1780808265.90 total 306.39 last 146.76) - open/close/unlink 47150 (time 1780808275.90 total 316.39 last 151.93) - open/close/unlink 48692 (time 1780808285.90 total 326.39 last 154.19) - open/close/unlink 50000 (time 1780808294.74 total 335.23 last 147.96) - open/close/unlink 51538 (time 1780808304.75 total 345.24 last 153.72) - open/close/unlink 53055 (time 1780808314.75 total 355.24 last 151.67) - open/close/unlink 54496 (time 1780808324.76 total 365.25 last 143.97) - open/close/unlink 56068 (time 1780808334.76 total 375.25 last 157.14) - open/close/unlink 57399 (time 1780808344.76 total 385.26 last 133.07) - open/close/unlink 58890 (time 1780808354.77 total 395.26 last 149.08) - open/close/unlink 60000 (time 1780808362.38 total 402.87 last 145.86) - open/close/unlink 61512 (time 1780808372.38 total 412.87 last 151.17) - open/close/unlink 63135 (time 1780808382.38 total 422.87 last 162.23) - open/close/unlink 64568 (time 1780808392.39 total 432.88 last 143.25) total: 64768 open/close/unlink in 434.85 seconds: 148.94 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete oleg159-server: cannot open 'lustre-ost2': no such pool pdsh@oleg159-client: oleg159-server: ssh exited with exit code 255 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg159-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server unloading modules via unload_modules_local on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local oleg159-server: modules unloaded. pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2