== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:16:14 (1709784974) start mds service on oleg346-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg346-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg346-client.virtnet: -o user_xattr,flock oleg346-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 883 (time 1709784999.48 total 10.78 last 81.90) - open/close 1406 (time 1709785009.60 total 20.90 last 51.68) - open/close 2010 (time 1709785019.98 total 31.28 last 58.19) - open/close 2654 (time 1709785030.50 total 41.80 last 61.20) - open/close 3206 (time 1709785040.84 total 52.14 last 53.43) - open/close 3766 (time 1709785051.27 total 62.57 last 53.69) - open/close 4274 (time 1709785061.61 total 72.90 last 49.14) - open/close 4849 (time 1709785072.25 total 83.55 last 54.02) - open/close 5371 (time 1709785082.70 total 94.00 last 49.95) - open/close 5818 (time 1709785093.01 total 104.31 last 43.35) - open/close 6244 (time 1709785103.33 total 114.63 last 41.29) - open/close 6721 (time 1709785113.82 total 125.12 last 45.46) - open/close 7202 (time 1709785124.55 total 135.85 last 44.84) - open/close 7622 (time 1709785134.93 total 146.23 last 40.46) - open/close 8028 (time 1709785145.43 total 156.73 last 38.67) - open/close 8453 (time 1709785156.14 total 167.44 last 39.67) - open/close 8861 (time 1709785166.70 total 178.00 last 38.64) - open/close 9301 (time 1709785176.70 total 188.00 last 44.00) - open/close 9755 (time 1709785186.71 total 198.01 last 45.38) - open/close 10000 (time 1709785192.73 total 204.03 last 40.69) - open/close 10464 (time 1709785202.73 total 214.03 last 46.39) - open/close 10821 (time 1709785213.31 total 224.61 last 33.76) - open/close 11239 (time 1709785223.37 total 234.67 last 41.52) - open/close 13400 (time 1709785233.38 total 244.68 last 216.03) - open/close 15235 (time 1709785243.38 total 254.68 last 183.45) - open/close 16784 (time 1709785253.38 total 264.68 last 154.87) - open/close 18280 (time 1709785263.39 total 274.69 last 149.50) - open/close 19668 (time 1709785273.39 total 284.69 last 138.80) - open/close 20000 (time 1709785276.24 total 287.54 last 116.25) - open/close 21140 (time 1709785286.25 total 297.55 last 113.95) - open/close 22334 (time 1709785296.26 total 307.55 last 119.32) - open/close 23520 (time 1709785306.26 total 317.56 last 118.55) - open/close 24684 (time 1709785316.27 total 327.57 last 116.30) - open/close 25775 (time 1709785326.27 total 337.57 last 109.05) - open/close 26808 (time 1709785336.28 total 347.58 last 103.22) - open/close 27834 (time 1709785346.29 total 357.59 last 102.54) - open/close 28782 (time 1709785356.29 total 367.59 last 94.74) - open/close 29722 (time 1709785366.29 total 377.59 last 93.99) - open/close 30000 (time 1709785369.49 total 380.79 last 87.07) - open/close 30873 (time 1709785379.49 total 390.79 last 87.28) - open/close 31676 (time 1709785389.49 total 400.79 last 80.28) - open/close 32456 (time 1709785399.50 total 410.80 last 77.96) - open/close 33139 (time 1709785409.51 total 420.81 last 68.22) - open/close 33832 (time 1709785419.52 total 430.81 last 69.25) - open/close 34513 (time 1709785429.53 total 440.82 last 68.03) - open/close 35197 (time 1709785439.54 total 450.84 last 68.31) - open/close 35865 (time 1709785449.54 total 460.84 last 66.77) - open/close 36519 (time 1709785459.56 total 470.86 last 65.27) - open/close 37166 (time 1709785469.58 total 480.88 last 64.60) - open/close 37801 (time 1709785479.58 total 490.88 last 63.48) - open/close 38390 (time 1709785489.58 total 500.88 last 58.88) - open/close 38969 (time 1709785499.60 total 510.90 last 57.82) - open/close 39539 (time 1709785509.61 total 520.91 last 56.95) - open/close 40000 (time 1709785518.01 total 529.31 last 54.88) - open/close 40552 (time 1709785528.02 total 539.32 last 55.13) - open/close 41113 (time 1709785538.03 total 549.32 last 56.08) - open/close 41650 (time 1709785548.03 total 559.33 last 53.67) - open/close 42206 (time 1709785558.04 total 569.34 last 55.54) - open/close 42717 (time 1709785568.05 total 579.35 last 51.06) - open/close 43275 (time 1709785578.06 total 589.35 last 55.77) - open/close 43812 (time 1709785588.08 total 599.38 last 53.56) - open/close 44334 (time 1709785598.10 total 609.39 last 52.13) - open/close 44854 (time 1709785608.10 total 619.40 last 51.98) - open/close 45365 (time 1709785618.11 total 629.41 last 51.06) - open/close 45843 (time 1709785628.12 total 639.42 last 47.73) - open/close 46294 (time 1709785638.13 total 649.43 last 45.05) - open/close 46766 (time 1709785648.15 total 659.45 last 47.14) - open/close 47242 (time 1709785658.15 total 669.45 last 47.57) - open/close 47732 (time 1709785668.17 total 679.47 last 48.92) - open/close 48214 (time 1709785678.19 total 689.48 last 48.13) - open/close 48664 (time 1709785688.20 total 699.50 last 44.94) - open/close 49119 (time 1709785698.21 total 709.51 last 45.43) - open/close 49497 (time 1709785708.23 total 719.53 last 37.74) - open/close 49906 (time 1709785718.23 total 729.53 last 40.90) total: 50000 open/close in 731.76 seconds: 68.33 ops/second - unlinked 0 (time 1709785721 ; total 0 ; last 0) - unlinked 10000 (time 1709785965 ; total 244 ; last 244) - unlinked 20000 (time 1709786150 ; total 429 ; last 185) - unlinked 30000 (time 1709786272 ; total 551 ; last 122) - unlinked 40000 (time 1709786368 ; total 647 ; last 96) total: 50000 unlinks in 688 seconds: 72.674416 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg346-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server stop mds service on oleg346-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server unloading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2