== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:18:25 (1709785105) start mds service on oleg140-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg140-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg140-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg140-client.virtnet: -o user_xattr,flock oleg140-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 789 (time 1709785133.92 total 10.75 last 73.38) - open/close 1514 (time 1709785144.65 total 21.48 last 67.56) - open/close 2144 (time 1709785155.32 total 32.15 last 59.07) - open/close 2767 (time 1709785165.88 total 42.72 last 58.97) - open/close 3353 (time 1709785176.56 total 53.40 last 54.87) - open/close 3922 (time 1709785189.20 total 66.04 last 45.01) - open/close 4503 (time 1709785199.90 total 76.73 last 54.32) - open/close 5061 (time 1709785210.61 total 87.44 last 52.11) - open/close 5650 (time 1709785221.34 total 98.17 last 54.89) - open/close 6027 (time 1709785231.71 total 108.54 last 36.36) - open/close 6490 (time 1709785242.29 total 119.12 last 43.75) - open/close 6934 (time 1709785252.84 total 129.68 last 42.08) - open/close 7364 (time 1709785263.35 total 140.19 last 40.90) - open/close 7805 (time 1709785273.91 total 150.74 last 41.78) - open/close 8193 (time 1709785284.50 total 161.33 last 36.65) - open/close 8537 (time 1709785295.02 total 171.85 last 32.70) - open/close 8997 (time 1709785305.94 total 182.77 last 42.11) - open/close 9408 (time 1709785316.59 total 193.43 last 38.57) - open/close 9867 (time 1709785327.51 total 204.34 last 42.05) - open/close 10000 (time 1709785330.09 total 206.92 last 51.60) - open/close 10367 (time 1709785340.71 total 217.54 last 34.56) - open/close 10831 (time 1709785350.74 total 227.58 last 46.23) - open/close 11171 (time 1709785361.17 total 238.00 last 32.61) - open/close 13572 (time 1709785371.17 total 248.01 last 240.05) - open/close 15727 (time 1709785381.17 total 258.01 last 215.47) - open/close 17548 (time 1709785391.18 total 268.01 last 182.03) - open/close 18807 (time 1709785401.18 total 278.01 last 125.86) - open/close 20000 (time 1709785408.88 total 285.71 last 154.92) - open/close 21392 (time 1709785418.89 total 295.72 last 139.14) - open/close 22704 (time 1709785428.89 total 305.73 last 131.10) - open/close 23950 (time 1709785438.90 total 315.74 last 124.49) - open/close 25231 (time 1709785448.91 total 325.74 last 128.02) - open/close 26199 (time 1709785458.91 total 335.75 last 96.74) - open/close 27370 (time 1709785468.92 total 345.75 last 117.03) - open/close 28462 (time 1709785478.92 total 355.76 last 109.17) - open/close 29469 (time 1709785488.93 total 365.76 last 100.63) - open/close 30000 (time 1709785495.04 total 371.87 last 86.95) - open/close 30781 (time 1709785505.04 total 381.88 last 78.05) - open/close 31660 (time 1709785515.05 total 391.88 last 87.86) - open/close 32496 (time 1709785525.05 total 401.89 last 83.54) - open/close 33277 (time 1709785535.05 total 411.89 last 78.10) - open/close 34135 (time 1709785545.06 total 421.90 last 85.72) - open/close 34976 (time 1709785555.07 total 431.91 last 84.03) - open/close 35742 (time 1709785565.09 total 441.92 last 76.50) - open/close 36531 (time 1709785575.09 total 451.93 last 78.85) - open/close 37313 (time 1709785585.10 total 461.93 last 78.13) - open/close 38087 (time 1709785595.11 total 471.94 last 77.35) - open/close 38784 (time 1709785605.11 total 481.95 last 69.66) - open/close 39474 (time 1709785615.12 total 491.95 last 68.98) - open/close 40000 (time 1709785622.47 total 499.30 last 71.53) - open/close 40678 (time 1709785632.48 total 509.31 last 67.73) - open/close 41372 (time 1709785642.49 total 519.32 last 69.33) - open/close 42014 (time 1709785652.49 total 529.32 last 64.20) - open/close 42653 (time 1709785662.49 total 539.33 last 63.89) - open/close 43299 (time 1709785672.50 total 549.33 last 64.54) - open/close 43888 (time 1709785682.52 total 559.35 last 58.79) - open/close 44468 (time 1709785692.52 total 569.36 last 57.98) - open/close 45019 (time 1709785702.53 total 579.37 last 55.04) - open/close 45614 (time 1709785712.55 total 589.38 last 59.42) - open/close 46183 (time 1709785722.56 total 599.40 last 56.81) - open/close 46758 (time 1709785732.57 total 609.40 last 57.49) - open/close 47368 (time 1709785742.57 total 619.41 last 60.95) - open/close 47930 (time 1709785752.58 total 629.41 last 56.19) - open/close 48498 (time 1709785762.58 total 639.41 last 56.79) - open/close 49044 (time 1709785772.59 total 649.42 last 54.55) - open/close 49610 (time 1709785782.60 total 659.44 last 56.51) total: 50000 open/close in 666.60 seconds: 75.01 ops/second - unlinked 0 (time 1709785790 ; total 0 ; last 0) - unlinked 10000 (time 1709785990 ; total 200 ; last 200) - unlinked 20000 (time 1709786137 ; total 347 ; last 147) - unlinked 30000 (time 1709786233 ; total 443 ; last 96) - unlinked 40000 (time 1709786295 ; total 505 ; last 62) total: 50000 unlinks in 539 seconds: 92.764381 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg140-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg140-server unloading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2