== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:17:28 (1709785048) start mds service on oleg451-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg451-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock oleg451-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 544 (time 1709785073.54 total 10.24 last 53.11) - open/close 936 (time 1709785083.65 total 20.36 last 38.75) - open/close 1388 (time 1709785093.65 total 30.36 last 45.20) - open/close 1868 (time 1709785103.81 total 40.52 last 47.25) - open/close 2502 (time 1709785114.29 total 51.00 last 60.50) - open/close 3075 (time 1709785124.71 total 61.42 last 54.98) - open/close 3667 (time 1709785135.36 total 72.06 last 55.63) - open/close 4275 (time 1709785146.00 total 82.70 last 57.12) - open/close 4812 (time 1709785156.67 total 93.37 last 50.34) - open/close 5334 (time 1709785167.25 total 103.96 last 49.30) - open/close 5733 (time 1709785177.59 total 114.29 last 38.60) - open/close 6154 (time 1709785188.24 total 124.94 last 39.55) - open/close 6483 (time 1709785198.43 total 135.14 last 32.27) - open/close 6773 (time 1709785208.54 total 145.24 last 28.70) - open/close 7239 (time 1709785219.15 total 155.85 last 43.91) - open/close 7677 (time 1709785229.71 total 166.41 last 41.49) - open/close 8043 (time 1709785240.15 total 176.85 last 35.07) - open/close 8521 (time 1709785251.11 total 187.81 last 43.60) - open/close 8954 (time 1709785261.98 total 198.68 last 39.83) - open/close 9464 (time 1709785272.17 total 208.87 last 50.05) - open/close 9825 (time 1709785283.13 total 219.84 last 32.93) - open/close 10000 (time 1709785287.97 total 224.67 last 36.20) - open/close 10456 (time 1709785298.32 total 235.02 last 44.05) - open/close 11989 (time 1709785308.32 total 245.02 last 153.29) - open/close 13580 (time 1709785318.32 total 255.03 last 159.03) - open/close 15115 (time 1709785328.33 total 265.03 last 153.42) - open/close 16604 (time 1709785338.34 total 275.04 last 148.79) - open/close 17930 (time 1709785348.34 total 285.04 last 132.53) - open/close 19245 (time 1709785358.34 total 295.04 last 131.49) - open/close 20000 (time 1709785364.90 total 301.60 last 115.21) - open/close 21082 (time 1709785374.90 total 311.60 last 108.14) - open/close 22090 (time 1709785384.91 total 321.61 last 100.70) - open/close 23017 (time 1709785394.92 total 331.62 last 92.64) - open/close 23979 (time 1709785404.92 total 341.63 last 96.14) - open/close 24857 (time 1709785414.93 total 351.64 last 87.71) - open/close 25863 (time 1709785424.93 total 361.64 last 100.59) - open/close 26737 (time 1709785434.93 total 371.64 last 87.40) - open/close 27627 (time 1709785444.94 total 381.64 last 88.93) - open/close 28417 (time 1709785454.95 total 391.65 last 78.98) - open/close 29251 (time 1709785464.95 total 401.66 last 83.34) - open/close 30000 (time 1709785474.85 total 411.55 last 75.71) - open/close 30661 (time 1709785484.85 total 421.56 last 66.04) - open/close 31399 (time 1709785494.86 total 431.56 last 73.78) - open/close 32063 (time 1709785504.86 total 441.57 last 66.35) - open/close 32792 (time 1709785514.88 total 451.58 last 72.81) - open/close 33446 (time 1709785524.89 total 461.59 last 65.31) - open/close 34129 (time 1709785534.89 total 471.60 last 68.27) - open/close 34748 (time 1709785544.90 total 481.60 last 61.87) - open/close 35392 (time 1709785554.91 total 491.62 last 64.31) - open/close 35968 (time 1709785564.92 total 501.62 last 57.59) - open/close 36597 (time 1709785574.92 total 511.62 last 62.89) - open/close 37254 (time 1709785584.94 total 521.64 last 65.58) - open/close 37815 (time 1709785594.95 total 531.65 last 56.01) - open/close 38390 (time 1709785604.96 total 541.66 last 57.47) - open/close 38924 (time 1709785614.97 total 551.67 last 53.33) - open/close 39450 (time 1709785624.98 total 561.68 last 52.53) - open/close 40000 (time 1709785634.92 total 571.62 last 55.34) - open/close 40590 (time 1709785644.92 total 581.63 last 58.99) - open/close 41117 (time 1709785654.93 total 591.63 last 52.67) - open/close 41720 (time 1709785664.93 total 601.63 last 60.28) - open/close 42293 (time 1709785674.93 total 611.64 last 57.29) - open/close 42825 (time 1709785684.94 total 621.65 last 53.15) - open/close 43326 (time 1709785694.95 total 631.66 last 50.04) - open/close 43813 (time 1709785704.96 total 641.66 last 48.68) - open/close 44325 (time 1709785714.97 total 651.67 last 51.13) - open/close 44786 (time 1709785724.98 total 661.69 last 46.04) - open/close 45305 (time 1709785735.00 total 671.70 last 51.81) - open/close 45731 (time 1709785745.01 total 681.71 last 42.57) - open/close 46169 (time 1709785755.03 total 691.73 last 43.72) - open/close 46640 (time 1709785765.03 total 701.73 last 47.07) - open/close 47152 (time 1709785775.05 total 711.75 last 51.11) - open/close 47646 (time 1709785785.06 total 721.76 last 49.36) - open/close 48128 (time 1709785795.07 total 731.77 last 48.14) - open/close 48594 (time 1709785805.08 total 741.78 last 46.55) - open/close 49033 (time 1709785815.09 total 751.79 last 43.86) - open/close 49530 (time 1709785825.11 total 761.81 last 49.59) total: 50000 open/close in 771.76 seconds: 64.79 ops/second - unlinked 0 (time 1709785835 ; total 0 ; last 0) - unlinked 10000 (time 1709786093 ; total 258 ; last 258) - unlinked 20000 (time 1709786280 ; total 445 ; last 187) - unlinked 30000 (time 1709786401 ; total 566 ; last 121) - unlinked 40000 (time 1709786467 ; total 632 ; last 66) total: 50000 unlinks in 657 seconds: 76.103500 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server unloading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2