== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:09:58 (1709784598) start mds service on oleg311-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg311-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0000 oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg311-client.virtnet: -o user_xattr,flock oleg311-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 258 (time 1709784630.86 total 10.39 last 24.82) - open/close 564 (time 1709784641.75 total 21.29 last 28.10) - open/close 882 (time 1709784651.75 total 31.29 last 31.79) - open/close 1334 (time 1709784662.29 total 41.82 last 42.91) - open/close 1821 (time 1709784672.57 total 52.11 last 47.36) - open/close 2448 (time 1709784683.13 total 62.67 last 59.36) - open/close 3010 (time 1709784693.62 total 73.15 last 53.60) - open/close 3467 (time 1709784704.03 total 83.56 last 43.91) - open/close 3827 (time 1709784714.38 total 93.92 last 34.77) - open/close 4107 (time 1709784724.95 total 104.48 last 26.50) - open/close 4498 (time 1709784735.54 total 115.07 last 36.91) - open/close 5016 (time 1709784745.54 total 125.07 last 51.80) - open/close 5276 (time 1709784755.78 total 135.31 last 25.39) - open/close 5715 (time 1709784765.78 total 145.32 last 43.89) - open/close 6281 (time 1709784776.05 total 155.58 last 55.14) - open/close 6643 (time 1709784786.51 total 166.04 last 34.60) - open/close 6894 (time 1709784796.59 total 176.13 last 24.88) - open/close 7182 (time 1709784806.59 total 186.13 last 28.80) - open/close 7618 (time 1709784817.27 total 196.80 last 40.84) - open/close 8120 (time 1709784827.75 total 207.29 last 47.90) - open/close 8370 (time 1709784838.16 total 217.70 last 24.01) - open/close 8777 (time 1709784848.17 total 227.70 last 40.69) - open/close 8966 (time 1709784858.31 total 237.85 last 18.63) - open/close 9342 (time 1709784869.21 total 248.75 last 34.49) - open/close 10000 (time 1709784872.42 total 251.95 last 205.33) - open/close 12339 (time 1709784882.42 total 261.96 last 233.83) - open/close 14512 (time 1709784892.43 total 271.96 last 217.19) - open/close 16578 (time 1709784902.43 total 281.96 last 206.58) - open/close 18407 (time 1709784912.43 total 291.97 last 182.83) - open/close 20000 (time 1709784921.88 total 301.42 last 168.59) - open/close 21296 (time 1709784931.88 total 311.42 last 129.59) - open/close 22614 (time 1709784941.89 total 321.42 last 131.72) - open/close 23881 (time 1709784951.89 total 331.43 last 126.63) - open/close 25072 (time 1709784961.90 total 341.43 last 119.03) - open/close 26266 (time 1709784971.90 total 351.43 last 119.40) - open/close 27349 (time 1709784981.91 total 361.44 last 108.20) - open/close 28342 (time 1709784991.91 total 371.45 last 99.25) - open/close 29340 (time 1709785001.92 total 381.45 last 99.74) - open/close 30000 (time 1709785008.07 total 387.60 last 107.33) - open/close 31105 (time 1709785018.08 total 397.61 last 110.39) - open/close 32028 (time 1709785028.08 total 407.61 last 92.29) - open/close 32918 (time 1709785038.09 total 417.62 last 88.93) - open/close 33763 (time 1709785048.09 total 427.63 last 84.44) - open/close 34689 (time 1709785058.10 total 437.64 last 92.55) - open/close 35364 (time 1709785068.10 total 447.64 last 67.48) - open/close 36212 (time 1709785078.11 total 457.64 last 84.77) - open/close 37094 (time 1709785088.12 total 467.65 last 88.11) - open/close 38001 (time 1709785098.12 total 477.66 last 90.66) - open/close 38811 (time 1709785108.13 total 487.66 last 80.94) - open/close 39605 (time 1709785118.13 total 497.67 last 79.37) - open/close 40000 (time 1709785123.48 total 503.01 last 73.92) - open/close 40761 (time 1709785133.49 total 513.02 last 76.02) - open/close 41484 (time 1709785143.50 total 523.03 last 72.23) - open/close 42180 (time 1709785153.51 total 533.04 last 69.54) - open/close 42909 (time 1709785163.51 total 543.05 last 72.86) - open/close 43564 (time 1709785173.52 total 553.05 last 65.44) - open/close 44194 (time 1709785183.52 total 563.06 last 62.98) - open/close 44911 (time 1709785193.53 total 573.07 last 71.64) - open/close 45574 (time 1709785203.54 total 583.07 last 66.26) - open/close 46212 (time 1709785213.55 total 593.08 last 63.73) - open/close 46817 (time 1709785223.56 total 603.09 last 60.44) - open/close 47388 (time 1709785233.57 total 613.11 last 57.02) - open/close 47972 (time 1709785243.57 total 623.11 last 58.39) - open/close 48517 (time 1709785253.58 total 633.11 last 54.49) - open/close 49043 (time 1709785263.58 total 643.11 last 52.60) - open/close 49502 (time 1709785273.60 total 653.14 last 45.79) - open/close 49978 (time 1709785283.60 total 663.14 last 47.60) total: 50000 open/close in 663.59 seconds: 75.35 ops/second - unlinked 0 (time 1709785285 ; total 1 ; last 1) - unlinked 10000 (time 1709785495 ; total 211 ; last 210) - unlinked 20000 (time 1709785662 ; total 378 ; last 167) - unlinked 30000 (time 1709785767 ; total 483 ; last 105) - unlinked 40000 (time 1709785829 ; total 545 ; last 62) total: 50000 unlinks in 569 seconds: 87.873459 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg311-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg311-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg311-server stop mds service on oleg311-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg311-server unloading modules on: 'oleg311-server' oleg311-server: oleg311-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg311-client: oleg311-client: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-client: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2