== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:47:54 (1788850074) start mds service on oleg654-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 8 01:49 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg654-server' oleg654-server: oleg654-server.virtnet: executing load_modules_local oleg654-server: kmemleak disabled oleg654-server: Loading modules from /home/green/git/lustre-release/lustre oleg654-server: detected 4 online CPUs by sysfs oleg654-server: MODOPTS_LIBCFS= oleg654-server: Force libcfs to create 2 CPU partitions oleg654-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg654-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg654-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg654-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg654-server: oleg654-server.virtnet: executing set_default_debug -1 all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg654-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg654-server: oleg654-server.virtnet: executing set_default_debug -1 all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-OST0000 oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 173 (time 1788850140.97 total 10.40 last 16.63) - open/close 317 (time 1788850150.98 total 20.41 last 14.39) - open/close 470 (time 1788850161.39 total 30.82 last 14.70) - open/close 636 (time 1788850171.40 total 40.83 last 16.57) - open/close 946 (time 1788850181.48 total 50.91 last 30.75) - open/close 1264 (time 1788850192.17 total 61.60 last 29.77) - open/close 1456 (time 1788850202.47 total 71.90 last 18.63) - open/close 1633 (time 1788850212.72 total 82.15 last 17.28) - open/close 1759 (time 1788850223.59 total 93.02 last 11.59) - open/close 1845 (time 1788850233.84 total 103.27 last 8.39) - open/close 2036 (time 1788850246.06 total 115.49 last 15.62) - open/close 2159 (time 1788850260.71 total 130.15 last 8.39) - open/close 2468 (time 1788850271.02 total 140.45 last 29.99) - open/close 2660 (time 1788850281.12 total 150.55 last 19.00) - open/close 3050 (time 1788850291.82 total 161.25 last 36.48) - open/close 3257 (time 1788850301.82 total 171.25 last 20.70) - open/close 3500 (time 1788850312.49 total 181.92 last 22.76) - open/close 3648 (time 1788850322.50 total 191.93 last 14.79) - open/close 3717 (time 1788850332.71 total 202.14 last 6.76) - open/close 3946 (time 1788850343.41 total 212.84 last 21.41) - open/close 4035 (time 1788850353.70 total 223.13 last 8.64) - open/close 4153 (time 1788850364.46 total 233.89 last 10.97) - open/close 4226 (time 1788850374.51 total 243.94 last 7.26) - open/close 4381 (time 1788850384.69 total 254.12 last 15.23) - open/close 4554 (time 1788850395.25 total 264.68 last 16.37) - open/close 4714 (time 1788850405.43 total 274.86 last 15.73) - open/close 4838 (time 1788850417.26 total 286.69 last 10.48) - open/close 5025 (time 1788850427.27 total 296.70 last 18.69) - open/close 5258 (time 1788850437.67 total 307.10 last 22.38) - open/close 5511 (time 1788850448.18 total 317.61 last 24.09) - open/close 5870 (time 1788850458.22 total 327.65 last 35.74) - open/close 6127 (time 1788850468.77 total 338.21 last 24.35) - open/close 6337 (time 1788850480.04 total 349.47 last 18.64) - open/close 6542 (time 1788850490.41 total 359.84 last 19.78) - open/close 6735 (time 1788850500.46 total 369.89 last 19.21) - open/close 7088 (time 1788850510.50 total 379.93 last 35.14) - open/close 7304 (time 1788850520.75 total 390.18 last 21.08) - open/close 7472 (time 1788850530.76 total 400.19 last 16.79) - open/close 7691 (time 1788850540.97 total 410.40 last 21.44) - open/close 7918 (time 1788850551.41 total 420.84 last 21.76) - open/close 8144 (time 1788850561.64 total 431.07 last 22.09) - open/close 8488 (time 1788850572.52 total 441.95 last 31.61) - open/close 8740 (time 1788850582.96 total 452.39 last 24.15) - open/close 9021 (time 1788850593.32 total 462.75 last 27.13) - open/close 9316 (time 1788850603.75 total 473.18 last 28.28) - open/close 9575 (time 1788850613.75 total 483.18 last 25.90) - open/close 9822 (time 1788850623.78 total 493.21 last 24.62) - open/close 10000 (time 1788850632.18 total 501.61 last 21.19) - open/close 10162 (time 1788850642.60 total 512.03 last 15.55) - open/close 10716 (time 1788850652.60 total 522.03 last 55.38) - open/close 12623 (time 1788850662.61 total 532.04 last 190.60) - open/close 14309 (time 1788850672.61 total 542.04 last 168.52) - open/close 16036 (time 1788850682.61 total 552.04 last 172.70) - open/close 17903 (time 1788850692.61 total 562.04 last 186.64) - open/close 19265 (time 1788850702.62 total 572.05 last 136.13) - open/close 20000 (time 1788850708.22 total 577.65 last 131.12) - open/close 21266 (time 1788850718.23 total 587.66 last 126.58) - open/close 22660 (time 1788850728.23 total 597.66 last 139.37) - open/close 24279 (time 1788850738.23 total 607.66 last 161.87) - open/close 26098 (time 1788850748.23 total 617.66 last 181.81) - open/close 27675 (time 1788850758.24 total 627.67 last 157.69) - open/close 29254 (time 1788850768.24 total 637.67 last 157.86) - open/close 30000 (time 1788850772.74 total 642.17 last 165.63) - open/close 31471 (time 1788850782.74 total 652.17 last 147.08) - open/close 32933 (time 1788850792.75 total 662.18 last 146.11) - open/close 34526 (time 1788850802.75 total 672.18 last 159.27) - open/close 36261 (time 1788850812.76 total 682.19 last 173.40) - open/close 38064 (time 1788850822.76 total 692.19 last 180.28) - open/close 39659 (time 1788850832.76 total 702.19 last 159.49) - open/close 40000 (time 1788850834.85 total 704.28 last 162.97) - open/close 41541 (time 1788850844.85 total 714.28 last 154.07) - open/close 42913 (time 1788850854.86 total 724.29 last 137.14) - open/close 43978 (time 1788850864.86 total 734.29 last 106.45) - open/close 45082 (time 1788850874.87 total 744.30 last 110.32) - open/close 46320 (time 1788850884.88 total 754.31 last 123.70) - open/close 47881 (time 1788850894.88 total 764.31 last 156.05) - open/close 49217 (time 1788850904.89 total 774.32 last 133.42) total: 50000 open/close in 781.76 seconds: 63.96 ops/second - unlinked 0 (time 1788850914 ; total 0 ; last 0) - unlinked 10000 (time 1788850965 ; total 51 ; last 51) - unlinked 20000 (time 1788851008 ; total 94 ; last 43) - unlinked 30000 (time 1788851046 ; total 132 ; last 38) - unlinked 40000 (time 1788851072 ; total 158 ; last 26) total: 50000 unlinks in 185 seconds: 270.270264 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg654-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg654-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg654-server stop mds service on oleg654-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg654-server unloading modules via unload_modules_local on: 'oleg654-server' oleg654-server: oleg654-server.virtnet: executing unload_modules_local oleg654-server: kmemleak disabled modules unloaded. pdsh@oleg654-client: oleg654-client: ssh exited with exit code 2 pdsh@oleg654-client: oleg654-server: ssh exited with exit code 2 pdsh@oleg654-client: oleg654-client: ssh exited with exit code 2 pdsh@oleg654-client: oleg654-server: ssh exited with exit code 2