== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:42:37 (1790714557) start mds service on oleg140-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1946968 Sep 29 04:58 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing load_modules_local oleg140-server: kmemleak disabled oleg140-server: Loading modules from /home/green/git/lustre-release/lustre oleg140-server: detected 4 online CPUs by sysfs oleg140-server: MODOPTS_LIBCFS= oleg140-server: Force libcfs to create 2 CPU partitions oleg140-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg140-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg140-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg140-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg140-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 232 (time 1790714634.85 total 10.01 last 23.19) - open/close 457 (time 1790714644.88 total 20.04 last 22.43) - open/close 688 (time 1790714655.15 total 30.30 last 22.50) - open/close 953 (time 1790714665.16 total 40.32 last 26.46) - open/close 1166 (time 1790714675.34 total 50.49 last 20.94) - open/close 1403 (time 1790714685.76 total 60.92 last 22.74) - open/close 1616 (time 1790714700.52 total 75.68 last 14.43) - open/close 1821 (time 1790714710.53 total 85.68 last 20.48) - open/close 2077 (time 1790714726.52 total 101.68 last 16.00) - open/close 2298 (time 1790714737.73 total 112.89 last 19.72) - open/close 2514 (time 1790714747.74 total 122.89 last 21.58) - open/close 2788 (time 1790714757.74 total 132.90 last 27.39) - open/close 3019 (time 1790714767.75 total 142.90 last 23.10) - open/close 3284 (time 1790714778.22 total 153.38 last 25.29) - open/close 3598 (time 1790714788.91 total 164.07 last 29.37) - open/close 3860 (time 1790714798.93 total 174.08 last 26.16) - open/close 4116 (time 1790714809.25 total 184.40 last 24.81) - open/close 4338 (time 1790714819.25 total 194.40 last 22.19) - open/close 4534 (time 1790714829.37 total 204.52 last 19.37) - open/close 4784 (time 1790714842.06 total 217.22 last 19.69) - open/close 4972 (time 1790714852.06 total 227.22 last 18.79) - open/close 5078 (time 1790714862.59 total 237.74 last 10.07) - open/close 5285 (time 1790714872.60 total 247.75 last 20.68) - open/close 5636 (time 1790714882.61 total 257.76 last 35.06) - open/close 5810 (time 1790714892.98 total 268.14 last 16.77) - open/close 6123 (time 1790714902.98 total 278.14 last 31.29) - open/close 6379 (time 1790714913.45 total 288.61 last 24.46) - open/close 6605 (time 1790714923.62 total 298.78 last 22.22) - open/close 6859 (time 1790714933.64 total 308.79 last 25.37) - open/close 7116 (time 1790714943.65 total 318.81 last 25.66) - open/close 7392 (time 1790714953.65 total 328.81 last 27.60) - open/close 7635 (time 1790714963.66 total 338.82 last 24.28) - open/close 7885 (time 1790714974.47 total 349.63 last 23.12) - open/close 8135 (time 1790714985.15 total 360.30 last 23.42) - open/close 8421 (time 1790714995.15 total 370.31 last 28.58) - open/close 8662 (time 1790715005.82 total 380.98 last 22.59) - open/close 8889 (time 1790715016.12 total 391.27 last 22.05) - open/close 9148 (time 1790715026.12 total 401.28 last 25.88) - open/close 9401 (time 1790715036.14 total 411.29 last 25.27) - open/close 9527 (time 1790715046.58 total 421.73 last 12.07) - open/close 9781 (time 1790715056.59 total 431.75 last 25.37) - open/close 9926 (time 1790715066.59 total 441.75 last 14.50) - open/close 10000 (time 1790715068.79 total 443.94 last 33.70) - open/close 10232 (time 1790715079.48 total 454.64 last 21.70) - open/close 10499 (time 1790715089.49 total 464.64 last 26.69) - open/close 10700 (time 1790715100.69 total 475.84 last 17.94) - open/close 10956 (time 1790715110.82 total 485.97 last 25.27) - open/close 11226 (time 1790715120.83 total 495.98 last 26.97) - open/close 11441 (time 1790715131.58 total 506.73 last 20.00) - open/close 11670 (time 1790715141.94 total 517.09 last 22.11) - open/close 11920 (time 1790715152.88 total 528.04 last 22.84) - open/close 12109 (time 1790715163.13 total 538.28 last 18.45) - open/close 12222 (time 1790715173.35 total 548.50 last 11.06) - open/close 12418 (time 1790715184.03 total 559.19 last 18.34) - open/close 12660 (time 1790715194.04 total 569.19 last 24.19) - open/close 12875 (time 1790715204.40 total 579.55 last 20.75) - open/close 13125 (time 1790715214.42 total 589.57 last 24.95) - open/close 13323 (time 1790715224.63 total 599.79 last 19.38) - open/close 13574 (time 1790715235.38 total 610.53 last 23.36) - open/close 13853 (time 1790715246.44 total 621.59 last 25.23) - open/close 14089 (time 1790715256.88 total 632.03 last 22.61) - open/close 14316 (time 1790715266.88 total 642.04 last 22.69) - open/close 14539 (time 1790715277.38 total 652.53 last 21.24) - open/close 14730 (time 1790715287.93 total 663.08 last 18.11) - open/close 14984 (time 1790715297.95 total 673.10 last 25.36) - open/close 15169 (time 1790715308.26 total 683.42 last 17.93) - open/close 15436 (time 1790715318.79 total 693.95 last 25.36) - open/close 15702 (time 1790715329.39 total 704.54 last 25.10) - open/close 15954 (time 1790715339.97 total 715.12 last 23.82) - open/close 16162 (time 1790715350.48 total 725.64 last 19.78) - open/close 16334 (time 1790715360.48 total 735.64 last 17.20) - open/close 16557 (time 1790715371.05 total 746.21 last 21.10) - open/close 16731 (time 1790715381.88 total 757.03 last 16.07) - open/close 16988 (time 1790715391.89 total 767.04 last 25.67) - open/close 17177 (time 1790715401.91 total 777.07 last 18.85) - open/close 17409 (time 1790715412.86 total 788.02 last 21.19) - open/close 17611 (time 1790715423.80 total 798.95 last 18.47) - open/close 17750 (time 1790715433.91 total 809.07 last 13.74) - open/close 18008 (time 1790715444.45 total 819.61 last 24.47) - open/close 18229 (time 1790715454.86 total 830.01 last 21.24) - open/close 18440 (time 1790715464.87 total 840.02 last 21.09) - open/close 18654 (time 1790715475.34 total 850.50 last 20.43) - open/close 18869 (time 1790715485.35 total 860.50 last 21.49) - open/close 19058 (time 1790715495.40 total 870.56 last 18.79) - open/close 19249 (time 1790715505.42 total 880.58 last 19.06) - open/close 19451 (time 1790715515.44 total 890.60 last 20.16) - open/close 19653 (time 1790715526.46 total 901.61 last 18.34) - open/close 19894 (time 1790715536.48 total 911.64 last 24.05) - open/close 20000 (time 1790715542.46 total 917.61 last 17.74) - open/close 20195 (time 1790715553.08 total 928.23 last 18.37) - open/close 20842 (time 1790715563.08 total 938.24 last 64.65) - open/close 21538 (time 1790715573.09 total 948.24 last 69.56) - open/close 22320 (time 1790715583.09 total 958.25 last 78.15) - open/close 23159 (time 1790715593.13 total 968.28 last 83.60) - open/close 23981 (time 1790715603.14 total 978.30 last 82.09) - open/close 24860 (time 1790715613.15 total 988.31 last 87.81) - open/close 25681 (time 1790715623.16 total 998.31 last 82.05) - open/close 26523 (time 1790715633.17 total 1008.33 last 84.10) - open/close 27265 (time 1790715643.18 total 1018.33 last 74.14) - open/close 28139 (time 1790715653.19 total 1028.34 last 87.35) - open/close 28846 (time 1790715663.21 total 1038.37 last 70.51) - open/close 29725 (time 1790715673.23 total 1048.38 last 87.79) - open/close 30000 (time 1790715677.46 total 1052.62 last 64.89) - open/close 30805 (time 1790715687.50 total 1062.65 last 80.22) - open/close 31632 (time 1790715697.51 total 1072.66 last 82.62) - open/close 32493 (time 1790715707.51 total 1082.67 last 86.08) - open/close 33325 (time 1790715717.52 total 1092.68 last 83.10) - open/close 34128 (time 1790715727.53 total 1102.68 last 80.28) - open/close 34913 (time 1790715737.53 total 1112.69 last 78.44) - open/close 35707 (time 1790715747.55 total 1122.71 last 79.26) - open/close 36529 (time 1790715757.56 total 1132.71 last 82.13) - open/close 37394 (time 1790715767.57 total 1142.72 last 86.46) - open/close 38131 (time 1790715777.58 total 1152.73 last 73.60) - open/close 38889 (time 1790715787.59 total 1162.74 last 75.73) - open/close 39726 (time 1790715797.60 total 1172.75 last 83.63) - open/close 40000 (time 1790715802.06 total 1177.22 last 61.36) - open/close 40759 (time 1790715812.07 total 1187.23 last 75.81) - open/close 41671 (time 1790715822.11 total 1197.27 last 90.84) - open/close 42453 (time 1790715832.12 total 1207.28 last 78.11) - open/close 43274 (time 1790715842.13 total 1217.29 last 82.04) - open/close 43967 (time 1790715852.14 total 1227.29 last 69.27) - open/close 44640 (time 1790715862.14 total 1237.30 last 67.26) - open/close 45442 (time 1790715872.15 total 1247.31 last 80.12) - open/close 46316 (time 1790715882.16 total 1257.31 last 87.34) - open/close 47109 (time 1790715892.16 total 1267.32 last 79.29) - open/close 47843 (time 1790715902.17 total 1277.32 last 73.34) - open/close 48488 (time 1790715912.17 total 1287.33 last 64.48) - open/close 49081 (time 1790715922.18 total 1297.33 last 59.26) - open/close 49792 (time 1790715932.18 total 1307.34 last 71.07) total: 50000 open/close in 1309.73 seconds: 38.18 ops/second - unlinked 0 (time 1790715939 ; total 0 ; last 0) - unlinked 10000 (time 1790716067 ; total 128 ; last 128) - unlinked 20000 (time 1790716199 ; total 260 ; last 132) - unlinked 30000 (time 1790716338 ; total 399 ; last 139) - unlinked 40000 (time 1790716473 ; total 534 ; last 135) total: 50000 unlinks in 664 seconds: 75.301208 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg140-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server unloading modules via unload_modules_local on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing unload_modules_local oleg140-server: kmemleak disabled modules unloaded. pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2