== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:12:47 (1788660767) start mds service on oleg331-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 5 07:32 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg331-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg331-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 182 (time 1788660853.30 total 10.45 last 17.42) - open/close 421 (time 1788660863.31 total 20.46 last 23.88) - open/close 622 (time 1788660873.33 total 30.48 last 20.05) - open/close 876 (time 1788660884.09 total 41.25 last 23.60) - open/close 1128 (time 1788660894.67 total 51.82 last 23.83) - open/close 1375 (time 1788660904.67 total 61.83 last 24.69) - open/close 1563 (time 1788660915.56 total 72.71 last 17.27) - open/close 1813 (time 1788660925.57 total 82.72 last 24.98) - open/close 1978 (time 1788660936.08 total 93.23 last 15.71) - open/close 2209 (time 1788660946.77 total 103.92 last 21.61) - open/close 2428 (time 1788660956.88 total 114.03 last 21.65) - open/close 2684 (time 1788660966.89 total 124.04 last 25.58) - open/close 2930 (time 1788660977.35 total 134.50 last 23.52) - open/close 3180 (time 1788660987.91 total 145.06 last 23.68) - open/close 3412 (time 1788660998.34 total 155.49 last 22.24) - open/close 3691 (time 1788661008.84 total 165.99 last 26.57) - open/close 3926 (time 1788661019.66 total 176.81 last 21.71) - open/close 4161 (time 1788661029.89 total 187.04 last 22.97) - open/close 4364 (time 1788661040.46 total 197.61 last 19.21) - open/close 4586 (time 1788661050.52 total 207.67 last 22.08) - open/close 4831 (time 1788661060.53 total 217.68 last 24.47) - open/close 5053 (time 1788661070.53 total 227.68 last 22.20) - open/close 5276 (time 1788661080.53 total 237.68 last 22.29) - open/close 5498 (time 1788661091.16 total 248.31 last 20.90) - open/close 5691 (time 1788661101.53 total 258.69 last 18.60) - open/close 5899 (time 1788661111.56 total 268.71 last 20.74) - open/close 6123 (time 1788661121.57 total 278.72 last 22.38) - open/close 6357 (time 1788661131.59 total 288.74 last 23.36) - open/close 6575 (time 1788661142.48 total 299.63 last 20.01) - open/close 6824 (time 1788661152.59 total 309.74 last 24.64) - open/close 7040 (time 1788661162.59 total 319.74 last 21.60) - open/close 7270 (time 1788661172.60 total 329.76 last 22.96) - open/close 7488 (time 1788661183.63 total 340.78 last 19.77) - open/close 7740 (time 1788661194.25 total 351.40 last 23.73) - open/close 7956 (time 1788661204.37 total 361.52 last 21.35) - open/close 8155 (time 1788661214.80 total 371.95 last 19.08) - open/close 8399 (time 1788661224.82 total 381.97 last 24.35) - open/close 8583 (time 1788661235.73 total 392.88 last 16.87) - open/close 8713 (time 1788661246.46 total 403.62 last 12.11) - open/close 8968 (time 1788661257.49 total 414.64 last 23.13) - open/close 9127 (time 1788661267.86 total 425.01 last 15.34) - open/close 9404 (time 1788661277.87 total 435.02 last 27.67) - open/close 9612 (time 1788661287.88 total 445.03 last 20.78) - open/close 9846 (time 1788661298.83 total 455.98 last 21.37) - open/close 9968 (time 1788661309.58 total 466.73 last 11.34) - open/close 10200 (time 1788661319.62 total 476.77 last 23.11) - open/close 10339 (time 1788661329.81 total 486.96 last 13.64) - open/close 10530 (time 1788661340.23 total 497.38 last 18.33) - open/close 10643 (time 1788661351.08 total 508.23 last 10.41) - open/close 10787 (time 1788661361.38 total 518.53 last 13.99) - open/close 11034 (time 1788661371.39 total 528.54 last 24.67) - open/close 11113 (time 1788661381.39 total 538.54 last 7.90) - open/close 11316 (time 1788661391.83 total 548.98 last 19.44) - open/close 11548 (time 1788661402.81 total 559.96 last 21.13) - open/close 11692 (time 1788661413.72 total 570.88 last 13.20) - open/close 11963 (time 1788661423.76 total 580.91 last 27.00) - open/close 12169 (time 1788661434.60 total 591.75 last 19.01) - open/close 12371 (time 1788661444.63 total 601.78 last 20.13) - open/close 12559 (time 1788661454.63 total 611.79 last 18.80) - open/close 12819 (time 1788661464.63 total 621.79 last 26.00) - open/close 12969 (time 1788661475.10 total 632.25 last 14.34) - open/close 13093 (time 1788661485.11 total 642.26 last 12.39) - open/close 13274 (time 1788661495.83 total 652.98 last 16.89) - open/close 13521 (time 1788661505.84 total 662.99 last 24.67) - open/close 13763 (time 1788661516.75 total 673.90 last 22.17) - open/close 13956 (time 1788661527.69 total 684.84 last 17.65) - open/close 14195 (time 1788661538.18 total 695.34 last 22.77) - open/close 14440 (time 1788661548.21 total 705.36 last 24.45) - open/close 14610 (time 1788661559.14 total 716.30 last 15.54) - open/close 14827 (time 1788661569.15 total 726.30 last 21.69) - open/close 14950 (time 1788661580.03 total 737.18 last 11.31) - open/close 15190 (time 1788661590.71 total 747.87 last 22.46) - open/close 15382 (time 1788661600.99 total 758.14 last 18.69) - open/close 15630 (time 1788661610.99 total 768.15 last 24.78) - open/close 15841 (time 1788661621.22 total 778.37 last 20.64) - open/close 16076 (time 1788661631.94 total 789.10 last 21.91) - open/close 16305 (time 1788661642.24 total 799.39 last 22.24) - open/close 16561 (time 1788661652.25 total 809.40 last 25.58) - open/close 16811 (time 1788661663.01 total 820.16 last 23.23) - open/close 17024 (time 1788661673.28 total 830.43 last 20.73) - open/close 17266 (time 1788661683.97 total 841.13 last 22.63) - open/close 17483 (time 1788661694.41 total 851.56 last 20.80) - open/close 17694 (time 1788661704.75 total 861.91 last 20.39) - open/close 17970 (time 1788661714.76 total 871.91 last 27.58) - open/close 18197 (time 1788661725.33 total 882.48 last 21.48) - open/close 18434 (time 1788661736.01 total 893.17 last 22.19) - open/close 18657 (time 1788661746.84 total 903.99 last 20.60) - open/close 18806 (time 1788661756.88 total 914.04 last 14.83) - open/close 19410 (time 1788661766.90 total 924.05 last 60.33) - open/close 20000 (time 1788661776.05 total 933.20 last 64.46) - open/close 20616 (time 1788661786.05 total 943.20 last 61.58) - open/close 21277 (time 1788661796.05 total 953.20 last 66.09) - open/close 21935 (time 1788661806.07 total 963.22 last 65.70) - open/close 22586 (time 1788661816.07 total 973.22 last 65.06) - open/close 23249 (time 1788661826.07 total 983.22 last 66.30) - open/close 23833 (time 1788661836.08 total 993.24 last 58.33) - open/close 24449 (time 1788661846.09 total 1003.24 last 61.57) - open/close 25071 (time 1788661856.10 total 1013.25 last 62.16) - open/close 25685 (time 1788661866.10 total 1023.26 last 61.34) - open/close 26296 (time 1788661876.11 total 1033.26 last 61.06) - open/close 26955 (time 1788661886.11 total 1043.27 last 65.88) - open/close 27597 (time 1788661896.14 total 1053.29 last 64.03) - open/close 28261 (time 1788661906.14 total 1063.29 last 66.38) - open/close 28908 (time 1788661916.16 total 1073.31 last 64.61) - open/close 29574 (time 1788661926.16 total 1083.31 last 66.57) - open/close 30000 (time 1788661933.40 total 1090.56 last 58.83) - open/close 30606 (time 1788661943.42 total 1100.57 last 60.53) - open/close 31228 (time 1788661953.43 total 1110.58 last 62.13) - open/close 31849 (time 1788661963.44 total 1120.59 last 62.05) - open/close 32507 (time 1788661973.45 total 1130.60 last 65.71) - open/close 33123 (time 1788661983.46 total 1140.61 last 61.52) - open/close 33795 (time 1788661993.47 total 1150.62 last 67.15) - open/close 34448 (time 1788662003.48 total 1160.63 last 65.23) - open/close 35091 (time 1788662013.50 total 1170.65 last 64.17) - open/close 35771 (time 1788662023.51 total 1180.66 last 67.96) - open/close 36432 (time 1788662033.53 total 1190.68 last 65.95) - open/close 37074 (time 1788662043.54 total 1200.69 last 64.16) - open/close 37729 (time 1788662053.55 total 1210.70 last 65.42) - open/close 38355 (time 1788662063.56 total 1220.71 last 62.52) - open/close 39000 (time 1788662073.57 total 1230.72 last 64.45) - open/close 39669 (time 1788662083.57 total 1240.72 last 66.87) - open/close 40000 (time 1788662088.60 total 1245.75 last 65.79) - open/close 40595 (time 1788662098.61 total 1255.76 last 59.47) - open/close 41224 (time 1788662108.62 total 1265.77 last 62.83) - open/close 41904 (time 1788662118.63 total 1275.78 last 67.95) - open/close 42576 (time 1788662128.64 total 1285.79 last 67.13) - open/close 43242 (time 1788662138.65 total 1295.80 last 66.54) - open/close 43853 (time 1788662148.66 total 1305.82 last 60.99) - open/close 44454 (time 1788662158.67 total 1315.83 last 60.05) - open/close 45079 (time 1788662168.68 total 1325.83 last 62.46) - open/close 45699 (time 1788662178.68 total 1335.83 last 61.99) - open/close 46275 (time 1788662188.68 total 1345.84 last 57.58) - open/close 46863 (time 1788662198.68 total 1355.84 last 58.80) - open/close 47454 (time 1788662208.69 total 1365.84 last 59.05) - open/close 48020 (time 1788662218.70 total 1375.86 last 56.53) - open/close 48567 (time 1788662228.71 total 1385.86 last 54.67) - open/close 49193 (time 1788662238.73 total 1395.88 last 62.48) - open/close 49844 (time 1788662248.73 total 1405.88 last 65.07) total: 50000 open/close in 1408.42 seconds: 35.50 ops/second - unlinked 0 (time 1788662255 ; total 0 ; last 0) - unlinked 10000 (time 1788662417 ; total 162 ; last 162) - unlinked 20000 (time 1788662580 ; total 325 ; last 163) - unlinked 30000 (time 1788662773 ; total 518 ; last 193) - unlinked 40000 (time 1788662947 ; total 692 ; last 174) total: 50000 unlinks in 869 seconds: 57.537399 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server unloading modules via unload_modules_local on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing unload_modules_local oleg331-server: kmemleak disabled modules unloaded. pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2