== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:46:45 (1788626805) start mds service on oleg359-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 4 17:42 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg359-server' oleg359-server: oleg359-server.virtnet: executing load_modules_local oleg359-server: kmemleak disabled oleg359-server: Loading modules from /home/green/git/lustre-release/lustre oleg359-server: detected 4 online CPUs by sysfs oleg359-server: MODOPTS_LIBCFS= oleg359-server: Force libcfs to create 2 CPU partitions oleg359-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg359-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg359-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg359-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg359-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg359-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0000 oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg359-client.virtnet: -o user_xattr,flock 192.168.203.159@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 179 (time 1788626917.63 total 10.92 last 16.39) - open/close 373 (time 1788626927.64 total 20.93 last 19.38) - open/close 528 (time 1788626938.04 total 31.34 last 14.89) - open/close 802 (time 1788626949.01 total 42.31 last 24.98) - open/close 1078 (time 1788626960.67 total 53.96 last 23.68) - open/close 1231 (time 1788626970.68 total 63.98 last 15.28) - open/close 1480 (time 1788626980.69 total 73.99 last 24.87) - open/close 1714 (time 1788626990.70 total 84.00 last 23.37) - open/close 1945 (time 1788627001.69 total 94.98 last 21.04) - open/close 2148 (time 1788627011.69 total 104.99 last 20.28) - open/close 2445 (time 1788627021.72 total 115.02 last 29.62) - open/close 2644 (time 1788627031.73 total 125.03 last 19.87) - open/close 2895 (time 1788627041.74 total 135.04 last 25.08) - open/close 3178 (time 1788627052.42 total 145.72 last 26.49) - open/close 3452 (time 1788627062.53 total 155.83 last 27.11) - open/close 3657 (time 1788627072.54 total 165.84 last 20.48) - open/close 3970 (time 1788627082.55 total 175.84 last 31.28) - open/close 4187 (time 1788627093.53 total 186.83 last 19.76) - open/close 4468 (time 1788627103.54 total 196.84 last 28.07) - open/close 4618 (time 1788627113.54 total 206.84 last 15.00) - open/close 4820 (time 1788627123.62 total 216.91 last 20.05) - open/close 5003 (time 1788627134.66 total 227.96 last 16.57) - open/close 5029 (time 1788627144.67 total 237.97 last 2.60) - open/close 5205 (time 1788627154.71 total 248.01 last 17.53) - open/close 5333 (time 1788627165.26 total 258.56 last 12.13) - open/close 5522 (time 1788627175.27 total 268.57 last 18.88) - open/close 5678 (time 1788627185.61 total 278.90 last 15.09) - open/close 5787 (time 1788627196.23 total 289.53 last 10.26) - open/close 5923 (time 1788627206.37 total 299.66 last 13.42) - open/close 6129 (time 1788627217.44 total 310.73 last 18.61) - open/close 6312 (time 1788627227.44 total 320.74 last 18.29) - open/close 6525 (time 1788627237.58 total 330.88 last 21.00) - open/close 6792 (time 1788627248.21 total 341.51 last 25.13) - open/close 7012 (time 1788627258.97 total 352.26 last 20.45) - open/close 7204 (time 1788627269.59 total 362.89 last 18.07) - open/close 7273 (time 1788627279.60 total 372.89 last 6.90) - open/close 7469 (time 1788627289.83 total 383.13 last 19.15) - open/close 7646 (time 1788627300.56 total 393.86 last 16.49) - open/close 7813 (time 1788627313.41 total 406.71 last 12.99) - open/close 7950 (time 1788627323.42 total 416.71 last 13.70) - open/close 8156 (time 1788627333.60 total 426.89 last 20.24) - open/close 8332 (time 1788627344.26 total 437.56 last 16.50) - open/close 8463 (time 1788627354.27 total 447.57 last 13.09) - open/close 8725 (time 1788627364.31 total 457.60 last 26.11) - open/close 8983 (time 1788627377.17 total 470.46 last 20.06) - open/close 9199 (time 1788627387.17 total 480.47 last 21.59) - open/close 9480 (time 1788627400.27 total 493.57 last 21.45) - open/close 9783 (time 1788627410.27 total 503.57 last 30.29) - open/close 10000 (time 1788627418.53 total 511.83 last 26.28) - open/close 10265 (time 1788627428.93 total 522.23 last 25.48) - open/close 10531 (time 1788627441.29 total 534.58 last 21.53) - open/close 10721 (time 1788627451.65 total 544.95 last 18.33) - open/close 10936 (time 1788627462.47 total 555.76 last 19.88) - open/close 11222 (time 1788627473.49 total 566.78 last 25.95) - open/close 11459 (time 1788627483.80 total 577.10 last 22.97) - open/close 11721 (time 1788627494.63 total 587.93 last 24.20) - open/close 11976 (time 1788627504.66 total 597.95 last 25.43) - open/close 12239 (time 1788627514.67 total 607.97 last 26.27) - open/close 12438 (time 1788627524.67 total 617.97 last 19.90) - open/close 12634 (time 1788627535.27 total 628.57 last 18.49) - open/close 12757 (time 1788627545.90 total 639.19 last 11.58) - open/close 12897 (time 1788627556.55 total 649.85 last 13.14) - open/close 13063 (time 1788627566.93 total 660.23 last 15.99) - open/close 13155 (time 1788627576.94 total 670.23 last 9.20) - open/close 13319 (time 1788627587.99 total 681.29 last 14.84) - open/close 13407 (time 1788627599.07 total 692.37 last 7.94) - open/close 13665 (time 1788627609.50 total 702.79 last 24.75) - open/close 13954 (time 1788627620.36 total 713.66 last 26.59) - open/close 14242 (time 1788627630.54 total 723.84 last 28.29) - open/close 14526 (time 1788627641.10 total 734.40 last 26.90) - open/close 14820 (time 1788627651.86 total 745.15 last 27.34) - open/close 15041 (time 1788627662.80 total 756.10 last 20.19) - open/close 15305 (time 1788627673.55 total 766.85 last 24.55) - open/close 15509 (time 1788627683.57 total 776.87 last 20.36) - open/close 15661 (time 1788627693.58 total 786.87 last 15.20) - open/close 15952 (time 1788627704.66 total 797.95 last 26.26) - open/close 16169 (time 1788627715.47 total 808.76 last 20.07) - open/close 16373 (time 1788627726.13 total 819.42 last 19.14) - open/close 16673 (time 1788627736.66 total 829.95 last 28.49) - open/close 16916 (time 1788627748.89 total 842.19 last 19.86) - open/close 17199 (time 1788627758.90 total 852.20 last 28.28) - open/close 17405 (time 1788627769.29 total 862.59 last 19.82) - open/close 17679 (time 1788627779.31 total 872.61 last 27.35) - open/close 17886 (time 1788627789.32 total 882.61 last 20.69) - open/close 18076 (time 1788627799.45 total 892.75 last 18.75) - open/close 18133 (time 1788627810.44 total 903.74 last 5.19) - open/close 18287 (time 1788627821.06 total 914.36 last 14.50) - open/close 18460 (time 1788627831.07 total 924.36 last 17.29) - open/close 18635 (time 1788627841.07 total 934.37 last 17.49) - open/close 18735 (time 1788627851.08 total 944.37 last 9.99) - open/close 18936 (time 1788627861.86 total 955.16 last 18.64) - open/close 19130 (time 1788627872.90 total 966.19 last 17.58) - open/close 19242 (time 1788627883.35 total 976.65 last 10.71) - open/close 19334 (time 1788627894.97 total 988.26 last 7.92) - open/close 19618 (time 1788627904.98 total 998.28 last 28.35) - open/close 19884 (time 1788627915.00 total 1008.29 last 26.57) - open/close 20000 (time 1788627917.00 total 1010.30 last 57.92) - open/close 20797 (time 1788627927.00 total 1020.30 last 79.69) - open/close 21762 (time 1788627937.02 total 1030.32 last 96.33) - open/close 22702 (time 1788627947.02 total 1040.32 last 93.98) - open/close 23532 (time 1788627957.03 total 1050.33 last 82.88) - open/close 24426 (time 1788627967.04 total 1060.34 last 89.35) - open/close 25328 (time 1788627977.05 total 1070.35 last 90.11) - open/close 26166 (time 1788627987.06 total 1080.36 last 83.71) - open/close 27041 (time 1788627997.07 total 1090.37 last 87.41) - open/close 27919 (time 1788628007.08 total 1100.38 last 87.69) - open/close 28824 (time 1788628017.09 total 1110.39 last 90.43) - open/close 29692 (time 1788628027.10 total 1120.40 last 86.70) - open/close 30000 (time 1788628030.41 total 1123.71 last 93.01) - open/close 30892 (time 1788628040.42 total 1133.72 last 89.10) - open/close 31790 (time 1788628050.44 total 1143.73 last 89.69) - open/close 32647 (time 1788628060.44 total 1153.74 last 85.65) - open/close 33436 (time 1788628070.45 total 1163.74 last 78.87) - open/close 34097 (time 1788628080.46 total 1173.75 last 66.03) - open/close 34949 (time 1788628090.48 total 1183.77 last 85.02) - open/close 35844 (time 1788628100.48 total 1193.78 last 89.44) - open/close 36716 (time 1788628110.49 total 1203.78 last 87.20) - open/close 37679 (time 1788628120.51 total 1213.81 last 96.05) - open/close 38666 (time 1788628130.52 total 1223.82 last 98.61) - open/close 39562 (time 1788628140.52 total 1233.82 last 89.59) - open/close 40000 (time 1788628145.63 total 1238.92 last 85.77) - open/close 40847 (time 1788628155.63 total 1248.93 last 84.67) - open/close 41604 (time 1788628165.64 total 1258.94 last 75.61) - open/close 42340 (time 1788628175.66 total 1268.96 last 73.50) - open/close 43216 (time 1788628185.67 total 1278.96 last 87.53) - open/close 43975 (time 1788628195.67 total 1288.97 last 75.85) - open/close 44795 (time 1788628205.68 total 1298.98 last 81.92) - open/close 45709 (time 1788628215.70 total 1308.99 last 91.26) - open/close 46531 (time 1788628225.70 total 1319.00 last 82.19) - open/close 47407 (time 1788628235.70 total 1329.00 last 87.56) - open/close 48223 (time 1788628245.71 total 1339.00 last 81.58) - open/close 48998 (time 1788628255.71 total 1349.01 last 77.47) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1356.16 seconds: 36.60 ops/second - unlinked 0 (time 1788628266 ; total 0 ; last 0) - unlinked 10000 (time 1788628380 ; total 114 ; last 114) - unlinked 20000 (time 1788628490 ; total 224 ; last 110) - unlinked 30000 (time 1788628555 ; total 289 ; last 65) - unlinked 40000 (time 1788628621 ; total 355 ; last 66) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 420 seconds: 118.169044 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg359-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg359-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg359-server stop mds service on oleg359-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg359-server stop mds service on oleg359-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg359-server unloading modules via unload_modules_local on: 'oleg359-server' oleg359-server: oleg359-server.virtnet: executing unload_modules_local oleg359-server: kmemleak disabled modules unloaded. pdsh@oleg359-client: oleg359-client: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-client: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2