== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:20:11 (1789068011) start mds service on oleg317-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 10 02:33 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: kmemleak disabled oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg317-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg317-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg317-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg317-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 225 (time 1789068108.65 total 10.02 last 22.45) - open/close 436 (time 1789068118.72 total 20.09 last 20.96) - open/close 682 (time 1789068128.75 total 30.12 last 24.53) - open/close 911 (time 1789068138.77 total 40.14 last 22.86) - open/close 1149 (time 1789068148.77 total 50.14 last 23.79) - open/close 1372 (time 1789068158.94 total 60.31 last 21.93) - open/close 1617 (time 1789068168.95 total 70.32 last 24.47) - open/close 1842 (time 1789068179.44 total 80.81 last 21.46) - open/close 2085 (time 1789068189.45 total 90.82 last 24.26) - open/close 2321 (time 1789068199.46 total 100.83 last 23.58) - open/close 2602 (time 1789068210.42 total 111.79 last 25.64) - open/close 2703 (time 1789068220.92 total 122.29 last 9.62) - open/close 2916 (time 1789068231.92 total 133.28 last 19.37) - open/close 3066 (time 1789068241.94 total 143.30 last 14.97) - open/close 3382 (time 1789068251.95 total 153.32 last 31.56) - open/close 3611 (time 1789068262.07 total 163.43 last 22.63) - open/close 3845 (time 1789068272.08 total 173.45 last 23.37) - open/close 4075 (time 1789068282.37 total 183.74 last 22.35) - open/close 4250 (time 1789068292.94 total 194.31 last 16.55) - open/close 4506 (time 1789068302.96 total 204.33 last 25.56) - open/close 4736 (time 1789068313.02 total 214.38 last 22.87) - open/close 5037 (time 1789068323.90 total 225.26 last 27.67) - open/close 5313 (time 1789068334.13 total 235.50 last 26.97) - open/close 5591 (time 1789068344.14 total 245.51 last 27.78) - open/close 5841 (time 1789068354.15 total 255.52 last 24.97) - open/close 6046 (time 1789068366.90 total 268.26 last 16.08) - open/close 6192 (time 1789068377.28 total 278.65 last 14.05) - open/close 6348 (time 1789068387.45 total 288.81 last 15.35) - open/close 6542 (time 1789068398.00 total 299.37 last 18.37) - open/close 6755 (time 1789068408.02 total 309.39 last 21.26) - open/close 6970 (time 1789068419.69 total 321.06 last 18.42) - open/close 7252 (time 1789068429.70 total 331.07 last 28.19) - open/close 7457 (time 1789068440.56 total 341.93 last 18.87) - open/close 7622 (time 1789068450.74 total 352.11 last 16.22) - open/close 7867 (time 1789068460.98 total 362.35 last 23.92) - open/close 8119 (time 1789068471.45 total 372.82 last 24.06) - open/close 8339 (time 1789068481.46 total 382.83 last 21.98) - open/close 8641 (time 1789068491.95 total 393.32 last 28.81) - open/close 8939 (time 1789068502.96 total 404.33 last 27.05) - open/close 9096 (time 1789068512.99 total 414.36 last 15.66) - open/close 9360 (time 1789068522.99 total 424.36 last 26.39) - open/close 9544 (time 1789068533.62 total 434.98 last 17.32) - open/close 9823 (time 1789068543.63 total 445.00 last 27.87) - open/close 9999 (time 1789068553.64 total 455.01 last 17.58) - open/close 10245 (time 1789068564.35 total 465.71 last 22.97) - open/close 10496 (time 1789068574.37 total 475.74 last 25.05) - open/close 10692 (time 1789068584.39 total 485.76 last 19.55) - open/close 10861 (time 1789068597.12 total 498.48 last 13.28) - open/close 11041 (time 1789068607.38 total 508.74 last 17.55) - open/close 11146 (time 1789068618.31 total 519.67 last 9.61) - open/close 11319 (time 1789068628.85 total 530.22 last 16.40) - open/close 11561 (time 1789068639.40 total 540.77 last 22.94) - open/close 11769 (time 1789068651.33 total 552.70 last 17.43) - open/close 12001 (time 1789068663.41 total 564.78 last 19.21) - open/close 12182 (time 1789068673.42 total 574.79 last 18.08) - open/close 12335 (time 1789068684.15 total 585.52 last 14.26) - open/close 12458 (time 1789068694.77 total 596.14 last 11.58) - open/close 12585 (time 1789068705.78 total 607.15 last 11.53) - open/close 12797 (time 1789068715.80 total 617.17 last 21.17) - open/close 12993 (time 1789068725.80 total 627.17 last 19.60) - open/close 13225 (time 1789068736.44 total 637.81 last 21.79) - open/close 13482 (time 1789068746.45 total 647.82 last 25.69) - open/close 13702 (time 1789068756.87 total 658.24 last 21.11) - open/close 13902 (time 1789068767.43 total 668.80 last 18.95) - open/close 14088 (time 1789068778.17 total 679.54 last 17.31) - open/close 14361 (time 1789068788.85 total 690.22 last 25.56) - open/close 14635 (time 1789068798.87 total 700.24 last 27.36) - open/close 14878 (time 1789068808.89 total 710.26 last 24.25) - open/close 15012 (time 1789068818.89 total 720.26 last 13.40) - open/close 15209 (time 1789068828.91 total 730.28 last 19.66) - open/close 15452 (time 1789068839.41 total 740.78 last 23.14) - open/close 15656 (time 1789068850.16 total 751.53 last 18.97) - open/close 15840 (time 1789068860.50 total 761.87 last 17.80) - open/close 16066 (time 1789068870.78 total 772.14 last 21.99) - open/close 16281 (time 1789068881.27 total 782.64 last 20.49) - open/close 16507 (time 1789068892.27 total 793.64 last 20.53) - open/close 16805 (time 1789068903.34 total 804.71 last 26.92) - open/close 17062 (time 1789068913.36 total 814.72 last 25.67) - open/close 17215 (time 1789068923.36 total 824.73 last 15.30) - open/close 17464 (time 1789068934.23 total 835.60 last 22.90) - open/close 17642 (time 1789068945.00 total 846.37 last 16.53) - open/close 17854 (time 1789068955.01 total 856.38 last 21.18) - open/close 18088 (time 1789068965.70 total 867.07 last 21.88) - open/close 18301 (time 1789068975.85 total 877.22 last 20.98) - open/close 18506 (time 1789068988.66 total 890.03 last 16.01) - open/close 18769 (time 1789068998.68 total 900.04 last 26.25) - open/close 19028 (time 1789069009.51 total 910.88 last 23.90) - open/close 19254 (time 1789069019.73 total 921.10 last 22.12) - open/close 19456 (time 1789069029.74 total 931.11 last 20.17) - open/close 19662 (time 1789069039.86 total 941.23 last 20.36) - open/close 20000 (time 1789069045.32 total 946.69 last 61.95) - open/close 20724 (time 1789069055.33 total 956.69 last 72.35) - open/close 21496 (time 1789069065.33 total 966.70 last 77.17) - open/close 22210 (time 1789069075.34 total 976.70 last 71.35) - open/close 22999 (time 1789069085.34 total 986.71 last 78.86) - open/close 23751 (time 1789069095.35 total 996.71 last 75.16) - open/close 24440 (time 1789069105.37 total 1006.74 last 68.71) - open/close 25246 (time 1789069115.38 total 1016.75 last 80.55) - open/close 25943 (time 1789069125.39 total 1026.75 last 69.65) - open/close 26617 (time 1789069135.39 total 1036.76 last 67.36) - open/close 27328 (time 1789069145.40 total 1046.77 last 71.05) - open/close 28045 (time 1789069155.41 total 1056.78 last 71.60) - open/close 28730 (time 1789069165.42 total 1066.79 last 68.43) - open/close 29375 (time 1789069175.43 total 1076.80 last 64.47) - open/close 30000 (time 1789069185.38 total 1086.75 last 62.79) - open/close 30618 (time 1789069195.41 total 1096.78 last 61.61) - open/close 31265 (time 1789069205.42 total 1106.79 last 64.63) - open/close 31846 (time 1789069215.43 total 1116.80 last 58.07) - open/close 32610 (time 1789069225.44 total 1126.80 last 76.34) - open/close 33345 (time 1789069235.46 total 1136.82 last 73.35) - open/close 33994 (time 1789069245.47 total 1146.84 last 64.83) - open/close 34582 (time 1789069255.49 total 1156.85 last 58.69) - open/close 35247 (time 1789069265.49 total 1166.86 last 66.48) - open/close 36078 (time 1789069275.49 total 1176.86 last 83.06) - open/close 36786 (time 1789069285.52 total 1186.88 last 70.65) - open/close 37522 (time 1789069295.52 total 1196.89 last 73.54) - open/close 38289 (time 1789069305.54 total 1206.90 last 76.61) - open/close 39049 (time 1789069315.54 total 1216.91 last 75.98) - open/close 39769 (time 1789069325.54 total 1226.91 last 71.99) - open/close 40000 (time 1789069328.63 total 1230.00 last 74.86) - open/close 40670 (time 1789069338.64 total 1240.00 last 66.94) - open/close 41346 (time 1789069348.65 total 1250.02 last 67.48) - open/close 41987 (time 1789069358.68 total 1260.05 last 63.92) - open/close 42625 (time 1789069368.69 total 1270.06 last 63.74) - open/close 43258 (time 1789069378.69 total 1280.06 last 63.29) - open/close 43917 (time 1789069388.70 total 1290.06 last 65.88) - open/close 44618 (time 1789069398.70 total 1300.07 last 70.08) - open/close 45270 (time 1789069408.70 total 1310.07 last 65.16) - open/close 46098 (time 1789069418.72 total 1320.09 last 82.67) - open/close 46840 (time 1789069428.73 total 1330.10 last 74.13) - open/close 47485 (time 1789069438.73 total 1340.10 last 64.49) - open/close 48170 (time 1789069448.73 total 1350.10 last 68.50) - open/close 48766 (time 1789069458.74 total 1360.11 last 59.53) - open/close 49378 (time 1789069468.74 total 1370.11 last 61.20) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1374.24 seconds: 36.12 ops/second - unlinked 0 (time 1789069477 ; total 0 ; last 0) - unlinked 10000 (time 1789069642 ; total 165 ; last 165) - unlinked 20000 (time 1789069787 ; total 310 ; last 145) - unlinked 30000 (time 1789069926 ; total 449 ; last 139) - unlinked 40000 (time 1789070073 ; total 596 ; last 147) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 724 seconds: 68.551102 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local oleg317-server: kmemleak disabled modules unloaded. pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2