== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:03:22 (1790715802) start mds service on oleg322-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1946968 Sep 29 04:58 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg322-server' oleg322-server: oleg322-server.virtnet: executing load_modules_local oleg322-server: kmemleak disabled oleg322-server: Loading modules from /home/green/git/lustre-release/lustre oleg322-server: detected 4 online CPUs by sysfs oleg322-server: MODOPTS_LIBCFS= oleg322-server: Force libcfs to create 2 CPU partitions oleg322-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg322-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg322-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg322-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg322-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg322-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 57 (time 1790715920.81 total 10.86 last 5.25) - open/close 181 (time 1790715930.83 total 20.88 last 12.37) - open/close 363 (time 1790715940.86 total 30.91 last 18.16) - open/close 589 (time 1790715951.46 total 41.51 last 21.32) - open/close 768 (time 1790715964.21 total 54.26 last 14.04) - open/close 915 (time 1790715975.13 total 65.18 last 13.46) - open/close 1031 (time 1790715985.72 total 75.77 last 10.96) - open/close 1214 (time 1790715996.39 total 86.44 last 17.15) - open/close 1353 (time 1790716006.70 total 96.75 last 13.48) - open/close 1595 (time 1790716016.71 total 106.76 last 24.18) - open/close 1797 (time 1790716027.21 total 117.26 last 19.23) - open/close 1991 (time 1790716037.28 total 127.33 last 19.26) - open/close 2135 (time 1790716047.30 total 137.35 last 14.38) - open/close 2260 (time 1790716057.30 total 147.35 last 12.49) - open/close 2384 (time 1790716067.37 total 157.42 last 12.31) - open/close 2584 (time 1790716077.37 total 167.42 last 20.00) - open/close 2727 (time 1790716087.37 total 177.43 last 14.30) - open/close 2893 (time 1790716100.06 total 190.11 last 13.09) - open/close 3009 (time 1790716112.73 total 202.79 last 9.15) - open/close 3126 (time 1790716123.41 total 213.46 last 10.96) - open/close 3198 (time 1790716133.43 total 223.49 last 7.18) - open/close 3415 (time 1790716144.25 total 234.30 last 20.07) - open/close 3588 (time 1790716155.11 total 245.16 last 15.93) - open/close 3771 (time 1790716165.99 total 256.04 last 16.82) - open/close 4050 (time 1790716176.61 total 266.67 last 26.25) - open/close 4204 (time 1790716186.61 total 276.67 last 15.40) - open/close 4465 (time 1790716197.07 total 287.13 last 24.95) - open/close 4754 (time 1790716207.59 total 297.64 last 27.49) - open/close 4972 (time 1790716217.59 total 307.64 last 21.79) - open/close 5216 (time 1790716228.14 total 318.19 last 23.14) - open/close 5518 (time 1790716238.49 total 328.55 last 29.16) - open/close 5764 (time 1790716248.51 total 338.56 last 24.57) - open/close 5977 (time 1790716258.96 total 349.01 last 20.38) - open/close 6167 (time 1790716268.97 total 359.03 last 18.97) - open/close 6332 (time 1790716279.51 total 369.56 last 15.66) - open/close 6552 (time 1790716289.88 total 379.93 last 21.21) - open/close 6744 (time 1790716300.19 total 390.24 last 18.63) - open/close 6870 (time 1790716310.20 total 400.25 last 12.59) - open/close 7120 (time 1790716321.26 total 411.31 last 22.61) - open/close 7229 (time 1790716332.02 total 422.08 last 10.13) - open/close 7317 (time 1790716342.88 total 432.93 last 8.10) - open/close 7425 (time 1790716352.95 total 443.00 last 10.73) - open/close 7513 (time 1790716362.95 total 453.01 last 8.79) - open/close 7667 (time 1790716372.96 total 463.02 last 15.38) - open/close 7782 (time 1790716383.35 total 473.40 last 11.07) - open/close 7939 (time 1790716393.58 total 483.63 last 15.35) - open/close 8124 (time 1790716403.59 total 493.64 last 18.48) - open/close 8387 (time 1790716414.25 total 504.31 last 24.66) - open/close 8608 (time 1790716424.42 total 514.48 last 21.73) - open/close 8710 (time 1790716434.73 total 524.78 last 9.90) - open/close 8862 (time 1790716444.96 total 535.01 last 14.85) - open/close 8938 (time 1790716455.78 total 545.83 last 7.03) - open/close 9027 (time 1790716466.40 total 556.45 last 8.38) - open/close 9177 (time 1790716476.41 total 566.47 last 14.98) - open/close 9309 (time 1790716489.31 total 579.36 last 10.24) - open/close 9409 (time 1790716499.34 total 589.40 last 9.96) - open/close 9556 (time 1790716511.77 total 601.82 last 11.83) - open/close 9727 (time 1790716522.73 total 612.78 last 15.60) - open/close 9920 (time 1790716533.96 total 624.01 last 17.18) - open/close 10000 (time 1790716536.43 total 626.48 last 32.38) - open/close 10172 (time 1790716547.12 total 637.17 last 16.09) - open/close 10298 (time 1790716558.04 total 648.09 last 11.54) - open/close 10392 (time 1790716568.54 total 658.59 last 8.95) - open/close 10638 (time 1790716578.55 total 668.60 last 24.58) - open/close 10851 (time 1790716588.84 total 678.89 last 20.71) - open/close 11017 (time 1790716599.72 total 689.77 last 15.26) - open/close 11199 (time 1790716610.65 total 700.70 last 16.65) - open/close 11445 (time 1790716622.44 total 712.49 last 20.87) - open/close 11609 (time 1790716632.45 total 722.50 last 16.39) - open/close 11707 (time 1790716642.45 total 732.50 last 9.80) - open/close 11899 (time 1790716653.45 total 743.50 last 17.45) - open/close 12138 (time 1790716664.22 total 754.27 last 22.20) - open/close 12317 (time 1790716674.22 total 764.27 last 17.90) - open/close 12490 (time 1790716684.85 total 774.90 last 16.28) - open/close 12580 (time 1790716694.95 total 785.00 last 8.91) - open/close 12809 (time 1790716707.55 total 797.60 last 18.17) - open/close 12909 (time 1790716717.84 total 807.90 last 9.72) - open/close 13003 (time 1790716729.95 total 820.00 last 7.76) - open/close 13240 (time 1790716740.37 total 830.43 last 22.74) - open/close 13479 (time 1790716750.88 total 840.93 last 22.75) - open/close 13715 (time 1790716763.39 total 853.44 last 18.87) - open/close 13873 (time 1790716773.40 total 863.45 last 15.78) - open/close 14121 (time 1790716784.36 total 874.41 last 22.62) - open/close 14448 (time 1790716794.38 total 884.43 last 32.66) - open/close 14648 (time 1790716804.58 total 894.63 last 19.60) - open/close 14877 (time 1790716814.58 total 904.63 last 22.89) - open/close 15087 (time 1790716825.15 total 915.20 last 19.86) - open/close 15284 (time 1790716835.16 total 925.21 last 19.69) - open/close 15319 (time 1790716845.34 total 935.40 last 3.44) - open/close 15569 (time 1790716855.98 total 946.03 last 23.51) - open/close 15735 (time 1790716866.33 total 956.38 last 16.04) - open/close 15836 (time 1790716876.63 total 966.68 last 9.80) - open/close 16087 (time 1790716886.63 total 976.68 last 25.10) - open/close 16327 (time 1790716896.65 total 986.70 last 23.96) - open/close 16451 (time 1790716906.66 total 996.71 last 12.39) - open/close 16651 (time 1790716916.82 total 1006.88 last 19.68) - open/close 16769 (time 1790716927.64 total 1017.69 last 10.91) - open/close 16850 (time 1790716938.66 total 1028.71 last 7.35) - open/close 16938 (time 1790716949.04 total 1039.09 last 8.48) - open/close 17140 (time 1790716959.97 total 1050.02 last 18.47) - open/close 17423 (time 1790716969.99 total 1060.04 last 28.24) - open/close 18116 (time 1790716979.99 total 1070.05 last 69.28) - open/close 18834 (time 1790716990.00 total 1080.05 last 71.79) - open/close 19468 (time 1790717000.02 total 1090.07 last 63.26) - open/close 20000 (time 1790717007.52 total 1097.57 last 70.95) - open/close 20737 (time 1790717017.53 total 1107.58 last 73.63) - open/close 21423 (time 1790717027.53 total 1117.58 last 68.56) - open/close 22198 (time 1790717037.53 total 1127.59 last 77.47) - open/close 22920 (time 1790717047.55 total 1137.60 last 72.08) - open/close 23604 (time 1790717057.55 total 1147.60 last 68.39) - open/close 24289 (time 1790717067.56 total 1157.61 last 68.48) - open/close 24976 (time 1790717077.57 total 1167.62 last 68.61) - open/close 25661 (time 1790717087.58 total 1177.63 last 68.42) - open/close 26371 (time 1790717097.59 total 1187.64 last 70.95) - open/close 27114 (time 1790717107.59 total 1197.64 last 74.28) - open/close 27886 (time 1790717117.60 total 1207.65 last 77.09) - open/close 28579 (time 1790717127.60 total 1217.66 last 69.29) - open/close 29306 (time 1790717137.62 total 1227.67 last 72.60) - open/close 29969 (time 1790717147.62 total 1237.68 last 66.26) - open/close 30739 (time 1790717157.63 total 1247.68 last 76.94) - open/close 31457 (time 1790717167.63 total 1257.68 last 71.79) - open/close 32143 (time 1790717177.64 total 1267.69 last 68.59) - open/close 32860 (time 1790717187.64 total 1277.69 last 71.65) - open/close 33529 (time 1790717197.66 total 1287.71 last 66.77) - open/close 34224 (time 1790717207.66 total 1297.71 last 69.50) - open/close 34877 (time 1790717217.67 total 1307.72 last 65.25) - open/close 35610 (time 1790717227.68 total 1317.74 last 73.19) - open/close 36329 (time 1790717237.69 total 1327.74 last 71.88) - open/close 36936 (time 1790717247.70 total 1337.76 last 60.60) - open/close 37559 (time 1790717257.70 total 1347.76 last 62.30) - open/close 38283 (time 1790717267.75 total 1357.80 last 72.05) - open/close 39020 (time 1790717277.79 total 1367.84 last 73.43) - open/close 39667 (time 1790717287.81 total 1377.86 last 64.60) - open/close 40000 (time 1790717293.72 total 1383.77 last 56.33) - open/close 40653 (time 1790717303.73 total 1393.78 last 65.24) - open/close 41225 (time 1790717313.74 total 1403.79 last 57.12) - open/close 41974 (time 1790717323.75 total 1413.80 last 74.84) - open/close 42716 (time 1790717333.76 total 1423.81 last 74.09) - open/close 43425 (time 1790717343.77 total 1433.82 last 70.88) - open/close 44213 (time 1790717353.77 total 1443.82 last 78.79) - open/close 45059 (time 1790717363.77 total 1453.82 last 84.55) - open/close 45878 (time 1790717373.78 total 1463.83 last 81.83) - open/close 46779 (time 1790717383.79 total 1473.84 last 90.00) - open/close 47530 (time 1790717393.81 total 1483.86 last 74.96) - open/close 48187 (time 1790717403.83 total 1493.89 last 65.54) - open/close 49058 (time 1790717413.83 total 1503.89 last 87.10) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1512.72 seconds: 32.81 ops/second - unlinked 0 (time 1790717427 ; total 0 ; last 0) - unlinked 10000 (time 1790717565 ; total 138 ; last 138) - unlinked 20000 (time 1790717712 ; total 285 ; last 147) - unlinked 30000 (time 1790717853 ; total 426 ; last 141) - unlinked 40000 (time 1790717992 ; total 565 ; last 139) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 707 seconds: 70.199432 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg322-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg322-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg322-server stop mds service on oleg322-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg322-server stop mds service on oleg322-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg322-server unloading modules via unload_modules_local on: 'oleg322-server' oleg322-server: oleg322-server.virtnet: executing unload_modules_local oleg322-server: kmemleak disabled modules unloaded. pdsh@oleg322-client: oleg322-client: ssh exited with exit code 2 pdsh@oleg322-client: oleg322-server: ssh exited with exit code 2 pdsh@oleg322-client: oleg322-client: ssh exited with exit code 2 pdsh@oleg322-client: oleg322-server: ssh exited with exit code 2