== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:03:42 (1788444222) start mds service on oleg226-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 3 07:51 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg226-server' oleg226-server: oleg226-server.virtnet: executing load_modules_local oleg226-server: kmemleak disabled oleg226-server: Loading modules from /home/green/git/lustre-release/lustre oleg226-server: detected 4 online CPUs by sysfs oleg226-server: MODOPTS_LIBCFS= oleg226-server: Force libcfs to create 2 CPU partitions oleg226-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg226-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg226-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg226-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg226-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg226-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0000 oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 225 (time 1788444331.41 total 10.18 last 22.11) - open/close 332 (time 1788444341.52 total 20.29 last 10.58) - open/close 570 (time 1788444351.73 total 30.49 last 23.32) - open/close 853 (time 1788444362.00 total 40.76 last 27.55) - open/close 1009 (time 1788444372.02 total 50.78 last 15.57) - open/close 1205 (time 1788444382.03 total 60.79 last 19.58) - open/close 1402 (time 1788444392.45 total 71.22 last 18.89) - open/close 1655 (time 1788444402.47 total 81.24 last 25.26) - open/close 1870 (time 1788444412.47 total 91.24 last 21.50) - open/close 2061 (time 1788444423.07 total 101.83 last 18.02) - open/close 2396 (time 1788444433.07 total 111.84 last 33.48) - open/close 2589 (time 1788444443.43 total 122.19 last 18.64) - open/close 2873 (time 1788444454.22 total 132.99 last 26.31) - open/close 3040 (time 1788444464.23 total 143.00 last 16.69) - open/close 3255 (time 1788444475.22 total 153.99 last 19.55) - open/close 3393 (time 1788444485.23 total 163.99 last 13.79) - open/close 3658 (time 1788444495.44 total 174.21 last 25.94) - open/close 3931 (time 1788444505.76 total 184.52 last 26.47) - open/close 4170 (time 1788444515.76 total 194.53 last 23.89) - open/close 4491 (time 1788444525.79 total 204.55 last 32.02) - open/close 4669 (time 1788444536.03 total 214.80 last 17.37) - open/close 4914 (time 1788444546.25 total 225.01 last 23.98) - open/close 5149 (time 1788444556.69 total 235.46 last 22.50) - open/close 5384 (time 1788444567.13 total 245.89 last 22.52) - open/close 5519 (time 1788444577.16 total 255.92 last 13.46) - open/close 5692 (time 1788444588.16 total 266.93 last 15.71) - open/close 5982 (time 1788444598.17 total 276.94 last 28.98) - open/close 6273 (time 1788444608.65 total 287.42 last 27.77) - open/close 6494 (time 1788444619.47 total 298.23 last 20.43) - open/close 6799 (time 1788444630.03 total 308.80 last 28.87) - open/close 7115 (time 1788444640.03 total 318.80 last 31.59) - open/close 7306 (time 1788444650.05 total 328.82 last 19.07) - open/close 7551 (time 1788444660.05 total 338.82 last 24.50) - open/close 7822 (time 1788444670.33 total 349.10 last 26.37) - open/close 8063 (time 1788444680.34 total 359.11 last 24.07) - open/close 8348 (time 1788444690.54 total 369.30 last 27.95) - open/close 8572 (time 1788444700.93 total 379.70 last 21.54) - open/close 8725 (time 1788444710.94 total 389.70 last 15.30) - open/close 8893 (time 1788444720.95 total 399.71 last 16.78) - open/close 9130 (time 1788444730.98 total 409.75 last 23.61) - open/close 9379 (time 1788444740.99 total 419.76 last 24.88) - open/close 9674 (time 1788444751.95 total 430.71 last 26.93) - open/close 9811 (time 1788444761.97 total 440.74 last 13.67) - open/close 9982 (time 1788444771.99 total 450.76 last 17.07) - open/close 10068 (time 1788444782.73 total 461.50 last 8.01) - open/close 10235 (time 1788444793.32 total 472.09 last 15.77) - open/close 10512 (time 1788444803.93 total 482.69 last 26.11) - open/close 10675 (time 1788444817.21 total 495.97 last 12.27) - open/close 10963 (time 1788444827.22 total 505.98 last 28.77) - open/close 11117 (time 1788444837.23 total 516.00 last 15.38) - open/close 11375 (time 1788444847.70 total 526.47 last 24.63) - open/close 11602 (time 1788444857.71 total 536.48 last 22.68) - open/close 11903 (time 1788444870.73 total 549.50 last 23.13) - open/close 12119 (time 1788444880.99 total 559.76 last 21.05) - open/close 12381 (time 1788444891.15 total 569.92 last 25.79) - open/close 12663 (time 1788444902.18 total 580.94 last 25.58) - open/close 12831 (time 1788444912.53 total 591.30 last 16.22) - open/close 12976 (time 1788444922.55 total 601.32 last 14.47) - open/close 13100 (time 1788444932.82 total 611.59 last 12.08) - open/close 13235 (time 1788444943.35 total 622.12 last 12.82) - open/close 13414 (time 1788444953.36 total 632.13 last 17.88) - open/close 13617 (time 1788444963.57 total 642.34 last 19.89) - open/close 13871 (time 1788444973.65 total 652.42 last 25.18) - open/close 14091 (time 1788444985.41 total 664.17 last 18.72) - open/close 14208 (time 1788444996.11 total 674.88 last 10.93) - open/close 14386 (time 1788445006.12 total 684.89 last 17.79) - open/close 14520 (time 1788445016.12 total 694.89 last 13.40) - open/close 14797 (time 1788445026.27 total 705.03 last 27.31) - open/close 14929 (time 1788445036.94 total 715.71 last 12.37) - open/close 15145 (time 1788445047.65 total 726.42 last 20.16) - open/close 15358 (time 1788445057.66 total 736.43 last 21.28) - open/close 15568 (time 1788445069.54 total 748.31 last 17.68) - open/close 15828 (time 1788445079.55 total 758.31 last 25.99) - open/close 16099 (time 1788445089.89 total 768.66 last 26.19) - open/close 16357 (time 1788445099.93 total 778.70 last 25.71) - open/close 16600 (time 1788445109.96 total 788.72 last 24.24) - open/close 16729 (time 1788445120.86 total 799.62 last 11.84) - open/close 16933 (time 1788445130.98 total 809.74 last 20.16) - open/close 17102 (time 1788445140.98 total 819.75 last 16.89) - open/close 17273 (time 1788445151.00 total 829.76 last 17.08) - open/close 17447 (time 1788445161.00 total 839.77 last 17.39) - open/close 17697 (time 1788445172.05 total 850.82 last 22.63) - open/close 17932 (time 1788445182.05 total 860.82 last 23.50) - open/close 18105 (time 1788445192.68 total 871.45 last 16.27) - open/close 18190 (time 1788445202.69 total 881.46 last 8.50) - open/close 18450 (time 1788445212.69 total 891.46 last 25.98) - open/close 18735 (time 1788445223.45 total 902.22 last 26.49) - open/close 19012 (time 1788445234.33 total 913.09 last 25.48) - open/close 19267 (time 1788445245.03 total 923.80 last 23.82) - open/close 19359 (time 1788445255.66 total 934.43 last 8.65) - open/close 19534 (time 1788445265.68 total 944.45 last 17.46) - open/close 19603 (time 1788445275.69 total 954.46 last 6.90) - open/close 19728 (time 1788445285.71 total 964.48 last 12.47) - open/close 19922 (time 1788445295.73 total 974.50 last 19.37) - open/close 20418 (time 1788445305.73 total 984.50 last 49.60) - open/close 21170 (time 1788445315.74 total 994.50 last 75.15) - open/close 21971 (time 1788445325.74 total 1004.51 last 80.07) - open/close 22856 (time 1788445335.74 total 1014.51 last 88.48) - open/close 23768 (time 1788445345.75 total 1024.52 last 91.14) - open/close 24639 (time 1788445355.75 total 1034.52 last 87.10) - open/close 25399 (time 1788445365.77 total 1044.54 last 75.86) - open/close 26309 (time 1788445375.77 total 1054.54 last 90.96) - open/close 27500 (time 1788445385.77 total 1064.54 last 119.10) - open/close 28509 (time 1788445395.77 total 1074.54 last 100.90) - open/close 29440 (time 1788445405.78 total 1084.54 last 93.06) - open/close 30000 (time 1788445411.43 total 1090.20 last 99.09) - open/close 30912 (time 1788445421.43 total 1100.20 last 91.18) - open/close 31675 (time 1788445431.44 total 1110.20 last 76.26) - open/close 32344 (time 1788445441.44 total 1120.21 last 66.84) - open/close 32827 (time 1788445451.45 total 1130.21 last 48.30) - open/close 33288 (time 1788445461.47 total 1140.23 last 46.01) - open/close 34025 (time 1788445471.48 total 1150.25 last 73.58) - open/close 34783 (time 1788445481.48 total 1160.25 last 75.78) - open/close 35631 (time 1788445491.49 total 1170.26 last 84.77) - open/close 36509 (time 1788445501.49 total 1180.26 last 87.77) - open/close 37343 (time 1788445511.49 total 1190.26 last 83.37) - open/close 38084 (time 1788445521.54 total 1200.30 last 73.80) - open/close 39021 (time 1788445531.55 total 1210.31 last 93.59) - open/close 39662 (time 1788445541.55 total 1220.32 last 64.07) - open/close 40000 (time 1788445545.88 total 1224.65 last 78.04) - open/close 40834 (time 1788445555.89 total 1234.65 last 83.37) - open/close 41640 (time 1788445565.89 total 1244.65 last 80.59) - open/close 42454 (time 1788445575.89 total 1254.66 last 81.38) - open/close 43300 (time 1788445585.90 total 1264.67 last 84.53) - open/close 44153 (time 1788445595.91 total 1274.68 last 85.20) - open/close 44932 (time 1788445605.92 total 1284.69 last 77.80) - open/close 45767 (time 1788445615.92 total 1294.69 last 83.50) - open/close 46578 (time 1788445625.93 total 1304.70 last 81.03) - open/close 47389 (time 1788445635.93 total 1314.70 last 81.08) - open/close 48041 (time 1788445645.94 total 1324.71 last 65.13) - open/close 48804 (time 1788445655.95 total 1334.72 last 76.25) - open/close 49580 (time 1788445665.95 total 1344.72 last 77.58) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1345.27 seconds: 36.89 ops/second - unlinked 0 (time 1788445671 ; total 0 ; last 0) - unlinked 10000 (time 1788445783 ; total 112 ; last 112) - unlinked 20000 (time 1788445901 ; total 230 ; last 118) - unlinked 30000 (time 1788445969 ; total 298 ; last 68) - unlinked 40000 (time 1788446032 ; total 361 ; last 63) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 419 seconds: 118.451073 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg226-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg226-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg226-server stop mds service on oleg226-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg226-server stop mds service on oleg226-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg226-server unloading modules via unload_modules_local on: 'oleg226-server' oleg226-server: oleg226-server.virtnet: executing unload_modules_local oleg226-server: kmemleak disabled modules unloaded. pdsh@oleg226-client: oleg226-client: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-client: ssh exited with exit code 2