== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:27:11 (1788528431) start mds service on oleg225-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 3 15:14 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing load_modules_local oleg225-server: kmemleak disabled oleg225-server: Loading modules from /home/green/git/lustre-release/lustre oleg225-server: detected 4 online CPUs by sysfs oleg225-server: MODOPTS_LIBCFS= oleg225-server: Force libcfs to create 2 CPU partitions oleg225-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg225-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg225-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg225-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg225-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg225-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-OST0000 oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 125 (time 1788528558.14 total 10.08 last 12.40) - open/close 356 (time 1788528568.71 total 20.65 last 21.85) - open/close 485 (time 1788528578.94 total 30.88 last 12.61) - open/close 570 (time 1788528588.95 total 40.89 last 8.49) - open/close 757 (time 1788528600.60 total 52.54 last 16.06) - open/close 990 (time 1788528611.02 total 62.96 last 22.35) - open/close 1178 (time 1788528621.68 total 73.62 last 17.63) - open/close 1433 (time 1788528631.70 total 83.64 last 25.47) - open/close 1637 (time 1788528641.71 total 93.65 last 20.38) - open/close 1849 (time 1788528652.74 total 104.68 last 19.22) - open/close 2062 (time 1788528662.75 total 114.69 last 21.28) - open/close 2297 (time 1788528673.80 total 125.74 last 21.26) - open/close 2591 (time 1788528686.59 total 138.53 last 22.99) - open/close 2699 (time 1788528696.99 total 148.93 last 10.39) - open/close 2881 (time 1788528707.26 total 159.20 last 17.72) - open/close 3105 (time 1788528717.26 total 169.20 last 22.39) - open/close 3350 (time 1788528727.27 total 179.20 last 24.49) - open/close 3573 (time 1788528737.35 total 189.29 last 22.11) - open/close 3811 (time 1788528748.41 total 200.35 last 21.53) - open/close 4042 (time 1788528758.92 total 210.85 last 21.98) - open/close 4223 (time 1788528768.99 total 220.93 last 17.97) - open/close 4456 (time 1788528779.58 total 231.52 last 21.99) - open/close 4578 (time 1788528789.92 total 241.86 last 11.81) - open/close 4643 (time 1788528799.94 total 251.87 last 6.49) - open/close 4778 (time 1788528809.95 total 261.89 last 13.49) - open/close 4992 (time 1788528819.96 total 271.90 last 21.37) - open/close 5241 (time 1788528830.74 total 282.68 last 23.10) - open/close 5502 (time 1788528840.74 total 292.68 last 26.09) - open/close 5770 (time 1788528851.63 total 303.57 last 24.62) - open/close 6063 (time 1788528862.53 total 314.47 last 26.87) - open/close 6298 (time 1788528872.76 total 324.70 last 22.97) - open/close 6465 (time 1788528882.78 total 334.72 last 16.67) - open/close 6722 (time 1788528893.37 total 345.31 last 24.26) - open/close 6970 (time 1788528904.43 total 356.37 last 22.42) - open/close 7186 (time 1788528914.82 total 366.76 last 20.79) - open/close 7271 (time 1788528924.87 total 376.81 last 8.46) - open/close 7534 (time 1788528935.71 total 387.65 last 24.27) - open/close 7813 (time 1788528945.71 total 397.65 last 27.88) - open/close 8027 (time 1788528955.74 total 407.68 last 21.34) - open/close 8274 (time 1788528965.75 total 417.69 last 24.67) - open/close 8471 (time 1788528975.77 total 427.71 last 19.66) - open/close 8662 (time 1788528985.79 total 437.73 last 19.07) - open/close 8836 (time 1788528996.29 total 448.23 last 16.56) - open/close 9073 (time 1788529006.65 total 458.59 last 22.89) - open/close 9312 (time 1788529016.65 total 468.59 last 23.90) - open/close 9526 (time 1788529026.82 total 478.76 last 21.05) - open/close 9645 (time 1788529037.79 total 489.73 last 10.85) - open/close 9808 (time 1788529048.52 total 500.46 last 15.19) - open/close 10000 (time 1788529055.77 total 507.71 last 26.47) - open/close 10266 (time 1788529065.78 total 517.71 last 26.59) - open/close 10533 (time 1788529081.78 total 533.72 last 16.68) - open/close 10809 (time 1788529092.09 total 544.03 last 26.76) - open/close 11015 (time 1788529102.10 total 554.04 last 20.58) - open/close 11241 (time 1788529112.52 total 564.45 last 21.70) - open/close 11513 (time 1788529123.59 total 575.53 last 24.57) - open/close 11763 (time 1788529133.60 total 585.54 last 24.98) - open/close 12011 (time 1788529143.61 total 595.55 last 24.77) - open/close 12206 (time 1788529154.38 total 606.32 last 18.10) - open/close 12449 (time 1788529165.40 total 617.34 last 22.05) - open/close 12609 (time 1788529175.69 total 627.63 last 15.55) - open/close 12882 (time 1788529186.11 total 638.05 last 26.19) - open/close 13051 (time 1788529196.44 total 648.38 last 16.37) - open/close 13230 (time 1788529206.45 total 658.39 last 17.88) - open/close 13455 (time 1788529216.46 total 668.40 last 22.48) - open/close 13667 (time 1788529226.76 total 678.70 last 20.58) - open/close 13844 (time 1788529236.82 total 688.76 last 17.59) - open/close 13955 (time 1788529247.75 total 699.68 last 10.16) - open/close 14014 (time 1788529260.79 total 712.73 last 4.52) - open/close 14210 (time 1788529271.24 total 723.18 last 18.76) - open/close 14382 (time 1788529281.36 total 733.30 last 16.99) - open/close 14464 (time 1788529291.37 total 743.31 last 8.19) - open/close 14652 (time 1788529301.39 total 753.33 last 18.77) - open/close 14779 (time 1788529311.45 total 763.39 last 12.62) - open/close 15074 (time 1788529321.46 total 773.40 last 29.47) - open/close 15280 (time 1788529331.47 total 783.41 last 20.58) - open/close 15388 (time 1788529341.90 total 793.84 last 10.35) - open/close 15538 (time 1788529352.01 total 803.95 last 14.84) - open/close 15696 (time 1788529362.99 total 814.92 last 14.40) - open/close 15893 (time 1788529373.85 total 825.79 last 18.13) - open/close 16052 (time 1788529384.15 total 836.08 last 15.44) - open/close 16230 (time 1788529394.15 total 846.09 last 17.78) - open/close 16396 (time 1788529406.73 total 858.67 last 13.20) - open/close 16538 (time 1788529417.36 total 869.30 last 13.36) - open/close 16682 (time 1788529427.65 total 879.59 last 14.00) - open/close 16873 (time 1788529438.68 total 890.62 last 17.32) - open/close 17083 (time 1788529448.78 total 900.71 last 20.79) - open/close 17324 (time 1788529458.78 total 910.72 last 24.10) - open/close 17519 (time 1788529469.93 total 921.87 last 17.49) - open/close 17651 (time 1788529480.79 total 932.72 last 12.16) - open/close 17845 (time 1788529491.03 total 942.97 last 18.94) - open/close 18005 (time 1788529501.39 total 953.33 last 15.44) - open/close 18095 (time 1788529511.74 total 963.68 last 8.70) - open/close 18186 (time 1788529522.27 total 974.21 last 8.64) - open/close 18404 (time 1788529532.89 total 984.83 last 20.53) - open/close 18591 (time 1788529543.56 total 995.50 last 17.54) - open/close 18832 (time 1788529554.55 total 1006.49 last 21.92) - open/close 19303 (time 1788529564.55 total 1016.49 last 47.10) - open/close 19899 (time 1788529574.57 total 1026.50 last 59.52) - open/close 20632 (time 1788529584.57 total 1036.50 last 73.30) - open/close 21252 (time 1788529594.57 total 1046.51 last 61.99) - open/close 21791 (time 1788529604.59 total 1056.52 last 53.81) - open/close 22323 (time 1788529614.59 total 1066.53 last 53.19) - open/close 22868 (time 1788529624.60 total 1076.54 last 54.41) - open/close 23556 (time 1788529634.61 total 1086.55 last 68.76) - open/close 24219 (time 1788529644.62 total 1096.56 last 66.26) - open/close 24992 (time 1788529654.63 total 1106.57 last 77.22) - open/close 25590 (time 1788529664.64 total 1116.58 last 59.73) - open/close 26309 (time 1788529674.64 total 1126.58 last 71.86) - open/close 26896 (time 1788529684.65 total 1136.58 last 58.69) - open/close 27719 (time 1788529694.65 total 1146.59 last 82.28) - open/close 28359 (time 1788529704.66 total 1156.60 last 63.94) - open/close 28912 (time 1788529714.68 total 1166.61 last 55.20) - open/close 29580 (time 1788529724.68 total 1176.62 last 66.76) - open/close 30000 (time 1788529731.55 total 1183.49 last 61.13) - open/close 30491 (time 1788529741.57 total 1193.51 last 49.03) - open/close 31072 (time 1788529751.58 total 1203.52 last 58.04) - open/close 31764 (time 1788529761.58 total 1213.52 last 69.16) - open/close 32647 (time 1788529771.60 total 1223.54 last 88.17) - open/close 33204 (time 1788529781.61 total 1233.55 last 55.63) - open/close 33677 (time 1788529791.62 total 1243.56 last 47.26) - open/close 34356 (time 1788529801.63 total 1253.57 last 67.85) - open/close 35081 (time 1788529811.63 total 1263.57 last 72.46) - open/close 35715 (time 1788529821.64 total 1273.58 last 63.37) - open/close 36371 (time 1788529831.66 total 1283.60 last 65.42) - open/close 37000 (time 1788529841.67 total 1293.61 last 62.88) - open/close 37497 (time 1788529851.70 total 1303.63 last 49.56) - open/close 38166 (time 1788529861.70 total 1313.63 last 66.90) - open/close 38774 (time 1788529871.70 total 1323.64 last 60.80) - open/close 39410 (time 1788529881.70 total 1333.64 last 63.56) - open/close 40000 (time 1788529890.99 total 1342.93 last 63.50) - open/close 40587 (time 1788529901.00 total 1352.94 last 58.66) - open/close 41246 (time 1788529911.00 total 1362.94 last 65.90) - open/close 41910 (time 1788529921.00 total 1372.94 last 66.38) - open/close 42515 (time 1788529931.01 total 1382.95 last 60.47) - open/close 43119 (time 1788529941.02 total 1392.96 last 60.34) - open/close 43667 (time 1788529951.02 total 1402.96 last 54.78) - open/close 44188 (time 1788529961.05 total 1412.98 last 51.98) - open/close 44762 (time 1788529971.06 total 1423.00 last 57.30) - open/close 45329 (time 1788529981.09 total 1433.03 last 56.57) - open/close 46068 (time 1788529991.11 total 1443.05 last 73.75) - open/close 46715 (time 1788530001.11 total 1453.05 last 64.67) - open/close 47273 (time 1788530011.11 total 1463.05 last 55.78) - open/close 47864 (time 1788530021.12 total 1473.06 last 59.09) - open/close 48524 (time 1788530031.12 total 1483.06 last 65.98) - open/close 49172 (time 1788530041.13 total 1493.07 last 64.72) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1499.76 seconds: 33.09 ops/second - unlinked 0 (time 1788530053 ; total 0 ; last 0) - unlinked 10000 (time 1788530206 ; total 153 ; last 153) - unlinked 20000 (time 1788530371 ; total 318 ; last 165) - unlinked 30000 (time 1788530518 ; total 465 ; last 147) - unlinked 40000 (time 1788530654 ; total 601 ; last 136) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 725 seconds: 68.456551 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg225-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg225-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server stop mds service on oleg225-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server stop mds service on oleg225-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg225-server unloading modules via unload_modules_local on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing unload_modules_local oleg225-server: kmemleak disabled modules unloaded. pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2