== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:47:32 (1788749252) start mds service on oleg457-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 6 20:25 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing load_modules_local oleg457-server: kmemleak disabled oleg457-server: Loading modules from /home/green/git/lustre-release/lustre oleg457-server: detected 4 online CPUs by sysfs oleg457-server: MODOPTS_LIBCFS= oleg457-server: Force libcfs to create 2 CPU partitions oleg457-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg457-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg457-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg457-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg457-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg457-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 195 (time 1788749344.60 total 10.20 last 19.13) - open/close 387 (time 1788749354.77 total 20.36 last 18.88) - open/close 586 (time 1788749364.77 total 30.37 last 19.89) - open/close 836 (time 1788749374.77 total 40.37 last 24.99) - open/close 1024 (time 1788749385.74 total 51.34 last 17.14) - open/close 1258 (time 1788749395.74 total 61.34 last 23.39) - open/close 1550 (time 1788749406.67 total 72.26 last 26.73) - open/close 1802 (time 1788749416.67 total 82.27 last 25.19) - open/close 1992 (time 1788749426.70 total 92.30 last 18.95) - open/close 2142 (time 1788749436.88 total 102.48 last 14.73) - open/close 2268 (time 1788749447.82 total 113.42 last 11.52) - open/close 2418 (time 1788749457.93 total 123.53 last 14.83) - open/close 2615 (time 1788749468.69 total 134.28 last 18.32) - open/close 2774 (time 1788749478.81 total 144.41 last 15.70) - open/close 2938 (time 1788749488.98 total 154.58 last 16.13) - open/close 3146 (time 1788749499.93 total 165.52 last 19.01) - open/close 3286 (time 1788749510.73 total 176.33 last 12.96) - open/close 3465 (time 1788749520.74 total 186.34 last 17.88) - open/close 3552 (time 1788749531.09 total 196.69 last 8.40) - open/close 3705 (time 1788749541.11 total 206.71 last 15.27) - open/close 3829 (time 1788749551.20 total 216.80 last 12.29) - open/close 3952 (time 1788749561.24 total 226.84 last 12.25) - open/close 4077 (time 1788749571.45 total 237.04 last 12.25) - open/close 4204 (time 1788749582.18 total 247.78 last 11.83) - open/close 4447 (time 1788749592.19 total 257.79 last 24.28) - open/close 4566 (time 1788749602.23 total 267.83 last 11.85) - open/close 4723 (time 1788749612.24 total 277.84 last 15.69) - open/close 4858 (time 1788749622.92 total 288.51 last 12.65) - open/close 5023 (time 1788749633.52 total 299.12 last 15.56) - open/close 5254 (time 1788749643.52 total 309.12 last 23.09) - open/close 5491 (time 1788749653.54 total 319.13 last 23.67) - open/close 5772 (time 1788749664.02 total 329.62 last 26.80) - open/close 6006 (time 1788749674.05 total 339.64 last 23.34) - open/close 6260 (time 1788749684.25 total 349.85 last 24.89) - open/close 6526 (time 1788749695.08 total 360.68 last 24.56) - open/close 6709 (time 1788749705.10 total 370.70 last 18.26) - open/close 6959 (time 1788749715.11 total 380.71 last 24.97) - open/close 7242 (time 1788749725.60 total 391.20 last 26.99) - open/close 7494 (time 1788749736.41 total 402.01 last 23.31) - open/close 7769 (time 1788749747.09 total 412.69 last 25.76) - open/close 8023 (time 1788749757.09 total 422.69 last 25.39) - open/close 8270 (time 1788749767.09 total 432.69 last 24.69) - open/close 8451 (time 1788749779.48 total 445.08 last 14.61) - open/close 8589 (time 1788749789.67 total 455.26 last 13.55) - open/close 8797 (time 1788749799.74 total 465.34 last 20.65) - open/close 9038 (time 1788749810.00 total 475.59 last 23.49) - open/close 9303 (time 1788749820.14 total 485.73 last 26.13) - open/close 9567 (time 1788749830.98 total 496.58 last 24.34) - open/close 9734 (time 1788749841.00 total 506.60 last 16.67) - open/close 10000 (time 1788749850.61 total 516.21 last 27.68) - open/close 10231 (time 1788749860.64 total 526.24 last 23.03) - open/close 10452 (time 1788749871.26 total 536.86 last 20.82) - open/close 10600 (time 1788749881.27 total 546.87 last 14.78) - open/close 10856 (time 1788749891.28 total 556.88 last 25.58) - open/close 11064 (time 1788749902.04 total 567.64 last 19.34) - open/close 11257 (time 1788749912.55 total 578.15 last 18.36) - open/close 11466 (time 1788749922.60 total 588.20 last 20.81) - open/close 11748 (time 1788749932.64 total 598.24 last 28.09) - open/close 11998 (time 1788749943.08 total 608.68 last 23.95) - open/close 12161 (time 1788749954.26 total 619.86 last 14.57) - open/close 12440 (time 1788749964.28 total 629.87 last 27.86) - open/close 12702 (time 1788749974.28 total 639.88 last 26.19) - open/close 12831 (time 1788749984.30 total 649.89 last 12.88) - open/close 13023 (time 1788749994.40 total 660.00 last 19.00) - open/close 13171 (time 1788750004.98 total 670.58 last 13.98) - open/close 13481 (time 1788750015.00 total 680.60 last 30.95) - open/close 13644 (time 1788750025.62 total 691.22 last 15.35) - open/close 13882 (time 1788750035.63 total 701.23 last 23.78) - open/close 14049 (time 1788750046.18 total 711.77 last 15.84) - open/close 14246 (time 1788750057.17 total 722.77 last 17.91) - open/close 14354 (time 1788750067.36 total 732.96 last 10.60) - open/close 14602 (time 1788750077.90 total 743.50 last 23.54) - open/close 14849 (time 1788750087.93 total 753.53 last 24.63) - open/close 14943 (time 1788750097.93 total 763.53 last 9.40) - open/close 15167 (time 1788750111.00 total 776.60 last 17.14) - open/close 15258 (time 1788750121.01 total 786.61 last 9.09) - open/close 15413 (time 1788750131.03 total 796.63 last 15.48) - open/close 15586 (time 1788750142.49 total 808.09 last 15.09) - open/close 15725 (time 1788750153.51 total 819.11 last 12.61) - open/close 15912 (time 1788750163.51 total 829.11 last 18.70) - open/close 16153 (time 1788750173.72 total 839.32 last 23.62) - open/close 16434 (time 1788750183.91 total 849.51 last 27.57) - open/close 16682 (time 1788750194.38 total 859.98 last 23.68) - open/close 16935 (time 1788750204.47 total 870.07 last 25.07) - open/close 17145 (time 1788750214.48 total 880.08 last 20.98) - open/close 17394 (time 1788750226.33 total 891.93 last 21.02) - open/close 18000 (time 1788750236.33 total 901.93 last 60.60) - open/close 18860 (time 1788750246.34 total 911.94 last 85.89) - open/close 19692 (time 1788750256.34 total 921.94 last 83.18) - open/close 20000 (time 1788750259.98 total 925.58 last 84.75) - open/close 20767 (time 1788750269.98 total 935.58 last 76.68) - open/close 21495 (time 1788750279.99 total 945.58 last 72.76) - open/close 22320 (time 1788750290.02 total 955.61 last 82.27) - open/close 23123 (time 1788750300.02 total 965.62 last 80.25) - open/close 23939 (time 1788750310.04 total 975.63 last 81.48) - open/close 24794 (time 1788750320.04 total 985.64 last 85.46) - open/close 25642 (time 1788750330.06 total 995.66 last 84.59) - open/close 26395 (time 1788750340.07 total 1005.67 last 75.26) - open/close 27217 (time 1788750350.08 total 1015.68 last 82.12) - open/close 28013 (time 1788750360.09 total 1025.68 last 79.55) - open/close 28822 (time 1788750370.09 total 1035.69 last 80.84) - open/close 29636 (time 1788750380.11 total 1045.70 last 81.29) - open/close 30000 (time 1788750384.45 total 1050.05 last 83.78) - open/close 30821 (time 1788750394.46 total 1060.06 last 82.05) - open/close 31625 (time 1788750404.46 total 1070.06 last 80.35) - open/close 32439 (time 1788750414.47 total 1080.07 last 81.35) - open/close 33276 (time 1788750424.47 total 1090.07 last 83.67) - open/close 34101 (time 1788750434.48 total 1100.08 last 82.42) - open/close 34915 (time 1788750444.49 total 1110.08 last 81.37) - open/close 35791 (time 1788750454.50 total 1120.10 last 87.49) - open/close 36698 (time 1788750464.51 total 1130.10 last 90.63) - open/close 37540 (time 1788750474.52 total 1140.12 last 84.09) - open/close 38405 (time 1788750484.53 total 1150.13 last 86.43) - open/close 39232 (time 1788750494.53 total 1160.13 last 82.65) - open/close 40000 (time 1788750503.50 total 1169.10 last 85.64) - open/close 40929 (time 1788750513.51 total 1179.11 last 92.82) - open/close 41860 (time 1788750523.51 total 1189.11 last 93.08) - open/close 42779 (time 1788750533.53 total 1199.13 last 91.72) - open/close 43632 (time 1788750543.55 total 1209.14 last 85.18) - open/close 44537 (time 1788750553.55 total 1219.15 last 90.44) - open/close 45410 (time 1788750563.56 total 1229.15 last 87.26) - open/close 46267 (time 1788750573.56 total 1239.16 last 85.69) - open/close 47083 (time 1788750583.57 total 1249.16 last 81.52) - open/close 48026 (time 1788750593.58 total 1259.18 last 94.20) - open/close 48896 (time 1788750603.58 total 1269.18 last 86.95) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1278.09 seconds: 38.83 ops/second - unlinked 0 (time 1788750616 ; total 0 ; last 0) - unlinked 10000 (time 1788750739 ; total 123 ; last 123) - unlinked 20000 (time 1788750869 ; total 253 ; last 130) - unlinked 30000 (time 1788750990 ; total 374 ; last 121) - unlinked 40000 (time 1788751117 ; total 501 ; last 127) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 612 seconds: 81.096405 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg457-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg457-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server stop mds service on oleg457-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg457-server stop mds service on oleg457-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg457-server unloading modules via unload_modules_local on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing unload_modules_local oleg457-server: kmemleak disabled modules unloaded. pdsh@oleg457-client: oleg457-client: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-client: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2