== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:14:07 (1788945247) start mds service on oleg421-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 8 22:36 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing load_modules_local oleg421-server: kmemleak disabled oleg421-server: Loading modules from /home/green/git/lustre-release/lustre oleg421-server: detected 4 online CPUs by sysfs oleg421-server: MODOPTS_LIBCFS= oleg421-server: Force libcfs to create 2 CPU partitions oleg421-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg421-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg421-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg421-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg421-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg421-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0000 oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 193 (time 1788945348.34 total 10.99 last 17.56) - open/close 472 (time 1788945359.29 total 21.94 last 25.48) - open/close 772 (time 1788945369.72 total 32.38 last 28.76) - open/close 1018 (time 1788945379.73 total 42.38 last 24.59) - open/close 1301 (time 1788945390.51 total 53.16 last 26.25) - open/close 1501 (time 1788945400.51 total 63.16 last 20.00) - open/close 1784 (time 1788945411.14 total 73.79 last 26.63) - open/close 2034 (time 1788945422.15 total 84.80 last 22.70) - open/close 2356 (time 1788945432.15 total 94.80 last 32.20) - open/close 2605 (time 1788945442.80 total 105.45 last 23.38) - open/close 2811 (time 1788945452.81 total 115.46 last 20.59) - open/close 3068 (time 1788945462.83 total 125.48 last 25.65) - open/close 3265 (time 1788945473.72 total 136.37 last 18.08) - open/close 3535 (time 1788945484.02 total 146.67 last 26.22) - open/close 3816 (time 1788945497.04 total 159.69 last 21.58) - open/close 4042 (time 1788945507.06 total 169.71 last 22.55) - open/close 4243 (time 1788945517.69 total 180.35 last 18.90) - open/close 4529 (time 1788945528.23 total 190.89 last 27.14) - open/close 4773 (time 1788945538.25 total 200.90 last 24.36) - open/close 5014 (time 1788945548.85 total 211.50 last 22.74) - open/close 5276 (time 1788945558.86 total 221.52 last 26.16) - open/close 5548 (time 1788945569.46 total 232.12 last 25.66) - open/close 5800 (time 1788945580.28 total 242.93 last 23.30) - open/close 6064 (time 1788945591.30 total 253.95 last 23.96) - open/close 6317 (time 1788945601.45 total 264.11 last 24.91) - open/close 6566 (time 1788945611.51 total 274.16 last 24.76) - open/close 6876 (time 1788945621.52 total 284.18 last 30.95) - open/close 7119 (time 1788945631.74 total 294.39 last 23.78) - open/close 7384 (time 1788945642.03 total 304.69 last 25.75) - open/close 7656 (time 1788945652.04 total 314.70 last 27.17) - open/close 7895 (time 1788945662.39 total 325.05 last 23.09) - open/close 8146 (time 1788945672.89 total 335.55 last 23.91) - open/close 8372 (time 1788945683.19 total 345.84 last 21.95) - open/close 8543 (time 1788945693.68 total 356.34 last 16.30) - open/close 8722 (time 1788945704.37 total 367.03 last 16.75) - open/close 9011 (time 1788945714.39 total 377.04 last 28.86) - open/close 9221 (time 1788945725.42 total 388.08 last 19.02) - open/close 9400 (time 1788945735.43 total 398.08 last 17.90) - open/close 9588 (time 1788945746.34 total 408.99 last 17.23) - open/close 9827 (time 1788945756.35 total 419.01 last 23.87) - open/close 10000 (time 1788945762.38 total 425.03 last 28.71) - open/close 10220 (time 1788945772.83 total 435.48 last 21.05) - open/close 10472 (time 1788945783.67 total 446.32 last 23.25) - open/close 10674 (time 1788945793.68 total 456.33 last 20.18) - open/close 10873 (time 1788945803.70 total 466.35 last 19.86) - open/close 11021 (time 1788945813.70 total 476.35 last 14.80) - open/close 11264 (time 1788945824.37 total 487.02 last 22.77) - open/close 11585 (time 1788945835.38 total 498.03 last 29.16) - open/close 11820 (time 1788945846.35 total 509.01 last 21.42) - open/close 12095 (time 1788945857.08 total 519.73 last 25.64) - open/close 12333 (time 1788945867.09 total 529.74 last 23.78) - open/close 12583 (time 1788945877.09 total 539.74 last 24.99) - open/close 12869 (time 1788945887.63 total 550.28 last 27.14) - open/close 13161 (time 1788945898.35 total 561.00 last 27.23) - open/close 13447 (time 1788945908.45 total 571.10 last 28.33) - open/close 13716 (time 1788945918.71 total 581.36 last 26.21) - open/close 13924 (time 1788945928.72 total 591.37 last 20.79) - open/close 14139 (time 1788945939.76 total 602.42 last 19.46) - open/close 14399 (time 1788945950.52 total 613.17 last 24.18) - open/close 14633 (time 1788945961.08 total 623.73 last 22.15) - open/close 14860 (time 1788945971.44 total 634.10 last 21.90) - open/close 15116 (time 1788945981.44 total 644.10 last 25.60) - open/close 15205 (time 1788945992.32 total 654.98 last 8.18) - open/close 15346 (time 1788946002.32 total 664.98 last 14.10) - open/close 15587 (time 1788946013.37 total 676.02 last 21.82) - open/close 15790 (time 1788946023.47 total 686.12 last 20.10) - open/close 16025 (time 1788946034.46 total 697.11 last 21.40) - open/close 16153 (time 1788946046.64 total 709.29 last 10.51) - open/close 16395 (time 1788946056.65 total 719.30 last 24.17) - open/close 16616 (time 1788946067.47 total 730.12 last 20.43) - open/close 16780 (time 1788946077.48 total 740.13 last 16.39) - open/close 17023 (time 1788946087.87 total 750.52 last 23.38) - open/close 17283 (time 1788946098.05 total 760.71 last 25.54) - open/close 17528 (time 1788946108.23 total 770.88 last 24.07) - open/close 17780 (time 1788946118.24 total 780.89 last 25.18) - open/close 17984 (time 1788946128.31 total 790.96 last 20.26) - open/close 18233 (time 1788946138.32 total 800.97 last 24.88) - open/close 18484 (time 1788946149.37 total 812.02 last 22.72) - open/close 18762 (time 1788946160.08 total 822.73 last 25.97) - open/close 19013 (time 1788946170.30 total 832.95 last 24.56) - open/close 19282 (time 1788946181.04 total 843.70 last 25.04) - open/close 20000 (time 1788946190.30 total 852.95 last 77.54) - open/close 20808 (time 1788946200.30 total 862.96 last 80.79) - open/close 21630 (time 1788946210.32 total 872.97 last 82.05) - open/close 22392 (time 1788946220.34 total 882.99 last 76.07) - open/close 23257 (time 1788946230.34 total 893.00 last 86.46) - open/close 24151 (time 1788946240.36 total 903.01 last 89.29) - open/close 25028 (time 1788946250.36 total 913.01 last 87.69) - open/close 25876 (time 1788946260.37 total 923.02 last 84.70) - open/close 26803 (time 1788946270.37 total 933.02 last 92.68) - open/close 27688 (time 1788946280.38 total 943.03 last 88.45) - open/close 28450 (time 1788946290.38 total 953.04 last 76.15) - open/close 29262 (time 1788946300.39 total 963.04 last 81.14) - open/close 30000 (time 1788946309.07 total 971.73 last 85.00) - open/close 30855 (time 1788946319.08 total 981.73 last 85.46) - open/close 31694 (time 1788946329.08 total 991.74 last 83.86) - open/close 32537 (time 1788946339.10 total 1001.76 last 84.12) - open/close 33330 (time 1788946349.12 total 1011.78 last 79.15) - open/close 34217 (time 1788946359.12 total 1021.78 last 88.70) - open/close 34990 (time 1788946369.13 total 1031.78 last 77.27) - open/close 35800 (time 1788946379.13 total 1041.78 last 80.99) - open/close 36588 (time 1788946389.13 total 1051.78 last 78.78) - open/close 37288 (time 1788946399.14 total 1061.79 last 69.93) - open/close 38057 (time 1788946409.15 total 1071.80 last 76.85) - open/close 38866 (time 1788946419.15 total 1081.80 last 80.86) - open/close 39704 (time 1788946429.15 total 1091.81 last 83.80) - open/close 40000 (time 1788946432.65 total 1095.30 last 84.66) - open/close 40828 (time 1788946442.65 total 1105.30 last 82.79) - open/close 41650 (time 1788946452.65 total 1115.31 last 82.18) - open/close 42552 (time 1788946462.66 total 1125.31 last 90.16) - open/close 43389 (time 1788946472.66 total 1135.31 last 83.67) - open/close 44178 (time 1788946482.66 total 1145.32 last 78.87) - open/close 44956 (time 1788946492.67 total 1155.32 last 77.76) - open/close 45787 (time 1788946502.67 total 1165.32 last 83.09) - open/close 46667 (time 1788946512.68 total 1175.33 last 87.91) - open/close 47550 (time 1788946522.69 total 1185.34 last 88.23) - open/close 48350 (time 1788946532.71 total 1195.37 last 79.81) - open/close 49246 (time 1788946542.72 total 1205.37 last 89.52) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1210.58 seconds: 41.00 ops/second - unlinked 0 (time 1788946552 ; total 0 ; last 0) - unlinked 10000 (time 1788946684 ; total 132 ; last 132) - unlinked 20000 (time 1788946813 ; total 261 ; last 129) - unlinked 30000 (time 1788946936 ; total 384 ; last 123) - unlinked 40000 (time 1788947061 ; total 509 ; last 125) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 632 seconds: 78.530060 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg421-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg421-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg421-server stop mds service on oleg421-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg421-server stop mds service on oleg421-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg421-server unloading modules via unload_modules_local on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing unload_modules_local oleg421-server: kmemleak disabled modules unloaded. pdsh@oleg421-client: oleg421-client: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-client: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 2