== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:46:44 (1789184804) start mds service on oleg218-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 11 09:35 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg218-server' oleg218-server: oleg218-server.virtnet: executing load_modules_local oleg218-server: kmemleak disabled oleg218-server: Loading modules from /home/green/git/lustre-release/lustre oleg218-server: detected 4 online CPUs by sysfs oleg218-server: MODOPTS_LIBCFS= oleg218-server: Force libcfs to create 2 CPU partitions oleg218-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg218-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg218-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg218-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg218-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg218-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0000 oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 112 (time 1789184913.95 total 10.69 last 10.48) - open/close 304 (time 1789184924.94 total 21.68 last 17.47) - open/close 548 (time 1789184935.56 total 32.30 last 22.97) - open/close 780 (time 1789184945.58 total 42.32 last 23.16) - open/close 1168 (time 1789184956.50 total 53.24 last 35.53) - open/close 1383 (time 1789184966.50 total 63.24 last 21.49) - open/close 1575 (time 1789184977.48 total 74.22 last 17.48) - open/close 1813 (time 1789184988.00 total 84.74 last 22.62) - open/close 2053 (time 1789184998.54 total 95.28 last 22.77) - open/close 2268 (time 1789185008.55 total 105.29 last 21.48) - open/close 2427 (time 1789185018.56 total 115.30 last 15.88) - open/close 2709 (time 1789185029.45 total 126.19 last 25.90) - open/close 2949 (time 1789185040.01 total 136.75 last 22.73) - open/close 3222 (time 1789185050.39 total 147.13 last 26.30) - open/close 3509 (time 1789185061.40 total 158.14 last 26.05) - open/close 3750 (time 1789185071.42 total 168.16 last 24.07) - open/close 4004 (time 1789185081.48 total 178.22 last 25.23) - open/close 4252 (time 1789185091.49 total 188.23 last 24.79) - open/close 4486 (time 1789185101.49 total 198.23 last 23.39) - open/close 4602 (time 1789185111.88 total 208.62 last 11.16) - open/close 4805 (time 1789185121.89 total 218.63 last 20.29) - open/close 5044 (time 1789185131.89 total 228.63 last 23.89) - open/close 5231 (time 1789185142.87 total 239.61 last 17.03) - open/close 5426 (time 1789185152.88 total 249.62 last 19.49) - open/close 5722 (time 1789185162.88 total 259.62 last 29.59) - open/close 5974 (time 1789185172.88 total 269.62 last 25.20) - open/close 6233 (time 1789185183.43 total 280.17 last 24.55) - open/close 6479 (time 1789185193.44 total 290.18 last 24.58) - open/close 6691 (time 1789185203.45 total 300.19 last 21.18) - open/close 6932 (time 1789185213.71 total 310.45 last 23.49) - open/close 7262 (time 1789185224.21 total 320.95 last 31.43) - open/close 7455 (time 1789185234.82 total 331.56 last 18.19) - open/close 7679 (time 1789185244.82 total 341.57 last 22.39) - open/close 7904 (time 1789185254.83 total 351.57 last 22.49) - open/close 8056 (time 1789185264.84 total 361.58 last 15.19) - open/close 8311 (time 1789185275.60 total 372.34 last 23.70) - open/close 8499 (time 1789185285.77 total 382.51 last 18.49) - open/close 8686 (time 1789185296.29 total 393.03 last 17.78) - open/close 8896 (time 1789185306.91 total 403.65 last 19.77) - open/close 9007 (time 1789185317.76 total 414.50 last 10.23) - open/close 9210 (time 1789185330.78 total 427.52 last 15.60) - open/close 9457 (time 1789185343.84 total 440.58 last 18.90) - open/close 9686 (time 1789185354.34 total 451.08 last 21.82) - open/close 9743 (time 1789185364.35 total 461.09 last 5.69) - open/close 9883 (time 1789185374.83 total 471.57 last 13.36) - open/close 10000 (time 1789185379.20 total 475.94 last 26.79) - open/close 10217 (time 1789185390.09 total 486.83 last 19.91) - open/close 10401 (time 1789185400.11 total 496.85 last 18.37) - open/close 10508 (time 1789185410.52 total 507.26 last 10.28) - open/close 10669 (time 1789185420.83 total 517.57 last 15.61) - open/close 10857 (time 1789185432.72 total 529.46 last 15.82) - open/close 10941 (time 1789185443.12 total 539.86 last 8.08) - open/close 11093 (time 1789185453.13 total 549.87 last 15.19) - open/close 11260 (time 1789185463.81 total 560.55 last 15.63) - open/close 11471 (time 1789185475.41 total 572.15 last 18.19) - open/close 11641 (time 1789185486.18 total 582.92 last 15.79) - open/close 11746 (time 1789185496.91 total 593.65 last 9.78) - open/close 11821 (time 1789185507.43 total 604.17 last 7.13) - open/close 11994 (time 1789185519.39 total 616.13 last 14.47) - open/close 12232 (time 1789185529.42 total 626.16 last 23.72) - open/close 12364 (time 1789185539.42 total 636.16 last 13.19) - open/close 12563 (time 1789185549.43 total 646.17 last 19.88) - open/close 12740 (time 1789185559.46 total 656.20 last 17.65) - open/close 12815 (time 1789185569.46 total 666.20 last 7.50) - open/close 13140 (time 1789185579.47 total 676.21 last 32.46) - open/close 13343 (time 1789185590.42 total 687.16 last 18.54) - open/close 13603 (time 1789185600.43 total 697.17 last 25.99) - open/close 13759 (time 1789185610.57 total 707.31 last 15.38) - open/close 13934 (time 1789185621.10 total 717.84 last 16.62) - open/close 14140 (time 1789185631.51 total 728.25 last 19.79) - open/close 14349 (time 1789185641.61 total 738.35 last 20.69) - open/close 14468 (time 1789185653.50 total 750.24 last 10.01) - open/close 14573 (time 1789185663.51 total 760.25 last 10.49) - open/close 14794 (time 1789185674.46 total 771.20 last 20.19) - open/close 15039 (time 1789185685.26 total 782.00 last 22.67) - open/close 15239 (time 1789185695.28 total 792.02 last 19.98) - open/close 15374 (time 1789185705.31 total 802.05 last 13.46) - open/close 15435 (time 1789185715.82 total 812.56 last 5.80) - open/close 15619 (time 1789185729.68 total 826.42 last 13.28) - open/close 15773 (time 1789185740.56 total 837.30 last 14.15) - open/close 15892 (time 1789185752.04 total 848.78 last 10.37) - open/close 16062 (time 1789185762.15 total 858.89 last 16.81) - open/close 16207 (time 1789185772.17 total 868.91 last 14.47) - open/close 16364 (time 1789185783.25 total 879.99 last 14.17) - open/close 16499 (time 1789185793.26 total 890.00 last 13.49) - open/close 16631 (time 1789185803.27 total 900.01 last 13.19) - open/close 16849 (time 1789185813.93 total 910.67 last 20.44) - open/close 16961 (time 1789185823.94 total 920.68 last 11.19) - open/close 17132 (time 1789185833.97 total 930.71 last 17.04) - open/close 17332 (time 1789185844.39 total 941.13 last 19.20) - open/close 17481 (time 1789185854.94 total 951.68 last 14.12) - open/close 17738 (time 1789185865.90 total 962.64 last 23.46) - open/close 17952 (time 1789185875.90 total 972.64 last 21.40) - open/close 18152 (time 1789185885.91 total 982.65 last 19.98) - open/close 18340 (time 1789185895.91 total 992.65 last 18.79) - open/close 19024 (time 1789185905.92 total 1002.66 last 68.33) - open/close 19795 (time 1789185915.93 total 1012.67 last 77.05) - open/close 20000 (time 1789185918.78 total 1015.52 last 71.79) - open/close 20703 (time 1789185928.80 total 1025.54 last 70.22) - open/close 21452 (time 1789185938.80 total 1035.54 last 74.90) - open/close 22131 (time 1789185948.80 total 1045.54 last 67.89) - open/close 22810 (time 1789185958.80 total 1055.54 last 67.87) - open/close 23535 (time 1789185968.81 total 1065.55 last 72.48) - open/close 24206 (time 1789185978.82 total 1075.56 last 67.01) - open/close 24869 (time 1789185988.82 total 1085.56 last 66.28) - open/close 25623 (time 1789185998.82 total 1095.57 last 75.37) - open/close 26307 (time 1789186008.84 total 1105.58 last 68.27) - open/close 27025 (time 1789186018.84 total 1115.58 last 71.80) - open/close 27958 (time 1789186028.85 total 1125.59 last 93.29) - open/close 28673 (time 1789186038.85 total 1135.59 last 71.44) - open/close 29369 (time 1789186048.86 total 1145.60 last 69.54) - open/close 30000 (time 1789186057.74 total 1154.48 last 71.09) - open/close 30635 (time 1789186067.74 total 1164.48 last 63.49) - open/close 31408 (time 1789186077.76 total 1174.50 last 77.13) - open/close 32142 (time 1789186087.77 total 1184.51 last 73.35) - open/close 32898 (time 1789186097.78 total 1194.52 last 75.51) - open/close 33750 (time 1789186107.79 total 1204.53 last 85.15) - open/close 34575 (time 1789186117.79 total 1214.53 last 82.43) - open/close 35344 (time 1789186127.81 total 1224.55 last 76.80) - open/close 36172 (time 1789186137.81 total 1234.55 last 82.78) - open/close 37005 (time 1789186147.81 total 1244.55 last 83.30) - open/close 37860 (time 1789186157.82 total 1254.56 last 85.39) - open/close 38733 (time 1789186167.83 total 1264.57 last 87.28) - open/close 39449 (time 1789186177.83 total 1274.57 last 71.60) - open/close 40000 (time 1789186184.44 total 1281.18 last 83.36) - open/close 40799 (time 1789186194.45 total 1291.19 last 79.82) - open/close 41491 (time 1789186204.45 total 1301.19 last 69.19) - open/close 42272 (time 1789186214.46 total 1311.20 last 78.04) - open/close 43056 (time 1789186224.47 total 1321.21 last 78.33) - open/close 43856 (time 1789186234.47 total 1331.21 last 79.99) - open/close 44613 (time 1789186244.47 total 1341.21 last 75.65) - open/close 45358 (time 1789186254.49 total 1351.23 last 74.36) - open/close 46169 (time 1789186264.50 total 1361.24 last 81.06) - open/close 46860 (time 1789186274.52 total 1371.26 last 68.96) - open/close 47743 (time 1789186284.52 total 1381.26 last 88.28) - open/close 48589 (time 1789186294.53 total 1391.27 last 84.53) - open/close 49421 (time 1789186304.53 total 1401.27 last 83.16) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1404.09 seconds: 35.35 ops/second - unlinked 0 (time 1789186311 ; total 0 ; last 0) - unlinked 10000 (time 1789186439 ; total 128 ; last 128) - unlinked 20000 (time 1789186554 ; total 243 ; last 115) - unlinked 30000 (time 1789186666 ; total 355 ; last 112) - unlinked 40000 (time 1789186741 ; total 430 ; last 75) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 483 seconds: 102.755692 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg218-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg218-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg218-server stop mds service on oleg218-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg218-server stop mds service on oleg218-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg218-server unloading modules via unload_modules_local on: 'oleg218-server' oleg218-server: oleg218-server.virtnet: executing unload_modules_local oleg218-server: kmemleak disabled modules unloaded. pdsh@oleg218-client: oleg218-server: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-client: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-client: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-server: ssh exited with exit code 2