== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:12:52 (1790100772) start mds service on oleg322-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1940568 Sep 22 12:15 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg322-server' oleg322-server: oleg322-server.virtnet: executing load_modules_local oleg322-server: kmemleak disabled oleg322-server: Loading modules from /home/green/git/lustre-release/lustre oleg322-server: detected 4 online CPUs by sysfs oleg322-server: MODOPTS_LIBCFS= oleg322-server: Force libcfs to create 2 CPU partitions oleg322-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg322-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg322-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg322-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg322-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg322-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 185 (time 1790100882.91 total 10.01 last 18.48) - open/close 413 (time 1790100893.47 total 20.57 last 21.60) - open/close 636 (time 1790100903.77 total 30.88 last 21.63) - open/close 904 (time 1790100913.78 total 40.89 last 26.77) - open/close 1102 (time 1790100923.79 total 50.89 last 19.79) - open/close 1421 (time 1790100933.79 total 60.90 last 31.90) - open/close 1651 (time 1790100944.18 total 71.28 last 22.15) - open/close 1781 (time 1790100955.18 total 82.28 last 11.82) - open/close 1955 (time 1790100965.18 total 92.28 last 17.40) - open/close 2034 (time 1790100975.20 total 102.30 last 7.89) - open/close 2169 (time 1790100985.21 total 112.31 last 13.48) - open/close 2336 (time 1790100995.81 total 122.91 last 15.75) - open/close 2525 (time 1790101006.06 total 133.17 last 18.43) - open/close 2731 (time 1790101016.07 total 143.17 last 20.59) - open/close 2927 (time 1790101026.07 total 153.17 last 19.60) - open/close 3144 (time 1790101037.09 total 164.20 last 19.69) - open/close 3297 (time 1790101047.11 total 174.21 last 15.27) - open/close 3491 (time 1790101057.14 total 184.24 last 19.35) - open/close 3656 (time 1790101067.15 total 194.26 last 16.47) - open/close 3856 (time 1790101077.20 total 204.30 last 19.92) - open/close 4161 (time 1790101087.22 total 214.32 last 30.44) - open/close 4350 (time 1790101099.49 total 226.59 last 15.40) - open/close 4551 (time 1790101109.69 total 236.79 last 19.71) - open/close 4753 (time 1790101121.44 total 248.54 last 17.19) - open/close 5018 (time 1790101131.45 total 258.55 last 26.48) - open/close 5192 (time 1790101142.37 total 269.47 last 15.94) - open/close 5390 (time 1790101152.37 total 279.47 last 19.79) - open/close 5600 (time 1790101162.37 total 289.48 last 21.00) - open/close 5725 (time 1790101172.37 total 299.48 last 12.50) - open/close 5909 (time 1790101182.50 total 309.60 last 18.18) - open/close 6033 (time 1790101193.32 total 320.42 last 11.46) - open/close 6242 (time 1790101204.06 total 331.17 last 19.45) - open/close 6488 (time 1790101214.07 total 341.17 last 24.59) - open/close 6658 (time 1790101224.35 total 351.45 last 16.54) - open/close 6897 (time 1790101234.35 total 361.46 last 23.89) - open/close 7036 (time 1790101244.36 total 371.46 last 13.89) - open/close 7299 (time 1790101255.14 total 382.24 last 24.40) - open/close 7492 (time 1790101265.15 total 392.25 last 19.29) - open/close 7755 (time 1790101275.99 total 403.10 last 24.25) - open/close 7928 (time 1790101286.85 total 413.95 last 15.93) - open/close 8163 (time 1790101297.44 total 424.54 last 22.20) - open/close 8398 (time 1790101307.45 total 434.55 last 23.46) - open/close 8573 (time 1790101318.21 total 445.31 last 16.26) - open/close 8802 (time 1790101328.25 total 455.35 last 22.81) - open/close 9063 (time 1790101338.99 total 466.10 last 24.30) - open/close 9213 (time 1790101349.00 total 476.11 last 14.98) - open/close 9445 (time 1790101359.01 total 486.11 last 23.18) - open/close 9670 (time 1790101369.01 total 496.12 last 22.49) - open/close 9942 (time 1790101379.02 total 506.12 last 27.18) - open/close 10000 (time 1790101382.12 total 509.22 last 18.72) - open/close 10272 (time 1790101392.13 total 519.24 last 27.16) - open/close 10521 (time 1790101402.13 total 529.24 last 24.90) - open/close 10667 (time 1790101413.19 total 540.29 last 13.21) - open/close 10924 (time 1790101423.91 total 551.02 last 23.96) - open/close 11075 (time 1790101436.33 total 563.43 last 12.16) - open/close 11277 (time 1790101446.36 total 573.47 last 20.13) - open/close 11470 (time 1790101457.38 total 584.48 last 17.52) - open/close 11698 (time 1790101467.56 total 594.67 last 22.38) - open/close 12006 (time 1790101478.44 total 605.54 last 28.33) - open/close 12287 (time 1790101488.44 total 615.55 last 28.09) - open/close 12511 (time 1790101499.17 total 626.27 last 20.88) - open/close 12731 (time 1790101510.00 total 637.11 last 20.30) - open/close 12937 (time 1790101520.10 total 647.20 last 20.41) - open/close 13205 (time 1790101530.86 total 657.96 last 24.91) - open/close 13472 (time 1790101540.86 total 667.96 last 26.69) - open/close 13738 (time 1790101550.86 total 677.96 last 26.60) - open/close 13950 (time 1790101560.87 total 687.98 last 21.17) - open/close 14206 (time 1790101571.50 total 698.60 last 24.09) - open/close 14438 (time 1790101581.52 total 708.62 last 23.16) - open/close 14691 (time 1790101591.98 total 719.08 last 24.19) - open/close 14946 (time 1790101602.85 total 729.95 last 23.46) - open/close 15137 (time 1790101613.35 total 740.46 last 18.18) - open/close 15391 (time 1790101623.90 total 751.00 last 24.09) - open/close 15661 (time 1790101634.81 total 761.91 last 24.75) - open/close 15894 (time 1790101645.04 total 772.14 last 22.78) - open/close 16188 (time 1790101655.05 total 782.15 last 29.36) - open/close 16376 (time 1790101665.31 total 792.41 last 18.32) - open/close 16620 (time 1790101675.63 total 802.73 last 23.64) - open/close 16863 (time 1790101685.79 total 812.90 last 23.91) - open/close 17081 (time 1790101695.79 total 822.90 last 21.80) - open/close 17237 (time 1790101706.34 total 833.44 last 14.79) - open/close 17422 (time 1790101716.76 total 843.86 last 17.75) - open/close 17624 (time 1790101726.76 total 853.87 last 20.19) - open/close 17783 (time 1790101737.80 total 864.90 last 14.41) - open/close 17901 (time 1790101748.53 total 875.64 last 10.99) - open/close 18072 (time 1790101758.71 total 885.81 last 16.80) - open/close 18232 (time 1790101768.80 total 895.90 last 15.86) - open/close 18472 (time 1790101779.28 total 906.39 last 22.88) - open/close 18665 (time 1790101789.30 total 916.40 last 19.28) - open/close 18881 (time 1790101800.09 total 927.20 last 20.01) - open/close 19140 (time 1790101810.09 total 937.20 last 25.90) - open/close 19362 (time 1790101820.12 total 947.22 last 22.15) - open/close 19582 (time 1790101832.35 total 959.46 last 17.98) - open/close 19845 (time 1790101842.36 total 969.46 last 26.29) - open/close 20000 (time 1790101849.96 total 977.06 last 20.38) - open/close 20631 (time 1790101859.96 total 987.07 last 63.09) - open/close 21395 (time 1790101869.97 total 997.08 last 76.33) - open/close 22188 (time 1790101880.00 total 1007.10 last 79.11) - open/close 22905 (time 1790101890.00 total 1017.10 last 71.67) - open/close 23607 (time 1790101900.02 total 1027.12 last 70.05) - open/close 24261 (time 1790101910.03 total 1037.13 last 65.36) - open/close 24944 (time 1790101920.03 total 1047.13 last 68.29) - open/close 25673 (time 1790101930.04 total 1057.14 last 72.85) - open/close 26528 (time 1790101940.04 total 1067.14 last 85.46) - open/close 27301 (time 1790101950.05 total 1077.15 last 77.27) - open/close 28059 (time 1790101960.07 total 1087.18 last 75.58) - open/close 28722 (time 1790101970.08 total 1097.19 last 66.25) - open/close 29456 (time 1790101980.09 total 1107.20 last 73.32) - open/close 30000 (time 1790101987.07 total 1114.18 last 77.94) - open/close 30725 (time 1790101997.07 total 1124.18 last 72.50) - open/close 31427 (time 1790102007.07 total 1134.18 last 70.19) - open/close 32208 (time 1790102017.08 total 1144.18 last 78.10) - open/close 32934 (time 1790102027.08 total 1154.18 last 72.59) - open/close 33622 (time 1790102037.08 total 1164.18 last 68.77) - open/close 34236 (time 1790102047.10 total 1174.21 last 61.27) - open/close 34803 (time 1790102057.14 total 1184.24 last 56.49) - open/close 35430 (time 1790102067.15 total 1194.25 last 62.64) - open/close 36076 (time 1790102077.17 total 1204.27 last 64.49) - open/close 36771 (time 1790102087.19 total 1214.29 last 69.34) - open/close 37503 (time 1790102097.20 total 1224.30 last 73.13) - open/close 38178 (time 1790102107.20 total 1234.31 last 67.46) - open/close 38808 (time 1790102117.25 total 1244.35 last 62.70) - open/close 39567 (time 1790102127.26 total 1254.37 last 75.82) - open/close 40000 (time 1790102133.53 total 1260.64 last 69.06) - open/close 40687 (time 1790102143.54 total 1270.64 last 68.68) - open/close 41419 (time 1790102153.55 total 1280.65 last 73.10) - open/close 42160 (time 1790102163.55 total 1290.66 last 74.06) - open/close 42901 (time 1790102173.57 total 1300.67 last 73.99) - open/close 43617 (time 1790102183.58 total 1310.69 last 71.50) - open/close 44328 (time 1790102193.61 total 1320.71 last 70.92) - open/close 45009 (time 1790102203.62 total 1330.72 last 68.04) - open/close 45748 (time 1790102213.63 total 1340.73 last 73.82) - open/close 46512 (time 1790102223.63 total 1350.74 last 76.37) - open/close 47105 (time 1790102233.63 total 1360.74 last 59.29) - open/close 47774 (time 1790102243.64 total 1370.74 last 66.89) - open/close 48514 (time 1790102253.65 total 1380.75 last 73.89) - open/close 49213 (time 1790102263.66 total 1390.76 last 69.85) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1397.99 seconds: 35.50 ops/second - unlinked 0 (time 1790102275 ; total 0 ; last 0) - unlinked 10000 (time 1790102419 ; total 144 ; last 144) - unlinked 20000 (time 1790102558 ; total 283 ; last 139) - unlinked 30000 (time 1790102707 ; total 432 ; last 149) - unlinked 40000 (time 1790102846 ; total 571 ; last 139) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 716 seconds: 69.317039 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg322-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg322-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg322-server stop mds service on oleg322-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg322-server stop mds service on oleg322-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg322-server unloading modules via unload_modules_local on: 'oleg322-server' oleg322-server: oleg322-server.virtnet: executing unload_modules_local oleg322-server: kmemleak disabled modules unloaded. pdsh@oleg322-client: oleg322-client: ssh exited with exit code 2 pdsh@oleg322-client: oleg322-server: ssh exited with exit code 2 pdsh@oleg322-client: oleg322-client: ssh exited with exit code 2 pdsh@oleg322-client: oleg322-server: ssh exited with exit code 2