== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:25:40 (1789431940) start mds service on oleg221-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 14 11:26 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing load_modules_local oleg221-server: kmemleak disabled oleg221-server: Loading modules from /home/green/git/lustre-release/lustre oleg221-server: detected 4 online CPUs by sysfs oleg221-server: MODOPTS_LIBCFS= oleg221-server: Force libcfs to create 2 CPU partitions oleg221-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg221-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg221-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg221-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg221-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg221-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-OST0000 oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg221-client.virtnet: -o user_xattr,flock 192.168.202.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 216 (time 1789432036.44 total 10.30 last 20.98) - open/close 405 (time 1789432046.45 total 20.30 last 18.90) - open/close 644 (time 1789432056.47 total 30.32 last 23.85) - open/close 864 (time 1789432066.47 total 40.32 last 21.99) - open/close 1006 (time 1789432076.65 total 50.50 last 13.95) - open/close 1182 (time 1789432087.40 total 61.25 last 16.37) - open/close 1453 (time 1789432098.14 total 71.99 last 25.24) - open/close 1762 (time 1789432108.96 total 82.82 last 28.54) - open/close 1946 (time 1789432119.65 total 93.51 last 17.21) - open/close 2204 (time 1789432130.28 total 104.13 last 24.29) - open/close 2447 (time 1789432140.83 total 114.68 last 23.02) - open/close 2709 (time 1789432150.83 total 124.68 last 26.20) - open/close 2866 (time 1789432160.85 total 134.70 last 15.68) - open/close 3128 (time 1789432170.85 total 144.70 last 26.19) - open/close 3447 (time 1789432181.59 total 155.45 last 29.69) - open/close 3713 (time 1789432192.02 total 165.87 last 25.51) - open/close 3995 (time 1789432202.03 total 175.88 last 28.17) - open/close 4201 (time 1789432212.90 total 186.75 last 18.95) - open/close 4366 (time 1789432222.94 total 196.79 last 16.43) - open/close 4571 (time 1789432233.96 total 207.81 last 18.60) - open/close 4740 (time 1789432244.38 total 218.24 last 16.21) - open/close 4929 (time 1789432254.39 total 228.24 last 18.89) - open/close 5118 (time 1789432267.07 total 240.92 last 14.91) - open/close 5355 (time 1789432277.09 total 250.94 last 23.65) - open/close 5568 (time 1789432287.10 total 260.95 last 21.27) - open/close 5783 (time 1789432297.11 total 270.97 last 21.48) - open/close 6017 (time 1789432307.11 total 280.97 last 23.40) - open/close 6238 (time 1789432317.93 total 291.78 last 20.44) - open/close 6475 (time 1789432328.10 total 301.96 last 23.29) - open/close 6731 (time 1789432338.41 total 312.26 last 24.84) - open/close 7008 (time 1789432349.22 total 323.08 last 25.61) - open/close 7165 (time 1789432359.73 total 333.58 last 14.95) - open/close 7431 (time 1789432370.02 total 343.87 last 25.85) - open/close 7535 (time 1789432380.45 total 354.30 last 9.97) - open/close 7712 (time 1789432390.47 total 364.32 last 17.67) - open/close 7891 (time 1789432401.26 total 375.12 last 16.58) - open/close 8007 (time 1789432411.37 total 385.22 last 11.48) - open/close 8226 (time 1789432421.37 total 395.23 last 21.89) - open/close 8302 (time 1789432431.39 total 405.24 last 7.59) - open/close 8457 (time 1789432441.70 total 415.56 last 15.03) - open/close 8560 (time 1789432452.23 total 426.08 last 9.78) - open/close 8831 (time 1789432462.24 total 436.09 last 27.08) - open/close 8999 (time 1789432472.75 total 446.60 last 15.98) - open/close 9294 (time 1789432483.55 total 457.40 last 27.32) - open/close 9556 (time 1789432493.55 total 467.41 last 26.18) - open/close 9779 (time 1789432504.03 total 477.89 last 21.28) - open/close 9928 (time 1789432516.44 total 490.29 last 12.01) - open/close 10000 (time 1789432518.63 total 492.48 last 32.88) - open/close 10201 (time 1789432529.51 total 503.37 last 18.47) - open/close 10451 (time 1789432540.45 total 514.30 last 22.86) - open/close 10744 (time 1789432550.48 total 524.33 last 29.22) - open/close 11078 (time 1789432560.73 total 534.58 last 32.58) - open/close 11362 (time 1789432570.74 total 544.59 last 28.37) - open/close 11633 (time 1789432583.33 total 557.18 last 21.53) - open/close 11878 (time 1789432594.28 total 568.14 last 22.36) - open/close 12174 (time 1789432604.64 total 578.49 last 28.59) - open/close 12407 (time 1789432614.90 total 588.75 last 22.71) - open/close 12683 (time 1789432624.90 total 598.76 last 27.58) - open/close 12921 (time 1789432634.92 total 608.77 last 23.77) - open/close 13151 (time 1789432644.92 total 618.77 last 22.99) - open/close 13366 (time 1789432655.45 total 629.30 last 20.42) - open/close 13620 (time 1789432665.75 total 639.61 last 24.65) - open/close 13856 (time 1789432676.33 total 650.19 last 22.31) - open/close 14131 (time 1789432688.95 total 662.81 last 21.79) - open/close 14346 (time 1789432698.96 total 672.81 last 21.50) - open/close 14581 (time 1789432708.96 total 682.81 last 23.49) - open/close 14828 (time 1789432719.13 total 692.99 last 24.28) - open/close 15075 (time 1789432729.14 total 702.99 last 24.68) - open/close 15313 (time 1789432739.44 total 713.30 last 23.10) - open/close 15605 (time 1789432749.46 total 723.32 last 29.14) - open/close 15854 (time 1789432759.48 total 733.33 last 24.86) - open/close 16044 (time 1789432770.34 total 744.19 last 17.50) - open/close 16311 (time 1789432781.02 total 754.88 last 24.98) - open/close 16536 (time 1789432791.04 total 764.89 last 22.46) - open/close 16746 (time 1789432801.05 total 774.90 last 20.99) - open/close 16985 (time 1789432811.17 total 785.02 last 23.61) - open/close 17185 (time 1789432821.86 total 795.72 last 18.70) - open/close 17467 (time 1789432832.55 total 806.41 last 26.38) - open/close 17667 (time 1789432843.50 total 817.35 last 18.27) - open/close 17900 (time 1789432854.25 total 828.10 last 21.69) - open/close 18077 (time 1789432865.24 total 839.09 last 16.10) - open/close 18313 (time 1789432875.57 total 849.43 last 22.83) - open/close 18589 (time 1789432885.74 total 859.59 last 27.15) - open/close 18829 (time 1789432895.84 total 869.69 last 23.76) - open/close 19080 (time 1789432906.14 total 879.99 last 24.37) - open/close 19307 (time 1789432916.59 total 890.44 last 21.72) - open/close 19551 (time 1789432926.72 total 900.58 last 24.07) - open/close 19810 (time 1789432937.16 total 911.01 last 24.82) - open/close 20000 (time 1789432944.94 total 918.79 last 24.41) - open/close 20693 (time 1789432954.94 total 928.79 last 69.30) - open/close 21601 (time 1789432964.94 total 938.80 last 90.76) - open/close 22491 (time 1789432974.95 total 948.81 last 88.93) - open/close 23305 (time 1789432984.96 total 958.81 last 81.36) - open/close 24085 (time 1789432994.96 total 968.81 last 77.97) - open/close 24880 (time 1789433004.97 total 978.82 last 79.44) - open/close 25791 (time 1789433014.97 total 988.83 last 91.05) - open/close 26610 (time 1789433024.98 total 998.83 last 81.86) - open/close 27474 (time 1789433034.98 total 1008.83 last 86.40) - open/close 28309 (time 1789433044.98 total 1018.83 last 83.48) - open/close 29096 (time 1789433054.99 total 1028.84 last 78.64) - open/close 29881 (time 1789433064.99 total 1038.84 last 78.50) - open/close 30827 (time 1789433074.99 total 1048.84 last 94.58) - open/close 31696 (time 1789433085.01 total 1058.87 last 86.72) - open/close 32437 (time 1789433095.01 total 1068.87 last 74.09) - open/close 33275 (time 1789433105.01 total 1078.87 last 83.79) - open/close 34234 (time 1789433115.03 total 1088.88 last 95.80) - open/close 35005 (time 1789433125.04 total 1098.89 last 77.02) - open/close 35864 (time 1789433135.04 total 1108.90 last 85.83) - open/close 36683 (time 1789433145.06 total 1118.91 last 81.80) - open/close 37457 (time 1789433155.07 total 1128.92 last 77.30) - open/close 38286 (time 1789433165.08 total 1138.93 last 82.82) - open/close 39140 (time 1789433175.08 total 1148.93 last 85.38) - open/close 39936 (time 1789433185.10 total 1158.95 last 79.43) - open/close 40677 (time 1789433195.11 total 1168.97 last 74.00) - open/close 41499 (time 1789433205.14 total 1178.99 last 82.01) - open/close 42298 (time 1789433215.15 total 1189.00 last 79.81) - open/close 42811 (time 1789433225.16 total 1199.01 last 51.25) - open/close 43368 (time 1789433235.17 total 1209.03 last 55.61) - open/close 44046 (time 1789433245.19 total 1219.04 last 67.69) - open/close 44787 (time 1789433255.19 total 1229.04 last 74.09) - open/close 45534 (time 1789433265.20 total 1239.05 last 74.66) - open/close 46385 (time 1789433275.20 total 1249.06 last 85.05) - open/close 47299 (time 1789433285.21 total 1259.07 last 91.29) - open/close 48063 (time 1789433295.22 total 1269.07 last 76.34) - open/close 48794 (time 1789433305.24 total 1279.09 last 72.96) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1288.33 seconds: 38.52 ops/second - unlinked 0 (time 1789433318 ; total 0 ; last 0) - unlinked 10000 (time 1789433440 ; total 122 ; last 122) - unlinked 20000 (time 1789433564 ; total 246 ; last 124) - unlinked 30000 (time 1789433689 ; total 371 ; last 125) - unlinked 40000 (time 1789433821 ; total 503 ; last 132) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 621 seconds: 79.921097 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg221-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg221-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg221-server stop mds service on oleg221-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg221-server stop mds service on oleg221-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg221-server unloading modules via unload_modules_local on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing unload_modules_local oleg221-server: kmemleak disabled modules unloaded. pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-client: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-client: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2