== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:28:06 (1788539286) start mds service on oleg336-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1923184 Sep 3 20:45 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing load_modules_local oleg336-server: kmemleak disabled oleg336-server: Loading modules from /home/green/git/lustre-release/lustre oleg336-server: detected 4 online CPUs by sysfs oleg336-server: MODOPTS_LIBCFS= oleg336-server: Force libcfs to create 2 CPU partitions oleg336-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg336-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg336-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg336-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg336-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg336-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 157 (time 1788539404.56 total 10.51 last 14.94) - open/close 292 (time 1788539415.55 total 21.51 last 12.28) - open/close 435 (time 1788539425.56 total 31.51 last 14.29) - open/close 643 (time 1788539435.78 total 41.73 last 20.35) - open/close 833 (time 1788539445.78 total 51.73 last 19.00) - open/close 1043 (time 1788539456.00 total 61.96 last 20.54) - open/close 1227 (time 1788539466.00 total 71.96 last 18.40) - open/close 1374 (time 1788539477.06 total 83.01 last 13.30) - open/close 1425 (time 1788539489.41 total 95.36 last 4.13) - open/close 1492 (time 1788539502.08 total 108.03 last 5.29) - open/close 1700 (time 1788539512.39 total 118.34 last 20.17) - open/close 1889 (time 1788539522.42 total 128.38 last 18.84) - open/close 2061 (time 1788539532.87 total 138.83 last 16.46) - open/close 2219 (time 1788539544.66 total 150.61 last 13.40) - open/close 2252 (time 1788539554.83 total 160.78 last 3.25) - open/close 2436 (time 1788539564.84 total 170.79 last 18.38) - open/close 2685 (time 1788539575.69 total 181.64 last 22.95) - open/close 2924 (time 1788539588.43 total 194.39 last 18.75) - open/close 3091 (time 1788539598.44 total 204.39 last 16.69) - open/close 3213 (time 1788539608.45 total 214.40 last 12.19) - open/close 3395 (time 1788539619.33 total 225.29 last 16.71) - open/close 3548 (time 1788539630.14 total 236.09 last 14.17) - open/close 3749 (time 1788539640.22 total 246.17 last 19.94) - open/close 3915 (time 1788539651.68 total 257.64 last 14.48) - open/close 4144 (time 1788539662.57 total 268.52 last 21.04) - open/close 4240 (time 1788539672.97 total 278.93 last 9.22) - open/close 4420 (time 1788539683.98 total 289.93 last 16.36) - open/close 4640 (time 1788539694.55 total 300.50 last 20.82) - open/close 4881 (time 1788539704.70 total 310.65 last 23.75) - open/close 5012 (time 1788539714.70 total 320.65 last 13.10) - open/close 5213 (time 1788539724.70 total 330.65 last 20.10) - open/close 5456 (time 1788539734.71 total 340.66 last 24.28) - open/close 5611 (time 1788539745.31 total 351.26 last 14.62) - open/close 5707 (time 1788539755.66 total 361.61 last 9.28) - open/close 5899 (time 1788539765.67 total 371.62 last 19.18) - open/close 6022 (time 1788539775.68 total 381.63 last 12.29) - open/close 6214 (time 1788539785.68 total 391.64 last 19.20) - open/close 6482 (time 1788539795.68 total 401.64 last 26.79) - open/close 6747 (time 1788539806.66 total 412.61 last 24.15) - open/close 6987 (time 1788539817.22 total 423.17 last 22.72) - open/close 7204 (time 1788539827.24 total 433.19 last 21.66) - open/close 7435 (time 1788539837.25 total 443.21 last 23.08) - open/close 7706 (time 1788539847.26 total 453.22 last 27.07) - open/close 7912 (time 1788539857.53 total 463.49 last 20.06) - open/close 8137 (time 1788539867.58 total 473.53 last 22.40) - open/close 8382 (time 1788539877.83 total 483.78 last 23.90) - open/close 8647 (time 1788539888.20 total 494.15 last 25.55) - open/close 8938 (time 1788539898.21 total 504.16 last 29.07) - open/close 9151 (time 1788539908.56 total 514.51 last 20.58) - open/close 9319 (time 1788539919.27 total 525.22 last 15.69) - open/close 9519 (time 1788539930.34 total 536.29 last 18.06) - open/close 9724 (time 1788539940.37 total 546.32 last 20.44) - open/close 9810 (time 1788539950.41 total 556.36 last 8.57) - open/close 10000 (time 1788539958.21 total 564.16 last 24.37) - open/close 10243 (time 1788539968.78 total 574.74 last 22.97) - open/close 10491 (time 1788539978.79 total 584.75 last 24.77) - open/close 10732 (time 1788539989.51 total 595.47 last 22.49) - open/close 10988 (time 1788539999.74 total 605.69 last 25.04) - open/close 11179 (time 1788540009.74 total 615.70 last 19.09) - open/close 11306 (time 1788540019.74 total 625.70 last 12.70) - open/close 11386 (time 1788540029.77 total 635.73 last 7.98) - open/close 11550 (time 1788540040.65 total 646.61 last 15.07) - open/close 11705 (time 1788540050.96 total 656.91 last 15.05) - open/close 11856 (time 1788540061.77 total 667.72 last 13.96) - open/close 11988 (time 1788540074.07 total 680.02 last 10.73) - open/close 12144 (time 1788540084.81 total 690.76 last 14.53) - open/close 12407 (time 1788540095.78 total 701.73 last 23.97) - open/close 12679 (time 1788540105.82 total 711.78 last 27.08) - open/close 12872 (time 1788540116.56 total 722.51 last 17.98) - open/close 12954 (time 1788540126.56 total 732.51 last 8.20) - open/close 13114 (time 1788540136.97 total 742.93 last 15.37) - open/close 13206 (time 1788540147.02 total 752.98 last 9.15) - open/close 13343 (time 1788540157.34 total 763.29 last 13.29) - open/close 13557 (time 1788540167.36 total 773.31 last 21.36) - open/close 13836 (time 1788540177.66 total 783.62 last 27.07) - open/close 13954 (time 1788540188.06 total 794.02 last 11.34) - open/close 14156 (time 1788540198.08 total 804.03 last 20.17) - open/close 14381 (time 1788540208.09 total 814.04 last 22.49) - open/close 14554 (time 1788540218.75 total 824.70 last 16.23) - open/close 14733 (time 1788540228.76 total 834.71 last 17.88) - open/close 14970 (time 1788540238.77 total 844.72 last 23.68) - open/close 15216 (time 1788540249.45 total 855.40 last 23.03) - open/close 15480 (time 1788540260.25 total 866.21 last 24.44) - open/close 15745 (time 1788540270.92 total 876.88 last 24.84) - open/close 16007 (time 1788540280.95 total 886.90 last 26.13) - open/close 16296 (time 1788540291.72 total 897.67 last 26.83) - open/close 16478 (time 1788540302.72 total 908.67 last 16.54) - open/close 16719 (time 1788540312.75 total 918.70 last 24.04) - open/close 16865 (time 1788540322.76 total 928.71 last 14.58) - open/close 17033 (time 1788540332.99 total 938.94 last 16.42) - open/close 17228 (time 1788540344.07 total 950.02 last 17.60) - open/close 17507 (time 1788540355.00 total 960.96 last 25.51) - open/close 17754 (time 1788540365.57 total 971.52 last 23.39) - open/close 17982 (time 1788540376.53 total 982.48 last 20.80) - open/close 18176 (time 1788540386.53 total 992.49 last 19.38) - open/close 18782 (time 1788540396.54 total 1002.49 last 60.58) - open/close 19523 (time 1788540406.55 total 1012.51 last 73.99) - open/close 20000 (time 1788540413.41 total 1019.37 last 69.51) - open/close 20722 (time 1788540423.42 total 1029.37 last 72.17) - open/close 21517 (time 1788540433.42 total 1039.37 last 79.49) - open/close 22337 (time 1788540443.42 total 1049.38 last 81.99) - open/close 23105 (time 1788540453.42 total 1059.38 last 76.79) - open/close 23822 (time 1788540463.43 total 1069.39 last 71.61) - open/close 24542 (time 1788540473.44 total 1079.40 last 71.95) - open/close 25208 (time 1788540483.47 total 1089.43 last 66.41) - open/close 25880 (time 1788540493.48 total 1099.43 last 67.15) - open/close 26636 (time 1788540503.48 total 1109.43 last 75.59) - open/close 27405 (time 1788540513.50 total 1119.45 last 76.79) - open/close 28178 (time 1788540523.51 total 1129.46 last 77.22) - open/close 28966 (time 1788540533.51 total 1139.46 last 78.76) - open/close 29767 (time 1788540543.51 total 1149.46 last 80.10) - open/close 30000 (time 1788540546.50 total 1152.46 last 77.86) - open/close 30790 (time 1788540556.51 total 1162.47 last 78.94) - open/close 31630 (time 1788540566.52 total 1172.48 last 83.91) - open/close 32385 (time 1788540576.53 total 1182.48 last 75.47) - open/close 33135 (time 1788540586.54 total 1192.49 last 74.93) - open/close 33875 (time 1788540596.54 total 1202.50 last 73.94) - open/close 34603 (time 1788540606.55 total 1212.50 last 72.78) - open/close 35311 (time 1788540616.56 total 1222.52 last 70.68) - open/close 36072 (time 1788540626.56 total 1232.52 last 76.10) - open/close 36728 (time 1788540636.57 total 1242.52 last 65.56) - open/close 37395 (time 1788540646.58 total 1252.54 last 66.62) - open/close 38073 (time 1788540656.59 total 1262.55 last 67.72) - open/close 38826 (time 1788540666.60 total 1272.55 last 75.26) - open/close 39625 (time 1788540676.61 total 1282.56 last 79.83) - open/close 40000 (time 1788540681.69 total 1287.65 last 73.76) - open/close 40712 (time 1788540691.70 total 1297.65 last 71.15) - open/close 41444 (time 1788540701.71 total 1307.67 last 73.11) - open/close 42113 (time 1788540711.73 total 1317.68 last 66.78) - open/close 42811 (time 1788540721.74 total 1327.69 last 69.75) - open/close 43424 (time 1788540731.75 total 1337.70 last 61.25) - open/close 44047 (time 1788540741.75 total 1347.71 last 62.25) - open/close 44684 (time 1788540751.76 total 1357.72 last 63.63) - open/close 45412 (time 1788540761.76 total 1367.72 last 72.80) - open/close 46161 (time 1788540771.78 total 1377.73 last 74.78) - open/close 46899 (time 1788540781.78 total 1387.74 last 73.78) - open/close 47688 (time 1788540791.78 total 1397.74 last 78.89) - open/close 48464 (time 1788540801.79 total 1407.74 last 77.59) - open/close 49168 (time 1788540811.79 total 1417.74 last 70.39) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1423.40 seconds: 34.87 ops/second - unlinked 0 (time 1788540822 ; total 0 ; last 0) - unlinked 10000 (time 1788540964 ; total 142 ; last 142) - unlinked 20000 (time 1788541103 ; total 281 ; last 139) - unlinked 30000 (time 1788541248 ; total 426 ; last 145) - unlinked 40000 (time 1788541382 ; total 560 ; last 134) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 689 seconds: 72.033379 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg336-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg336-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg336-server stop mds service on oleg336-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg336-server stop mds service on oleg336-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg336-server unloading modules via unload_modules_local on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing unload_modules_local oleg336-server: kmemleak disabled modules unloaded. pdsh@oleg336-client: oleg336-client: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-client: ssh exited with exit code 2