== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 06:04:02 (1788689042) start mds service on oleg118-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 5 18:57 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing load_modules_local oleg118-server: kmemleak disabled oleg118-server: Loading modules from /home/green/git/lustre-release/lustre oleg118-server: detected 4 online CPUs by sysfs oleg118-server: MODOPTS_LIBCFS= oleg118-server: Force libcfs to create 2 CPU partitions oleg118-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg118-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg118-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg118-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg118-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg118-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 140 (time 1788689155.42 total 10.01 last 13.99) - open/close 271 (time 1788689167.25 total 21.84 last 11.07) - open/close 381 (time 1788689180.18 total 34.77 last 8.51) - open/close 589 (time 1788689191.23 total 45.82 last 18.82) - open/close 826 (time 1788689201.26 total 55.85 last 23.65) - open/close 1095 (time 1788689212.23 total 66.82 last 24.51) - open/close 1304 (time 1788689222.25 total 76.84 last 20.87) - open/close 1535 (time 1788689232.39 total 86.98 last 22.78) - open/close 1808 (time 1788689242.39 total 96.98 last 27.28) - open/close 2080 (time 1788689253.39 total 107.98 last 24.73) - open/close 2284 (time 1788689264.19 total 118.78 last 18.89) - open/close 2476 (time 1788689274.70 total 129.29 last 18.27) - open/close 2592 (time 1788689285.18 total 139.77 last 11.07) - open/close 2820 (time 1788689295.62 total 150.21 last 21.85) - open/close 3101 (time 1788689306.62 total 161.21 last 25.54) - open/close 3349 (time 1788689317.66 total 172.25 last 22.47) - open/close 3563 (time 1788689327.67 total 182.26 last 21.37) - open/close 3714 (time 1788689338.05 total 192.64 last 14.55) - open/close 3942 (time 1788689348.07 total 202.66 last 22.75) - open/close 4164 (time 1788689358.28 total 212.87 last 21.74) - open/close 4409 (time 1788689368.63 total 223.22 last 23.67) - open/close 4462 (time 1788689379.09 total 233.68 last 5.07) - open/close 4628 (time 1788689391.95 total 246.54 last 12.91) - open/close 4876 (time 1788689402.52 total 257.11 last 23.46) - open/close 5129 (time 1788689413.49 total 268.08 last 23.06) - open/close 5314 (time 1788689423.50 total 278.09 last 18.50) - open/close 5531 (time 1788689433.50 total 288.09 last 21.69) - open/close 5760 (time 1788689443.84 total 298.43 last 22.15) - open/close 5959 (time 1788689453.84 total 308.43 last 19.90) - open/close 6155 (time 1788689464.71 total 319.30 last 18.03) - open/close 6320 (time 1788689475.72 total 330.31 last 14.99) - open/close 6432 (time 1788689488.07 total 342.66 last 9.07) - open/close 6652 (time 1788689498.18 total 352.77 last 21.75) - open/close 6943 (time 1788689508.19 total 362.78 last 29.08) - open/close 7168 (time 1788689518.20 total 372.79 last 22.47) - open/close 7401 (time 1788689528.20 total 382.79 last 23.30) - open/close 7636 (time 1788689538.22 total 392.81 last 23.45) - open/close 7849 (time 1788689551.17 total 405.76 last 16.45) - open/close 7924 (time 1788689562.19 total 416.78 last 6.80) - open/close 8206 (time 1788689572.29 total 426.88 last 27.91) - open/close 8424 (time 1788689582.29 total 436.88 last 21.80) - open/close 8541 (time 1788689592.30 total 446.89 last 11.69) - open/close 8752 (time 1788689602.90 total 457.49 last 19.90) - open/close 8881 (time 1788689613.08 total 467.67 last 12.68) - open/close 8996 (time 1788689623.84 total 478.43 last 10.69) - open/close 9171 (time 1788689633.85 total 488.44 last 17.48) - open/close 9242 (time 1788689643.88 total 498.47 last 7.08) - open/close 9380 (time 1788689653.89 total 508.48 last 13.79) - open/close 9493 (time 1788689664.60 total 519.19 last 10.55) - open/close 9624 (time 1788689675.34 total 529.93 last 12.20) - open/close 9755 (time 1788689686.28 total 540.87 last 11.97) - open/close 9936 (time 1788689696.31 total 550.90 last 18.06) - open/close 10000 (time 1788689698.37 total 552.96 last 30.99) - open/close 10168 (time 1788689708.84 total 563.43 last 16.04) - open/close 10436 (time 1788689721.22 total 575.81 last 21.64) - open/close 10637 (time 1788689732.29 total 586.88 last 18.17) - open/close 10768 (time 1788689742.31 total 596.90 last 13.07) - open/close 11048 (time 1788689752.32 total 606.91 last 27.97) - open/close 11354 (time 1788689762.34 total 616.93 last 30.54) - open/close 11594 (time 1788689774.93 total 629.52 last 19.06) - open/close 11772 (time 1788689785.36 total 639.95 last 17.08) - open/close 12026 (time 1788689796.05 total 650.64 last 23.76) - open/close 12233 (time 1788689807.82 total 662.41 last 17.58) - open/close 12416 (time 1788689817.82 total 672.41 last 18.29) - open/close 12725 (time 1788689827.84 total 682.43 last 30.86) - open/close 12894 (time 1788689839.14 total 693.73 last 14.95) - open/close 13013 (time 1788689852.11 total 706.70 last 9.18) - open/close 13250 (time 1788689862.72 total 717.31 last 22.33) - open/close 13454 (time 1788689873.49 total 728.08 last 18.95) - open/close 13635 (time 1788689884.42 total 739.01 last 16.55) - open/close 13759 (time 1788689894.46 total 749.05 last 12.35) - open/close 13931 (time 1788689905.02 total 759.61 last 16.30) - open/close 14106 (time 1788689915.78 total 770.37 last 16.26) - open/close 14327 (time 1788689927.97 total 782.56 last 18.13) - open/close 14520 (time 1788689939.66 total 794.24 last 16.51) - open/close 14606 (time 1788689949.66 total 804.25 last 8.59) - open/close 14832 (time 1788689959.95 total 814.54 last 21.98) - open/close 15078 (time 1788689970.37 total 824.96 last 23.59) - open/close 15221 (time 1788689980.44 total 835.03 last 14.21) - open/close 15360 (time 1788689991.36 total 845.95 last 12.73) - open/close 15600 (time 1788690002.38 total 856.97 last 21.78) - open/close 15660 (time 1788690013.20 total 867.79 last 5.55) - open/close 15883 (time 1788690025.54 total 880.13 last 18.07) - open/close 15979 (time 1788690035.55 total 890.14 last 9.59) - open/close 16178 (time 1788690045.56 total 900.15 last 19.89) - open/close 16390 (time 1788690056.00 total 910.59 last 20.31) - open/close 16570 (time 1788690066.59 total 921.18 last 16.99) - open/close 16723 (time 1788690076.60 total 931.19 last 15.28) - open/close 17053 (time 1788690086.61 total 941.20 last 32.96) - open/close 17222 (time 1788690096.79 total 951.38 last 16.60) - open/close 17438 (time 1788690106.80 total 961.39 last 21.58) - open/close 17562 (time 1788690116.95 total 971.54 last 12.22) - open/close 17687 (time 1788690127.69 total 982.28 last 11.63) - open/close 17811 (time 1788690137.70 total 992.29 last 12.38) - open/close 18080 (time 1788690147.71 total 1002.30 last 26.88) - open/close 18240 (time 1788690157.77 total 1012.36 last 15.91) - open/close 18417 (time 1788690167.77 total 1022.36 last 17.69) - open/close 19184 (time 1788690177.78 total 1032.37 last 76.67) - open/close 20000 (time 1788690186.65 total 1041.24 last 91.95) - open/close 20840 (time 1788690196.66 total 1051.25 last 83.95) - open/close 21801 (time 1788690206.66 total 1061.25 last 96.06) - open/close 22727 (time 1788690216.67 total 1071.26 last 92.54) - open/close 23734 (time 1788690226.67 total 1081.26 last 100.65) - open/close 24611 (time 1788690236.68 total 1091.27 last 87.67) - open/close 25626 (time 1788690246.69 total 1101.28 last 101.40) - open/close 26542 (time 1788690256.69 total 1111.28 last 91.56) - open/close 27365 (time 1788690266.69 total 1121.28 last 82.27) - open/close 28259 (time 1788690276.71 total 1131.30 last 89.28) - open/close 29077 (time 1788690286.71 total 1141.30 last 81.76) - open/close 29979 (time 1788690296.71 total 1151.30 last 90.19) - open/close 30826 (time 1788690306.71 total 1161.30 last 84.69) - open/close 31585 (time 1788690316.72 total 1171.31 last 75.83) - open/close 32403 (time 1788690326.73 total 1181.32 last 81.79) - open/close 33286 (time 1788690336.73 total 1191.32 last 88.25) - open/close 34201 (time 1788690346.75 total 1201.34 last 91.30) - open/close 34929 (time 1788690356.76 total 1211.35 last 72.79) - open/close 35828 (time 1788690366.76 total 1221.35 last 89.87) - open/close 36765 (time 1788690376.77 total 1231.36 last 93.60) - open/close 37693 (time 1788690386.78 total 1241.37 last 92.72) - open/close 38517 (time 1788690396.78 total 1251.37 last 82.35) - open/close 39230 (time 1788690406.79 total 1261.38 last 71.27) - open/close 39908 (time 1788690416.79 total 1271.38 last 67.75) - open/close 40614 (time 1788690426.81 total 1281.40 last 70.50) - open/close 41500 (time 1788690436.81 total 1291.40 last 88.58) - open/close 42396 (time 1788690446.81 total 1301.40 last 89.58) - open/close 43367 (time 1788690456.82 total 1311.41 last 97.06) - open/close 44450 (time 1788690466.83 total 1321.42 last 108.21) - open/close 45385 (time 1788690476.83 total 1331.42 last 93.49) - open/close 46276 (time 1788690486.83 total 1341.42 last 89.08) - open/close 47087 (time 1788690496.83 total 1351.42 last 81.07) - open/close 48069 (time 1788690506.84 total 1361.43 last 98.12) - open/close 48990 (time 1788690516.84 total 1371.43 last 92.08) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1378.94 seconds: 35.99 ops/second - unlinked 0 (time 1788690529 ; total 0 ; last 0) - unlinked 10000 (time 1788690641 ; total 112 ; last 112) - unlinked 20000 (time 1788690743 ; total 214 ; last 102) - unlinked 30000 (time 1788690812 ; total 283 ; last 69) - unlinked 40000 (time 1788690873 ; total 344 ; last 61) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 406 seconds: 122.243843 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg118-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg118-server unloading modules via unload_modules_local on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing unload_modules_local oleg118-server: kmemleak disabled modules unloaded. pdsh@oleg118-client: oleg118-client: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-server: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-server: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-client: ssh exited with exit code 2