== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:25:27 (1787887527) start mds service on oleg343-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754920 Aug 26 15:28 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg343-server' oleg343-server: oleg343-server.virtnet: executing load_modules_local oleg343-server: kmemleak disabled oleg343-server: Loading modules from /home/green/git/lustre-release/lustre oleg343-server: detected 4 online CPUs by sysfs oleg343-server: MODOPTS_LIBCFS= oleg343-server: Force libcfs to create 2 CPU partitions oleg343-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg343-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg343-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg343-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg343-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg343-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-OST0000 oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 71 (time 1787887630.36 total 10.68 last 6.65) - open/close 329 (time 1787887640.37 total 20.69 last 25.77) - open/close 580 (time 1787887650.89 total 31.21 last 23.84) - open/close 817 (time 1787887661.52 total 41.84 last 22.30) - open/close 1106 (time 1787887671.53 total 51.85 last 28.88) - open/close 1344 (time 1787887682.01 total 62.33 last 22.71) - open/close 1604 (time 1787887692.55 total 72.87 last 24.66) - open/close 1752 (time 1787887705.37 total 85.68 last 11.55) - open/close 1958 (time 1787887716.21 total 96.53 last 19.00) - open/close 2138 (time 1787887726.81 total 107.13 last 16.98) - open/close 2301 (time 1787887739.35 total 119.67 last 13.00) - open/close 2556 (time 1787887749.36 total 129.68 last 25.48) - open/close 2823 (time 1787887759.96 total 140.28 last 25.18) - open/close 3117 (time 1787887770.08 total 150.39 last 29.07) - open/close 3392 (time 1787887780.09 total 160.41 last 27.46) - open/close 3669 (time 1787887790.09 total 170.41 last 27.70) - open/close 3938 (time 1787887800.10 total 180.41 last 26.89) - open/close 4160 (time 1787887810.89 total 191.21 last 20.57) - open/close 4424 (time 1787887821.64 total 201.96 last 24.55) - open/close 4641 (time 1787887831.71 total 212.03 last 21.56) - open/close 4863 (time 1787887841.72 total 222.04 last 22.19) - open/close 5113 (time 1787887852.69 total 233.01 last 22.77) - open/close 5238 (time 1787887863.45 total 243.77 last 11.62) - open/close 5487 (time 1787887874.15 total 254.46 last 23.29) - open/close 5746 (time 1787887884.15 total 264.47 last 25.88) - open/close 5981 (time 1787887894.87 total 275.19 last 21.94) - open/close 6228 (time 1787887905.82 total 286.13 last 22.56) - open/close 6463 (time 1787887915.83 total 296.15 last 23.47) - open/close 6683 (time 1787887926.67 total 306.99 last 20.29) - open/close 6927 (time 1787887936.67 total 316.99 last 24.40) - open/close 7176 (time 1787887946.97 total 327.29 last 24.18) - open/close 7427 (time 1787887956.98 total 337.30 last 25.08) - open/close 7687 (time 1787887967.24 total 347.56 last 25.34) - open/close 7884 (time 1787887977.94 total 358.26 last 18.41) - open/close 8137 (time 1787887988.44 total 368.76 last 24.10) - open/close 8380 (time 1787887998.60 total 378.92 last 23.92) - open/close 8610 (time 1787888008.98 total 389.29 last 22.17) - open/close 8836 (time 1787888019.00 total 399.32 last 22.54) - open/close 9076 (time 1787888029.52 total 409.84 last 22.82) - open/close 9304 (time 1787888039.53 total 419.85 last 22.77) - open/close 9515 (time 1787888050.15 total 430.47 last 19.87) - open/close 9673 (time 1787888060.45 total 440.77 last 15.34) - open/close 9869 (time 1787888070.47 total 450.78 last 19.58) - open/close 10000 (time 1787888077.10 total 457.41 last 19.76) - open/close 10260 (time 1787888087.11 total 467.43 last 25.95) - open/close 10519 (time 1787888097.37 total 477.68 last 25.27) - open/close 10781 (time 1787888107.87 total 488.18 last 24.95) - open/close 11068 (time 1787888118.62 total 498.94 last 26.68) - open/close 11329 (time 1787888129.37 total 509.69 last 24.29) - open/close 11569 (time 1787888142.22 total 522.54 last 18.67) - open/close 11812 (time 1787888152.23 total 532.54 last 24.29) - open/close 12075 (time 1787888162.39 total 542.71 last 25.88) - open/close 12335 (time 1787888173.06 total 553.38 last 24.35) - open/close 12548 (time 1787888183.08 total 563.39 last 21.27) - open/close 12661 (time 1787888193.81 total 574.13 last 10.53) - open/close 12881 (time 1787888206.37 total 586.69 last 17.51) - open/close 13198 (time 1787888219.05 total 599.37 last 25.00) - open/close 13411 (time 1787888229.14 total 609.46 last 21.11) - open/close 13653 (time 1787888240.03 total 620.35 last 22.22) - open/close 13875 (time 1787888250.06 total 630.37 last 22.15) - open/close 14038 (time 1787888260.84 total 641.15 last 15.12) - open/close 14320 (time 1787888271.77 total 652.09 last 25.78) - open/close 14567 (time 1787888282.15 total 662.47 last 23.80) - open/close 14833 (time 1787888292.17 total 672.49 last 26.56) - open/close 14976 (time 1787888302.19 total 682.50 last 14.28) - open/close 15155 (time 1787888313.82 total 694.14 last 15.38) - open/close 15329 (time 1787888323.85 total 704.17 last 17.35) - open/close 15526 (time 1787888335.76 total 716.08 last 16.55) - open/close 15681 (time 1787888345.78 total 726.10 last 15.47) - open/close 15863 (time 1787888355.78 total 736.10 last 18.19) - open/close 16075 (time 1787888365.79 total 746.11 last 21.18) - open/close 16270 (time 1787888376.99 total 757.31 last 17.41) - open/close 16469 (time 1787888387.11 total 767.43 last 19.67) - open/close 16725 (time 1787888397.12 total 777.44 last 25.56) - open/close 16980 (time 1787888408.02 total 788.34 last 23.40) - open/close 17185 (time 1787888418.27 total 798.58 last 20.01) - open/close 17275 (time 1787888428.56 total 808.88 last 8.74) - open/close 17440 (time 1787888439.58 total 819.90 last 14.98) - open/close 17684 (time 1787888449.59 total 829.91 last 24.37) - open/close 17830 (time 1787888459.73 total 840.05 last 14.40) - open/close 18044 (time 1787888469.74 total 850.06 last 21.38) - open/close 18310 (time 1787888480.66 total 860.98 last 24.35) - open/close 18577 (time 1787888490.67 total 870.98 last 26.69) - open/close 18774 (time 1787888500.67 total 880.99 last 19.69) - open/close 19044 (time 1787888510.68 total 891.00 last 26.96) - open/close 19275 (time 1787888522.07 total 902.39 last 20.29) - open/close 19600 (time 1787888533.02 total 913.34 last 29.67) - open/close 19880 (time 1787888546.06 total 926.38 last 21.47) - open/close 20000 (time 1787888550.15 total 930.46 last 29.39) - open/close 20212 (time 1787888560.34 total 940.66 last 20.80) - open/close 20899 (time 1787888570.34 total 950.66 last 68.70) - open/close 21642 (time 1787888580.34 total 960.66 last 74.29) - open/close 22362 (time 1787888590.34 total 970.66 last 71.98) - open/close 23061 (time 1787888600.35 total 980.67 last 69.85) - open/close 23783 (time 1787888610.36 total 990.68 last 72.15) - open/close 24502 (time 1787888620.36 total 1000.68 last 71.90) - open/close 25309 (time 1787888630.37 total 1010.69 last 80.58) - open/close 26043 (time 1787888640.40 total 1020.71 last 73.23) - open/close 26825 (time 1787888650.40 total 1030.72 last 78.13) - open/close 27602 (time 1787888660.41 total 1040.73 last 77.67) - open/close 28393 (time 1787888670.42 total 1050.74 last 78.98) - open/close 29117 (time 1787888680.43 total 1060.74 last 72.39) - open/close 29841 (time 1787888690.43 total 1070.75 last 72.35) - open/close 30000 (time 1787888692.95 total 1073.27 last 63.06) - open/close 30774 (time 1787888702.96 total 1083.28 last 77.37) - open/close 31522 (time 1787888712.96 total 1093.28 last 74.80) - open/close 32315 (time 1787888722.97 total 1103.28 last 79.25) - open/close 33098 (time 1787888732.97 total 1113.29 last 78.25) - open/close 33870 (time 1787888742.98 total 1123.29 last 77.17) - open/close 34622 (time 1787888752.98 total 1133.30 last 75.17) - open/close 35349 (time 1787888762.99 total 1143.31 last 72.60) - open/close 35997 (time 1787888773.00 total 1153.31 last 64.79) - open/close 36645 (time 1787888783.02 total 1163.33 last 64.67) - open/close 37332 (time 1787888793.02 total 1173.34 last 68.65) - open/close 38070 (time 1787888803.03 total 1183.35 last 73.78) - open/close 38799 (time 1787888813.05 total 1193.37 last 72.74) - open/close 39462 (time 1787888823.06 total 1203.38 last 66.24) - open/close 40000 (time 1787888831.14 total 1211.46 last 66.58) - open/close 40672 (time 1787888841.14 total 1221.46 last 67.20) - open/close 41429 (time 1787888851.18 total 1231.50 last 75.41) - open/close 42094 (time 1787888861.18 total 1241.50 last 66.47) - open/close 42825 (time 1787888871.20 total 1251.51 last 73.00) - open/close 43579 (time 1787888881.20 total 1261.52 last 75.38) - open/close 44314 (time 1787888891.20 total 1271.52 last 73.49) - open/close 44927 (time 1787888901.23 total 1281.55 last 61.13) - open/close 45568 (time 1787888911.23 total 1291.55 last 64.09) - open/close 46315 (time 1787888921.25 total 1301.57 last 74.55) - open/close 47070 (time 1787888931.25 total 1311.57 last 75.49) - open/close 47819 (time 1787888941.25 total 1321.57 last 74.87) - open/close 48598 (time 1787888951.25 total 1331.57 last 77.89) - open/close 49324 (time 1787888961.26 total 1341.57 last 72.59) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1345.62 seconds: 36.88 ops/second - unlinked 0 (time 1787888969 ; total 0 ; last 0) - unlinked 10000 (time 1787889109 ; total 140 ; last 140) - unlinked 20000 (time 1787889243 ; total 274 ; last 134) - unlinked 30000 (time 1787889389 ; total 420 ; last 146) - unlinked 40000 (time 1787889534 ; total 565 ; last 145) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 705 seconds: 70.398582 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg343-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg343-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg343-server stop mds service on oleg343-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg343-server stop mds service on oleg343-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg343-server unloading modules via unload_modules_local on: 'oleg343-server' oleg343-server: oleg343-server.virtnet: executing unload_modules_local oleg343-server: kmemleak disabled modules unloaded. pdsh@oleg343-client: oleg343-client: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-server: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-client: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-server: ssh exited with exit code 2