== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:44:31 (1788835471) start mds service on oleg248-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1754936 Sep 7 17:53 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg248-server' oleg248-server: oleg248-server.virtnet: executing load_modules_local oleg248-server: kmemleak disabled oleg248-server: Loading modules from /home/green/git/lustre-release/lustre oleg248-server: detected 4 online CPUs by sysfs oleg248-server: MODOPTS_LIBCFS= oleg248-server: Force libcfs to create 2 CPU partitions oleg248-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg248-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg248-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg248-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg248-server: oleg248-server.virtnet: executing set_default_debug -1 all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg248-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg248-server: oleg248-server.virtnet: executing set_default_debug -1 all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg248-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug -1 all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0000 oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 150 (time 1788835595.38 total 10.02 last 14.97) - open/close 335 (time 1788835605.72 total 20.36 last 17.89) - open/close 567 (time 1788835615.72 total 30.36 last 23.20) - open/close 674 (time 1788835626.36 total 40.99 last 10.06) - open/close 743 (time 1788835637.20 total 51.84 last 6.36) - open/close 854 (time 1788835647.41 total 62.04 last 10.88) - open/close 998 (time 1788835658.48 total 73.11 last 13.01) - open/close 1281 (time 1788835671.49 total 86.13 last 21.75) - open/close 1356 (time 1788835681.69 total 96.33 last 7.35) - open/close 1596 (time 1788835691.70 total 106.34 last 23.98) - open/close 1697 (time 1788835701.86 total 116.50 last 9.94) - open/close 1970 (time 1788835712.58 total 127.21 last 25.47) - open/close 2108 (time 1788835722.61 total 137.24 last 13.76) - open/close 2260 (time 1788835733.23 total 147.87 last 14.30) - open/close 2452 (time 1788835746.23 total 160.87 last 14.77) - open/close 2570 (time 1788835756.24 total 170.87 last 11.79) - open/close 2837 (time 1788835766.25 total 180.88 last 26.67) - open/close 3118 (time 1788835776.27 total 190.91 last 28.02) - open/close 3343 (time 1788835786.77 total 201.41 last 21.43) - open/close 3512 (time 1788835796.94 total 211.57 last 16.63) - open/close 3709 (time 1788835807.61 total 222.25 last 18.46) - open/close 3885 (time 1788835817.90 total 232.54 last 17.10) - open/close 4032 (time 1788835828.64 total 243.28 last 13.68) - open/close 4217 (time 1788835838.66 total 253.29 last 18.48) - open/close 4503 (time 1788835848.66 total 263.30 last 28.57) - open/close 4623 (time 1788835861.55 total 276.19 last 9.31) - open/close 4863 (time 1788835871.79 total 286.43 last 23.44) - open/close 5024 (time 1788835885.36 total 299.99 last 11.87) - open/close 5109 (time 1788835896.30 total 310.94 last 7.77) - open/close 5413 (time 1788835906.88 total 321.51 last 28.75) - open/close 5658 (time 1788835918.51 total 333.15 last 21.06) - open/close 5800 (time 1788835928.93 total 343.57 last 13.62) - open/close 6149 (time 1788835941.20 total 355.83 last 28.46) - open/close 6344 (time 1788835951.21 total 365.85 last 19.47) - open/close 6548 (time 1788835962.17 total 376.80 last 18.62) - open/close 6714 (time 1788835973.22 total 387.86 last 15.01) - open/close 6758 (time 1788835983.66 total 398.29 last 4.22) - open/close 6955 (time 1788835993.66 total 408.30 last 19.69) - open/close 7218 (time 1788836004.68 total 419.32 last 23.88) - open/close 7529 (time 1788836014.68 total 429.32 last 31.09) - open/close 7736 (time 1788836024.94 total 439.58 last 20.18) - open/close 7897 (time 1788836034.94 total 449.58 last 16.09) - open/close 8006 (time 1788836044.96 total 459.60 last 10.88) - open/close 8238 (time 1788836055.02 total 469.65 last 23.08) - open/close 8417 (time 1788836065.03 total 479.67 last 17.88) - open/close 8523 (time 1788836075.82 total 490.45 last 9.83) - open/close 8635 (time 1788836085.83 total 500.47 last 11.18) - open/close 8882 (time 1788836095.94 total 510.58 last 24.44) - open/close 9095 (time 1788836106.55 total 521.19 last 20.07) - open/close 9215 (time 1788836117.43 total 532.07 last 11.03) - open/close 9471 (time 1788836127.43 total 542.07 last 25.60) - open/close 9693 (time 1788836138.36 total 552.99 last 20.32) - open/close 9950 (time 1788836148.38 total 563.02 last 25.64) - open/close 10237 (time 1788836159.15 total 573.79 last 26.64) - open/close 10334 (time 1788836169.52 total 584.16 last 9.35) - open/close 10587 (time 1788836179.95 total 594.59 last 24.26) - open/close 10728 (time 1788836190.96 total 605.59 last 12.81) - open/close 10844 (time 1788836201.77 total 616.40 last 10.73) - open/close 11064 (time 1788836212.29 total 626.93 last 20.91) - open/close 11335 (time 1788836222.86 total 637.50 last 25.63) - open/close 11570 (time 1788836233.91 total 648.55 last 21.28) - open/close 11742 (time 1788836243.91 total 658.55 last 17.19) - open/close 11873 (time 1788836253.92 total 668.56 last 13.09) - open/close 12050 (time 1788836264.04 total 678.68 last 17.49) - open/close 12229 (time 1788836274.94 total 689.58 last 16.42) - open/close 12375 (time 1788836284.95 total 699.59 last 14.59) - open/close 12646 (time 1788836295.78 total 710.42 last 25.02) - open/close 12844 (time 1788836306.29 total 720.93 last 18.83) - open/close 13098 (time 1788836317.15 total 731.79 last 23.40) - open/close 13341 (time 1788836327.84 total 742.48 last 22.72) - open/close 13578 (time 1788836338.85 total 753.49 last 21.53) - open/close 13785 (time 1788836348.88 total 763.51 last 20.65) - open/close 14053 (time 1788836359.90 total 774.53 last 24.32) - open/close 14375 (time 1788836370.99 total 785.63 last 29.03) - open/close 14672 (time 1788836381.00 total 795.64 last 29.67) - open/close 14888 (time 1788836391.15 total 805.78 last 21.29) - open/close 15000 (time 1788836401.15 total 815.78 last 11.20) - open/close 15173 (time 1788836411.17 total 825.80 last 17.27) - open/close 15313 (time 1788836421.88 total 836.51 last 13.07) - open/close 15450 (time 1788836431.93 total 846.57 last 13.62) - open/close 15622 (time 1788836442.72 total 857.35 last 15.95) - open/close 15759 (time 1788836452.74 total 867.38 last 13.67) - open/close 15900 (time 1788836463.47 total 878.11 last 13.14) - open/close 16091 (time 1788836474.09 total 888.73 last 17.99) - open/close 16161 (time 1788836484.10 total 898.73 last 7.00) - open/close 16375 (time 1788836494.69 total 909.33 last 20.19) - open/close 16485 (time 1788836505.39 total 920.03 last 10.28) - open/close 16657 (time 1788836516.34 total 930.97 last 15.72) - open/close 16752 (time 1788836526.57 total 941.20 last 9.29) - open/close 16823 (time 1788836537.58 total 952.22 last 6.45) - open/close 16849 (time 1788836547.83 total 962.47 last 2.54) - open/close 16956 (time 1788836558.62 total 973.25 last 9.92) - open/close 17167 (time 1788836568.96 total 983.60 last 20.39) - open/close 17242 (time 1788836579.82 total 994.46 last 6.91) - open/close 17388 (time 1788836589.83 total 1004.47 last 14.60) - open/close 17603 (time 1788836599.83 total 1014.47 last 21.49) - open/close 17770 (time 1788836609.86 total 1024.49 last 16.66) - open/close 17886 (time 1788836619.96 total 1034.60 last 11.48) - open/close 17946 (time 1788836630.50 total 1045.14 last 5.69) - open/close 18384 (time 1788836640.51 total 1055.15 last 43.75) - open/close 19058 (time 1788836650.53 total 1065.16 last 67.33) - open/close 19868 (time 1788836660.53 total 1075.16 last 81.00) - open/close 20658 (time 1788836670.53 total 1085.16 last 78.99) - open/close 21526 (time 1788836680.54 total 1095.18 last 86.67) - open/close 22430 (time 1788836690.56 total 1105.20 last 90.22) - open/close 23394 (time 1788836700.57 total 1115.21 last 96.30) - open/close 23995 (time 1788836710.59 total 1125.23 last 60.01) - open/close 24680 (time 1788836720.59 total 1135.23 last 68.47) - open/close 25387 (time 1788836730.60 total 1145.23 last 70.68) - open/close 26259 (time 1788836740.60 total 1155.24 last 87.13) - open/close 27188 (time 1788836750.61 total 1165.25 last 92.81) - open/close 28199 (time 1788836760.62 total 1175.25 last 101.07) - open/close 29087 (time 1788836770.63 total 1185.26 last 88.71) - open/close 29832 (time 1788836780.64 total 1195.28 last 74.42) - open/close 30000 (time 1788836783.26 total 1197.89 last 64.17) - open/close 30924 (time 1788836793.26 total 1207.90 last 92.37) - open/close 31795 (time 1788836803.26 total 1217.90 last 87.07) - open/close 32785 (time 1788836813.27 total 1227.91 last 98.93) - open/close 33684 (time 1788836823.27 total 1237.91 last 89.88) - open/close 34611 (time 1788836833.28 total 1247.92 last 92.60) - open/close 35452 (time 1788836843.29 total 1257.93 last 84.07) - open/close 36166 (time 1788836853.30 total 1267.94 last 71.33) - open/close 36749 (time 1788836863.31 total 1277.94 last 58.25) - open/close 37736 (time 1788836873.31 total 1287.94 last 98.70) - open/close 38802 (time 1788836883.31 total 1297.94 last 106.59) - open/close 39681 (time 1788836893.31 total 1307.95 last 87.85) - open/close 40000 (time 1788836896.78 total 1311.42 last 92.01) - open/close 40920 (time 1788836906.81 total 1321.45 last 91.72) - open/close 41665 (time 1788836916.83 total 1331.47 last 74.35) - open/close 42444 (time 1788836926.83 total 1341.47 last 77.90) - open/close 43332 (time 1788836936.84 total 1351.47 last 88.75) - open/close 44065 (time 1788836946.85 total 1361.49 last 73.21) - open/close 44891 (time 1788836956.87 total 1371.51 last 82.42) - open/close 45668 (time 1788836966.88 total 1381.51 last 77.67) - open/close 46517 (time 1788836976.88 total 1391.52 last 84.88) - open/close 47364 (time 1788836986.89 total 1401.52 last 84.62) - open/close 48113 (time 1788836996.90 total 1411.53 last 74.84) - open/close 48921 (time 1788837006.90 total 1421.54 last 80.78) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1430.85 seconds: 34.69 ops/second - unlinked 0 (time 1788837021 ; total 0 ; last 0) - unlinked 10000 (time 1788837135 ; total 114 ; last 114) - unlinked 20000 (time 1788837227 ; total 206 ; last 92) - unlinked 30000 (time 1788837287 ; total 266 ; last 60) - unlinked 40000 (time 1788837344 ; total 323 ; last 57) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 378 seconds: 131.298935 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg248-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg248-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg248-server stop mds service on oleg248-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg248-server stop mds service on oleg248-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg248-server unloading modules via unload_modules_local on: 'oleg248-server' oleg248-server: oleg248-server.virtnet: executing unload_modules_local oleg248-server: kmemleak disabled modules unloaded. pdsh@oleg248-client: oleg248-client: ssh exited with exit code 2 pdsh@oleg248-client: oleg248-server: ssh exited with exit code 2 pdsh@oleg248-client: oleg248-client: ssh exited with exit code 2 pdsh@oleg248-client: oleg248-server: ssh exited with exit code 2