== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:25:06 (1790897106) start mds service on oleg245-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1940568 Oct 1 17:41 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: kmemleak disabled oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: MODOPTS_LIBCFS= oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg245-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg245-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg245-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg245-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg245-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 242 (time 1790897220.58 total 10.01 last 24.19) - open/close 455 (time 1790897230.59 total 20.01 last 21.28) - open/close 712 (time 1790897240.60 total 30.02 last 25.69) - open/close 893 (time 1790897250.65 total 40.07 last 18.01) - open/close 1179 (time 1790897261.15 total 50.57 last 27.24) - open/close 1488 (time 1790897271.76 total 61.19 last 29.11) - open/close 1795 (time 1790897282.58 total 72.01 last 28.37) - open/close 2137 (time 1790897292.60 total 82.02 last 34.15) - open/close 2411 (time 1790897303.25 total 92.67 last 25.72) - open/close 2704 (time 1790897313.88 total 103.30 last 27.56) - open/close 2944 (time 1790897323.94 total 113.36 last 23.87) - open/close 3195 (time 1790897334.24 total 123.67 last 24.35) - open/close 3458 (time 1790897345.25 total 134.68 last 23.89) - open/close 3748 (time 1790897355.26 total 144.68 last 29.00) - open/close 4006 (time 1790897365.86 total 155.28 last 24.33) - open/close 4184 (time 1790897376.17 total 165.59 last 17.27) - open/close 4451 (time 1790897386.18 total 175.60 last 26.67) - open/close 4719 (time 1790897396.18 total 185.60 last 26.80) - open/close 4949 (time 1790897406.19 total 195.61 last 22.98) - open/close 5201 (time 1790897416.85 total 206.27 last 23.64) - open/close 5452 (time 1790897427.76 total 217.18 last 23.01) - open/close 5660 (time 1790897438.11 total 227.53 last 20.10) - open/close 5873 (time 1790897448.33 total 237.76 last 20.83) - open/close 6053 (time 1790897458.34 total 247.76 last 17.99) - open/close 6254 (time 1790897468.37 total 257.79 last 20.04) - open/close 6551 (time 1790897478.39 total 267.81 last 29.63) - open/close 6779 (time 1790897489.63 total 279.05 last 20.29) - open/close 6902 (time 1790897500.57 total 290.00 last 11.24) - open/close 7085 (time 1790897511.00 total 300.43 last 17.55) - open/close 7283 (time 1790897521.40 total 310.82 last 19.05) - open/close 7417 (time 1790897531.41 total 320.84 last 13.38) - open/close 7627 (time 1790897541.42 total 330.85 last 20.98) - open/close 7866 (time 1790897551.73 total 341.15 last 23.20) - open/close 8034 (time 1790897564.65 total 354.07 last 13.00) - open/close 8239 (time 1790897576.68 total 366.10 last 17.04) - open/close 8508 (time 1790897587.47 total 376.89 last 24.92) - open/close 8615 (time 1790897597.78 total 387.20 last 10.38) - open/close 8804 (time 1790897608.14 total 397.56 last 18.24) - open/close 9055 (time 1790897618.98 total 408.40 last 23.17) - open/close 9228 (time 1790897629.47 total 418.89 last 16.49) - open/close 9522 (time 1790897640.29 total 429.71 last 27.17) - open/close 9675 (time 1790897650.53 total 439.95 last 14.95) - open/close 9888 (time 1790897660.54 total 449.96 last 21.27) - open/close 10000 (time 1790897665.36 total 454.79 last 23.22) - open/close 10237 (time 1790897675.69 total 465.11 last 22.96) - open/close 10407 (time 1790897685.93 total 475.35 last 16.60) - open/close 10563 (time 1790897698.07 total 487.49 last 12.85) - open/close 10679 (time 1790897708.45 total 497.87 last 11.18) - open/close 10828 (time 1790897719.15 total 508.57 last 13.92) - open/close 11009 (time 1790897729.16 total 518.58 last 18.10) - open/close 11246 (time 1790897740.41 total 529.83 last 21.05) - open/close 11428 (time 1790897750.45 total 539.88 last 18.12) - open/close 11571 (time 1790897760.46 total 549.88 last 14.30) - open/close 11833 (time 1790897771.23 total 560.65 last 24.32) - open/close 12087 (time 1790897781.24 total 570.66 last 25.38) - open/close 12181 (time 1790897792.29 total 581.71 last 8.50) - open/close 12320 (time 1790897802.31 total 591.73 last 13.88) - open/close 12478 (time 1790897813.22 total 602.64 last 14.48) - open/close 12650 (time 1790897823.89 total 613.32 last 16.12) - open/close 12801 (time 1790897836.57 total 625.99 last 11.91) - open/close 13029 (time 1790897847.32 total 636.74 last 21.21) - open/close 13190 (time 1790897859.10 total 648.52 last 13.67) - open/close 13431 (time 1790897870.51 total 659.93 last 21.12) - open/close 13517 (time 1790897880.79 total 670.21 last 8.37) - open/close 13789 (time 1790897891.68 total 681.10 last 24.99) - open/close 14050 (time 1790897902.27 total 691.69 last 24.64) - open/close 14230 (time 1790897913.05 total 702.47 last 16.71) - open/close 14392 (time 1790897923.62 total 713.04 last 15.32) - open/close 14548 (time 1790897933.64 total 723.06 last 15.57) - open/close 14789 (time 1790897943.71 total 733.14 last 23.92) - open/close 14935 (time 1790897953.72 total 743.15 last 14.59) - open/close 15199 (time 1790897963.73 total 753.15 last 26.39) - open/close 15451 (time 1790897973.73 total 763.15 last 25.19) - open/close 15597 (time 1790897983.74 total 773.16 last 14.59) - open/close 15741 (time 1790897993.92 total 783.34 last 14.14) - open/close 15994 (time 1790898004.26 total 793.68 last 24.47) - open/close 16170 (time 1790898016.94 total 806.37 last 13.87) - open/close 16426 (time 1790898028.02 total 817.44 last 23.11) - open/close 16604 (time 1790898038.04 total 827.47 last 17.76) - open/close 16731 (time 1790898048.06 total 837.48 last 12.68) - open/close 16954 (time 1790898058.07 total 847.49 last 22.28) - open/close 17170 (time 1790898068.45 total 857.87 last 20.81) - open/close 17407 (time 1790898079.27 total 868.70 last 21.89) - open/close 17770 (time 1790898089.78 total 879.20 last 34.56) - open/close 17957 (time 1790898099.78 total 889.20 last 18.69) - open/close 18140 (time 1790898109.79 total 899.21 last 18.29) - open/close 18369 (time 1790898120.10 total 909.52 last 22.20) - open/close 18630 (time 1790898135.92 total 925.34 last 16.50) - open/close 18928 (time 1790898146.79 total 936.22 last 27.40) - open/close 19195 (time 1790898156.81 total 946.24 last 26.65) - open/close 19406 (time 1790898166.82 total 956.24 last 21.09) - open/close 19817 (time 1790898176.82 total 966.24 last 41.08) - open/close 20000 (time 1790898178.92 total 968.34 last 87.34) - open/close 20888 (time 1790898188.92 total 978.34 last 88.76) - open/close 21854 (time 1790898198.92 total 988.34 last 96.59) - open/close 22811 (time 1790898208.93 total 998.36 last 95.59) - open/close 23548 (time 1790898218.94 total 1008.36 last 73.69) - open/close 24413 (time 1790898228.94 total 1018.36 last 86.45) - open/close 25158 (time 1790898238.95 total 1028.37 last 74.42) - open/close 26023 (time 1790898248.96 total 1038.38 last 86.47) - open/close 26950 (time 1790898258.96 total 1048.38 last 92.66) - open/close 27922 (time 1790898268.99 total 1058.41 last 96.94) - open/close 28762 (time 1790898278.99 total 1068.41 last 83.98) - open/close 29670 (time 1790898288.99 total 1078.41 last 90.79) - open/close 30000 (time 1790898293.20 total 1082.62 last 78.38) - open/close 30954 (time 1790898303.20 total 1092.62 last 95.40) - open/close 31847 (time 1790898313.20 total 1102.62 last 89.29) - open/close 32677 (time 1790898323.21 total 1112.63 last 82.96) - open/close 33611 (time 1790898333.21 total 1122.63 last 93.40) - open/close 34624 (time 1790898343.21 total 1132.63 last 101.25) - open/close 35568 (time 1790898353.22 total 1142.64 last 94.35) - open/close 36558 (time 1790898363.23 total 1152.65 last 98.91) - open/close 37455 (time 1790898373.23 total 1162.65 last 89.64) - open/close 38340 (time 1790898383.24 total 1172.66 last 88.44) - open/close 39255 (time 1790898393.24 total 1182.66 last 91.47) - open/close 40000 (time 1790898401.15 total 1190.57 last 94.25) - open/close 40989 (time 1790898411.15 total 1200.58 last 98.84) - open/close 41942 (time 1790898421.16 total 1210.58 last 95.27) - open/close 42818 (time 1790898431.16 total 1220.59 last 87.53) - open/close 43806 (time 1790898441.17 total 1230.59 last 98.79) - open/close 44776 (time 1790898451.17 total 1240.59 last 96.99) - open/close 45753 (time 1790898461.17 total 1250.59 last 97.65) - open/close 46756 (time 1790898471.17 total 1260.60 last 100.27) - open/close 47730 (time 1790898481.18 total 1270.60 last 97.39) - open/close 48657 (time 1790898491.19 total 1280.61 last 92.56) - open/close 49530 (time 1790898501.21 total 1290.63 last 87.16) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1291.94 seconds: 38.42 ops/second - unlinked 0 (time 1790898506 ; total 0 ; last 0) - unlinked 10000 (time 1790898608 ; total 102 ; last 102) - unlinked 20000 (time 1790898715 ; total 209 ; last 107) - unlinked 30000 (time 1790898779 ; total 273 ; last 64) - unlinked 40000 (time 1790898839 ; total 333 ; last 60) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 392 seconds: 126.609695 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local oleg245-server: kmemleak disabled modules unloaded. pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2