== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:50:56 (1788623456) start mds service on oleg638-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 4 17:39 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg638-server' oleg638-server: oleg638-server.virtnet: executing load_modules_local oleg638-server: kmemleak disabled oleg638-server: Loading modules from /home/green/git/lustre-release/lustre oleg638-server: detected 4 online CPUs by sysfs oleg638-server: MODOPTS_LIBCFS= oleg638-server: Force libcfs to create 2 CPU partitions oleg638-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg638-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg638-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg638-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg638-server: oleg638-server.virtnet: executing set_default_debug -1 all pdsh@oleg638-client: oleg638-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg638-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg638-server: oleg638-server.virtnet: executing set_default_debug -1 all pdsh@oleg638-client: oleg638-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg638-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg638-server: oleg638-server.virtnet: executing set_default_debug -1 all pdsh@oleg638-client: oleg638-server: ssh exited with exit code 1 Started lustre-OST0000 oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg638-client.virtnet: -o user_xattr,flock 192.168.206.138@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 207 (time 1788623559.78 total 10.00 last 20.70) - open/close 446 (time 1788623572.41 total 22.63 last 18.92) - open/close 688 (time 1788623582.84 total 33.06 last 23.21) - open/close 913 (time 1788623593.85 total 44.07 last 20.44) - open/close 1146 (time 1788623604.36 total 54.58 last 22.17) - open/close 1414 (time 1788623615.30 total 65.52 last 24.49) - open/close 1641 (time 1788623625.53 total 75.75 last 22.19) - open/close 1910 (time 1788623636.40 total 86.62 last 24.75) - open/close 2184 (time 1788623646.40 total 96.62 last 27.39) - open/close 2442 (time 1788623656.82 total 107.04 last 24.77) - open/close 2704 (time 1788623666.82 total 117.04 last 26.19) - open/close 2936 (time 1788623676.98 total 127.20 last 22.84) - open/close 3212 (time 1788623687.92 total 138.14 last 25.24) - open/close 3386 (time 1788623698.18 total 148.40 last 16.95) - open/close 3646 (time 1788623708.41 total 158.63 last 25.42) - open/close 3870 (time 1788623718.59 total 168.81 last 22.01) - open/close 4144 (time 1788623728.61 total 178.83 last 27.32) - open/close 4428 (time 1788623739.63 total 189.86 last 25.77) - open/close 4618 (time 1788623750.52 total 200.74 last 17.46) - open/close 4762 (time 1788623760.52 total 210.74 last 14.39) - open/close 5038 (time 1788623770.52 total 220.75 last 27.60) - open/close 5234 (time 1788623781.26 total 231.48 last 18.26) - open/close 5397 (time 1788623791.87 total 242.09 last 15.35) - open/close 5570 (time 1788623802.90 total 253.12 last 15.69) - open/close 5745 (time 1788623812.94 total 263.16 last 17.43) - open/close 6017 (time 1788623822.95 total 273.17 last 27.17) - open/close 6213 (time 1788623832.95 total 283.17 last 19.60) - open/close 6440 (time 1788623842.97 total 293.19 last 22.66) - open/close 6677 (time 1788623853.61 total 303.83 last 22.27) - open/close 6889 (time 1788623864.64 total 314.86 last 19.22) - open/close 7150 (time 1788623874.68 total 324.90 last 26.01) - open/close 7287 (time 1788623884.68 total 334.90 last 13.69) - open/close 7485 (time 1788623894.68 total 344.90 last 19.80) - open/close 7706 (time 1788623905.66 total 355.88 last 20.14) - open/close 7928 (time 1788623915.67 total 365.89 last 22.17) - open/close 8177 (time 1788623925.67 total 375.90 last 24.89) - open/close 8401 (time 1788623936.12 total 386.34 last 21.44) - open/close 8654 (time 1788623946.16 total 396.38 last 25.22) - open/close 8950 (time 1788623956.16 total 406.38 last 29.59) - open/close 9222 (time 1788623966.16 total 416.38 last 27.19) - open/close 9457 (time 1788623976.63 total 426.85 last 22.46) - open/close 9712 (time 1788623986.68 total 436.90 last 25.36) - open/close 9912 (time 1788623997.65 total 447.87 last 18.24) - open/close 10000 (time 1788624000.40 total 450.62 last 32.00) - open/close 10239 (time 1788624010.68 total 460.90 last 23.25) - open/close 10493 (time 1788624021.36 total 471.58 last 23.78) - open/close 10732 (time 1788624031.37 total 481.59 last 23.88) - open/close 11062 (time 1788624042.18 total 492.40 last 30.52) - open/close 11332 (time 1788624052.18 total 502.40 last 27.00) - open/close 11582 (time 1788624062.19 total 512.41 last 24.97) - open/close 11805 (time 1788624072.20 total 522.42 last 22.28) - open/close 12060 (time 1788624082.29 total 532.52 last 25.26) - open/close 12312 (time 1788624092.47 total 542.69 last 24.77) - open/close 12592 (time 1788624102.90 total 553.12 last 26.83) - open/close 12762 (time 1788624112.91 total 563.13 last 16.98) - open/close 12944 (time 1788624123.07 total 573.30 last 17.91) - open/close 13144 (time 1788624133.08 total 583.30 last 19.98) - open/close 13332 (time 1788624143.22 total 593.44 last 18.55) - open/close 13574 (time 1788624153.29 total 603.51 last 24.03) - open/close 13873 (time 1788624164.14 total 614.36 last 27.55) - open/close 14056 (time 1788624174.15 total 624.37 last 18.27) - open/close 14288 (time 1788624184.18 total 634.40 last 23.14) - open/close 14486 (time 1788624195.00 total 645.22 last 18.31) - open/close 14738 (time 1788624205.17 total 655.40 last 24.76) - open/close 15017 (time 1788624215.63 total 665.85 last 26.69) - open/close 15284 (time 1788624225.63 total 675.85 last 26.69) - open/close 15519 (time 1788624235.64 total 685.86 last 23.49) - open/close 15734 (time 1788624245.64 total 695.86 last 21.50) - open/close 16009 (time 1788624255.65 total 705.87 last 27.46) - open/close 16342 (time 1788624265.98 total 716.20 last 32.24) - open/close 16618 (time 1788624276.55 total 726.77 last 26.12) - open/close 16912 (time 1788624286.55 total 736.77 last 29.39) - open/close 17166 (time 1788624296.56 total 746.78 last 25.39) - open/close 17408 (time 1788624306.99 total 757.21 last 23.21) - open/close 17638 (time 1788624317.82 total 768.04 last 21.23) - open/close 17792 (time 1788624327.83 total 778.06 last 15.37) - open/close 18003 (time 1788624337.87 total 788.09 last 21.02) - open/close 18248 (time 1788624348.42 total 798.64 last 23.24) - open/close 18412 (time 1788624358.42 total 808.65 last 16.39) - open/close 18658 (time 1788624368.70 total 818.92 last 23.93) - open/close 18900 (time 1788624378.71 total 828.93 last 24.18) - open/close 19096 (time 1788624389.44 total 839.66 last 18.27) - open/close 19261 (time 1788624399.44 total 849.67 last 16.49) - open/close 19476 (time 1788624409.45 total 859.67 last 21.48) - open/close 19732 (time 1788624419.56 total 869.79 last 25.32) - open/close 19973 (time 1788624429.75 total 879.97 last 23.67) - open/close 20213 (time 1788624439.75 total 889.97 last 23.99) - open/close 20484 (time 1788624449.76 total 899.98 last 27.08) - open/close 20726 (time 1788624460.40 total 910.62 last 22.74) - open/close 21382 (time 1788624470.41 total 920.63 last 65.54) - open/close 22149 (time 1788624480.42 total 930.64 last 76.62) - open/close 22895 (time 1788624490.42 total 940.64 last 74.56) - open/close 23712 (time 1788624500.43 total 950.65 last 81.68) - open/close 24567 (time 1788624510.43 total 960.65 last 85.48) - open/close 25403 (time 1788624520.43 total 970.66 last 83.55) - open/close 26230 (time 1788624530.44 total 980.66 last 82.65) - open/close 27043 (time 1788624540.44 total 990.67 last 81.26) - open/close 27871 (time 1788624550.45 total 1000.67 last 82.80) - open/close 28694 (time 1788624560.45 total 1010.67 last 82.29) - open/close 29507 (time 1788624570.45 total 1020.67 last 81.28) - open/close 30000 (time 1788624576.39 total 1026.61 last 83.04) - open/close 30688 (time 1788624586.39 total 1036.61 last 68.75) - open/close 31382 (time 1788624596.41 total 1046.63 last 69.28) - open/close 32092 (time 1788624606.41 total 1056.64 last 70.98) - open/close 32803 (time 1788624616.43 total 1066.65 last 71.02) - open/close 33482 (time 1788624626.43 total 1076.65 last 67.88) - open/close 34223 (time 1788624636.44 total 1086.66 last 74.03) - open/close 34959 (time 1788624646.44 total 1096.66 last 73.59) - open/close 35650 (time 1788624656.44 total 1106.66 last 69.08) - open/close 36441 (time 1788624666.46 total 1116.68 last 79.00) - open/close 37227 (time 1788624676.49 total 1126.71 last 78.31) - open/close 38055 (time 1788624686.49 total 1136.71 last 82.80) - open/close 38814 (time 1788624696.50 total 1146.72 last 75.88) - open/close 39646 (time 1788624706.50 total 1156.72 last 83.19) - open/close 40000 (time 1788624710.98 total 1161.20 last 79.05) - open/close 40832 (time 1788624720.99 total 1171.21 last 83.11) - open/close 41549 (time 1788624730.99 total 1181.22 last 71.64) - open/close 42356 (time 1788624741.00 total 1191.22 last 80.68) - open/close 43157 (time 1788624751.00 total 1201.22 last 80.05) - open/close 43973 (time 1788624761.03 total 1211.25 last 81.38) - open/close 44695 (time 1788624771.04 total 1221.26 last 72.16) - open/close 45455 (time 1788624781.04 total 1231.26 last 75.98) - open/close 46249 (time 1788624791.05 total 1241.27 last 79.34) - open/close 47023 (time 1788624801.06 total 1251.28 last 77.33) - open/close 47817 (time 1788624811.06 total 1261.28 last 79.33) - open/close 48565 (time 1788624821.07 total 1271.29 last 74.73) - open/close 49403 (time 1788624831.08 total 1281.30 last 83.77) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1283.99 seconds: 38.65 ops/second - unlinked 0 (time 1788624838 ; total 0 ; last 0) - unlinked 10000 (time 1788624967 ; total 129 ; last 129) - unlinked 20000 (time 1788625097 ; total 259 ; last 130) - unlinked 30000 (time 1788625224 ; total 386 ; last 127) - unlinked 40000 (time 1788625353 ; total 515 ; last 129) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 634 seconds: 78.282333 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg638-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg638-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg638-server stop mds service on oleg638-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg638-server stop mds service on oleg638-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg638-server unloading modules via unload_modules_local on: 'oleg638-server' oleg638-server: oleg638-server.virtnet: executing unload_modules_local oleg638-server: kmemleak disabled modules unloaded. pdsh@oleg638-client: oleg638-client: ssh exited with exit code 2 pdsh@oleg638-client: oleg638-server: ssh exited with exit code 2 pdsh@oleg638-client: oleg638-server: ssh exited with exit code 2 pdsh@oleg638-client: oleg638-client: ssh exited with exit code 2