== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:43:52 (1788443032) start mds service on oleg233-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 3 07:51 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg233-server' oleg233-server: oleg233-server.virtnet: executing load_modules_local oleg233-server: kmemleak disabled oleg233-server: Loading modules from /home/green/git/lustre-release/lustre oleg233-server: detected 4 online CPUs by sysfs oleg233-server: MODOPTS_LIBCFS= oleg233-server: Force libcfs to create 2 CPU partitions oleg233-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg233-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg233-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg233-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg233-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg233-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 174 (time 1788443143.98 total 10.04 last 17.34) - open/close 340 (time 1788443154.00 total 20.05 last 16.58) - open/close 548 (time 1788443164.00 total 30.05 last 20.80) - open/close 822 (time 1788443174.29 total 40.34 last 26.62) - open/close 1027 (time 1788443184.32 total 50.37 last 20.45) - open/close 1145 (time 1788443194.32 total 60.38 last 11.79) - open/close 1297 (time 1788443204.95 total 71.00 last 14.31) - open/close 1488 (time 1788443214.95 total 81.00 last 19.09) - open/close 1702 (time 1788443225.59 total 91.65 last 20.11) - open/close 1841 (time 1788443235.60 total 101.65 last 13.89) - open/close 2092 (time 1788443245.75 total 111.81 last 24.72) - open/close 2323 (time 1788443256.40 total 122.45 last 21.70) - open/close 2344 (time 1788443267.17 total 133.22 last 1.95) - open/close 2558 (time 1788443277.83 total 143.88 last 20.06) - open/close 2818 (time 1788443287.84 total 153.89 last 25.98) - open/close 3088 (time 1788443297.93 total 163.99 last 26.75) - open/close 3376 (time 1788443307.95 total 174.00 last 28.75) - open/close 3488 (time 1788443318.03 total 184.08 last 11.11) - open/close 3527 (time 1788443328.33 total 194.39 last 3.79) - open/close 3725 (time 1788443339.21 total 205.26 last 18.21) - open/close 4025 (time 1788443349.22 total 215.27 last 29.97) - open/close 4286 (time 1788443359.24 total 225.30 last 26.03) - open/close 4548 (time 1788443369.69 total 235.74 last 25.08) - open/close 4709 (time 1788443380.00 total 246.05 last 15.61) - open/close 4931 (time 1788443390.01 total 256.06 last 22.19) - open/close 5109 (time 1788443400.56 total 266.61 last 16.88) - open/close 5272 (time 1788443410.56 total 276.61 last 16.29) - open/close 5527 (time 1788443421.11 total 287.16 last 24.18) - open/close 5760 (time 1788443431.41 total 297.46 last 22.62) - open/close 5891 (time 1788443442.40 total 308.45 last 11.92) - open/close 6096 (time 1788443452.40 total 318.45 last 20.49) - open/close 6326 (time 1788443462.79 total 328.84 last 22.13) - open/close 6643 (time 1788443473.55 total 339.60 last 29.48) - open/close 6945 (time 1788443483.56 total 349.61 last 30.17) - open/close 7112 (time 1788443493.57 total 359.62 last 16.69) - open/close 7431 (time 1788443503.57 total 369.62 last 31.89) - open/close 7614 (time 1788443514.33 total 380.38 last 17.01) - open/close 7828 (time 1788443525.19 total 391.25 last 19.69) - open/close 8094 (time 1788443535.20 total 401.25 last 26.58) - open/close 8380 (time 1788443545.21 total 411.26 last 28.58) - open/close 8667 (time 1788443555.85 total 421.90 last 26.99) - open/close 8922 (time 1788443566.37 total 432.42 last 24.23) - open/close 9032 (time 1788443578.92 total 444.97 last 8.76) - open/close 9216 (time 1788443589.78 total 455.84 last 16.94) - open/close 9515 (time 1788443599.80 total 465.85 last 29.85) - open/close 9727 (time 1788443609.81 total 475.86 last 21.18) - open/close 9969 (time 1788443620.50 total 486.55 last 22.65) - open/close 10158 (time 1788443630.94 total 497.00 last 18.09) - open/close 10436 (time 1788443642.03 total 508.08 last 25.08) - open/close 10703 (time 1788443652.82 total 518.87 last 24.76) - open/close 10957 (time 1788443663.50 total 529.55 last 23.77) - open/close 11164 (time 1788443673.51 total 539.56 last 20.68) - open/close 11412 (time 1788443683.52 total 549.57 last 24.78) - open/close 11668 (time 1788443693.85 total 559.90 last 24.79) - open/close 11929 (time 1788443703.85 total 569.90 last 26.10) - open/close 12103 (time 1788443715.27 total 581.33 last 15.23) - open/close 12285 (time 1788443725.77 total 591.82 last 17.35) - open/close 12396 (time 1788443736.09 total 602.14 last 10.76) - open/close 12514 (time 1788443746.71 total 612.76 last 11.11) - open/close 12657 (time 1788443757.20 total 623.25 last 13.63) - open/close 12870 (time 1788443767.25 total 633.30 last 21.21) - open/close 13028 (time 1788443777.50 total 643.55 last 15.41) - open/close 13252 (time 1788443788.25 total 654.30 last 20.85) - open/close 13439 (time 1788443798.26 total 664.31 last 18.68) - open/close 13665 (time 1788443808.28 total 674.33 last 22.54) - open/close 13931 (time 1788443818.31 total 684.36 last 26.53) - open/close 14123 (time 1788443828.32 total 694.37 last 19.17) - open/close 14264 (time 1788443838.34 total 704.39 last 14.07) - open/close 14542 (time 1788443848.34 total 714.40 last 27.79) - open/close 14669 (time 1788443859.35 total 725.40 last 11.54) - open/close 14831 (time 1788443869.52 total 735.57 last 15.92) - open/close 15160 (time 1788443879.54 total 745.59 last 32.85) - open/close 15440 (time 1788443889.56 total 755.61 last 27.93) - open/close 15736 (time 1788443899.60 total 765.65 last 29.48) - open/close 15949 (time 1788443909.62 total 775.67 last 21.26) - open/close 16063 (time 1788443919.98 total 786.03 last 11.01) - open/close 16261 (time 1788443930.84 total 796.89 last 18.23) - open/close 16424 (time 1788443942.11 total 808.17 last 14.45) - open/close 16672 (time 1788443952.21 total 818.26 last 24.58) - open/close 16822 (time 1788443963.50 total 829.55 last 13.28) - open/close 16986 (time 1788443974.41 total 840.46 last 15.03) - open/close 17128 (time 1788443984.73 total 850.78 last 13.77) - open/close 17298 (time 1788443994.74 total 860.80 last 16.97) - open/close 17519 (time 1788444005.78 total 871.84 last 20.02) - open/close 17738 (time 1788444015.81 total 881.86 last 21.84) - open/close 17994 (time 1788444026.48 total 892.53 last 24.00) - open/close 18273 (time 1788444036.49 total 902.54 last 27.86) - open/close 18476 (time 1788444049.55 total 915.61 last 15.54) - open/close 18743 (time 1788444060.04 total 926.09 last 25.46) - open/close 18887 (time 1788444070.76 total 936.81 last 13.43) - open/close 19045 (time 1788444080.77 total 946.82 last 15.79) - open/close 19238 (time 1788444090.79 total 956.85 last 19.24) - open/close 19351 (time 1788444101.57 total 967.62 last 10.49) - open/close 19435 (time 1788444111.58 total 977.63 last 8.39) - open/close 19571 (time 1788444122.38 total 988.43 last 12.60) - open/close 20000 (time 1788444126.58 total 992.64 last 101.96) - open/close 21001 (time 1788444136.60 total 1002.65 last 99.98) - open/close 21905 (time 1788444146.60 total 1012.65 last 90.39) - open/close 22756 (time 1788444156.60 total 1022.66 last 85.05) - open/close 23829 (time 1788444166.62 total 1032.67 last 107.10) - open/close 24876 (time 1788444176.63 total 1042.68 last 104.67) - open/close 25625 (time 1788444186.65 total 1052.70 last 74.74) - open/close 26453 (time 1788444196.65 total 1062.70 last 82.76) - open/close 27330 (time 1788444206.65 total 1072.71 last 87.68) - open/close 28213 (time 1788444216.66 total 1082.71 last 88.27) - open/close 28997 (time 1788444226.66 total 1092.71 last 78.37) - open/close 29838 (time 1788444236.66 total 1102.71 last 84.09) - open/close 30880 (time 1788444246.67 total 1112.72 last 104.16) - open/close 31799 (time 1788444256.67 total 1122.72 last 91.90) - open/close 32797 (time 1788444266.67 total 1132.72 last 99.78) - open/close 33915 (time 1788444276.67 total 1142.72 last 111.78) - open/close 34737 (time 1788444286.68 total 1152.73 last 82.14) - open/close 35647 (time 1788444296.68 total 1162.73 last 90.95) - open/close 36597 (time 1788444306.69 total 1172.74 last 94.96) - open/close 37593 (time 1788444316.70 total 1182.75 last 99.50) - open/close 38541 (time 1788444326.70 total 1192.75 last 94.78) - open/close 39329 (time 1788444336.70 total 1202.76 last 78.76) - open/close 40000 (time 1788444343.09 total 1209.14 last 105.13) - open/close 40730 (time 1788444353.09 total 1219.15 last 72.95) - open/close 41536 (time 1788444363.10 total 1229.15 last 80.59) - open/close 42296 (time 1788444373.10 total 1239.15 last 76.00) - open/close 43178 (time 1788444383.10 total 1249.16 last 88.12) - open/close 43956 (time 1788444393.11 total 1259.17 last 77.72) - open/close 44869 (time 1788444403.12 total 1269.17 last 91.28) - open/close 45598 (time 1788444413.12 total 1279.17 last 72.86) - open/close 46452 (time 1788444423.14 total 1289.19 last 85.28) - open/close 47397 (time 1788444433.14 total 1299.19 last 94.46) - open/close 48340 (time 1788444443.15 total 1309.20 last 94.17) - open/close 49249 (time 1788444453.15 total 1319.21 last 90.90) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1323.60 seconds: 37.50 ops/second - unlinked 0 (time 1788444462 ; total 0 ; last 0) - unlinked 10000 (time 1788444578 ; total 116 ; last 116) - unlinked 20000 (time 1788444681 ; total 219 ; last 103) - unlinked 30000 (time 1788444735 ; total 273 ; last 54) - unlinked 40000 (time 1788444798 ; total 336 ; last 63) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 393 seconds: 126.287529 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg233-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg233-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg233-server stop mds service on oleg233-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg233-server stop mds service on oleg233-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg233-server unloading modules via unload_modules_local on: 'oleg233-server' oleg233-server: oleg233-server.virtnet: executing unload_modules_local oleg233-server: kmemleak disabled modules unloaded. pdsh@oleg233-client: oleg233-client: ssh exited with exit code 2 pdsh@oleg233-client: oleg233-server: ssh exited with exit code 2 pdsh@oleg233-client: oleg233-server: ssh exited with exit code 2 pdsh@oleg233-client: oleg233-client: ssh exited with exit code 2