== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:19:47 (1788848387) start mds service on oleg211-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 7 17:48 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg211-server' oleg211-server: oleg211-server.virtnet: executing load_modules_local oleg211-server: kmemleak disabled oleg211-server: Loading modules from /home/green/git/lustre-release/lustre oleg211-server: detected 4 online CPUs by sysfs oleg211-server: MODOPTS_LIBCFS= oleg211-server: Force libcfs to create 2 CPU partitions oleg211-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg211-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg211-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg211-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg211-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg211-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg211-client.virtnet: -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 171 (time 1788848490.37 total 10.25 last 16.68) - open/close 378 (time 1788848501.32 total 21.20 last 18.90) - open/close 555 (time 1788848512.01 total 31.88 last 16.57) - open/close 825 (time 1788848522.02 total 41.90 last 26.96) - open/close 1067 (time 1788848532.62 total 52.50 last 22.83) - open/close 1140 (time 1788848542.80 total 62.68 last 7.17) - open/close 1245 (time 1788848553.45 total 73.32 last 9.86) - open/close 1337 (time 1788848564.48 total 84.36 last 8.34) - open/close 1561 (time 1788848574.50 total 94.38 last 22.36) - open/close 1743 (time 1788848584.50 total 104.38 last 18.20) - open/close 1941 (time 1788848594.77 total 114.64 last 19.29) - open/close 2054 (time 1788848605.45 total 125.32 last 10.58) - open/close 2248 (time 1788848615.45 total 135.33 last 19.40) - open/close 2416 (time 1788848625.46 total 145.33 last 16.79) - open/close 2577 (time 1788848636.26 total 156.14 last 14.90) - open/close 2729 (time 1788848647.90 total 167.77 last 13.06) - open/close 2972 (time 1788848657.93 total 177.81 last 24.21) - open/close 3234 (time 1788848668.75 total 188.62 last 24.23) - open/close 3517 (time 1788848679.06 total 198.93 last 27.45) - open/close 3766 (time 1788848690.62 total 210.49 last 21.54) - open/close 3999 (time 1788848700.63 total 220.51 last 23.27) - open/close 4263 (time 1788848711.36 total 231.24 last 24.59) - open/close 4457 (time 1788848722.39 total 242.27 last 17.60) - open/close 4574 (time 1788848733.08 total 252.95 last 10.95) - open/close 4716 (time 1788848743.12 total 263.00 last 14.14) - open/close 4878 (time 1788848753.74 total 273.61 last 15.26) - open/close 5083 (time 1788848763.92 total 283.79 last 20.13) - open/close 5312 (time 1788848774.99 total 294.86 last 20.69) - open/close 5511 (time 1788848785.00 total 304.87 last 19.88) - open/close 5663 (time 1788848795.00 total 314.88 last 15.19) - open/close 5834 (time 1788848805.01 total 324.88 last 17.10) - open/close 6049 (time 1788848815.50 total 335.38 last 20.49) - open/close 6307 (time 1788848825.51 total 345.38 last 25.79) - open/close 6494 (time 1788848836.12 total 356.00 last 17.62) - open/close 6751 (time 1788848846.99 total 366.86 last 23.65) - open/close 7019 (time 1788848857.00 total 376.87 last 26.77) - open/close 7263 (time 1788848867.02 total 386.89 last 24.35) - open/close 7445 (time 1788848877.02 total 396.90 last 18.19) - open/close 7689 (time 1788848887.04 total 406.92 last 24.36) - open/close 7908 (time 1788848897.04 total 416.92 last 21.89) - open/close 8164 (time 1788848907.46 total 427.34 last 24.58) - open/close 8422 (time 1788848918.16 total 438.04 last 24.11) - open/close 8664 (time 1788848928.27 total 448.15 last 23.94) - open/close 8934 (time 1788848938.27 total 458.15 last 27.00) - open/close 9149 (time 1788848948.90 total 468.78 last 20.22) - open/close 9396 (time 1788848960.90 total 480.78 last 20.58) - open/close 9642 (time 1788848971.08 total 490.96 last 24.17) - open/close 9818 (time 1788848981.26 total 501.14 last 17.28) - open/close 10000 (time 1788848988.35 total 508.23 last 25.68) - open/close 10272 (time 1788848998.35 total 518.23 last 27.20) - open/close 10498 (time 1788849008.37 total 528.24 last 22.56) - open/close 10748 (time 1788849018.96 total 538.83 last 23.61) - open/close 10991 (time 1788849029.19 total 549.07 last 23.74) - open/close 11269 (time 1788849039.95 total 559.83 last 25.84) - open/close 11519 (time 1788849049.97 total 569.84 last 24.96) - open/close 11779 (time 1788849060.27 total 580.15 last 25.23) - open/close 12067 (time 1788849071.06 total 590.94 last 26.70) - open/close 12331 (time 1788849081.06 total 600.94 last 26.40) - open/close 12584 (time 1788849091.78 total 611.66 last 23.60) - open/close 12834 (time 1788849102.47 total 622.34 last 23.40) - open/close 13091 (time 1788849112.53 total 632.41 last 25.53) - open/close 13310 (time 1788849122.77 total 642.65 last 21.38) - open/close 13590 (time 1788849132.79 total 652.66 last 27.97) - open/close 13848 (time 1788849142.80 total 662.67 last 25.77) - open/close 14012 (time 1788849155.15 total 675.03 last 13.28) - open/close 14183 (time 1788849165.77 total 685.64 last 16.10) - open/close 14395 (time 1788849175.92 total 695.80 last 20.88) - open/close 14578 (time 1788849186.18 total 706.06 last 17.83) - open/close 14779 (time 1788849196.94 total 716.82 last 18.69) - open/close 14968 (time 1788849206.95 total 726.82 last 18.89) - open/close 15197 (time 1788849217.61 total 737.49 last 21.48) - open/close 15402 (time 1788849228.26 total 748.14 last 19.25) - open/close 15601 (time 1788849241.35 total 761.22 last 15.21) - open/close 15797 (time 1788849252.82 total 772.70 last 17.08) - open/close 16049 (time 1788849263.06 total 782.93 last 24.62) - open/close 16307 (time 1788849273.56 total 793.44 last 24.57) - open/close 16563 (time 1788849283.57 total 803.45 last 25.57) - open/close 16795 (time 1788849293.58 total 813.46 last 23.18) - open/close 17038 (time 1788849303.58 total 823.46 last 24.29) - open/close 17296 (time 1788849314.00 total 833.88 last 24.77) - open/close 17564 (time 1788849324.02 total 843.90 last 26.75) - open/close 17799 (time 1788849334.04 total 853.91 last 23.46) - open/close 18022 (time 1788849344.25 total 864.13 last 21.83) - open/close 18225 (time 1788849354.27 total 874.14 last 20.27) - open/close 18477 (time 1788849364.82 total 884.70 last 23.88) - open/close 18695 (time 1788849374.84 total 894.72 last 21.76) - open/close 18900 (time 1788849384.96 total 904.84 last 20.25) - open/close 19167 (time 1788849394.97 total 914.85 last 26.67) - open/close 19384 (time 1788849405.58 total 925.46 last 20.46) - open/close 20000 (time 1788849414.37 total 934.25 last 70.09) - open/close 21021 (time 1788849424.38 total 944.25 last 102.05) - open/close 22046 (time 1788849434.38 total 954.25 last 102.48) - open/close 23083 (time 1788849444.38 total 964.26 last 103.68) - open/close 24050 (time 1788849454.38 total 974.26 last 96.65) - open/close 25000 (time 1788849464.39 total 984.27 last 94.92) - open/close 25972 (time 1788849474.40 total 994.27 last 97.15) - open/close 27014 (time 1788849484.40 total 1004.28 last 104.20) - open/close 27945 (time 1788849494.40 total 1014.28 last 93.04) - open/close 28862 (time 1788849504.41 total 1024.29 last 91.66) - open/close 29671 (time 1788849514.42 total 1034.29 last 80.84) - open/close 30000 (time 1788849518.33 total 1038.21 last 84.08) - open/close 31072 (time 1788849528.33 total 1048.21 last 107.17) - open/close 32119 (time 1788849538.33 total 1058.21 last 104.70) - open/close 33134 (time 1788849548.34 total 1068.22 last 101.42) - open/close 34213 (time 1788849558.34 total 1078.22 last 107.90) - open/close 35274 (time 1788849568.34 total 1088.22 last 106.10) - open/close 36305 (time 1788849578.35 total 1098.23 last 102.99) - open/close 37363 (time 1788849588.35 total 1108.23 last 105.80) - open/close 38471 (time 1788849598.36 total 1118.23 last 110.77) - open/close 39469 (time 1788849608.36 total 1128.23 last 99.77) - open/close 40000 (time 1788849615.64 total 1135.51 last 72.96) - open/close 40956 (time 1788849625.64 total 1145.51 last 95.59) - open/close 41954 (time 1788849635.64 total 1155.52 last 99.75) - open/close 43045 (time 1788849645.65 total 1165.52 last 109.05) - open/close 44112 (time 1788849655.65 total 1175.52 last 106.69) - open/close 45179 (time 1788849665.65 total 1185.53 last 106.68) - open/close 46194 (time 1788849675.66 total 1195.53 last 101.42) - open/close 47069 (time 1788849685.66 total 1205.54 last 87.47) - open/close 48107 (time 1788849695.67 total 1215.55 last 103.72) - open/close 49034 (time 1788849705.68 total 1225.55 last 92.62) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1233.12 seconds: 40.25 ops/second - unlinked 0 (time 1788849716 ; total 0 ; last 0) - unlinked 10000 (time 1788849817 ; total 101 ; last 101) - unlinked 20000 (time 1788849901 ; total 185 ; last 84) - unlinked 30000 (time 1788849959 ; total 243 ; last 58) - unlinked 40000 (time 1788850014 ; total 298 ; last 55) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 348 seconds: 142.617813 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg211-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg211-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server stop mds service on oleg211-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server stop mds service on oleg211-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg211-server unloading modules via unload_modules_local on: 'oleg211-server' oleg211-server: oleg211-server.virtnet: executing unload_modules_local oleg211-server: kmemleak disabled modules unloaded. pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2