== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:11:03 (1788657063) start mds service on oleg157-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921680 Sep 5 00:29 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg157-server' oleg157-server: oleg157-server.virtnet: executing load_modules_local oleg157-server: kmemleak disabled oleg157-server: Loading modules from /home/green/git/lustre-release/lustre oleg157-server: detected 4 online CPUs by sysfs oleg157-server: MODOPTS_LIBCFS= oleg157-server: Force libcfs to create 2 CPU partitions oleg157-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg157-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg157-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg157-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg157-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg157-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0000 oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 243 (time 1788657163.22 total 11.00 last 22.10) - open/close 517 (time 1788657173.38 total 21.16 last 26.96) - open/close 754 (time 1788657183.40 total 31.18 last 23.66) - open/close 943 (time 1788657193.59 total 41.37 last 18.54) - open/close 1175 (time 1788657203.93 total 51.71 last 22.44) - open/close 1435 (time 1788657214.48 total 62.25 last 24.66) - open/close 1738 (time 1788657224.48 total 72.26 last 30.29) - open/close 1991 (time 1788657235.08 total 82.86 last 23.87) - open/close 2271 (time 1788657245.83 total 93.60 last 26.05) - open/close 2533 (time 1788657255.83 total 103.61 last 26.19) - open/close 2807 (time 1788657265.83 total 113.61 last 27.39) - open/close 3116 (time 1788657276.60 total 124.38 last 28.70) - open/close 3384 (time 1788657286.87 total 134.65 last 26.10) - open/close 3696 (time 1788657296.87 total 144.65 last 31.19) - open/close 3942 (time 1788657306.87 total 154.65 last 24.60) - open/close 4202 (time 1788657316.88 total 164.66 last 25.98) - open/close 4458 (time 1788657326.97 total 174.75 last 25.36) - open/close 4668 (time 1788657336.99 total 184.76 last 20.97) - open/close 4841 (time 1788657347.46 total 195.24 last 16.52) - open/close 5050 (time 1788657359.28 total 207.05 last 17.69) - open/close 5326 (time 1788657369.93 total 217.71 last 25.90) - open/close 5595 (time 1788657379.94 total 227.72 last 26.87) - open/close 5829 (time 1788657390.46 total 238.24 last 22.25) - open/close 6097 (time 1788657401.35 total 249.13 last 24.61) - open/close 6333 (time 1788657411.35 total 259.13 last 23.59) - open/close 6543 (time 1788657422.03 total 269.80 last 19.67) - open/close 6692 (time 1788657432.09 total 279.87 last 14.81) - open/close 6929 (time 1788657442.10 total 289.88 last 23.68) - open/close 7233 (time 1788657452.11 total 299.89 last 30.37) - open/close 7543 (time 1788657463.04 total 310.82 last 28.35) - open/close 7815 (time 1788657473.99 total 321.77 last 24.84) - open/close 8094 (time 1788657483.99 total 331.77 last 27.90) - open/close 8367 (time 1788657494.40 total 342.18 last 26.23) - open/close 8639 (time 1788657504.43 total 352.20 last 27.13) - open/close 8824 (time 1788657514.88 total 362.65 last 17.70) - open/close 9097 (time 1788657524.89 total 372.66 last 27.28) - open/close 9353 (time 1788657535.29 total 383.07 last 24.61) - open/close 9646 (time 1788657545.42 total 393.20 last 28.91) - open/close 9939 (time 1788657556.43 total 404.21 last 26.62) - open/close 10267 (time 1788657567.42 total 415.20 last 29.85) - open/close 10513 (time 1788657578.31 total 426.09 last 22.58) - open/close 10690 (time 1788657589.27 total 437.05 last 16.15) - open/close 10988 (time 1788657599.29 total 447.07 last 29.75) - open/close 11241 (time 1788657609.29 total 457.07 last 25.30) - open/close 11458 (time 1788657619.91 total 467.68 last 20.44) - open/close 11708 (time 1788657630.15 total 477.92 last 24.42) - open/close 11995 (time 1788657640.66 total 488.44 last 27.29) - open/close 12256 (time 1788657650.75 total 498.53 last 25.88) - open/close 12505 (time 1788657660.78 total 508.55 last 24.83) - open/close 12786 (time 1788657671.61 total 519.38 last 25.95) - open/close 13044 (time 1788657681.61 total 529.39 last 25.78) - open/close 13193 (time 1788657691.62 total 539.40 last 14.89) - open/close 13462 (time 1788657702.23 total 550.01 last 25.34) - open/close 13762 (time 1788657712.25 total 560.02 last 29.96) - open/close 13866 (time 1788657722.53 total 570.31 last 10.11) - open/close 14046 (time 1788657732.95 total 580.73 last 17.28) - open/close 14151 (time 1788657743.78 total 591.56 last 9.70) - open/close 14473 (time 1788657754.13 total 601.91 last 31.10) - open/close 14825 (time 1788657764.17 total 611.95 last 35.07) - open/close 15007 (time 1788657774.21 total 621.99 last 18.13) - open/close 15129 (time 1788657785.17 total 632.95 last 11.13) - open/close 15374 (time 1788657795.18 total 642.95 last 24.49) - open/close 15541 (time 1788657805.45 total 653.23 last 16.26) - open/close 15765 (time 1788657815.45 total 663.23 last 22.39) - open/close 15938 (time 1788657826.04 total 673.82 last 16.34) - open/close 16247 (time 1788657836.05 total 683.83 last 30.87) - open/close 16475 (time 1788657846.07 total 693.85 last 22.75) - open/close 16740 (time 1788657856.45 total 704.23 last 25.53) - open/close 16972 (time 1788657869.27 total 717.05 last 18.10) - open/close 17155 (time 1788657879.78 total 727.56 last 17.41) - open/close 17396 (time 1788657889.79 total 737.57 last 24.08) - open/close 17626 (time 1788657899.92 total 747.70 last 22.71) - open/close 17913 (time 1788657909.92 total 757.70 last 28.69) - open/close 18184 (time 1788657920.77 total 768.55 last 24.98) - open/close 18387 (time 1788657931.75 total 779.53 last 18.49) - open/close 18625 (time 1788657941.84 total 789.62 last 23.58) - open/close 18893 (time 1788657952.07 total 799.84 last 26.22) - open/close 19178 (time 1788657962.34 total 810.11 last 27.75) - open/close 19372 (time 1788657972.54 total 820.32 last 19.01) - open/close 19571 (time 1788657982.56 total 830.33 last 19.87) - open/close 19743 (time 1788657992.57 total 840.35 last 17.18) - open/close 20000 (time 1788658002.13 total 849.91 last 26.86) - open/close 20163 (time 1788658012.19 total 859.96 last 16.22) - open/close 20336 (time 1788658022.91 total 870.69 last 16.13) - open/close 20561 (time 1788658033.69 total 881.46 last 20.89) - open/close 20773 (time 1788658047.96 total 895.74 last 14.86) - open/close 21033 (time 1788658058.14 total 905.91 last 25.54) - open/close 21414 (time 1788658068.15 total 915.93 last 38.03) - open/close 22314 (time 1788658078.16 total 925.94 last 89.92) - open/close 23278 (time 1788658088.17 total 935.94 last 96.38) - open/close 24295 (time 1788658098.17 total 945.95 last 101.63) - open/close 25153 (time 1788658108.20 total 955.97 last 85.60) - open/close 25931 (time 1788658118.20 total 965.98 last 77.74) - open/close 26827 (time 1788658128.21 total 975.99 last 89.56) - open/close 27781 (time 1788658138.21 total 985.99 last 95.37) - open/close 28678 (time 1788658148.22 total 996.00 last 89.59) - open/close 29572 (time 1788658158.23 total 1006.01 last 89.36) - open/close 30000 (time 1788658162.89 total 1010.67 last 91.84) - open/close 30823 (time 1788658172.92 total 1020.70 last 82.05) - open/close 31894 (time 1788658182.92 total 1030.70 last 107.08) - open/close 32956 (time 1788658192.92 total 1040.70 last 106.18) - open/close 33992 (time 1788658202.93 total 1050.71 last 103.51) - open/close 34919 (time 1788658212.94 total 1060.71 last 92.66) - open/close 35863 (time 1788658222.95 total 1070.72 last 94.31) - open/close 36772 (time 1788658232.95 total 1080.73 last 90.82) - open/close 37648 (time 1788658242.96 total 1090.74 last 87.56) - open/close 38587 (time 1788658252.96 total 1100.74 last 93.87) - open/close 39480 (time 1788658262.97 total 1110.74 last 89.26) - open/close 40000 (time 1788658269.51 total 1117.29 last 79.45) - open/close 40989 (time 1788658279.52 total 1127.30 last 98.83) - open/close 41754 (time 1788658289.54 total 1137.31 last 76.36) - open/close 42394 (time 1788658299.55 total 1147.32 last 63.94) - open/close 43251 (time 1788658309.55 total 1157.33 last 85.68) - open/close 44360 (time 1788658319.55 total 1167.33 last 110.87) - open/close 45182 (time 1788658329.55 total 1177.33 last 82.20) - open/close 46270 (time 1788658339.56 total 1187.34 last 108.69) - open/close 47303 (time 1788658349.57 total 1197.35 last 103.23) - open/close 48252 (time 1788658359.57 total 1207.35 last 94.85) - open/close 49057 (time 1788658369.58 total 1217.36 last 80.46) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1224.27 seconds: 40.54 ops/second - unlinked 0 (time 1788658381 ; total 0 ; last 0) - unlinked 10000 (time 1788658490 ; total 109 ; last 109) - unlinked 20000 (time 1788658590 ; total 209 ; last 100) - unlinked 30000 (time 1788658646 ; total 265 ; last 56) - unlinked 40000 (time 1788658705 ; total 324 ; last 59) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 378 seconds: 131.298935 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg157-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg157-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg157-server stop mds service on oleg157-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg157-server stop mds service on oleg157-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg157-server unloading modules via unload_modules_local on: 'oleg157-server' oleg157-server: oleg157-server.virtnet: executing unload_modules_local oleg157-server: kmemleak disabled modules unloaded. pdsh@oleg157-client: oleg157-client: ssh exited with exit code 2 pdsh@oleg157-client: oleg157-server: ssh exited with exit code 2 pdsh@oleg157-client: oleg157-client: ssh exited with exit code 2 pdsh@oleg157-client: oleg157-server: ssh exited with exit code 2