== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:48:13 (1788662893) start mds service on oleg637-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 5 07:32 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg637-server' oleg637-server: oleg637-server.virtnet: executing load_modules_local oleg637-server: kmemleak disabled oleg637-server: Loading modules from /home/green/git/lustre-release/lustre oleg637-server: detected 4 online CPUs by sysfs oleg637-server: MODOPTS_LIBCFS= oleg637-server: Force libcfs to create 2 CPU partitions oleg637-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg637-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg637-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg637-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg637-server: oleg637-server.virtnet: executing set_default_debug -1 all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg637-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg637-server: oleg637-server.virtnet: executing set_default_debug -1 all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg637-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg637-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg637-server: oleg637-server.virtnet: executing set_default_debug -1 all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-OST0000 oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 191 (time 1788663001.65 total 10.00 last 19.10) - open/close 381 (time 1788663011.68 total 20.03 last 18.95) - open/close 509 (time 1788663022.63 total 30.98 last 11.69) - open/close 665 (time 1788663033.63 total 41.99 last 14.17) - open/close 933 (time 1788663043.64 total 52.00 last 26.77) - open/close 1084 (time 1788663053.65 total 62.00 last 15.09) - open/close 1278 (time 1788663064.49 total 72.85 last 17.89) - open/close 1483 (time 1788663074.54 total 82.90 last 20.40) - open/close 1634 (time 1788663084.58 total 92.93 last 15.05) - open/close 1830 (time 1788663095.14 total 103.49 last 18.56) - open/close 1960 (time 1788663105.17 total 113.52 last 12.96) - open/close 2168 (time 1788663115.94 total 124.29 last 19.32) - open/close 2377 (time 1788663125.94 total 134.29 last 20.89) - open/close 2616 (time 1788663136.54 total 144.89 last 22.55) - open/close 2751 (time 1788663146.54 total 154.89 last 13.50) - open/close 3023 (time 1788663156.78 total 165.14 last 26.55) - open/close 3195 (time 1788663166.79 total 175.14 last 17.20) - open/close 3338 (time 1788663176.93 total 185.28 last 14.10) - open/close 3534 (time 1788663186.94 total 195.29 last 19.57) - open/close 3743 (time 1788663197.08 total 205.43 last 20.61) - open/close 3889 (time 1788663207.08 total 215.43 last 14.60) - open/close 4058 (time 1788663217.09 total 225.44 last 16.89) - open/close 4273 (time 1788663227.10 total 235.45 last 21.48) - open/close 4524 (time 1788663237.74 total 246.09 last 23.59) - open/close 4822 (time 1788663248.55 total 256.90 last 27.57) - open/close 5117 (time 1788663258.56 total 266.91 last 29.47) - open/close 5380 (time 1788663269.03 total 277.39 last 25.10) - open/close 5615 (time 1788663279.04 total 287.39 last 23.49) - open/close 5850 (time 1788663289.04 total 297.40 last 23.49) - open/close 6112 (time 1788663299.94 total 308.29 last 24.05) - open/close 6385 (time 1788663310.40 total 318.76 last 26.09) - open/close 6641 (time 1788663321.13 total 329.48 last 23.87) - open/close 6892 (time 1788663331.14 total 339.49 last 25.06) - open/close 7150 (time 1788663341.15 total 349.50 last 25.78) - open/close 7320 (time 1788663351.79 total 360.14 last 15.98) - open/close 7574 (time 1788663362.35 total 370.70 last 24.05) - open/close 7772 (time 1788663373.01 total 381.36 last 18.57) - open/close 8032 (time 1788663383.84 total 392.20 last 24.00) - open/close 8239 (time 1788663393.85 total 402.21 last 20.68) - open/close 8478 (time 1788663405.03 total 413.38 last 21.39) - open/close 8780 (time 1788663415.07 total 423.42 last 30.07) - open/close 9001 (time 1788663425.28 total 433.63 last 21.66) - open/close 9281 (time 1788663436.32 total 444.67 last 25.35) - open/close 9567 (time 1788663447.32 total 455.68 last 26.00) - open/close 9810 (time 1788663457.51 total 465.87 last 23.85) - open/close 9977 (time 1788663468.52 total 476.87 last 15.18) - open/close 10165 (time 1788663478.87 total 487.22 last 18.16) - open/close 10268 (time 1788663488.93 total 497.28 last 10.24) - open/close 10504 (time 1788663499.86 total 508.22 last 21.58) - open/close 10752 (time 1788663509.99 total 518.34 last 24.49) - open/close 10974 (time 1788663520.01 total 528.36 last 22.16) - open/close 11190 (time 1788663530.03 total 538.38 last 21.56) - open/close 11429 (time 1788663540.04 total 548.39 last 23.88) - open/close 11622 (time 1788663550.63 total 558.99 last 18.21) - open/close 11785 (time 1788663560.69 total 569.04 last 16.21) - open/close 11988 (time 1788663570.69 total 579.04 last 20.30) - open/close 12254 (time 1788663580.73 total 589.08 last 26.50) - open/close 12393 (time 1788663591.42 total 599.77 last 13.00) - open/close 12683 (time 1788663601.44 total 609.80 last 28.93) - open/close 12997 (time 1788663612.52 total 620.88 last 28.34) - open/close 13298 (time 1788663622.53 total 630.88 last 30.09) - open/close 13562 (time 1788663633.12 total 641.47 last 24.92) - open/close 13837 (time 1788663643.93 total 652.28 last 25.45) - open/close 14090 (time 1788663654.01 total 662.36 last 25.09) - open/close 14345 (time 1788663664.29 total 672.65 last 24.79) - open/close 14633 (time 1788663674.92 total 683.27 last 27.11) - open/close 14900 (time 1788663684.92 total 693.27 last 26.70) - open/close 15152 (time 1788663695.09 total 703.45 last 24.77) - open/close 15372 (time 1788663705.84 total 714.19 last 20.47) - open/close 15635 (time 1788663715.84 total 724.20 last 26.29) - open/close 15745 (time 1788663725.86 total 734.21 last 10.98) - open/close 15943 (time 1788663736.15 total 744.51 last 19.24) - open/close 16220 (time 1788663746.84 total 755.19 last 25.92) - open/close 16505 (time 1788663759.87 total 768.22 last 21.87) - open/close 16726 (time 1788663769.91 total 778.26 last 22.02) - open/close 16925 (time 1788663779.91 total 788.26 last 19.90) - open/close 17039 (time 1788663790.18 total 798.54 last 11.09) - open/close 17222 (time 1788663800.19 total 808.54 last 18.30) - open/close 17440 (time 1788663810.26 total 818.62 last 21.63) - open/close 17647 (time 1788663820.27 total 828.62 last 20.69) - open/close 17852 (time 1788663830.37 total 838.73 last 20.28) - open/close 18128 (time 1788663840.38 total 848.73 last 27.59) - open/close 18369 (time 1788663850.58 total 858.94 last 23.62) - open/close 18646 (time 1788663863.65 total 872.00 last 21.20) - open/close 18914 (time 1788663874.46 total 882.82 last 24.78) - open/close 19160 (time 1788663884.53 total 892.89 last 24.43) - open/close 19403 (time 1788663895.23 total 903.59 last 22.71) - open/close 19650 (time 1788663905.24 total 913.59 last 24.69) - open/close 19901 (time 1788663915.24 total 923.59 last 25.10) - open/close 20000 (time 1788663918.87 total 927.22 last 27.25) - open/close 20276 (time 1788663929.45 total 937.80 last 26.10) - open/close 21063 (time 1788663939.46 total 947.81 last 78.60) - open/close 21897 (time 1788663949.46 total 957.81 last 83.39) - open/close 22727 (time 1788663959.46 total 967.82 last 82.98) - open/close 23622 (time 1788663969.47 total 977.82 last 89.45) - open/close 24522 (time 1788663979.48 total 987.83 last 89.90) - open/close 25421 (time 1788663989.49 total 997.84 last 89.85) - open/close 26264 (time 1788663999.49 total 1007.85 last 84.24) - open/close 27159 (time 1788664009.50 total 1017.85 last 89.45) - open/close 27988 (time 1788664019.50 total 1027.85 last 82.89) - open/close 28780 (time 1788664029.50 total 1037.86 last 79.16) - open/close 29508 (time 1788664039.52 total 1047.87 last 72.71) - open/close 30000 (time 1788664046.41 total 1054.76 last 71.36) - open/close 30843 (time 1788664056.43 total 1064.78 last 84.17) - open/close 31493 (time 1788664066.43 total 1074.79 last 64.96) - open/close 32204 (time 1788664076.45 total 1084.81 last 70.96) - open/close 32924 (time 1788664086.46 total 1094.81 last 71.95) - open/close 33640 (time 1788664096.47 total 1104.82 last 71.52) - open/close 34398 (time 1788664106.48 total 1114.84 last 75.70) - open/close 35207 (time 1788664116.49 total 1124.84 last 80.87) - open/close 36000 (time 1788664126.50 total 1134.85 last 79.20) - open/close 36789 (time 1788664136.51 total 1144.86 last 78.83) - open/close 37576 (time 1788664146.52 total 1154.87 last 78.66) - open/close 38418 (time 1788664156.52 total 1164.87 last 84.17) - open/close 39185 (time 1788664166.52 total 1174.88 last 76.67) - open/close 39994 (time 1788664176.52 total 1184.88 last 80.89) - open/close 40858 (time 1788664186.53 total 1194.88 last 86.33) - open/close 41670 (time 1788664196.54 total 1204.89 last 81.15) - open/close 42475 (time 1788664206.54 total 1214.89 last 80.49) - open/close 43294 (time 1788664216.55 total 1224.90 last 81.84) - open/close 44140 (time 1788664226.55 total 1234.90 last 84.56) - open/close 45023 (time 1788664236.55 total 1244.91 last 88.28) - open/close 45787 (time 1788664246.56 total 1254.91 last 76.37) - open/close 46543 (time 1788664256.57 total 1264.92 last 75.50) - open/close 47224 (time 1788664266.58 total 1274.93 last 68.04) - open/close 48057 (time 1788664276.58 total 1284.94 last 83.28) - open/close 48859 (time 1788664286.59 total 1294.94 last 80.13) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1302.79 seconds: 38.10 ops/second - unlinked 0 (time 1788664300 ; total 0 ; last 0) - unlinked 10000 (time 1788664423 ; total 123 ; last 123) - unlinked 20000 (time 1788664551 ; total 251 ; last 128) - unlinked 30000 (time 1788664678 ; total 378 ; last 127) - unlinked 40000 (time 1788664802 ; total 502 ; last 124) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 625 seconds: 79.409599 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg637-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg637-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg637-server stop mds service on oleg637-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg637-server stop mds service on oleg637-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg637-server unloading modules via unload_modules_local on: 'oleg637-server' oleg637-server: oleg637-server.virtnet: executing unload_modules_local oleg637-server: kmemleak disabled modules unloaded. pdsh@oleg637-client: oleg637-client: ssh exited with exit code 2 pdsh@oleg637-client: oleg637-server: ssh exited with exit code 2 pdsh@oleg637-client: oleg637-client: ssh exited with exit code 2 pdsh@oleg637-client: oleg637-server: ssh exited with exit code 2