== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:16:08 (1788801368) start mds service on oleg360-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 7 10:22 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg360-server' oleg360-server: oleg360-server.virtnet: executing load_modules_local oleg360-server: kmemleak disabled oleg360-server: Loading modules from /home/green/git/lustre-release/lustre oleg360-server: detected 4 online CPUs by sysfs oleg360-server: MODOPTS_LIBCFS= oleg360-server: Force libcfs to create 2 CPU partitions oleg360-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg360-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg360-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg360-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg360-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg360-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-OST0000 oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 142 (time 1788801476.73 total 10.01 last 14.18) - open/close 255 (time 1788801487.43 total 20.71 last 10.56) - open/close 379 (time 1788801497.44 total 30.72 last 12.38) - open/close 450 (time 1788801508.29 total 41.58 last 6.54) - open/close 586 (time 1788801518.30 total 51.58 last 13.59) - open/close 785 (time 1788801528.31 total 61.60 last 19.87) - open/close 964 (time 1788801538.32 total 71.60 last 17.89) - open/close 1039 (time 1788801548.36 total 81.64 last 7.47) - open/close 1192 (time 1788801558.84 total 92.12 last 14.60) - open/close 1430 (time 1788801568.85 total 102.13 last 23.77) - open/close 1671 (time 1788801579.78 total 113.06 last 22.06) - open/close 1913 (time 1788801589.78 total 123.06 last 24.20) - open/close 2146 (time 1788801599.80 total 133.08 last 23.26) - open/close 2381 (time 1788801609.92 total 143.20 last 23.21) - open/close 2661 (time 1788801620.80 total 154.08 last 25.74) - open/close 2863 (time 1788801630.80 total 164.09 last 20.19) - open/close 3032 (time 1788801646.15 total 179.44 last 11.01) - open/close 3285 (time 1788801657.10 total 190.38 last 23.11) - open/close 3524 (time 1788801667.92 total 201.21 last 22.08) - open/close 3778 (time 1788801677.94 total 211.22 last 25.36) - open/close 3987 (time 1788801687.95 total 221.24 last 20.87) - open/close 4129 (time 1788801698.88 total 232.17 last 12.99) - open/close 4220 (time 1788801710.33 total 243.62 last 7.95) - open/close 4354 (time 1788801720.38 total 253.67 last 13.33) - open/close 4609 (time 1788801730.39 total 263.68 last 25.48) - open/close 4852 (time 1788801741.06 total 274.34 last 22.78) - open/close 5110 (time 1788801752.01 total 285.29 last 23.57) - open/close 5382 (time 1788801762.57 total 295.85 last 25.75) - open/close 5607 (time 1788801772.92 total 306.21 last 21.73) - open/close 5764 (time 1788801783.57 total 316.85 last 14.75) - open/close 5863 (time 1788801794.50 total 327.78 last 9.06) - open/close 6037 (time 1788801804.85 total 338.14 last 16.80) - open/close 6293 (time 1788801814.86 total 348.14 last 25.59) - open/close 6501 (time 1788801827.41 total 360.70 last 16.57) - open/close 6641 (time 1788801837.43 total 370.71 last 13.97) - open/close 6804 (time 1788801847.88 total 381.17 last 15.60) - open/close 7055 (time 1788801858.70 total 391.98 last 23.21) - open/close 7272 (time 1788801868.70 total 401.98 last 21.69) - open/close 7489 (time 1788801879.76 total 413.04 last 19.62) - open/close 7710 (time 1788801891.22 total 424.50 last 19.29) - open/close 7891 (time 1788801901.97 total 435.25 last 16.84) - open/close 8068 (time 1788801912.52 total 445.81 last 16.77) - open/close 8339 (time 1788801924.48 total 457.77 last 22.66) - open/close 8590 (time 1788801934.50 total 467.78 last 25.06) - open/close 8792 (time 1788801945.36 total 478.65 last 18.59) - open/close 8869 (time 1788801956.35 total 489.63 last 7.01) - open/close 8970 (time 1788801966.59 total 499.88 last 9.86) - open/close 9134 (time 1788801976.78 total 510.06 last 16.10) - open/close 9308 (time 1788801987.46 total 520.74 last 16.29) - open/close 9514 (time 1788801998.40 total 531.68 last 18.82) - open/close 9766 (time 1788802008.41 total 541.70 last 25.17) - open/close 10000 (time 1788802018.34 total 551.62 last 23.58) - open/close 10272 (time 1788802028.34 total 561.63 last 27.19) - open/close 10502 (time 1788802038.97 total 572.26 last 21.63) - open/close 10732 (time 1788802049.35 total 582.64 last 22.16) - open/close 11041 (time 1788802060.10 total 593.39 last 28.74) - open/close 11276 (time 1788802071.42 total 604.71 last 20.76) - open/close 11560 (time 1788802082.26 total 615.55 last 26.20) - open/close 11816 (time 1788802092.93 total 626.21 last 24.00) - open/close 12044 (time 1788802103.76 total 637.04 last 21.06) - open/close 12305 (time 1788802113.76 total 647.04 last 26.09) - open/close 12547 (time 1788802124.15 total 657.44 last 23.28) - open/close 12755 (time 1788802134.16 total 667.44 last 20.79) - open/close 13021 (time 1788802144.17 total 677.45 last 26.57) - open/close 13245 (time 1788802154.17 total 687.46 last 22.39) - open/close 13496 (time 1788802165.02 total 698.30 last 23.15) - open/close 13771 (time 1788802175.49 total 708.77 last 26.26) - open/close 14030 (time 1788802186.18 total 719.46 last 24.22) - open/close 14307 (time 1788802196.87 total 730.15 last 25.92) - open/close 14529 (time 1788802206.89 total 740.17 last 22.15) - open/close 14607 (time 1788802216.89 total 750.18 last 7.80) - open/close 14888 (time 1788802226.91 total 760.19 last 28.06) - open/close 15094 (time 1788802237.39 total 770.68 last 19.65) - open/close 15289 (time 1788802247.70 total 780.98 last 18.92) - open/close 15493 (time 1788802257.71 total 790.99 last 20.38) - open/close 15660 (time 1788802267.71 total 800.99 last 16.70) - open/close 15794 (time 1788802277.73 total 811.01 last 13.37) - open/close 16003 (time 1788802288.01 total 821.30 last 20.32) - open/close 16265 (time 1788802298.89 total 832.17 last 24.10) - open/close 16520 (time 1788802308.90 total 842.19 last 25.46) - open/close 16753 (time 1788802318.92 total 852.20 last 23.27) - open/close 17013 (time 1788802329.28 total 862.57 last 25.08) - open/close 17219 (time 1788802340.04 total 873.32 last 19.15) - open/close 17493 (time 1788802350.05 total 883.33 last 27.37) - open/close 17726 (time 1788802362.32 total 895.61 last 18.99) - open/close 17888 (time 1788802372.33 total 905.61 last 16.19) - open/close 18144 (time 1788802382.34 total 915.62 last 25.58) - open/close 18384 (time 1788802392.71 total 926.00 last 23.13) - open/close 18667 (time 1788802403.26 total 936.54 last 26.85) - open/close 18890 (time 1788802413.43 total 946.72 last 21.91) - open/close 19184 (time 1788802424.39 total 957.67 last 26.84) - open/close 19858 (time 1788802434.41 total 967.69 last 67.26) - open/close 20625 (time 1788802444.42 total 977.70 last 76.64) - open/close 21391 (time 1788802454.42 total 987.70 last 76.60) - open/close 22202 (time 1788802464.42 total 997.70 last 81.08) - open/close 22948 (time 1788802474.43 total 1007.71 last 74.55) - open/close 23647 (time 1788802484.44 total 1017.72 last 69.82) - open/close 24426 (time 1788802494.44 total 1027.72 last 77.88) - open/close 25147 (time 1788802504.45 total 1037.73 last 72.06) - open/close 25919 (time 1788802514.46 total 1047.74 last 77.13) - open/close 26747 (time 1788802524.46 total 1057.74 last 82.79) - open/close 27480 (time 1788802534.46 total 1067.74 last 73.30) - open/close 28176 (time 1788802544.46 total 1077.74 last 69.60) - open/close 28828 (time 1788802554.46 total 1087.74 last 65.18) - open/close 29566 (time 1788802564.47 total 1097.75 last 73.75) - open/close 30000 (time 1788802571.45 total 1104.74 last 62.11) - open/close 30822 (time 1788802581.47 total 1114.75 last 82.08) - open/close 31544 (time 1788802591.47 total 1124.75 last 72.19) - open/close 32302 (time 1788802601.48 total 1134.76 last 75.74) - open/close 33139 (time 1788802611.49 total 1144.77 last 83.60) - open/close 34027 (time 1788802621.49 total 1154.78 last 88.78) - open/close 34823 (time 1788802631.50 total 1164.79 last 79.54) - open/close 35599 (time 1788802641.51 total 1174.79 last 77.54) - open/close 36347 (time 1788802651.52 total 1184.81 last 74.70) - open/close 37067 (time 1788802661.53 total 1194.81 last 71.94) - open/close 37782 (time 1788802671.54 total 1204.83 last 71.41) - open/close 38466 (time 1788802681.55 total 1214.83 last 68.36) - open/close 39146 (time 1788802691.56 total 1224.84 last 67.95) - open/close 39876 (time 1788802701.57 total 1234.85 last 72.94) - open/close 40639 (time 1788802711.57 total 1244.85 last 76.26) - open/close 41322 (time 1788802721.58 total 1254.87 last 68.22) - open/close 42092 (time 1788802731.60 total 1264.88 last 76.88) - open/close 42830 (time 1788802741.61 total 1274.90 last 73.69) - open/close 43641 (time 1788802751.62 total 1284.90 last 81.04) - open/close 44402 (time 1788802761.63 total 1294.92 last 75.99) - open/close 45169 (time 1788802771.65 total 1304.94 last 76.55) - open/close 45950 (time 1788802781.66 total 1314.95 last 78.03) - open/close 46742 (time 1788802791.68 total 1324.96 last 79.09) - open/close 47478 (time 1788802801.69 total 1334.97 last 73.53) - open/close 48239 (time 1788802811.69 total 1344.97 last 76.10) - open/close 49002 (time 1788802821.70 total 1354.99 last 76.17) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1363.59 seconds: 36.40 ops/second - unlinked 0 (time 1788802834 ; total 0 ; last 0) - unlinked 10000 (time 1788802967 ; total 133 ; last 133) - unlinked 20000 (time 1788803098 ; total 264 ; last 131) - unlinked 30000 (time 1788803231 ; total 397 ; last 133) - unlinked 40000 (time 1788803360 ; total 526 ; last 129) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 653 seconds: 76.004593 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg360-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg360-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg360-server stop mds service on oleg360-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg360-server stop mds service on oleg360-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg360-server unloading modules via unload_modules_local on: 'oleg360-server' oleg360-server: oleg360-server.virtnet: executing unload_modules_local oleg360-server: kmemleak disabled modules unloaded. pdsh@oleg360-client: oleg360-client: ssh exited with exit code 2 pdsh@oleg360-client: oleg360-server: ssh exited with exit code 2 pdsh@oleg360-client: oleg360-client: ssh exited with exit code 2 pdsh@oleg360-client: oleg360-server: ssh exited with exit code 2