== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:35:58 (1788428158) start mds service on oleg201-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921464 Sep 3 03:21 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing load_modules_local oleg201-server: kmemleak disabled oleg201-server: Loading modules from /home/green/git/lustre-release/lustre oleg201-server: detected 4 online CPUs by sysfs oleg201-server: MODOPTS_LIBCFS= oleg201-server: Force libcfs to create 2 CPU partitions oleg201-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg201-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg201-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg201-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg201-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg201-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 204 (time 1788428282.95 total 10.80 last 18.89) - open/close 307 (time 1788428294.38 total 22.24 last 9.01) - open/close 524 (time 1788428304.40 total 32.26 last 21.66) - open/close 626 (time 1788428314.42 total 42.28 last 10.18) - open/close 717 (time 1788428327.42 total 55.28 last 7.00) - open/close 875 (time 1788428337.43 total 65.28 last 15.79) - open/close 1044 (time 1788428348.32 total 76.17 last 15.53) - open/close 1179 (time 1788428358.34 total 86.19 last 13.47) - open/close 1281 (time 1788428370.57 total 98.42 last 8.34) - open/close 1481 (time 1788428380.57 total 108.43 last 20.00) - open/close 1594 (time 1788428390.68 total 118.53 last 11.19) - open/close 1779 (time 1788428400.68 total 128.53 last 18.49) - open/close 1910 (time 1788428411.10 total 138.96 last 12.57) - open/close 1957 (time 1788428421.64 total 149.49 last 4.46) - open/close 2062 (time 1788428431.65 total 159.50 last 10.49) - open/close 2208 (time 1788428441.76 total 169.61 last 14.44) - open/close 2417 (time 1788428452.33 total 180.18 last 19.77) - open/close 2526 (time 1788428463.18 total 191.04 last 10.05) - open/close 2657 (time 1788428473.74 total 201.60 last 12.40) - open/close 2774 (time 1788428484.49 total 212.34 last 10.89) - open/close 2902 (time 1788428494.51 total 222.36 last 12.78) - open/close 3046 (time 1788428504.91 total 232.77 last 13.84) - open/close 3146 (time 1788428515.01 total 242.87 last 9.90) - open/close 3285 (time 1788428525.64 total 253.49 last 13.08) - open/close 3446 (time 1788428535.64 total 263.49 last 16.10) - open/close 3603 (time 1788428546.56 total 274.42 last 14.38) - open/close 3733 (time 1788428556.71 total 284.56 last 12.81) - open/close 3976 (time 1788428566.93 total 294.78 last 23.77) - open/close 4043 (time 1788428576.98 total 304.83 last 6.67) - open/close 4215 (time 1788428587.92 total 315.78 last 15.71) - open/close 4353 (time 1788428597.93 total 325.79 last 13.79) - open/close 4517 (time 1788428609.55 total 337.40 last 14.12) - open/close 4630 (time 1788428620.32 total 348.17 last 10.49) - open/close 4863 (time 1788428631.18 total 359.03 last 21.46) - open/close 5116 (time 1788428641.19 total 369.04 last 25.28) - open/close 5345 (time 1788428653.97 total 381.82 last 17.91) - open/close 5591 (time 1788428663.98 total 391.84 last 24.57) - open/close 5848 (time 1788428674.89 total 402.75 last 23.55) - open/close 6122 (time 1788428684.92 total 412.77 last 27.32) - open/close 6234 (time 1788428696.46 total 424.31 last 9.71) - open/close 6462 (time 1788428706.46 total 434.31 last 22.80) - open/close 6668 (time 1788428716.47 total 444.32 last 20.58) - open/close 6938 (time 1788428726.47 total 454.32 last 27.00) - open/close 7158 (time 1788428736.48 total 464.33 last 21.98) - open/close 7336 (time 1788428746.48 total 474.33 last 17.79) - open/close 7527 (time 1788428756.71 total 484.56 last 18.68) - open/close 7771 (time 1788428767.06 total 494.91 last 23.57) - open/close 7999 (time 1788428777.56 total 505.41 last 21.71) - open/close 8190 (time 1788428788.37 total 516.22 last 17.68) - open/close 8411 (time 1788428798.61 total 526.47 last 21.57) - open/close 8621 (time 1788428808.84 total 536.70 last 20.53) - open/close 8754 (time 1788428819.04 total 546.89 last 13.05) - open/close 8865 (time 1788428829.06 total 556.91 last 11.08) - open/close 8958 (time 1788428839.74 total 567.59 last 8.71) - open/close 9238 (time 1788428850.51 total 578.37 last 25.98) - open/close 9451 (time 1788428860.52 total 588.38 last 21.28) - open/close 9633 (time 1788428870.53 total 598.38 last 18.19) - open/close 9731 (time 1788428880.56 total 608.41 last 9.77) - open/close 9833 (time 1788428890.79 total 618.64 last 9.97) - open/close 10000 (time 1788428897.69 total 625.54 last 24.20) - open/close 10185 (time 1788428907.69 total 635.54 last 18.49) - open/close 10387 (time 1788428918.23 total 646.09 last 19.16) - open/close 10630 (time 1788428928.32 total 656.17 last 24.09) - open/close 10799 (time 1788428939.77 total 667.63 last 14.76) - open/close 10918 (time 1788428952.84 total 680.69 last 9.11) - open/close 11136 (time 1788428963.68 total 691.54 last 20.11) - open/close 11380 (time 1788428973.69 total 701.55 last 24.37) - open/close 11573 (time 1788428983.70 total 711.56 last 19.28) - open/close 11756 (time 1788428993.70 total 721.56 last 18.30) - open/close 11922 (time 1788429004.23 total 732.08 last 15.78) - open/close 12120 (time 1788429014.23 total 742.09 last 19.78) - open/close 12374 (time 1788429024.25 total 752.11 last 25.35) - open/close 12574 (time 1788429034.27 total 762.12 last 19.97) - open/close 12791 (time 1788429044.28 total 772.13 last 21.67) - open/close 12920 (time 1788429055.95 total 783.81 last 11.05) - open/close 13060 (time 1788429066.01 total 793.86 last 13.92) - open/close 13234 (time 1788429076.03 total 803.89 last 17.36) - open/close 13337 (time 1788429086.45 total 814.30 last 9.89) - open/close 13420 (time 1788429098.76 total 826.61 last 6.74) - open/close 13631 (time 1788429109.51 total 837.37 last 19.62) - open/close 13793 (time 1788429120.14 total 847.99 last 15.25) - open/close 13938 (time 1788429130.18 total 858.03 last 14.44) - open/close 14125 (time 1788429140.19 total 868.05 last 18.68) - open/close 14337 (time 1788429150.99 total 878.84 last 19.64) - open/close 14526 (time 1788429160.99 total 888.84 last 18.90) - open/close 14732 (time 1788429171.00 total 898.85 last 20.58) - open/close 14872 (time 1788429181.00 total 908.86 last 13.99) - open/close 15068 (time 1788429191.91 total 919.76 last 17.98) - open/close 15215 (time 1788429201.92 total 929.77 last 14.68) - open/close 15333 (time 1788429213.80 total 941.65 last 9.93) - open/close 15463 (time 1788429223.82 total 951.67 last 12.97) - open/close 15621 (time 1788429234.21 total 962.06 last 15.21) - open/close 15860 (time 1788429244.21 total 972.06 last 23.90) - open/close 16123 (time 1788429254.93 total 982.78 last 24.54) - open/close 16355 (time 1788429265.74 total 993.59 last 21.46) - open/close 16565 (time 1788429276.05 total 1003.91 last 20.36) - open/close 16770 (time 1788429286.06 total 1013.91 last 20.48) - open/close 16865 (time 1788429296.45 total 1024.31 last 9.14) - open/close 17011 (time 1788429307.36 total 1035.21 last 13.39) - open/close 17187 (time 1788429318.11 total 1045.96 last 16.37) - open/close 17323 (time 1788429328.13 total 1055.98 last 13.58) - open/close 17519 (time 1788429338.14 total 1066.00 last 19.57) - open/close 17898 (time 1788429348.14 total 1076.00 last 37.89) - open/close 18682 (time 1788429358.15 total 1086.00 last 78.39) - open/close 19426 (time 1788429368.17 total 1096.02 last 74.21) - open/close 20000 (time 1788429375.88 total 1103.73 last 74.46) - open/close 20838 (time 1788429385.88 total 1113.73 last 83.80) - open/close 21679 (time 1788429395.89 total 1123.75 last 84.00) - open/close 22587 (time 1788429405.90 total 1133.75 last 90.78) - open/close 23365 (time 1788429415.91 total 1143.76 last 77.72) - open/close 24134 (time 1788429425.91 total 1153.76 last 76.86) - open/close 24822 (time 1788429435.91 total 1163.77 last 68.79) - open/close 25578 (time 1788429445.92 total 1173.77 last 75.54) - open/close 26355 (time 1788429455.92 total 1183.78 last 77.68) - open/close 27016 (time 1788429465.94 total 1193.79 last 66.01) - open/close 27701 (time 1788429475.94 total 1203.79 last 68.48) - open/close 28364 (time 1788429485.94 total 1213.79 last 66.30) - open/close 29036 (time 1788429495.98 total 1223.83 last 66.95) - open/close 29810 (time 1788429505.99 total 1233.84 last 77.35) - open/close 30000 (time 1788429508.82 total 1236.67 last 67.11) - open/close 30952 (time 1788429518.82 total 1246.68 last 95.13) - open/close 31672 (time 1788429528.83 total 1256.68 last 71.96) - open/close 32462 (time 1788429538.85 total 1266.70 last 78.88) - open/close 33306 (time 1788429548.85 total 1276.71 last 84.33) - open/close 34117 (time 1788429558.86 total 1286.71 last 81.04) - open/close 35042 (time 1788429568.87 total 1296.72 last 92.42) - open/close 35861 (time 1788429578.87 total 1306.72 last 81.89) - open/close 36574 (time 1788429588.88 total 1316.73 last 71.26) - open/close 37302 (time 1788429598.88 total 1326.74 last 72.75) - open/close 38018 (time 1788429608.89 total 1336.74 last 71.59) - open/close 38810 (time 1788429618.89 total 1346.75 last 79.13) - open/close 39438 (time 1788429628.91 total 1356.77 last 62.69) - open/close 40000 (time 1788429636.35 total 1364.21 last 75.54) - open/close 40774 (time 1788429646.35 total 1374.21 last 77.38) - open/close 41694 (time 1788429656.37 total 1384.22 last 91.90) - open/close 42396 (time 1788429666.37 total 1394.22 last 70.18) - open/close 42972 (time 1788429676.38 total 1404.23 last 57.53) - open/close 43590 (time 1788429686.40 total 1414.25 last 61.68) - open/close 44199 (time 1788429696.42 total 1424.27 last 60.80) - open/close 44910 (time 1788429706.43 total 1434.28 last 70.99) - open/close 45709 (time 1788429716.43 total 1444.29 last 79.88) - open/close 46523 (time 1788429726.44 total 1454.29 last 81.36) - open/close 47086 (time 1788429736.44 total 1464.30 last 56.27) - open/close 47750 (time 1788429746.46 total 1474.31 last 66.32) - open/close 48561 (time 1788429756.46 total 1484.32 last 81.05) - open/close 49401 (time 1788429766.47 total 1494.32 last 83.94) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1496.70 seconds: 33.16 ops/second - unlinked 0 (time 1788429774 ; total 0 ; last 0) - unlinked 10000 (time 1788429907 ; total 133 ; last 133) - unlinked 20000 (time 1788430039 ; total 265 ; last 132) - unlinked 30000 (time 1788430176 ; total 402 ; last 137) - unlinked 40000 (time 1788430312 ; total 538 ; last 136) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 664 seconds: 74.745483 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg201-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server stop mds service on oleg201-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server stop mds service on oleg201-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg201-server unloading modules via unload_modules_local on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing unload_modules_local oleg201-server: kmemleak disabled modules unloaded. pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2