== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:59:51 (1788598791) start mds service on oleg429-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1924424 Sep 4 13:25 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg429-server' oleg429-server: oleg429-server.virtnet: executing load_modules_local oleg429-server: kmemleak disabled oleg429-server: Loading modules from /home/green/git/lustre-release/lustre oleg429-server: detected 4 online CPUs by sysfs oleg429-server: MODOPTS_LIBCFS= oleg429-server: Force libcfs to create 2 CPU partitions oleg429-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg429-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg429-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg429-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg429-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg429-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 159 (time 1788598917.32 total 10.34 last 15.38) - open/close 423 (time 1788598928.27 total 21.30 last 24.09) - open/close 558 (time 1788598938.29 total 31.31 last 13.48) - open/close 778 (time 1788598949.32 total 42.34 last 19.95) - open/close 828 (time 1788598959.92 total 52.94 last 4.72) - open/close 996 (time 1788598971.87 total 64.89 last 14.06) - open/close 1030 (time 1788598983.07 total 76.09 last 3.04) - open/close 1119 (time 1788598993.30 total 86.32 last 8.70) - open/close 1226 (time 1788599003.97 total 96.99 last 10.02) - open/close 1385 (time 1788599014.65 total 107.68 last 14.89) - open/close 1569 (time 1788599025.66 total 118.68 last 16.72) - open/close 1675 (time 1788599035.66 total 128.68 last 10.60) - open/close 1783 (time 1788599046.15 total 139.17 last 10.30) - open/close 1973 (time 1788599056.15 total 149.17 last 19.00) - open/close 2177 (time 1788599068.36 total 161.38 last 16.71) - open/close 2438 (time 1788599079.17 total 172.19 last 24.15) - open/close 2719 (time 1788599090.12 total 183.14 last 25.66) - open/close 2865 (time 1788599100.50 total 193.52 last 14.06) - open/close 3156 (time 1788599110.74 total 203.76 last 28.43) - open/close 3438 (time 1788599120.75 total 213.77 last 28.17) - open/close 3667 (time 1788599131.34 total 224.37 last 21.61) - open/close 3895 (time 1788599141.37 total 234.39 last 22.75) - open/close 4030 (time 1788599152.07 total 245.09 last 12.61) - open/close 4300 (time 1788599162.08 total 255.10 last 26.99) - open/close 4558 (time 1788599172.09 total 265.11 last 25.76) - open/close 4789 (time 1788599183.01 total 276.03 last 21.15) - open/close 4976 (time 1788599193.03 total 286.05 last 18.66) - open/close 5195 (time 1788599203.14 total 296.16 last 21.67) - open/close 5346 (time 1788599213.14 total 306.16 last 15.10) - open/close 5509 (time 1788599223.68 total 316.70 last 15.47) - open/close 5797 (time 1788599233.68 total 326.70 last 28.79) - open/close 5997 (time 1788599244.24 total 337.26 last 18.94) - open/close 6174 (time 1788599254.71 total 347.73 last 16.89) - open/close 6361 (time 1788599266.43 total 359.45 last 15.96) - open/close 6460 (time 1788599276.61 total 369.63 last 9.72) - open/close 6601 (time 1788599286.61 total 379.63 last 14.10) - open/close 6831 (time 1788599296.84 total 389.86 last 22.48) - open/close 7120 (time 1788599306.85 total 399.87 last 28.88) - open/close 7286 (time 1788599316.85 total 409.87 last 16.59) - open/close 7482 (time 1788599327.56 total 420.58 last 18.30) - open/close 7730 (time 1788599338.31 total 431.33 last 23.08) - open/close 7997 (time 1788599348.87 total 441.90 last 25.27) - open/close 8124 (time 1788599359.63 total 452.66 last 11.80) - open/close 8206 (time 1788599370.63 total 463.65 last 7.46) - open/close 8483 (time 1788599380.64 total 473.67 last 27.66) - open/close 8733 (time 1788599390.65 total 483.67 last 24.99) - open/close 8903 (time 1788599400.65 total 493.67 last 17.00) - open/close 9183 (time 1788599412.48 total 505.51 last 23.66) - open/close 9368 (time 1788599422.80 total 515.82 last 17.93) - open/close 9597 (time 1788599432.80 total 525.83 last 22.89) - open/close 9832 (time 1788599442.82 total 535.84 last 23.46) - open/close 10000 (time 1788599449.64 total 542.66 last 24.64) - open/close 10292 (time 1788599459.64 total 552.66 last 29.20) - open/close 10301 (time 1788599470.36 total 563.38 last 0.84) - open/close 10456 (time 1788599480.46 total 573.48 last 15.35) - open/close 10549 (time 1788599490.83 total 583.85 last 8.97) - open/close 10715 (time 1788599501.53 total 594.55 last 15.51) - open/close 10901 (time 1788599512.52 total 605.54 last 16.93) - open/close 11074 (time 1788599522.54 total 615.57 last 17.26) - open/close 11271 (time 1788599533.24 total 626.26 last 18.42) - open/close 11461 (time 1788599543.24 total 636.27 last 19.00) - open/close 11682 (time 1788599553.43 total 646.45 last 21.70) - open/close 11941 (time 1788599563.44 total 656.46 last 25.88) - open/close 12139 (time 1788599576.10 total 669.12 last 15.64) - open/close 12321 (time 1788599586.11 total 679.13 last 18.18) - open/close 12560 (time 1788599596.36 total 689.38 last 23.33) - open/close 12839 (time 1788599607.36 total 700.39 last 25.35) - open/close 13159 (time 1788599617.37 total 710.39 last 31.99) - open/close 13417 (time 1788599627.40 total 720.42 last 25.73) - open/close 13635 (time 1788599637.42 total 730.44 last 21.74) - open/close 13924 (time 1788599648.45 total 741.47 last 26.22) - open/close 14199 (time 1788599659.21 total 752.23 last 25.55) - open/close 14460 (time 1788599669.98 total 763.00 last 24.23) - open/close 14757 (time 1788599679.99 total 773.01 last 29.68) - open/close 14997 (time 1788599689.99 total 783.02 last 23.98) - open/close 15264 (time 1788599700.00 total 793.03 last 26.68) - open/close 15529 (time 1788599710.84 total 803.86 last 24.45) - open/close 15712 (time 1788599720.84 total 813.87 last 18.30) - open/close 15925 (time 1788599731.70 total 824.73 last 19.61) - open/close 16125 (time 1788599741.72 total 834.74 last 19.98) - open/close 16385 (time 1788599754.56 total 847.58 last 20.24) - open/close 16473 (time 1788599764.81 total 857.83 last 8.59) - open/close 16620 (time 1788599777.48 total 870.50 last 11.60) - open/close 16821 (time 1788599787.49 total 880.51 last 20.09) - open/close 16898 (time 1788599798.53 total 891.56 last 6.97) - open/close 17034 (time 1788599808.54 total 901.56 last 13.60) - open/close 17102 (time 1788599819.24 total 912.26 last 6.35) - open/close 17310 (time 1788599829.34 total 922.36 last 20.60) - open/close 17490 (time 1788599839.89 total 932.91 last 17.05) - open/close 17615 (time 1788599850.60 total 943.62 last 11.67) - open/close 17766 (time 1788599860.61 total 953.63 last 15.09) - open/close 17894 (time 1788599870.61 total 963.64 last 12.79) - open/close 18057 (time 1788599881.06 total 974.08 last 15.60) - open/close 18142 (time 1788599891.29 total 984.32 last 8.30) - open/close 18290 (time 1788599904.09 total 997.11 last 11.57) - open/close 18496 (time 1788599914.10 total 1007.12 last 20.57) - open/close 18685 (time 1788599924.94 total 1017.96 last 17.44) - open/close 18886 (time 1788599934.95 total 1027.97 last 20.09) - open/close 19751 (time 1788599944.95 total 1037.97 last 86.47) - open/close 20000 (time 1788599947.37 total 1040.39 last 103.12) - open/close 20951 (time 1788599957.37 total 1050.39 last 95.07) - open/close 21762 (time 1788599967.38 total 1060.40 last 81.04) - open/close 22510 (time 1788599977.41 total 1070.43 last 74.56) - open/close 23298 (time 1788599987.41 total 1080.44 last 78.76) - open/close 24008 (time 1788599997.42 total 1090.44 last 70.96) - open/close 24673 (time 1788600007.44 total 1100.46 last 66.39) - open/close 25383 (time 1788600017.45 total 1110.47 last 70.91) - open/close 26220 (time 1788600027.45 total 1120.47 last 83.70) - open/close 26986 (time 1788600037.45 total 1130.47 last 76.59) - open/close 27750 (time 1788600047.45 total 1140.47 last 76.38) - open/close 28542 (time 1788600057.46 total 1150.49 last 79.11) - open/close 29371 (time 1788600067.47 total 1160.49 last 82.83) - open/close 30000 (time 1788600075.75 total 1168.77 last 75.98) - open/close 30919 (time 1788600085.75 total 1178.77 last 91.88) - open/close 31706 (time 1788600095.76 total 1188.78 last 78.66) - open/close 32513 (time 1788600105.76 total 1198.78 last 80.65) - open/close 33367 (time 1788600115.76 total 1208.79 last 85.39) - open/close 34258 (time 1788600125.79 total 1218.81 last 88.87) - open/close 35104 (time 1788600135.80 total 1228.82 last 84.53) - open/close 35972 (time 1788600145.80 total 1238.82 last 86.79) - open/close 36835 (time 1788600155.80 total 1248.82 last 86.29) - open/close 37615 (time 1788600165.81 total 1258.83 last 77.93) - open/close 38426 (time 1788600175.82 total 1268.84 last 81.02) - open/close 39240 (time 1788600185.82 total 1278.85 last 81.35) - open/close 39997 (time 1788600195.83 total 1288.85 last 75.66) - open/close 40820 (time 1788600205.83 total 1298.85 last 82.29) - open/close 41651 (time 1788600215.84 total 1308.86 last 83.04) - open/close 42487 (time 1788600225.84 total 1318.86 last 83.56) - open/close 43362 (time 1788600235.85 total 1328.87 last 87.46) - open/close 44208 (time 1788600245.85 total 1338.87 last 84.56) - open/close 45116 (time 1788600255.86 total 1348.88 last 90.72) - open/close 45972 (time 1788600265.86 total 1358.88 last 85.59) - open/close 46814 (time 1788600275.87 total 1368.89 last 84.11) - open/close 47606 (time 1788600285.88 total 1378.90 last 79.13) - open/close 48447 (time 1788600295.89 total 1388.91 last 84.04) - open/close 49294 (time 1788600305.89 total 1398.91 last 84.66) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1402.82 seconds: 35.38 ops/second - unlinked 0 (time 1788600314 ; total 0 ; last 0) - unlinked 10000 (time 1788600431 ; total 117 ; last 117) - unlinked 20000 (time 1788600565 ; total 251 ; last 134) - unlinked 30000 (time 1788600689 ; total 375 ; last 124) - unlinked 40000 (time 1788600812 ; total 498 ; last 123) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 610 seconds: 81.362297 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg429-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg429-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg429-server stop mds service on oleg429-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg429-server stop mds service on oleg429-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg429-server unloading modules via unload_modules_local on: 'oleg429-server' oleg429-server: oleg429-server.virtnet: executing unload_modules_local oleg429-server: kmemleak disabled modules unloaded. pdsh@oleg429-client: oleg429-client: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-server: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-server: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-client: ssh exited with exit code 2