== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:39:25 (1789047565) start mds service on oleg350-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 10 00:34 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: kmemleak disabled oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg350-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 226 (time 1789047666.08 total 10.01 last 22.59) - open/close 425 (time 1789047676.36 total 20.29 last 19.36) - open/close 704 (time 1789047686.84 total 30.77 last 26.60) - open/close 983 (time 1789047697.86 total 41.79 last 25.33) - open/close 1276 (time 1789047708.54 total 52.47 last 27.42) - open/close 1582 (time 1789047719.43 total 63.36 last 28.11) - open/close 1846 (time 1789047729.45 total 73.38 last 26.35) - open/close 2105 (time 1789047739.46 total 83.39 last 25.88) - open/close 2333 (time 1789047749.63 total 93.56 last 22.41) - open/close 2619 (time 1789047760.48 total 104.41 last 26.36) - open/close 2861 (time 1789047772.36 total 116.29 last 20.37) - open/close 3068 (time 1789047782.37 total 126.30 last 20.67) - open/close 3258 (time 1789047792.38 total 136.31 last 18.99) - open/close 3361 (time 1789047802.55 total 146.48 last 10.12) - open/close 3597 (time 1789047815.38 total 159.31 last 18.39) - open/close 3792 (time 1789047825.64 total 169.57 last 19.01) - open/close 4016 (time 1789047836.30 total 180.23 last 21.02) - open/close 4203 (time 1789047846.31 total 190.24 last 18.69) - open/close 4252 (time 1789047856.97 total 200.90 last 4.60) - open/close 4416 (time 1789047867.94 total 211.87 last 14.94) - open/close 4660 (time 1789047878.53 total 222.46 last 23.05) - open/close 4838 (time 1789047888.55 total 232.48 last 17.77) - open/close 5041 (time 1789047899.51 total 243.44 last 18.52) - open/close 5298 (time 1789047909.84 total 253.77 last 24.88) - open/close 5546 (time 1789047920.39 total 264.32 last 23.51) - open/close 5844 (time 1789047930.40 total 274.33 last 29.77) - open/close 6063 (time 1789047941.18 total 285.11 last 20.30) - open/close 6222 (time 1789047951.19 total 295.12 last 15.89) - open/close 6458 (time 1789047961.20 total 305.13 last 23.56) - open/close 6710 (time 1789047971.29 total 315.22 last 24.98) - open/close 6942 (time 1789047981.47 total 325.40 last 22.80) - open/close 7273 (time 1789047992.33 total 336.26 last 30.47) - open/close 7531 (time 1789048003.07 total 347.00 last 24.01) - open/close 7674 (time 1789048014.09 total 358.02 last 12.98) - open/close 7955 (time 1789048024.09 total 368.02 last 28.10) - open/close 8124 (time 1789048034.37 total 378.30 last 16.44) - open/close 8214 (time 1789048045.52 total 389.45 last 8.08) - open/close 8455 (time 1789048056.08 total 400.01 last 22.82) - open/close 8735 (time 1789048066.76 total 410.69 last 26.21) - open/close 8993 (time 1789048076.77 total 420.70 last 25.78) - open/close 9224 (time 1789048086.79 total 430.72 last 23.07) - open/close 9488 (time 1789048096.79 total 440.72 last 26.39) - open/close 9756 (time 1789048107.13 total 451.06 last 25.92) - open/close 10000 (time 1789048116.86 total 460.79 last 25.08) - open/close 10254 (time 1789048126.87 total 470.80 last 25.37) - open/close 10498 (time 1789048136.87 total 480.80 last 24.39) - open/close 10750 (time 1789048146.88 total 490.81 last 25.18) - open/close 10982 (time 1789048157.77 total 501.70 last 21.31) - open/close 11249 (time 1789048168.58 total 512.51 last 24.70) - open/close 11564 (time 1789048178.58 total 522.51 last 31.48) - open/close 11796 (time 1789048188.59 total 532.52 last 23.18) - open/close 12065 (time 1789048198.60 total 542.53 last 26.89) - open/close 12307 (time 1789048208.61 total 552.54 last 24.16) - open/close 12508 (time 1789048219.46 total 563.39 last 18.53) - open/close 12614 (time 1789048229.46 total 573.39 last 10.60) - open/close 12719 (time 1789048241.48 total 585.41 last 8.74) - open/close 12876 (time 1789048251.48 total 595.41 last 15.69) - open/close 13065 (time 1789048261.51 total 605.44 last 18.85) - open/close 13372 (time 1789048272.25 total 616.18 last 28.57) - open/close 13623 (time 1789048282.25 total 626.18 last 25.10) - open/close 13860 (time 1789048292.28 total 636.21 last 23.64) - open/close 14102 (time 1789048302.50 total 646.43 last 23.68) - open/close 14387 (time 1789048313.28 total 657.21 last 26.42) - open/close 14605 (time 1789048324.16 total 668.09 last 20.04) - open/close 14895 (time 1789048334.68 total 678.61 last 27.58) - open/close 14964 (time 1789048345.60 total 689.53 last 6.32) - open/close 15113 (time 1789048355.70 total 699.63 last 14.75) - open/close 15349 (time 1789048365.89 total 709.82 last 23.17) - open/close 15599 (time 1789048376.55 total 720.48 last 23.45) - open/close 15811 (time 1789048386.55 total 730.48 last 21.19) - open/close 16065 (time 1789048396.56 total 740.49 last 25.40) - open/close 16337 (time 1789048407.26 total 751.19 last 25.40) - open/close 16618 (time 1789048417.28 total 761.21 last 28.07) - open/close 16810 (time 1789048428.05 total 771.98 last 17.83) - open/close 16964 (time 1789048438.06 total 781.99 last 15.38) - open/close 17209 (time 1789048448.51 total 792.44 last 23.45) - open/close 17534 (time 1789048458.51 total 802.44 last 32.48) - open/close 17731 (time 1789048471.09 total 815.02 last 15.66) - open/close 18029 (time 1789048481.60 total 825.53 last 28.35) - open/close 18296 (time 1789048491.65 total 835.58 last 26.58) - open/close 18548 (time 1789048501.66 total 845.59 last 25.17) - open/close 18738 (time 1789048512.21 total 856.14 last 18.02) - open/close 19082 (time 1789048522.21 total 866.14 last 34.39) - open/close 19291 (time 1789048532.22 total 876.15 last 20.87) - open/close 19531 (time 1789048542.47 total 886.40 last 23.41) - open/close 19802 (time 1789048553.41 total 897.34 last 24.79) - open/close 20000 (time 1789048562.73 total 906.66 last 21.23) - open/close 20217 (time 1789048572.74 total 916.67 last 21.69) - open/close 20458 (time 1789048583.18 total 927.11 last 23.07) - open/close 20712 (time 1789048593.37 total 937.30 last 24.94) - open/close 21412 (time 1789048603.37 total 947.30 last 69.95) - open/close 22063 (time 1789048613.38 total 957.31 last 65.07) - open/close 22778 (time 1789048623.39 total 967.32 last 71.45) - open/close 23487 (time 1789048633.39 total 977.32 last 70.88) - open/close 24193 (time 1789048643.39 total 987.32 last 70.59) - open/close 24730 (time 1789048653.40 total 997.33 last 53.65) - open/close 25377 (time 1789048663.41 total 1007.34 last 64.64) - open/close 26136 (time 1789048673.41 total 1017.34 last 75.88) - open/close 26901 (time 1789048683.42 total 1027.35 last 76.42) - open/close 27749 (time 1789048693.44 total 1037.37 last 84.65) - open/close 28603 (time 1789048703.45 total 1047.38 last 85.28) - open/close 29407 (time 1789048713.47 total 1057.40 last 80.29) - open/close 30000 (time 1789048721.64 total 1065.57 last 72.52) - open/close 30796 (time 1789048731.65 total 1075.58 last 79.52) - open/close 31574 (time 1789048741.66 total 1085.59 last 77.72) - open/close 32394 (time 1789048751.68 total 1095.61 last 81.84) - open/close 33154 (time 1789048761.69 total 1105.62 last 75.98) - open/close 34108 (time 1789048771.69 total 1115.62 last 95.34) - open/close 34990 (time 1789048781.69 total 1125.62 last 88.19) - open/close 35813 (time 1789048791.70 total 1135.63 last 82.22) - open/close 36442 (time 1789048801.71 total 1145.64 last 62.83) - open/close 37231 (time 1789048811.74 total 1155.67 last 78.74) - open/close 38028 (time 1789048821.75 total 1165.68 last 79.60) - open/close 38866 (time 1789048831.76 total 1175.69 last 83.74) - open/close 39589 (time 1789048841.76 total 1185.69 last 72.30) - open/close 40000 (time 1789048846.64 total 1190.57 last 84.10) - open/close 40716 (time 1789048856.65 total 1200.58 last 71.57) - open/close 41467 (time 1789048866.66 total 1210.59 last 74.99) - open/close 42368 (time 1789048876.67 total 1220.60 last 90.01) - open/close 43290 (time 1789048886.69 total 1230.62 last 92.04) - open/close 44142 (time 1789048896.69 total 1240.62 last 85.19) - open/close 45010 (time 1789048906.70 total 1250.63 last 86.67) - open/close 45804 (time 1789048916.71 total 1260.64 last 79.39) - open/close 46624 (time 1789048926.71 total 1270.64 last 81.99) - open/close 47498 (time 1789048936.71 total 1280.64 last 87.37) - open/close 48190 (time 1789048946.72 total 1290.65 last 69.15) - open/close 49002 (time 1789048956.72 total 1300.65 last 81.15) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1308.58 seconds: 37.93 ops/second - unlinked 0 (time 1789048969 ; total 0 ; last 0) - unlinked 10000 (time 1789049091 ; total 122 ; last 122) - unlinked 20000 (time 1789049211 ; total 242 ; last 120) - unlinked 30000 (time 1789049328 ; total 359 ; last 117) - unlinked 40000 (time 1789049444 ; total 475 ; last 116) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 542 seconds: 91.570114 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: kmemleak disabled modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2