== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:32:14 (1789687934) start mds service on oleg101-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 17 16:20 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg101-server' oleg101-server: oleg101-server.virtnet: executing load_modules_local oleg101-server: kmemleak disabled oleg101-server: Loading modules from /home/green/git/lustre-release/lustre oleg101-server: detected 4 online CPUs by sysfs oleg101-server: MODOPTS_LIBCFS= oleg101-server: Force libcfs to create 2 CPU partitions oleg101-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg101-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg101-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg101-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg101-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg101-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-OST0000 oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg101-client.virtnet: -o user_xattr,flock 192.168.201.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 180 (time 1789688054.74 total 10.17 last 17.70) - open/close 386 (time 1789688065.74 total 21.17 last 18.72) - open/close 583 (time 1789688078.54 total 33.97 last 15.40) - open/close 821 (time 1789688089.24 total 44.67 last 22.24) - open/close 1051 (time 1789688099.24 total 54.67 last 22.99) - open/close 1220 (time 1789688111.79 total 67.22 last 13.47) - open/close 1338 (time 1789688124.70 total 80.13 last 9.14) - open/close 1544 (time 1789688135.25 total 90.68 last 19.53) - open/close 1744 (time 1789688147.12 total 102.55 last 16.86) - open/close 1968 (time 1789688157.40 total 112.83 last 21.79) - open/close 2240 (time 1789688170.36 total 125.79 last 20.99) - open/close 2303 (time 1789688181.39 total 136.82 last 5.71) - open/close 2527 (time 1789688193.44 total 148.87 last 18.59) - open/close 2797 (time 1789688203.67 total 159.10 last 26.39) - open/close 3065 (time 1789688213.68 total 169.11 last 26.77) - open/close 3276 (time 1789688224.64 total 180.07 last 19.25) - open/close 3497 (time 1789688234.66 total 190.09 last 22.04) - open/close 3704 (time 1789688244.68 total 200.11 last 20.68) - open/close 3869 (time 1789688254.68 total 210.11 last 16.49) - open/close 4128 (time 1789688264.68 total 220.11 last 25.89) - open/close 4356 (time 1789688275.08 total 230.51 last 21.93) - open/close 4499 (time 1789688285.41 total 240.84 last 13.85) - open/close 4777 (time 1789688295.42 total 250.85 last 27.77) - open/close 5060 (time 1789688305.44 total 260.87 last 28.23) - open/close 5279 (time 1789688315.47 total 270.90 last 21.83) - open/close 5435 (time 1789688325.74 total 281.17 last 15.20) - open/close 5563 (time 1789688335.74 total 291.17 last 12.79) - open/close 5806 (time 1789688345.77 total 301.20 last 24.24) - open/close 6087 (time 1789688355.78 total 311.21 last 28.04) - open/close 6359 (time 1789688365.79 total 321.22 last 27.17) - open/close 6595 (time 1789688375.81 total 331.24 last 23.56) - open/close 6866 (time 1789688385.82 total 341.25 last 27.08) - open/close 7024 (time 1789688396.07 total 351.50 last 15.42) - open/close 7235 (time 1789688406.96 total 362.40 last 19.36) - open/close 7507 (time 1789688419.82 total 375.25 last 21.15) - open/close 7836 (time 1789688429.83 total 385.26 last 32.87) - open/close 8138 (time 1789688440.71 total 396.14 last 27.77) - open/close 8429 (time 1789688450.72 total 406.15 last 29.08) - open/close 8703 (time 1789688460.74 total 416.17 last 27.34) - open/close 8950 (time 1789688470.95 total 426.38 last 24.19) - open/close 9210 (time 1789688481.24 total 436.67 last 25.27) - open/close 9477 (time 1789688491.55 total 446.98 last 25.89) - open/close 9786 (time 1789688501.56 total 456.99 last 30.87) - open/close 10000 (time 1789688509.73 total 465.16 last 26.19) - open/close 10259 (time 1789688519.74 total 475.17 last 25.88) - open/close 10585 (time 1789688530.09 total 485.52 last 31.49) - open/close 10842 (time 1789688540.11 total 495.54 last 25.67) - open/close 11080 (time 1789688550.12 total 505.55 last 23.77) - open/close 11314 (time 1789688560.13 total 515.56 last 23.38) - open/close 11502 (time 1789688570.84 total 526.27 last 17.56) - open/close 11671 (time 1789688581.28 total 536.71 last 16.19) - open/close 11899 (time 1789688591.65 total 547.08 last 21.98) - open/close 12146 (time 1789688603.61 total 559.04 last 20.65) - open/close 12380 (time 1789688613.62 total 569.05 last 23.37) - open/close 12651 (time 1789688624.18 total 579.61 last 25.66) - open/close 12947 (time 1789688634.65 total 590.08 last 28.29) - open/close 13204 (time 1789688647.63 total 603.06 last 19.79) - open/close 13443 (time 1789688657.65 total 613.08 last 23.87) - open/close 13620 (time 1789688667.66 total 623.09 last 17.68) - open/close 13920 (time 1789688677.67 total 633.10 last 29.95) - open/close 14099 (time 1789688687.68 total 643.11 last 17.89) - open/close 14328 (time 1789688697.94 total 653.37 last 22.33) - open/close 14529 (time 1789688708.95 total 664.38 last 18.24) - open/close 14729 (time 1789688719.92 total 675.35 last 18.23) - open/close 14930 (time 1789688730.44 total 685.88 last 19.10) - open/close 15152 (time 1789688742.94 total 698.37 last 17.77) - open/close 15381 (time 1789688753.44 total 708.87 last 21.79) - open/close 15593 (time 1789688764.50 total 719.93 last 19.18) - open/close 15770 (time 1789688774.73 total 730.16 last 17.30) - open/close 16006 (time 1789688785.11 total 740.54 last 22.75) - open/close 16223 (time 1789688795.12 total 750.55 last 21.66) - open/close 16476 (time 1789688805.12 total 760.55 last 25.30) - open/close 16720 (time 1789688815.12 total 770.56 last 24.40) - open/close 16940 (time 1789688825.14 total 780.57 last 21.97) - open/close 17156 (time 1789688835.16 total 790.59 last 21.55) - open/close 17387 (time 1789688845.84 total 801.27 last 21.63) - open/close 17635 (time 1789688856.54 total 811.97 last 23.18) - open/close 17896 (time 1789688866.56 total 821.99 last 26.05) - open/close 18104 (time 1789688876.57 total 832.00 last 20.78) - open/close 18374 (time 1789688887.45 total 842.88 last 24.82) - open/close 18645 (time 1789688898.35 total 853.78 last 24.85) - open/close 18844 (time 1789688908.94 total 864.37 last 18.80) - open/close 19095 (time 1789688921.03 total 876.46 last 20.77) - open/close 19327 (time 1789688931.03 total 886.46 last 23.19) - open/close 19580 (time 1789688941.52 total 896.95 last 24.13) - open/close 19839 (time 1789688952.28 total 907.71 last 24.08) - open/close 20000 (time 1789688957.23 total 912.66 last 32.54) - open/close 20267 (time 1789688967.23 total 922.66 last 26.69) - open/close 20493 (time 1789688977.23 total 932.66 last 22.59) - open/close 20968 (time 1789688987.24 total 942.67 last 47.45) - open/close 21928 (time 1789688997.25 total 952.68 last 95.93) - open/close 22810 (time 1789689007.26 total 962.69 last 88.15) - open/close 23805 (time 1789689017.26 total 972.69 last 99.48) - open/close 24789 (time 1789689027.27 total 982.70 last 98.32) - open/close 25772 (time 1789689037.27 total 992.70 last 98.26) - open/close 26705 (time 1789689047.28 total 1002.71 last 93.21) - open/close 27594 (time 1789689057.28 total 1012.71 last 88.88) - open/close 28601 (time 1789689067.29 total 1022.72 last 100.64) - open/close 29507 (time 1789689077.29 total 1032.72 last 90.57) - open/close 30000 (time 1789689082.24 total 1037.67 last 99.64) - open/close 30872 (time 1789689092.24 total 1047.67 last 87.20) - open/close 31670 (time 1789689102.26 total 1057.69 last 79.61) - open/close 32482 (time 1789689112.26 total 1067.69 last 81.19) - open/close 33215 (time 1789689122.27 total 1077.70 last 73.28) - open/close 34158 (time 1789689132.27 total 1087.70 last 94.28) - open/close 35083 (time 1789689142.27 total 1097.70 last 92.48) - open/close 36052 (time 1789689152.27 total 1107.70 last 96.89) - open/close 37049 (time 1789689162.29 total 1117.72 last 99.57) - open/close 37957 (time 1789689172.29 total 1127.72 last 90.73) - open/close 38750 (time 1789689182.32 total 1137.75 last 79.13) - open/close 39542 (time 1789689192.32 total 1147.75 last 79.19) - open/close 40000 (time 1789689197.77 total 1153.20 last 83.97) - open/close 40693 (time 1789689207.78 total 1163.21 last 69.24) - open/close 41482 (time 1789689217.79 total 1173.22 last 78.81) - open/close 42272 (time 1789689227.80 total 1183.23 last 78.92) - open/close 43140 (time 1789689237.80 total 1193.23 last 86.78) - open/close 44026 (time 1789689247.81 total 1203.24 last 88.53) - open/close 44821 (time 1789689257.83 total 1213.26 last 79.36) - open/close 45685 (time 1789689267.83 total 1223.26 last 86.38) - open/close 46571 (time 1789689277.83 total 1233.26 last 88.57) - open/close 47247 (time 1789689287.85 total 1243.28 last 67.51) - open/close 48111 (time 1789689297.85 total 1253.28 last 86.38) - open/close 49113 (time 1789689307.85 total 1263.28 last 100.19) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1268.82 seconds: 39.12 ops/second - unlinked 0 (time 1789689317 ; total 0 ; last 0) - unlinked 10000 (time 1789689429 ; total 112 ; last 112) - unlinked 20000 (time 1789689543 ; total 226 ; last 114) - unlinked 30000 (time 1789689604 ; total 287 ; last 61) - unlinked 40000 (time 1789689665 ; total 348 ; last 61) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 412 seconds: 120.463593 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg101-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg101-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg101-server stop mds service on oleg101-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg101-server stop mds service on oleg101-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg101-server unloading modules via unload_modules_local on: 'oleg101-server' oleg101-server: oleg101-server.virtnet: executing unload_modules_local oleg101-server: kmemleak disabled modules unloaded. pdsh@oleg101-client: oleg101-client: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-server: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-client: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-server: ssh exited with exit code 2