== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:35:50 (1790739350) start mds service on oleg145-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1947032 Sep 29 13:13 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg145-server' oleg145-server: oleg145-server.virtnet: executing load_modules_local oleg145-server: kmemleak disabled oleg145-server: Loading modules from /home/green/git/lustre-release/lustre oleg145-server: detected 4 online CPUs by sysfs oleg145-server: MODOPTS_LIBCFS= oleg145-server: Force libcfs to create 2 CPU partitions oleg145-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg145-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg145-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg145-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg145-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg145-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-OST0000 oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg145-client.virtnet: -o user_xattr,flock 192.168.201.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 175 (time 1790739462.43 total 10.31 last 16.97) - open/close 447 (time 1790739472.44 total 20.31 last 27.19) - open/close 573 (time 1790739484.45 total 32.33 last 10.49) - open/close 758 (time 1790739494.75 total 42.62 last 17.97) - open/close 996 (time 1790739504.75 total 52.63 last 23.79) - open/close 1243 (time 1790739514.76 total 62.63 last 24.69) - open/close 1435 (time 1790739524.90 total 72.77 last 18.93) - open/close 1696 (time 1790739534.90 total 82.78 last 26.08) - open/close 1985 (time 1790739545.74 total 93.62 last 26.67) - open/close 2169 (time 1790739555.76 total 103.64 last 18.36) - open/close 2274 (time 1790739565.78 total 113.66 last 10.48) - open/close 2522 (time 1790739576.42 total 124.30 last 23.30) - open/close 2758 (time 1790739586.45 total 134.33 last 23.54) - open/close 2926 (time 1790739597.47 total 145.35 last 15.25) - open/close 3255 (time 1790739607.57 total 155.45 last 32.58) - open/close 3504 (time 1790739617.98 total 165.86 last 23.91) - open/close 3774 (time 1790739628.00 total 175.87 last 26.97) - open/close 4033 (time 1790739638.87 total 186.75 last 23.82) - open/close 4294 (time 1790739650.73 total 198.60 last 22.01) - open/close 4466 (time 1790739661.56 total 209.44 last 15.88) - open/close 4635 (time 1790739672.47 total 220.34 last 15.50) - open/close 4915 (time 1790739682.48 total 230.36 last 27.95) - open/close 5143 (time 1790739693.41 total 241.28 last 20.87) - open/close 5417 (time 1790739703.41 total 251.28 last 27.40) - open/close 5644 (time 1790739713.43 total 261.31 last 22.65) - open/close 5900 (time 1790739724.05 total 271.93 last 24.11) - open/close 6091 (time 1790739734.75 total 282.63 last 17.85) - open/close 6338 (time 1790739745.45 total 293.33 last 23.08) - open/close 6568 (time 1790739755.48 total 303.35 last 22.94) - open/close 6790 (time 1790739765.50 total 313.37 last 22.16) - open/close 7065 (time 1790739775.51 total 323.39 last 27.45) - open/close 7294 (time 1790739785.95 total 333.82 last 21.95) - open/close 7477 (time 1790739796.43 total 344.31 last 17.45) - open/close 7742 (time 1790739806.68 total 354.56 last 25.85) - open/close 7962 (time 1790739817.55 total 365.43 last 20.24) - open/close 8199 (time 1790739827.55 total 375.43 last 23.70) - open/close 8447 (time 1790739837.81 total 385.68 last 24.19) - open/close 8685 (time 1790739847.81 total 395.69 last 23.78) - open/close 8963 (time 1790739858.75 total 406.62 last 25.43) - open/close 9260 (time 1790739868.77 total 416.64 last 29.64) - open/close 9386 (time 1790739879.60 total 427.48 last 11.63) - open/close 9535 (time 1790739890.28 total 438.16 last 13.95) - open/close 9621 (time 1790739901.21 total 449.09 last 7.87) - open/close 9805 (time 1790739911.50 total 459.38 last 17.88) - open/close 9932 (time 1790739922.26 total 470.13 last 11.81) - open/close 10000 (time 1790739924.46 total 472.33 last 30.92) - open/close 10211 (time 1790739934.63 total 482.50 last 20.75) - open/close 10468 (time 1790739944.66 total 492.54 last 25.62) - open/close 10645 (time 1790739954.67 total 502.55 last 17.67) - open/close 10859 (time 1790739964.68 total 512.55 last 21.40) - open/close 11082 (time 1790739974.68 total 522.56 last 22.29) - open/close 11337 (time 1790739985.16 total 533.03 last 24.34) - open/close 11529 (time 1790739995.47 total 543.34 last 18.62) - open/close 11742 (time 1790740005.47 total 553.35 last 21.28) - open/close 11965 (time 1790740015.61 total 563.48 last 22.01) - open/close 12093 (time 1790740026.45 total 574.33 last 11.81) - open/close 12262 (time 1790740036.46 total 584.33 last 16.89) - open/close 12391 (time 1790740046.47 total 594.35 last 12.88) - open/close 12506 (time 1790740056.48 total 604.36 last 11.49) - open/close 12743 (time 1790740066.48 total 614.36 last 23.69) - open/close 12968 (time 1790740078.66 total 626.54 last 18.47) - open/close 13268 (time 1790740088.67 total 636.55 last 29.98) - open/close 13526 (time 1790740098.97 total 646.85 last 25.05) - open/close 13781 (time 1790740109.58 total 657.46 last 24.03) - open/close 13961 (time 1790740119.83 total 667.71 last 17.56) - open/close 14207 (time 1790740129.90 total 677.78 last 24.43) - open/close 14503 (time 1790740139.92 total 687.79 last 29.56) - open/close 14679 (time 1790740150.24 total 698.11 last 17.05) - open/close 14881 (time 1790740160.72 total 708.60 last 19.26) - open/close 15115 (time 1790740171.49 total 719.36 last 21.74) - open/close 15385 (time 1790740181.65 total 729.53 last 26.57) - open/close 15662 (time 1790740192.67 total 740.55 last 25.13) - open/close 15861 (time 1790740202.67 total 750.55 last 19.89) - open/close 16109 (time 1790740213.05 total 760.92 last 23.91) - open/close 16307 (time 1790740223.06 total 770.93 last 19.78) - open/close 16446 (time 1790740233.07 total 780.95 last 13.88) - open/close 16566 (time 1790740244.06 total 791.94 last 10.92) - open/close 16797 (time 1790740254.11 total 801.99 last 23.00) - open/close 17033 (time 1790740264.27 total 812.15 last 23.22) - open/close 17336 (time 1790740275.02 total 822.90 last 28.19) - open/close 17600 (time 1790740285.03 total 832.91 last 26.38) - open/close 17858 (time 1790740295.04 total 842.92 last 25.78) - open/close 18018 (time 1790740305.06 total 852.93 last 15.97) - open/close 18215 (time 1790740315.07 total 862.95 last 19.68) - open/close 18487 (time 1790740325.07 total 872.95 last 27.19) - open/close 18726 (time 1790740335.59 total 883.46 last 22.73) - open/close 19024 (time 1790740345.59 total 893.47 last 29.79) - open/close 19264 (time 1790740355.93 total 903.81 last 23.21) - open/close 19418 (time 1790740368.30 total 916.17 last 12.46) - open/close 19594 (time 1790740380.17 total 928.05 last 14.82) - open/close 19673 (time 1790740390.49 total 938.36 last 7.66) - open/close 19835 (time 1790740402.89 total 950.76 last 13.06) - open/close 19922 (time 1790740413.53 total 961.41 last 8.17) - open/close 20940 (time 1790740423.54 total 971.41 last 101.76) - open/close 22016 (time 1790740433.54 total 981.41 last 107.58) - open/close 22837 (time 1790740443.55 total 991.42 last 82.04) - open/close 23526 (time 1790740453.55 total 1001.43 last 68.85) - open/close 24256 (time 1790740463.56 total 1011.43 last 72.98) - open/close 24996 (time 1790740473.56 total 1021.43 last 74.00) - open/close 25835 (time 1790740483.56 total 1031.44 last 83.86) - open/close 26825 (time 1790740493.57 total 1041.45 last 98.88) - open/close 27908 (time 1790740503.58 total 1051.45 last 108.26) - open/close 28945 (time 1790740513.58 total 1061.46 last 103.66) - open/close 29986 (time 1790740523.58 total 1071.46 last 104.08) - open/close 30970 (time 1790740533.59 total 1081.46 last 98.37) - open/close 31914 (time 1790740543.59 total 1091.47 last 94.32) - open/close 32897 (time 1790740553.60 total 1101.48 last 98.22) - open/close 33734 (time 1790740563.61 total 1111.49 last 83.63) - open/close 34655 (time 1790740573.61 total 1121.49 last 92.07) - open/close 35594 (time 1790740583.62 total 1131.50 last 93.83) - open/close 36645 (time 1790740593.62 total 1141.50 last 105.07) - open/close 37617 (time 1790740603.63 total 1151.51 last 97.15) - open/close 38592 (time 1790740613.64 total 1161.52 last 97.40) - open/close 39553 (time 1790740623.64 total 1171.52 last 96.06) - open/close 40000 (time 1790740629.01 total 1176.89 last 83.26) - open/close 40931 (time 1790740639.01 total 1186.89 last 93.08) - open/close 41873 (time 1790740649.02 total 1196.90 last 94.13) - open/close 42815 (time 1790740659.03 total 1206.90 last 94.16) - open/close 43700 (time 1790740669.04 total 1216.91 last 88.40) - open/close 44538 (time 1790740679.04 total 1226.91 last 83.80) - open/close 45401 (time 1790740689.04 total 1236.92 last 86.25) - open/close 46378 (time 1790740699.05 total 1246.92 last 97.67) - open/close 47306 (time 1790740709.05 total 1256.93 last 92.77) - open/close 48312 (time 1790740719.06 total 1266.94 last 100.49) - open/close 49251 (time 1790740729.07 total 1276.95 last 93.83) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1280.40 seconds: 38.76 ops/second - unlinked 0 (time 1790740736 ; total 0 ; last 0) - unlinked 10000 (time 1790740850 ; total 114 ; last 114) - unlinked 20000 (time 1790740961 ; total 225 ; last 111) - unlinked 30000 (time 1790741023 ; total 287 ; last 62) - unlinked 40000 (time 1790741087 ; total 351 ; last 64) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 409 seconds: 121.347191 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg145-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg145-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server stop mds service on oleg145-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg145-server stop mds service on oleg145-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg145-server unloading modules via unload_modules_local on: 'oleg145-server' oleg145-server: oleg145-server.virtnet: executing unload_modules_local oleg145-server: kmemleak disabled modules unloaded. pdsh@oleg145-client: oleg145-client: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-client: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2