== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:37:44 (1789547864) start mds service on oleg248-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 16 03:17 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg248-server' oleg248-server: oleg248-server.virtnet: executing load_modules_local oleg248-server: kmemleak disabled oleg248-server: Loading modules from /home/green/git/lustre-release/lustre oleg248-server: detected 4 online CPUs by sysfs oleg248-server: MODOPTS_LIBCFS= oleg248-server: Force libcfs to create 2 CPU partitions oleg248-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg248-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg248-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg248-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg248-server: oleg248-server.virtnet: executing set_default_debug -1 all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg248-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg248-server: oleg248-server.virtnet: executing set_default_debug -1 all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg248-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug -1 all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0000 oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 152 (time 1789547932.32 total 10.05 last 15.13) - open/close 301 (time 1789547942.37 total 20.09 last 14.83) - open/close 369 (time 1789547952.61 total 30.33 last 6.65) - open/close 507 (time 1789547963.62 total 41.34 last 12.53) - open/close 651 (time 1789547974.50 total 52.22 last 13.24) - open/close 758 (time 1789547984.95 total 62.67 last 10.23) - open/close 912 (time 1789547995.95 total 73.67 last 14.01) - open/close 1056 (time 1789548005.96 total 83.68 last 14.38) - open/close 1200 (time 1789548016.24 total 93.96 last 14.01) - open/close 1352 (time 1789548026.53 total 104.25 last 14.77) - open/close 1563 (time 1789548036.78 total 114.50 last 20.58) - open/close 1755 (time 1789548046.79 total 124.51 last 19.18) - open/close 2000 (time 1789548057.25 total 134.97 last 23.41) - open/close 2141 (time 1789548068.13 total 145.85 last 12.97) - open/close 2284 (time 1789548079.07 total 156.79 last 13.07) - open/close 2458 (time 1789548089.24 total 166.96 last 17.10) - open/close 2507 (time 1789548100.25 total 177.97 last 4.45) - open/close 2716 (time 1789548110.76 total 188.48 last 19.89) - open/close 2955 (time 1789548120.76 total 198.48 last 23.89) - open/close 3184 (time 1789548131.55 total 209.27 last 21.23) - open/close 3386 (time 1789548141.87 total 219.59 last 19.58) - open/close 3559 (time 1789548151.97 total 229.69 last 17.11) - open/close 3764 (time 1789548161.99 total 239.71 last 20.47) - open/close 3905 (time 1789548172.70 total 250.42 last 13.16) - open/close 4077 (time 1789548184.23 total 261.95 last 14.92) - open/close 4209 (time 1789548195.32 total 273.04 last 11.90) - open/close 4361 (time 1789548205.32 total 283.04 last 15.20) - open/close 4520 (time 1789548216.30 total 294.02 last 14.48) - open/close 4635 (time 1789548226.80 total 304.52 last 10.95) - open/close 4755 (time 1789548237.80 total 315.52 last 10.91) - open/close 4897 (time 1789548247.84 total 325.56 last 14.15) - open/close 5045 (time 1789548258.81 total 336.53 last 13.49) - open/close 5332 (time 1789548269.52 total 347.24 last 26.81) - open/close 5565 (time 1789548280.30 total 358.02 last 21.62) - open/close 5882 (time 1789548291.38 total 369.11 last 28.59) - open/close 6116 (time 1789548301.55 total 379.27 last 23.02) - open/close 6417 (time 1789548312.37 total 390.09 last 27.82) - open/close 6664 (time 1789548322.59 total 400.31 last 24.16) - open/close 6978 (time 1789548333.71 total 411.43 last 28.25) - open/close 7187 (time 1789548343.94 total 421.66 last 20.43) - open/close 7390 (time 1789548354.21 total 431.93 last 19.77) - open/close 7678 (time 1789548364.21 total 441.93 last 28.78) - open/close 7909 (time 1789548374.94 total 452.66 last 21.53) - open/close 8117 (time 1789548385.12 total 462.85 last 20.43) - open/close 8365 (time 1789548395.13 total 472.86 last 24.78) - open/close 8607 (time 1789548405.14 total 482.86 last 24.18) - open/close 8858 (time 1789548416.09 total 493.81 last 22.93) - open/close 9093 (time 1789548426.11 total 503.83 last 23.45) - open/close 9312 (time 1789548436.57 total 514.30 last 20.93) - open/close 9622 (time 1789548446.88 total 524.60 last 30.07) - open/close 9864 (time 1789548457.59 total 535.31 last 22.60) - open/close 10000 (time 1789548464.71 total 542.44 last 19.09) - open/close 10210 (time 1789548475.11 total 552.83 last 20.21) - open/close 10416 (time 1789548485.11 total 562.83 last 20.59) - open/close 10604 (time 1789548495.31 total 573.03 last 18.44) - open/close 10838 (time 1789548505.95 total 583.67 last 21.99) - open/close 11129 (time 1789548516.39 total 594.11 last 27.89) - open/close 11356 (time 1789548527.33 total 605.05 last 20.74) - open/close 11488 (time 1789548537.34 total 615.06 last 13.20) - open/close 11634 (time 1789548547.36 total 625.08 last 14.57) - open/close 11775 (time 1789548558.11 total 635.83 last 13.12) - open/close 11972 (time 1789548570.26 total 647.99 last 16.21) - open/close 12220 (time 1789548580.97 total 658.69 last 23.18) - open/close 12582 (time 1789548591.26 total 668.98 last 35.16) - open/close 12967 (time 1789548602.26 total 679.98 last 35.00) - open/close 13201 (time 1789548613.15 total 690.87 last 21.50) - open/close 13430 (time 1789548623.15 total 700.88 last 22.88) - open/close 14199 (time 1789548633.16 total 710.88 last 76.88) - open/close 16049 (time 1789548643.16 total 720.88 last 184.91) - open/close 18319 (time 1789548653.16 total 730.89 last 226.95) - open/close 20000 (time 1789548662.07 total 739.79 last 188.81) - open/close 22026 (time 1789548672.07 total 749.79 last 202.54) - open/close 24593 (time 1789548682.07 total 759.79 last 256.62) - open/close 26452 (time 1789548692.08 total 769.80 last 185.81) - open/close 28369 (time 1789548702.08 total 779.80 last 191.69) - open/close 29575 (time 1789548712.08 total 789.80 last 120.56) - open/close 30000 (time 1789548716.07 total 793.80 last 106.46) - open/close 31058 (time 1789548726.08 total 803.80 last 105.78) - open/close 32164 (time 1789548736.08 total 813.80 last 110.58) - open/close 33251 (time 1789548746.09 total 823.81 last 108.60) - open/close 34330 (time 1789548756.09 total 833.81 last 107.83) - open/close 35191 (time 1789548766.10 total 843.82 last 86.05) - open/close 36401 (time 1789548776.11 total 853.83 last 120.92) - open/close 37610 (time 1789548786.11 total 863.83 last 120.83) - open/close 38526 (time 1789548796.12 total 873.84 last 91.53) - open/close 39634 (time 1789548806.12 total 883.84 last 110.80) - open/close 40000 (time 1789548809.34 total 887.06 last 113.75) - open/close 41070 (time 1789548819.35 total 897.07 last 106.86) - open/close 42226 (time 1789548829.36 total 907.08 last 115.48) - open/close 43297 (time 1789548839.36 total 917.09 last 107.05) - open/close 44321 (time 1789548849.37 total 927.09 last 102.33) - open/close 45329 (time 1789548859.37 total 937.10 last 100.76) - open/close 46432 (time 1789548869.38 total 947.10 last 110.23) - open/close 47622 (time 1789548879.38 total 957.11 last 118.97) - open/close 48710 (time 1789548889.39 total 967.11 last 108.74) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 974.85 seconds: 50.91 ops/second - unlinked 0 (time 1789548899 ; total 0 ; last 0) - unlinked 10000 (time 1789548978 ; total 79 ; last 79) - unlinked 20000 (time 1789549024 ; total 125 ; last 46) - unlinked 30000 (time 1789549071 ; total 172 ; last 47) - unlinked 40000 (time 1789549113 ; total 214 ; last 42) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 260 seconds: 190.888458 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg248-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg248-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg248-server stop mds service on oleg248-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg248-server stop mds service on oleg248-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg248-server unloading modules via unload_modules_local on: 'oleg248-server' oleg248-server: oleg248-server.virtnet: executing unload_modules_local oleg248-server: kmemleak disabled modules unloaded. pdsh@oleg248-client: oleg248-client: ssh exited with exit code 2 pdsh@oleg248-client: oleg248-server: ssh exited with exit code 2 pdsh@oleg248-client: oleg248-server: ssh exited with exit code 2 pdsh@oleg248-client: oleg248-client: ssh exited with exit code 2