== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:26:25 (1788596785) start mds service on oleg140-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1924424 Sep 4 13:25 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing load_modules_local oleg140-server: kmemleak disabled oleg140-server: Loading modules from /home/green/git/lustre-release/lustre oleg140-server: detected 4 online CPUs by sysfs oleg140-server: MODOPTS_LIBCFS= oleg140-server: Force libcfs to create 2 CPU partitions oleg140-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg140-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg140-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg140-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg140-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 196 (time 1788596878.61 total 10.36 last 18.91) - open/close 381 (time 1788596888.78 total 20.53 last 18.19) - open/close 623 (time 1788596899.07 total 30.82 last 23.52) - open/close 750 (time 1788596910.08 total 41.84 last 11.53) - open/close 963 (time 1788596921.01 total 52.76 last 19.50) - open/close 1198 (time 1788596931.04 total 62.79 last 23.42) - open/close 1371 (time 1788596941.06 total 72.81 last 17.27) - open/close 1512 (time 1788596951.17 total 82.92 last 13.94) - open/close 1688 (time 1788596961.48 total 93.23 last 17.08) - open/close 1950 (time 1788596971.52 total 103.28 last 26.08) - open/close 2136 (time 1788596981.55 total 113.31 last 18.54) - open/close 2294 (time 1788596991.55 total 123.31 last 15.79) - open/close 2434 (time 1788597001.97 total 133.72 last 13.45) - open/close 2550 (time 1788597012.29 total 144.04 last 11.24) - open/close 2674 (time 1788597022.30 total 154.06 last 12.38) - open/close 2784 (time 1788597032.51 total 164.26 last 10.78) - open/close 2899 (time 1788597042.51 total 174.26 last 11.50) - open/close 3089 (time 1788597052.51 total 184.26 last 19.00) - open/close 3195 (time 1788597062.98 total 194.73 last 10.13) - open/close 3369 (time 1788597073.90 total 205.65 last 15.93) - open/close 3467 (time 1788597084.16 total 215.91 last 9.55) - open/close 3577 (time 1788597094.16 total 225.92 last 10.99) - open/close 3664 (time 1788597105.22 total 236.97 last 7.87) - open/close 3848 (time 1788597115.39 total 247.14 last 18.09) - open/close 3957 (time 1788597125.39 total 257.14 last 10.90) - open/close 3994 (time 1788597141.29 total 273.04 last 2.33) - open/close 4121 (time 1788597152.24 total 284.00 last 11.59) - open/close 4220 (time 1788597162.43 total 294.18 last 9.72) - open/close 4357 (time 1788597172.44 total 304.19 last 13.69) - open/close 4520 (time 1788597182.45 total 314.20 last 16.28) - open/close 4607 (time 1788597193.08 total 324.84 last 8.18) - open/close 4679 (time 1788597203.27 total 335.03 last 7.07) - open/close 4865 (time 1788597214.08 total 345.84 last 17.20) - open/close 5062 (time 1788597224.10 total 355.86 last 19.66) - open/close 5259 (time 1788597234.11 total 365.86 last 19.70) - open/close 5425 (time 1788597244.29 total 376.04 last 16.31) - open/close 5622 (time 1788597254.40 total 386.16 last 19.48) - open/close 5865 (time 1788597265.36 total 397.11 last 22.18) - open/close 5967 (time 1788597276.91 total 408.66 last 8.83) - open/close 6145 (time 1788597288.02 total 419.78 last 16.01) - open/close 6223 (time 1788597298.11 total 429.87 last 7.73) - open/close 6461 (time 1788597308.66 total 440.41 last 22.57) - open/close 6675 (time 1788597318.92 total 450.67 last 20.85) - open/close 6938 (time 1788597328.92 total 460.68 last 26.29) - open/close 7191 (time 1788597338.95 total 470.71 last 25.22) - open/close 7410 (time 1788597349.45 total 481.21 last 20.86) - open/close 7530 (time 1788597359.89 total 491.65 last 11.50) - open/close 7569 (time 1788597370.50 total 502.26 last 3.67) - open/close 7807 (time 1788597381.22 total 512.98 last 22.21) - open/close 8011 (time 1788597391.93 total 523.68 last 19.06) - open/close 8196 (time 1788597402.29 total 534.05 last 17.84) - open/close 8378 (time 1788597412.56 total 544.31 last 17.74) - open/close 8508 (time 1788597423.36 total 555.11 last 12.03) - open/close 8697 (time 1788597435.45 total 567.20 last 15.63) - open/close 8892 (time 1788597445.48 total 577.23 last 19.45) - open/close 9033 (time 1788597455.76 total 587.52 last 13.71) - open/close 9210 (time 1788597467.51 total 599.27 last 15.06) - open/close 9413 (time 1788597477.57 total 609.32 last 20.19) - open/close 9590 (time 1788597488.12 total 619.88 last 16.77) - open/close 9719 (time 1788597498.16 total 629.92 last 12.85) - open/close 9917 (time 1788597508.17 total 639.93 last 19.78) - open/close 10000 (time 1788597513.23 total 644.98 last 16.43) - open/close 10191 (time 1788597523.24 total 655.00 last 19.07) - open/close 10320 (time 1788597533.27 total 665.02 last 12.87) - open/close 10400 (time 1788597544.30 total 676.06 last 7.25) - open/close 10497 (time 1788597554.95 total 686.71 last 9.11) - open/close 10642 (time 1788597564.96 total 696.72 last 14.49) - open/close 10818 (time 1788597576.35 total 708.10 last 15.45) - open/close 11011 (time 1788597587.00 total 718.76 last 18.12) - open/close 11239 (time 1788597597.02 total 728.78 last 22.76) - open/close 11428 (time 1788597607.25 total 739.00 last 18.48) - open/close 11544 (time 1788597617.27 total 749.02 last 11.57) - open/close 11666 (time 1788597627.29 total 759.04 last 12.18) - open/close 11770 (time 1788597637.32 total 769.08 last 10.37) - open/close 11934 (time 1788597648.32 total 780.07 last 14.92) - open/close 12083 (time 1788597659.02 total 790.77 last 13.92) - open/close 12205 (time 1788597669.21 total 800.97 last 11.97) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-12305) error: No space left on device total: 12305 open/close in 804.71 seconds: 15.29 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9030: kill: (76030) - No such process - unlinked 0 (time 1788597687 ; total 0 ; last 0) - unlinked 10000 (time 1788597820 ; total 133 ; last 133) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-12305) error: No such file or directory total: 12305 unlinks in 158 seconds: 77.879745 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg140-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server unloading modules via unload_modules_local on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing unload_modules_local oleg140-server: kmemleak disabled modules unloaded. pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2