== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:44:47 (1790729087) start mds service on oleg260-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1946856 Sep 29 10:18 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: kmemleak disabled oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions oleg260-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg260-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg260-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg260-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg260-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 193 (time 1790729175.28 total 10.91 last 17.69) - open/close 442 (time 1790729185.51 total 21.14 last 24.35) - open/close 634 (time 1790729195.78 total 31.40 last 18.70) - open/close 899 (time 1790729206.63 total 42.26 last 24.41) - open/close 1121 (time 1790729216.64 total 52.26 last 22.19) - open/close 1361 (time 1790729226.87 total 62.49 last 23.46) - open/close 1583 (time 1790729236.93 total 72.55 last 22.06) - open/close 1764 (time 1790729246.93 total 82.56 last 18.09) - open/close 1997 (time 1790729257.27 total 92.89 last 22.55) - open/close 2286 (time 1790729268.26 total 103.89 last 26.28) - open/close 2520 (time 1790729278.26 total 113.89 last 23.40) - open/close 2767 (time 1790729288.74 total 124.37 last 23.56) - open/close 3041 (time 1790729299.63 total 135.26 last 25.17) - open/close 3296 (time 1790729310.20 total 145.82 last 24.13) - open/close 3526 (time 1790729320.75 total 156.38 last 21.80) - open/close 3726 (time 1790729331.24 total 166.87 last 19.07) - open/close 3917 (time 1790729341.25 total 176.87 last 19.09) - open/close 4155 (time 1790729351.63 total 187.26 last 22.91) - open/close 4346 (time 1790729361.66 total 197.29 last 19.05) - open/close 4566 (time 1790729372.48 total 208.11 last 20.32) - open/close 4766 (time 1790729382.71 total 218.34 last 19.56) - open/close 5078 (time 1790729392.73 total 228.35 last 31.15) - open/close 5286 (time 1790729403.57 total 239.19 last 19.19) - open/close 5432 (time 1790729414.32 total 249.95 last 13.57) - open/close 5534 (time 1790729424.35 total 259.97 last 10.18) - open/close 5642 (time 1790729434.35 total 269.97 last 10.80) - open/close 5799 (time 1790729447.18 total 282.81 last 12.23) - open/close 5939 (time 1790729457.19 total 292.82 last 13.99) - open/close 6144 (time 1790729467.20 total 302.83 last 20.47) - open/close 6260 (time 1790729477.20 total 312.83 last 11.60) - open/close 6404 (time 1790729488.05 total 323.67 last 13.28) - open/close 6562 (time 1790729498.86 total 334.49 last 14.61) - open/close 6760 (time 1790729509.44 total 345.07 last 18.72) - open/close 6916 (time 1790729520.31 total 355.93 last 14.36) - open/close 7074 (time 1790729531.12 total 366.75 last 14.61) - open/close 7321 (time 1790729541.79 total 377.41 last 23.15) - open/close 7434 (time 1790729554.03 total 389.66 last 9.23) - open/close 7502 (time 1790729564.74 total 400.37 last 6.35) - open/close 7721 (time 1790729575.53 total 411.16 last 20.29) - open/close 7826 (time 1790729586.29 total 421.91 last 9.77) - open/close 7951 (time 1790729596.78 total 432.40 last 11.91) - open/close 8029 (time 1790729607.16 total 442.79 last 7.51) - open/close 8255 (time 1790729617.16 total 452.79 last 22.59) - open/close 8301 (time 1790729627.56 total 463.19 last 4.42) - open/close 8439 (time 1790729638.06 total 473.69 last 13.14) - open/close 8553 (time 1790729648.77 total 484.40 last 10.64) - open/close 8825 (time 1790729658.79 total 494.42 last 27.15) - open/close 9071 (time 1790729668.81 total 504.44 last 24.56) - open/close 9232 (time 1790729678.82 total 514.45 last 16.08) - open/close 9322 (time 1790729689.58 total 525.20 last 8.37) - open/close 9423 (time 1790729700.31 total 535.93 last 9.41) - open/close 9591 (time 1790729710.32 total 545.95 last 16.78) - open/close 9780 (time 1790729720.81 total 556.43 last 18.02) - open/close 9873 (time 1790729731.82 total 567.45 last 8.44) - open/close 9977 (time 1790729742.50 total 578.13 last 9.74) - open/close 10157 (time 1790729752.53 total 588.16 last 17.95) - open/close 10425 (time 1790729762.54 total 598.16 last 26.78) - open/close 10587 (time 1790729772.92 total 608.55 last 15.60) - open/close 10856 (time 1790729782.93 total 618.56 last 26.87) - open/close 11085 (time 1790729792.93 total 628.56 last 22.90) - open/close 11260 (time 1790729805.55 total 641.18 last 13.87) - open/close 11422 (time 1790729816.17 total 651.80 last 15.26) - open/close 11622 (time 1790729826.57 total 662.19 last 19.23) - open/close 11808 (time 1790729836.64 total 672.27 last 18.46) - open/close 11910 (time 1790729847.39 total 683.02 last 9.49) - open/close 12041 (time 1790729858.30 total 693.92 last 12.01) - open/close 12279 (time 1790729868.31 total 703.93 last 23.78) - open/close 12439 (time 1790729879.08 total 714.71 last 14.85) - open/close 12666 (time 1790729889.79 total 725.41 last 21.20) - open/close 12927 (time 1790729900.14 total 735.77 last 25.21) - open/close 13174 (time 1790729910.15 total 745.77 last 24.69) - open/close 13350 (time 1790729920.62 total 756.25 last 16.80) - open/close 13597 (time 1790729931.06 total 766.68 last 23.68) - open/close 13866 (time 1790729941.96 total 777.59 last 24.68) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-14033) error: No space left on device total: 14033 open/close in 786.14 seconds: 17.85 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9030: kill: (76236) - No such process - unlinked 0 (time 1790729954 ; total 0 ; last 0) - unlinked 10000 (time 1790730053 ; total 99 ; last 99) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-14033) error: No such file or directory total: 14033 unlinks in 130 seconds: 107.946152 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg260-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server unloading modules via unload_modules_local on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing unload_modules_local oleg260-server: kmemleak disabled modules unloaded. pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2