== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:47:06 (1788616026) start mds service on oleg254-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 4 17:26 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: kmemleak disabled oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions oleg254-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg254-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg254-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg254-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg254-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 228 (time 1788616108.53 total 11.01 last 20.71) - open/close 476 (time 1788616119.41 total 21.88 last 22.81) - open/close 775 (time 1788616129.42 total 31.89 last 29.87) - open/close 986 (time 1788616140.36 total 42.84 last 19.28) - open/close 1113 (time 1788616150.70 total 53.18 last 12.28) - open/close 1384 (time 1788616160.71 total 63.18 last 27.08) - open/close 1574 (time 1788616171.12 total 73.60 last 18.25) - open/close 1789 (time 1788616181.95 total 84.43 last 19.86) - open/close 1959 (time 1788616192.69 total 95.17 last 15.83) - open/close 2130 (time 1788616202.70 total 105.18 last 17.08) - open/close 2373 (time 1788616213.20 total 115.67 last 23.15) - open/close 2556 (time 1788616223.73 total 126.20 last 17.38) - open/close 2688 (time 1788616233.98 total 136.46 last 12.87) - open/close 2822 (time 1788616246.51 total 148.99 last 10.69) - open/close 2915 (time 1788616256.77 total 159.24 last 9.07) - open/close 3086 (time 1788616266.78 total 169.25 last 17.08) - open/close 3330 (time 1788616277.29 total 179.77 last 23.21) - open/close 3473 (time 1788616287.31 total 189.78 last 14.28) - open/close 3593 (time 1788616297.31 total 199.78 last 12.00) - open/close 3725 (time 1788616307.77 total 210.24 last 12.62) - open/close 3932 (time 1788616318.50 total 220.97 last 19.29) - open/close 4083 (time 1788616328.52 total 230.99 last 15.06) - open/close 4221 (time 1788616338.86 total 241.33 last 13.35) - open/close 4407 (time 1788616349.38 total 251.85 last 17.68) - open/close 4493 (time 1788616359.91 total 262.39 last 8.16) - open/close 4565 (time 1788616369.91 total 272.39 last 7.20) - open/close 4798 (time 1788616380.92 total 283.39 last 21.18) - open/close 4969 (time 1788616390.92 total 293.39 last 17.09) - open/close 5130 (time 1788616401.27 total 303.74 last 15.56) - open/close 5320 (time 1788616411.28 total 313.75 last 18.99) - open/close 5530 (time 1788616421.67 total 324.14 last 20.20) - open/close 5711 (time 1788616432.07 total 334.55 last 17.40) - open/close 5852 (time 1788616442.08 total 344.56 last 14.09) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-5905) error: No space left on device total: 5905 open/close in 346.52 seconds: 17.04 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9030: kill: (76055) - No such process - unlinked 0 (time 1788616450 ; total 0 ; last 0) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-5905) error: No such file or directory total: 5905 unlinks in 51 seconds: 115.784317 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg254-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server unloading modules via unload_modules_local on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing unload_modules_local oleg254-server: kmemleak disabled modules unloaded. pdsh@oleg254-client: oleg254-client: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-client: ssh exited with exit code 2