== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:03:06 (1789520586) start mds service on oleg460-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 15 17:59 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing load_modules_local oleg460-server: kmemleak disabled oleg460-server: Loading modules from /home/green/git/lustre-release/lustre oleg460-server: detected 4 online CPUs by sysfs oleg460-server: MODOPTS_LIBCFS= oleg460-server: Force libcfs to create 2 CPU partitions oleg460-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg460-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg460-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg460-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg460-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 157 (time 1789520671.82 total 10.03 last 15.66) - open/close 267 (time 1789520683.55 total 21.76 last 9.38) - open/close 490 (time 1789520694.56 total 32.77 last 20.25) - open/close 643 (time 1789520705.24 total 43.45 last 14.32) - open/close 770 (time 1789520716.71 total 54.92 last 11.08) - open/close 867 (time 1789520727.21 total 65.42 last 9.24) - open/close 1049 (time 1789520737.23 total 75.44 last 18.16) - open/close 1283 (time 1789520747.48 total 85.69 last 22.82) - open/close 1476 (time 1789520757.88 total 96.09 last 18.57) - open/close 1709 (time 1789520767.90 total 106.11 last 23.24) - open/close 1889 (time 1789520778.93 total 117.14 last 16.32) - open/close 2038 (time 1789520788.93 total 127.14 last 14.89) - open/close 2125 (time 1789520799.00 total 137.21 last 8.64) - open/close 2280 (time 1789520809.13 total 147.34 last 15.31) - open/close 2465 (time 1789520821.39 total 159.60 last 15.08) - open/close 2619 (time 1789520831.42 total 169.63 last 15.36) - open/close 2768 (time 1789520841.45 total 179.66 last 14.85) - open/close 2898 (time 1789520851.45 total 189.66 last 13.00) - open/close 3100 (time 1789520863.62 total 201.83 last 16.60) - open/close 3330 (time 1789520873.62 total 211.83 last 23.00) - open/close 3528 (time 1789520883.64 total 221.85 last 19.77) - open/close 3675 (time 1789520894.81 total 233.02 last 13.16) - open/close 3823 (time 1789520905.85 total 244.06 last 13.40) - open/close 3903 (time 1789520916.65 total 254.86 last 7.41) - open/close 4056 (time 1789520927.31 total 265.53 last 14.35) - open/close 4228 (time 1789520937.40 total 275.61 last 17.06) - open/close 4483 (time 1789520947.41 total 285.62 last 25.47) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-4753) error: No space left on device total: 4753 open/close in 295.29 seconds: 16.10 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9030: kill: (75991) - No such process - unlinked 0 (time 1789520969 ; total 0 ; last 0) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-4753) error: No such file or directory total: 4753 unlinks in 43 seconds: 110.534882 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg460-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg460-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server stop mds service on oleg460-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg460-server unloading modules via unload_modules_local on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing unload_modules_local oleg460-server: kmemleak disabled modules unloaded. pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2