== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:55:58 (1788666958) start mds service on oleg659-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1922328 Sep 5 15:39 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg659-server' oleg659-server: oleg659-server.virtnet: executing load_modules_local oleg659-server: kmemleak disabled oleg659-server: Loading modules from /home/green/git/lustre-release/lustre oleg659-server: detected 4 online CPUs by sysfs oleg659-server: MODOPTS_LIBCFS= oleg659-server: Force libcfs to create 2 CPU partitions oleg659-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg659-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg659-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg659-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg659-server: oleg659-server.virtnet: executing set_default_debug -1 all pdsh@oleg659-client: oleg659-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg659-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg659-server: oleg659-server.virtnet: executing set_default_debug -1 all pdsh@oleg659-client: oleg659-server: ssh exited with exit code 1 Started lustre-OST0000 oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg659-client.virtnet: -o user_xattr,flock 192.168.206.159@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 210 (time 1788667029.03 total 10.55 last 19.90) - open/close 536 (time 1788667039.03 total 20.56 last 32.59) - open/close 815 (time 1788667049.19 total 30.71 last 27.47) - open/close 1125 (time 1788667059.75 total 41.27 last 29.36) - open/close 1320 (time 1788667070.25 total 51.78 last 18.56) - open/close 1585 (time 1788667080.30 total 61.82 last 26.38) - open/close 1847 (time 1788667091.18 total 72.70 last 24.08) - open/close 2166 (time 1788667101.30 total 82.83 last 31.52) - open/close 2417 (time 1788667111.31 total 92.83 last 25.08) - open/close 2681 (time 1788667122.31 total 103.83 last 24.00) - open/close 2934 (time 1788667132.59 total 114.11 last 24.61) - open/close 3174 (time 1788667142.69 total 124.22 last 23.76) - open/close 3452 (time 1788667153.13 total 134.65 last 26.64) - open/close 3753 (time 1788667164.08 total 145.61 last 27.48) - open/close 4037 (time 1788667174.10 total 155.63 last 28.34) - open/close 4292 (time 1788667184.11 total 165.63 last 25.48) - open/close 4515 (time 1788667194.60 total 176.13 last 21.24) - open/close 4786 (time 1788667204.61 total 186.13 last 27.10) - open/close 5035 (time 1788667214.61 total 196.14 last 24.88) - open/close 5272 (time 1788667224.62 total 206.14 last 23.69) - open/close 5524 (time 1788667235.38 total 216.90 last 23.41) - open/close 5787 (time 1788667245.38 total 226.91 last 26.29) - open/close 6043 (time 1788667255.85 total 237.38 last 24.44) - open/close 6268 (time 1788667265.97 total 247.50 last 22.24) - open/close 6437 (time 1788667275.99 total 257.52 last 16.87) - open/close 6703 (time 1788667286.31 total 267.83 last 25.79) - open/close 7009 (time 1788667296.42 total 277.95 last 30.26) - open/close 7277 (time 1788667306.43 total 287.95 last 26.78) - open/close 7503 (time 1788667317.36 total 298.88 last 20.68) - open/close 7754 (time 1788667327.72 total 309.24 last 24.22) - open/close 8000 (time 1788667337.88 total 319.41 last 24.21) - open/close 8213 (time 1788667348.22 total 329.74 last 20.61) - open/close 8477 (time 1788667359.16 total 340.68 last 24.14) - open/close 8736 (time 1788667370.10 total 351.62 last 23.67) - open/close 8983 (time 1788667380.90 total 362.42 last 22.87) - open/close 9216 (time 1788667391.53 total 373.06 last 21.91) - open/close 9462 (time 1788667401.55 total 383.08 last 24.55) - open/close 9746 (time 1788667412.40 total 393.93 last 26.17) - open/close 10000 (time 1788667421.63 total 403.15 last 27.53) - open/close 10215 (time 1788667431.72 total 413.25 last 21.30) - open/close 10456 (time 1788667441.83 total 423.36 last 23.83) - open/close 10715 (time 1788667452.81 total 434.33 last 23.60) - open/close 10950 (time 1788667463.33 total 444.85 last 22.33) - open/close 11208 (time 1788667473.72 total 455.24 last 24.83) - open/close 11418 (time 1788667483.73 total 465.26 last 20.97) - open/close 11656 (time 1788667494.83 total 476.35 last 21.45) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-11761) error: No space left on device total: 11761 open/close in 480.05 seconds: 24.50 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9030: kill: (75965) - No such process - unlinked 0 (time 1788667505 ; total 0 ; last 0) - unlinked 10000 (time 1788667581 ; total 76 ; last 76) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-11761) error: No such file or directory total: 11761 unlinks in 88 seconds: 133.647720 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg659-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg659-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg659-server stop mds service on oleg659-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg659-server unloading modules via unload_modules_local on: 'oleg659-server' oleg659-server: oleg659-server.virtnet: executing unload_modules_local oleg659-server: kmemleak disabled modules unloaded. pdsh@oleg659-client: oleg659-server: ssh exited with exit code 2 pdsh@oleg659-client: oleg659-client: ssh exited with exit code 2 pdsh@oleg659-client: oleg659-client: ssh exited with exit code 2 pdsh@oleg659-client: oleg659-server: ssh exited with exit code 2