== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:48:32 (1789998512) start mds service on oleg352-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1940192 Sep 21 05:46 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: kmemleak disabled oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions oleg352-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg352-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg352-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg352-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg352-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-OST0000 oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg352-client.virtnet: -o user_xattr,flock 192.168.203.152@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 245 (time 1789998582.65 total 10.01 last 24.48) - open/close 455 (time 1789998592.65 total 20.01 last 21.00) - open/close 680 (time 1789998602.66 total 30.02 last 22.47) - open/close 897 (time 1789998612.76 total 40.12 last 21.49) - open/close 1137 (time 1789998622.77 total 50.13 last 23.98) - open/close 1353 (time 1789998633.16 total 60.52 last 20.79) - open/close 1577 (time 1789998643.29 total 70.65 last 22.11) - open/close 1837 (time 1789998653.31 total 80.67 last 25.93) - open/close 2050 (time 1789998663.90 total 91.26 last 20.12) - open/close 2289 (time 1789998673.92 total 101.28 last 23.85) - open/close 2492 (time 1789998684.29 total 111.65 last 19.58) - open/close 2746 (time 1789998694.29 total 121.65 last 25.39) - open/close 2941 (time 1789998704.71 total 132.07 last 18.72) - open/close 3230 (time 1789998714.72 total 142.08 last 28.88) - open/close 3443 (time 1789998725.18 total 152.54 last 20.36) - open/close 3631 (time 1789998735.20 total 162.56 last 18.77) - open/close 3871 (time 1789998745.79 total 173.15 last 22.66) - open/close 4085 (time 1789998756.30 total 183.66 last 20.37) - open/close 4315 (time 1789998766.54 total 193.90 last 22.45) - open/close 4531 (time 1789998776.90 total 204.26 last 20.86) - open/close 4751 (time 1789998787.26 total 214.62 last 21.22) - open/close 4999 (time 1789998797.97 total 225.33 last 23.16) - open/close 5245 (time 1789998808.91 total 236.27 last 22.49) - open/close 5462 (time 1789998819.78 total 247.14 last 19.96) - open/close 5716 (time 1789998830.67 total 258.03 last 23.32) - open/close 5942 (time 1789998840.68 total 268.04 last 22.59) - open/close 6112 (time 1789998850.69 total 278.05 last 16.98) - open/close 6318 (time 1789998860.74 total 288.10 last 20.50) - open/close 6566 (time 1789998870.75 total 298.10 last 24.78) - open/close 6786 (time 1789998881.15 total 308.51 last 21.14) - open/close 7030 (time 1789998891.71 total 319.07 last 23.10) - open/close 7242 (time 1789998902.28 total 329.64 last 20.06) - open/close 7426 (time 1789998913.09 total 340.44 last 17.03) - open/close 7628 (time 1789998923.09 total 350.45 last 20.19) - open/close 7865 (time 1789998933.10 total 360.46 last 23.68) - open/close 8091 (time 1789998943.55 total 370.91 last 21.61) - open/close 8345 (time 1789998953.57 total 380.93 last 25.36) - open/close 8554 (time 1789998963.85 total 391.21 last 20.34) - open/close 8805 (time 1789998974.74 total 402.10 last 23.04) - open/close 9014 (time 1789998984.97 total 412.33 last 20.44) - open/close 9247 (time 1789998995.59 total 422.95 last 21.93) - open/close 9473 (time 1789999005.95 total 433.31 last 21.81) - open/close 9663 (time 1789999016.77 total 444.13 last 17.57) - open/close 9868 (time 1789999027.78 total 455.14 last 18.62) - open/close 10000 (time 1789999033.76 total 461.12 last 22.06) - open/close 10186 (time 1789999043.79 total 471.15 last 18.56) - open/close 10284 (time 1789999054.58 total 481.94 last 9.08) - open/close 10398 (time 1789999065.48 total 492.84 last 10.46) - open/close 10533 (time 1789999078.32 total 505.68 last 10.52) - open/close 10637 (time 1789999089.05 total 516.41 last 9.69) - open/close 10767 (time 1789999099.07 total 526.43 last 12.98) - open/close 10937 (time 1789999109.12 total 536.48 last 16.91) - open/close 11151 (time 1789999119.31 total 546.67 last 21.00) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-11377) error: No space left on device total: 11377 open/close in 555.22 seconds: 20.49 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9030: kill: (76197) - No such process - unlinked 0 (time 1789999133 ; total 0 ; last 0) - unlinked 10000 (time 1789999217 ; total 84 ; last 84) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-11377) error: No such file or directory total: 11377 unlinks in 95 seconds: 119.757896 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg352-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server unloading modules via unload_modules_local on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing unload_modules_local oleg352-server: kmemleak disabled modules unloaded. pdsh@oleg352-client: oleg352-client: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-client: ssh exited with exit code 2