== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:54:59 (1788911699) start mds service on oleg219-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1757936 Sep 8 18:39 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing load_modules_local oleg219-server: kmemleak disabled oleg219-server: Loading modules from /home/green/git/lustre-release/lustre oleg219-server: detected 4 online CPUs by sysfs oleg219-server: MODOPTS_LIBCFS= oleg219-server: Force libcfs to create 2 CPU partitions oleg219-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg219-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg219-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg219-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg219-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg219-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 275 (time 1788911755.46 total 10.91 last 25.20) - open/close 504 (time 1788911766.40 total 21.85 last 20.93) - open/close 828 (time 1788911777.22 total 32.68 last 29.94) - open/close 979 (time 1788911787.46 total 42.92 last 14.75) - open/close 1235 (time 1788911797.57 total 53.03 last 25.32) - open/close 1566 (time 1788911807.69 total 63.14 last 32.71) - open/close 1822 (time 1788911817.69 total 73.15 last 25.59) - open/close 2051 (time 1788911828.30 total 83.75 last 21.60) - open/close 2299 (time 1788911839.11 total 94.57 last 22.93) - open/close 2436 (time 1788911849.16 total 104.61 last 13.64) - open/close 2581 (time 1788911859.17 total 114.62 last 14.48) - open/close 2767 (time 1788911869.64 total 125.10 last 17.75) - open/close 3056 (time 1788911880.14 total 135.59 last 27.54) - open/close 3276 (time 1788911890.43 total 145.89 last 21.37) - open/close 3532 (time 1788911901.25 total 156.70 last 23.67) - open/close 3787 (time 1788911911.94 total 167.39 last 23.86) - open/close 4033 (time 1788911922.88 total 178.34 last 22.47) - open/close 4312 (time 1788911933.51 total 188.96 last 26.26) - open/close 4516 (time 1788911943.94 total 199.39 last 19.55) - open/close 4688 (time 1788911954.11 total 209.57 last 16.91) - open/close 5033 (time 1788911964.67 total 220.13 last 32.67) - open/close 5377 (time 1788911980.48 total 235.93 last 21.76) - open/close 5658 (time 1788911990.72 total 246.17 last 27.45) - open/close 5873 (time 1788912000.72 total 256.17 last 21.50) - open/close 5977 (time 1788912011.64 total 267.10 last 9.52) - open/close 6561 (time 1788912021.89 total 277.34 last 57.01) - open/close 6979 (time 1788912032.58 total 288.03 last 39.11) - open/close 7604 (time 1788912042.75 total 298.20 last 61.44) - open/close 8132 (time 1788912053.50 total 308.96 last 49.10) - open/close 8765 (time 1788912063.55 total 319.01 last 63.00) - open/close 9393 (time 1788912073.79 total 329.24 last 61.34) - open/close 9972 (time 1788912083.79 total 339.24 last 57.90) - open/close 10590 (time 1788912094.53 total 349.98 last 57.55) - open/close 11270 (time 1788912104.53 total 359.98 last 67.99) - open/close 15729 (time 1788912114.53 total 369.98 last 445.83) - open/close 20000 (time 1788912124.04 total 379.50 last 448.83) - open/close 24094 (time 1788912134.05 total 389.50 last 409.38) - open/close 28416 (time 1788912144.05 total 399.50 last 431.99) - open/close 30000 (time 1788912147.27 total 402.73 last 491.50) - open/close 34400 (time 1788912157.27 total 412.73 last 439.99) - open/close 39035 (time 1788912167.27 total 422.73 last 463.42) - open/close 40000 (time 1788912169.36 total 424.82 last 462.72) - open/close 44558 (time 1788912179.36 total 434.82 last 455.73) - open/close 49007 (time 1788912189.36 total 444.82 last 444.81) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 446.16 seconds: 111.24 ops/second - unlinked 0 (time 1788912192 ; total 0 ; last 0) - unlinked 10000 (time 1788912205 ; total 13 ; last 13) - unlinked 20000 (time 1788912216 ; total 24 ; last 11) - unlinked 30000 (time 1788912227 ; total 35 ; last 11) - unlinked 40000 (time 1788912238 ; total 46 ; last 11) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 56 seconds: 886.267883 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg219-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg219-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server stop mds service on oleg219-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg219-server stop mds service on oleg219-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg219-server unloading modules via unload_modules_local on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing unload_modules_local oleg219-server: kmemleak disabled modules unloaded. pdsh@oleg219-client: oleg219-client: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-client: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2