== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:44:40 (1788849880) start mds service on oleg336-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 7 17:48 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing load_modules_local oleg336-server: kmemleak disabled oleg336-server: Loading modules from /home/green/git/lustre-release/lustre oleg336-server: detected 4 online CPUs by sysfs oleg336-server: MODOPTS_LIBCFS= oleg336-server: Force libcfs to create 2 CPU partitions oleg336-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg336-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg336-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg336-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg336-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg336-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 180 (time 1788849972.61 total 10.01 last 17.98) - open/close 384 (time 1788849982.86 total 20.26 last 19.90) - open/close 510 (time 1788849993.72 total 31.12 last 11.60) - open/close 704 (time 1788850004.48 total 41.88 last 18.04) - open/close 848 (time 1788850014.59 total 51.99 last 14.24) - open/close 992 (time 1788850024.60 total 62.00 last 14.39) - open/close 1172 (time 1788850035.20 total 72.60 last 16.98) - open/close 1384 (time 1788850046.01 total 83.41 last 19.61) - open/close 1531 (time 1788850056.07 total 93.47 last 14.62) - open/close 1615 (time 1788850066.30 total 103.69 last 8.21) - open/close 1747 (time 1788850077.19 total 114.58 last 12.12) - open/close 1868 (time 1788850088.06 total 125.46 last 11.13) - open/close 1975 (time 1788850098.68 total 136.08 last 10.08) - open/close 2063 (time 1788850111.17 total 148.57 last 7.05) - open/close 2169 (time 1788850121.59 total 158.99 last 10.17) - open/close 2404 (time 1788850132.28 total 169.68 last 21.98) - open/close 2603 (time 1788850142.52 total 179.92 last 19.43) - open/close 2831 (time 1788850153.09 total 190.49 last 21.58) - open/close 2872 (time 1788850163.91 total 201.31 last 3.79) - open/close 3086 (time 1788850174.85 total 212.25 last 19.56) - open/close 3273 (time 1788850185.03 total 222.43 last 18.37) - open/close 3531 (time 1788850195.45 total 232.85 last 24.76) - open/close 3733 (time 1788850205.76 total 243.16 last 19.59) - open/close 4058 (time 1788850215.81 total 253.21 last 32.34) - open/close 4371 (time 1788850227.71 total 265.11 last 26.29) - open/close 4650 (time 1788850237.95 total 275.35 last 27.25) - open/close 4961 (time 1788850247.96 total 285.36 last 31.08) - open/close 5289 (time 1788850257.96 total 295.36 last 32.79) - open/close 5597 (time 1788850267.97 total 305.37 last 30.77) - open/close 5803 (time 1788850278.01 total 315.41 last 20.51) - open/close 6163 (time 1788850289.01 total 326.41 last 32.72) - open/close 6420 (time 1788850299.26 total 336.66 last 25.07) - open/close 6738 (time 1788850310.07 total 347.47 last 29.42) - open/close 7063 (time 1788850320.76 total 358.16 last 30.41) - open/close 7240 (time 1788850331.65 total 369.05 last 16.26) - open/close 7509 (time 1788850341.94 total 379.34 last 26.13) - open/close 7840 (time 1788850351.94 total 389.34 last 33.10) - open/close 8159 (time 1788850362.68 total 400.08 last 29.71) - open/close 8485 (time 1788850372.69 total 410.09 last 32.58) - open/close 8785 (time 1788850383.16 total 420.56 last 28.64) - open/close 9005 (time 1788850393.72 total 431.12 last 20.84) - open/close 9265 (time 1788850403.84 total 441.23 last 25.70) - open/close 9559 (time 1788850414.58 total 451.98 last 27.35) - open/close 9868 (time 1788850424.59 total 461.99 last 30.89) - open/close 10000 (time 1788850429.66 total 467.06 last 26.02) - open/close 10242 (time 1788850440.63 total 478.03 last 22.06) - open/close 10459 (time 1788850451.19 total 488.59 last 20.55) - open/close 10659 (time 1788850461.69 total 499.09 last 19.05) - open/close 11783 (time 1788850471.69 total 509.09 last 112.39) - open/close 13353 (time 1788850481.69 total 519.09 last 156.92) - open/close 14975 (time 1788850491.70 total 529.10 last 162.15) - open/close 16758 (time 1788850501.70 total 539.10 last 178.26) - open/close 18541 (time 1788850511.70 total 549.10 last 178.23) - open/close 20000 (time 1788850519.49 total 556.89 last 187.48) - open/close 21696 (time 1788850529.49 total 566.89 last 169.59) - open/close 23202 (time 1788850539.49 total 576.89 last 150.51) - open/close 24694 (time 1788850549.49 total 586.89 last 149.19) - open/close 26394 (time 1788850559.50 total 596.90 last 169.94) - open/close 28118 (time 1788850569.50 total 606.90 last 172.35) - open/close 29803 (time 1788850579.50 total 616.90 last 168.49) - open/close 31669 (time 1788850589.50 total 626.90 last 186.52) - open/close 33665 (time 1788850599.51 total 636.91 last 199.45) - open/close 35546 (time 1788850609.52 total 646.92 last 187.95) - open/close 37329 (time 1788850619.52 total 656.92 last 178.26) - open/close 39490 (time 1788850629.52 total 666.92 last 216.09) - open/close 40000 (time 1788850632.07 total 669.47 last 199.96) - open/close 42169 (time 1788850642.08 total 679.48 last 216.85) - open/close 44924 (time 1788850652.08 total 689.48 last 275.39) - open/close 46318 (time 1788850662.09 total 699.49 last 139.32) - open/close 47240 (time 1788850672.09 total 709.49 last 92.15) - open/close 48319 (time 1788850682.10 total 719.50 last 107.83) - open/close 49168 (time 1788850692.14 total 729.54 last 84.50) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 734.40 seconds: 67.58 ops/second - unlinked 0 (time 1788850701 ; total 0 ; last 0) - unlinked 10000 (time 1788850785 ; total 84 ; last 84) - unlinked 20000 (time 1788850866 ; total 165 ; last 81) - unlinked 30000 (time 1788850944 ; total 243 ; last 78) - unlinked 40000 (time 1788851019 ; total 318 ; last 75) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 393 seconds: 126.287529 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg336-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg336-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg336-server stop mds service on oleg336-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg336-server stop mds service on oleg336-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg336-server unloading modules via unload_modules_local on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing unload_modules_local oleg336-server: kmemleak disabled modules unloaded. pdsh@oleg336-client: oleg336-client: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-client: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 2