== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:48:36 (1788806916) start mds service on oleg246-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 7 11:22 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing load_modules_local oleg246-server: kmemleak disabled oleg246-server: Loading modules from /home/green/git/lustre-release/lustre oleg246-server: detected 4 online CPUs by sysfs oleg246-server: MODOPTS_LIBCFS= oleg246-server: Force libcfs to create 2 CPU partitions oleg246-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg246-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg246-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg246-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg246-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg246-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-OST0000 oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg246-client.virtnet: -o user_xattr,flock 192.168.202.146@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 276 (time 1788806982.36 total 10.99 last 25.12) - open/close 445 (time 1788806992.37 total 20.99 last 16.90) - open/close 645 (time 1788807002.40 total 31.02 last 19.94) - open/close 876 (time 1788807012.83 total 41.46 last 22.14) - open/close 1104 (time 1788807022.84 total 51.46 last 22.79) - open/close 1337 (time 1788807033.25 total 61.88 last 22.37) - open/close 1661 (time 1788807043.36 total 71.98 last 32.05) - open/close 1971 (time 1788807054.36 total 82.99 last 28.18) - open/close 2314 (time 1788807065.37 total 94.00 last 31.15) - open/close 2725 (time 1788807075.38 total 104.00 last 41.09) - open/close 2931 (time 1788807085.47 total 114.09 last 20.41) - open/close 3144 (time 1788807095.84 total 124.47 last 20.54) - open/close 3293 (time 1788807108.85 total 137.47 last 11.46) - open/close 3555 (time 1788807119.59 total 148.21 last 24.39) - open/close 3837 (time 1788807130.28 total 158.90 last 26.38) - open/close 4262 (time 1788807140.90 total 169.52 last 40.02) - open/close 4512 (time 1788807151.60 total 180.22 last 23.37) - open/close 4751 (time 1788807162.04 total 190.66 last 22.89) - open/close 4930 (time 1788807172.04 total 200.66 last 17.90) - open/close 5071 (time 1788807182.73 total 211.35 last 13.19) - open/close 5222 (time 1788807192.73 total 221.36 last 15.10) - open/close 5337 (time 1788807202.77 total 231.39 last 11.46) - open/close 5496 (time 1788807215.54 total 244.17 last 12.45) - open/close 5672 (time 1788807225.55 total 254.17 last 17.58) - open/close 5768 (time 1788807237.17 total 265.79 last 8.27) - open/close 5979 (time 1788807247.68 total 276.31 last 20.06) - open/close 6219 (time 1788807258.48 total 287.10 last 22.23) - open/close 6483 (time 1788807268.93 total 297.55 last 25.27) - open/close 6795 (time 1788807279.85 total 308.48 last 28.57) - open/close 7072 (time 1788807290.85 total 319.48 last 25.18) - open/close 7389 (time 1788807301.81 total 330.44 last 28.92) - open/close 7654 (time 1788807311.82 total 340.44 last 26.48) - open/close 7913 (time 1788807322.56 total 351.18 last 24.12) - open/close 8181 (time 1788807335.40 total 364.03 last 20.87) - open/close 8374 (time 1788807345.40 total 374.03 last 19.29) - open/close 8659 (time 1788807356.40 total 385.03 last 25.91) - open/close 8941 (time 1788807367.25 total 395.87 last 26.00) - open/close 9161 (time 1788807377.26 total 405.88 last 21.98) - open/close 9374 (time 1788807387.69 total 416.32 last 20.42) - open/close 9608 (time 1788807397.70 total 426.32 last 23.38) - open/close 9894 (time 1788807407.71 total 436.33 last 28.57) - open/close 10000 (time 1788807412.51 total 441.13 last 22.09) - open/close 10218 (time 1788807422.56 total 451.18 last 21.69) - open/close 10475 (time 1788807433.51 total 462.13 last 23.47) - open/close 10751 (time 1788807444.26 total 472.88 last 25.68) - open/close 11044 (time 1788807454.83 total 483.45 last 27.71) - open/close 11258 (time 1788807465.48 total 494.10 last 20.10) - open/close 11500 (time 1788807475.49 total 504.11 last 24.18) - open/close 11732 (time 1788807486.26 total 514.88 last 21.53) - open/close 11974 (time 1788807497.15 total 525.77 last 22.23) - open/close 12220 (time 1788807507.64 total 536.26 last 23.45) - open/close 12509 (time 1788807517.86 total 546.49 last 28.26) - open/close 12781 (time 1788807528.34 total 556.96 last 25.96) - open/close 13013 (time 1788807538.35 total 566.98 last 23.18) - open/close 13271 (time 1788807548.76 total 577.39 last 24.78) - open/close 13530 (time 1788807558.77 total 587.39 last 25.89) - open/close 13712 (time 1788807568.78 total 597.41 last 18.18) - open/close 14044 (time 1788807578.80 total 607.43 last 33.13) - open/close 14233 (time 1788807588.81 total 617.44 last 18.88) - open/close 14438 (time 1788807599.02 total 627.64 last 20.10) - open/close 14652 (time 1788807609.06 total 637.69 last 21.30) - open/close 14842 (time 1788807620.01 total 648.64 last 17.35) - open/close 15053 (time 1788807630.05 total 658.68 last 21.01) - open/close 15266 (time 1788807640.07 total 668.69 last 21.26) - open/close 15483 (time 1788807650.35 total 678.97 last 21.12) - open/close 15661 (time 1788807660.73 total 689.36 last 17.14) - open/close 15889 (time 1788807671.21 total 699.83 last 21.76) - open/close 16098 (time 1788807681.77 total 710.39 last 19.79) - open/close 16294 (time 1788807692.19 total 720.81 last 18.81) - open/close 16517 (time 1788807702.19 total 730.82 last 22.29) - open/close 17067 (time 1788807712.20 total 740.82 last 54.99) - open/close 17918 (time 1788807722.20 total 750.82 last 85.06) - open/close 18772 (time 1788807732.21 total 760.83 last 85.36) - open/close 19582 (time 1788807742.21 total 770.83 last 80.95) - open/close 20000 (time 1788807747.03 total 775.65 last 86.78) - open/close 20814 (time 1788807757.05 total 785.67 last 81.25) - open/close 21647 (time 1788807767.05 total 795.68 last 83.25) - open/close 22510 (time 1788807777.07 total 805.69 last 86.18) - open/close 23335 (time 1788807787.07 total 815.69 last 82.49) - open/close 24249 (time 1788807797.07 total 825.69 last 91.38) - open/close 25258 (time 1788807807.07 total 835.70 last 100.85) - open/close 26134 (time 1788807817.08 total 845.70 last 87.58) - open/close 27066 (time 1788807827.08 total 855.70 last 93.20) - open/close 28039 (time 1788807837.08 total 865.70 last 97.29) - open/close 28920 (time 1788807847.09 total 875.71 last 88.03) - open/close 29772 (time 1788807857.09 total 885.72 last 85.14) - open/close 30000 (time 1788807860.02 total 888.65 last 77.79) - open/close 30856 (time 1788807870.03 total 898.66 last 85.53) - open/close 31766 (time 1788807880.05 total 908.68 last 90.82) - open/close 32658 (time 1788807890.05 total 918.68 last 89.20) - open/close 33569 (time 1788807900.06 total 928.68 last 91.05) - open/close 34403 (time 1788807910.06 total 938.69 last 83.35) - open/close 35320 (time 1788807920.06 total 948.69 last 91.69) - open/close 36219 (time 1788807930.07 total 958.69 last 89.84) - open/close 37297 (time 1788807940.08 total 968.70 last 107.73) - open/close 38214 (time 1788807950.08 total 978.70 last 91.68) - open/close 39077 (time 1788807960.08 total 988.71 last 86.27) - open/close 39961 (time 1788807970.09 total 998.71 last 88.38) - open/close 40944 (time 1788807980.09 total 1008.71 last 98.27) - open/close 41866 (time 1788807990.09 total 1018.71 last 92.19) - open/close 42771 (time 1788808000.10 total 1028.72 last 90.43) - open/close 43714 (time 1788808010.10 total 1038.73 last 94.25) - open/close 44643 (time 1788808020.11 total 1048.74 last 92.81) - open/close 45655 (time 1788808030.12 total 1058.74 last 101.13) - open/close 46592 (time 1788808040.13 total 1068.75 last 93.65) - open/close 47524 (time 1788808050.13 total 1078.75 last 93.20) - open/close 48421 (time 1788808060.13 total 1088.75 last 89.68) - open/close 49325 (time 1788808070.14 total 1098.76 last 90.33) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1102.43 seconds: 45.02 ops/second - unlinked 0 (time 1788808077 ; total 0 ; last 0) - unlinked 10000 (time 1788808180 ; total 103 ; last 103) - unlinked 20000 (time 1788808250 ; total 173 ; last 70) - unlinked 30000 (time 1788808299 ; total 222 ; last 49) - unlinked 40000 (time 1788808347 ; total 270 ; last 48) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 318 seconds: 156.072327 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg246-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg246-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg246-server stop mds service on oleg246-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg246-server stop mds service on oleg246-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg246-server unloading modules via unload_modules_local on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing unload_modules_local oleg246-server: kmemleak disabled modules unloaded. pdsh@oleg246-client: oleg246-client: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-client: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 2