== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:32:30 (1788629550) start mds service on oleg631-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 4 17:56 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg631-server' oleg631-server: oleg631-server.virtnet: executing load_modules_local oleg631-server: kmemleak disabled oleg631-server: Loading modules from /home/green/git/lustre-release/lustre oleg631-server: detected 4 online CPUs by sysfs oleg631-server: MODOPTS_LIBCFS= oleg631-server: Force libcfs to create 2 CPU partitions oleg631-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg631-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg631-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg631-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg631-server: oleg631-server.virtnet: executing set_default_debug -1 all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg631-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg631-server: oleg631-server.virtnet: executing set_default_debug -1 all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg631-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg631-server: oleg631-server.virtnet: executing set_default_debug -1 all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-OST0000 oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg631-client.virtnet: -o user_xattr,flock 192.168.206.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 246 (time 1788629646.18 total 10.02 last 24.55) - open/close 511 (time 1788629657.14 total 20.98 last 24.18) - open/close 762 (time 1788629667.14 total 30.99 last 25.09) - open/close 1008 (time 1788629677.15 total 40.99 last 24.59) - open/close 1251 (time 1788629687.15 total 50.99 last 24.30) - open/close 1512 (time 1788629697.16 total 61.01 last 26.06) - open/close 1761 (time 1788629707.17 total 71.01 last 24.88) - open/close 2003 (time 1788629717.24 total 81.08 last 24.04) - open/close 2263 (time 1788629727.24 total 91.08 last 26.00) - open/close 2546 (time 1788629737.53 total 101.38 last 27.49) - open/close 2752 (time 1788629747.98 total 111.83 last 19.71) - open/close 3033 (time 1788629758.78 total 122.62 last 26.03) - open/close 3323 (time 1788629769.42 total 133.26 last 27.26) - open/close 3575 (time 1788629779.43 total 143.27 last 25.18) - open/close 3828 (time 1788629789.43 total 153.27 last 25.29) - open/close 4095 (time 1788629799.80 total 163.64 last 25.75) - open/close 4377 (time 1788629809.80 total 173.64 last 28.19) - open/close 4624 (time 1788629820.66 total 184.51 last 22.74) - open/close 4925 (time 1788629830.67 total 194.51 last 30.09) - open/close 5166 (time 1788629840.68 total 204.52 last 24.07) - open/close 5435 (time 1788629853.18 total 217.03 last 21.51) - open/close 5677 (time 1788629863.29 total 227.13 last 23.94) - open/close 5937 (time 1788629873.29 total 237.13 last 25.99) - open/close 6187 (time 1788629883.29 total 247.14 last 25.00) - open/close 6453 (time 1788629893.63 total 257.47 last 25.74) - open/close 6768 (time 1788629904.33 total 268.17 last 29.43) - open/close 7054 (time 1788629914.33 total 278.18 last 28.59) - open/close 7316 (time 1788629925.17 total 289.01 last 24.18) - open/close 7577 (time 1788629935.18 total 299.02 last 26.08) - open/close 7838 (time 1788629945.78 total 309.62 last 24.62) - open/close 8111 (time 1788629956.74 total 320.58 last 24.92) - open/close 8348 (time 1788629966.74 total 330.58 last 23.69) - open/close 8596 (time 1788629976.74 total 340.58 last 24.79) - open/close 8837 (time 1788629987.32 total 351.17 last 22.77) - open/close 9042 (time 1788629997.33 total 361.17 last 20.49) - open/close 9297 (time 1788630007.33 total 371.17 last 25.49) - open/close 9563 (time 1788630017.33 total 381.18 last 26.60) - open/close 9841 (time 1788630027.33 total 391.18 last 27.80) - open/close 10000 (time 1788630032.80 total 396.64 last 29.08) - open/close 10313 (time 1788630042.80 total 406.65 last 31.29) - open/close 10530 (time 1788630053.31 total 417.15 last 20.66) - open/close 10860 (time 1788630063.33 total 427.17 last 32.93) - open/close 11116 (time 1788630073.41 total 437.25 last 25.39) - open/close 11314 (time 1788630084.41 total 448.25 last 18.00) - open/close 11581 (time 1788630094.43 total 458.27 last 26.66) - open/close 11843 (time 1788630104.92 total 468.76 last 24.97) - open/close 12092 (time 1788630114.94 total 478.78 last 24.85) - open/close 12379 (time 1788630124.95 total 488.79 last 28.68) - open/close 12590 (time 1788630134.95 total 498.80 last 21.09) - open/close 12806 (time 1788630144.95 total 508.80 last 21.60) - open/close 13090 (time 1788630155.07 total 518.91 last 28.08) - open/close 13329 (time 1788630165.08 total 528.92 last 23.87) - open/close 13577 (time 1788630175.08 total 538.92 last 24.79) - open/close 13733 (time 1788630185.09 total 548.93 last 15.58) - open/close 13962 (time 1788630195.11 total 558.95 last 22.86) - open/close 14281 (time 1788630205.11 total 568.95 last 31.89) - open/close 14574 (time 1788630217.34 total 581.18 last 23.96) - open/close 14824 (time 1788630228.16 total 592.00 last 23.11) - open/close 15084 (time 1788630239.17 total 603.02 last 23.60) - open/close 15387 (time 1788630249.84 total 613.69 last 28.39) - open/close 15653 (time 1788630260.02 total 623.87 last 26.13) - open/close 15980 (time 1788630273.05 total 636.89 last 25.11) - open/close 16286 (time 1788630283.06 total 646.91 last 30.55) - open/close 16552 (time 1788630293.07 total 656.92 last 26.57) - open/close 16835 (time 1788630304.06 total 667.90 last 25.76) - open/close 17101 (time 1788630314.25 total 678.10 last 26.09) - open/close 17372 (time 1788630324.67 total 688.52 last 26.01) - open/close 17616 (time 1788630335.42 total 699.27 last 22.70) - open/close 17876 (time 1788630346.11 total 709.95 last 24.34) - open/close 18089 (time 1788630356.15 total 719.99 last 21.20) - open/close 18322 (time 1788630366.65 total 730.49 last 22.20) - open/close 18628 (time 1788630376.65 total 740.49 last 30.59) - open/close 18846 (time 1788630386.66 total 750.50 last 21.79) - open/close 19074 (time 1788630397.09 total 760.93 last 21.86) - open/close 19354 (time 1788630407.35 total 771.19 last 27.28) - open/close 19616 (time 1788630418.29 total 782.14 last 23.94) - open/close 19877 (time 1788630428.30 total 792.14 last 26.09) - open/close 20000 (time 1788630432.19 total 796.04 last 31.59) - open/close 20265 (time 1788630442.20 total 806.04 last 26.48) - open/close 20518 (time 1788630452.92 total 816.77 last 23.59) - open/close 20724 (time 1788630463.36 total 827.20 last 19.74) - open/close 21718 (time 1788630473.37 total 837.21 last 99.29) - open/close 22660 (time 1788630483.37 total 847.22 last 94.14) - open/close 23631 (time 1788630493.38 total 857.22 last 97.02) - open/close 24629 (time 1788630503.39 total 867.23 last 99.77) - open/close 25577 (time 1788630513.39 total 877.23 last 94.80) - open/close 26491 (time 1788630523.40 total 887.24 last 91.30) - open/close 27403 (time 1788630533.40 total 897.25 last 91.14) - open/close 28380 (time 1788630543.42 total 907.26 last 97.57) - open/close 29556 (time 1788630553.42 total 917.27 last 117.52) - open/close 30000 (time 1788630557.99 total 921.84 last 97.16) - open/close 30969 (time 1788630568.01 total 931.85 last 96.77) - open/close 31951 (time 1788630578.01 total 941.85 last 98.16) - open/close 32949 (time 1788630588.01 total 951.86 last 99.77) - open/close 33915 (time 1788630598.02 total 961.86 last 96.57) - open/close 34861 (time 1788630608.03 total 971.87 last 94.48) - open/close 35872 (time 1788630618.03 total 981.88 last 101.03) - open/close 36893 (time 1788630628.04 total 991.88 last 102.03) - open/close 37873 (time 1788630638.04 total 1001.89 last 97.99) - open/close 38823 (time 1788630648.05 total 1011.89 last 94.94) - open/close 39788 (time 1788630658.05 total 1021.89 last 96.50) - open/close 40000 (time 1788630660.05 total 1023.89 last 105.95) - open/close 40937 (time 1788630670.06 total 1033.90 last 93.65) - open/close 41908 (time 1788630680.06 total 1043.90 last 97.09) - open/close 42822 (time 1788630690.07 total 1053.91 last 91.29) - open/close 43718 (time 1788630700.08 total 1063.92 last 89.52) - open/close 44685 (time 1788630710.09 total 1073.93 last 96.58) - open/close 45614 (time 1788630720.10 total 1083.95 last 92.79) - open/close 46405 (time 1788630730.11 total 1093.95 last 79.02) - open/close 47241 (time 1788630740.12 total 1103.96 last 83.57) - open/close 48209 (time 1788630750.13 total 1113.97 last 96.65) - open/close 49150 (time 1788630760.14 total 1123.98 last 94.05) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1130.07 seconds: 43.92 ops/second - unlinked 0 (time 1788630769 ; total 0 ; last 0) - unlinked 10000 (time 1788630886 ; total 117 ; last 117) - unlinked 20000 (time 1788630994 ; total 225 ; last 108) - unlinked 30000 (time 1788631061 ; total 292 ; last 67) - unlinked 40000 (time 1788631128 ; total 359 ; last 67) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 425 seconds: 116.778824 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg631-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg631-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg631-server stop mds service on oleg631-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg631-server stop mds service on oleg631-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg631-server unloading modules via unload_modules_local on: 'oleg631-server' oleg631-server: oleg631-server.virtnet: executing unload_modules_local oleg631-server: kmemleak disabled modules unloaded. pdsh@oleg631-client: oleg631-client: ssh exited with exit code 2 pdsh@oleg631-client: oleg631-server: ssh exited with exit code 2 pdsh@oleg631-client: oleg631-server: ssh exited with exit code 2 pdsh@oleg631-client: oleg631-client: ssh exited with exit code 2