== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:59:03 (1790949543) start mds service on oleg253-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1946968 Oct 2 03:41 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg253-server' oleg253-server: oleg253-server.virtnet: executing load_modules_local oleg253-server: kmemleak disabled oleg253-server: Loading modules from /home/green/git/lustre-release/lustre oleg253-server: detected 4 online CPUs by sysfs oleg253-server: MODOPTS_LIBCFS= oleg253-server: Force libcfs to create 2 CPU partitions oleg253-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg253-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg253-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg253-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg253-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg253-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-OST0000 oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg253-client.virtnet: -o user_xattr,flock 192.168.202.153@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 224 (time 1790949655.53 total 10.00 last 22.39) - open/close 381 (time 1790949665.55 total 20.02 last 15.67) - open/close 645 (time 1790949676.13 total 30.60 last 24.95) - open/close 867 (time 1790949687.01 total 41.48 last 20.40) - open/close 1054 (time 1790949697.18 total 51.65 last 18.39) - open/close 1254 (time 1790949707.20 total 61.67 last 19.97) - open/close 1470 (time 1790949717.21 total 71.68 last 21.57) - open/close 1676 (time 1790949727.21 total 81.68 last 20.59) - open/close 1803 (time 1790949737.22 total 91.69 last 12.70) - open/close 2040 (time 1790949747.57 total 102.04 last 22.89) - open/close 2202 (time 1790949758.09 total 112.56 last 15.40) - open/close 2425 (time 1790949768.46 total 122.93 last 21.51) - open/close 2718 (time 1790949778.83 total 133.30 last 28.26) - open/close 2981 (time 1790949788.84 total 143.31 last 26.28) - open/close 3189 (time 1790949798.85 total 153.32 last 20.78) - open/close 3402 (time 1790949809.75 total 164.22 last 19.53) - open/close 3670 (time 1790949819.76 total 174.23 last 26.79) - open/close 3816 (time 1790949829.77 total 184.24 last 14.58) - open/close 4129 (time 1790949839.79 total 194.26 last 31.25) - open/close 4296 (time 1790949850.28 total 204.74 last 15.92) - open/close 4505 (time 1790949860.31 total 214.78 last 20.83) - open/close 4794 (time 1790949870.31 total 224.78 last 28.89) - open/close 5065 (time 1790949880.33 total 234.80 last 27.05) - open/close 5310 (time 1790949890.83 total 245.30 last 23.33) - open/close 5489 (time 1790949900.84 total 255.31 last 17.88) - open/close 5693 (time 1790949911.90 total 266.36 last 18.45) - open/close 5873 (time 1790949922.35 total 276.82 last 17.22) - open/close 6143 (time 1790949932.36 total 286.83 last 26.98) - open/close 6404 (time 1790949942.67 total 297.14 last 25.31) - open/close 6594 (time 1790949953.42 total 307.89 last 17.67) - open/close 6762 (time 1790949963.74 total 318.21 last 16.28) - open/close 6954 (time 1790949974.15 total 328.62 last 18.44) - open/close 7133 (time 1790949988.31 total 342.78 last 12.64) - open/close 7398 (time 1790949998.68 total 353.15 last 25.55) - open/close 7644 (time 1790950008.69 total 363.16 last 24.57) - open/close 7766 (time 1790950020.75 total 375.22 last 10.12) - open/close 7907 (time 1790950032.86 total 387.33 last 11.64) - open/close 8053 (time 1790950042.88 total 397.35 last 14.58) - open/close 8234 (time 1790950053.27 total 407.74 last 17.42) - open/close 8479 (time 1790950064.29 total 418.76 last 22.23) - open/close 8651 (time 1790950074.39 total 428.86 last 17.03) - open/close 8868 (time 1790950084.69 total 439.15 last 21.07) - open/close 9173 (time 1790950094.69 total 449.16 last 30.49) - open/close 9430 (time 1790950105.49 total 459.96 last 23.80) - open/close 9581 (time 1790950115.49 total 469.96 last 15.10) - open/close 9761 (time 1790950125.49 total 479.96 last 18.00) - open/close 10000 (time 1790950131.49 total 485.96 last 39.84) - open/close 10208 (time 1790950141.53 total 496.00 last 20.71) - open/close 10396 (time 1790950152.27 total 506.74 last 17.51) - open/close 10554 (time 1790950163.22 total 517.69 last 14.42) - open/close 10657 (time 1790950173.41 total 527.88 last 10.11) - open/close 10869 (time 1790950183.41 total 537.88 last 21.19) - open/close 11131 (time 1790950193.87 total 548.34 last 25.06) - open/close 11428 (time 1790950204.88 total 559.35 last 26.98) - open/close 11711 (time 1790950214.88 total 569.35 last 28.30) - open/close 11991 (time 1790950224.90 total 579.37 last 27.94) - open/close 12227 (time 1790950235.69 total 590.16 last 21.88) - open/close 12541 (time 1790950246.69 total 601.16 last 28.55) - open/close 12639 (time 1790950257.56 total 612.02 last 9.02) - open/close 12770 (time 1790950267.58 total 622.05 last 13.06) - open/close 12878 (time 1790950278.35 total 632.82 last 10.03) - open/close 13029 (time 1790950288.37 total 642.84 last 15.07) - open/close 13375 (time 1790950299.27 total 653.74 last 31.72) - open/close 13616 (time 1790950310.29 total 664.76 last 21.88) - open/close 13888 (time 1790950320.83 total 675.29 last 25.81) - open/close 14098 (time 1790950333.91 total 688.38 last 16.04) - open/close 14161 (time 1790950345.93 total 700.40 last 5.24) - open/close 14281 (time 1790950357.46 total 711.93 last 10.41) - open/close 14399 (time 1790950367.96 total 722.43 last 11.24) - open/close 14630 (time 1790950377.97 total 732.44 last 23.07) - open/close 14812 (time 1790950388.24 total 742.71 last 17.72) - open/close 15038 (time 1790950399.78 total 754.25 last 19.58) - open/close 15284 (time 1790950409.93 total 764.40 last 24.24) - open/close 15472 (time 1790950420.92 total 775.39 last 17.12) - open/close 15667 (time 1790950430.92 total 785.39 last 19.50) - open/close 15868 (time 1790950441.16 total 795.63 last 19.63) - open/close 16163 (time 1790950451.17 total 805.64 last 29.46) - open/close 16424 (time 1790950461.51 total 815.98 last 25.24) - open/close 16673 (time 1790950471.52 total 825.99 last 24.87) - open/close 16969 (time 1790950481.54 total 836.01 last 29.55) - open/close 17209 (time 1790950491.56 total 846.03 last 23.95) - open/close 17445 (time 1790950501.56 total 856.03 last 23.60) - open/close 17670 (time 1790950511.58 total 866.05 last 22.46) - open/close 17974 (time 1790950522.39 total 876.86 last 28.11) - open/close 18190 (time 1790950532.40 total 886.87 last 21.58) - open/close 18438 (time 1790950542.41 total 896.88 last 24.78) - open/close 18662 (time 1790950552.85 total 907.32 last 21.46) - open/close 18858 (time 1790950562.90 total 917.37 last 19.50) - open/close 19136 (time 1790950572.91 total 927.38 last 27.78) - open/close 19365 (time 1790950582.92 total 937.39 last 22.88) - open/close 19705 (time 1790950596.02 total 950.49 last 25.96) - open/close 19823 (time 1790950607.09 total 961.56 last 10.66) - open/close 20000 (time 1790950615.19 total 969.66 last 21.83) - open/close 20863 (time 1790950625.21 total 979.68 last 86.19) - open/close 21738 (time 1790950635.22 total 989.68 last 87.41) - open/close 22704 (time 1790950645.22 total 999.69 last 96.52) - open/close 23624 (time 1790950655.23 total 1009.70 last 91.97) - open/close 24498 (time 1790950665.24 total 1019.71 last 87.29) - open/close 25409 (time 1790950675.26 total 1029.73 last 90.94) - open/close 26274 (time 1790950685.27 total 1039.73 last 86.43) - open/close 27118 (time 1790950695.27 total 1049.74 last 84.35) - open/close 28129 (time 1790950705.28 total 1059.75 last 101.01) - open/close 29158 (time 1790950715.29 total 1069.75 last 102.85) - open/close 30000 (time 1790950722.99 total 1077.45 last 109.36) - open/close 31139 (time 1790950732.99 total 1087.46 last 113.81) - open/close 32166 (time 1790950743.00 total 1097.47 last 102.66) - open/close 32923 (time 1790950753.01 total 1107.48 last 75.62) - open/close 33778 (time 1790950763.01 total 1117.48 last 85.50) - open/close 34764 (time 1790950773.01 total 1127.48 last 98.58) - open/close 35821 (time 1790950783.02 total 1137.48 last 105.65) - open/close 36810 (time 1790950793.02 total 1147.48 last 98.90) - open/close 37724 (time 1790950803.02 total 1157.49 last 91.33) - open/close 38717 (time 1790950813.03 total 1167.50 last 99.21) - open/close 39724 (time 1790950823.04 total 1177.51 last 100.60) - open/close 40000 (time 1790950825.86 total 1180.33 last 97.82) - open/close 40998 (time 1790950835.87 total 1190.34 last 99.76) - open/close 42045 (time 1790950845.87 total 1200.34 last 104.65) - open/close 42985 (time 1790950855.87 total 1210.34 last 94.00) - open/close 43888 (time 1790950865.88 total 1220.35 last 90.23) - open/close 44864 (time 1790950875.89 total 1230.36 last 97.55) - open/close 45761 (time 1790950885.89 total 1240.36 last 89.68) - open/close 46693 (time 1790950895.89 total 1250.36 last 93.20) - open/close 47686 (time 1790950905.90 total 1260.36 last 99.23) - open/close 48701 (time 1790950915.90 total 1270.36 last 101.50) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1280.33 seconds: 38.76 ops/second - unlinked 0 (time 1790950929 ; total 0 ; last 0) - unlinked 10000 (time 1790951028 ; total 99 ; last 99) - unlinked 20000 (time 1790951121 ; total 192 ; last 93) - unlinked 30000 (time 1790951181 ; total 252 ; last 60) - unlinked 40000 (time 1790951239 ; total 310 ; last 58) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 375 seconds: 132.349335 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg253-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg253-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg253-server stop mds service on oleg253-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg253-server stop mds service on oleg253-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg253-server unloading modules via unload_modules_local on: 'oleg253-server' oleg253-server: oleg253-server.virtnet: executing unload_modules_local oleg253-server: kmemleak disabled modules unloaded. pdsh@oleg253-client: oleg253-client: ssh exited with exit code 2 pdsh@oleg253-client: oleg253-server: ssh exited with exit code 2 pdsh@oleg253-client: oleg253-server: ssh exited with exit code 2 pdsh@oleg253-client: oleg253-client: ssh exited with exit code 2