== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:07:32 (1788444452) start mds service on oleg154-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 3 07:51 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: kmemleak disabled oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: MODOPTS_LIBCFS= oleg154-server: Force libcfs to create 2 CPU partitions oleg154-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg154-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg154-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg154-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg154-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg154-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-OST0000 oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 113 (time 1788444562.39 total 10.01 last 11.29) - open/close 256 (time 1788444572.42 total 20.03 last 14.26) - open/close 388 (time 1788444583.36 total 30.98 last 12.06) - open/close 593 (time 1788444593.54 total 41.16 last 20.14) - open/close 722 (time 1788444604.23 total 51.85 last 12.07) - open/close 1001 (time 1788444616.26 total 63.88 last 23.18) - open/close 1128 (time 1788444628.92 total 76.54 last 10.03) - open/close 1367 (time 1788444638.96 total 86.58 last 23.80) - open/close 1603 (time 1788444648.97 total 96.59 last 23.58) - open/close 1853 (time 1788444659.08 total 106.69 last 24.74) - open/close 2098 (time 1788444669.35 total 116.97 last 23.84) - open/close 2243 (time 1788444679.37 total 126.99 last 14.47) - open/close 2394 (time 1788444689.65 total 137.27 last 14.69) - open/close 2649 (time 1788444700.56 total 148.18 last 23.38) - open/close 2908 (time 1788444711.25 total 158.87 last 24.23) - open/close 3073 (time 1788444721.29 total 168.91 last 16.44) - open/close 3188 (time 1788444731.78 total 179.40 last 10.96) - open/close 3397 (time 1788444741.79 total 189.41 last 20.89) - open/close 3529 (time 1788444751.79 total 199.41 last 13.20) - open/close 3703 (time 1788444762.50 total 210.12 last 16.25) - open/close 3833 (time 1788444772.51 total 220.13 last 12.98) - open/close 4165 (time 1788444782.53 total 230.14 last 33.16) - open/close 4324 (time 1788444792.89 total 240.51 last 15.34) - open/close 4552 (time 1788444803.90 total 251.52 last 20.71) - open/close 4797 (time 1788444814.27 total 261.89 last 23.63) - open/close 5025 (time 1788444824.28 total 271.90 last 22.77) - open/close 5271 (time 1788444834.71 total 282.33 last 23.60) - open/close 5520 (time 1788444846.10 total 293.72 last 21.86) - open/close 5806 (time 1788444856.10 total 303.72 last 28.59) - open/close 5950 (time 1788444866.13 total 313.74 last 14.37) - open/close 6026 (time 1788444876.56 total 324.18 last 7.28) - open/close 6172 (time 1788444886.56 total 334.18 last 14.60) - open/close 6353 (time 1788444897.29 total 344.91 last 16.88) - open/close 6553 (time 1788444907.35 total 354.97 last 19.87) - open/close 6799 (time 1788444917.63 total 365.25 last 23.93) - open/close 7025 (time 1788444927.69 total 375.31 last 22.47) - open/close 7211 (time 1788444937.98 total 385.60 last 18.08) - open/close 7391 (time 1788444948.32 total 395.94 last 17.41) - open/close 7718 (time 1788444958.33 total 405.95 last 32.67) - open/close 7855 (time 1788444968.34 total 415.96 last 13.69) - open/close 8069 (time 1788444978.94 total 426.56 last 20.18) - open/close 8211 (time 1788444988.98 total 436.59 last 14.16) - open/close 8400 (time 1788444998.98 total 446.60 last 18.88) - open/close 8632 (time 1788445009.00 total 456.62 last 23.16) - open/close 8876 (time 1788445019.36 total 466.98 last 23.56) - open/close 9054 (time 1788445030.22 total 477.84 last 16.39) - open/close 9289 (time 1788445040.72 total 488.34 last 22.39) - open/close 9477 (time 1788445051.44 total 499.06 last 17.54) - open/close 9658 (time 1788445061.44 total 509.06 last 18.10) - open/close 9952 (time 1788445072.27 total 519.89 last 27.15) - open/close 10164 (time 1788445083.13 total 530.75 last 19.51) - open/close 10377 (time 1788445093.13 total 540.75 last 21.30) - open/close 10540 (time 1788445103.15 total 550.77 last 16.27) - open/close 10813 (time 1788445113.16 total 560.78 last 27.29) - open/close 11106 (time 1788445123.81 total 571.43 last 27.51) - open/close 11360 (time 1788445133.81 total 581.43 last 25.40) - open/close 11610 (time 1788445144.64 total 592.26 last 23.09) - open/close 11852 (time 1788445155.21 total 602.83 last 22.89) - open/close 12060 (time 1788445165.75 total 613.37 last 19.72) - open/close 12229 (time 1788445175.76 total 623.38 last 16.89) - open/close 12362 (time 1788445185.92 total 633.54 last 13.09) - open/close 12615 (time 1788445196.16 total 643.78 last 24.71) - open/close 12877 (time 1788445209.15 total 656.77 last 20.16) - open/close 12948 (time 1788445219.18 total 666.80 last 7.08) - open/close 13253 (time 1788445229.19 total 676.81 last 30.48) - open/close 13488 (time 1788445239.53 total 687.14 last 22.73) - open/close 13622 (time 1788445250.06 total 697.68 last 12.71) - open/close 13896 (time 1788445260.07 total 707.68 last 27.40) - open/close 14159 (time 1788445270.86 total 718.48 last 24.37) - open/close 14315 (time 1788445281.36 total 728.98 last 14.85) - open/close 14461 (time 1788445291.38 total 739.00 last 14.58) - open/close 14578 (time 1788445301.53 total 749.15 last 11.52) - open/close 14753 (time 1788445311.54 total 759.16 last 17.48) - open/close 14975 (time 1788445322.45 total 770.07 last 20.36) - open/close 15236 (time 1788445332.82 total 780.44 last 25.16) - open/close 15495 (time 1788445343.11 total 790.73 last 25.18) - open/close 15590 (time 1788445358.38 total 806.00 last 6.22) - open/close 15746 (time 1788445368.40 total 816.01 last 15.58) - open/close 15948 (time 1788445378.50 total 826.11 last 20.00) - open/close 16067 (time 1788445389.31 total 836.93 last 11.00) - open/close 16195 (time 1788445399.31 total 846.93 last 12.80) - open/close 16316 (time 1788445410.12 total 857.73 last 11.20) - open/close 16478 (time 1788445420.12 total 867.74 last 16.19) - open/close 16729 (time 1788445430.13 total 877.74 last 25.09) - open/close 17004 (time 1788445441.16 total 888.78 last 24.92) - open/close 17193 (time 1788445451.17 total 898.79 last 18.88) - open/close 17322 (time 1788445461.58 total 909.20 last 12.39) - open/close 17439 (time 1788445471.59 total 919.20 last 11.70) - open/close 17542 (time 1788445481.87 total 929.49 last 10.01) - open/close 17785 (time 1788445492.43 total 940.05 last 23.02) - open/close 18075 (time 1788445503.38 total 951.00 last 26.48) - open/close 18323 (time 1788445514.14 total 961.76 last 23.05) - open/close 18596 (time 1788445524.61 total 972.23 last 26.07) - open/close 18856 (time 1788445535.01 total 982.62 last 25.01) - open/close 19018 (time 1788445545.46 total 993.08 last 15.50) - open/close 19135 (time 1788445556.18 total 1003.79 last 10.92) - open/close 19319 (time 1788445566.81 total 1014.43 last 17.30) - open/close 19472 (time 1788445577.05 total 1024.67 last 14.95) - open/close 19571 (time 1788445587.08 total 1034.70 last 9.87) - open/close 19755 (time 1788445597.14 total 1044.76 last 18.28) - open/close 20000 (time 1788445599.26 total 1046.88 last 115.61) - open/close 20776 (time 1788445609.27 total 1056.89 last 77.56) - open/close 21659 (time 1788445619.28 total 1066.90 last 88.20) - open/close 22607 (time 1788445629.28 total 1076.90 last 94.79) - open/close 23622 (time 1788445639.28 total 1086.90 last 101.47) - open/close 24785 (time 1788445649.29 total 1096.91 last 116.24) - open/close 25769 (time 1788445659.30 total 1106.92 last 98.27) - open/close 26720 (time 1788445669.31 total 1116.92 last 95.06) - open/close 27568 (time 1788445679.31 total 1126.93 last 84.76) - open/close 28399 (time 1788445689.32 total 1136.94 last 83.04) - open/close 29357 (time 1788445699.33 total 1146.95 last 95.69) - open/close 30000 (time 1788445708.15 total 1155.77 last 72.88) - open/close 30986 (time 1788445718.16 total 1165.77 last 98.55) - open/close 31763 (time 1788445728.16 total 1175.78 last 77.67) - open/close 32765 (time 1788445738.16 total 1185.78 last 100.17) - open/close 33843 (time 1788445748.18 total 1195.80 last 107.57) - open/close 34894 (time 1788445758.20 total 1205.82 last 104.92) - open/close 35813 (time 1788445768.22 total 1215.84 last 91.69) - open/close 36756 (time 1788445778.24 total 1225.85 last 94.19) - open/close 37612 (time 1788445788.24 total 1235.86 last 85.55) - open/close 38299 (time 1788445798.25 total 1245.86 last 68.67) - open/close 39140 (time 1788445808.25 total 1255.87 last 84.10) - open/close 40000 (time 1788445816.22 total 1263.84 last 107.80) - open/close 40949 (time 1788445826.23 total 1273.84 last 94.88) - open/close 42011 (time 1788445836.24 total 1283.86 last 106.09) - open/close 42970 (time 1788445846.27 total 1293.88 last 95.62) - open/close 43747 (time 1788445856.27 total 1303.88 last 77.70) - open/close 44694 (time 1788445866.27 total 1313.89 last 94.64) - open/close 45617 (time 1788445876.28 total 1323.90 last 92.23) - open/close 46719 (time 1788445886.29 total 1333.91 last 110.10) - open/close 47677 (time 1788445896.29 total 1343.91 last 95.74) - open/close 48799 (time 1788445906.30 total 1353.92 last 112.14) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1362.92 seconds: 36.42 ops/second - unlinked 0 (time 1788445920 ; total 0 ; last 0) - unlinked 10000 (time 1788446032 ; total 112 ; last 112) - unlinked 20000 (time 1788446139 ; total 219 ; last 107) - unlinked 30000 (time 1788446196 ; total 276 ; last 57) - unlinked 40000 (time 1788446252 ; total 332 ; last 56) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 384 seconds: 129.247391 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg154-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg154-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg154-server stop mds service on oleg154-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg154-server stop mds service on oleg154-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg154-server unloading modules via unload_modules_local on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing unload_modules_local oleg154-server: kmemleak disabled modules unloaded. pdsh@oleg154-client: oleg154-client: ssh exited with exit code 2 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 2 pdsh@oleg154-client: oleg154-client: ssh exited with exit code 2 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 2