== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 06:21:54 (1789381314) start mds service on oleg218-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 14 04:14 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg218-server' oleg218-server: oleg218-server.virtnet: executing load_modules_local oleg218-server: kmemleak disabled oleg218-server: Loading modules from /home/green/git/lustre-release/lustre oleg218-server: detected 4 online CPUs by sysfs oleg218-server: MODOPTS_LIBCFS= oleg218-server: Force libcfs to create 2 CPU partitions oleg218-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg218-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg218-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg218-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg218-server: params: OBD_IOC_LLOG_PRINT failed: No such file or directory pdsh@oleg218-client: oleg218-server: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-server: ssh exited with exit code 123 allow_register=unlimited start mds service on oleg218-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg218-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0000 oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 134 (time 1789381444.75 total 10.02 last 13.37) - open/close 339 (time 1789381454.75 total 20.03 last 20.50) - open/close 469 (time 1789381464.95 total 30.22 last 12.75) - open/close 601 (time 1789381475.29 total 40.57 last 12.76) - open/close 800 (time 1789381485.70 total 50.97 last 19.13) - open/close 1030 (time 1789381496.65 total 61.93 last 21.00) - open/close 1097 (time 1789381508.81 total 74.09 last 5.51) - open/close 1266 (time 1789381518.83 total 84.11 last 16.87) - open/close 1398 (time 1789381529.29 total 94.57 last 12.62) - open/close 1479 (time 1789381539.91 total 105.19 last 7.63) - open/close 1603 (time 1789381550.59 total 115.86 last 11.62) - open/close 1672 (time 1789381561.14 total 126.41 last 6.54) - open/close 1798 (time 1789381572.16 total 137.43 last 11.43) - open/close 1924 (time 1789381583.80 total 149.08 last 10.82) - open/close 1950 (time 1789381594.75 total 160.02 last 2.38) - open/close 2139 (time 1789381604.76 total 170.03 last 18.88) - open/close 2328 (time 1789381614.86 total 180.13 last 18.71) - open/close 2595 (time 1789381624.86 total 190.14 last 26.69) - open/close 2750 (time 1789381634.87 total 200.14 last 15.49) - open/close 2916 (time 1789381644.89 total 210.16 last 16.57) - open/close 2998 (time 1789381655.62 total 220.89 last 7.64) - open/close 3155 (time 1789381666.38 total 231.65 last 14.59) - open/close 3266 (time 1789381676.39 total 241.67 last 11.08) - open/close 3409 (time 1789381687.36 total 252.63 last 13.04) - open/close 3490 (time 1789381697.52 total 262.80 last 7.97) - open/close 3589 (time 1789381707.55 total 272.82 last 9.88) - open/close 3702 (time 1789381717.58 total 282.86 last 11.26) - open/close 3800 (time 1789381727.62 total 292.90 last 9.76) - open/close 3974 (time 1789381737.64 total 302.91 last 17.37) - open/close 4112 (time 1789381748.38 total 313.66 last 12.84) - open/close 4230 (time 1789381758.40 total 323.67 last 11.78) - open/close 4401 (time 1789381769.22 total 334.50 last 15.79) - open/close 4523 (time 1789381779.23 total 344.51 last 12.19) - open/close 4667 (time 1789381789.82 total 355.09 last 13.61) - open/close 4826 (time 1789381800.44 total 365.71 last 14.97) - open/close 4953 (time 1789381810.63 total 375.90 last 12.46) - open/close 5143 (time 1789381820.63 total 385.91 last 18.99) - open/close 5270 (time 1789381830.63 total 395.91 last 12.70) - open/close 5328 (time 1789381841.35 total 406.63 last 5.41) - open/close 5524 (time 1789381851.35 total 416.63 last 19.60) - open/close 5615 (time 1789381861.37 total 426.64 last 9.08) - open/close 5752 (time 1789381871.86 total 437.13 last 13.06) - open/close 5976 (time 1789381882.76 total 448.03 last 20.55) - open/close 6132 (time 1789381893.76 total 459.03 last 14.19) - open/close 6324 (time 1789381903.93 total 469.20 last 18.88) - open/close 6481 (time 1789381916.47 total 481.74 last 12.52) - open/close 6639 (time 1789381926.90 total 492.17 last 15.14) - open/close 6814 (time 1789381936.92 total 502.19 last 17.47) - open/close 7072 (time 1789381946.92 total 512.20 last 25.79) - open/close 7220 (time 1789381957.69 total 522.96 last 13.75) - open/close 7452 (time 1789381968.58 total 533.85 last 21.31) - open/close 7632 (time 1789381979.11 total 544.39 last 17.08) - open/close 7757 (time 1789381989.12 total 554.39 last 12.49) - open/close 7947 (time 1789381999.19 total 564.46 last 18.88) - open/close 8187 (time 1789382009.21 total 574.49 last 23.93) - open/close 8437 (time 1789382019.77 total 585.05 last 23.68) - open/close 8645 (time 1789382031.10 total 596.37 last 18.36) - open/close 8872 (time 1789382041.82 total 607.09 last 21.17) - open/close 9023 (time 1789382052.30 total 617.57 last 14.41) - open/close 9192 (time 1789382062.30 total 627.58 last 16.89) - open/close 9415 (time 1789382073.10 total 638.37 last 20.66) - open/close 9677 (time 1789382084.03 total 649.30 last 23.96) - open/close 9810 (time 1789382094.68 total 659.95 last 12.49) - open/close 9949 (time 1789382105.02 total 670.30 last 13.43) - open/close 10000 (time 1789382107.54 total 672.81 last 20.27) - open/close 10142 (time 1789382118.08 total 683.35 last 13.48) - open/close 10402 (time 1789382128.08 total 693.35 last 25.99) - open/close 10556 (time 1789382138.08 total 703.36 last 15.40) - open/close 10745 (time 1789382148.40 total 713.68 last 18.32) - open/close 10884 (time 1789382158.78 total 724.05 last 13.39) - open/close 11034 (time 1789382169.11 total 734.39 last 14.51) - open/close 11270 (time 1789382179.13 total 744.40 last 23.57) - open/close 11427 (time 1789382189.70 total 754.97 last 14.85) - open/close 11531 (time 1789382202.37 total 767.64 last 8.21) - open/close 11613 (time 1789382212.79 total 778.06 last 7.87) - open/close 11749 (time 1789382223.30 total 788.58 last 12.93) - open/close 11826 (time 1789382233.35 total 798.62 last 7.66) - open/close 11970 (time 1789382243.70 total 808.97 last 13.92) - open/close 12172 (time 1789382253.71 total 818.98 last 20.17) - open/close 12412 (time 1789382263.72 total 828.99 last 23.97) - open/close 12486 (time 1789382274.75 total 840.03 last 6.71) - open/close 12720 (time 1789382284.76 total 850.03 last 23.39) - open/close 12948 (time 1789382295.22 total 860.49 last 21.79) - open/close 13090 (time 1789382306.29 total 871.56 last 12.83) - open/close 13346 (time 1789382316.30 total 881.57 last 25.56) - open/close 13563 (time 1789382326.31 total 891.59 last 21.68) - open/close 13705 (time 1789382337.27 total 902.55 last 12.95) - open/close 13916 (time 1789382347.61 total 912.89 last 20.41) - open/close 14032 (time 1789382358.10 total 923.37 last 11.06) - open/close 14182 (time 1789382368.10 total 933.37 last 14.99) - open/close 14413 (time 1789382378.12 total 943.40 last 23.05) - open/close 14647 (time 1789382389.85 total 955.12 last 19.96) - open/close 14824 (time 1789382400.92 total 966.20 last 15.98) - open/close 15017 (time 1789382410.93 total 976.20 last 19.29) - open/close 15123 (time 1789382421.55 total 986.82 last 9.98) - open/close 15332 (time 1789382431.92 total 997.19 last 20.16) - open/close 15435 (time 1789382442.71 total 1007.99 last 9.54) - open/close 15570 (time 1789382452.73 total 1018.01 last 13.48) - open/close 15821 (time 1789382462.76 total 1028.03 last 25.04) - open/close 15980 (time 1789382475.40 total 1040.67 last 12.58) - open/close 16212 (time 1789382486.35 total 1051.63 last 21.18) - open/close 16458 (time 1789382496.36 total 1061.64 last 24.58) - open/close 16719 (time 1789382507.03 total 1072.31 last 24.46) - open/close 17068 (time 1789382517.04 total 1082.32 last 34.86) - open/close 17269 (time 1789382528.02 total 1093.29 last 18.32) - open/close 17486 (time 1789382538.03 total 1103.30 last 21.68) - open/close 18190 (time 1789382548.04 total 1113.31 last 70.33) - open/close 18975 (time 1789382558.04 total 1123.32 last 78.45) - open/close 19816 (time 1789382568.04 total 1133.32 last 84.10) - open/close 20000 (time 1789382570.34 total 1135.62 last 80.05) - open/close 20777 (time 1789382580.34 total 1145.62 last 77.69) - open/close 21573 (time 1789382590.36 total 1155.64 last 79.44) - open/close 22400 (time 1789382600.37 total 1165.65 last 82.63) - open/close 23249 (time 1789382610.38 total 1175.65 last 84.84) - open/close 24006 (time 1789382620.38 total 1185.66 last 75.68) - open/close 24760 (time 1789382630.39 total 1195.66 last 75.36) - open/close 25604 (time 1789382640.39 total 1205.67 last 84.35) - open/close 26296 (time 1789382650.40 total 1215.67 last 69.16) - open/close 27109 (time 1789382660.40 total 1225.67 last 81.30) - open/close 27830 (time 1789382670.40 total 1235.68 last 72.08) - open/close 28637 (time 1789382680.40 total 1245.68 last 80.68) - open/close 29392 (time 1789382690.42 total 1255.69 last 75.40) - open/close 30000 (time 1789382697.87 total 1263.14 last 81.62) - open/close 30688 (time 1789382707.87 total 1273.14 last 68.79) - open/close 31307 (time 1789382717.89 total 1283.17 last 61.75) - open/close 32063 (time 1789382727.90 total 1293.17 last 75.57) - open/close 32731 (time 1789382737.91 total 1303.18 last 66.74) - open/close 33377 (time 1789382747.91 total 1313.19 last 64.56) - open/close 34124 (time 1789382757.92 total 1323.19 last 74.65) - open/close 34936 (time 1789382767.92 total 1333.20 last 81.18) - open/close 35592 (time 1789382777.93 total 1343.20 last 65.55) - open/close 36307 (time 1789382787.94 total 1353.21 last 71.45) - open/close 37101 (time 1789382797.94 total 1363.21 last 79.39) - open/close 37869 (time 1789382807.95 total 1373.23 last 76.69) - open/close 38459 (time 1789382817.97 total 1383.25 last 58.87) - open/close 39220 (time 1789382827.98 total 1393.26 last 76.05) - open/close 39967 (time 1789382837.99 total 1403.27 last 74.62) - open/close 40613 (time 1789382848.01 total 1413.29 last 64.47) - open/close 41241 (time 1789382858.01 total 1423.29 last 62.80) - open/close 41965 (time 1789382868.02 total 1433.29 last 72.37) - open/close 42543 (time 1789382878.03 total 1443.30 last 57.72) - open/close 43270 (time 1789382888.03 total 1453.31 last 72.69) - open/close 43981 (time 1789382898.04 total 1463.32 last 71.02) - open/close 44621 (time 1789382908.04 total 1473.32 last 64.00) - open/close 45392 (time 1789382918.05 total 1483.32 last 77.07) - open/close 46088 (time 1789382928.06 total 1493.33 last 69.52) - open/close 46840 (time 1789382938.08 total 1503.35 last 75.08) - open/close 47524 (time 1789382948.08 total 1513.36 last 68.34) - open/close 48218 (time 1789382958.10 total 1523.37 last 69.29) - open/close 48922 (time 1789382968.11 total 1533.38 last 70.35) - open/close 49625 (time 1789382978.11 total 1543.38 last 70.27) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1543.45 seconds: 32.16 ops/second - unlinked 0 (time 1789382983 ; total 0 ; last 0) - unlinked 10000 (time 1789383135 ; total 152 ; last 152) - unlinked 20000 (time 1789383277 ; total 294 ; last 142) - unlinked 30000 (time 1789383417 ; total 434 ; last 140) - unlinked 40000 (time 1789383560 ; total 577 ; last 143) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 714 seconds: 69.511208 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg218-client.virtnet /mnt/lustre (opts:) oleg218-server: params: OBD_IOC_LLOG_PRINT failed: No such file or directory pdsh@oleg218-client: oleg218-server: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-server: ssh exited with exit code 123 allow_register=unlimited stop ost1 service on oleg218-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg218-server stop mds service on oleg218-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg218-server stop mds service on oleg218-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg218-server unloading modules via unload_modules_local on: 'oleg218-server' oleg218-server: oleg218-server.virtnet: executing unload_modules_local oleg218-server: kmemleak disabled modules unloaded. pdsh@oleg218-client: oleg218-client: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-server: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-client: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-server: ssh exited with exit code 2