== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:39:36 (1788467976) start mds service on oleg349-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 3 03:55 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg349-server' oleg349-server: oleg349-server.virtnet: executing load_modules_local oleg349-server: kmemleak disabled oleg349-server: Loading modules from /home/green/git/lustre-release/lustre oleg349-server: detected 4 online CPUs by sysfs oleg349-server: MODOPTS_LIBCFS= oleg349-server: Force libcfs to create 2 CPU partitions oleg349-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg349-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg349-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg349-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg349-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg349-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 186 (time 1788468092.03 total 10.00 last 18.60) - open/close 327 (time 1788468102.03 total 20.01 last 14.09) - open/close 479 (time 1788468112.03 total 30.01 last 15.20) - open/close 739 (time 1788468122.79 total 40.76 last 24.17) - open/close 951 (time 1788468132.80 total 50.77 last 21.19) - open/close 1143 (time 1788468144.30 total 62.27 last 16.70) - open/close 1328 (time 1788468155.25 total 73.22 last 16.89) - open/close 1542 (time 1788468165.67 total 83.64 last 20.54) - open/close 1808 (time 1788468175.67 total 93.65 last 26.59) - open/close 2039 (time 1788468186.02 total 103.99 last 22.33) - open/close 2237 (time 1788468196.26 total 114.24 last 19.32) - open/close 2483 (time 1788468208.21 total 126.18 last 20.60) - open/close 2713 (time 1788468221.02 total 138.99 last 17.95) - open/close 2852 (time 1788468231.26 total 149.23 last 13.58) - open/close 2993 (time 1788468241.72 total 159.70 last 13.47) - open/close 3061 (time 1788468251.82 total 169.80 last 6.73) - open/close 3239 (time 1788468261.84 total 179.81 last 17.78) - open/close 3366 (time 1788468271.86 total 189.83 last 12.68) - open/close 3558 (time 1788468282.72 total 200.70 last 17.67) - open/close 3852 (time 1788468293.76 total 211.73 last 26.65) - open/close 4001 (time 1788468303.97 total 221.94 last 14.59) - open/close 4168 (time 1788468313.97 total 231.94 last 16.69) - open/close 4264 (time 1788468324.25 total 242.22 last 9.34) - open/close 4478 (time 1788468334.95 total 252.92 last 20.00) - open/close 4631 (time 1788468347.87 total 265.84 last 11.84) - open/close 4753 (time 1788468358.63 total 276.60 last 11.34) - open/close 4882 (time 1788468368.64 total 286.61 last 12.89) - open/close 5060 (time 1788468379.42 total 297.39 last 16.50) - open/close 5265 (time 1788468389.42 total 307.40 last 20.49) - open/close 5481 (time 1788468400.16 total 318.13 last 20.12) - open/close 5739 (time 1788468410.98 total 328.95 last 23.85) - open/close 5947 (time 1788468420.98 total 338.95 last 20.79) - open/close 6100 (time 1788468431.03 total 349.00 last 15.23) - open/close 6313 (time 1788468441.06 total 359.03 last 21.23) - open/close 6531 (time 1788468451.37 total 369.34 last 21.15) - open/close 6769 (time 1788468461.37 total 379.34 last 23.80) - open/close 7031 (time 1788468471.38 total 389.36 last 26.16) - open/close 7119 (time 1788468481.50 total 399.47 last 8.70) - open/close 7191 (time 1788468491.50 total 409.47 last 7.20) - open/close 7383 (time 1788468502.61 total 420.58 last 17.28) - open/close 7544 (time 1788468512.63 total 430.60 last 16.06) - open/close 7795 (time 1788468522.64 total 440.62 last 25.07) - open/close 7939 (time 1788468532.64 total 450.62 last 14.40) - open/close 8211 (time 1788468543.52 total 461.49 last 25.02) - open/close 8414 (time 1788468554.29 total 472.27 last 18.83) - open/close 8632 (time 1788468564.52 total 482.50 last 21.31) - open/close 8877 (time 1788468574.56 total 492.53 last 24.41) - open/close 9059 (time 1788468584.57 total 502.54 last 18.18) - open/close 9181 (time 1788468595.10 total 513.07 last 11.59) - open/close 9386 (time 1788468605.98 total 523.95 last 18.84) - open/close 9601 (time 1788468616.79 total 534.76 last 19.88) - open/close 9857 (time 1788468627.45 total 545.42 last 24.01) - open/close 10000 (time 1788468633.40 total 551.37 last 24.06) - open/close 10130 (time 1788468644.40 total 562.38 last 11.81) - open/close 10382 (time 1788468655.47 total 573.45 last 22.76) - open/close 10627 (time 1788468665.89 total 583.86 last 23.53) - open/close 10813 (time 1788468676.09 total 594.06 last 18.23) - open/close 10987 (time 1788468686.69 total 604.66 last 16.42) - open/close 11194 (time 1788468696.70 total 614.67 last 20.67) - open/close 11338 (time 1788468706.99 total 624.96 last 13.99) - open/close 11543 (time 1788468718.00 total 635.97 last 18.62) - open/close 11777 (time 1788468728.01 total 645.98 last 23.39) - open/close 11998 (time 1788468740.19 total 658.16 last 18.14) - open/close 12246 (time 1788468753.13 total 671.10 last 19.17) - open/close 12403 (time 1788468763.13 total 681.11 last 15.69) - open/close 12572 (time 1788468773.13 total 691.11 last 16.90) - open/close 12714 (time 1788468783.26 total 701.24 last 14.02) - open/close 12972 (time 1788468793.77 total 711.74 last 24.56) - open/close 13195 (time 1788468804.64 total 722.61 last 20.52) - open/close 13424 (time 1788468814.65 total 732.62 last 22.88) - open/close 13653 (time 1788468824.65 total 742.62 last 22.89) - open/close 13908 (time 1788468834.65 total 752.62 last 25.50) - open/close 14172 (time 1788468844.90 total 762.87 last 25.75) - open/close 14342 (time 1788468855.15 total 773.12 last 16.59) - open/close 14619 (time 1788468865.16 total 783.13 last 27.67) - open/close 14804 (time 1788468875.16 total 793.13 last 18.50) - open/close 14957 (time 1788468885.91 total 803.88 last 14.23) - open/close 15110 (time 1788468896.29 total 814.26 last 14.75) - open/close 15379 (time 1788468906.31 total 824.28 last 26.84) - open/close 15620 (time 1788468916.65 total 834.63 last 23.29) - open/close 15775 (time 1788468926.66 total 844.63 last 15.49) - open/close 16022 (time 1788468936.69 total 854.67 last 24.62) - open/close 16258 (time 1788468946.70 total 864.67 last 23.59) - open/close 16386 (time 1788468956.71 total 874.68 last 12.79) - open/close 16577 (time 1788468969.28 total 887.25 last 15.19) - open/close 16688 (time 1788468980.20 total 898.17 last 10.17) - open/close 16793 (time 1788468990.88 total 908.85 last 9.83) - open/close 16930 (time 1788469003.94 total 921.91 last 10.49) - open/close 17187 (time 1788469014.12 total 932.09 last 25.24) - open/close 17442 (time 1788469024.13 total 942.10 last 25.48) - open/close 17624 (time 1788469034.34 total 952.31 last 17.82) - open/close 17842 (time 1788469044.36 total 962.33 last 21.76) - open/close 18082 (time 1788469054.44 total 972.41 last 23.81) - open/close 18315 (time 1788469064.45 total 982.42 last 23.28) - open/close 18805 (time 1788469074.46 total 992.43 last 48.96) - open/close 19614 (time 1788469084.46 total 1002.43 last 80.88) - open/close 20000 (time 1788469089.57 total 1007.54 last 75.60) - open/close 20599 (time 1788469099.57 total 1017.54 last 59.86) - open/close 21235 (time 1788469109.60 total 1027.57 last 63.43) - open/close 21974 (time 1788469119.61 total 1037.58 last 73.84) - open/close 22710 (time 1788469129.61 total 1047.58 last 73.56) - open/close 23513 (time 1788469139.62 total 1057.59 last 80.21) - open/close 24293 (time 1788469149.63 total 1067.60 last 77.97) - open/close 25038 (time 1788469159.63 total 1077.60 last 74.49) - open/close 25767 (time 1788469169.64 total 1087.61 last 72.85) - open/close 26451 (time 1788469179.64 total 1097.61 last 68.36) - open/close 27164 (time 1788469189.65 total 1107.62 last 71.24) - open/close 27876 (time 1788469199.67 total 1117.64 last 71.09) - open/close 28637 (time 1788469209.67 total 1127.64 last 76.07) - open/close 29446 (time 1788469219.70 total 1137.67 last 80.65) - open/close 30000 (time 1788469227.26 total 1145.23 last 73.30) - open/close 30673 (time 1788469237.27 total 1155.24 last 67.23) - open/close 31379 (time 1788469247.27 total 1165.24 last 70.58) - open/close 32163 (time 1788469257.28 total 1175.25 last 78.33) - open/close 32907 (time 1788469267.28 total 1185.26 last 74.37) - open/close 33706 (time 1788469277.28 total 1195.26 last 79.89) - open/close 34492 (time 1788469287.30 total 1205.27 last 78.50) - open/close 35177 (time 1788469297.30 total 1215.27 last 68.47) - open/close 35849 (time 1788469307.31 total 1225.28 last 67.15) - open/close 36598 (time 1788469317.32 total 1235.29 last 74.83) - open/close 37398 (time 1788469327.32 total 1245.30 last 79.95) - open/close 38145 (time 1788469337.33 total 1255.31 last 74.64) - open/close 38859 (time 1788469347.35 total 1265.33 last 71.26) - open/close 39591 (time 1788469357.36 total 1275.33 last 73.13) - open/close 40000 (time 1788469363.88 total 1281.85 last 62.79) - open/close 40678 (time 1788469373.88 total 1291.85 last 67.76) - open/close 41471 (time 1788469383.89 total 1301.86 last 79.28) - open/close 42170 (time 1788469393.90 total 1311.87 last 69.78) - open/close 42854 (time 1788469403.90 total 1321.88 last 68.39) - open/close 43522 (time 1788469413.92 total 1331.89 last 66.73) - open/close 44123 (time 1788469423.92 total 1341.89 last 60.07) - open/close 44797 (time 1788469433.93 total 1351.90 last 67.34) - open/close 45501 (time 1788469443.94 total 1361.91 last 70.33) - open/close 46075 (time 1788469453.94 total 1371.91 last 57.39) - open/close 46619 (time 1788469463.94 total 1381.91 last 54.39) - open/close 47274 (time 1788469473.95 total 1391.93 last 65.42) - open/close 47868 (time 1788469483.95 total 1401.93 last 59.39) - open/close 48498 (time 1788469493.96 total 1411.93 last 62.99) - open/close 49175 (time 1788469503.96 total 1421.94 last 67.65) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1429.04 seconds: 34.73 ops/second - unlinked 0 (time 1788469516 ; total 0 ; last 0) - unlinked 10000 (time 1788469660 ; total 144 ; last 144) - unlinked 20000 (time 1788469812 ; total 296 ; last 152) - unlinked 30000 (time 1788469978 ; total 462 ; last 166) - unlinked 40000 (time 1788470130 ; total 614 ; last 152) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 760 seconds: 65.303947 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg349-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg349-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg349-server stop mds service on oleg349-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg349-server stop mds service on oleg349-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg349-server unloading modules via unload_modules_local on: 'oleg349-server' oleg349-server: oleg349-server.virtnet: executing unload_modules_local oleg349-server: kmemleak disabled modules unloaded. pdsh@oleg349-client: oleg349-client: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-client: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 2