== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:32:18 (1788845538) start mds service on oleg410-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 7 17:41 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg410-server' oleg410-server: oleg410-server.virtnet: executing load_modules_local oleg410-server: kmemleak disabled oleg410-server: Loading modules from /home/green/git/lustre-release/lustre oleg410-server: detected 4 online CPUs by sysfs oleg410-server: MODOPTS_LIBCFS= oleg410-server: Force libcfs to create 2 CPU partitions oleg410-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg410-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg410-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg410-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg410-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg410-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-OST0000 oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 58 (time 1788845649.12 total 10.52 last 5.51) - open/close 274 (time 1788845659.78 total 21.18 last 20.26) - open/close 464 (time 1788845670.58 total 31.98 last 17.59) - open/close 738 (time 1788845680.58 total 41.99 last 27.39) - open/close 937 (time 1788845690.96 total 52.36 last 19.18) - open/close 1197 (time 1788845700.98 total 62.39 last 25.94) - open/close 1386 (time 1788845711.17 total 72.58 last 18.54) - open/close 1582 (time 1788845721.27 total 82.67 last 19.41) - open/close 1729 (time 1788845731.27 total 92.68 last 14.70) - open/close 1841 (time 1788845741.30 total 102.70 last 11.17) - open/close 1987 (time 1788845752.13 total 113.53 last 13.48) - open/close 2206 (time 1788845762.15 total 123.56 last 21.85) - open/close 2318 (time 1788845773.00 total 134.41 last 10.32) - open/close 2526 (time 1788845785.42 total 146.83 last 16.75) - open/close 2562 (time 1788845795.71 total 157.11 last 3.50) - open/close 2768 (time 1788845805.72 total 167.13 last 20.58) - open/close 2945 (time 1788845815.75 total 177.15 last 17.65) - open/close 3069 (time 1788845826.05 total 187.45 last 12.04) - open/close 3252 (time 1788845836.08 total 197.48 last 18.25) - open/close 3372 (time 1788845846.40 total 207.80 last 11.63) - open/close 3466 (time 1788845856.66 total 218.06 last 9.16) - open/close 3605 (time 1788845866.66 total 228.07 last 13.89) - open/close 3870 (time 1788845877.62 total 239.03 last 24.19) - open/close 4016 (time 1788845887.84 total 249.24 last 14.29) - open/close 4178 (time 1788845898.63 total 260.03 last 15.01) - open/close 4361 (time 1788845910.74 total 272.14 last 15.11) - open/close 4446 (time 1788845920.96 total 282.37 last 8.31) - open/close 4524 (time 1788845930.98 total 292.38 last 7.79) - open/close 4645 (time 1788845941.76 total 303.17 last 11.22) - open/close 4785 (time 1788845952.50 total 313.91 last 13.04) - open/close 4853 (time 1788845965.38 total 326.79 last 5.28) - open/close 5117 (time 1788845975.76 total 337.17 last 25.43) - open/close 5379 (time 1788845985.77 total 347.17 last 26.18) - open/close 5586 (time 1788845995.78 total 357.18 last 20.69) - open/close 5748 (time 1788846006.20 total 367.61 last 15.54) - open/close 5994 (time 1788846016.21 total 377.62 last 24.56) - open/close 6170 (time 1788846028.67 total 390.08 last 14.13) - open/close 6397 (time 1788846039.17 total 400.57 last 21.63) - open/close 6506 (time 1788846049.17 total 410.58 last 10.89) - open/close 6615 (time 1788846059.91 total 421.31 last 10.16) - open/close 6865 (time 1788846072.83 total 434.23 last 19.35) - open/close 7034 (time 1788846083.46 total 444.87 last 15.89) - open/close 7332 (time 1788846094.48 total 455.88 last 27.06) - open/close 7564 (time 1788846106.16 total 467.56 last 19.86) - open/close 7768 (time 1788846117.12 total 478.53 last 18.60) - open/close 7856 (time 1788846127.16 total 488.56 last 8.77) - open/close 8144 (time 1788846139.77 total 501.18 last 22.82) - open/close 8304 (time 1788846150.08 total 511.49 last 15.53) - open/close 8477 (time 1788846160.46 total 521.86 last 16.67) - open/close 8736 (time 1788846174.91 total 536.31 last 17.92) - open/close 8939 (time 1788846185.31 total 546.72 last 19.51) - open/close 9131 (time 1788846195.32 total 556.72 last 19.20) - open/close 9356 (time 1788846205.89 total 567.30 last 21.27) - open/close 9460 (time 1788846215.90 total 577.30 last 10.39) - open/close 9479 (time 1788846226.71 total 588.12 last 1.76) - open/close 9679 (time 1788846237.12 total 598.53 last 19.21) - open/close 9851 (time 1788846247.95 total 609.35 last 15.88) - open/close 9955 (time 1788846258.55 total 619.95 last 9.81) - open/close 10000 (time 1788846265.94 total 627.35 last 6.09) - open/close 10090 (time 1788846276.58 total 637.99 last 8.46) - open/close 10170 (time 1788846287.51 total 648.91 last 7.33) - open/close 10339 (time 1788846298.25 total 659.66 last 15.73) - open/close 10527 (time 1788846308.26 total 669.67 last 18.78) - open/close 10640 (time 1788846319.06 total 680.47 last 10.46) - open/close 10708 (time 1788846329.07 total 690.47 last 6.80) - open/close 10915 (time 1788846339.08 total 700.48 last 20.68) - open/close 11042 (time 1788846349.09 total 710.49 last 12.69) - open/close 11234 (time 1788846359.09 total 720.50 last 19.19) - open/close 11416 (time 1788846369.55 total 730.96 last 17.40) - open/close 11553 (time 1788846379.56 total 740.96 last 13.70) - open/close 11679 (time 1788846389.77 total 751.17 last 12.34) - open/close 11742 (time 1788846400.59 total 762.00 last 5.82) - open/close 11932 (time 1788846410.61 total 772.01 last 18.97) - open/close 12158 (time 1788846426.17 total 787.57 last 14.52) - open/close 12294 (time 1788846438.24 total 799.64 last 11.27) - open/close 12508 (time 1788846448.58 total 809.99 last 20.69) - open/close 12700 (time 1788846458.59 total 820.00 last 19.18) - open/close 12914 (time 1788846468.61 total 830.02 last 21.35) - open/close 13135 (time 1788846478.72 total 840.13 last 21.86) - open/close 13343 (time 1788846488.74 total 850.14 last 20.77) - open/close 13457 (time 1788846499.58 total 860.98 last 10.52) - open/close 13559 (time 1788846510.07 total 871.47 last 9.72) - open/close 13751 (time 1788846520.08 total 881.48 last 19.18) - open/close 13885 (time 1788846530.44 total 891.85 last 12.93) - open/close 14016 (time 1788846540.79 total 902.19 last 12.67) - open/close 14111 (time 1788846551.27 total 912.67 last 9.06) - open/close 14290 (time 1788846562.33 total 923.74 last 16.18) - open/close 14483 (time 1788846572.36 total 933.76 last 19.25) - open/close 14646 (time 1788846584.91 total 946.31 last 12.99) - open/close 14818 (time 1788846595.11 total 956.51 last 16.87) - open/close 14935 (time 1788846606.16 total 967.56 last 10.59) - open/close 15144 (time 1788846616.16 total 977.57 last 20.89) - open/close 15270 (time 1788846626.30 total 987.71 last 12.43) - open/close 15440 (time 1788846637.07 total 998.47 last 15.79) - open/close 15531 (time 1788846647.09 total 1008.50 last 9.08) - open/close 15737 (time 1788846658.12 total 1019.53 last 18.67) - open/close 16037 (time 1788846668.68 total 1030.08 last 28.42) - open/close 16195 (time 1788846678.68 total 1040.09 last 15.80) - open/close 16359 (time 1788846690.37 total 1051.78 last 14.03) - open/close 16463 (time 1788846700.99 total 1062.39 last 9.80) - open/close 16804 (time 1788846711.01 total 1072.42 last 34.01) - open/close 17360 (time 1788846721.02 total 1082.43 last 55.55) - open/close 17929 (time 1788846731.03 total 1092.44 last 56.83) - open/close 18536 (time 1788846741.03 total 1102.44 last 60.69) - open/close 19097 (time 1788846751.05 total 1112.45 last 56.03) - open/close 19772 (time 1788846761.05 total 1122.46 last 67.46) - open/close 20000 (time 1788846764.62 total 1126.02 last 64.00) - open/close 20698 (time 1788846774.62 total 1136.02 last 69.77) - open/close 21360 (time 1788846784.63 total 1146.04 last 66.11) - open/close 22000 (time 1788846794.65 total 1156.06 last 63.87) - open/close 22711 (time 1788846804.66 total 1166.07 last 71.03) - open/close 23431 (time 1788846814.67 total 1176.08 last 71.93) - open/close 24014 (time 1788846824.69 total 1186.09 last 58.21) - open/close 24702 (time 1788846834.70 total 1196.10 last 68.75) - open/close 25339 (time 1788846844.71 total 1206.11 last 63.62) - open/close 26034 (time 1788846854.72 total 1216.12 last 69.43) - open/close 26747 (time 1788846864.73 total 1226.13 last 71.24) - open/close 27367 (time 1788846874.73 total 1236.14 last 61.97) - open/close 28063 (time 1788846884.74 total 1246.15 last 69.54) - open/close 28796 (time 1788846894.74 total 1256.15 last 73.29) - open/close 29503 (time 1788846904.75 total 1266.15 last 70.64) - open/close 30000 (time 1788846911.30 total 1272.71 last 75.86) - open/close 30708 (time 1788846921.34 total 1282.74 last 70.55) - open/close 31402 (time 1788846931.34 total 1292.75 last 69.36) - open/close 32113 (time 1788846941.34 total 1302.75 last 71.09) - open/close 32848 (time 1788846951.35 total 1312.76 last 73.43) - open/close 33391 (time 1788846961.36 total 1322.77 last 54.24) - open/close 34056 (time 1788846971.36 total 1332.77 last 66.49) - open/close 34687 (time 1788846981.37 total 1342.77 last 63.08) - open/close 35291 (time 1788846991.40 total 1352.80 last 60.20) - open/close 35986 (time 1788847001.41 total 1362.81 last 69.44) - open/close 36618 (time 1788847011.41 total 1372.82 last 63.17) - open/close 37306 (time 1788847021.44 total 1382.84 last 68.62) - open/close 38004 (time 1788847031.45 total 1392.85 last 69.74) - open/close 38671 (time 1788847041.45 total 1402.85 last 66.68) - open/close 39315 (time 1788847051.46 total 1412.87 last 64.31) - open/close 39976 (time 1788847061.47 total 1422.87 last 66.08) - open/close 40576 (time 1788847071.47 total 1432.87 last 59.98) - open/close 41167 (time 1788847081.47 total 1442.88 last 59.10) - open/close 41789 (time 1788847091.47 total 1452.88 last 62.20) - open/close 42451 (time 1788847101.48 total 1462.88 last 66.14) - open/close 43019 (time 1788847111.49 total 1472.89 last 56.77) - open/close 43702 (time 1788847121.49 total 1482.89 last 68.28) - open/close 44410 (time 1788847131.49 total 1492.89 last 70.79) - open/close 45093 (time 1788847141.50 total 1502.90 last 68.26) - open/close 45751 (time 1788847151.50 total 1512.91 last 65.76) - open/close 46393 (time 1788847161.50 total 1522.91 last 64.19) - open/close 47072 (time 1788847171.51 total 1532.92 last 67.82) - open/close 47677 (time 1788847181.53 total 1542.93 last 60.43) - open/close 48327 (time 1788847191.54 total 1552.94 last 64.92) - open/close 49009 (time 1788847201.55 total 1562.95 last 68.12) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1572.61 seconds: 31.56 ops/second - unlinked 0 (time 1788847216 ; total 0 ; last 0) - unlinked 10000 (time 1788847377 ; total 161 ; last 161) - unlinked 20000 (time 1788847530 ; total 314 ; last 153) - unlinked 30000 (time 1788847687 ; total 471 ; last 157) - unlinked 40000 (time 1788847831 ; total 615 ; last 144) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 750 seconds: 66.174667 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg410-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg410-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg410-server stop mds service on oleg410-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg410-server stop mds service on oleg410-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg410-server unloading modules via unload_modules_local on: 'oleg410-server' oleg410-server: oleg410-server.virtnet: executing unload_modules_local oleg410-server: kmemleak disabled modules unloaded. pdsh@oleg410-client: oleg410-client: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-server: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-client: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-server: ssh exited with exit code 2