== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:23:42 (1788801822) start mds service on oleg411-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 7 09:50 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing load_modules_local oleg411-server: kmemleak disabled oleg411-server: Loading modules from /home/green/git/lustre-release/lustre oleg411-server: detected 4 online CPUs by sysfs oleg411-server: MODOPTS_LIBCFS= oleg411-server: Force libcfs to create 2 CPU partitions oleg411-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg411-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg411-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg411-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg411-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg411-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0000 oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 258 (time 1788801931.46 total 10.70 last 24.11) - open/close 532 (time 1788801942.39 total 21.63 last 25.08) - open/close 817 (time 1788801952.40 total 31.64 last 28.47) - open/close 1068 (time 1788801962.41 total 41.65 last 25.07) - open/close 1222 (time 1788801972.54 total 51.78 last 15.20) - open/close 1485 (time 1788801983.03 total 62.27 last 25.06) - open/close 1761 (time 1788801995.87 total 75.11 last 21.50) - open/close 2065 (time 1788802005.87 total 85.11 last 30.38) - open/close 2335 (time 1788802016.51 total 95.75 last 25.38) - open/close 2593 (time 1788802026.52 total 105.76 last 25.78) - open/close 2825 (time 1788802036.53 total 115.77 last 23.17) - open/close 3060 (time 1788802046.54 total 125.78 last 23.47) - open/close 3217 (time 1788802057.92 total 137.16 last 13.80) - open/close 3390 (time 1788802067.98 total 147.22 last 17.21) - open/close 3452 (time 1788802077.99 total 157.23 last 6.19) - open/close 3653 (time 1788802088.95 total 168.19 last 18.34) - open/close 3833 (time 1788802099.89 total 179.13 last 16.46) - open/close 4003 (time 1788802109.90 total 189.14 last 16.98) - open/close 4126 (time 1788802120.01 total 199.25 last 12.17) - open/close 4347 (time 1788802130.37 total 209.61 last 21.33) - open/close 4462 (time 1788802140.75 total 219.99 last 11.08) - open/close 4736 (time 1788802151.68 total 230.92 last 25.06) - open/close 5018 (time 1788802161.69 total 240.93 last 28.18) - open/close 5232 (time 1788802171.69 total 250.93 last 21.40) - open/close 5509 (time 1788802184.74 total 263.97 last 21.24) - open/close 5773 (time 1788802195.62 total 274.86 last 24.25) - open/close 6020 (time 1788802205.88 total 285.12 last 24.06) - open/close 6212 (time 1788802216.77 total 296.00 last 17.64) - open/close 6374 (time 1788802227.89 total 307.13 last 14.56) - open/close 6533 (time 1788802237.91 total 317.15 last 15.87) - open/close 6642 (time 1788802248.45 total 327.69 last 10.35) - open/close 6895 (time 1788802258.46 total 337.69 last 25.28) - open/close 7134 (time 1788802270.79 total 350.03 last 19.38) - open/close 7393 (time 1788802280.80 total 360.04 last 25.86) - open/close 7546 (time 1788802290.81 total 370.05 last 15.29) - open/close 7735 (time 1788802300.82 total 380.06 last 18.88) - open/close 7861 (time 1788802310.99 total 390.22 last 12.40) - open/close 7989 (time 1788802322.05 total 401.29 last 11.56) - open/close 8119 (time 1788802332.80 total 412.04 last 12.09) - open/close 8431 (time 1788802342.98 total 422.22 last 30.65) - open/close 8568 (time 1788802353.08 total 432.31 last 13.57) - open/close 8645 (time 1788802363.27 total 442.51 last 7.55) - open/close 8778 (time 1788802373.45 total 452.69 last 13.07) - open/close 8902 (time 1788802384.29 total 463.53 last 11.43) - open/close 9004 (time 1788802394.39 total 473.63 last 10.10) - open/close 9313 (time 1788802405.19 total 484.43 last 28.60) - open/close 9490 (time 1788802415.93 total 495.17 last 16.48) - open/close 9615 (time 1788802425.94 total 505.18 last 12.49) - open/close 9847 (time 1788802436.68 total 515.92 last 21.60) - open/close 10000 (time 1788802444.94 total 524.18 last 18.52) - open/close 10241 (time 1788802455.83 total 535.07 last 22.14) - open/close 10457 (time 1788802465.83 total 545.07 last 21.59) - open/close 10578 (time 1788802477.65 total 556.89 last 10.24) - open/close 10668 (time 1788802487.83 total 567.07 last 8.84) - open/close 10867 (time 1788802498.24 total 577.48 last 19.10) - open/close 10980 (time 1788802508.24 total 587.48 last 11.30) - open/close 11114 (time 1788802518.25 total 597.49 last 13.39) - open/close 11398 (time 1788802528.33 total 607.57 last 28.17) - open/close 11526 (time 1788802538.34 total 617.57 last 12.79) - open/close 11698 (time 1788802548.34 total 627.58 last 17.19) - open/close 11767 (time 1788802559.24 total 638.48 last 6.33) - open/close 11866 (time 1788802570.12 total 649.36 last 9.10) - open/close 12082 (time 1788802580.94 total 660.17 last 19.98) - open/close 12215 (time 1788802590.97 total 670.21 last 13.25) - open/close 12319 (time 1788802601.34 total 680.58 last 10.03) - open/close 12385 (time 1788802611.47 total 690.71 last 6.52) - open/close 12478 (time 1788802621.47 total 700.71 last 9.30) - open/close 12584 (time 1788802633.34 total 712.58 last 8.93) - open/close 12779 (time 1788802643.56 total 722.80 last 19.09) - open/close 12916 (time 1788802653.96 total 733.19 last 13.17) - open/close 13087 (time 1788802664.43 total 743.67 last 16.32) - open/close 13247 (time 1788802675.07 total 754.31 last 15.04) - open/close 13377 (time 1788802685.19 total 764.43 last 12.85) - open/close 13491 (time 1788802695.21 total 774.45 last 11.38) - open/close 13615 (time 1788802705.72 total 784.96 last 11.79) - open/close 13776 (time 1788802715.75 total 794.98 last 16.06) - open/close 13862 (time 1788802725.83 total 805.07 last 8.53) - open/close 13986 (time 1788802736.59 total 815.83 last 11.52) - open/close 14134 (time 1788802746.60 total 825.83 last 14.79) - open/close 14292 (time 1788802757.35 total 836.59 last 14.69) - open/close 14426 (time 1788802767.63 total 846.86 last 13.04) - open/close 14603 (time 1788802777.64 total 856.88 last 17.67) - open/close 14760 (time 1788802788.43 total 867.67 last 14.56) - open/close 14852 (time 1788802799.87 total 879.11 last 8.04) - open/close 14950 (time 1788802810.37 total 889.61 last 9.34) - open/close 15036 (time 1788802821.64 total 900.88 last 7.63) - open/close 15239 (time 1788802831.64 total 910.88 last 20.30) - open/close 15373 (time 1788802841.66 total 920.90 last 13.37) - open/close 15498 (time 1788802852.49 total 931.72 last 11.55) - open/close 15685 (time 1788802862.50 total 941.74 last 18.67) - open/close 15769 (time 1788802872.51 total 951.74 last 8.40) - open/close 15901 (time 1788802883.54 total 962.78 last 11.97) - open/close 16086 (time 1788802893.54 total 972.78 last 18.49) - open/close 16319 (time 1788802903.56 total 982.80 last 23.26) - open/close 16422 (time 1788802914.36 total 993.59 last 9.54) - open/close 16651 (time 1788802924.38 total 1003.62 last 22.85) - open/close 16789 (time 1788802934.39 total 1013.63 last 13.78) - open/close 16932 (time 1788802945.41 total 1024.65 last 12.98) - open/close 17099 (time 1788802956.15 total 1035.39 last 15.55) - open/close 17302 (time 1788802966.61 total 1045.85 last 19.41) - open/close 17376 (time 1788802976.62 total 1055.86 last 7.39) - open/close 17546 (time 1788802986.63 total 1065.87 last 16.97) - open/close 17669 (time 1788802997.51 total 1076.75 last 11.31) - open/close 18012 (time 1788803007.52 total 1086.76 last 34.28) - open/close 18521 (time 1788803017.53 total 1096.77 last 50.83) - open/close 18932 (time 1788803027.54 total 1106.78 last 41.05) - open/close 19488 (time 1788803037.55 total 1116.79 last 55.54) - open/close 20000 (time 1788803045.68 total 1124.91 last 63.04) - open/close 20607 (time 1788803055.69 total 1134.93 last 60.63) - open/close 21370 (time 1788803065.70 total 1144.93 last 76.23) - open/close 22205 (time 1788803075.71 total 1154.95 last 83.40) - open/close 22976 (time 1788803085.71 total 1164.95 last 77.08) - open/close 23903 (time 1788803095.72 total 1174.96 last 92.62) - open/close 24726 (time 1788803105.72 total 1184.96 last 82.29) - open/close 25577 (time 1788803115.73 total 1194.97 last 84.99) - open/close 26381 (time 1788803125.76 total 1204.99 last 80.21) - open/close 27226 (time 1788803135.76 total 1215.00 last 84.49) - open/close 28048 (time 1788803145.76 total 1225.00 last 82.18) - open/close 28854 (time 1788803155.77 total 1235.01 last 80.49) - open/close 29438 (time 1788803165.81 total 1245.05 last 58.20) - open/close 30000 (time 1788803174.64 total 1253.88 last 63.61) - open/close 30686 (time 1788803184.65 total 1263.88 last 68.58) - open/close 31475 (time 1788803194.66 total 1273.89 last 78.81) - open/close 32383 (time 1788803204.66 total 1283.90 last 90.75) - open/close 33258 (time 1788803214.67 total 1293.90 last 87.47) - open/close 34149 (time 1788803224.67 total 1303.90 last 89.10) - open/close 34849 (time 1788803234.68 total 1313.92 last 69.90) - open/close 35680 (time 1788803244.71 total 1323.95 last 82.84) - open/close 36571 (time 1788803254.71 total 1333.95 last 89.09) - open/close 37425 (time 1788803264.71 total 1343.95 last 85.40) - open/close 38219 (time 1788803274.72 total 1353.95 last 79.37) - open/close 38988 (time 1788803284.72 total 1363.96 last 76.87) - open/close 39854 (time 1788803294.73 total 1373.97 last 86.53) - open/close 40741 (time 1788803304.73 total 1383.97 last 88.69) - open/close 41619 (time 1788803314.74 total 1393.98 last 87.72) - open/close 42448 (time 1788803324.76 total 1404.00 last 82.74) - open/close 43261 (time 1788803334.81 total 1414.05 last 80.89) - open/close 43986 (time 1788803344.83 total 1424.07 last 72.32) - open/close 44796 (time 1788803354.84 total 1434.08 last 80.91) - open/close 45643 (time 1788803364.85 total 1444.09 last 84.66) - open/close 46509 (time 1788803374.91 total 1454.15 last 86.04) - open/close 47354 (time 1788803384.92 total 1464.16 last 84.46) - open/close 48171 (time 1788803394.95 total 1474.19 last 81.47) - open/close 49040 (time 1788803404.96 total 1484.20 last 86.75) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1490.99 seconds: 33.29 ops/second - unlinked 0 (time 1788803416 ; total 0 ; last 0) - unlinked 10000 (time 1788803552 ; total 136 ; last 136) - unlinked 20000 (time 1788803672 ; total 256 ; last 120) - unlinked 30000 (time 1788803789 ; total 373 ; last 117) - unlinked 40000 (time 1788803910 ; total 494 ; last 121) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 611 seconds: 81.229134 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg411-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg411-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server stop mds service on oleg411-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg411-server stop mds service on oleg411-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg411-server unloading modules via unload_modules_local on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing unload_modules_local oleg411-server: kmemleak disabled modules unloaded. pdsh@oleg411-client: oleg411-client: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-client: ssh exited with exit code 2