== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:41:11 (1788525671) start mds service on oleg453-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 3 15:04 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg453-server' oleg453-server: oleg453-server.virtnet: executing load_modules_local oleg453-server: kmemleak disabled oleg453-server: Loading modules from /home/green/git/lustre-release/lustre oleg453-server: detected 4 online CPUs by sysfs oleg453-server: MODOPTS_LIBCFS= oleg453-server: Force libcfs to create 2 CPU partitions oleg453-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg453-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg453-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg453-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg453-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg453-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-OST0000 oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg453-client.virtnet: -o user_xattr,flock 192.168.204.153@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 94 (time 1788525780.22 total 10.03 last 9.37) - open/close 282 (time 1788525790.22 total 20.03 last 18.79) - open/close 438 (time 1788525800.23 total 30.04 last 15.58) - open/close 673 (time 1788525810.57 total 40.38 last 22.74) - open/close 933 (time 1788525821.31 total 51.12 last 24.20) - open/close 1163 (time 1788525831.31 total 61.12 last 23.00) - open/close 1353 (time 1788525842.24 total 72.05 last 17.38) - open/close 1472 (time 1788525852.75 total 82.56 last 11.33) - open/close 1703 (time 1788525862.77 total 92.58 last 23.04) - open/close 1857 (time 1788525873.09 total 102.90 last 14.93) - open/close 2116 (time 1788525883.26 total 113.07 last 25.45) - open/close 2357 (time 1788525893.29 total 123.10 last 24.04) - open/close 2610 (time 1788525903.52 total 133.33 last 24.73) - open/close 2825 (time 1788525913.52 total 143.33 last 21.49) - open/close 3023 (time 1788525923.52 total 153.33 last 19.80) - open/close 3224 (time 1788525933.60 total 163.41 last 19.94) - open/close 3345 (time 1788525944.21 total 174.02 last 11.40) - open/close 3649 (time 1788525954.22 total 184.03 last 30.37) - open/close 3944 (time 1788525964.23 total 194.04 last 29.47) - open/close 4168 (time 1788525975.49 total 205.30 last 19.89) - open/close 4330 (time 1788525985.61 total 215.42 last 16.01) - open/close 4604 (time 1788525995.61 total 225.42 last 27.40) - open/close 4745 (time 1788526005.61 total 235.42 last 14.10) - open/close 4872 (time 1788526016.13 total 245.94 last 12.07) - open/close 5015 (time 1788526026.90 total 256.71 last 13.28) - open/close 5236 (time 1788526038.35 total 268.16 last 19.31) - open/close 5454 (time 1788526048.36 total 278.17 last 21.76) - open/close 5534 (time 1788526058.64 total 288.45 last 7.79) - open/close 5628 (time 1788526068.65 total 298.46 last 9.39) - open/close 5845 (time 1788526078.93 total 308.74 last 21.10) - open/close 5948 (time 1788526088.93 total 318.74 last 10.30) - open/close 6145 (time 1788526099.08 total 328.89 last 19.42) - open/close 6277 (time 1788526109.16 total 338.97 last 13.09) - open/close 6408 (time 1788526119.57 total 349.38 last 12.58) - open/close 6588 (time 1788526129.94 total 359.75 last 17.37) - open/close 6742 (time 1788526144.15 total 373.96 last 10.83) - open/close 6971 (time 1788526154.16 total 383.97 last 22.88) - open/close 7128 (time 1788526167.04 total 396.85 last 12.19) - open/close 7273 (time 1788526177.07 total 406.88 last 14.46) - open/close 7415 (time 1788526188.11 total 417.92 last 12.87) - open/close 7643 (time 1788526199.68 total 429.49 last 19.69) - open/close 7796 (time 1788526209.86 total 439.67 last 15.04) - open/close 8056 (time 1788526219.90 total 449.71 last 25.89) - open/close 8290 (time 1788526229.92 total 459.73 last 23.36) - open/close 8510 (time 1788526239.92 total 469.73 last 21.99) - open/close 8763 (time 1788526250.30 total 480.11 last 24.38) - open/close 9044 (time 1788526260.97 total 490.78 last 26.33) - open/close 9219 (time 1788526271.98 total 501.79 last 15.89) - open/close 9350 (time 1788526282.00 total 511.81 last 13.08) - open/close 9485 (time 1788526293.00 total 522.81 last 12.27) - open/close 9741 (time 1788526303.89 total 533.70 last 23.51) - open/close 9977 (time 1788526313.89 total 543.70 last 23.59) - open/close 10066 (time 1788526324.58 total 554.39 last 8.33) - open/close 10181 (time 1788526334.98 total 564.79 last 11.05) - open/close 10343 (time 1788526345.80 total 575.61 last 14.97) - open/close 10518 (time 1788526355.81 total 585.62 last 17.48) - open/close 10767 (time 1788526366.22 total 596.03 last 23.92) - open/close 10886 (time 1788526376.40 total 606.21 last 11.69) - open/close 11147 (time 1788526386.40 total 616.21 last 26.09) - open/close 11362 (time 1788526396.43 total 626.24 last 21.44) - open/close 11615 (time 1788526407.32 total 637.13 last 23.24) - open/close 11780 (time 1788526418.38 total 648.19 last 14.92) - open/close 11983 (time 1788526429.38 total 659.19 last 18.46) - open/close 12243 (time 1788526440.40 total 670.21 last 23.59) - open/close 12370 (time 1788526450.42 total 680.23 last 12.67) - open/close 12595 (time 1788526461.53 total 691.34 last 20.27) - open/close 12811 (time 1788526471.84 total 701.65 last 20.93) - open/close 12984 (time 1788526482.63 total 712.44 last 16.04) - open/close 13225 (time 1788526492.65 total 722.46 last 24.04) - open/close 13358 (time 1788526503.30 total 733.11 last 12.50) - open/close 13555 (time 1788526514.92 total 744.73 last 16.96) - open/close 13773 (time 1788526525.52 total 755.33 last 20.56) - open/close 13945 (time 1788526535.75 total 765.56 last 16.81) - open/close 14038 (time 1788526547.29 total 777.10 last 8.06) - open/close 14141 (time 1788526557.31 total 787.12 last 10.28) - open/close 14218 (time 1788526567.45 total 797.26 last 7.59) - open/close 14432 (time 1788526577.45 total 807.26 last 21.40) - open/close 14504 (time 1788526587.47 total 817.28 last 7.19) - open/close 14614 (time 1788526598.35 total 828.16 last 10.11) - open/close 14818 (time 1788526608.74 total 838.55 last 19.63) - open/close 14958 (time 1788526619.05 total 848.86 last 13.57) - open/close 15119 (time 1788526629.58 total 859.39 last 15.30) - open/close 15320 (time 1788526640.33 total 870.14 last 18.70) - open/close 15460 (time 1788526650.72 total 880.53 last 13.47) - open/close 15653 (time 1788526663.63 total 893.44 last 14.96) - open/close 15741 (time 1788526673.64 total 903.45 last 8.79) - open/close 15967 (time 1788526683.67 total 913.48 last 22.53) - open/close 16158 (time 1788526693.90 total 923.71 last 18.66) - open/close 16206 (time 1788526704.39 total 934.20 last 4.58) - open/close 16375 (time 1788526714.43 total 944.24 last 16.83) - open/close 16572 (time 1788526727.40 total 957.21 last 15.19) - open/close 16690 (time 1788526737.74 total 967.55 last 11.41) - open/close 16955 (time 1788526748.17 total 977.98 last 25.41) - open/close 17224 (time 1788526758.98 total 988.79 last 24.88) - open/close 17500 (time 1788526769.62 total 999.43 last 25.94) - open/close 17663 (time 1788526780.03 total 1009.84 last 15.66) - open/close 17833 (time 1788526790.71 total 1020.52 last 15.91) - open/close 17911 (time 1788526800.71 total 1030.52 last 7.80) - open/close 18347 (time 1788526810.73 total 1040.54 last 43.54) - open/close 19132 (time 1788526820.73 total 1050.54 last 78.49) - open/close 19914 (time 1788526830.73 total 1060.54 last 78.19) - open/close 20720 (time 1788526840.74 total 1070.55 last 80.52) - open/close 21515 (time 1788526850.74 total 1080.55 last 79.48) - open/close 22271 (time 1788526860.75 total 1090.56 last 75.57) - open/close 23001 (time 1788526870.75 total 1100.56 last 72.96) - open/close 23748 (time 1788526880.76 total 1110.57 last 74.64) - open/close 24580 (time 1788526890.77 total 1120.58 last 83.13) - open/close 25389 (time 1788526900.78 total 1130.59 last 80.79) - open/close 26171 (time 1788526910.80 total 1140.61 last 78.10) - open/close 26964 (time 1788526920.82 total 1150.63 last 79.09) - open/close 27732 (time 1788526930.83 total 1160.64 last 76.76) - open/close 28584 (time 1788526940.83 total 1170.64 last 85.15) - open/close 29426 (time 1788526950.83 total 1180.64 last 84.20) - open/close 30000 (time 1788526957.30 total 1187.11 last 88.78) - open/close 30786 (time 1788526967.30 total 1197.11 last 78.56) - open/close 31675 (time 1788526977.31 total 1207.12 last 88.83) - open/close 32553 (time 1788526987.32 total 1217.13 last 87.77) - open/close 33390 (time 1788526997.32 total 1227.13 last 83.64) - open/close 34239 (time 1788527007.33 total 1237.14 last 84.81) - open/close 35047 (time 1788527017.34 total 1247.15 last 80.77) - open/close 35835 (time 1788527027.34 total 1257.15 last 78.77) - open/close 36664 (time 1788527037.35 total 1267.16 last 82.81) - open/close 37527 (time 1788527047.35 total 1277.16 last 86.28) - open/close 38437 (time 1788527057.36 total 1287.17 last 90.93) - open/close 39285 (time 1788527067.37 total 1297.18 last 84.75) - open/close 40000 (time 1788527075.79 total 1305.60 last 84.92) - open/close 40829 (time 1788527085.79 total 1315.60 last 82.88) - open/close 41644 (time 1788527095.80 total 1325.61 last 81.39) - open/close 42519 (time 1788527105.81 total 1335.62 last 87.45) - open/close 43342 (time 1788527115.82 total 1345.63 last 82.21) - open/close 44230 (time 1788527125.82 total 1355.63 last 88.77) - open/close 45001 (time 1788527135.82 total 1365.63 last 77.09) - open/close 45741 (time 1788527145.83 total 1375.64 last 73.96) - open/close 46588 (time 1788527155.83 total 1385.64 last 84.68) - open/close 47418 (time 1788527165.84 total 1395.65 last 82.94) - open/close 48313 (time 1788527175.85 total 1405.66 last 89.42) - open/close 49153 (time 1788527185.86 total 1415.67 last 83.91) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1421.94 seconds: 34.90 ops/second - unlinked 0 (time 1788527196 ; total 0 ; last 0) - unlinked 10000 (time 1788527324 ; total 128 ; last 128) - unlinked 20000 (time 1788527451 ; total 255 ; last 127) - unlinked 30000 (time 1788527574 ; total 378 ; last 123) - unlinked 40000 (time 1788527695 ; total 499 ; last 121) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 618 seconds: 80.309059 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg453-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg453-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg453-server stop mds service on oleg453-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg453-server stop mds service on oleg453-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg453-server unloading modules via unload_modules_local on: 'oleg453-server' oleg453-server: oleg453-server.virtnet: executing unload_modules_local oleg453-server: kmemleak disabled modules unloaded. pdsh@oleg453-client: oleg453-server: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-client: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-client: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 2