== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:46:11 (1789303571) start mds service on oleg437-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 12 19:15 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing load_modules_local oleg437-server: kmemleak disabled oleg437-server: Loading modules from /home/green/git/lustre-release/lustre oleg437-server: detected 4 online CPUs by sysfs oleg437-server: MODOPTS_LIBCFS= oleg437-server: Force libcfs to create 2 CPU partitions oleg437-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg437-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg437-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg437-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg437-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg437-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0000 oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 178 (time 1789303696.73 total 10.02 last 17.77) - open/close 208 (time 1789303706.75 total 20.04 last 3.00) - open/close 461 (time 1789303717.15 total 30.44 last 24.32) - open/close 590 (time 1789303727.17 total 40.46 last 12.87) - open/close 823 (time 1789303737.63 total 50.92 last 22.27) - open/close 1076 (time 1789303748.06 total 61.35 last 24.26) - open/close 1304 (time 1789303758.31 total 71.60 last 22.24) - open/close 1419 (time 1789303768.79 total 82.08 last 10.97) - open/close 1494 (time 1789303778.81 total 92.09 last 7.49) - open/close 1729 (time 1789303788.82 total 102.11 last 23.46) - open/close 1935 (time 1789303798.83 total 112.12 last 20.58) - open/close 2129 (time 1789303808.84 total 122.13 last 19.38) - open/close 2259 (time 1789303819.76 total 133.05 last 11.90) - open/close 2386 (time 1789303830.56 total 143.85 last 11.76) - open/close 2518 (time 1789303840.57 total 153.86 last 13.19) - open/close 2712 (time 1789303850.98 total 164.27 last 18.63) - open/close 2895 (time 1789303861.61 total 174.90 last 17.22) - open/close 3086 (time 1789303872.38 total 185.66 last 17.74) - open/close 3234 (time 1789303882.68 total 195.96 last 14.37) - open/close 3365 (time 1789303892.70 total 205.98 last 13.07) - open/close 3455 (time 1789303902.70 total 215.99 last 8.99) - open/close 3692 (time 1789303913.27 total 226.55 last 22.43) - open/close 3876 (time 1789303923.29 total 236.57 last 18.37) - open/close 4090 (time 1789303933.29 total 246.58 last 21.38) - open/close 4309 (time 1789303945.64 total 258.93 last 17.74) - open/close 4580 (time 1789303956.38 total 269.67 last 25.23) - open/close 4792 (time 1789303968.61 total 281.90 last 17.34) - open/close 4988 (time 1789303978.93 total 292.21 last 19.00) - open/close 5226 (time 1789303989.88 total 303.17 last 21.72) - open/close 5462 (time 1789304000.70 total 313.99 last 21.82) - open/close 5666 (time 1789304010.72 total 324.00 last 20.37) - open/close 5917 (time 1789304020.73 total 334.02 last 25.05) - open/close 6144 (time 1789304030.75 total 344.03 last 22.67) - open/close 6330 (time 1789304042.27 total 355.56 last 16.14) - open/close 6532 (time 1789304053.35 total 366.63 last 18.24) - open/close 6726 (time 1789304064.31 total 377.59 last 17.70) - open/close 6897 (time 1789304074.99 total 388.28 last 16.00) - open/close 7103 (time 1789304085.00 total 398.28 last 20.59) - open/close 7313 (time 1789304095.19 total 408.48 last 20.60) - open/close 7530 (time 1789304105.58 total 418.87 last 20.89) - open/close 7742 (time 1789304118.32 total 431.61 last 16.64) - open/close 7932 (time 1789304128.34 total 441.62 last 18.97) - open/close 8186 (time 1789304139.06 total 452.34 last 23.69) - open/close 8344 (time 1789304149.62 total 462.90 last 14.96) - open/close 8445 (time 1789304159.77 total 473.06 last 9.94) - open/close 8618 (time 1789304169.97 total 483.25 last 16.97) - open/close 8796 (time 1789304180.78 total 494.06 last 16.47) - open/close 9085 (time 1789304191.21 total 504.49 last 27.71) - open/close 9192 (time 1789304202.55 total 515.84 last 9.43) - open/close 9361 (time 1789304215.27 total 528.56 last 13.29) - open/close 9535 (time 1789304226.29 total 539.58 last 15.79) - open/close 9617 (time 1789304238.56 total 551.85 last 6.68) - open/close 9809 (time 1789304248.57 total 561.86 last 19.18) - open/close 9950 (time 1789304259.55 total 572.83 last 12.85) - open/close 10000 (time 1789304262.81 total 576.10 last 15.31) - open/close 10157 (time 1789304272.81 total 586.10 last 15.70) - open/close 10256 (time 1789304283.36 total 596.65 last 9.39) - open/close 10441 (time 1789304293.38 total 606.67 last 18.46) - open/close 10622 (time 1789304303.39 total 616.68 last 18.09) - open/close 10798 (time 1789304313.39 total 626.68 last 17.60) - open/close 10944 (time 1789304323.40 total 636.68 last 14.59) - open/close 11109 (time 1789304336.29 total 649.58 last 12.80) - open/close 11290 (time 1789304347.31 total 660.60 last 16.42) - open/close 11451 (time 1789304357.92 total 671.21 last 15.18) - open/close 11525 (time 1789304367.93 total 681.22 last 7.39) - open/close 11683 (time 1789304378.41 total 691.70 last 15.07) - open/close 11852 (time 1789304389.21 total 702.50 last 15.65) - open/close 11952 (time 1789304399.54 total 712.83 last 9.68) - open/close 12099 (time 1789304409.70 total 722.99 last 14.47) - open/close 12213 (time 1789304420.58 total 733.86 last 10.48) - open/close 12320 (time 1789304431.45 total 744.74 last 9.84) - open/close 12373 (time 1789304442.14 total 755.43 last 4.96) - open/close 12524 (time 1789304453.85 total 767.14 last 12.89) - open/close 12593 (time 1789304464.14 total 777.43 last 6.70) - open/close 12712 (time 1789304474.47 total 787.75 last 11.53) - open/close 12875 (time 1789304484.50 total 797.79 last 16.24) - open/close 13016 (time 1789304494.69 total 807.98 last 13.84) - open/close 13181 (time 1789304504.69 total 817.98 last 16.50) - open/close 13415 (time 1789304514.70 total 827.99 last 23.38) - open/close 13574 (time 1789304525.28 total 838.57 last 15.03) - open/close 13776 (time 1789304535.29 total 848.58 last 20.19) - open/close 13917 (time 1789304545.57 total 858.85 last 13.72) - open/close 14036 (time 1789304556.00 total 869.29 last 11.41) - open/close 14165 (time 1789304567.01 total 880.30 last 11.71) - open/close 14304 (time 1789304577.03 total 890.32 last 13.87) - open/close 14507 (time 1789304587.16 total 900.45 last 20.04) - open/close 14665 (time 1789304597.98 total 911.27 last 14.60) - open/close 14817 (time 1789304608.30 total 921.59 last 14.73) - open/close 15026 (time 1789304618.72 total 932.01 last 20.06) - open/close 15199 (time 1789304628.73 total 942.02 last 17.29) - open/close 15350 (time 1789304638.90 total 952.19 last 14.85) - open/close 15518 (time 1789304649.45 total 962.74 last 15.92) - open/close 15716 (time 1789304659.46 total 972.75 last 19.77) - open/close 16039 (time 1789304669.48 total 982.76 last 32.26) - open/close 16284 (time 1789304679.48 total 992.77 last 24.49) - open/close 16499 (time 1789304689.95 total 1003.23 last 20.55) - open/close 16738 (time 1789304700.51 total 1013.79 last 22.64) - open/close 16974 (time 1789304710.89 total 1024.18 last 22.72) - open/close 17298 (time 1789304720.89 total 1034.18 last 32.40) - open/close 18065 (time 1789304730.90 total 1044.18 last 76.68) - open/close 18846 (time 1789304740.92 total 1054.20 last 77.95) - open/close 19623 (time 1789304750.92 total 1064.20 last 77.68) - open/close 20000 (time 1789304755.73 total 1069.02 last 78.32) - open/close 20733 (time 1789304765.73 total 1079.02 last 73.29) - open/close 21415 (time 1789304775.73 total 1089.02 last 68.20) - open/close 22177 (time 1789304785.74 total 1099.03 last 76.12) - open/close 22922 (time 1789304795.75 total 1109.03 last 74.47) - open/close 23754 (time 1789304805.76 total 1119.04 last 83.13) - open/close 24477 (time 1789304815.76 total 1129.05 last 72.25) - open/close 25300 (time 1789304825.76 total 1139.05 last 82.28) - open/close 26039 (time 1789304835.78 total 1149.06 last 73.80) - open/close 26843 (time 1789304845.78 total 1159.07 last 80.39) - open/close 27614 (time 1789304855.79 total 1169.07 last 77.04) - open/close 28478 (time 1789304865.79 total 1179.08 last 86.37) - open/close 29388 (time 1789304875.80 total 1189.09 last 90.93) - open/close 30000 (time 1789304883.01 total 1196.30 last 84.88) - open/close 30744 (time 1789304893.01 total 1206.30 last 74.36) - open/close 31581 (time 1789304903.03 total 1216.32 last 83.54) - open/close 32332 (time 1789304913.04 total 1226.33 last 75.05) - open/close 33126 (time 1789304923.05 total 1236.33 last 79.34) - open/close 33938 (time 1789304933.05 total 1246.34 last 81.17) - open/close 34730 (time 1789304943.06 total 1256.35 last 79.09) - open/close 35601 (time 1789304953.07 total 1266.36 last 87.07) - open/close 36386 (time 1789304963.07 total 1276.36 last 78.45) - open/close 37130 (time 1789304973.09 total 1286.38 last 74.26) - open/close 37909 (time 1789304983.10 total 1296.39 last 77.86) - open/close 38660 (time 1789304993.11 total 1306.40 last 75.02) - open/close 39431 (time 1789305003.12 total 1316.40 last 77.04) - open/close 40000 (time 1789305010.37 total 1323.66 last 78.41) - open/close 40755 (time 1789305020.38 total 1333.67 last 75.42) - open/close 41512 (time 1789305030.39 total 1343.67 last 75.67) - open/close 42289 (time 1789305040.39 total 1353.68 last 77.66) - open/close 43057 (time 1789305050.39 total 1363.68 last 76.79) - open/close 43796 (time 1789305060.40 total 1373.68 last 73.87) - open/close 44595 (time 1789305070.40 total 1383.69 last 79.85) - open/close 45370 (time 1789305080.42 total 1393.70 last 77.40) - open/close 46140 (time 1789305090.42 total 1403.71 last 76.95) - open/close 47009 (time 1789305100.43 total 1413.72 last 86.82) - open/close 47820 (time 1789305110.44 total 1423.72 last 81.06) - open/close 48695 (time 1789305120.45 total 1433.74 last 87.38) - open/close 49499 (time 1789305130.46 total 1443.74 last 80.35) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1445.70 seconds: 34.33 ops/second - unlinked 0 (time 1789305136 ; total 0 ; last 0) - unlinked 10000 (time 1789305255 ; total 119 ; last 119) - unlinked 20000 (time 1789305369 ; total 233 ; last 114) - unlinked 30000 (time 1789305485 ; total 349 ; last 116) - unlinked 40000 (time 1789305552 ; total 416 ; last 67) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 478 seconds: 103.830544 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg437-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg437-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg437-server stop mds service on oleg437-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg437-server stop mds service on oleg437-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg437-server unloading modules via unload_modules_local on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing unload_modules_local oleg437-server: kmemleak disabled modules unloaded. pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2