== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:33:38 (1789054418) start mds service on oleg349-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931568 Sep 10 01:50 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg349-server' oleg349-server: oleg349-server.virtnet: executing load_modules_local oleg349-server: kmemleak disabled oleg349-server: Loading modules from /home/green/git/lustre-release/lustre oleg349-server: detected 4 online CPUs by sysfs oleg349-server: MODOPTS_LIBCFS= oleg349-server: Force libcfs to create 2 CPU partitions oleg349-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg349-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg349-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg349-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg349-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg349-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 159 (time 1789054535.07 total 11.07 last 14.36) - open/close 271 (time 1789054545.51 total 21.52 last 10.72) - open/close 525 (time 1789054555.52 total 31.53 last 25.38) - open/close 788 (time 1789054565.52 total 41.53 last 26.29) - open/close 1065 (time 1789054575.53 total 51.53 last 27.69) - open/close 1263 (time 1789054585.70 total 61.71 last 19.46) - open/close 1478 (time 1789054595.71 total 71.72 last 21.48) - open/close 1687 (time 1789054605.73 total 81.73 last 20.87) - open/close 1949 (time 1789054615.74 total 91.75 last 26.17) - open/close 2099 (time 1789054625.75 total 101.76 last 14.98) - open/close 2263 (time 1789054636.21 total 112.22 last 15.67) - open/close 2420 (time 1789054646.50 total 122.51 last 15.26) - open/close 2643 (time 1789054658.85 total 134.85 last 18.06) - open/close 2860 (time 1789054668.86 total 144.87 last 21.67) - open/close 3064 (time 1789054678.87 total 154.88 last 20.38) - open/close 3363 (time 1789054688.88 total 164.89 last 29.86) - open/close 3553 (time 1789054699.50 total 175.51 last 17.90) - open/close 3755 (time 1789054709.85 total 185.86 last 19.52) - open/close 3966 (time 1789054720.60 total 196.61 last 19.62) - open/close 4254 (time 1789054730.62 total 206.63 last 28.75) - open/close 4446 (time 1789054741.47 total 217.48 last 17.70) - open/close 4640 (time 1789054752.41 total 228.42 last 17.73) - open/close 4846 (time 1789054763.76 total 239.77 last 18.15) - open/close 5028 (time 1789054773.77 total 249.77 last 18.19) - open/close 5279 (time 1789054783.78 total 259.79 last 25.07) - open/close 5409 (time 1789054795.37 total 271.38 last 11.22) - open/close 5682 (time 1789054806.36 total 282.37 last 24.84) - open/close 5944 (time 1789054816.93 total 292.94 last 24.79) - open/close 6163 (time 1789054828.00 total 304.01 last 19.78) - open/close 6351 (time 1789054838.01 total 314.02 last 18.79) - open/close 6525 (time 1789054848.01 total 324.02 last 17.40) - open/close 6799 (time 1789054858.40 total 334.41 last 26.37) - open/close 7059 (time 1789054868.40 total 344.41 last 26.00) - open/close 7252 (time 1789054878.41 total 354.42 last 19.27) - open/close 7436 (time 1789054889.11 total 365.12 last 17.20) - open/close 7619 (time 1789054899.50 total 375.50 last 17.62) - open/close 7897 (time 1789054909.51 total 385.52 last 27.75) - open/close 8119 (time 1789054919.54 total 395.54 last 22.15) - open/close 8295 (time 1789054930.54 total 406.54 last 16.00) - open/close 8430 (time 1789054940.54 total 416.55 last 13.49) - open/close 8683 (time 1789054950.57 total 426.57 last 25.24) - open/close 8833 (time 1789054960.90 total 436.91 last 14.51) - open/close 9004 (time 1789054971.37 total 447.38 last 16.34) - open/close 9119 (time 1789054982.01 total 458.01 last 10.81) - open/close 9328 (time 1789054992.49 total 468.50 last 19.94) - open/close 9487 (time 1789055003.52 total 479.53 last 14.41) - open/close 9628 (time 1789055013.54 total 489.54 last 14.08) - open/close 9781 (time 1789055023.79 total 499.79 last 14.93) - open/close 9980 (time 1789055034.53 total 510.54 last 18.52) - open/close 10122 (time 1789055044.55 total 520.56 last 14.17) - open/close 10208 (time 1789055056.74 total 532.75 last 7.05) - open/close 10336 (time 1789055066.79 total 542.80 last 12.74) - open/close 10413 (time 1789055076.79 total 552.80 last 7.70) - open/close 10565 (time 1789055088.09 total 564.10 last 13.45) - open/close 10786 (time 1789055098.21 total 574.22 last 21.85) - open/close 10945 (time 1789055108.38 total 584.39 last 15.63) - open/close 11195 (time 1789055119.15 total 595.15 last 23.22) - open/close 11450 (time 1789055130.03 total 606.03 last 23.44) - open/close 11730 (time 1789055140.03 total 616.04 last 27.98) - open/close 11835 (time 1789055150.04 total 626.04 last 10.50) - open/close 12049 (time 1789055160.04 total 636.05 last 21.39) - open/close 12269 (time 1789055170.05 total 646.06 last 21.98) - open/close 12537 (time 1789055180.96 total 656.97 last 24.57) - open/close 12857 (time 1789055192.03 total 668.04 last 28.91) - open/close 13121 (time 1789055202.79 total 678.79 last 24.54) - open/close 13325 (time 1789055213.22 total 689.23 last 19.54) - open/close 13611 (time 1789055223.85 total 699.86 last 26.92) - open/close 13832 (time 1789055234.14 total 710.15 last 21.46) - open/close 14073 (time 1789055244.57 total 720.57 last 23.12) - open/close 14253 (time 1789055255.14 total 731.15 last 17.02) - open/close 14503 (time 1789055266.15 total 742.16 last 22.72) - open/close 14672 (time 1789055276.81 total 752.82 last 15.85) - open/close 14913 (time 1789055286.82 total 762.82 last 24.09) - open/close 15145 (time 1789055296.83 total 772.83 last 23.18) - open/close 15401 (time 1789055307.56 total 783.57 last 23.85) - open/close 15601 (time 1789055317.58 total 793.59 last 19.95) - open/close 15843 (time 1789055327.59 total 803.60 last 24.19) - open/close 16060 (time 1789055338.31 total 814.31 last 20.25) - open/close 16278 (time 1789055348.40 total 824.41 last 21.59) - open/close 16521 (time 1789055358.88 total 834.89 last 23.19) - open/close 16793 (time 1789055369.82 total 845.83 last 24.86) - open/close 17073 (time 1789055379.82 total 855.83 last 28.00) - open/close 17325 (time 1789055389.84 total 865.85 last 25.15) - open/close 17593 (time 1789055399.86 total 875.87 last 26.75) - open/close 17735 (time 1789055409.88 total 885.89 last 14.18) - open/close 17886 (time 1789055419.89 total 895.89 last 15.09) - open/close 18109 (time 1789055431.81 total 907.82 last 18.70) - open/close 18285 (time 1789055441.85 total 917.85 last 17.54) - open/close 18464 (time 1789055451.85 total 927.86 last 17.90) - open/close 18671 (time 1789055461.98 total 937.99 last 20.43) - open/close 18828 (time 1789055471.98 total 947.99 last 15.70) - open/close 18969 (time 1789055482.46 total 958.47 last 13.46) - open/close 19113 (time 1789055492.65 total 968.65 last 14.14) - open/close 19245 (time 1789055502.88 total 978.88 last 12.90) - open/close 19951 (time 1789055512.89 total 988.89 last 70.54) - open/close 20593 (time 1789055522.90 total 998.90 last 64.14) - open/close 21241 (time 1789055532.90 total 1008.91 last 64.77) - open/close 21767 (time 1789055542.93 total 1018.94 last 52.45) - open/close 22258 (time 1789055552.94 total 1028.94 last 49.07) - open/close 22869 (time 1789055562.94 total 1038.95 last 61.06) - open/close 23547 (time 1789055572.95 total 1048.96 last 67.74) - open/close 24122 (time 1789055582.97 total 1058.97 last 57.42) - open/close 24728 (time 1789055593.00 total 1069.00 last 60.43) - open/close 25272 (time 1789055603.01 total 1079.02 last 54.30) - open/close 25811 (time 1789055613.01 total 1089.02 last 53.89) - open/close 26466 (time 1789055623.02 total 1099.03 last 65.46) - open/close 26939 (time 1789055633.04 total 1109.05 last 47.22) - open/close 27574 (time 1789055643.05 total 1119.06 last 63.42) - open/close 28193 (time 1789055653.06 total 1129.07 last 61.83) - open/close 28733 (time 1789055663.08 total 1139.09 last 53.88) - open/close 29402 (time 1789055673.09 total 1149.09 last 66.88) - open/close 29979 (time 1789055683.10 total 1159.10 last 57.65) - open/close 30561 (time 1789055693.11 total 1169.11 last 58.14) - open/close 31287 (time 1789055703.12 total 1179.13 last 72.50) - open/close 31959 (time 1789055713.12 total 1189.13 last 67.17) - open/close 32595 (time 1789055723.14 total 1199.15 last 63.49) - open/close 33112 (time 1789055733.14 total 1209.15 last 51.69) - open/close 33666 (time 1789055743.15 total 1219.16 last 55.35) - open/close 34354 (time 1789055753.16 total 1229.17 last 68.74) - open/close 35015 (time 1789055763.16 total 1239.17 last 66.09) - open/close 35641 (time 1789055773.17 total 1249.18 last 62.54) - open/close 36325 (time 1789055783.18 total 1259.19 last 68.32) - open/close 36925 (time 1789055793.20 total 1269.20 last 59.94) - open/close 37591 (time 1789055803.20 total 1279.21 last 66.56) - open/close 38197 (time 1789055813.21 total 1289.21 last 60.57) - open/close 38966 (time 1789055823.21 total 1299.22 last 76.85) - open/close 39718 (time 1789055833.21 total 1309.22 last 75.19) - open/close 40000 (time 1789055836.93 total 1312.93 last 75.97) - open/close 40746 (time 1789055846.93 total 1322.94 last 74.54) - open/close 41421 (time 1789055856.95 total 1332.96 last 67.39) - open/close 42069 (time 1789055866.97 total 1342.98 last 64.68) - open/close 42721 (time 1789055876.97 total 1352.98 last 65.18) - open/close 43429 (time 1789055886.98 total 1362.99 last 70.74) - open/close 44193 (time 1789055896.98 total 1372.99 last 76.38) - open/close 44857 (time 1789055906.99 total 1383.00 last 66.33) - open/close 45636 (time 1789055916.99 total 1393.00 last 77.89) - open/close 46320 (time 1789055927.00 total 1403.01 last 68.34) - open/close 46958 (time 1789055937.01 total 1413.01 last 63.78) - open/close 47624 (time 1789055947.01 total 1423.02 last 66.58) - open/close 48128 (time 1789055957.02 total 1433.03 last 50.34) - open/close 48608 (time 1789055967.02 total 1443.03 last 47.99) - open/close 49106 (time 1789055977.03 total 1453.03 last 49.78) - open/close 49625 (time 1789055987.05 total 1463.05 last 51.80) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1463.18 seconds: 33.92 ops/second - unlinked 0 (time 1789055991 ; total 0 ; last 0) - unlinked 10000 (time 1789056138 ; total 147 ; last 147) - unlinked 20000 (time 1789056310 ; total 319 ; last 172) - unlinked 30000 (time 1789056461 ; total 470 ; last 151) - unlinked 40000 (time 1789056628 ; total 637 ; last 167) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 797 seconds: 62.272270 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg349-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg349-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg349-server stop mds service on oleg349-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg349-server stop mds service on oleg349-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg349-server unloading modules via unload_modules_local on: 'oleg349-server' oleg349-server: oleg349-server.virtnet: executing unload_modules_local oleg349-server: kmemleak disabled modules unloaded. pdsh@oleg349-client: oleg349-server: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-client: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-client: ssh exited with exit code 2