== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:09:20 (1790968160) start mds service on oleg355-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Oct 2 13:44 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg355-server' oleg355-server: oleg355-server.virtnet: executing load_modules_local oleg355-server: kmemleak disabled oleg355-server: Loading modules from /home/green/git/lustre-release/lustre oleg355-server: detected 4 online CPUs by sysfs oleg355-server: MODOPTS_LIBCFS= oleg355-server: Force libcfs to create 2 CPU partitions oleg355-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg355-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg355-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg355-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg355-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg355-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 290 (time 1790968271.31 total 10.97 last 26.44) - open/close 552 (time 1790968281.55 total 21.20 last 25.61) - open/close 822 (time 1790968291.58 total 31.23 last 26.91) - open/close 1011 (time 1790968304.35 total 44.00 last 14.80) - open/close 1283 (time 1790968314.83 total 54.48 last 25.94) - open/close 1567 (time 1790968324.85 total 64.50 last 28.35) - open/close 1820 (time 1790968335.19 total 74.85 last 24.46) - open/close 2089 (time 1790968345.74 total 85.39 last 25.51) - open/close 2254 (time 1790968356.11 total 95.76 last 15.91) - open/close 2432 (time 1790968367.50 total 107.16 last 15.62) - open/close 2571 (time 1790968377.51 total 117.17 last 13.89) - open/close 2715 (time 1790968388.56 total 128.22 last 13.03) - open/close 2959 (time 1790968399.39 total 139.04 last 22.55) - open/close 3079 (time 1790968411.12 total 150.77 last 10.23) - open/close 3345 (time 1790968421.90 total 161.56 last 24.66) - open/close 3638 (time 1790968432.35 total 172.01 last 28.04) - open/close 3962 (time 1790968443.28 total 182.94 last 29.65) - open/close 4217 (time 1790968453.28 total 192.94 last 25.50) - open/close 4520 (time 1790968464.22 total 203.87 last 27.72) - open/close 4766 (time 1790968474.31 total 213.97 last 24.36) - open/close 5045 (time 1790968485.13 total 224.79 last 25.78) - open/close 5324 (time 1790968495.14 total 234.79 last 27.89) - open/close 5613 (time 1790968505.62 total 245.27 last 27.57) - open/close 5883 (time 1790968516.39 total 256.04 last 25.07) - open/close 6038 (time 1790968526.73 total 266.39 last 14.98) - open/close 6183 (time 1790968536.87 total 276.52 last 14.30) - open/close 6391 (time 1790968547.16 total 286.81 last 20.22) - open/close 6705 (time 1790968557.45 total 297.10 last 30.51) - open/close 6987 (time 1790968568.20 total 307.86 last 26.22) - open/close 7271 (time 1790968578.20 total 317.86 last 28.40) - open/close 7498 (time 1790968588.37 total 328.02 last 22.33) - open/close 7750 (time 1790968599.13 total 338.79 last 23.41) - open/close 7979 (time 1790968609.16 total 348.82 last 22.83) - open/close 8219 (time 1790968619.18 total 358.83 last 23.96) - open/close 8521 (time 1790968630.04 total 369.70 last 27.80) - open/close 8823 (time 1790968640.06 total 379.71 last 30.16) - open/close 9081 (time 1790968650.29 total 389.94 last 25.22) - open/close 9301 (time 1790968661.14 total 400.79 last 20.27) - open/close 9588 (time 1790968672.15 total 411.80 last 26.07) - open/close 9868 (time 1790968682.78 total 422.43 last 26.35) - open/close 10000 (time 1790968687.84 total 427.49 last 26.06) - open/close 10228 (time 1790968697.85 total 437.50 last 22.78) - open/close 10407 (time 1790968708.39 total 448.04 last 16.99) - open/close 10669 (time 1790968719.33 total 458.98 last 23.95) - open/close 11014 (time 1790968729.62 total 469.28 last 33.51) - open/close 11298 (time 1790968739.87 total 479.52 last 27.71) - open/close 11574 (time 1790968749.96 total 489.62 last 27.35) - open/close 11870 (time 1790968759.97 total 499.62 last 29.57) - open/close 12178 (time 1790968769.98 total 509.63 last 30.77) - open/close 12365 (time 1790968780.06 total 519.72 last 18.55) - open/close 12675 (time 1790968790.92 total 530.57 last 28.55) - open/close 12975 (time 1790968801.95 total 541.60 last 27.21) - open/close 13249 (time 1790968812.17 total 551.82 last 26.81) - open/close 13524 (time 1790968822.17 total 561.83 last 27.49) - open/close 13765 (time 1790968832.54 total 572.19 last 23.25) - open/close 14016 (time 1790968842.54 total 582.20 last 25.08) - open/close 14267 (time 1790968852.56 total 592.21 last 25.07) - open/close 14514 (time 1790968862.56 total 602.22 last 24.69) - open/close 14782 (time 1790968872.57 total 612.23 last 26.77) - open/close 15053 (time 1790968882.58 total 622.23 last 27.08) - open/close 15192 (time 1790968892.60 total 632.26 last 13.87) - open/close 15416 (time 1790968902.61 total 642.26 last 22.39) - open/close 15588 (time 1790968912.61 total 652.26 last 17.20) - open/close 15813 (time 1790968922.74 total 662.40 last 22.20) - open/close 16100 (time 1790968932.75 total 672.40 last 28.69) - open/close 16435 (time 1790968942.75 total 682.41 last 33.47) - open/close 16687 (time 1790968952.94 total 692.59 last 24.74) - open/close 16917 (time 1790968963.28 total 702.94 last 22.23) - open/close 17221 (time 1790968974.09 total 713.75 last 28.13) - open/close 17428 (time 1790968984.11 total 723.76 last 20.67) - open/close 17541 (time 1790968994.15 total 733.80 last 11.25) - open/close 17758 (time 1790969005.90 total 745.55 last 18.46) - open/close 18005 (time 1790969016.97 total 756.62 last 22.31) - open/close 18257 (time 1790969026.98 total 766.63 last 25.18) - open/close 18544 (time 1790969037.52 total 777.18 last 27.22) - open/close 18850 (time 1790969047.56 total 787.22 last 30.48) - open/close 19073 (time 1790969058.01 total 797.66 last 21.35) - open/close 19254 (time 1790969068.50 total 808.15 last 17.25) - open/close 19518 (time 1790969079.48 total 819.14 last 24.03) - open/close 19816 (time 1790969089.50 total 829.15 last 29.76) - open/close 20000 (time 1790969097.54 total 837.20 last 22.87) - open/close 20205 (time 1790969108.13 total 847.78 last 19.36) - open/close 20428 (time 1790969119.13 total 858.78 last 20.28) - open/close 20692 (time 1790969129.70 total 869.35 last 24.97) - open/close 20900 (time 1790969139.70 total 879.36 last 20.79) - open/close 21145 (time 1790969149.72 total 889.37 last 24.47) - open/close 21470 (time 1790969159.73 total 899.38 last 32.47) - open/close 21724 (time 1790969170.58 total 910.23 last 23.40) - open/close 22009 (time 1790969180.59 total 920.25 last 28.47) - open/close 22604 (time 1790969190.60 total 930.25 last 59.46) - open/close 23430 (time 1790969200.60 total 940.26 last 82.56) - open/close 24363 (time 1790969210.61 total 950.27 last 93.24) - open/close 25244 (time 1790969220.63 total 960.28 last 87.97) - open/close 26130 (time 1790969230.64 total 970.29 last 88.52) - open/close 26874 (time 1790969240.65 total 980.30 last 74.33) - open/close 27629 (time 1790969250.65 total 990.31 last 75.43) - open/close 28621 (time 1790969260.66 total 1000.31 last 99.14) - open/close 29482 (time 1790969270.66 total 1010.32 last 86.09) - open/close 30000 (time 1790969276.33 total 1015.99 last 91.31) - open/close 30969 (time 1790969286.34 total 1025.99 last 96.84) - open/close 31804 (time 1790969296.35 total 1036.00 last 83.46) - open/close 32751 (time 1790969306.35 total 1046.01 last 94.61) - open/close 33684 (time 1790969316.36 total 1056.01 last 93.24) - open/close 34569 (time 1790969326.39 total 1066.04 last 88.28) - open/close 35360 (time 1790969336.39 total 1076.04 last 79.07) - open/close 36183 (time 1790969346.39 total 1086.05 last 82.28) - open/close 37029 (time 1790969356.40 total 1096.05 last 84.55) - open/close 38008 (time 1790969366.41 total 1106.06 last 97.80) - open/close 38865 (time 1790969376.41 total 1116.07 last 85.66) - open/close 39732 (time 1790969386.41 total 1126.07 last 86.70) - open/close 40000 (time 1790969389.87 total 1129.52 last 77.51) - open/close 40777 (time 1790969399.87 total 1139.53 last 77.69) - open/close 41617 (time 1790969409.89 total 1149.54 last 83.89) - open/close 42504 (time 1790969419.90 total 1159.55 last 88.61) - open/close 43218 (time 1790969429.90 total 1169.55 last 71.39) - open/close 43992 (time 1790969439.92 total 1179.57 last 77.26) - open/close 44890 (time 1790969449.92 total 1189.57 last 89.75) - open/close 45811 (time 1790969459.93 total 1199.58 last 92.01) - open/close 46715 (time 1790969469.94 total 1209.59 last 90.36) - open/close 47550 (time 1790969479.94 total 1219.60 last 83.43) - open/close 48429 (time 1790969489.94 total 1229.60 last 87.89) - open/close 49296 (time 1790969499.98 total 1239.63 last 86.41) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1243.34 seconds: 39.92 ops/second - unlinked 0 (time 1790969508 ; total 0 ; last 0) - unlinked 10000 (time 1790969624 ; total 116 ; last 116) - unlinked 20000 (time 1790969742 ; total 234 ; last 118) - unlinked 30000 (time 1790969818 ; total 310 ; last 76) - unlinked 40000 (time 1790969887 ; total 379 ; last 69) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 446 seconds: 111.280266 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg355-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg355-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg355-server stop mds service on oleg355-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg355-server stop mds service on oleg355-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg355-server unloading modules via unload_modules_local on: 'oleg355-server' oleg355-server: oleg355-server.virtnet: executing unload_modules_local oleg355-server: kmemleak disabled modules unloaded. pdsh@oleg355-client: oleg355-client: ssh exited with exit code 2 pdsh@oleg355-client: oleg355-server: ssh exited with exit code 2 pdsh@oleg355-client: oleg355-client: ssh exited with exit code 2 pdsh@oleg355-client: oleg355-server: ssh exited with exit code 2