== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:54:06 (1788573246) start mds service on oleg606-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 4 03:00 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg606-server' oleg606-server: oleg606-server.virtnet: executing load_modules_local oleg606-server: kmemleak disabled oleg606-server: Loading modules from /home/green/git/lustre-release/lustre oleg606-server: detected 4 online CPUs by sysfs oleg606-server: MODOPTS_LIBCFS= oleg606-server: Force libcfs to create 2 CPU partitions oleg606-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg606-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg606-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg606-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg606-server: oleg606-server.virtnet: executing set_default_debug -1 all pdsh@oleg606-client: oleg606-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg606-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg606-server: oleg606-server.virtnet: executing set_default_debug -1 all pdsh@oleg606-client: oleg606-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg606-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg606-server: oleg606-server.virtnet: executing set_default_debug -1 all pdsh@oleg606-client: oleg606-server: ssh exited with exit code 1 Started lustre-OST0000 oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg606-client.virtnet: -o user_xattr,flock 192.168.206.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 233 (time 1788573356.03 total 10.91 last 21.36) - open/close 402 (time 1788573366.81 total 21.68 last 15.69) - open/close 554 (time 1788573376.83 total 31.70 last 15.17) - open/close 706 (time 1788573386.84 total 41.72 last 15.18) - open/close 874 (time 1788573396.85 total 51.73 last 16.78) - open/close 1102 (time 1788573406.86 total 61.74 last 22.79) - open/close 1205 (time 1788573416.99 total 71.86 last 10.17) - open/close 1304 (time 1788573427.16 total 82.03 last 9.73) - open/close 1567 (time 1788573437.16 total 92.04 last 26.29) - open/close 1841 (time 1788573447.26 total 102.14 last 27.13) - open/close 1994 (time 1788573457.30 total 112.17 last 15.25) - open/close 2211 (time 1788573468.05 total 122.92 last 20.19) - open/close 2379 (time 1788573478.36 total 133.23 last 16.30) - open/close 2633 (time 1788573490.31 total 145.18 last 21.25) - open/close 2802 (time 1788573501.14 total 156.01 last 15.61) - open/close 2928 (time 1788573511.16 total 166.03 last 12.57) - open/close 3143 (time 1788573521.30 total 176.17 last 21.21) - open/close 3455 (time 1788573531.30 total 186.18 last 31.19) - open/close 3657 (time 1788573541.31 total 196.19 last 20.18) - open/close 3918 (time 1788573551.96 total 206.84 last 24.51) - open/close 4186 (time 1788573561.97 total 216.84 last 26.78) - open/close 4288 (time 1788573572.11 total 226.98 last 10.06) - open/close 4458 (time 1788573584.51 total 239.38 last 13.71) - open/close 4729 (time 1788573594.51 total 249.39 last 27.09) - open/close 4869 (time 1788573604.55 total 259.42 last 13.95) - open/close 5041 (time 1788573615.40 total 270.28 last 15.84) - open/close 5192 (time 1788573625.41 total 280.29 last 15.09) - open/close 5371 (time 1788573636.23 total 291.11 last 16.55) - open/close 5660 (time 1788573646.24 total 301.12 last 28.87) - open/close 5924 (time 1788573656.84 total 311.71 last 24.92) - open/close 6145 (time 1788573666.84 total 321.71 last 22.10) - open/close 6423 (time 1788573676.93 total 331.81 last 27.55) - open/close 6638 (time 1788573687.04 total 341.92 last 21.26) - open/close 6944 (time 1788573697.05 total 351.93 last 30.57) - open/close 7221 (time 1788573707.05 total 361.93 last 27.70) - open/close 7483 (time 1788573718.04 total 372.91 last 23.85) - open/close 7760 (time 1788573728.53 total 383.41 last 26.39) - open/close 8037 (time 1788573739.22 total 394.09 last 25.93) - open/close 8317 (time 1788573750.02 total 404.90 last 25.91) - open/close 8542 (time 1788573760.91 total 415.78 last 20.67) - open/close 8788 (time 1788573771.35 total 426.22 last 23.56) - open/close 9070 (time 1788573781.36 total 436.23 last 28.17) - open/close 9303 (time 1788573791.94 total 446.81 last 22.03) - open/close 9507 (time 1788573802.57 total 457.45 last 19.18) - open/close 9713 (time 1788573813.57 total 468.45 last 18.73) - open/close 9949 (time 1788573823.95 total 478.83 last 22.74) - open/close 10222 (time 1788573834.70 total 489.58 last 25.39) - open/close 10498 (time 1788573845.27 total 500.15 last 26.11) - open/close 10736 (time 1788573855.62 total 510.50 last 23.00) - open/close 11019 (time 1788573866.63 total 521.51 last 25.70) - open/close 11253 (time 1788573877.51 total 532.38 last 21.52) - open/close 11348 (time 1788573888.59 total 543.47 last 8.57) - open/close 11561 (time 1788573898.60 total 553.47 last 21.30) - open/close 11754 (time 1788573909.32 total 564.20 last 17.99) - open/close 11891 (time 1788573919.33 total 574.21 last 13.69) - open/close 12046 (time 1788573929.61 total 584.48 last 15.09) - open/close 12238 (time 1788573940.05 total 594.93 last 18.38) - open/close 12370 (time 1788573953.06 total 607.93 last 10.15) - open/close 12655 (time 1788573963.06 total 617.93 last 28.50) - open/close 12783 (time 1788573973.06 total 627.94 last 12.79) - open/close 12943 (time 1788573983.10 total 637.98 last 15.94) - open/close 13106 (time 1788573993.49 total 648.37 last 15.69) - open/close 13345 (time 1788574003.68 total 658.56 last 23.45) - open/close 13559 (time 1788574014.28 total 669.15 last 20.19) - open/close 13751 (time 1788574024.28 total 679.16 last 19.19) - open/close 13965 (time 1788574034.29 total 689.16 last 21.40) - open/close 14097 (time 1788574045.27 total 700.15 last 12.01) - open/close 14288 (time 1788574055.43 total 710.31 last 18.80) - open/close 14538 (time 1788574066.13 total 721.01 last 23.36) - open/close 14823 (time 1788574076.14 total 731.01 last 28.48) - open/close 15109 (time 1788574086.99 total 741.86 last 26.36) - open/close 15344 (time 1788574098.12 total 753.00 last 21.10) - open/close 15600 (time 1788574108.13 total 763.01 last 25.58) - open/close 15814 (time 1788574118.23 total 773.10 last 21.19) - open/close 16101 (time 1788574129.10 total 783.98 last 26.39) - open/close 16373 (time 1788574139.13 total 794.00 last 27.13) - open/close 16592 (time 1788574150.13 total 805.00 last 19.91) - open/close 16737 (time 1788574160.14 total 815.02 last 14.48) - open/close 16983 (time 1788574170.26 total 825.14 last 24.31) - open/close 17202 (time 1788574180.27 total 835.14 last 21.89) - open/close 17361 (time 1788574190.27 total 845.14 last 15.89) - open/close 17613 (time 1788574200.71 total 855.58 last 24.14) - open/close 17846 (time 1788574210.73 total 865.60 last 23.26) - open/close 18091 (time 1788574221.13 total 876.00 last 23.55) - open/close 18340 (time 1788574231.23 total 886.11 last 24.65) - open/close 18626 (time 1788574241.24 total 896.11 last 28.58) - open/close 18860 (time 1788574251.34 total 906.22 last 23.17) - open/close 19099 (time 1788574261.65 total 916.53 last 23.17) - open/close 19292 (time 1788574272.28 total 927.16 last 18.15) - open/close 19548 (time 1788574282.29 total 937.17 last 25.58) - open/close 19715 (time 1788574292.61 total 947.48 last 16.19) - open/close 19912 (time 1788574302.63 total 957.50 last 19.66) - open/close 20000 (time 1788574306.50 total 961.38 last 22.71) - open/close 20437 (time 1788574316.51 total 971.38 last 43.69) - open/close 21115 (time 1788574326.52 total 981.39 last 67.73) - open/close 21736 (time 1788574336.53 total 991.41 last 62.01) - open/close 22260 (time 1788574346.54 total 1001.42 last 52.33) - open/close 22802 (time 1788574356.55 total 1011.43 last 54.16) - open/close 23504 (time 1788574366.56 total 1021.43 last 70.15) - open/close 24233 (time 1788574376.56 total 1031.43 last 72.90) - open/close 24976 (time 1788574386.57 total 1041.44 last 74.24) - open/close 25681 (time 1788574396.57 total 1051.45 last 70.49) - open/close 26434 (time 1788574406.57 total 1061.45 last 75.29) - open/close 27115 (time 1788574416.57 total 1071.45 last 68.09) - open/close 27887 (time 1788574426.57 total 1081.45 last 77.19) - open/close 28676 (time 1788574436.58 total 1091.46 last 78.84) - open/close 29465 (time 1788574446.60 total 1101.47 last 78.79) - open/close 30000 (time 1788574453.19 total 1108.06 last 81.13) - open/close 30792 (time 1788574463.19 total 1118.07 last 79.18) - open/close 31649 (time 1788574473.19 total 1128.07 last 85.69) - open/close 32531 (time 1788574483.21 total 1138.08 last 88.06) - open/close 33237 (time 1788574493.21 total 1148.09 last 70.59) - open/close 33996 (time 1788574503.22 total 1158.09 last 75.86) - open/close 34780 (time 1788574513.22 total 1168.10 last 78.37) - open/close 35596 (time 1788574523.23 total 1178.10 last 81.54) - open/close 36460 (time 1788574533.23 total 1188.11 last 86.39) - open/close 37227 (time 1788574543.25 total 1198.12 last 76.56) - open/close 37865 (time 1788574553.25 total 1208.13 last 63.76) - open/close 38612 (time 1788574563.26 total 1218.13 last 74.69) - open/close 39321 (time 1788574573.27 total 1228.14 last 70.81) - open/close 39974 (time 1788574583.28 total 1238.16 last 65.20) - open/close 40533 (time 1788574593.29 total 1248.16 last 55.88) - open/close 41130 (time 1788574603.29 total 1258.17 last 59.69) - open/close 41870 (time 1788574613.30 total 1268.17 last 73.95) - open/close 42629 (time 1788574623.31 total 1278.18 last 75.82) - open/close 43299 (time 1788574633.31 total 1288.18 last 66.98) - open/close 44154 (time 1788574643.31 total 1298.19 last 85.47) - open/close 44881 (time 1788574653.31 total 1308.19 last 72.68) - open/close 45660 (time 1788574663.32 total 1318.19 last 77.87) - open/close 46478 (time 1788574673.32 total 1328.20 last 81.76) - open/close 47281 (time 1788574683.33 total 1338.20 last 80.26) - open/close 48085 (time 1788574693.33 total 1348.21 last 80.38) - open/close 48758 (time 1788574703.34 total 1358.21 last 67.27) - open/close 49440 (time 1788574713.35 total 1368.23 last 68.10) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1370.58 seconds: 36.21 ops/second - unlinked 0 (time 1788574719 ; total 0 ; last 0) - unlinked 10000 (time 1788574849 ; total 130 ; last 130) - unlinked 20000 (time 1788574977 ; total 258 ; last 128) - unlinked 30000 (time 1788575122 ; total 403 ; last 145) - unlinked 40000 (time 1788575264 ; total 545 ; last 142) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 684 seconds: 72.559944 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg606-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg606-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg606-server stop mds service on oleg606-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg606-server stop mds service on oleg606-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg606-server unloading modules via unload_modules_local on: 'oleg606-server' oleg606-server: oleg606-server.virtnet: executing unload_modules_local oleg606-server: kmemleak disabled modules unloaded. pdsh@oleg606-client: oleg606-client: ssh exited with exit code 2 pdsh@oleg606-client: oleg606-server: ssh exited with exit code 2 pdsh@oleg606-client: oleg606-client: ssh exited with exit code 2 pdsh@oleg606-client: oleg606-server: ssh exited with exit code 2