== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:03:15 (1790776995) start mds service on oleg238-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1952976 Sep 29 14:03 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing load_modules_local oleg238-server: kmemleak disabled oleg238-server: Loading modules from /home/green/git/lustre-release/lustre oleg238-server: detected 4 online CPUs by sysfs oleg238-server: MODOPTS_LIBCFS= oleg238-server: Force libcfs to create 2 CPU partitions oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg238-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg238-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg238-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg238-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 167 (time 1790777111.15 total 10.82 last 15.43) - open/close 312 (time 1790777121.68 total 21.35 last 13.77) - open/close 434 (time 1790777132.68 total 32.36 last 11.09) - open/close 528 (time 1790777142.86 total 42.53 last 9.24) - open/close 791 (time 1790777153.03 total 52.71 last 25.86) - open/close 1044 (time 1790777163.06 total 62.73 last 25.23) - open/close 1303 (time 1790777173.78 total 73.46 last 24.15) - open/close 1568 (time 1790777183.79 total 83.47 last 26.48) - open/close 1744 (time 1790777193.79 total 93.47 last 17.60) - open/close 2034 (time 1790777204.66 total 104.33 last 26.70) - open/close 2263 (time 1790777214.72 total 114.40 last 22.75) - open/close 2419 (time 1790777225.11 total 124.79 last 15.01) - open/close 2728 (time 1790777235.56 total 135.24 last 29.57) - open/close 2866 (time 1790777245.61 total 145.28 last 13.74) - open/close 3147 (time 1790777255.61 total 155.29 last 28.08) - open/close 3389 (time 1790777266.13 total 165.80 last 23.01) - open/close 3575 (time 1790777276.46 total 176.14 last 18.00) - open/close 3697 (time 1790777289.46 total 189.13 last 9.39) - open/close 3890 (time 1790777299.60 total 199.28 last 19.02) - open/close 4061 (time 1790777310.48 total 210.16 last 15.72) - open/close 4303 (time 1790777321.42 total 221.09 last 22.13) - open/close 4527 (time 1790777333.58 total 233.25 last 18.43) - open/close 4713 (time 1790777343.96 total 243.63 last 17.91) - open/close 5005 (time 1790777353.98 total 253.66 last 29.13) - open/close 5141 (time 1790777363.99 total 263.66 last 13.59) - open/close 5330 (time 1790777374.82 total 274.49 last 17.46) - open/close 5490 (time 1790777384.84 total 284.52 last 15.96) - open/close 5579 (time 1790777395.10 total 294.78 last 8.67) - open/close 5736 (time 1790777406.03 total 305.70 last 14.37) - open/close 5853 (time 1790777416.45 total 316.12 last 11.23) - open/close 6146 (time 1790777426.45 total 326.13 last 29.30) - open/close 6436 (time 1790777436.47 total 336.14 last 28.95) - open/close 6641 (time 1790777447.28 total 346.95 last 18.97) - open/close 6892 (time 1790777457.29 total 356.97 last 25.06) - open/close 7077 (time 1790777467.30 total 366.98 last 18.48) - open/close 7150 (time 1790777478.35 total 378.03 last 6.61) - open/close 7446 (time 1790777488.36 total 388.03 last 29.59) - open/close 7675 (time 1790777498.79 total 398.47 last 21.94) - open/close 7942 (time 1790777508.90 total 408.57 last 26.43) - open/close 8134 (time 1790777518.91 total 418.59 last 19.17) - open/close 8315 (time 1790777529.21 total 428.89 last 17.57) - open/close 8495 (time 1790777539.48 total 439.15 last 17.54) - open/close 8709 (time 1790777549.48 total 449.15 last 21.39) - open/close 9016 (time 1790777560.46 total 460.14 last 27.95) - open/close 9279 (time 1790777570.46 total 470.14 last 26.30) - open/close 9556 (time 1790777581.19 total 480.86 last 25.83) - open/close 9774 (time 1790777591.58 total 491.25 last 20.98) - open/close 9983 (time 1790777601.89 total 501.57 last 20.26) - open/close 10221 (time 1790777611.97 total 511.65 last 23.61) - open/close 10482 (time 1790777622.61 total 522.28 last 24.54) - open/close 10703 (time 1790777633.15 total 532.83 last 20.96) - open/close 10989 (time 1790777643.18 total 542.85 last 28.52) - open/close 11250 (time 1790777653.77 total 553.45 last 24.64) - open/close 11568 (time 1790777663.78 total 563.46 last 31.77) - open/close 11777 (time 1790777673.80 total 573.47 last 20.87) - open/close 12025 (time 1790777684.63 total 584.30 last 22.91) - open/close 12267 (time 1790777695.65 total 595.32 last 21.96) - open/close 12486 (time 1790777705.81 total 605.48 last 21.55) - open/close 12741 (time 1790777716.26 total 615.93 last 24.41) - open/close 12997 (time 1790777726.27 total 625.95 last 25.55) - open/close 13195 (time 1790777736.28 total 635.96 last 19.78) - open/close 13270 (time 1790777746.71 total 646.38 last 7.19) - open/close 13521 (time 1790777756.72 total 656.39 last 25.07) - open/close 13578 (time 1790777767.06 total 666.74 last 5.51) - open/close 13752 (time 1790777777.67 total 677.35 last 16.40) - open/close 14055 (time 1790777787.67 total 687.35 last 30.30) - open/close 14254 (time 1790777797.94 total 697.62 last 19.38) - open/close 14432 (time 1790777808.34 total 708.01 last 17.13) - open/close 14692 (time 1790777818.69 total 718.36 last 25.11) - open/close 14937 (time 1790777829.06 total 728.73 last 23.63) - open/close 15232 (time 1790777839.07 total 738.75 last 29.45) - open/close 15512 (time 1790777849.75 total 749.43 last 26.22) - open/close 15739 (time 1790777862.22 total 761.89 last 18.21) - open/close 15967 (time 1790777872.22 total 771.90 last 22.78) - open/close 16248 (time 1790777883.10 total 782.78 last 25.83) - open/close 16463 (time 1790777893.11 total 792.78 last 21.48) - open/close 16586 (time 1790777903.57 total 803.24 last 11.76) - open/close 16817 (time 1790777913.73 total 813.41 last 22.72) - open/close 17036 (time 1790777924.37 total 824.04 last 20.60) - open/close 17282 (time 1790777934.92 total 834.59 last 23.32) - open/close 17599 (time 1790777944.92 total 844.59 last 31.70) - open/close 17887 (time 1790777955.61 total 855.28 last 26.94) - open/close 18113 (time 1790777965.68 total 865.35 last 22.44) - open/close 18380 (time 1790777975.69 total 875.36 last 26.67) - open/close 18486 (time 1790777986.20 total 885.87 last 10.09) - open/close 18783 (time 1790777996.22 total 895.89 last 29.62) - open/close 18983 (time 1790778006.80 total 906.47 last 18.91) - open/close 19267 (time 1790778017.68 total 917.36 last 26.10) - open/close 19421 (time 1790778027.70 total 927.37 last 15.38) - open/close 19691 (time 1790778037.70 total 937.37 last 27.00) - open/close 19904 (time 1790778047.71 total 947.38 last 21.27) - open/close 20000 (time 1790778051.18 total 950.85 last 27.70) - open/close 20213 (time 1790778061.84 total 961.52 last 19.97) - open/close 20856 (time 1790778071.85 total 971.53 last 64.25) - open/close 21832 (time 1790778081.85 total 981.53 last 97.58) - open/close 22960 (time 1790778091.86 total 991.53 last 112.76) - open/close 23919 (time 1790778101.86 total 1001.54 last 95.84) - open/close 24970 (time 1790778111.88 total 1011.55 last 104.92) - open/close 25850 (time 1790778121.90 total 1021.57 last 87.85) - open/close 26890 (time 1790778131.90 total 1031.58 last 103.94) - open/close 27866 (time 1790778141.91 total 1041.58 last 97.54) - open/close 28798 (time 1790778151.91 total 1051.58 last 93.19) - open/close 29659 (time 1790778161.92 total 1061.59 last 86.03) - open/close 30000 (time 1790778165.92 total 1065.59 last 85.29) - open/close 31037 (time 1790778175.93 total 1075.60 last 103.58) - open/close 31759 (time 1790778185.93 total 1085.61 last 72.16) - open/close 32433 (time 1790778195.94 total 1095.61 last 67.38) - open/close 33204 (time 1790778205.95 total 1105.62 last 77.02) - open/close 34115 (time 1790778215.95 total 1115.62 last 91.08) - open/close 34942 (time 1790778225.95 total 1125.62 last 82.69) - open/close 36082 (time 1790778235.95 total 1135.63 last 113.96) - open/close 37142 (time 1790778245.96 total 1145.63 last 105.97) - open/close 38130 (time 1790778255.97 total 1155.64 last 98.70) - open/close 38957 (time 1790778265.97 total 1165.64 last 82.70) - open/close 40000 (time 1790778274.87 total 1174.55 last 117.12) - open/close 41133 (time 1790778284.87 total 1184.55 last 113.29) - open/close 42024 (time 1790778294.89 total 1194.56 last 89.00) - open/close 42842 (time 1790778304.89 total 1204.56 last 81.76) - open/close 43573 (time 1790778314.89 total 1214.57 last 73.07) - open/close 44459 (time 1790778324.90 total 1224.58 last 88.53) - open/close 45150 (time 1790778334.90 total 1234.58 last 69.09) - open/close 45869 (time 1790778344.90 total 1244.58 last 71.89) - open/close 46791 (time 1790778354.91 total 1254.58 last 92.13) - open/close 47744 (time 1790778364.93 total 1264.60 last 95.13) - open/close 48571 (time 1790778374.93 total 1274.61 last 82.67) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1283.64 seconds: 38.66 ops/second - unlinked 0 (time 1790778387 ; total 0 ; last 0) - unlinked 10000 (time 1790778496 ; total 109 ; last 109) - unlinked 20000 (time 1790778607 ; total 220 ; last 111) - unlinked 30000 (time 1790778675 ; total 288 ; last 68) - unlinked 40000 (time 1790778739 ; total 352 ; last 64) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 415 seconds: 119.592773 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server stop mds service on oleg238-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server stop mds service on oleg238-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server unloading modules via unload_modules_local on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing unload_modules_local oleg238-server: kmemleak disabled modules unloaded. pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2