== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:04:52 (1788869092) start mds service on oleg630-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 8 04:31 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg630-server' oleg630-server: oleg630-server.virtnet: executing load_modules_local oleg630-server: kmemleak disabled oleg630-server: Loading modules from /home/green/git/lustre-release/lustre oleg630-server: detected 4 online CPUs by sysfs oleg630-server: MODOPTS_LIBCFS= oleg630-server: Force libcfs to create 2 CPU partitions oleg630-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg630-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg630-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg630-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg630-server: oleg630-server.virtnet: executing set_default_debug -1 all pdsh@oleg630-client: oleg630-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg630-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg630-server: oleg630-server.virtnet: executing set_default_debug -1 all pdsh@oleg630-client: oleg630-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg630-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg630-server: oleg630-server.virtnet: executing set_default_debug -1 all pdsh@oleg630-client: oleg630-server: ssh exited with exit code 1 Started lustre-OST0000 oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg630-client.virtnet: -o user_xattr,flock 192.168.206.130@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 252 (time 1788869156.49 total 10.54 last 23.91) - open/close 513 (time 1788869166.49 total 20.54 last 26.10) - open/close 752 (time 1788869177.36 total 31.41 last 22.00) - open/close 1005 (time 1788869188.30 total 42.35 last 23.12) - open/close 1300 (time 1788869198.67 total 52.72 last 28.45) - open/close 1580 (time 1788869208.84 total 62.89 last 27.52) - open/close 1875 (time 1788869219.15 total 73.20 last 28.61) - open/close 2242 (time 1788869229.16 total 83.21 last 36.68) - open/close 2557 (time 1788869239.17 total 93.22 last 31.47) - open/close 2919 (time 1788869250.07 total 104.12 last 33.21) - open/close 3247 (time 1788869260.88 total 114.93 last 30.34) - open/close 3569 (time 1788869271.43 total 125.48 last 30.51) - open/close 3913 (time 1788869281.99 total 136.04 last 32.57) - open/close 4157 (time 1788869292.11 total 146.16 last 24.12) - open/close 4424 (time 1788869302.99 total 157.03 last 24.54) - open/close 4645 (time 1788869313.67 total 167.72 last 20.68) - open/close 4898 (time 1788869324.43 total 178.48 last 23.52) - open/close 5250 (time 1788869335.19 total 189.24 last 32.72) - open/close 5564 (time 1788869345.61 total 199.66 last 30.13) - open/close 5831 (time 1788869356.62 total 210.67 last 24.26) - open/close 6074 (time 1788869367.31 total 221.36 last 22.73) - open/close 6321 (time 1788869378.31 total 232.36 last 22.45) - open/close 6616 (time 1788869388.62 total 242.67 last 28.62) - open/close 6871 (time 1788869398.73 total 252.78 last 25.22) - open/close 7198 (time 1788869409.43 total 263.48 last 30.56) - open/close 7493 (time 1788869419.92 total 273.97 last 28.13) - open/close 7776 (time 1788869430.15 total 284.20 last 27.64) - open/close 8070 (time 1788869440.46 total 294.51 last 28.53) - open/close 8316 (time 1788869451.15 total 305.20 last 23.02) - open/close 8632 (time 1788869461.45 total 315.50 last 30.68) - open/close 8880 (time 1788869472.33 total 326.38 last 22.80) - open/close 9077 (time 1788869482.77 total 336.82 last 18.87) - open/close 9340 (time 1788869493.70 total 347.75 last 24.05) - open/close 9675 (time 1788869503.71 total 357.76 last 33.49) - open/close 9974 (time 1788869514.13 total 368.17 last 28.70) - open/close 10312 (time 1788869524.68 total 378.73 last 32.01) - open/close 10519 (time 1788869535.31 total 389.36 last 19.48) - open/close 10768 (time 1788869545.31 total 399.36 last 24.89) - open/close 11019 (time 1788869555.34 total 409.39 last 25.05) - open/close 11523 (time 1788869565.34 total 419.39 last 50.39) - open/close 13264 (time 1788869575.34 total 429.39 last 174.06) - open/close 14915 (time 1788869585.34 total 439.39 last 165.07) - open/close 16357 (time 1788869595.35 total 449.40 last 144.14) - open/close 17785 (time 1788869605.35 total 459.40 last 142.76) - open/close 19430 (time 1788869615.35 total 469.40 last 164.43) - open/close 20000 (time 1788869619.05 total 473.10 last 154.12) - open/close 21706 (time 1788869629.06 total 483.10 last 170.54) - open/close 23343 (time 1788869639.06 total 493.11 last 163.67) - open/close 24987 (time 1788869649.06 total 503.11 last 164.36) - open/close 27027 (time 1788869659.06 total 513.11 last 204.00) - open/close 28903 (time 1788869669.06 total 523.11 last 187.60) - open/close 30000 (time 1788869675.54 total 529.59 last 169.34) - open/close 31898 (time 1788869685.54 total 539.59 last 189.77) - open/close 33788 (time 1788869695.55 total 549.59 last 188.89) - open/close 35424 (time 1788869705.55 total 559.60 last 163.56) - open/close 37323 (time 1788869715.55 total 569.60 last 189.87) - open/close 39085 (time 1788869725.56 total 579.61 last 176.09) - open/close 40000 (time 1788869731.21 total 585.26 last 161.83) - open/close 41882 (time 1788869741.21 total 595.26 last 188.14) - open/close 43731 (time 1788869751.22 total 605.27 last 184.82) - open/close 45371 (time 1788869761.22 total 615.27 last 163.95) - open/close 47220 (time 1788869771.22 total 625.27 last 184.84) - open/close 49439 (time 1788869781.23 total 635.27 last 221.87) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 636.34 seconds: 77.99 ops/second - unlinked 0 (time 1788869784 ; total 0 ; last 0) - unlinked 10000 (time 1788869815 ; total 31 ; last 31) - unlinked 20000 (time 1788869841 ; total 57 ; last 26) - unlinked 30000 (time 1788869869 ; total 85 ; last 28) - unlinked 40000 (time 1788869899 ; total 115 ; last 30) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 142 seconds: 349.514099 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg630-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg630-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg630-server stop mds service on oleg630-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg630-server stop mds service on oleg630-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg630-server unloading modules via unload_modules_local on: 'oleg630-server' oleg630-server: oleg630-server.virtnet: executing unload_modules_local oleg630-server: kmemleak disabled modules unloaded. pdsh@oleg630-client: oleg630-client: ssh exited with exit code 2 pdsh@oleg630-client: oleg630-server: ssh exited with exit code 2 pdsh@oleg630-client: oleg630-client: ssh exited with exit code 2 pdsh@oleg630-client: oleg630-server: ssh exited with exit code 2