== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:39:34 (1787841574) start mds service on oleg628-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754920 Aug 27 09:47 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg628-server' oleg628-server: oleg628-server.virtnet: executing load_modules_local oleg628-server: kmemleak disabled oleg628-server: Loading modules from /home/green/git/lustre-release/lustre oleg628-server: detected 4 online CPUs by sysfs oleg628-server: MODOPTS_LIBCFS= oleg628-server: Force libcfs to create 2 CPU partitions oleg628-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg628-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg628-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg628-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg628-server: oleg628-server.virtnet: executing set_default_debug -1 all pdsh@oleg628-client: oleg628-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg628-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg628-server: oleg628-server.virtnet: executing set_default_debug -1 all pdsh@oleg628-client: oleg628-server: ssh exited with exit code 1 Started lustre-OST0000 oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg628-client.virtnet: -o user_xattr,flock 192.168.206.128@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 284 (time 1787841659.79 total 10.01 last 28.38) - open/close 464 (time 1787841671.43 total 21.64 last 15.47) - open/close 727 (time 1787841681.44 total 31.65 last 26.28) - open/close 983 (time 1787841692.30 total 42.52 last 23.56) - open/close 1259 (time 1787841702.32 total 52.53 last 27.56) - open/close 1514 (time 1787841713.24 total 63.45 last 23.35) - open/close 1652 (time 1787841727.32 total 77.53 last 9.80) - open/close 1910 (time 1787841737.32 total 87.54 last 25.80) - open/close 2064 (time 1787841747.35 total 97.57 last 15.35) - open/close 2268 (time 1787841757.36 total 107.57 last 20.39) - open/close 2510 (time 1787841767.36 total 117.57 last 24.19) - open/close 2720 (time 1787841777.56 total 127.77 last 20.59) - open/close 2921 (time 1787841787.57 total 137.79 last 20.07) - open/close 3159 (time 1787841798.10 total 148.31 last 22.61) - open/close 3411 (time 1787841808.12 total 158.34 last 25.15) - open/close 3648 (time 1787841818.84 total 169.06 last 22.11) - open/close 3873 (time 1787841828.85 total 179.06 last 22.48) - open/close 4060 (time 1787841841.92 total 192.14 last 14.30) - open/close 4238 (time 1787841851.96 total 202.17 last 17.73) - open/close 4361 (time 1787841861.97 total 212.19 last 12.29) - open/close 4498 (time 1787841871.99 total 222.21 last 13.67) - open/close 4635 (time 1787841882.01 total 232.22 last 13.68) - open/close 4773 (time 1787841892.03 total 242.25 last 13.77) - open/close 4854 (time 1787841902.03 total 252.25 last 8.10) - open/close 5126 (time 1787841913.10 total 263.32 last 24.57) - open/close 5370 (time 1787841923.86 total 274.07 last 22.69) - open/close 5595 (time 1787841933.86 total 284.08 last 22.49) - open/close 5846 (time 1787841944.14 total 294.35 last 24.44) - open/close 6041 (time 1787841954.14 total 304.36 last 19.49) - open/close 6198 (time 1787841964.17 total 314.39 last 15.65) - open/close 6309 (time 1787841974.54 total 324.76 last 10.70) - open/close 6433 (time 1787841984.55 total 334.76 last 12.39) - open/close 6663 (time 1787841995.12 total 345.34 last 21.75) - open/close 6844 (time 1787842005.15 total 355.37 last 18.05) - open/close 7090 (time 1787842016.15 total 366.37 last 22.36) - open/close 7336 (time 1787842026.16 total 376.38 last 24.58) - open/close 7569 (time 1787842039.04 total 389.26 last 18.09) - open/close 7757 (time 1787842049.06 total 399.27 last 18.77) - open/close 7948 (time 1787842059.35 total 409.56 last 18.57) - open/close 8217 (time 1787842069.35 total 419.57 last 26.89) - open/close 8487 (time 1787842079.90 total 430.12 last 25.60) - open/close 8712 (time 1787842090.07 total 440.28 last 22.13) - open/close 8949 (time 1787842100.96 total 451.18 last 21.75) - open/close 9182 (time 1787842111.76 total 461.97 last 21.58) - open/close 9375 (time 1787842121.76 total 471.98 last 19.30) - open/close 9594 (time 1787842131.76 total 481.98 last 21.89) - open/close 9814 (time 1787842141.78 total 491.99 last 21.96) - open/close 10000 (time 1787842150.27 total 500.48 last 21.91) - open/close 10203 (time 1787842160.40 total 510.62 last 20.03) - open/close 10443 (time 1787842170.41 total 520.63 last 23.97) - open/close 10645 (time 1787842180.42 total 530.64 last 20.19) - open/close 10785 (time 1787842190.43 total 540.65 last 13.99) - open/close 11008 (time 1787842200.44 total 550.65 last 22.28) - open/close 11208 (time 1787842210.76 total 560.97 last 19.38) - open/close 11438 (time 1787842221.14 total 571.36 last 22.15) - open/close 11676 (time 1787842231.87 total 582.09 last 22.18) - open/close 11892 (time 1787842241.93 total 592.14 last 21.48) - open/close 12115 (time 1787842251.94 total 602.15 last 22.28) - open/close 12352 (time 1787842262.47 total 612.68 last 22.51) - open/close 12567 (time 1787842272.48 total 622.69 last 21.47) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-12593) error: No space left on device total: 12593 open/close in 623.20 seconds: 20.21 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9017: kill: (75944) - No such process - unlinked 0 (time 1787842283 ; total 1 ; last 1) - unlinked 10000 (time 1787842379 ; total 97 ; last 96) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-12593) error: No such file or directory total: 12593 unlinks in 121 seconds: 104.074379 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg628-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg628-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg628-server stop mds service on oleg628-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg628-server unloading modules via unload_modules_local on: 'oleg628-server' oleg628-server: oleg628-server.virtnet: executing unload_modules_local oleg628-server: kmemleak disabled modules unloaded. pdsh@oleg628-client: oleg628-client: ssh exited with exit code 2 pdsh@oleg628-client: oleg628-server: ssh exited with exit code 2 pdsh@oleg628-client: oleg628-server: ssh exited with exit code 2 pdsh@oleg628-client: oleg628-client: ssh exited with exit code 2