== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:04:00 (1790744640) start mds service on oleg247-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1947032 Sep 29 13:24 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing load_modules_local oleg247-server: kmemleak disabled oleg247-server: Loading modules from /home/green/git/lustre-release/lustre oleg247-server: detected 4 online CPUs by sysfs oleg247-server: MODOPTS_LIBCFS= oleg247-server: Force libcfs to create 2 CPU partitions oleg247-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg247-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg247-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg247-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg247-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg247-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 242 (time 1790744765.23 total 10.81 last 22.38) - open/close 398 (time 1790744775.48 total 21.06 last 15.22) - open/close 621 (time 1790744785.48 total 31.07 last 22.29) - open/close 765 (time 1790744796.03 total 41.61 last 13.66) - open/close 975 (time 1790744808.05 total 53.63 last 17.47) - open/close 1222 (time 1790744818.05 total 63.63 last 24.70) - open/close 1463 (time 1790744828.75 total 74.33 last 22.53) - open/close 1702 (time 1790744839.75 total 85.33 last 21.72) - open/close 1964 (time 1790744850.66 total 96.25 last 24.01) - open/close 2078 (time 1790744860.68 total 106.27 last 11.38) - open/close 2292 (time 1790744871.03 total 116.61 last 20.69) - open/close 2356 (time 1790744881.58 total 127.16 last 6.07) - open/close 2547 (time 1790744893.55 total 139.13 last 15.96) - open/close 2746 (time 1790744903.56 total 149.14 last 19.87) - open/close 2880 (time 1790744913.56 total 159.15 last 13.39) - open/close 3036 (time 1790744923.59 total 169.17 last 15.57) - open/close 3203 (time 1790744934.50 total 180.09 last 15.30) - open/close 3402 (time 1790744945.33 total 190.91 last 18.39) - open/close 3572 (time 1790744955.34 total 200.92 last 16.99) - open/close 3741 (time 1790744965.54 total 211.12 last 16.56) - open/close 3946 (time 1790744975.94 total 221.52 last 19.72) - open/close 4119 (time 1790744986.85 total 232.44 last 15.85) - open/close 4252 (time 1790744996.86 total 242.45 last 13.29) - open/close 4355 (time 1790745007.40 total 252.98 last 9.78) - open/close 4541 (time 1790745017.40 total 262.98 last 18.60) - open/close 4683 (time 1790745028.21 total 273.79 last 13.14) - open/close 4809 (time 1790745038.76 total 284.34 last 11.94) - open/close 5043 (time 1790745049.27 total 294.85 last 22.27) - open/close 5280 (time 1790745059.38 total 304.97 last 23.44) - open/close 5496 (time 1790745069.56 total 315.14 last 21.22) - open/close 5776 (time 1790745080.36 total 325.94 last 25.92) - open/close 6062 (time 1790745091.26 total 336.84 last 26.25) - open/close 6334 (time 1790745101.54 total 347.12 last 26.45) - open/close 6578 (time 1790745112.04 total 357.62 last 23.25) - open/close 6863 (time 1790745122.04 total 367.62 last 28.49) - open/close 7021 (time 1790745132.06 total 377.64 last 15.77) - open/close 7234 (time 1790745142.38 total 387.96 last 20.65) - open/close 7511 (time 1790745152.49 total 398.07 last 27.41) - open/close 7767 (time 1790745163.37 total 408.95 last 23.53) - open/close 8034 (time 1790745173.37 total 418.95 last 26.68) - open/close 8236 (time 1790745183.40 total 428.99 last 20.13) - open/close 8483 (time 1790745193.41 total 438.99 last 24.68) - open/close 8738 (time 1790745204.08 total 449.66 last 23.91) - open/close 8978 (time 1790745214.38 total 459.96 last 23.29) - open/close 9140 (time 1790745224.89 total 470.48 last 15.41) - open/close 9352 (time 1790745235.13 total 480.71 last 20.71) - open/close 9625 (time 1790745246.21 total 491.79 last 24.63) - open/close 9798 (time 1790745259.84 total 505.42 last 12.70) - open/close 10000 (time 1790745267.82 total 513.40 last 25.31) - open/close 10160 (time 1790745280.11 total 525.69 last 13.02) - open/close 10251 (time 1790745292.16 total 537.74 last 7.55) - open/close 10366 (time 1790745302.16 total 547.74 last 11.49) - open/close 10639 (time 1790745312.17 total 557.75 last 27.28) - open/close 10876 (time 1790745322.87 total 568.45 last 22.16) - open/close 11077 (time 1790745332.89 total 578.48 last 20.04) - open/close 11100 (time 1790745343.72 total 589.30 last 2.12) - open/close 11353 (time 1790745354.54 total 600.12 last 23.39) - open/close 11624 (time 1790745365.48 total 611.06 last 24.77) - open/close 11895 (time 1790745375.48 total 621.07 last 27.09) - open/close 12156 (time 1790745386.35 total 631.93 last 24.01) - open/close 12388 (time 1790745396.35 total 641.94 last 23.20) - open/close 12644 (time 1790745407.34 total 652.93 last 23.30) - open/close 12802 (time 1790745417.97 total 663.55 last 14.87) - open/close 12950 (time 1790745427.99 total 673.57 last 14.77) - open/close 13194 (time 1790745438.00 total 683.58 last 24.37) - open/close 13375 (time 1790745448.70 total 694.28 last 16.92) - open/close 13619 (time 1790745459.37 total 704.95 last 22.87) - open/close 13861 (time 1790745470.18 total 715.76 last 22.39) - open/close 14100 (time 1790745480.18 total 725.77 last 23.89) - open/close 14371 (time 1790745490.67 total 736.25 last 25.84) - open/close 14604 (time 1790745500.67 total 746.26 last 23.29) - open/close 14821 (time 1790745510.85 total 756.43 last 21.33) - open/close 15088 (time 1790745520.87 total 766.45 last 26.64) - open/close 15317 (time 1790745530.99 total 776.57 last 22.64) - open/close 15538 (time 1790745541.00 total 786.58 last 22.07) - open/close 15702 (time 1790745551.02 total 796.60 last 16.37) - open/close 15965 (time 1790745561.33 total 806.92 last 25.50) - open/close 16115 (time 1790745571.85 total 817.43 last 14.27) - open/close 16182 (time 1790745582.13 total 827.71 last 6.52) - open/close 16377 (time 1790745594.36 total 839.94 last 15.94) - open/close 16523 (time 1790745604.44 total 850.02 last 14.49) - open/close 16648 (time 1790745615.93 total 861.51 last 10.88) - open/close 16759 (time 1790745626.88 total 872.46 last 10.14) - open/close 17039 (time 1790745637.24 total 882.82 last 27.03) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-17041) error: No space left on device total: 17041 open/close in 882.86 seconds: 19.30 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9030: kill: (115058) - No such process - unlinked 0 (time 1790745644 ; total 0 ; last 0) - unlinked 10000 (time 1790745759 ; total 115 ; last 115) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-17041) error: No such file or directory total: 17041 unlinks in 170 seconds: 100.241173 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg247-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg247-server unloading modules via unload_modules_local on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing unload_modules_local oleg247-server: kmemleak disabled modules unloaded. pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-client: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-client: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2