== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:44:50 (1789393490) start mds service on oleg213-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 14 04:14 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: kmemleak disabled oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions oleg213-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg213-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg213-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg213-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg213-server: params: OBD_IOC_LLOG_PRINT failed: No such file or directory pdsh@oleg213-client: oleg213-server: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 123 allow_register=unlimited start mds service on oleg213-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg213-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-OST0000 oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 203 (time 1789393618.12 total 10.99 last 18.46) - open/close 422 (time 1789393629.33 total 22.21 last 19.53) - open/close 680 (time 1789393640.21 total 33.08 last 23.72) - open/close 967 (time 1789393650.22 total 43.10 last 28.65) - open/close 1207 (time 1789393660.72 total 53.60 last 22.86) - open/close 1451 (time 1789393670.75 total 63.62 last 24.34) - open/close 1627 (time 1789393680.77 total 73.64 last 17.56) - open/close 1880 (time 1789393691.46 total 84.34 last 23.66) - open/close 2134 (time 1789393702.28 total 95.16 last 23.48) - open/close 2415 (time 1789393712.85 total 105.73 last 26.59) - open/close 2638 (time 1789393722.87 total 115.74 last 22.26) - open/close 2907 (time 1789393733.57 total 126.44 last 25.14) - open/close 3150 (time 1789393743.68 total 136.56 last 24.03) - open/close 3405 (time 1789393753.69 total 146.56 last 25.48) - open/close 3592 (time 1789393763.79 total 156.67 last 18.51) - open/close 3801 (time 1789393774.62 total 167.49 last 19.31) - open/close 4057 (time 1789393785.10 total 177.98 last 24.41) - open/close 4303 (time 1789393796.96 total 189.84 last 20.75) - open/close 4548 (time 1789393807.89 total 200.77 last 22.41) - open/close 4822 (time 1789393817.90 total 210.78 last 27.38) - open/close 5028 (time 1789393827.90 total 220.78 last 20.59) - open/close 5253 (time 1789393837.93 total 230.81 last 22.43) - open/close 5523 (time 1789393847.94 total 240.82 last 26.98) - open/close 5681 (time 1789393857.95 total 250.82 last 15.79) - open/close 5832 (time 1789393868.33 total 261.21 last 14.54) - open/close 6110 (time 1789393878.33 total 271.21 last 27.79) - open/close 6384 (time 1789393889.16 total 282.04 last 25.31) - open/close 6598 (time 1789393899.23 total 292.11 last 21.25) - open/close 6887 (time 1789393909.25 total 302.13 last 28.85) - open/close 7122 (time 1789393919.94 total 312.81 last 21.99) - open/close 7364 (time 1789393930.95 total 323.83 last 21.97) - open/close 7636 (time 1789393941.72 total 334.60 last 25.26) - open/close 7844 (time 1789393952.60 total 345.47 last 19.12) - open/close 8134 (time 1789393962.88 total 355.76 last 28.20) - open/close 8378 (time 1789393973.67 total 366.55 last 22.61) - open/close 8552 (time 1789393985.89 total 378.77 last 14.24) - open/close 8764 (time 1789393995.90 total 388.77 last 21.19) - open/close 9004 (time 1789394006.60 total 399.48 last 22.41) - open/close 9256 (time 1789394017.57 total 410.44 last 22.99) - open/close 9509 (time 1789394027.91 total 420.79 last 24.45) - open/close 9736 (time 1789394042.21 total 435.09 last 15.87) - open/close 9973 (time 1789394052.51 total 445.39 last 23.01) - open/close 10245 (time 1789394062.52 total 455.40 last 27.19) - open/close 10502 (time 1789394072.84 total 465.71 last 24.90) - open/close 10718 (time 1789394082.85 total 475.72 last 21.58) - open/close 10968 (time 1789394093.64 total 486.52 last 23.15) - open/close 11159 (time 1789394104.45 total 497.32 last 17.68) - open/close 11374 (time 1789394114.45 total 507.33 last 21.49) - open/close 11593 (time 1789394125.38 total 518.26 last 20.03) - open/close 11826 (time 1789394135.39 total 528.27 last 23.28) - open/close 12066 (time 1789394146.43 total 539.30 last 21.75) - open/close 12342 (time 1789394157.21 total 550.09 last 25.60) - open/close 12565 (time 1789394167.67 total 560.55 last 21.31) - open/close 12779 (time 1789394177.73 total 570.60 last 21.29) - open/close 13019 (time 1789394187.92 total 580.79 last 23.55) - open/close 13136 (time 1789394198.02 total 590.90 last 11.58) - open/close 13351 (time 1789394208.03 total 600.91 last 21.48) - open/close 13634 (time 1789394218.84 total 611.72 last 26.18) - open/close 13879 (time 1789394228.88 total 621.76 last 24.41) - open/close 14147 (time 1789394238.89 total 631.77 last 26.77) - open/close 14417 (time 1789394248.90 total 641.78 last 26.97) - open/close 14662 (time 1789394259.54 total 652.41 last 23.03) - open/close 14975 (time 1789394270.29 total 663.16 last 29.12) - open/close 15271 (time 1789394281.10 total 673.97 last 27.38) - open/close 15504 (time 1789394291.80 total 684.68 last 21.77) - open/close 15802 (time 1789394302.85 total 695.72 last 26.98) - open/close 16054 (time 1789394313.48 total 706.36 last 23.70) - open/close 16310 (time 1789394323.49 total 716.37 last 25.57) - open/close 16524 (time 1789394334.48 total 727.35 last 19.49) - open/close 16813 (time 1789394344.91 total 737.78 last 27.70) - open/close 17021 (time 1789394355.01 total 747.89 last 20.58) - open/close 17259 (time 1789394365.31 total 758.19 last 23.11) - open/close 17548 (time 1789394375.32 total 768.19 last 28.89) - open/close 17826 (time 1789394385.61 total 778.48 last 27.01) - open/close 18073 (time 1789394396.03 total 788.91 last 23.70) - open/close 18347 (time 1789394406.05 total 798.92 last 27.36) - open/close 18521 (time 1789394416.22 total 809.09 last 17.10) - open/close 18799 (time 1789394426.89 total 819.76 last 26.05) - open/close 19021 (time 1789394437.64 total 830.52 last 20.64) - open/close 19220 (time 1789394447.66 total 840.54 last 19.86) - open/close 19461 (time 1789394458.01 total 850.89 last 23.29) - open/close 19755 (time 1789394468.02 total 860.90 last 29.38) - open/close 19867 (time 1789394478.02 total 870.90 last 11.20) - open/close 20000 (time 1789394484.13 total 877.01 last 21.78) - open/close 20178 (time 1789394494.37 total 887.25 last 17.38) - open/close 20469 (time 1789394504.37 total 897.25 last 29.10) - open/close 21359 (time 1789394514.37 total 907.25 last 88.99) - open/close 22284 (time 1789394524.38 total 917.25 last 92.48) - open/close 23229 (time 1789394534.40 total 927.27 last 94.31) - open/close 24202 (time 1789394544.40 total 937.27 last 97.30) - open/close 25171 (time 1789394554.40 total 947.27 last 96.90) - open/close 26028 (time 1789394564.40 total 957.28 last 85.65) - open/close 26896 (time 1789394574.41 total 967.29 last 86.72) - open/close 27832 (time 1789394584.41 total 977.29 last 93.56) - open/close 28676 (time 1789394594.42 total 987.29 last 84.38) - open/close 29569 (time 1789394604.42 total 997.30 last 89.24) - open/close 30000 (time 1789394610.45 total 1003.33 last 71.51) - open/close 30880 (time 1789394620.46 total 1013.34 last 87.90) - open/close 31689 (time 1789394630.48 total 1023.35 last 80.78) - open/close 32439 (time 1789394640.48 total 1033.36 last 74.95) - open/close 33269 (time 1789394650.49 total 1043.36 last 82.98) - open/close 34071 (time 1789394660.49 total 1053.37 last 80.14) - open/close 34847 (time 1789394670.50 total 1063.38 last 77.56) - open/close 35767 (time 1789394680.50 total 1073.38 last 91.98) - open/close 36647 (time 1789394690.50 total 1083.38 last 87.98) - open/close 37563 (time 1789394700.51 total 1093.39 last 91.55) - open/close 38377 (time 1789394710.51 total 1103.39 last 81.38) - open/close 39036 (time 1789394720.52 total 1113.40 last 65.86) - open/close 39661 (time 1789394730.53 total 1123.40 last 62.46) - open/close 40000 (time 1789394734.65 total 1127.52 last 82.23) - open/close 40849 (time 1789394744.66 total 1137.53 last 84.81) - open/close 41506 (time 1789394754.69 total 1147.57 last 65.48) - open/close 42171 (time 1789394764.70 total 1157.58 last 66.45) - open/close 42969 (time 1789394774.71 total 1167.59 last 79.70) - open/close 43816 (time 1789394784.72 total 1177.59 last 84.66) - open/close 44625 (time 1789394794.72 total 1187.59 last 80.89) - open/close 45164 (time 1789394804.72 total 1197.60 last 53.87) - open/close 45828 (time 1789394814.73 total 1207.60 last 66.37) - open/close 46538 (time 1789394824.74 total 1217.61 last 70.95) - open/close 47280 (time 1789394834.74 total 1227.62 last 74.15) - open/close 48078 (time 1789394844.75 total 1237.63 last 79.73) - open/close 48817 (time 1789394854.75 total 1247.63 last 73.90) - open/close 49510 (time 1789394864.77 total 1257.64 last 69.21) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1259.75 seconds: 39.40 ops/second - unlinked 0 (time 1789394872 ; total 0 ; last 0) - unlinked 10000 (time 1789395020 ; total 148 ; last 148) - unlinked 20000 (time 1789395162 ; total 290 ; last 142) - unlinked 30000 (time 1789395299 ; total 427 ; last 137) - unlinked 40000 (time 1789395439 ; total 567 ; last 140) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 705 seconds: 70.398582 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg213-client.virtnet /mnt/lustre (opts:) oleg213-server: params: OBD_IOC_LLOG_PRINT failed: No such file or directory pdsh@oleg213-client: oleg213-server: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 123 allow_register=unlimited stop ost1 service on oleg213-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg213-server stop mds service on oleg213-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg213-server stop mds service on oleg213-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg213-server unloading modules via unload_modules_local on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing unload_modules_local oleg213-server: kmemleak disabled modules unloaded. pdsh@oleg213-client: oleg213-client: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-client: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 2