== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:04:23 (1787371463) start mds service on oleg111-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions oleg111-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg111-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg111-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg111-server: oleg111-server.virtnet: executing set_default_debug -1 all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg111-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg111-server: oleg111-server.virtnet: executing set_default_debug -1 all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg111-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg111-server: oleg111-server.virtnet: executing set_default_debug -1 all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-OST0000 oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 225 (time 1787371598.24 total 15.77 last 14.26) - open/close 326 (time 1787371608.81 total 26.34 last 9.56) - open/close 572 (time 1787371619.87 total 37.40 last 22.22) - open/close 776 (time 1787371630.57 total 48.10 last 19.08) - open/close 987 (time 1787371641.31 total 58.84 last 19.65) - open/close 1085 (time 1787371651.75 total 69.28 last 9.39) - open/close 1222 (time 1787371662.59 total 80.12 last 12.64) - open/close 1485 (time 1787371672.60 total 90.13 last 26.28) - open/close 1734 (time 1787371682.79 total 100.32 last 24.42) - open/close 1960 (time 1787371692.79 total 110.32 last 22.60) - open/close 2209 (time 1787371703.02 total 120.55 last 24.34) - open/close 2355 (time 1787371713.08 total 130.61 last 14.52) - open/close 2625 (time 1787371723.08 total 140.61 last 26.99) - open/close 2928 (time 1787371733.09 total 150.62 last 30.28) - open/close 3244 (time 1787371744.03 total 161.56 last 28.88) - open/close 3399 (time 1787371754.08 total 171.61 last 15.43) - open/close 3662 (time 1787371764.65 total 182.18 last 24.88) - open/close 3854 (time 1787371774.67 total 192.20 last 19.16) - open/close 3982 (time 1787371785.45 total 202.98 last 11.87) - open/close 4225 (time 1787371795.52 total 213.05 last 24.14) - open/close 4475 (time 1787371805.53 total 223.06 last 24.98) - open/close 4715 (time 1787371815.53 total 233.06 last 23.99) - open/close 4907 (time 1787371825.56 total 243.09 last 19.14) - open/close 5172 (time 1787371836.41 total 253.94 last 24.43) - open/close 5409 (time 1787371846.41 total 263.94 last 23.69) - open/close 5711 (time 1787371857.32 total 274.85 last 27.68) - open/close 5886 (time 1787371867.62 total 285.15 last 17.00) - open/close 6085 (time 1787371877.80 total 295.33 last 19.54) - open/close 6255 (time 1787371887.81 total 305.34 last 16.99) - open/close 6410 (time 1787371898.52 total 316.05 last 14.47) - open/close 6660 (time 1787371908.80 total 326.33 last 24.32) - open/close 6866 (time 1787371918.80 total 336.33 last 20.60) - open/close 7108 (time 1787371928.87 total 346.40 last 24.02) - open/close 7273 (time 1787371938.91 total 356.44 last 16.43) - open/close 7447 (time 1787371948.92 total 366.45 last 17.39) - open/close 7652 (time 1787371959.73 total 377.26 last 18.97) - open/close 7847 (time 1787371970.55 total 388.08 last 18.02) - open/close 8078 (time 1787371980.91 total 398.44 last 22.30) - open/close 8211 (time 1787371990.91 total 408.44 last 13.29) - open/close 8350 (time 1787372002.55 total 420.08 last 11.94) - open/close 8456 (time 1787372013.37 total 430.90 last 9.79) - open/close 8659 (time 1787372023.41 total 440.94 last 20.22) - open/close 8869 (time 1787372034.29 total 451.82 last 19.31) - open/close 9198 (time 1787372044.78 total 462.31 last 31.35) - open/close 9450 (time 1787372055.29 total 472.82 last 23.97) - open/close 9668 (time 1787372065.30 total 482.83 last 21.78) - open/close 9942 (time 1787372076.32 total 493.85 last 24.88) - open/close 10207 (time 1787372089.27 total 506.80 last 20.47) - open/close 10393 (time 1787372099.28 total 516.81 last 18.57) - open/close 10655 (time 1787372109.31 total 526.84 last 26.13) - open/close 10896 (time 1787372121.86 total 539.39 last 19.20) - open/close 11142 (time 1787372131.88 total 549.41 last 24.57) - open/close 11366 (time 1787372142.82 total 560.35 last 20.46) - open/close 11645 (time 1787372153.74 total 571.27 last 25.56) - open/close 11937 (time 1787372163.75 total 581.28 last 29.18) - open/close 12202 (time 1787372173.93 total 591.46 last 26.02) - open/close 12480 (time 1787372184.49 total 602.02 last 26.32) - open/close 12741 (time 1787372195.33 total 612.86 last 24.08) - open/close 13004 (time 1787372205.73 total 623.26 last 25.27) - open/close 13137 (time 1787372216.65 total 634.18 last 12.18) - open/close 13401 (time 1787372227.57 total 645.10 last 24.19) - open/close 13613 (time 1787372238.00 total 655.53 last 20.33) - open/close 13848 (time 1787372248.00 total 665.53 last 23.49) - open/close 14124 (time 1787372258.10 total 675.63 last 27.32)