== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:24:40 (1788575080) start mds service on oleg156-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg156-server' oleg156-server: oleg156-server.virtnet: executing load_modules_local oleg156-server: Loading modules from /home/green/git/lustre-release/lustre oleg156-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg156-server: detected 4 online CPUs by sysfs oleg156-server: MODOPTS_LIBCFS= oleg156-server: Force libcfs to create 2 CPU partitions oleg156-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg156-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg156-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg156-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg156-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Started lustre-OST0000 oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg156-client.virtnet: -o user_xattr,flock 192.168.201.156@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 116 (time 1788575202.87 total 10.74 last 10.80) - open/close 350 (time 1788575213.37 total 21.23 last 22.30) - open/close 549 (time 1788575225.78 total 33.65 last 16.03) - open/close 710 (time 1788575236.75 total 44.61 last 14.68) - open/close 800 (time 1788575246.77 total 54.63 last 8.98) - open/close 959 (time 1788575257.21 total 65.07 last 15.24) - open/close 1142 (time 1788575268.02 total 75.89 last 16.92) - open/close 1320 (time 1788575278.53 total 86.39 last 16.94) - open/close 1426 (time 1788575290.88 total 98.74 last 8.58) - open/close 1574 (time 1788575300.89 total 108.75 last 14.79) - open/close 1867 (time 1788575310.91 total 118.77 last 29.24) - open/close 2137 (time 1788575320.91 total 128.77 last 27.00) - open/close 2354 (time 1788575331.97 total 139.84 last 19.60) - open/close 2534 (time 1788575342.00 total 149.86 last 17.96) - open/close 2730 (time 1788575352.02 total 159.88 last 19.55) - open/close 2790 (time 1788575362.89 total 170.75 last 5.52) - open/close 2993 (time 1788575373.62 total 181.49 last 18.91) - open/close 3199 (time 1788575383.64 total 191.51 last 20.56) - open/close 3390 (time 1788575394.16 total 202.03 last 18.16) - open/close 3596 (time 1788575404.17 total 212.03 last 20.59) - open/close 3767 (time 1788575414.17 total 222.03 last 17.10) - open/close 3959 (time 1788575424.28 total 232.14 last 18.99) - open/close 4144 (time 1788575434.28 total 242.15 last 18.49) - open/close 4266 (time 1788575444.29 total 252.15 last 12.20) - open/close 4442 (time 1788575455.04 total 262.90 last 16.37) - open/close 4578 (time 1788575465.48 total 273.34 last 13.02) - open/close 4951 (time 1788575476.03 total 283.90 last 35.34) - open/close 5259 (time 1788575486.10 total 293.96 last 30.60) - open/close 5374 (time 1788575496.14 total 304.00 last 11.46) - open/close 5620 (time 1788575506.14 total 314.00 last 24.60) - open/close 5836 (time 1788575516.49 total 324.35 last 20.86) - open/close 6130 (time 1788575527.29 total 335.15 last 27.23) - open/close 6329 (time 1788575537.30 total 345.16 last 19.87) - open/close 6527 (time 1788575549.30 total 357.17 last 16.49) - open/close 6771 (time 1788575559.75 total 367.61 last 23.37) - open/close 6854 (time 1788575569.77 total 377.63 last 8.28) - open/close 6989 (time 1788575582.85 total 390.71 last 10.32) - open/close 7137 (time 1788575593.16 total 401.03 last 14.35) - open/close 7340 (time 1788575603.17 total 411.03 last 20.28) - open/close 7587 (time 1788575613.39 total 421.26 last 24.16) - open/close 7842 (time 1788575623.93 total 431.79 last 24.20) - open/close 8002 (time 1788575634.42 total 442.29 last 15.25) - open/close 8167 (time 1788575644.60 total 452.46 last 16.21) - open/close 8378 (time 1788575654.61 total 462.47 last 21.08) - open/close 8611 (time 1788575665.46 total 473.33 last 21.46) - open/close 8871 (time 1788575675.51 total 483.38 last 25.88) - open/close 9056 (time 1788575686.39 total 494.25 last 17.01) - open/close 9495 (time 1788575696.40 total 504.26 last 43.87) - open/close 9796 (time 1788575706.42 total 514.28 last 30.04) - open/close 10000 (time 1788575712.51 total 520.38 last 33.45) - open/close 10261 (time 1788575722.54 total 530.41 last 26.03) - open/close 10420 (time 1788575733.76 total 541.63 last 14.17) - open/close 10646 (time 1788575743.88 total 551.74 last 22.34) - open/close 10840 (time 1788575753.88 total 561.74 last 19.40) - open/close 11036 (time 1788575764.51 total 572.37 last 18.43) - open/close 11243 (time 1788575774.52 total 582.39 last 20.67) - open/close 11494 (time 1788575784.53 total 592.39 last 25.08) - open/close 11707 (time 1788575795.19 total 603.06 last 19.98) - open/close 11975 (time 1788575805.21 total 613.08 last 26.75) - open/close 12164 (time 1788575815.39 total 623.25 last 18.58) - open/close 12371 (time 1788575826.33 total 634.19 last 18.92) - open/close 12610 (time 1788575836.34 total 644.20 last 23.88) - open/close 12858 (time 1788575846.53 total 654.40 last 24.32) - open/close 13087 (time 1788575857.47 total 665.33 last 20.95) - open/close 13395 (time 1788575867.63 total 675.50 last 30.29) - open/close 13624 (time 1788575877.65 total 685.52 last 22.86) - open/close 13874 (time 1788575887.66 total 695.53 last 24.97) - open/close 14134 (time 1788575898.11 total 705.98 last 24.88) - open/close 14459 (time 1788575908.69 total 716.56 last 30.73) - open/close 14699 (time 1788575919.16 total 727.03 last 22.92) - open/close 14963 (time 1788575929.17 total 737.03 last 26.38) - open/close 15189 (time 1788575939.95 total 747.81 last 20.96) - open/close 15465 (time 1788575950.54 total 758.41 last 26.06) - open/close 15683 (time 1788575963.57 total 771.43 last 16.74) - open/close 15910 (time 1788575973.59 total 781.45 last 22.65) - open/close 16093 (time 1788575983.61 total 791.47 last 18.27) - open/close 16322 (time 1788575993.98 total 801.84 last 22.08) - open/close 16593 (time 1788576005.02 total 812.88 last 24.56) - open/close 16885 (time 1788576015.98 total 823.85 last 26.63) - open/close 17178 (time 1788576027.91 total 835.77 last 24.56) - open/close 17430 (time 1788576037.94 total 845.80 last 25.13) - open/close 17659 (time 1788576053.11 total 860.98 last 15.09) - open/close 17877 (time 1788576063.54 total 871.41 last 20.91) - open/close 18017 (time 1788576073.55 total 881.41 last 13.99) - open/close 18286 (time 1788576084.59 total 892.46 last 24.36) - open/close 18549 (time 1788576096.27 total 904.13 last 22.53) - open/close 18765 (time 1788576107.09 total 914.95 last 19.95) - open/close 18930 (time 1788576117.26 total 925.12 last 16.23) - open/close 19196 (time 1788576129.05 total 936.91 last 22.56) - open/close 19470 (time 1788576139.05 total 946.92 last 27.38) - open/close 19696 (time 1788576149.07 total 956.93 last 22.57) - open/close 19848 (time 1788576159.08 total 966.94 last 15.18) - open/close 20000 (time 1788576167.96 total 975.83 last 17.11) - open/close 20164 (time 1788576178.50 total 986.37 last 15.56) - open/close 20448 (time 1788576189.23 total 997.09 last 26.48) - open/close 20622 (time 1788576199.82 total 1007.68 last 16.44) - open/close 21316 (time 1788576209.82 total 1017.69 last 69.37) - open/close 22315 (time 1788576219.82 total 1027.69 last 99.89) - open/close 23288 (time 1788576229.83 total 1037.69 last 97.26) - open/close 24267 (time 1788576239.83 total 1047.70 last 97.85) - open/close 25307 (time 1788576249.84 total 1057.70 last 103.94) - open/close 26368 (time 1788576259.86 total 1067.72 last 105.87) - open/close 27261 (time 1788576269.86 total 1077.72 last 89.30) - open/close 28049 (time 1788576279.87 total 1087.74 last 78.69) - open/close 28846 (time 1788576289.88 total 1097.74 last 79.66) - open/close 29829 (time 1788576299.88 total 1107.74 last 98.29) - open/close 30799 (time 1788576309.88 total 1117.74 last 97.00) - open/close 31866 (time 1788576319.88 total 1127.74 last 106.69) - open/close 32776 (time 1788576329.89 total 1137.75 last 90.95) - open/close 33664 (time 1788576339.90 total 1147.76 last 88.68) - open/close 34441 (time 1788576349.91 total 1157.77 last 77.64) - open/close 35230 (time 1788576359.91 total 1167.77 last 78.89) - open/close 36167 (time 1788576369.92 total 1177.78 last 93.63) - open/close 37086 (time 1788576379.92 total 1187.78 last 91.89) - open/close 38091 (time 1788576389.92 total 1197.79 last 100.45) - open/close 39102 (time 1788576399.92 total 1207.79 last 101.09) - open/close 40000 (time 1788576409.48 total 1217.34 last 93.98) - open/close 40907 (time 1788576419.49 total 1227.35 last 90.63) - open/close 41756 (time 1788576429.49 total 1237.36 last 84.84) - open/close 42737 (time 1788576439.50 total 1247.36 last 98.06) - open/close 43785 (time 1788576449.51 total 1257.38 last 104.64) - open/close 44625 (time 1788576459.51 total 1267.38 last 84.00) - open/close 45463 (time 1788576469.52 total 1277.38 last 83.76) - open/close 46298 (time 1788576479.53 total 1287.39 last 83.43) - open/close 47118 (time 1788576489.53 total 1297.40 last 81.96) - open/close 48016 (time 1788576499.53 total 1307.40 last 89.78) - open/close 49094 (time 1788576509.55 total 1317.42 last 107.60) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1323.36 seconds: 37.50 ops/second - unlinked 0 (time 1788576519 ; total 0 ; last 0) - unlinked 10000 (time 1788576634 ; total 115 ; last 115) - unlinked 20000 (time 1788576738 ; total 219 ; last 104) - unlinked 30000 (time 1788576804 ; total 285 ; last 66) - unlinked 40000 (time 1788576873 ; total 354 ; last 69) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 409 seconds: 121.347191 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg156-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg156-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg156-server stop mds service on oleg156-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg156-server stop mds service on oleg156-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg156-server unloading modules via unload_modules_local on: 'oleg156-server' oleg156-server: oleg156-server.virtnet: executing unload_modules_local oleg156-server: modules unloaded. pdsh@oleg156-client: oleg156-server: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-client: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-client: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-server: ssh exited with exit code 2