== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:40:18 (1787618418) start mds service on oleg316-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg316-server' oleg316-server: oleg316-server.virtnet: executing load_modules_local oleg316-server: Loading modules from /home/green/git/lustre-release/lustre oleg316-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg316-server: detected 4 online CPUs by sysfs oleg316-server: MODOPTS_LIBCFS= oleg316-server: Force libcfs to create 2 CPU partitions oleg316-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg316-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg316-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg316-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg316-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0000 oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 194 (time 1787618519.51 total 10.38 last 18.69) - open/close 424 (time 1787618529.51 total 20.38 last 23.00) - open/close 729 (time 1787618540.55 total 31.42 last 27.63) - open/close 962 (time 1787618550.56 total 41.43 last 23.28) - open/close 1197 (time 1787618561.02 total 51.89 last 22.46) - open/close 1368 (time 1787618571.84 total 62.72 last 15.79) - open/close 1522 (time 1787618582.71 total 73.59 last 14.17) - open/close 1688 (time 1787618593.54 total 84.41 last 15.33) - open/close 1922 (time 1787618603.56 total 94.44 last 23.34) - open/close 2149 (time 1787618613.58 total 104.45 last 22.67) - open/close 2393 (time 1787618624.33 total 115.21 last 22.68) - open/close 2639 (time 1787618634.76 total 125.64 last 23.60) - open/close 2884 (time 1787618645.82 total 136.70 last 22.15) - open/close 3165 (time 1787618655.84 total 146.72 last 28.04) - open/close 3386 (time 1787618666.11 total 156.99 last 21.52) - open/close 3628 (time 1787618676.35 total 167.22 last 23.65) - open/close 3890 (time 1787618687.11 total 177.99 last 24.34) - open/close 3943 (time 1787618697.61 total 188.49 last 5.05) - open/close 4152 (time 1787618707.90 total 198.78 last 20.30) - open/close 4379 (time 1787618717.92 total 208.79 last 22.67) - open/close 4617 (time 1787618727.93 total 218.81 last 23.77) - open/close 4876 (time 1787618737.95 total 228.83 last 25.85) - open/close 5058 (time 1787618748.48 total 239.36 last 17.29) - open/close 5240 (time 1787618758.87 total 249.74 last 17.52) - open/close 5450 (time 1787618769.61 total 260.49 last 19.55) - open/close 5611 (time 1787618779.79 total 270.66 last 15.82) - open/close 5825 (time 1787618789.81 total 280.69 last 21.35) - open/close 6052 (time 1787618800.77 total 291.64 last 20.73) - open/close 6300 (time 1787618811.01 total 301.89 last 24.21) - open/close 6576 (time 1787618821.49 total 312.37 last 26.33) - open/close 6771 (time 1787618832.44 total 323.31 last 17.82) - open/close 6902 (time 1787618844.55 total 335.43 last 10.81) - open/close 7058 (time 1787618856.72 total 347.59 last 12.82) - open/close 7210 (time 1787618867.47 total 358.34 last 14.14) - open/close 7458 (time 1787618877.47 total 368.35 last 24.78) - open/close 7705 (time 1787618888.25 total 379.13 last 22.91) - open/close 7892 (time 1787618899.08 total 389.96 last 17.27) - open/close 8072 (time 1787618909.09 total 399.96 last 17.99) - open/close 8332 (time 1787618919.10 total 409.98 last 25.97) - open/close 8515 (time 1787618929.62 total 420.50 last 17.39) - open/close 8715 (time 1787618939.63 total 430.51 last 19.98) - open/close 8923 (time 1787618949.69 total 440.57 last 20.68) - open/close 9037 (time 1787618960.25 total 451.12 last 10.80) - open/close 9226 (time 1787618971.60 total 462.47 last 16.65) - open/close 9361 (time 1787618981.64 total 472.52 last 13.44) - open/close 9541 (time 1787618991.75 total 482.62 last 17.81) - open/close 9662 (time 1787619002.63 total 493.51 last 11.12) - open/close 9858 (time 1787619012.63 total 503.51 last 19.60) - open/close 10000 (time 1787619022.62 total 513.49 last 14.22) - open/close 10221 (time 1787619033.67 total 524.54 last 20.00) - open/close 10456 (time 1787619044.32 total 535.20 last 22.06) - open/close 10618 (time 1787619054.67 total 545.54 last 15.66) - open/close 10920 (time 1787619064.68 total 555.56 last 30.15) - open/close 11064 (time 1787619075.30 total 566.17 last 13.57) - open/close 11169 (time 1787619085.51 total 576.39 last 10.28) - open/close 11304 (time 1787619095.82 total 586.70 last 13.10) - open/close 11418 (time 1787619105.83 total 596.70 last 11.39) - open/close 11491 (time 1787619115.83 total 606.71 last 7.30) - open/close 11668 (time 1787619125.84 total 616.71 last 17.69) - open/close 11943 (time 1787619135.85 total 626.73 last 27.47) - open/close 12172 (time 1787619146.33 total 637.21 last 21.85) - open/close 12414 (time 1787619156.53 total 647.41 last 23.72) - open/close 12691 (time 1787619166.72 total 657.59 last 27.20) - open/close 12813 (time 1787619178.32 total 669.20 last 10.51) - open/close 12937 (time 1787619188.36 total 679.23 last 12.35) - open/close 13130 (time 1787619198.86 total 689.73 last 18.39) - open/close 13334 (time 1787619209.49 total 700.36 last 19.19) - open/close 13566 (time 1787619220.29 total 711.17 last 21.47) - open/close 13648 (time 1787619230.36 total 721.24 last 8.14) - open/close 13916 (time 1787619240.37 total 731.25 last 26.76) - open/close 14089 (time 1787619250.69 total 741.57 last 16.77) - open/close 14328 (time 1787619261.26 total 752.13 last 22.62) - open/close 14560 (time 1787619271.94 total 762.82 last 21.71) - open/close 14685 (time 1787619282.69 total 773.57 last 11.62) - open/close 14842 (time 1787619292.70 total 783.58 last 15.69) - open/close 15079 (time 1787619303.42 total 794.30 last 22.10) - open/close 15308 (time 1787619313.43 total 804.30 last 22.89) - open/close 15568 (time 1787619324.22 total 815.10 last 24.08) - open/close 15774 (time 1787619334.24 total 825.12 last 20.56) - open/close 16010 (time 1787619344.51 total 835.38 last 22.99) - open/close 16139 (time 1787619354.54 total 845.42 last 12.86) - open/close 16342 (time 1787619364.55 total 855.42 last 20.29) - open/close 16571 (time 1787619375.17 total 866.04 last 21.56) - open/close 16845 (time 1787619385.18 total 876.05 last 27.38) - open/close 17096 (time 1787619395.91 total 886.79 last 23.38) - open/close 17269 (time 1787619406.21 total 897.08 last 16.80) - open/close 17503 (time 1787619416.21 total 907.08 last 23.40) - open/close 17734 (time 1787619427.14 total 918.01 last 21.14) - open/close 17944 (time 1787619437.38 total 928.26 last 20.49) - open/close 18301 (time 1787619447.40 total 938.27 last 35.66) - open/close 19103 (time 1787619457.42 total 948.29 last 80.03) - open/close 19952 (time 1787619467.44 total 958.31 last 84.71) - open/close 20754 (time 1787619477.44 total 968.32 last 80.18) - open/close 21571 (time 1787619487.45 total 978.32 last 81.65) - open/close 22317 (time 1787619497.45 total 988.33 last 74.59) - open/close 23116 (time 1787619507.45 total 998.33 last 79.89) - open/close 23890 (time 1787619517.45 total 1008.33 last 77.39) - open/close 24693 (time 1787619527.47 total 1018.34 last 80.18) - open/close 25459 (time 1787619537.47 total 1028.34 last 76.60) - open/close 26161 (time 1787619547.47 total 1038.35 last 70.19) - open/close 26889 (time 1787619557.48 total 1048.36 last 72.72) - open/close 27655 (time 1787619567.48 total 1058.36 last 76.59) - open/close 28472 (time 1787619577.48 total 1068.36 last 81.69) - open/close 29321 (time 1787619587.48 total 1078.36 last 84.90) - open/close 30000 (time 1787619597.34 total 1088.21 last 68.91) - open/close 30573 (time 1787619607.36 total 1098.24 last 57.15) - open/close 31303 (time 1787619617.37 total 1108.25 last 72.94) - open/close 32066 (time 1787619627.37 total 1118.25 last 76.29) - open/close 32910 (time 1787619637.39 total 1128.26 last 84.28) - open/close 33693 (time 1787619647.39 total 1138.27 last 78.27) - open/close 34290 (time 1787619657.40 total 1148.28 last 59.63) - open/close 34923 (time 1787619667.43 total 1158.30 last 63.15) - open/close 35598 (time 1787619677.43 total 1168.31 last 67.47) - open/close 36338 (time 1787619687.44 total 1178.31 last 73.95) - open/close 37178 (time 1787619697.44 total 1188.32 last 83.96) - open/close 37918 (time 1787619707.45 total 1198.32 last 73.96) - open/close 38592 (time 1787619717.47 total 1208.35 last 67.25) - open/close 39162 (time 1787619727.47 total 1218.35 last 57.00) - open/close 39790 (time 1787619737.48 total 1228.35 last 62.77) - open/close 40000 (time 1787619740.65 total 1231.53 last 66.19) - open/close 40709 (time 1787619750.65 total 1241.53 last 70.88) - open/close 41545 (time 1787619760.66 total 1251.53 last 83.57) - open/close 42363 (time 1787619770.66 total 1261.54 last 81.77) - open/close 43115 (time 1787619780.66 total 1271.54 last 75.16) - open/close 43864 (time 1787619790.67 total 1281.54 last 74.88) - open/close 44665 (time 1787619800.68 total 1291.56 last 80.00) - open/close 45448 (time 1787619810.68 total 1301.56 last 78.26) - open/close 46305 (time 1787619820.69 total 1311.57 last 85.62) - open/close 47024 (time 1787619830.70 total 1321.58 last 71.85) - open/close 47771 (time 1787619840.70 total 1331.58 last 74.69) - open/close 48486 (time 1787619850.72 total 1341.59 last 71.39) - open/close 49117 (time 1787619860.72 total 1351.60 last 63.09) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1358.15 seconds: 36.54 ops/second - unlinked 0 (time 1787619871 ; total 0 ; last 0) - unlinked 10000 (time 1787620020 ; total 149 ; last 149) - unlinked 20000 (time 1787620185 ; total 314 ; last 165) - unlinked 30000 (time 1787620335 ; total 464 ; last 150) - unlinked 40000 (time 1787620499 ; total 628 ; last 164) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 791 seconds: 62.744629 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg316-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg316-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg316-server stop mds service on oleg316-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg316-server stop mds service on oleg316-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg316-server unloading modules via unload_modules_local on: 'oleg316-server' oleg316-server: oleg316-server.virtnet: executing unload_modules_local oleg316-server: modules unloaded. pdsh@oleg316-client: oleg316-server: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-client: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-server: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-client: ssh exited with exit code 2