== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:50:26 (1788447026) start mds service on oleg229-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg229-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg229-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg229-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg229-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 271 (time 1788447136.12 total 10.99 last 24.65) - open/close 454 (time 1788447146.67 total 21.54 last 17.36) - open/close 752 (time 1788447157.75 total 32.62 last 26.89) - open/close 971 (time 1788447168.75 total 43.62 last 19.91) - open/close 1231 (time 1788447181.55 total 56.42 last 20.31) - open/close 1485 (time 1788447192.37 total 67.23 last 23.49) - open/close 1727 (time 1788447202.75 total 77.62 last 23.31) - open/close 1910 (time 1788447213.68 total 88.55 last 16.74) - open/close 2097 (time 1788447223.73 total 98.60 last 18.61) - open/close 2246 (time 1788447233.74 total 108.61 last 14.89) - open/close 2497 (time 1788447244.47 total 119.34 last 23.38) - open/close 2787 (time 1788447255.04 total 129.91 last 27.45) - open/close 3020 (time 1788447265.27 total 140.14 last 22.77) - open/close 3270 (time 1788447275.31 total 150.18 last 24.91) - open/close 3564 (time 1788447285.31 total 160.18 last 29.39) - open/close 3808 (time 1788447296.25 total 171.12 last 22.30) - open/close 4056 (time 1788447306.49 total 181.36 last 24.22) - open/close 4256 (time 1788447316.85 total 191.71 last 19.32) - open/close 4562 (time 1788447327.79 total 202.66 last 27.95) - open/close 4794 (time 1788447337.80 total 212.66 last 23.19) - open/close 4950 (time 1788447348.25 total 223.12 last 14.93) - open/close 5109 (time 1788447358.89 total 233.76 last 14.94) - open/close 5285 (time 1788447369.58 total 244.45 last 16.47) - open/close 5585 (time 1788447380.28 total 255.15 last 28.03) - open/close 5851 (time 1788447390.40 total 265.26 last 26.30) - open/close 6069 (time 1788447400.42 total 275.29 last 21.75) - open/close 6258 (time 1788447411.48 total 286.35 last 17.08) - open/close 6409 (time 1788447424.77 total 299.64 last 11.36) - open/close 6609 (time 1788447435.35 total 310.22 last 18.91) - open/close 6870 (time 1788447445.36 total 320.23 last 26.07) - open/close 7016 (time 1788447455.37 total 330.24 last 14.58) - open/close 7304 (time 1788447465.40 total 340.26 last 28.74) - open/close 7493 (time 1788447476.02 total 350.89 last 17.79) - open/close 7749 (time 1788447486.50 total 361.37 last 24.42) - open/close 7998 (time 1788447496.95 total 371.82 last 23.84) - open/close 8244 (time 1788447506.97 total 381.84 last 24.54) - open/close 8501 (time 1788447517.18 total 392.05 last 25.18) - open/close 8735 (time 1788447527.19 total 402.06 last 23.37) - open/close 8977 (time 1788447538.19 total 413.06 last 22.01) - open/close 9122 (time 1788447548.19 total 423.06 last 14.49) - open/close 9311 (time 1788447559.11 total 433.97 last 17.32) - open/close 9488 (time 1788447569.11 total 443.98 last 17.69) - open/close 9657 (time 1788447579.31 total 454.18 last 16.58) - open/close 9906 (time 1788447589.76 total 464.63 last 23.82) - open/close 10000 (time 1788447595.81 total 470.67 last 15.54) - open/close 10183 (time 1788447605.83 total 480.69 last 18.26) - open/close 10247 (time 1788447615.89 total 490.76 last 6.36) - open/close 10363 (time 1788447627.65 total 502.52 last 9.87) - open/close 10551 (time 1788447637.69 total 512.56 last 18.72) - open/close 10740 (time 1788447648.70 total 523.57 last 17.16) - open/close 10919 (time 1788447659.47 total 534.34 last 16.62) - open/close 11052 (time 1788447669.56 total 544.43 last 13.18) - open/close 11173 (time 1788447680.46 total 555.33 last 11.10) - open/close 11303 (time 1788447690.48 total 565.35 last 12.98) - open/close 11525 (time 1788447701.75 total 576.62 last 19.70) - open/close 11748 (time 1788447711.76 total 586.63 last 22.29) - open/close 11989 (time 1788447722.36 total 597.23 last 22.73) - open/close 12203 (time 1788447732.36 total 607.23 last 21.39) - open/close 12448 (time 1788447742.52 total 617.38 last 24.13) - open/close 12720 (time 1788447753.27 total 628.14 last 25.29) - open/close 12978 (time 1788447763.85 total 638.72 last 24.39) - open/close 13210 (time 1788447774.39 total 649.26 last 22.01) - open/close 13446 (time 1788447784.40 total 659.27 last 23.58) - open/close 13724 (time 1788447794.42 total 669.29 last 27.73) - open/close 14000 (time 1788447805.43 total 680.30 last 25.08) - open/close 14266 (time 1788447815.92 total 690.79 last 25.34) - open/close 14520 (time 1788447826.62 total 701.49 last 23.74) - open/close 14623 (time 1788447836.63 total 711.50 last 10.29) - open/close 14796 (time 1788447847.54 total 722.41 last 15.86) - open/close 14937 (time 1788447857.57 total 732.44 last 14.06) - open/close 15097 (time 1788447867.59 total 742.46 last 15.97) - open/close 15151 (time 1788447878.26 total 753.13 last 5.06) - open/close 15347 (time 1788447888.53 total 763.40 last 19.09) - open/close 15609 (time 1788447898.56 total 773.42 last 26.13) - open/close 15749 (time 1788447910.06 total 784.93 last 12.17) - open/close 15904 (time 1788447920.78 total 795.65 last 14.46) - open/close 16126 (time 1788447930.95 total 805.82 last 21.82) - open/close 16284 (time 1788447941.57 total 816.44 last 14.88) - open/close 16422 (time 1788447951.73 total 826.60 last 13.58) - open/close 16539 (time 1788447962.50 total 837.37 last 10.86) - open/close 16687 (time 1788447972.52 total 847.39 last 14.77) - open/close 16804 (time 1788447983.50 total 858.37 last 10.66) - open/close 16935 (time 1788447994.15 total 869.02 last 12.29) - open/close 17113 (time 1788448004.80 total 879.67 last 16.72) - open/close 17252 (time 1788448014.86 total 889.73 last 13.81) - open/close 17396 (time 1788448025.27 total 900.14 last 13.84) - open/close 17592 (time 1788448037.37 total 912.24 last 16.20) - open/close 17812 (time 1788448047.80 total 922.67 last 21.09) - open/close 18085 (time 1788448058.67 total 933.54 last 25.12) - open/close 18333 (time 1788448069.50 total 944.37 last 22.91) - open/close 18465 (time 1788448079.64 total 954.51 last 13.02) - open/close 18584 (time 1788448090.62 total 965.49 last 10.83) - open/close 18785 (time 1788448100.62 total 975.49 last 20.10) - open/close 18945 (time 1788448110.64 total 985.51 last 15.98) - open/close 19787 (time 1788448120.65 total 995.52 last 84.14) - open/close 20000 (time 1788448123.08 total 997.95 last 87.60) - open/close 20646 (time 1788448133.08 total 1007.95 last 64.58) - open/close 21602 (time 1788448143.08 total 1017.95 last 95.58) - open/close 22463 (time 1788448153.10 total 1027.97 last 85.93) - open/close 23105 (time 1788448163.14 total 1038.00 last 63.99) - open/close 23674 (time 1788448173.14 total 1048.01 last 56.87) - open/close 24417 (time 1788448183.14 total 1058.01 last 74.30) - open/close 25454 (time 1788448193.14 total 1068.01 last 103.69) - open/close 26279 (time 1788448203.15 total 1078.02 last 82.44) - open/close 27353 (time 1788448213.16 total 1088.03 last 107.31) - open/close 28309 (time 1788448223.16 total 1098.03 last 95.58) - open/close 29364 (time 1788448233.17 total 1108.04 last 105.43) - open/close 30000 (time 1788448239.89 total 1114.76 last 94.55) - open/close 31059 (time 1788448249.90 total 1124.77 last 105.83) - open/close 32012 (time 1788448259.92 total 1134.79 last 95.11) - open/close 33028 (time 1788448269.92 total 1144.79 last 101.56) - open/close 34182 (time 1788448279.94 total 1154.80 last 115.27) - open/close 35029 (time 1788448289.94 total 1164.81 last 84.66) - open/close 35875 (time 1788448299.95 total 1174.82 last 84.53) - open/close 36833 (time 1788448309.95 total 1184.82 last 95.76) - open/close 37773 (time 1788448319.96 total 1194.83 last 93.97) - open/close 38689 (time 1788448329.96 total 1204.83 last 91.54) - open/close 39489 (time 1788448339.97 total 1214.84 last 79.93) - open/close 40000 (time 1788448346.21 total 1221.08 last 81.94) - open/close 40886 (time 1788448356.21 total 1231.08 last 88.54) - open/close 41885 (time 1788448366.22 total 1241.09 last 99.88) - open/close 42719 (time 1788448376.23 total 1251.09 last 83.33) - open/close 43652 (time 1788448386.23 total 1261.10 last 93.26) - open/close 44576 (time 1788448396.24 total 1271.11 last 92.33) - open/close 45485 (time 1788448406.24 total 1281.11 last 90.90) - open/close 46421 (time 1788448416.24 total 1291.11 last 93.58) - open/close 47204 (time 1788448426.24 total 1301.11 last 78.29) - open/close 48059 (time 1788448436.24 total 1311.11 last 85.46) - open/close 48765 (time 1788448446.25 total 1321.12 last 70.54) - open/close 49329 (time 1788448456.28 total 1331.15 last 56.23) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1337.03 seconds: 37.12 ops/second - unlinked 0 (time 1788448467 ; total 0 ; last 0) - unlinked 10000 (time 1788448578 ; total 111 ; last 111) - unlinked 20000 (time 1788448681 ; total 214 ; last 103) - unlinked 30000 (time 1788448742 ; total 275 ; last 61) - unlinked 40000 (time 1788448808 ; total 341 ; last 66) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 403 seconds: 123.153847 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server unloading modules via unload_modules_local on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing unload_modules_local oleg229-server: modules unloaded. pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2