== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:44:17 (1782488657) start mds service on oleg454-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg454-server' oleg454-server: oleg454-server.virtnet: executing load_modules_local oleg454-server: Loading modules from /home/green/git/lustre-release/lustre oleg454-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg454-server: detected 4 online CPUs by sysfs oleg454-server: MODOPTS_LIBCFS= oleg454-server: Force libcfs to create 2 CPU partitions oleg454-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg454-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg454-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg454-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg454-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-OST0000 oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg454-client.virtnet: -o user_xattr,flock 192.168.204.154@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 516 (time 1782488714.09 total 10.06 last 51.27) - open/close 1177 (time 1782488724.45 total 20.43 last 63.79) - open/close 1825 (time 1782488735.39 total 31.37 last 59.21) - open/close 2230 (time 1782488746.36 total 42.34 last 36.93) - open/close 2397 (time 1782488756.46 total 52.44 last 16.54) - open/close 2593 (time 1782488766.46 total 62.44 last 19.59) - open/close 2806 (time 1782488776.49 total 72.46 last 21.25) - open/close 2999 (time 1782488786.50 total 82.48 last 19.28) - open/close 3174 (time 1782488796.78 total 92.76 last 17.02) - open/close 3424 (time 1782488806.82 total 102.80 last 24.89) - open/close 3660 (time 1782488817.38 total 113.36 last 22.35) - open/close 3918 (time 1782488827.39 total 123.37 last 25.77) - open/close 4111 (time 1782488837.40 total 133.38 last 19.29) - open/close 4256 (time 1782488847.40 total 143.38 last 14.49) - open/close 4442 (time 1782488857.72 total 153.70 last 18.04) - open/close 4987 (time 1782488868.46 total 164.44 last 50.74) - open/close 5100 (time 1782488879.08 total 175.06 last 10.63) - open/close 5240 (time 1782488889.32 total 185.30 last 13.68) - open/close 5363 (time 1782488899.68 total 195.66 last 11.87) - open/close 5497 (time 1782488910.38 total 206.36 last 12.52) - open/close 5764 (time 1782488921.07 total 217.04 last 24.99) - open/close 6000 (time 1782488931.89 total 227.87 last 21.81) - open/close 6231 (time 1782488941.93 total 237.90 last 23.01) - open/close 6464 (time 1782488952.49 total 248.46 last 22.06) - open/close 6597 (time 1782488963.11 total 259.09 last 12.51) - open/close 6796 (time 1782488973.48 total 269.46 last 19.20) - open/close 6914 (time 1782488984.18 total 280.15 last 11.03) - open/close 6992 (time 1782488994.35 total 290.33 last 7.66) - open/close 7181 (time 1782489004.52 total 300.50 last 18.59) - open/close 7378 (time 1782489014.89 total 310.87 last 19.00) - open/close 7506 (time 1782489025.57 total 321.55 last 11.98) - open/close 7738 (time 1782489036.01 total 331.99 last 22.23) - open/close 8071 (time 1782489046.32 total 342.30 last 32.30) - open/close 8247 (time 1782489056.75 total 352.73 last 16.87) - open/close 8350 (time 1782489067.11 total 363.09 last 9.94) - open/close 8663 (time 1782489077.99 total 373.97 last 28.76) - open/close 8908 (time 1782489089.06 total 385.04 last 22.14) - open/close 9109 (time 1782489099.09 total 395.07 last 20.04) - open/close 9204 (time 1782489109.11 total 405.08 last 9.48) - open/close 9348 (time 1782489119.42 total 415.40 last 13.96) - open/close 9510 (time 1782489129.44 total 425.42 last 16.17) - open/close 9630 (time 1782489139.47 total 435.45 last 11.96) - open/close 9745 (time 1782489150.21 total 446.19 last 10.71) - open/close 9880 (time 1782489160.96 total 456.94 last 12.56) - open/close 10000 (time 1782489168.59 total 464.57 last 15.74) - open/close 10235 (time 1782489181.61 total 477.59 last 18.05) - open/close 10371 (time 1782489191.63 total 487.61 last 13.57) - open/close 10480 (time 1782489201.65 total 497.62 last 10.88) - open/close 10536 (time 1782489212.10 total 508.08 last 5.36) - open/close 10696 (time 1782489223.09 total 519.07 last 14.55) - open/close 10879 (time 1782489233.10 total 529.08 last 18.29) - open/close 11062 (time 1782489243.12 total 539.09 last 18.27) - open/close 11257 (time 1782489254.06 total 550.04 last 17.82) - open/close 11501 (time 1782489264.07 total 560.05 last 24.38) - open/close 11741 (time 1782489274.07 total 570.05 last 23.99) - open/close 11924 (time 1782489284.09 total 580.07 last 18.27) - open/close 12067 (time 1782489294.45 total 590.43 last 13.80) - open/close 12348 (time 1782489304.46 total 600.44 last 28.08) - open/close 12507 (time 1782489314.94 total 610.92 last 15.17) - open/close 12760 (time 1782489324.96 total 620.94 last 25.26) - open/close 12999 (time 1782489335.88 total 631.86 last 21.89) - open/close 13127 (time 1782489345.89 total 641.87 last 12.78) - open/close 13234 (time 1782489356.67 total 652.64 last 9.93) - open/close 13348 (time 1782489367.83 total 663.80 last 10.21) - open/close 13482 (time 1782489378.01 total 673.98 last 13.16) - open/close 13676 (time 1782489388.62 total 684.59 last 18.28) - open/close 13768 (time 1782489399.09 total 695.07 last 8.79) - open/close 14025 (time 1782489409.10 total 705.07 last 25.68) - open/close 14045 (time 1782489419.95 total 715.93 last 1.84) - open/close 14649 (time 1782489429.96 total 725.94 last 60.35) - open/close 15317 (time 1782489439.97 total 735.95 last 66.72) - open/close 15975 (time 1782489449.98 total 745.96 last 65.75) - open/close 16665 (time 1782489459.99 total 755.97 last 68.96) - open/close 17351 (time 1782489469.99 total 765.97 last 68.55) - open/close 18084 (time 1782489480.00 total 775.98 last 73.28) - open/close 18752 (time 1782489490.00 total 785.98 last 66.77) - open/close 19334 (time 1782489500.02 total 796.00 last 58.09) - open/close 19930 (time 1782489510.02 total 806.00 last 59.58) - open/close 20589 (time 1782489520.04 total 816.02 last 65.77) - open/close 21176 (time 1782489530.05 total 826.03 last 58.67) - open/close 21819 (time 1782489540.05 total 836.03 last 64.30) - open/close 22472 (time 1782489550.06 total 846.04 last 65.25) - open/close 23167 (time 1782489560.07 total 856.04 last 69.45) - open/close 23794 (time 1782489570.07 total 866.05 last 62.69) - open/close 24460 (time 1782489580.07 total 876.05 last 66.55) - open/close 25108 (time 1782489590.08 total 886.06 last 64.73) - open/close 25793 (time 1782489600.09 total 896.06 last 68.49) - open/close 26438 (time 1782489610.10 total 906.07 last 64.44) - open/close 27032 (time 1782489620.11 total 916.09 last 59.30) - open/close 27609 (time 1782489630.11 total 926.09 last 57.68) - open/close 28155 (time 1782489640.12 total 936.10 last 54.55) - open/close 28732 (time 1782489650.15 total 946.13 last 57.53) - open/close 29380 (time 1782489660.16 total 956.14 last 64.78) - open/close 30000 (time 1782489670.00 total 965.98 last 63.01) - open/close 30664 (time 1782489680.01 total 975.99 last 66.30) - open/close 31238 (time 1782489690.03 total 986.01 last 57.29) - open/close 31708 (time 1782489700.04 total 996.02 last 46.95) - open/close 32271 (time 1782489710.04 total 1006.02 last 56.29) - open/close 32877 (time 1782489720.05 total 1016.03 last 60.55) - open/close 33344 (time 1782489730.06 total 1026.03 last 46.68) - open/close 33844 (time 1782489740.06 total 1036.04 last 49.96) - open/close 34566 (time 1782489750.08 total 1046.05 last 72.11) - open/close 35096 (time 1782489760.09 total 1056.07 last 52.90) - open/close 35602 (time 1782489770.11 total 1066.09 last 50.50) - open/close 36091 (time 1782489780.15 total 1076.12 last 48.74) - open/close 36528 (time 1782489790.17 total 1086.15 last 43.59) - open/close 36932 (time 1782489800.20 total 1096.18 last 40.27) - open/close 37404 (time 1782489810.21 total 1106.19 last 47.16) - open/close 37848 (time 1782489820.22 total 1116.20 last 44.37) - open/close 38376 (time 1782489830.24 total 1126.22 last 52.69) - open/close 38967 (time 1782489840.25 total 1136.22 last 59.06) - open/close 39494 (time 1782489850.26 total 1146.23 last 52.65) - open/close 40000 (time 1782489859.19 total 1155.17 last 56.63) - open/close 40643 (time 1782489869.20 total 1165.17 last 64.27) - open/close 41349 (time 1782489879.20 total 1175.18 last 70.54) - open/close 42047 (time 1782489889.21 total 1185.19 last 69.77) - open/close 42640 (time 1782489899.21 total 1195.19 last 59.29) - open/close 43185 (time 1782489909.23 total 1205.21 last 54.41) - open/close 43700 (time 1782489919.24 total 1215.22 last 51.44) - open/close 44235 (time 1782489929.25 total 1225.22 last 53.46) - open/close 44745 (time 1782489939.27 total 1235.25 last 50.87) - open/close 45362 (time 1782489949.27 total 1245.25 last 61.70) - open/close 46006 (time 1782489959.28 total 1255.26 last 64.34) - open/close 46694 (time 1782489969.29 total 1265.27 last 68.76) - open/close 47347 (time 1782489979.30 total 1275.27 last 65.25) - open/close 47999 (time 1782489989.30 total 1285.27 last 65.19) - open/close 48701 (time 1782489999.30 total 1295.28 last 70.18) - open/close 49365 (time 1782490009.30 total 1305.28 last 66.38) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1309.32 seconds: 37.91 ops/second - unlinked 0 (time 1782490017 ; total 0 ; last 0) - unlinked 10000 (time 1782490178 ; total 161 ; last 161) - unlinked 20000 (time 1782490355 ; total 338 ; last 177) - unlinked 30000 (time 1782490553 ; total 536 ; last 198) - unlinked 40000 (time 1782490724 ; total 707 ; last 171) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 874 seconds: 56.786041 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg454-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg454-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg454-server stop mds service on oleg454-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg454-server stop mds service on oleg454-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg454-server unloading modules via unload_modules_local on: 'oleg454-server' oleg454-server: oleg454-server.virtnet: executing unload_modules_local oleg454-server: modules unloaded. pdsh@oleg454-client: oleg454-server: ssh exited with exit code 2 pdsh@oleg454-client: oleg454-client: ssh exited with exit code 2 pdsh@oleg454-client: oleg454-server: ssh exited with exit code 2 pdsh@oleg454-client: oleg454-client: ssh exited with exit code 2