== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:17:45 (1787120265) start mds service on oleg255-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg255-server: detected 4 online CPUs by sysfs oleg255-server: MODOPTS_LIBCFS= oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg255-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg255-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg255-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg255-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 238 (time 1787120374.01 total 10.48 last 22.72) - open/close 489 (time 1787120384.01 total 20.48 last 25.09) - open/close 728 (time 1787120394.22 total 30.68 last 23.43) - open/close 914 (time 1787120404.45 total 40.91 last 18.18) - open/close 1154 (time 1787120415.39 total 51.86 last 21.92) - open/close 1386 (time 1787120425.95 total 62.42 last 21.97) - open/close 1631 (time 1787120436.33 total 72.79 last 23.61) - open/close 1872 (time 1787120446.51 total 82.97 last 23.68) - open/close 2113 (time 1787120456.51 total 92.98 last 24.09) - open/close 2233 (time 1787120466.52 total 102.99 last 11.99) - open/close 2395 (time 1787120476.54 total 113.01 last 16.17) - open/close 2661 (time 1787120487.14 total 123.61 last 25.09) - open/close 2885 (time 1787120497.17 total 133.64 last 22.34) - open/close 3102 (time 1787120508.14 total 144.60 last 19.79) - open/close 3333 (time 1787120518.15 total 154.61 last 23.08) - open/close 3589 (time 1787120529.08 total 165.55 last 23.40) - open/close 3856 (time 1787120539.08 total 175.55 last 26.70) - open/close 4067 (time 1787120549.12 total 185.59 last 21.02) - open/close 4403 (time 1787120559.13 total 195.60 last 33.57) - open/close 4647 (time 1787120569.14 total 205.60 last 24.38) - open/close 4799 (time 1787120579.15 total 215.61 last 15.19) - open/close 5094 (time 1787120589.36 total 225.82 last 28.90) - open/close 5350 (time 1787120599.93 total 236.40 last 24.21) - open/close 5583 (time 1787120609.96 total 246.43 last 23.23) - open/close 5843 (time 1787120620.64 total 257.11 last 24.35) - open/close 6057 (time 1787120631.48 total 267.94 last 19.75) - open/close 6298 (time 1787120642.04 total 278.50 last 22.82) - open/close 6561 (time 1787120652.87 total 289.33 last 24.28) - open/close 6843 (time 1787120662.87 total 299.34 last 28.19) - open/close 7046 (time 1787120672.87 total 309.34 last 20.30) - open/close 7353 (time 1787120683.44 total 319.91 last 29.05) - open/close 7524 (time 1787120693.76 total 330.22 last 16.57) - open/close 7744 (time 1787120703.77 total 340.23 last 21.98) - open/close 7952 (time 1787120713.77 total 350.24 last 20.79) - open/close 8186 (time 1787120723.79 total 360.26 last 23.36) - open/close 8433 (time 1787120734.21 total 370.68 last 23.70) - open/close 8630 (time 1787120744.99 total 381.46 last 18.28) - open/close 8906 (time 1787120755.84 total 392.31 last 25.44) - open/close 9224 (time 1787120766.70 total 403.17 last 29.27) - open/close 9440 (time 1787120776.71 total 413.17 last 21.59) - open/close 9592 (time 1787120786.72 total 423.18 last 15.19) - open/close 9819 (time 1787120796.98 total 433.45 last 22.12) - open/close 10000 (time 1787120805.32 total 441.78 last 21.71) - open/close 10219 (time 1787120815.41 total 451.87 last 21.71) - open/close 10408 (time 1787120826.09 total 462.55 last 17.69) - open/close 10607 (time 1787120836.74 total 473.20 last 18.69) - open/close 10875 (time 1787120847.08 total 483.54 last 25.92) - open/close 11110 (time 1787120857.08 total 493.55 last 23.50) - open/close 11323 (time 1787120867.09 total 503.55 last 21.28) - open/close 11541 (time 1787120877.10 total 513.56 last 21.78) - open/close 11748 (time 1787120887.14 total 523.61 last 20.61) - open/close 12039 (time 1787120897.17 total 533.63 last 29.03) - open/close 12280 (time 1787120907.18 total 543.64 last 24.07) - open/close 12513 (time 1787120917.30 total 553.77 last 23.01) - open/close 12711 (time 1787120928.14 total 564.60 last 18.28) - open/close 12929 (time 1787120938.14 total 574.60 last 21.79) - open/close 13167 (time 1787120948.68 total 585.14 last 22.58) - open/close 13378 (time 1787120959.31 total 595.77 last 19.85) - open/close 13630 (time 1787120969.33 total 605.79 last 25.15) - open/close 13821 (time 1787120979.54 total 616.01 last 18.69) - open/close 13954 (time 1787120989.65 total 626.12 last 13.16) - open/close 14243 (time 1787121000.00 total 636.46 last 27.94) - open/close 14395 (time 1787121010.34 total 646.81 last 14.69) - open/close 14621 (time 1787121020.84 total 657.30 last 21.54) - open/close 14848 (time 1787121030.90 total 667.37 last 22.55) - open/close 15020 (time 1787121040.96 total 677.43 last 17.10) - open/close 15157 (time 1787121050.97 total 687.43 last 13.69) - open/close 15319 (time 1787121060.97 total 697.44 last 16.19) - open/close 15524 (time 1787121070.98 total 707.44 last 20.50) - open/close 15722 (time 1787121081.52 total 717.98 last 18.78) - open/close 15951 (time 1787121091.82 total 728.28 last 22.23) - open/close 16203 (time 1787121101.83 total 738.29 last 25.17) - open/close 16412 (time 1787121111.84 total 748.30 last 20.88) - open/close 16558 (time 1787121122.50 total 758.97 last 13.69) - open/close 16723 (time 1787121132.74 total 769.20 last 16.12) - open/close 16970 (time 1787121148.09 total 784.55 last 16.09) - open/close 17191 (time 1787121158.10 total 794.56 last 22.07) - open/close 17392 (time 1787121168.10 total 804.57 last 20.10) - open/close 17571 (time 1787121178.42 total 814.89 last 17.34) - open/close 17673 (time 1787121189.24 total 825.70 last 9.43) - open/close 17798 (time 1787121199.41 total 835.87 last 12.29) - open/close 17953 (time 1787121210.18 total 846.64 last 14.39) - open/close 18210 (time 1787121220.18 total 856.65 last 25.69) - open/close 18297 (time 1787121230.32 total 866.78 last 8.59) - open/close 18550 (time 1787121241.02 total 877.48 last 23.64) - open/close 18699 (time 1787121251.06 total 887.52 last 14.84) - open/close 18780 (time 1787121261.20 total 897.67 last 7.99) - open/close 19010 (time 1787121272.50 total 908.97 last 20.35) - open/close 19210 (time 1787121282.52 total 918.98 last 19.97) - open/close 19493 (time 1787121292.53 total 929.00 last 28.25) - open/close 20000 (time 1787121298.15 total 934.61 last 90.29) - open/close 20839 (time 1787121308.15 total 944.62 last 83.89) - open/close 21634 (time 1787121318.16 total 954.63 last 79.40) - open/close 22364 (time 1787121328.17 total 964.63 last 72.99) - open/close 22963 (time 1787121338.17 total 974.63 last 59.89) - open/close 23643 (time 1787121348.17 total 984.64 last 67.97) - open/close 24413 (time 1787121358.18 total 994.65 last 76.91) - open/close 25079 (time 1787121368.19 total 1004.66 last 66.56) - open/close 25825 (time 1787121378.20 total 1014.66 last 74.54) - open/close 26498 (time 1787121388.21 total 1024.67 last 67.22) - open/close 27202 (time 1787121398.22 total 1034.69 last 70.31) - open/close 27932 (time 1787121408.22 total 1044.69 last 72.99) - open/close 28702 (time 1787121418.24 total 1054.70 last 76.88) - open/close 29434 (time 1787121428.25 total 1064.71 last 73.13) - open/close 30000 (time 1787121435.35 total 1071.81 last 79.70) - open/close 30561 (time 1787121445.36 total 1081.83 last 56.03) - open/close 31140 (time 1787121455.38 total 1091.84 last 57.81) - open/close 31878 (time 1787121465.38 total 1101.84 last 73.80) - open/close 32454 (time 1787121475.38 total 1111.84 last 57.60) - open/close 33267 (time 1787121485.39 total 1121.85 last 81.23) - open/close 34000 (time 1787121495.39 total 1131.86 last 73.27) - open/close 34636 (time 1787121505.40 total 1141.86 last 63.55) - open/close 35115 (time 1787121515.41 total 1151.87 last 47.85) - open/close 35625 (time 1787121525.41 total 1161.88 last 50.99) - open/close 36208 (time 1787121535.42 total 1171.88 last 58.28) - open/close 36767 (time 1787121545.42 total 1181.89 last 55.85) - open/close 37586 (time 1787121555.43 total 1191.90 last 81.82) - open/close 38218 (time 1787121565.44 total 1201.91 last 63.13) - open/close 38917 (time 1787121575.45 total 1211.92 last 69.84) - open/close 39577 (time 1787121585.46 total 1221.92 last 65.97) - open/close 40000 (time 1787121590.85 total 1227.32 last 78.40) - open/close 40926 (time 1787121600.86 total 1237.33 last 92.51) - open/close 41644 (time 1787121610.88 total 1247.35 last 71.66) - open/close 42403 (time 1787121620.88 total 1257.35 last 75.89) - open/close 43148 (time 1787121630.89 total 1267.35 last 74.47) - open/close 43855 (time 1787121640.90 total 1277.36 last 70.61) - open/close 44660 (time 1787121650.91 total 1287.37 last 80.44) - open/close 45476 (time 1787121660.91 total 1297.37 last 81.60) - open/close 46192 (time 1787121670.91 total 1307.38 last 71.58) - open/close 46929 (time 1787121680.92 total 1317.39 last 73.61) - open/close 47758 (time 1787121690.93 total 1327.39 last 82.86) - open/close 48476 (time 1787121700.96 total 1337.42 last 71.58) - open/close 49290 (time 1787121710.99 total 1347.46 last 81.11) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1353.33 seconds: 36.67 ops/second - unlinked 0 (time 1787121722 ; total 0 ; last 0) - unlinked 10000 (time 1787121864 ; total 142 ; last 142) - unlinked 20000 (time 1787122006 ; total 284 ; last 142) - unlinked 30000 (time 1787122144 ; total 422 ; last 138) - unlinked 40000 (time 1787122288 ; total 566 ; last 144) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 694 seconds: 71.514412 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg255-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server unloading modules via unload_modules_local on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing unload_modules_local oleg255-server: modules unloaded. pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2