== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:22:12 (1775769732) start mds service on oleg421-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing load_modules_local oleg421-server: Loading modules from /home/green/git/lustre-release/lustre oleg421-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg421-server: detected 4 online CPUs by sysfs oleg421-server: MODOPTS_LIBCFS= oleg421-server: Force libcfs to create 2 CPU partitions oleg421-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg421-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg421-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg421-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg421-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0000 oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 131 (time 1775769850.60 total 10.06 last 13.02) - open/close 253 (time 1775769861.11 total 20.57 last 11.61) - open/close 399 (time 1775769871.41 total 30.87 last 14.18) - open/close 513 (time 1775769881.95 total 41.41 last 10.81) - open/close 699 (time 1775769891.96 total 51.42 last 18.59) - open/close 913 (time 1775769902.60 total 62.06 last 20.11) - open/close 1089 (time 1775769913.14 total 72.61 last 16.69) - open/close 1279 (time 1775769923.16 total 82.62 last 18.97) - open/close 1531 (time 1775769934.06 total 93.52 last 23.11) - open/close 1722 (time 1775769944.07 total 103.53 last 19.09) - open/close 1868 (time 1775769954.28 total 113.75 last 14.29) - open/close 2069 (time 1775769964.59 total 124.06 last 19.50) - open/close 2317 (time 1775769974.61 total 134.07 last 24.77) - open/close 2540 (time 1775769985.46 total 144.92 last 20.55) - open/close 2718 (time 1775769996.20 total 155.66 last 16.57) - open/close 2953 (time 1775770006.20 total 165.66 last 23.50) - open/close 3093 (time 1775770017.20 total 176.67 last 12.72) - open/close 3289 (time 1775770027.37 total 186.83 last 19.28) - open/close 3458 (time 1775770038.05 total 197.52 last 15.82) - open/close 3555 (time 1775770050.59 total 210.06 last 7.74) - open/close 3791 (time 1775770061.67 total 221.13 last 21.31) - open/close 3932 (time 1775770071.68 total 231.14 last 14.08) - open/close 4168 (time 1775770081.69 total 241.15 last 23.59) - open/close 4368 (time 1775770092.12 total 251.59 last 19.16) - open/close 4555 (time 1775770102.13 total 261.59 last 18.69) - open/close 4741 (time 1775770112.14 total 271.60 last 18.59) - open/close 4780 (time 1775770122.49 total 281.95 last 3.77) - open/close 4873 (time 1775770132.49 total 291.95 last 9.30) - open/close 5041 (time 1775770142.63 total 302.09 last 16.56) - open/close 5168 (time 1775770153.65 total 313.11 last 11.53) - open/close 5379 (time 1775770163.66 total 323.12 last 21.08) - open/close 5542 (time 1775770174.17 total 333.63 last 15.50) - open/close 5653 (time 1775770184.36 total 343.82 last 10.90) - open/close 5772 (time 1775770195.36 total 354.82 last 10.81) - open/close 5926 (time 1775770205.38 total 364.85 last 15.37) - open/close 6041 (time 1775770215.39 total 374.85 last 11.49) - open/close 6163 (time 1775770226.34 total 385.80 last 11.14) - open/close 6363 (time 1775770238.88 total 398.35 last 15.94) - open/close 6473 (time 1775770248.88 total 408.35 last 11.00) - open/close 6573 (time 1775770258.91 total 418.37 last 9.97) - open/close 6636 (time 1775770269.16 total 428.62 last 6.15) - open/close 6711 (time 1775770279.71 total 439.17 last 7.11) - open/close 6800 (time 1775770294.07 total 453.53 last 6.20) - open/close 6874 (time 1775770304.94 total 464.41 last 6.80) - open/close 7056 (time 1775770314.96 total 474.42 last 18.17) - open/close 7189 (time 1775770325.29 total 484.75 last 12.87) - open/close 7298 (time 1775770335.30 total 494.76 last 10.89) - open/close 7504 (time 1775770348.09 total 507.55 last 16.10) - open/close 7661 (time 1775770362.81 total 522.27 last 10.66) - open/close 7830 (time 1775770373.71 total 533.17 last 15.51) - open/close 8000 (time 1775770385.51 total 544.97 last 14.40) - open/close 8159 (time 1775770396.12 total 555.58 last 14.99) - open/close 8299 (time 1775770408.16 total 567.62 last 11.63) - open/close 8526 (time 1775770418.16 total 577.63 last 22.68) - open/close 8646 (time 1775770429.15 total 588.61 last 10.92) - open/close 8802 (time 1775770440.83 total 600.29 last 13.36) - open/close 9032 (time 1775770451.37 total 610.83 last 21.81) - open/close 9145 (time 1775770461.43 total 620.89 last 11.24) - open/close 9252 (time 1775770472.36 total 631.82 last 9.79) - open/close 9435 (time 1775770482.37 total 641.83 last 18.29) - open/close 9510 (time 1775770492.64 total 652.10 last 7.30) - open/close 9741 (time 1775770505.60 total 665.06 last 17.82) - open/close 9929 (time 1775770516.68 total 676.14 last 16.97) - open/close 10000 (time 1775770524.67 total 684.14 last 8.88) - open/close 10113 (time 1775770535.67 total 695.13 last 10.27) - open/close 10212 (time 1775770546.35 total 705.81 last 9.27) - open/close 10394 (time 1775770556.37 total 715.83 last 18.15) - open/close 10548 (time 1775770567.04 total 726.50 last 14.44) - open/close 10671 (time 1775770577.96 total 737.43 last 11.26) - open/close 10828 (time 1775770588.32 total 747.79 last 15.15) - open/close 11003 (time 1775770599.15 total 758.61 last 16.17) - open/close 11127 (time 1775770609.17 total 768.63 last 12.37) - open/close 11251 (time 1775770619.88 total 779.34 last 11.58) - open/close 11434 (time 1775770629.90 total 789.36 last 18.25) - open/close 11555 (time 1775770640.24 total 799.71 last 11.70) - open/close 11723 (time 1775770650.54 total 810.00 last 16.32) - open/close 11958 (time 1775770661.28 total 820.75 last 21.87) - open/close 12191 (time 1775770672.30 total 831.76 last 21.15) - open/close 12314 (time 1775770682.32 total 841.78 last 12.28) - open/close 12420 (time 1775770693.09 total 852.55 last 9.84) - open/close 12603 (time 1775770703.15 total 862.61 last 18.18) - open/close 12738 (time 1775770713.32 total 872.78 last 13.28) - open/close 12925 (time 1775770723.52 total 882.98 last 18.34) - open/close 13129 (time 1775770734.31 total 893.78 last 18.89) - open/close 13237 (time 1775770744.33 total 903.79 last 10.78) - open/close 13430 (time 1775770754.51 total 913.97 last 18.96) - open/close 13559 (time 1775770765.51 total 924.97 last 11.73) - open/close 13665 (time 1775770775.78 total 935.25 last 10.32) - open/close 13796 (time 1775770785.81 total 945.27 last 13.07) - open/close 13865 (time 1775770795.81 total 955.27 last 6.90) - open/close 13999 (time 1775770806.11 total 965.57 last 13.01) - open/close 14074 (time 1775770816.23 total 975.69 last 7.41) - open/close 14247 (time 1775770826.73 total 986.20 last 16.47) - open/close 14295 (time 1775770837.55 total 997.02 last 4.44) - open/close 14354 (time 1775770850.60 total 1010.06 last 4.52) - open/close 14466 (time 1775770861.43 total 1020.89 last 10.35) - open/close 14642 (time 1775770871.43 total 1030.89 last 17.60) - open/close 14780 (time 1775770881.44 total 1040.90 last 13.79) - open/close 14843 (time 1775770892.26 total 1051.72 last 5.82) - open/close 15058 (time 1775770903.01 total 1062.48 last 19.99) - open/close 15298 (time 1775770913.13 total 1072.59 last 23.72) - open/close 15724 (time 1775770923.14 total 1082.60 last 42.58) - open/close 16382 (time 1775770933.14 total 1092.60 last 65.78) - open/close 17044 (time 1775770943.14 total 1102.60 last 66.19) - open/close 17728 (time 1775770953.16 total 1112.62 last 68.29) - open/close 18440 (time 1775770963.16 total 1122.63 last 71.14) - open/close 19071 (time 1775770973.17 total 1132.63 last 63.06) - open/close 19705 (time 1775770983.18 total 1142.64 last 63.36) - open/close 20000 (time 1775770987.18 total 1146.64 last 73.64) - open/close 20742 (time 1775770997.19 total 1156.65 last 74.15) - open/close 21491 (time 1775771007.19 total 1166.66 last 74.86) - open/close 22204 (time 1775771017.21 total 1176.67 last 71.20) - open/close 22900 (time 1775771027.22 total 1186.68 last 69.51) - open/close 23469 (time 1775771037.23 total 1196.69 last 56.84) - open/close 23996 (time 1775771047.23 total 1206.70 last 52.68) - open/close 24597 (time 1775771057.24 total 1216.70 last 60.08) - open/close 25262 (time 1775771067.25 total 1226.71 last 66.44) - open/close 25943 (time 1775771077.26 total 1236.72 last 68.00) - open/close 26601 (time 1775771087.26 total 1246.72 last 65.79) - open/close 27252 (time 1775771097.28 total 1256.74 last 65.00) - open/close 27884 (time 1775771107.29 total 1266.76 last 63.10) - open/close 28473 (time 1775771117.31 total 1276.77 last 58.82) - open/close 29152 (time 1775771127.31 total 1286.77 last 67.87) - open/close 29755 (time 1775771137.32 total 1296.79 last 60.24) - open/close 30000 (time 1775771140.73 total 1300.19 last 71.97) - open/close 30649 (time 1775771150.76 total 1310.22 last 64.68) - open/close 31244 (time 1775771160.81 total 1320.27 last 59.21) - open/close 31873 (time 1775771170.83 total 1330.29 last 62.80) - open/close 32442 (time 1775771180.84 total 1340.31 last 56.81) - open/close 32977 (time 1775771190.85 total 1350.31 last 53.48) - open/close 33563 (time 1775771200.86 total 1360.32 last 58.53) - open/close 34099 (time 1775771210.86 total 1370.33 last 53.57) - open/close 34683 (time 1775771220.88 total 1380.34 last 58.32) - open/close 35194 (time 1775771230.88 total 1390.34 last 51.09) - open/close 35818 (time 1775771240.91 total 1400.37 last 62.24) - open/close 36382 (time 1775771250.92 total 1410.38 last 56.35) - open/close 36983 (time 1775771260.92 total 1420.39 last 60.05) - open/close 37619 (time 1775771270.93 total 1430.40 last 63.54) - open/close 38209 (time 1775771280.94 total 1440.40 last 58.99) - open/close 38899 (time 1775771290.95 total 1450.41 last 68.91) - open/close 39486 (time 1775771300.96 total 1460.42 last 58.64) - open/close 40000 (time 1775771310.14 total 1469.60 last 55.97) - open/close 40692 (time 1775771320.14 total 1479.61 last 69.19) - open/close 41305 (time 1775771330.14 total 1489.61 last 61.30) - open/close 41937 (time 1775771340.15 total 1499.61 last 63.16) - open/close 42553 (time 1775771350.18 total 1509.64 last 61.41) - open/close 43146 (time 1775771360.19 total 1519.65 last 59.25) - open/close 43748 (time 1775771370.20 total 1529.66 last 60.12) - open/close 44397 (time 1775771380.21 total 1539.67 last 64.84) - open/close 45048 (time 1775771390.23 total 1549.69 last 64.98) - open/close 45503 (time 1775771400.25 total 1559.71 last 45.42) - open/close 45912 (time 1775771410.25 total 1569.71 last 40.87) - open/close 46375 (time 1775771420.27 total 1579.73 last 46.23) - open/close 46908 (time 1775771430.27 total 1589.73 last 53.28) - open/close 47509 (time 1775771440.29 total 1599.76 last 59.95) - open/close 48159 (time 1775771450.30 total 1609.76 last 64.98) - open/close 48790 (time 1775771460.32 total 1619.78 last 62.99) - open/close 49492 (time 1775771470.32 total 1629.78 last 70.17) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1631.98 seconds: 30.41 ops/second - unlinked 0 (time 1775771477 ; total 0 ; last 0) - unlinked 10000 (time 1775771638 ; total 161 ; last 161) - unlinked 20000 (time 1775771805 ; total 328 ; last 167) - unlinked 30000 (time 1775771972 ; total 495 ; last 167) - unlinked 40000 (time 1775772141 ; total 664 ; last 169) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 810 seconds: 61.274075 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg421-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg421-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg421-server stop mds service on oleg421-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg421-server stop mds service on oleg421-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg421-server unloading modules via unload_modules_local on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing unload_modules_local oleg421-server: modules unloaded. pdsh@oleg421-client: oleg421-server: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-client: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-client: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 2