== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:47:31 (1786420051) start mds service on oleg250-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg250-server' oleg250-server: oleg250-server.virtnet: executing load_modules_local oleg250-server: Loading modules from /home/green/git/lustre-release/lustre oleg250-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg250-server: detected 4 online CPUs by sysfs oleg250-server: MODOPTS_LIBCFS= oleg250-server: Force libcfs to create 2 CPU partitions oleg250-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg250-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg250-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg250-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg250-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 219 (time 1786420165.72 total 10.62 last 20.63) - open/close 506 (time 1786420175.73 total 20.63 last 28.66) - open/close 781 (time 1786420186.50 total 31.40 last 25.55) - open/close 1003 (time 1786420196.51 total 41.41 last 22.17) - open/close 1228 (time 1786420206.52 total 51.41 last 22.49) - open/close 1449 (time 1786420217.54 total 62.44 last 20.05) - open/close 1730 (time 1786420228.28 total 73.18 last 26.15) - open/close 2061 (time 1786420239.24 total 84.14 last 30.22) - open/close 2259 (time 1786420250.31 total 95.21 last 17.88) - open/close 2504 (time 1786420261.05 total 105.95 last 22.81) - open/close 2689 (time 1786420271.07 total 115.97 last 18.46) - open/close 2968 (time 1786420281.08 total 125.97 last 27.89) - open/close 3156 (time 1786420291.32 total 136.22 last 18.34) - open/close 3409 (time 1786420302.15 total 147.05 last 23.37) - open/close 3664 (time 1786420312.84 total 157.74 last 23.85) - open/close 3878 (time 1786420322.92 total 167.82 last 21.23) - open/close 4166 (time 1786420333.83 total 178.73 last 26.38) - open/close 4361 (time 1786420344.61 total 189.51 last 18.09) - open/close 4554 (time 1786420354.62 total 199.51 last 19.29) - open/close 4874 (time 1786420365.26 total 210.16 last 30.07) - open/close 5055 (time 1786420375.83 total 220.72 last 17.13) - open/close 5214 (time 1786420386.90 total 231.80 last 14.36) - open/close 5276 (time 1786420397.51 total 242.41 last 5.84) - open/close 5366 (time 1786420409.72 total 254.62 last 7.37) - open/close 5586 (time 1786420419.88 total 264.78 last 21.64) - open/close 5856 (time 1786420429.89 total 274.79 last 26.99) - open/close 6054 (time 1786420439.91 total 284.81 last 19.76) - open/close 6217 (time 1786420450.83 total 295.73 last 14.92) - open/close 6375 (time 1786420460.85 total 305.75 last 15.77) - open/close 6580 (time 1786420471.94 total 316.84 last 18.48) - open/close 6762 (time 1786420481.95 total 326.85 last 18.19) - open/close 6892 (time 1786420491.96 total 336.86 last 12.98) - open/close 6985 (time 1786420501.97 total 346.87 last 9.29) - open/close 7175 (time 1786420511.98 total 356.88 last 18.98) - open/close 7251 (time 1786420521.99 total 366.89 last 7.59) - open/close 7456 (time 1786420532.00 total 376.90 last 20.49) - open/close 7653 (time 1786420542.97 total 387.86 last 17.96) - open/close 7960 (time 1786420553.46 total 398.36 last 29.25) - open/close 8124 (time 1786420563.48 total 408.38 last 16.36) - open/close 8221 (time 1786420573.49 total 418.38 last 9.70) - open/close 8314 (time 1786420584.00 total 428.90 last 8.85) - open/close 8510 (time 1786420594.00 total 438.90 last 19.59) - open/close 8715 (time 1786420605.05 total 449.95 last 18.55) - open/close 8888 (time 1786420615.23 total 460.13 last 16.99) - open/close 9117 (time 1786420626.13 total 471.03 last 21.02) - open/close 9349 (time 1786420636.14 total 481.04 last 23.18) - open/close 9558 (time 1786420646.14 total 491.04 last 20.89) - open/close 9724 (time 1786420656.48 total 501.38 last 16.06) - open/close 9857 (time 1786420666.49 total 511.39 last 13.29) - open/close 9987 (time 1786420676.49 total 521.39 last 13.00) - open/close 10138 (time 1786420686.53 total 531.43 last 15.04) - open/close 10270 (time 1786420697.46 total 542.35 last 12.08) - open/close 10462 (time 1786420707.47 total 552.37 last 19.18) - open/close 10615 (time 1786420717.49 total 562.38 last 15.27) - open/close 10776 (time 1786420727.52 total 572.41 last 16.05) - open/close 10942 (time 1786420737.54 total 582.44 last 16.56) - open/close 11105 (time 1786420747.57 total 592.46 last 16.26) - open/close 11297 (time 1786420758.45 total 603.35 last 17.63) - open/close 11482 (time 1786420768.89 total 613.79 last 17.72) - open/close 11789 (time 1786420781.90 total 626.80 last 23.61) - open/close 11984 (time 1786420792.63 total 637.53 last 18.17) - open/close 12146 (time 1786420802.65 total 647.55 last 16.17) - open/close 12288 (time 1786420813.13 total 658.02 last 13.56) - open/close 12462 (time 1786420823.13 total 668.03 last 17.39) - open/close 12616 (time 1786420833.15 total 678.05 last 15.38) - open/close 12713 (time 1786420844.11 total 689.00 last 8.85) - open/close 12895 (time 1786420854.47 total 699.36 last 17.57) - open/close 13006 (time 1786420865.50 total 710.39 last 10.06) - open/close 13110 (time 1786420876.04 total 720.93 last 9.87) - open/close 13279 (time 1786420886.98 total 731.88 last 15.45) - open/close 13548 (time 1786420897.39 total 742.29 last 25.82) - open/close 13827 (time 1786420907.71 total 752.61 last 27.05) - open/close 13965 (time 1786420917.71 total 762.61 last 13.79) - open/close 14138 (time 1786420928.12 total 773.02 last 16.62) - open/close 14319 (time 1786420938.62 total 783.52 last 17.25) - open/close 14484 (time 1786420951.04 total 795.93 last 13.29) - open/close 14664 (time 1786420961.04 total 805.94 last 17.99) - open/close 14768 (time 1786420972.14 total 817.04 last 9.37) - open/close 14974 (time 1786420982.17 total 827.06 last 20.56) - open/close 15139 (time 1786420992.80 total 837.70 last 15.52) - open/close 15442 (time 1786421003.28 total 848.18 last 28.90) - open/close 15691 (time 1786421013.58 total 858.48 last 24.17) - open/close 15921 (time 1786421023.61 total 868.51 last 22.94) - open/close 15984 (time 1786421033.62 total 878.52 last 6.29) - open/close 16097 (time 1786421043.64 total 888.54 last 11.28) - open/close 16346 (time 1786421054.41 total 899.31 last 23.12) - open/close 16533 (time 1786421065.37 total 910.26 last 17.07) - open/close 16639 (time 1786421075.37 total 920.27 last 10.59) - open/close 16777 (time 1786421086.44 total 931.34 last 12.47) - open/close 16920 (time 1786421096.58 total 941.47 last 14.10) - open/close 17147 (time 1786421107.07 total 951.97 last 21.64) - open/close 17329 (time 1786421117.53 total 962.43 last 17.40) - open/close 17476 (time 1786421128.26 total 973.16 last 13.69) - open/close 17656 (time 1786421138.93 total 983.83 last 16.87) - open/close 17822 (time 1786421150.11 total 995.01 last 14.85) - open/close 18052 (time 1786421160.12 total 1005.02 last 22.98) - open/close 18278 (time 1786421170.67 total 1015.57 last 21.42) - open/close 18530 (time 1786421180.67 total 1025.57 last 25.19) - open/close 18794 (time 1786421190.67 total 1035.57 last 26.39) - open/close 19134 (time 1786421200.68 total 1045.58 last 33.99) - open/close 19920 (time 1786421210.70 total 1055.60 last 78.45) - open/close 20598 (time 1786421220.72 total 1065.62 last 67.64) - open/close 21280 (time 1786421230.74 total 1075.63 last 68.10) - open/close 22167 (time 1786421240.74 total 1085.64 last 88.66) - open/close 23133 (time 1786421250.74 total 1095.64 last 96.56) - open/close 23950 (time 1786421260.75 total 1105.65 last 81.66) - open/close 24765 (time 1786421270.76 total 1115.65 last 81.45) - open/close 25669 (time 1786421280.76 total 1125.66 last 90.38) - open/close 26561 (time 1786421290.77 total 1135.67 last 89.06) - open/close 27284 (time 1786421300.78 total 1145.68 last 72.26) - open/close 27933 (time 1786421310.78 total 1155.68 last 64.89) - open/close 28629 (time 1786421320.78 total 1165.68 last 69.56) - open/close 29582 (time 1786421330.79 total 1175.69 last 95.26) - open/close 30000 (time 1786421335.06 total 1179.96 last 97.85) - open/close 30942 (time 1786421345.07 total 1189.96 last 94.15) - open/close 31965 (time 1786421355.07 total 1199.96 last 102.28) - open/close 32905 (time 1786421365.07 total 1209.97 last 93.99) - open/close 33850 (time 1786421375.07 total 1219.97 last 94.49) - open/close 34566 (time 1786421385.08 total 1229.98 last 71.51) - open/close 35284 (time 1786421395.10 total 1240.00 last 71.64) - open/close 35831 (time 1786421405.11 total 1250.00 last 54.70) - open/close 36503 (time 1786421415.11 total 1260.01 last 67.17) - open/close 37139 (time 1786421425.12 total 1270.02 last 63.54) - open/close 37778 (time 1786421435.13 total 1280.02 last 63.86) - open/close 38496 (time 1786421445.14 total 1290.04 last 71.72) - open/close 39256 (time 1786421455.14 total 1300.04 last 75.99) - open/close 40000 (time 1786421464.63 total 1309.53 last 78.41) - open/close 40786 (time 1786421474.64 total 1319.54 last 78.49) - open/close 41438 (time 1786421484.64 total 1329.54 last 65.19) - open/close 42320 (time 1786421494.65 total 1339.55 last 88.14) - open/close 43071 (time 1786421504.67 total 1349.57 last 74.96) - open/close 43929 (time 1786421514.67 total 1359.57 last 85.77) - open/close 44693 (time 1786421524.68 total 1369.58 last 76.34) - open/close 45474 (time 1786421534.69 total 1379.59 last 78.03) - open/close 46273 (time 1786421544.69 total 1389.59 last 79.88) - open/close 47126 (time 1786421554.71 total 1399.60 last 85.18) - open/close 48141 (time 1786421564.72 total 1409.62 last 101.35) - open/close 48779 (time 1786421574.73 total 1419.63 last 63.73) - open/close 49304 (time 1786421584.74 total 1429.63 last 52.48) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1434.11 seconds: 34.61 ops/second - unlinked 0 (time 1786421594 ; total 0 ; last 0) - unlinked 10000 (time 1786421718 ; total 124 ; last 124) - unlinked 20000 (time 1786421834 ; total 240 ; last 116) - unlinked 30000 (time 1786421946 ; total 352 ; last 112) - unlinked 40000 (time 1786422069 ; total 475 ; last 123) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 550 seconds: 90.238182 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg250-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg250-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg250-server stop mds service on oleg250-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg250-server stop mds service on oleg250-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg250-server unloading modules via unload_modules_local on: 'oleg250-server' oleg250-server: oleg250-server.virtnet: executing unload_modules_local oleg250-server: modules unloaded. pdsh@oleg250-client: oleg250-client: ssh exited with exit code 2 pdsh@oleg250-client: oleg250-server: ssh exited with exit code 2 pdsh@oleg250-client: oleg250-client: ssh exited with exit code 2 pdsh@oleg250-client: oleg250-server: ssh exited with exit code 2