== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:35:10 (1787798110) start mds service on oleg158-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg158-server' oleg158-server: oleg158-server.virtnet: executing load_modules_local oleg158-server: Loading modules from /home/green/git/lustre-release/lustre oleg158-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg158-server: detected 4 online CPUs by sysfs oleg158-server: MODOPTS_LIBCFS= oleg158-server: Force libcfs to create 2 CPU partitions oleg158-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg158-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg158-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg158-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg158-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-OST0000 oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 195 (time 1787798212.52 total 10.80 last 18.06) - open/close 444 (time 1787798222.63 total 20.91 last 24.63) - open/close 697 (time 1787798233.20 total 31.48 last 23.94) - open/close 942 (time 1787798244.20 total 42.48 last 22.26) - open/close 1164 (time 1787798254.23 total 52.50 last 22.15) - open/close 1399 (time 1787798264.30 total 62.57 last 23.34) - open/close 1621 (time 1787798276.20 total 74.48 last 18.65) - open/close 1894 (time 1787798286.81 total 85.09 last 25.73) - open/close 2164 (time 1787798297.41 total 95.69 last 25.47) - open/close 2432 (time 1787798308.46 total 106.73 last 24.26) - open/close 2709 (time 1787798319.08 total 117.36 last 26.06) - open/close 2963 (time 1787798329.19 total 127.47 last 25.14) - open/close 3265 (time 1787798340.21 total 138.49 last 27.40) - open/close 3545 (time 1787798350.76 total 149.03 last 26.55) - open/close 3850 (time 1787798360.76 total 159.04 last 30.49) - open/close 4126 (time 1787798370.76 total 169.04 last 27.59) - open/close 4370 (time 1787798381.73 total 180.01 last 22.24) - open/close 4594 (time 1787798391.74 total 190.02 last 22.39) - open/close 4827 (time 1787798401.74 total 200.02 last 23.29) - open/close 5067 (time 1787798412.07 total 210.35 last 23.24) - open/close 5244 (time 1787798422.09 total 220.37 last 17.67) - open/close 5439 (time 1787798432.94 total 231.22 last 17.96) - open/close 5652 (time 1787798442.95 total 241.23 last 21.29) - open/close 5868 (time 1787798452.96 total 251.24 last 21.57) - open/close 6137 (time 1787798463.97 total 262.25 last 24.43) - open/close 6415 (time 1787798474.69 total 272.97 last 25.94) - open/close 6677 (time 1787798484.70 total 282.98 last 26.17) - open/close 6898 (time 1787798494.72 total 293.00 last 22.05) - open/close 7079 (time 1787798505.01 total 303.28 last 17.60) - open/close 7381 (time 1787798515.01 total 313.29 last 30.17) - open/close 7622 (time 1787798525.99 total 324.27 last 21.96) - open/close 7866 (time 1787798536.00 total 334.28 last 24.37) - open/close 8127 (time 1787798547.07 total 345.34 last 23.59) - open/close 8261 (time 1787798557.24 total 355.52 last 13.17) - open/close 8477 (time 1787798567.25 total 365.53 last 21.58) - open/close 8636 (time 1787798578.22 total 376.50 last 14.49) - open/close 8823 (time 1787798588.23 total 386.51 last 18.68) - open/close 9049 (time 1787798598.50 total 396.78 last 22.01) - open/close 9260 (time 1787798611.55 total 409.83 last 16.17) - open/close 9424 (time 1787798621.93 total 420.21 last 15.80) - open/close 9541 (time 1787798632.29 total 430.57 last 11.30) - open/close 9789 (time 1787798642.93 total 441.20 last 23.31) - open/close 10000 (time 1787798652.09 total 450.37 last 23.03) - open/close 10216 (time 1787798662.09 total 460.37 last 21.59) - open/close 10458 (time 1787798672.11 total 470.38 last 24.17) - open/close 10662 (time 1787798682.65 total 480.93 last 19.34) - open/close 10897 (time 1787798692.66 total 490.94 last 23.49) - open/close 11124 (time 1787798702.67 total 500.94 last 22.68) - open/close 11330 (time 1787798713.12 total 511.40 last 19.70) - open/close 11604 (time 1787798724.00 total 522.28 last 25.19) - open/close 11868 (time 1787798734.24 total 532.51 last 25.79) - open/close 12119 (time 1787798744.30 total 542.57 last 24.95) - open/close 12402 (time 1787798755.20 total 553.47 last 25.96) - open/close 12624 (time 1787798765.68 total 563.95 last 21.19) - open/close 12822 (time 1787798776.00 total 574.28 last 19.17) - open/close 13113 (time 1787798786.36 total 584.64 last 28.10) - open/close 13353 (time 1787798796.37 total 594.64 last 23.98) - open/close 13568 (time 1787798807.21 total 605.49 last 19.82) - open/close 13827 (time 1787798817.21 total 615.49 last 25.89) - open/close 14081 (time 1787798827.23 total 625.51 last 25.35) - open/close 14311 (time 1787798837.47 total 635.75 last 22.46) - open/close 14516 (time 1787798847.50 total 645.77 last 20.46) - open/close 14841 (time 1787798858.34 total 656.62 last 29.96) - open/close 15096 (time 1787798868.34 total 666.62 last 25.50) - open/close 15351 (time 1787798878.51 total 676.79 last 25.07) - open/close 15581 (time 1787798889.24 total 687.52 last 21.44) - open/close 15839 (time 1787798900.07 total 698.35 last 23.82) - open/close 16077 (time 1787798910.50 total 708.78 last 22.82) - open/close 16217 (time 1787798920.51 total 718.79 last 13.99) - open/close 16359 (time 1787798933.09 total 731.37 last 11.29) - open/close 16464 (time 1787798943.52 total 741.80 last 10.06) - open/close 16661 (time 1787798953.56 total 751.84 last 19.63) - open/close 16860 (time 1787798964.20 total 762.47 last 18.71) - open/close 16893 (time 1787798974.92 total 773.20 last 3.08) - open/close 17209 (time 1787798985.08 total 783.36 last 31.11) - open/close 17435 (time 1787798996.27 total 794.54 last 20.20) - open/close 17664 (time 1787799006.29 total 804.57 last 22.84) - open/close 17894 (time 1787799016.88 total 815.16 last 21.72) - open/close 18118 (time 1787799027.76 total 826.03 last 20.60) - open/close 18339 (time 1787799037.77 total 836.05 last 22.07) - open/close 18548 (time 1787799047.80 total 846.08 last 20.84) - open/close 18693 (time 1787799057.80 total 856.08 last 14.50) - open/close 18938 (time 1787799067.80 total 866.08 last 24.49) - open/close 19097 (time 1787799077.81 total 876.08 last 15.89) - open/close 19317 (time 1787799087.92 total 886.19 last 21.76) - open/close 20000 (time 1787799096.83 total 895.10 last 76.65) - open/close 20884 (time 1787799106.84 total 905.11 last 88.31) - open/close 21820 (time 1787799116.84 total 915.12 last 93.60) - open/close 22747 (time 1787799126.84 total 925.12 last 92.67) - open/close 23676 (time 1787799136.85 total 935.13 last 92.80) - open/close 24533 (time 1787799146.85 total 945.13 last 85.68) - open/close 25385 (time 1787799156.86 total 955.13 last 85.19) - open/close 26237 (time 1787799166.86 total 965.13 last 85.19) - open/close 27198 (time 1787799176.86 total 975.14 last 96.06) - open/close 28039 (time 1787799186.87 total 985.14 last 84.05) - open/close 29003 (time 1787799196.87 total 995.15 last 96.36) - open/close 29811 (time 1787799206.87 total 1005.15 last 80.78) - open/close 30000 (time 1787799209.28 total 1007.56 last 78.48) - open/close 30905 (time 1787799219.29 total 1017.56 last 90.46) - open/close 31740 (time 1787799229.30 total 1027.58 last 83.40) - open/close 32583 (time 1787799239.30 total 1037.58 last 84.26) - open/close 33587 (time 1787799249.30 total 1047.58 last 100.37) - open/close 34399 (time 1787799259.31 total 1057.58 last 81.19) - open/close 35177 (time 1787799269.31 total 1067.59 last 77.75) - open/close 36005 (time 1787799279.33 total 1077.61 last 82.68) - open/close 36947 (time 1787799289.34 total 1087.61 last 94.11) - open/close 37627 (time 1787799299.34 total 1097.62 last 67.94) - open/close 38418 (time 1787799309.35 total 1107.63 last 79.04) - open/close 39105 (time 1787799319.36 total 1117.64 last 68.64) - open/close 39826 (time 1787799329.37 total 1127.65 last 72.03) - open/close 40000 (time 1787799331.63 total 1129.91 last 76.95) - open/close 40807 (time 1787799341.65 total 1139.93 last 80.55) - open/close 41527 (time 1787799351.67 total 1149.95 last 71.86) - open/close 42200 (time 1787799361.68 total 1159.95 last 67.27) - open/close 43174 (time 1787799371.68 total 1169.96 last 97.35) - open/close 44112 (time 1787799381.70 total 1179.97 last 93.67) - open/close 44880 (time 1787799391.70 total 1189.97 last 76.79) - open/close 45653 (time 1787799401.70 total 1199.98 last 77.25) - open/close 46369 (time 1787799411.71 total 1209.99 last 71.53) - open/close 47159 (time 1787799421.72 total 1220.00 last 78.96) - open/close 47847 (time 1787799431.72 total 1230.00 last 68.76) - open/close 48748 (time 1787799441.73 total 1240.01 last 90.05) - open/close 49504 (time 1787799451.74 total 1250.01 last 75.55) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1251.77 seconds: 39.65 ops/second - unlinked 0 (time 1787799458 ; total 0 ; last 0) - unlinked 10000 (time 1787799579 ; total 121 ; last 121) - unlinked 20000 (time 1787799693 ; total 235 ; last 114) - unlinked 30000 (time 1787799755 ; total 297 ; last 62) - unlinked 40000 (time 1787799827 ; total 369 ; last 72) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 429 seconds: 115.689980 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg158-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg158-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg158-server stop mds service on oleg158-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg158-server stop mds service on oleg158-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg158-server unloading modules via unload_modules_local on: 'oleg158-server' oleg158-server: oleg158-server.virtnet: executing unload_modules_local oleg158-server: modules unloaded. pdsh@oleg158-client: oleg158-server: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-client: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-client: ssh exited with exit code 2