== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:09:59 (1787342999) start mds service on oleg442-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing load_modules_local oleg442-server: Loading modules from /home/green/git/lustre-release/lustre oleg442-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg442-server: detected 4 online CPUs by sysfs oleg442-server: MODOPTS_LIBCFS= oleg442-server: Force libcfs to create 2 CPU partitions oleg442-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg442-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg442-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg442-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg442-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0000 oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 89 (time 1787343112.47 total 10.02 last 8.88) - open/close 249 (time 1787343124.70 total 22.25 last 13.08) - open/close 441 (time 1787343134.70 total 32.25 last 19.20) - open/close 705 (time 1787343145.37 total 42.92 last 24.73) - open/close 914 (time 1787343155.38 total 52.93 last 20.90) - open/close 1137 (time 1787343165.39 total 62.94 last 22.27) - open/close 1398 (time 1787343176.46 total 74.01 last 23.57) - open/close 1634 (time 1787343186.49 total 84.04 last 23.55) - open/close 1844 (time 1787343196.51 total 94.07 last 20.94) - open/close 2045 (time 1787343206.55 total 104.10 last 20.03) - open/close 2242 (time 1787343216.85 total 114.40 last 19.13) - open/close 2349 (time 1787343226.86 total 124.41 last 10.69) - open/close 2512 (time 1787343236.87 total 134.42 last 16.28) - open/close 2777 (time 1787343246.87 total 144.42 last 26.49) - open/close 2966 (time 1787343257.50 total 155.05 last 17.78) - open/close 3158 (time 1787343268.33 total 165.88 last 17.74) - open/close 3413 (time 1787343280.16 total 177.71 last 21.55) - open/close 3691 (time 1787343290.16 total 187.71 last 27.80) - open/close 3918 (time 1787343301.08 total 198.63 last 20.78) - open/close 4158 (time 1787343311.09 total 208.64 last 24.00) - open/close 4377 (time 1787343321.69 total 219.24 last 20.65) - open/close 4652 (time 1787343331.70 total 229.25 last 27.48) - open/close 4884 (time 1787343341.71 total 239.26 last 23.17) - open/close 5129 (time 1787343352.65 total 250.20 last 22.39) - open/close 5413 (time 1787343362.66 total 260.21 last 28.37) - open/close 5600 (time 1787343372.67 total 270.22 last 18.68) - open/close 5789 (time 1787343382.70 total 280.25 last 18.85) - open/close 5959 (time 1787343392.92 total 290.47 last 16.63) - open/close 6189 (time 1787343403.14 total 300.69 last 22.49) - open/close 6414 (time 1787343413.59 total 311.14 last 21.55) - open/close 6602 (time 1787343423.59 total 321.14 last 18.79) - open/close 6763 (time 1787343434.22 total 331.77 last 15.15) - open/close 6991 (time 1787343444.22 total 341.77 last 22.79) - open/close 7216 (time 1787343454.37 total 351.92 last 22.18) - open/close 7403 (time 1787343464.85 total 362.40 last 17.83) - open/close 7672 (time 1787343474.90 total 372.45 last 26.79) - open/close 7857 (time 1787343485.78 total 383.33 last 16.99) - open/close 7989 (time 1787343496.09 total 393.64 last 12.80) - open/close 8158 (time 1787343506.72 total 404.27 last 15.90) - open/close 8426 (time 1787343517.27 total 414.82 last 25.41) - open/close 8686 (time 1787343527.88 total 425.43 last 24.49) - open/close 8974 (time 1787343537.89 total 435.44 last 28.79) - open/close 9153 (time 1787343548.77 total 446.32 last 16.45) - open/close 9282 (time 1787343558.78 total 456.33 last 12.89) - open/close 9517 (time 1787343569.30 total 466.85 last 22.32) - open/close 9734 (time 1787343579.38 total 476.93 last 21.54) - open/close 9979 (time 1787343589.39 total 486.94 last 24.47) - open/close 10213 (time 1787343599.41 total 496.96 last 23.36) - open/close 10487 (time 1787343610.38 total 507.93 last 24.96) - open/close 10735 (time 1787343620.41 total 517.96 last 24.74) - open/close 10952 (time 1787343631.25 total 528.80 last 20.01) - open/close 11188 (time 1787343641.64 total 539.19 last 22.72) - open/close 11426 (time 1787343651.98 total 549.53 last 23.01) - open/close 11660 (time 1787343662.48 total 560.03 last 22.29) - open/close 11815 (time 1787343673.17 total 570.72 last 14.51) - open/close 11986 (time 1787343683.34 total 580.89 last 16.82) - open/close 12181 (time 1787343693.35 total 590.90 last 19.47) - open/close 12202 (time 1787343704.22 total 601.77 last 1.93) - open/close 12331 (time 1787343715.24 total 612.79 last 11.70) - open/close 12477 (time 1787343726.11 total 623.66 last 13.43) - open/close 12620 (time 1787343736.46 total 634.01 last 13.82) - open/close 12866 (time 1787343746.65 total 644.20 last 24.15) - open/close 13038 (time 1787343756.66 total 654.21 last 17.18) - open/close 13256 (time 1787343766.66 total 664.21 last 21.78) - open/close 13462 (time 1787343776.71 total 674.26 last 20.51) - open/close 13706 (time 1787343787.29 total 684.84 last 23.06) - open/close 13854 (time 1787343797.29 total 694.84 last 14.80) - open/close 14111 (time 1787343807.77 total 705.32 last 24.53) - open/close 14359 (time 1787343817.78 total 715.33 last 24.76) - open/close 14600 (time 1787343827.79 total 725.34 last 24.09) - open/close 14831 (time 1787343838.11 total 735.66 last 22.37) - open/close 15110 (time 1787343848.80 total 746.35 last 26.12) - open/close 15372 (time 1787343859.69 total 757.24 last 24.05) - open/close 15617 (time 1787343869.84 total 767.39 last 24.14) - open/close 15866 (time 1787343880.08 total 777.63 last 24.32) - open/close 16154 (time 1787343890.10 total 787.65 last 28.75) - open/close 16415 (time 1787343900.96 total 798.51 last 24.04) - open/close 16574 (time 1787343911.90 total 809.45 last 14.54) - open/close 16643 (time 1787343922.82 total 820.37 last 6.32) - open/close 16743 (time 1787343933.47 total 831.02 last 9.39) - open/close 16939 (time 1787343943.48 total 841.03 last 19.59) - open/close 17178 (time 1787343953.95 total 851.50 last 22.82) - open/close 17463 (time 1787343963.95 total 861.50 last 28.50) - open/close 17720 (time 1787343974.62 total 872.17 last 24.09) - open/close 17917 (time 1787343985.43 total 882.98 last 18.22) - open/close 18186 (time 1787343996.27 total 893.82 last 24.82) - open/close 18362 (time 1787344006.64 total 904.19 last 16.96) - open/close 18565 (time 1787344017.71 total 915.26 last 18.34) - open/close 18830 (time 1787344027.72 total 925.27 last 26.47) - open/close 19013 (time 1787344038.09 total 935.64 last 17.65) - open/close 19379 (time 1787344048.10 total 945.65 last 36.59) - open/close 20000 (time 1787344056.69 total 954.24 last 72.23) - open/close 20743 (time 1787344066.70 total 964.25 last 74.25) - open/close 21430 (time 1787344076.71 total 974.26 last 68.67) - open/close 22172 (time 1787344086.71 total 984.26 last 74.17) - open/close 22890 (time 1787344096.72 total 994.27 last 71.74) - open/close 23612 (time 1787344106.72 total 1004.27 last 72.20) - open/close 24349 (time 1787344116.74 total 1014.29 last 73.57) - open/close 25085 (time 1787344126.75 total 1024.30 last 73.52) - open/close 25824 (time 1787344136.75 total 1034.30 last 73.87) - open/close 26574 (time 1787344146.75 total 1044.30 last 74.98) - open/close 27162 (time 1787344156.77 total 1054.32 last 58.70) - open/close 27790 (time 1787344166.77 total 1064.32 last 62.79) - open/close 28445 (time 1787344176.77 total 1074.32 last 65.49) - open/close 29132 (time 1787344186.78 total 1084.33 last 68.64) - open/close 29896 (time 1787344196.81 total 1094.36 last 76.17) - open/close 30620 (time 1787344206.83 total 1104.38 last 72.29) - open/close 31317 (time 1787344216.83 total 1114.38 last 69.69) - open/close 32018 (time 1787344226.84 total 1124.39 last 70.03) - open/close 32694 (time 1787344236.84 total 1134.40 last 67.56) - open/close 33361 (time 1787344246.85 total 1144.40 last 66.70) - open/close 33938 (time 1787344256.85 total 1154.40 last 57.68) - open/close 34549 (time 1787344266.85 total 1164.40 last 61.08) - open/close 35245 (time 1787344276.85 total 1174.40 last 69.59) - open/close 35989 (time 1787344286.86 total 1184.41 last 74.39) - open/close 36667 (time 1787344296.86 total 1194.41 last 67.76) - open/close 37456 (time 1787344306.89 total 1204.44 last 78.69) - open/close 38183 (time 1787344316.90 total 1214.45 last 72.62) - open/close 38905 (time 1787344326.90 total 1224.46 last 72.16) - open/close 39654 (time 1787344336.91 total 1234.46 last 74.84) - open/close 40000 (time 1787344342.04 total 1239.59 last 67.42) - open/close 40710 (time 1787344352.05 total 1249.60 last 70.95) - open/close 41485 (time 1787344362.08 total 1259.63 last 77.27) - open/close 42120 (time 1787344372.11 total 1269.66 last 63.30) - open/close 42863 (time 1787344382.12 total 1279.67 last 74.26) - open/close 43620 (time 1787344392.13 total 1289.68 last 75.65) - open/close 44310 (time 1787344402.14 total 1299.69 last 68.93) - open/close 45093 (time 1787344412.14 total 1309.69 last 78.28) - open/close 45869 (time 1787344422.14 total 1319.69 last 77.58) - open/close 46492 (time 1787344432.15 total 1329.70 last 62.26) - open/close 47128 (time 1787344442.15 total 1339.70 last 63.56) - open/close 47757 (time 1787344452.16 total 1349.71 last 62.84) - open/close 48368 (time 1787344462.16 total 1359.71 last 61.10) - open/close 48973 (time 1787344472.17 total 1369.72 last 60.48) - open/close 49601 (time 1787344482.18 total 1379.73 last 62.75) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1380.10 seconds: 35.96 ops/second - unlinked 0 (time 1787344487 ; total 0 ; last 0) - unlinked 10000 (time 1787344626 ; total 139 ; last 139) - unlinked 20000 (time 1787344773 ; total 286 ; last 147) - unlinked 30000 (time 1787344932 ; total 445 ; last 159) - unlinked 40000 (time 1787345079 ; total 592 ; last 147) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 729 seconds: 68.080933 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg442-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg442-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg442-server stop mds service on oleg442-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg442-server stop mds service on oleg442-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg442-server unloading modules via unload_modules_local on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing unload_modules_local oleg442-server: modules unloaded. pdsh@oleg442-client: oleg442-client: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-server: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-client: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-server: ssh exited with exit code 2