== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:53:26 (1775850806) start mds service on oleg227-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing load_modules_local oleg227-server: Loading modules from /home/green/git/lustre-release/lustre oleg227-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg227-server: detected 4 online CPUs by sysfs oleg227-server: MODOPTS_LIBCFS= oleg227-server: Force libcfs to create 2 CPU partitions oleg227-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg227-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg227-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg227-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg227-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-OST0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg227-client.virtnet: -o user_xattr,flock 192.168.202.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 201 (time 1775850918.88 total 10.02 last 20.06) - open/close 453 (time 1775850928.91 total 20.05 last 25.13) - open/close 576 (time 1775850938.92 total 30.05 last 12.29) - open/close 754 (time 1775850948.94 total 40.08 last 17.76) - open/close 899 (time 1775850958.95 total 50.08 last 14.49) - open/close 1044 (time 1775850969.03 total 60.16 last 14.39) - open/close 1192 (time 1775850979.03 total 70.17 last 14.80) - open/close 1439 (time 1775850989.58 total 80.71 last 23.43) - open/close 1619 (time 1775850999.59 total 90.72 last 17.98) - open/close 1749 (time 1775851009.72 total 100.85 last 12.83) - open/close 1939 (time 1775851020.21 total 111.34 last 18.11) - open/close 2121 (time 1775851030.22 total 121.35 last 18.18) - open/close 2262 (time 1775851043.17 total 134.31 last 10.89) - open/close 2416 (time 1775851053.38 total 144.52 last 15.08) - open/close 2655 (time 1775851064.20 total 155.34 last 22.09) - open/close 2915 (time 1775851074.75 total 165.88 last 24.65) - open/close 3145 (time 1775851085.13 total 176.26 last 22.17) - open/close 3331 (time 1775851095.28 total 186.42 last 18.31) - open/close 3570 (time 1775851105.29 total 196.43 last 23.88) - open/close 3807 (time 1775851116.29 total 207.42 last 21.55) - open/close 3971 (time 1775851126.84 total 217.97 last 15.55) - open/close 4149 (time 1775851136.85 total 227.98 last 17.78) - open/close 4322 (time 1775851146.95 total 238.08 last 17.12) - open/close 4451 (time 1775851156.96 total 248.10 last 12.88) - open/close 4642 (time 1775851167.80 total 258.93 last 17.62) - open/close 4869 (time 1775851177.81 total 268.94 last 22.69) - open/close 5096 (time 1775851188.15 total 279.28 last 21.95) - open/close 5325 (time 1775851198.40 total 289.53 last 22.34) - open/close 5483 (time 1775851208.69 total 299.82 last 15.35) - open/close 5598 (time 1775851219.12 total 310.26 last 11.02) - open/close 5744 (time 1775851231.50 total 322.63 last 11.80) - open/close 5869 (time 1775851242.42 total 333.55 last 11.44) - open/close 5907 (time 1775851255.08 total 346.21 last 3.00) - open/close 6078 (time 1775851266.05 total 357.19 last 15.58) - open/close 6189 (time 1775851276.28 total 367.41 last 10.85) - open/close 6488 (time 1775851286.97 total 378.10 last 27.98) - open/close 6595 (time 1775851297.60 total 388.73 last 10.06) - open/close 6735 (time 1775851307.68 total 398.81 last 13.90) - open/close 6875 (time 1775851318.36 total 409.50 last 13.10) - open/close 7069 (time 1775851328.37 total 419.50 last 19.38) - open/close 7270 (time 1775851338.81 total 429.94 last 19.26) - open/close 7524 (time 1775851349.19 total 440.33 last 24.45) - open/close 7757 (time 1775851359.36 total 450.50 last 22.91) - open/close 8075 (time 1775851369.38 total 460.51 last 31.74) - open/close 8278 (time 1775851379.40 total 470.53 last 20.27) - open/close 8428 (time 1775851390.32 total 481.45 last 13.73) - open/close 8550 (time 1775851400.45 total 491.58 last 12.05) - open/close 8648 (time 1775851411.02 total 502.15 last 9.27) - open/close 8896 (time 1775851422.47 total 513.60 last 21.66) - open/close 9135 (time 1775851432.76 total 523.89 last 23.22) - open/close 9278 (time 1775851443.27 total 534.41 last 13.60) - open/close 9497 (time 1775851453.29 total 544.43 last 21.86) - open/close 9709 (time 1775851463.31 total 554.44 last 21.16) - open/close 9943 (time 1775851473.34 total 564.47 last 23.34) - open/close 10000 (time 1775851483.01 total 574.14 last 5.89) - open/close 10141 (time 1775851493.32 total 584.46 last 13.67) - open/close 10332 (time 1775851503.34 total 594.48 last 19.07) - open/close 10397 (time 1775851513.40 total 604.53 last 6.46) - open/close 10486 (time 1775851524.13 total 615.27 last 8.29) - open/close 10738 (time 1775851534.14 total 625.27 last 25.20) - open/close 10910 (time 1775851544.78 total 635.91 last 16.17) - open/close 11031 (time 1775851554.80 total 645.93 last 12.07) - open/close 11200 (time 1775851565.07 total 656.20 last 16.45) - open/close 11331 (time 1775851575.09 total 666.22 last 13.07) - open/close 11488 (time 1775851585.10 total 676.23 last 15.69) - open/close 11617 (time 1775851595.10 total 686.23 last 12.90) - open/close 11762 (time 1775851605.10 total 696.24 last 14.50) - open/close 11876 (time 1775851616.08 total 707.21 last 10.39) - open/close 12048 (time 1775851626.19 total 717.32 last 17.01) - open/close 12243 (time 1775851636.22 total 727.35 last 19.44) - open/close 12372 (time 1775851646.41 total 737.54 last 12.66) - open/close 12576 (time 1775851657.24 total 748.37 last 18.83) - open/close 12771 (time 1775851667.24 total 758.37 last 19.50) - open/close 12872 (time 1775851679.89 total 771.02 last 7.99) - open/close 13027 (time 1775851690.55 total 781.69 last 14.53) - open/close 13199 (time 1775851700.62 total 791.76 last 17.08) - open/close 13350 (time 1775851711.36 total 802.50 last 14.06) - open/close 13504 (time 1775851723.00 total 814.13 last 13.24) - open/close 13756 (time 1775851733.00 total 824.13 last 25.19) - open/close 13866 (time 1775851743.07 total 834.20 last 10.93) - open/close 14100 (time 1775851753.08 total 844.21 last 23.36) - open/close 14272 (time 1775851765.94 total 857.07 last 13.38) - open/close 14348 (time 1775851775.95 total 867.09 last 7.59) - open/close 14560 (time 1775851785.96 total 877.09 last 21.18) - open/close 14660 (time 1775851796.60 total 887.74 last 9.40) - open/close 14893 (time 1775851806.62 total 897.75 last 23.26) - open/close 15102 (time 1775851817.52 total 908.65 last 19.18) - open/close 15228 (time 1775851827.58 total 918.71 last 12.52) - open/close 15444 (time 1775851837.59 total 928.73 last 21.57) - open/close 15670 (time 1775851848.51 total 939.64 last 20.70) - open/close 15780 (time 1775851858.62 total 949.75 last 10.88) - open/close 15945 (time 1775851868.65 total 959.78 last 16.46) - open/close 16092 (time 1775851878.91 total 970.05 last 14.32) - open/close 16290 (time 1775851888.91 total 980.05 last 19.80) - open/close 16413 (time 1775851899.28 total 990.41 last 11.87) - open/close 16548 (time 1775851910.35 total 1001.49 last 12.19) - open/close 16787 (time 1775851922.89 total 1014.02 last 19.07) - open/close 16878 (time 1775851932.99 total 1024.12 last 9.01) - open/close 17048 (time 1775851943.00 total 1034.13 last 16.99) - open/close 17297 (time 1775851953.59 total 1044.72 last 23.51) - open/close 17452 (time 1775851965.94 total 1057.07 last 12.55) - open/close 17597 (time 1775851976.09 total 1067.22 last 14.29) - open/close 17757 (time 1775851986.10 total 1077.24 last 15.97) - open/close 18322 (time 1775851996.12 total 1087.25 last 56.44) - open/close 18904 (time 1775852006.12 total 1097.25 last 58.17) - open/close 19664 (time 1775852016.13 total 1107.26 last 75.96) - open/close 20000 (time 1775852020.67 total 1111.81 last 73.90) - open/close 20718 (time 1775852030.68 total 1121.81 last 71.78) - open/close 21492 (time 1775852040.69 total 1131.82 last 77.33) - open/close 22170 (time 1775852050.72 total 1141.86 last 67.53) - open/close 22749 (time 1775852060.73 total 1151.86 last 57.88) - open/close 23330 (time 1775852070.73 total 1161.86 last 58.09) - open/close 23923 (time 1775852080.73 total 1171.86 last 59.29) - open/close 24647 (time 1775852090.74 total 1181.87 last 72.36) - open/close 25422 (time 1775852100.74 total 1191.87 last 77.50) - open/close 26108 (time 1775852110.74 total 1201.87 last 68.59) - open/close 26905 (time 1775852120.75 total 1211.88 last 79.63) - open/close 27681 (time 1775852130.75 total 1221.88 last 77.58) - open/close 28407 (time 1775852140.75 total 1231.89 last 72.59) - open/close 29182 (time 1775852150.76 total 1241.89 last 77.47) - open/close 30000 (time 1775852160.54 total 1251.68 last 83.59) - open/close 30773 (time 1775852170.56 total 1261.69 last 77.17) - open/close 31543 (time 1775852180.56 total 1271.70 last 76.99) - open/close 32430 (time 1775852190.57 total 1281.70 last 88.67) - open/close 33261 (time 1775852200.58 total 1291.71 last 83.01) - open/close 34009 (time 1775852210.58 total 1301.72 last 74.75) - open/close 34746 (time 1775852220.59 total 1311.72 last 73.66) - open/close 35475 (time 1775852230.59 total 1321.73 last 72.87) - open/close 36161 (time 1775852240.60 total 1331.73 last 68.58) - open/close 36697 (time 1775852250.60 total 1341.74 last 53.57) - open/close 37463 (time 1775852260.61 total 1351.74 last 76.54) - open/close 38239 (time 1775852270.63 total 1361.76 last 77.47) - open/close 38952 (time 1775852280.64 total 1371.77 last 71.23) - open/close 39702 (time 1775852290.65 total 1381.79 last 74.88) - open/close 40000 (time 1775852294.61 total 1385.74 last 75.36) - open/close 40731 (time 1775852304.61 total 1395.74 last 73.08) - open/close 41530 (time 1775852314.62 total 1405.75 last 79.85) - open/close 42167 (time 1775852324.62 total 1415.75 last 63.67) - open/close 42878 (time 1775852334.63 total 1425.77 last 71.00) - open/close 43509 (time 1775852344.65 total 1435.78 last 63.03) - open/close 44144 (time 1775852354.65 total 1445.78 last 63.50) - open/close 44823 (time 1775852364.65 total 1455.78 last 67.89) - open/close 45482 (time 1775852374.65 total 1465.78 last 65.89) - open/close 46137 (time 1775852384.65 total 1475.79 last 65.47) - open/close 46900 (time 1775852394.67 total 1485.80 last 76.21) - open/close 47579 (time 1775852404.67 total 1495.80 last 67.88) - open/close 48328 (time 1775852414.68 total 1505.81 last 74.84) - open/close 49053 (time 1775852424.69 total 1515.82 last 72.43) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1524.38 seconds: 32.56 ops/second - unlinked 0 (time 1775852439 ; total 1 ; last 1) - unlinked 10000 (time 1775852591 ; total 153 ; last 152) - unlinked 20000 (time 1775852737 ; total 299 ; last 146) - unlinked 30000 (time 1775852888 ; total 450 ; last 151) - unlinked 40000 (time 1775853039 ; total 601 ; last 151) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 735 seconds: 67.526527 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg227-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg227-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg227-server unloading modules via unload_modules_local on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing unload_modules_local oleg227-server: modules unloaded. pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-client: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-client: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2