== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:51:05 (1788159065) start mds service on oleg234-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing load_modules_local oleg234-server: Loading modules from /home/green/git/lustre-release/lustre oleg234-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg234-server: detected 4 online CPUs by sysfs oleg234-server: MODOPTS_LIBCFS= oleg234-server: Force libcfs to create 2 CPU partitions oleg234-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg234-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg234-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg234-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg234-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 135 (time 1788159180.91 total 10.93 last 12.36) - open/close 350 (time 1788159190.97 total 20.99 last 21.36) - open/close 501 (time 1788159201.33 total 31.35 last 14.58) - open/close 674 (time 1788159212.13 total 42.15 last 16.02) - open/close 877 (time 1788159222.15 total 52.17 last 20.24) - open/close 1126 (time 1788159232.15 total 62.18 last 24.90) - open/close 1267 (time 1788159242.16 total 72.18 last 14.09) - open/close 1390 (time 1788159256.05 total 86.07 last 8.86) - open/close 1459 (time 1788159266.05 total 96.07 last 6.89) - open/close 1574 (time 1788159276.07 total 106.09 last 11.48) - open/close 1632 (time 1788159286.07 total 116.10 last 5.80) - open/close 1741 (time 1788159296.37 total 126.39 last 10.59) - open/close 1915 (time 1788159307.19 total 137.21 last 16.08) - open/close 2089 (time 1788159317.20 total 147.22 last 17.39) - open/close 2170 (time 1788159327.52 total 157.54 last 7.85) - open/close 2417 (time 1788159337.52 total 167.54 last 24.70) - open/close 2628 (time 1788159348.15 total 178.18 last 19.84) - open/close 2755 (time 1788159358.18 total 188.20 last 12.66) - open/close 2907 (time 1788159369.06 total 199.09 last 13.97) - open/close 3185 (time 1788159379.07 total 209.09 last 27.78) - open/close 3402 (time 1788159389.55 total 219.57 last 20.71) - open/close 3637 (time 1788159399.94 total 229.96 last 22.63) - open/close 3861 (time 1788159410.28 total 240.30 last 21.67) - open/close 3959 (time 1788159420.91 total 250.93 last 9.21) - open/close 4092 (time 1788159430.91 total 260.93 last 13.30) - open/close 4247 (time 1788159441.47 total 271.49 last 14.68) - open/close 4446 (time 1788159452.34 total 282.36 last 18.30) - open/close 4586 (time 1788159462.36 total 292.38 last 13.98) - open/close 4826 (time 1788159473.18 total 303.21 last 22.18) - open/close 4995 (time 1788159484.21 total 314.23 last 15.32) - open/close 5119 (time 1788159494.27 total 324.30 last 12.32) - open/close 5333 (time 1788159506.40 total 336.42 last 17.66) - open/close 5504 (time 1788159516.40 total 346.42 last 17.10) - open/close 5729 (time 1788159526.41 total 356.43 last 22.47) - open/close 5847 (time 1788159536.43 total 366.45 last 11.77) - open/close 5938 (time 1788159546.45 total 376.47 last 9.09) - open/close 6044 (time 1788159557.16 total 387.18 last 9.89) - open/close 6192 (time 1788159567.94 total 397.96 last 13.73) - open/close 6365 (time 1788159578.08 total 408.10 last 17.07) - open/close 6485 (time 1788159588.10 total 418.12 last 11.98) - open/close 6561 (time 1788159598.26 total 428.29 last 7.48) - open/close 6713 (time 1788159608.60 total 438.62 last 14.70) - open/close 6865 (time 1788159619.23 total 449.25 last 14.30) - open/close 7017 (time 1788159629.24 total 459.26 last 15.19) - open/close 7220 (time 1788159640.06 total 470.08 last 18.76) - open/close 7340 (time 1788159650.74 total 480.76 last 11.23) - open/close 7445 (time 1788159663.73 total 493.75 last 8.09) - open/close 7682 (time 1788159675.16 total 505.19 last 20.72) - open/close 7813 (time 1788159685.19 total 515.21 last 13.07) - open/close 8009 (time 1788159695.79 total 525.81 last 18.49) - open/close 8083 (time 1788159705.79 total 535.81 last 7.40) - open/close 8300 (time 1788159716.47 total 546.50 last 20.31) - open/close 8576 (time 1788159726.56 total 556.59 last 27.35) - open/close 8741 (time 1788159736.59 total 566.61 last 16.46) - open/close 8986 (time 1788159749.59 total 579.61 last 18.84) - open/close 9179 (time 1788159759.60 total 589.63 last 19.28) - open/close 9405 (time 1788159770.04 total 600.06 last 21.67) - open/close 9663 (time 1788159780.52 total 610.54 last 24.60) - open/close 9958 (time 1788159790.63 total 620.65 last 29.19) - open/close 10233 (time 1788159801.57 total 631.59 last 25.14) - open/close 10424 (time 1788159811.87 total 641.89 last 18.53) - open/close 10712 (time 1788159821.88 total 651.90 last 28.78) - open/close 10996 (time 1788159831.89 total 661.91 last 28.39) - open/close 11231 (time 1788159842.40 total 672.42 last 22.36) - open/close 11505 (time 1788159855.06 total 685.08 last 21.64) - open/close 11770 (time 1788159865.06 total 695.08 last 26.49) - open/close 12001 (time 1788159876.01 total 706.03 last 21.10) - open/close 12092 (time 1788159886.23 total 716.25 last 8.90) - open/close 12327 (time 1788159896.96 total 726.98 last 21.91) - open/close 12432 (time 1788159907.49 total 737.51 last 9.97) - open/close 12510 (time 1788159917.86 total 747.88 last 7.52) - open/close 12790 (time 1788159927.86 total 757.88 last 27.99) - open/close 13041 (time 1788159938.54 total 768.56 last 23.51) - open/close 13306 (time 1788159948.64 total 778.66 last 26.22) - open/close 13506 (time 1788159959.58 total 789.60 last 18.28) - open/close 13724 (time 1788159970.41 total 800.43 last 20.13) - open/close 13941 (time 1788159980.89 total 810.91 last 20.71) - open/close 14181 (time 1788159991.33 total 821.35 last 23.00) - open/close 14435 (time 1788160002.17 total 832.19 last 23.44) - open/close 14603 (time 1788160014.32 total 844.35 last 13.82) - open/close 14877 (time 1788160024.33 total 854.35 last 27.38) - open/close 15116 (time 1788160034.34 total 864.36 last 23.88) - open/close 15327 (time 1788160044.38 total 874.40 last 21.02) - open/close 15558 (time 1788160055.40 total 885.42 last 20.96) - open/close 15783 (time 1788160065.40 total 895.42 last 22.50) - open/close 16046 (time 1788160075.40 total 905.42 last 26.29) - open/close 16333 (time 1788160085.53 total 915.55 last 28.33) - open/close 16584 (time 1788160096.15 total 926.18 last 23.63) - open/close 16867 (time 1788160106.16 total 936.18 last 28.28) - open/close 17095 (time 1788160116.72 total 946.74 last 21.59) - open/close 17378 (time 1788160126.73 total 956.75 last 28.28) - open/close 17598 (time 1788160136.73 total 966.75 last 21.99) - open/close 17851 (time 1788160147.03 total 977.05 last 24.56) - open/close 18064 (time 1788160157.04 total 987.06 last 21.28) - open/close 18572 (time 1788160167.05 total 997.07 last 50.77) - open/close 19321 (time 1788160177.05 total 1007.07 last 74.85) - open/close 20000 (time 1788160183.93 total 1013.95 last 98.78) - open/close 20819 (time 1788160193.94 total 1023.96 last 81.80) - open/close 21633 (time 1788160203.95 total 1033.97 last 81.32) - open/close 22484 (time 1788160213.95 total 1043.97 last 85.07) - open/close 23388 (time 1788160223.96 total 1053.98 last 90.30) - open/close 24342 (time 1788160233.97 total 1063.99 last 95.36) - open/close 25045 (time 1788160243.97 total 1073.99 last 70.25) - open/close 25867 (time 1788160253.97 total 1083.99 last 82.20) - open/close 26716 (time 1788160263.98 total 1094.00 last 84.87) - open/close 27576 (time 1788160273.98 total 1104.00 last 85.95) - open/close 28436 (time 1788160283.99 total 1114.01 last 85.92) - open/close 29362 (time 1788160293.99 total 1124.01 last 92.59) - open/close 30000 (time 1788160301.64 total 1131.66 last 83.44) - open/close 30902 (time 1788160311.65 total 1141.67 last 90.15) - open/close 31680 (time 1788160321.66 total 1151.68 last 77.71) - open/close 32576 (time 1788160331.66 total 1161.69 last 89.54) - open/close 33302 (time 1788160341.68 total 1171.70 last 72.49) - open/close 34079 (time 1788160351.69 total 1181.71 last 77.61) - open/close 34852 (time 1788160361.69 total 1191.71 last 77.29) - open/close 35763 (time 1788160371.70 total 1201.72 last 91.05) - open/close 36543 (time 1788160381.71 total 1211.73 last 77.93) - open/close 37295 (time 1788160391.71 total 1221.73 last 75.17) - open/close 38037 (time 1788160401.72 total 1231.74 last 74.14) - open/close 38842 (time 1788160411.72 total 1241.74 last 80.50) - open/close 39633 (time 1788160421.72 total 1251.74 last 79.08) - open/close 40000 (time 1788160425.76 total 1255.78 last 90.89) - open/close 40870 (time 1788160435.77 total 1265.79 last 86.91) - open/close 41717 (time 1788160445.77 total 1275.80 last 84.66) - open/close 42622 (time 1788160455.78 total 1285.81 last 90.41) - open/close 43433 (time 1788160465.79 total 1295.81 last 81.02) - open/close 44282 (time 1788160475.80 total 1305.82 last 84.88) - open/close 45083 (time 1788160485.80 total 1315.82 last 80.06) - open/close 45951 (time 1788160495.80 total 1325.82 last 86.78) - open/close 46884 (time 1788160505.81 total 1335.83 last 93.24) - open/close 47702 (time 1788160515.81 total 1345.83 last 81.78) - open/close 48532 (time 1788160525.82 total 1355.84 last 82.96) - open/close 49266 (time 1788160535.82 total 1365.84 last 73.37) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1369.97 seconds: 36.23 ops/second - unlinked 0 (time 1788160544 ; total 0 ; last 0) - unlinked 10000 (time 1788160680 ; total 136 ; last 136) - unlinked 20000 (time 1788160816 ; total 272 ; last 136) - unlinked 30000 (time 1788160952 ; total 408 ; last 136) - unlinked 40000 (time 1788161076 ; total 532 ; last 124) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 651 seconds: 76.238098 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg234-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg234-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg234-server unloading modules via unload_modules_local on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing unload_modules_local oleg234-server: modules unloaded. pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2