== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:45:40 (1787553940) start mds service on oleg115-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg115-server' oleg115-server: oleg115-server.virtnet: executing load_modules_local oleg115-server: Loading modules from /home/green/git/lustre-release/lustre oleg115-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg115-server: detected 4 online CPUs by sysfs oleg115-server: MODOPTS_LIBCFS= oleg115-server: Force libcfs to create 2 CPU partitions oleg115-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg115-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg115-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg115-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg115-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-OST0000 oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg115-client.virtnet: -o user_xattr,flock 192.168.201.115@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 163 (time 1787554068.36 total 11.66 last 13.98) - open/close 336 (time 1787554078.37 total 21.67 last 17.28) - open/close 483 (time 1787554089.16 total 32.45 last 13.63) - open/close 660 (time 1787554099.39 total 42.69 last 17.29) - open/close 791 (time 1787554110.22 total 53.52 last 12.10) - open/close 866 (time 1787554121.74 total 65.03 last 6.51) - open/close 1129 (time 1787554131.75 total 75.05 last 26.26) - open/close 1257 (time 1787554142.76 total 86.06 last 11.63) - open/close 1412 (time 1787554152.76 total 96.06 last 15.49) - open/close 1545 (time 1787554165.48 total 108.78 last 10.46) - open/close 1711 (time 1787554176.01 total 119.31 last 15.75) - open/close 1850 (time 1787554186.95 total 130.25 last 12.71) - open/close 2043 (time 1787554196.96 total 140.26 last 19.29) - open/close 2120 (time 1787554207.58 total 150.88 last 7.25) - open/close 2187 (time 1787554217.59 total 160.89 last 6.69) - open/close 2350 (time 1787554227.59 total 170.89 last 16.30) - open/close 2433 (time 1787554237.90 total 181.20 last 8.05) - open/close 2544 (time 1787554247.92 total 191.21 last 11.08) - open/close 2691 (time 1787554258.89 total 202.19 last 13.39) - open/close 2773 (time 1787554269.45 total 212.74 last 7.77) - open/close 2912 (time 1787554280.48 total 223.77 last 12.60) - open/close 2999 (time 1787554290.88 total 234.18 last 8.36) - open/close 3052 (time 1787554301.52 total 244.82 last 4.98) - open/close 3221 (time 1787554311.53 total 254.82 last 16.89) - open/close 3277 (time 1787554322.22 total 265.52 last 5.24) - open/close 3414 (time 1787554332.48 total 275.78 last 13.36) - open/close 3533 (time 1787554342.48 total 285.78 last 11.90) - open/close 3737 (time 1787554352.51 total 295.81 last 20.34) - open/close 3862 (time 1787554362.52 total 305.81 last 12.49) - open/close 4000 (time 1787554372.75 total 316.05 last 13.48) - open/close 4220 (time 1787554383.84 total 327.14 last 19.85) - open/close 4429 (time 1787554393.85 total 337.14 last 20.88) - open/close 4683 (time 1787554403.98 total 347.28 last 25.06) - open/close 4955 (time 1787554413.99 total 357.28 last 27.19) - open/close 5045 (time 1787554424.05 total 367.35 last 8.94) - open/close 5225 (time 1787554434.36 total 377.66 last 17.46) - open/close 5403 (time 1787554445.39 total 388.68 last 16.15) - open/close 5627 (time 1787554455.40 total 398.70 last 22.37) - open/close 5825 (time 1787554465.40 total 408.70 last 19.79) - open/close 5922 (time 1787554476.17 total 419.47 last 9.01) - open/close 6016 (time 1787554486.98 total 430.28 last 8.69) - open/close 6178 (time 1787554497.79 total 441.09 last 14.99) - open/close 6283 (time 1787554511.24 total 454.54 last 7.81) - open/close 6484 (time 1787554523.97 total 467.27 last 15.79) - open/close 6687 (time 1787554534.29 total 477.59 last 19.66) - open/close 6845 (time 1787554544.85 total 488.14 last 14.97) - open/close 6979 (time 1787554554.88 total 498.18 last 13.35) - open/close 7153 (time 1787554565.53 total 508.83 last 16.34) - open/close 7385 (time 1787554576.46 total 519.76 last 21.23) - open/close 7437 (time 1787554587.11 total 530.40 last 4.89) - open/close 7572 (time 1787554597.11 total 540.41 last 13.49) - open/close 7839 (time 1787554608.01 total 551.31 last 24.49) - open/close 8073 (time 1787554618.50 total 561.80 last 22.31) - open/close 8311 (time 1787554628.50 total 571.80 last 23.80) - open/close 8628 (time 1787554638.85 total 582.15 last 30.64) - open/close 8903 (time 1787554648.87 total 592.17 last 27.45) - open/close 9089 (time 1787554659.08 total 602.38 last 18.21) - open/close 9348 (time 1787554672.20 total 615.50 last 19.75) - open/close 9464 (time 1787554682.33 total 625.63 last 11.45) - open/close 9643 (time 1787554692.67 total 635.97 last 17.31) - open/close 9788 (time 1787554703.63 total 646.93 last 13.23) - open/close 10000 (time 1787554710.67 total 653.97 last 30.11) - open/close 10148 (time 1787554721.56 total 664.86 last 13.60) - open/close 10367 (time 1787554731.59 total 674.89 last 21.83) - open/close 10406 (time 1787554742.08 total 685.38 last 3.72) - open/close 10444 (time 1787554752.37 total 695.67 last 3.69) - open/close 10610 (time 1787554762.41 total 705.71 last 16.55) - open/close 10701 (time 1787554772.73 total 716.03 last 8.81) - open/close 10814 (time 1787554782.75 total 726.04 last 11.29) - open/close 10925 (time 1787554793.73 total 737.02 last 10.11) - open/close 11046 (time 1787554803.79 total 747.08 last 12.03) - open/close 11188 (time 1787554816.19 total 759.49 last 11.44) - open/close 11397 (time 1787554826.75 total 770.04 last 19.81) - open/close 11487 (time 1787554836.82 total 780.12 last 8.93) - open/close 11612 (time 1787554848.86 total 792.16 last 10.38) - open/close 11798 (time 1787554858.87 total 802.17 last 18.58) - open/close 11875 (time 1787554869.00 total 812.30 last 7.60) - open/close 11984 (time 1787554879.96 total 823.26 last 9.94) - open/close 12086 (time 1787554890.51 total 833.80 last 9.68) - open/close 12186 (time 1787554901.06 total 844.36 last 9.47) - open/close 12258 (time 1787554911.07 total 854.36 last 7.20) - open/close 12429 (time 1787554921.07 total 864.37 last 17.09) - open/close 12586 (time 1787554931.46 total 874.76 last 15.11) - open/close 12689 (time 1787554941.47 total 884.77 last 10.29) - open/close 12746 (time 1787554951.49 total 894.79 last 5.69) - open/close 12929 (time 1787554962.00 total 905.30 last 17.40) - open/close 13045 (time 1787554973.02 total 916.32 last 10.53) - open/close 13295 (time 1787554983.02 total 926.32 last 25.00) - open/close 13528 (time 1787554993.02 total 936.32 last 23.29) - open/close 13655 (time 1787555003.04 total 946.34 last 12.68) - open/close 13746 (time 1787555013.32 total 956.61 last 8.85) - open/close 13935 (time 1787555024.27 total 967.57 last 17.25) - open/close 14095 (time 1787555035.11 total 978.41 last 14.76) - open/close 14344 (time 1787555045.11 total 988.41 last 24.90) - open/close 14421 (time 1787555056.13 total 999.43 last 6.99) - open/close 14594 (time 1787555066.38 total 1009.68 last 16.88) - open/close 14725 (time 1787555076.38 total 1019.68 last 13.10) - open/close 14876 (time 1787555087.04 total 1030.34 last 14.16) - open/close 14964 (time 1787555097.05 total 1040.35 last 8.80) - open/close 15015 (time 1787555107.87 total 1051.17 last 4.71) - open/close 15176 (time 1787555117.96 total 1061.26 last 15.96) - open/close 15273 (time 1787555128.70 total 1072.00 last 9.03) - open/close 15436 (time 1787555139.39 total 1082.69 last 15.25) - open/close 15529 (time 1787555149.92 total 1093.22 last 8.84) - open/close 15690 (time 1787555159.93 total 1103.22 last 16.09) - open/close 15764 (time 1787555169.93 total 1113.23 last 7.40) - open/close 16493 (time 1787555179.94 total 1123.23 last 72.85) - open/close 17179 (time 1787555189.94 total 1133.24 last 68.57) - open/close 17923 (time 1787555199.95 total 1143.24 last 74.34) - open/close 18705 (time 1787555209.97 total 1153.27 last 78.02) - open/close 19491 (time 1787555219.98 total 1163.28 last 78.49) - open/close 20000 (time 1787555227.21 total 1170.51 last 70.45) - open/close 20731 (time 1787555237.21 total 1180.51 last 73.07) - open/close 21553 (time 1787555247.22 total 1190.52 last 82.16) - open/close 22261 (time 1787555257.22 total 1200.52 last 70.78) - open/close 22978 (time 1787555267.24 total 1210.53 last 71.60) - open/close 23663 (time 1787555277.24 total 1220.54 last 68.44) - open/close 24467 (time 1787555287.24 total 1230.54 last 80.39) - open/close 25254 (time 1787555297.25 total 1240.55 last 78.67) - open/close 25921 (time 1787555307.26 total 1250.56 last 66.63) - open/close 26631 (time 1787555317.27 total 1260.57 last 70.92) - open/close 27476 (time 1787555327.27 total 1270.57 last 84.49) - open/close 28175 (time 1787555337.27 total 1280.57 last 69.89) - open/close 28907 (time 1787555347.27 total 1290.57 last 73.20) - open/close 29655 (time 1787555357.28 total 1300.58 last 74.71) - open/close 30000 (time 1787555361.11 total 1304.40 last 90.28) - open/close 30822 (time 1787555371.12 total 1314.42 last 82.08) - open/close 31625 (time 1787555381.13 total 1324.42 last 80.26) - open/close 32355 (time 1787555391.13 total 1334.43 last 72.97) - open/close 32972 (time 1787555401.13 total 1344.43 last 61.68) - open/close 33693 (time 1787555411.14 total 1354.44 last 72.05) - open/close 34458 (time 1787555421.15 total 1364.45 last 76.40) - open/close 35073 (time 1787555431.16 total 1374.46 last 61.46) - open/close 35693 (time 1787555441.18 total 1384.48 last 61.87) - open/close 36338 (time 1787555451.19 total 1394.48 last 64.46) - open/close 37052 (time 1787555461.20 total 1404.49 last 71.34) - open/close 37725 (time 1787555471.21 total 1414.50 last 67.22) - open/close 38446 (time 1787555481.21 total 1424.51 last 72.07) - open/close 38981 (time 1787555491.23 total 1434.53 last 53.37) - open/close 39497 (time 1787555501.25 total 1444.55 last 51.50) - open/close 40000 (time 1787555510.31 total 1453.61 last 55.56) - open/close 40723 (time 1787555520.32 total 1463.62 last 72.19) - open/close 41422 (time 1787555530.33 total 1473.63 last 69.82) - open/close 42147 (time 1787555540.34 total 1483.64 last 72.44) - open/close 42926 (time 1787555550.35 total 1493.65 last 77.85) - open/close 43669 (time 1787555560.35 total 1503.65 last 74.28) - open/close 44400 (time 1787555570.35 total 1513.65 last 73.09) - open/close 45097 (time 1787555580.36 total 1523.66 last 69.65) - open/close 45762 (time 1787555590.36 total 1533.66 last 66.47) - open/close 46534 (time 1787555600.36 total 1543.66 last 77.20) - open/close 47287 (time 1787555610.37 total 1553.67 last 75.27) - open/close 48042 (time 1787555620.38 total 1563.68 last 75.39) - open/close 48816 (time 1787555630.42 total 1573.71 last 77.14) - open/close 49562 (time 1787555640.43 total 1583.73 last 74.49) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1584.68 seconds: 31.32 ops/second - unlinked 0 (time 1787555646 ; total 0 ; last 0) - unlinked 10000 (time 1787555785 ; total 139 ; last 139) - unlinked 20000 (time 1787555927 ; total 281 ; last 142) - unlinked 30000 (time 1787556055 ; total 409 ; last 128) - unlinked 40000 (time 1787556184 ; total 538 ; last 129) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 660 seconds: 75.198486 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg115-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg115-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg115-server stop mds service on oleg115-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg115-server stop mds service on oleg115-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg115-server unloading modules via unload_modules_local on: 'oleg115-server' oleg115-server: oleg115-server.virtnet: executing unload_modules_local oleg115-server: modules unloaded. pdsh@oleg115-client: oleg115-server: ssh exited with exit code 2 pdsh@oleg115-client: oleg115-client: ssh exited with exit code 2 pdsh@oleg115-client: oleg115-client: ssh exited with exit code 2 pdsh@oleg115-client: oleg115-server: ssh exited with exit code 2