== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:00:34 (1784588434) start mds service on oleg335-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing load_modules_local oleg335-server: Loading modules from /home/green/git/lustre-release/lustre oleg335-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg335-server: detected 4 online CPUs by sysfs oleg335-server: MODOPTS_LIBCFS= oleg335-server: Force libcfs to create 2 CPU partitions oleg335-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg335-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg335-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg335-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg335-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 208 (time 1784588533.07 total 10.40 last 20.00) - open/close 317 (time 1784588543.56 total 20.89 last 10.39) - open/close 414 (time 1784588554.16 total 31.50 last 9.15) - open/close 674 (time 1784588564.17 total 41.50 last 25.99) - open/close 948 (time 1784588574.18 total 51.51 last 27.37) - open/close 1244 (time 1784588585.15 total 62.48 last 26.98) - open/close 1534 (time 1784588595.39 total 72.73 last 28.30) - open/close 1782 (time 1784588605.39 total 82.73 last 24.80) - open/close 2025 (time 1784588616.06 total 93.39 last 22.78) - open/close 2247 (time 1784588626.07 total 103.40 last 22.18) - open/close 2431 (time 1784588636.94 total 114.28 last 16.92) - open/close 2672 (time 1784588646.95 total 124.28 last 24.09) - open/close 2926 (time 1784588656.96 total 134.30 last 25.36) - open/close 3021 (time 1784588667.40 total 144.74 last 9.10) - open/close 3183 (time 1784588677.44 total 154.78 last 16.14) - open/close 3384 (time 1784588687.49 total 164.82 last 20.01) - open/close 3593 (time 1784588697.78 total 175.11 last 20.30) - open/close 3850 (time 1784588708.54 total 185.87 last 23.89) - open/close 4066 (time 1784588718.84 total 196.18 last 20.97) - open/close 4314 (time 1784588728.85 total 206.18 last 24.79) - open/close 4522 (time 1784588738.86 total 216.20 last 20.77) - open/close 4816 (time 1784588749.55 total 226.89 last 27.50) - open/close 5069 (time 1784588759.56 total 236.89 last 25.28) - open/close 5340 (time 1784588770.36 total 247.69 last 25.10) - open/close 5587 (time 1784588780.61 total 257.94 last 24.09) - open/close 5838 (time 1784588790.62 total 267.96 last 25.06) - open/close 6049 (time 1784588800.63 total 277.96 last 21.09) - open/close 6304 (time 1784588811.19 total 288.52 last 24.15) - open/close 6544 (time 1784588821.76 total 299.09 last 22.71) - open/close 6741 (time 1784588831.76 total 309.09 last 19.70) - open/close 6903 (time 1784588841.87 total 319.20 last 16.03) - open/close 7097 (time 1784588852.34 total 329.68 last 18.52) - open/close 7367 (time 1784588862.35 total 339.68 last 27.00) - open/close 7595 (time 1784588872.77 total 350.10 last 21.87) - open/close 7820 (time 1784588882.77 total 360.11 last 22.50) - open/close 8134 (time 1784588893.81 total 371.14 last 28.45) - open/close 8422 (time 1784588904.64 total 381.97 last 26.60) - open/close 8612 (time 1784588915.33 total 392.66 last 17.77) - open/close 8814 (time 1784588925.51 total 402.84 last 19.84) - open/close 9023 (time 1784588936.45 total 413.78 last 19.10) - open/close 9281 (time 1784588947.19 total 424.52 last 24.02) - open/close 9509 (time 1784588957.19 total 434.53 last 22.80) - open/close 9785 (time 1784588967.20 total 444.53 last 27.59) - open/close 10000 (time 1784588976.58 total 453.91 last 22.91) - open/close 10259 (time 1784588986.75 total 464.09 last 25.46) - open/close 10346 (time 1784588997.37 total 474.70 last 8.20) - open/close 10614 (time 1784589007.62 total 484.95 last 26.15) - open/close 10850 (time 1784589017.78 total 495.11 last 23.23) - open/close 10932 (time 1784589028.79 total 506.13 last 7.44) - open/close 11038 (time 1784589038.82 total 516.15 last 10.58) - open/close 11310 (time 1784589049.72 total 527.05 last 24.96) - open/close 11609 (time 1784589059.72 total 537.05 last 29.89) - open/close 11885 (time 1784589070.13 total 547.47 last 26.50) - open/close 12128 (time 1784589080.17 total 557.50 last 24.21) - open/close 12354 (time 1784589090.30 total 567.63 last 22.31) - open/close 12636 (time 1784589101.25 total 578.58 last 25.76) - open/close 12764 (time 1784589112.19 total 589.53 last 11.69) - open/close 12987 (time 1784589122.59 total 599.92 last 21.46) - open/close 13212 (time 1784589132.59 total 609.92 last 22.50) - open/close 13496 (time 1784589142.66 total 619.99 last 28.20) - open/close 13777 (time 1784589153.21 total 630.54 last 26.63) - open/close 14034 (time 1784589163.74 total 641.08 last 24.40) - open/close 14292 (time 1784589174.79 total 652.12 last 23.36) - open/close 14569 (time 1784589185.42 total 662.76 last 26.04) - open/close 14825 (time 1784589196.04 total 673.37 last 24.12) - open/close 15125 (time 1784589206.06 total 683.39 last 29.93) - open/close 15355 (time 1784589217.03 total 694.36 last 20.97) - open/close 15582 (time 1784589227.04 total 704.37 last 22.67) - open/close 15878 (time 1784589237.05 total 714.39 last 29.56) - open/close 16130 (time 1784589247.43 total 724.76 last 24.29) - open/close 16366 (time 1784589257.44 total 734.77 last 23.57) - open/close 16553 (time 1784589267.70 total 745.03 last 18.23) - open/close 16736 (time 1784589277.89 total 755.22 last 17.96) - open/close 16933 (time 1784589287.90 total 765.23 last 19.68) - open/close 17018 (time 1784589298.30 total 775.64 last 8.17) - open/close 17251 (time 1784589308.32 total 785.65 last 23.27) - open/close 17453 (time 1784589318.99 total 796.32 last 18.93) - open/close 17753 (time 1784589328.99 total 806.32 last 30.00) - open/close 17970 (time 1784589339.91 total 817.24 last 19.88) - open/close 18241 (time 1784589349.92 total 827.25 last 27.08) - open/close 18514 (time 1784589359.93 total 837.26 last 27.27) - open/close 18810 (time 1784589370.68 total 848.01 last 27.53) - open/close 19066 (time 1784589380.69 total 858.02 last 25.59) - open/close 20000 (time 1784589390.66 total 867.99 last 93.63) - open/close 20814 (time 1784589400.67 total 878.00 last 81.36) - open/close 21524 (time 1784589410.67 total 888.00 last 70.98) - open/close 22236 (time 1784589420.71 total 898.05 last 70.89) - open/close 23223 (time 1784589430.72 total 908.05 last 98.66) - open/close 24162 (time 1784589440.72 total 918.05 last 93.88) - open/close 25111 (time 1784589450.72 total 928.05 last 94.88) - open/close 26051 (time 1784589460.73 total 938.06 last 93.91) - open/close 26912 (time 1784589470.74 total 948.07 last 86.05) - open/close 27796 (time 1784589480.74 total 958.07 last 88.36) - open/close 28684 (time 1784589490.74 total 968.07 last 88.79) - open/close 29596 (time 1784589500.75 total 978.08 last 91.14) - open/close 30000 (time 1784589505.41 total 982.74 last 86.73) - open/close 30893 (time 1784589515.41 total 992.74 last 89.27) - open/close 31786 (time 1784589525.41 total 1002.74 last 89.29) - open/close 32685 (time 1784589535.43 total 1012.76 last 89.77) - open/close 33571 (time 1784589545.44 total 1022.77 last 88.51) - open/close 34422 (time 1784589555.44 total 1032.77 last 85.07) - open/close 35072 (time 1784589565.45 total 1042.78 last 64.96) - open/close 35817 (time 1784589575.46 total 1052.79 last 74.40) - open/close 36781 (time 1784589585.46 total 1062.80 last 96.36) - open/close 37685 (time 1784589595.46 total 1072.80 last 90.40) - open/close 38435 (time 1784589605.46 total 1082.80 last 75.00) - open/close 39310 (time 1784589615.47 total 1092.80 last 87.49) - open/close 40000 (time 1784589624.20 total 1101.54 last 78.97) - open/close 40639 (time 1784589634.21 total 1111.54 last 63.88) - open/close 41408 (time 1784589644.22 total 1121.55 last 76.83) - open/close 42291 (time 1784589654.22 total 1131.56 last 88.23) - open/close 43183 (time 1784589664.23 total 1141.57 last 89.11) - open/close 44064 (time 1784589674.24 total 1151.57 last 88.07) - open/close 44941 (time 1784589684.25 total 1161.58 last 87.59) - open/close 45866 (time 1784589694.25 total 1171.59 last 92.48) - open/close 46739 (time 1784589704.26 total 1181.59 last 87.28) - open/close 47650 (time 1784589714.26 total 1191.59 last 91.07) - open/close 48482 (time 1784589724.26 total 1201.60 last 83.17) - open/close 49382 (time 1784589734.27 total 1211.60 last 89.94) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1214.75 seconds: 40.86 ops/second - unlinked 0 (time 1784589741 ; total 0 ; last 0) - unlinked 10000 (time 1784589869 ; total 128 ; last 128) - unlinked 20000 (time 1784589998 ; total 257 ; last 129) - unlinked 30000 (time 1784590122 ; total 381 ; last 124) - unlinked 40000 (time 1784590250 ; total 509 ; last 128) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 629 seconds: 78.904610 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg335-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server stop mds service on oleg335-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg335-server stop mds service on oleg335-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg335-server unloading modules via unload_modules_local on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing unload_modules_local oleg335-server: modules unloaded. pdsh@oleg335-client: oleg335-client: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-client: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2