== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:21:49 (1781241709) start mds service on oleg210-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing load_modules_local oleg210-server: Loading modules from /home/green/git/lustre-release/lustre oleg210-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg210-server: detected 4 online CPUs by sysfs oleg210-server: MODOPTS_LIBCFS= oleg210-server: Force libcfs to create 2 CPU partitions oleg210-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg210-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg210-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg210-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg210-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 204 (time 1781241813.34 total 10.08 last 20.23) - open/close 461 (time 1781241823.36 total 20.10 last 25.65) - open/close 602 (time 1781241833.39 total 30.13 last 14.06) - open/close 848 (time 1781241843.40 total 40.15 last 24.56) - open/close 1101 (time 1781241853.58 total 50.32 last 24.87) - open/close 1337 (time 1781241864.78 total 61.53 last 21.06) - open/close 1552 (time 1781241874.80 total 71.54 last 21.47) - open/close 1717 (time 1781241884.81 total 81.56 last 16.47) - open/close 1893 (time 1781241895.31 total 92.05 last 16.77) - open/close 2145 (time 1781241906.26 total 103.01 last 23.00) - open/close 2295 (time 1781241917.33 total 114.07 last 13.56) - open/close 2487 (time 1781241927.35 total 124.09 last 19.16) - open/close 2800 (time 1781241938.31 total 135.06 last 28.54) - open/close 2993 (time 1781241948.76 total 145.50 last 18.48) - open/close 3228 (time 1781241959.26 total 156.01 last 22.37) - open/close 3507 (time 1781241969.27 total 166.01 last 27.89) - open/close 3735 (time 1781241979.27 total 176.02 last 22.79) - open/close 3945 (time 1781241989.28 total 186.02 last 21.00) - open/close 4197 (time 1781242000.08 total 196.83 last 23.31) - open/close 4442 (time 1781242010.46 total 207.20 last 23.62) - open/close 4646 (time 1781242020.46 total 217.21 last 20.38) - open/close 4787 (time 1781242030.47 total 227.21 last 14.10) - open/close 5057 (time 1781242041.36 total 238.10 last 24.78) - open/close 5222 (time 1781242051.99 total 248.73 last 15.53) - open/close 5417 (time 1781242061.99 total 258.74 last 19.49) - open/close 5649 (time 1781242072.74 total 269.48 last 21.59) - open/close 5930 (time 1781242082.76 total 279.50 last 28.05) - open/close 6066 (time 1781242092.76 total 289.51 last 13.59) - open/close 6303 (time 1781242103.25 total 299.99 last 22.60) - open/close 6595 (time 1781242113.26 total 310.00 last 29.17) - open/close 6813 (time 1781242123.35 total 320.09 last 21.60) - open/close 7089 (time 1781242133.35 total 330.10 last 27.59) - open/close 7231 (time 1781242143.38 total 340.12 last 14.17) - open/close 7497 (time 1781242154.47 total 351.21 last 23.99) - open/close 7768 (time 1781242164.84 total 361.59 last 26.12) - open/close 7942 (time 1781242175.44 total 372.19 last 16.41) - open/close 8107 (time 1781242186.08 total 382.82 last 15.51) - open/close 8383 (time 1781242196.08 total 392.82 last 27.60) - open/close 8658 (time 1781242207.06 total 403.80 last 25.05) - open/close 8868 (time 1781242217.07 total 413.81 last 20.99) - open/close 9069 (time 1781242227.07 total 423.81 last 20.10) - open/close 9292 (time 1781242237.08 total 433.82 last 22.28) - open/close 9504 (time 1781242247.08 total 443.82 last 21.20) - open/close 9700 (time 1781242257.17 total 453.91 last 19.42) - open/close 9837 (time 1781242267.18 total 463.92 last 13.69) - open/close 9959 (time 1781242277.46 total 474.20 last 11.86) - open/close 10066 (time 1781242287.49 total 484.23 last 10.67) - open/close 10209 (time 1781242297.93 total 494.67 last 13.69) - open/close 10283 (time 1781242308.69 total 505.44 last 6.88) - open/close 10411 (time 1781242319.32 total 516.06 last 12.05) - open/close 10591 (time 1781242329.67 total 526.42 last 17.38) - open/close 10772 (time 1781242340.23 total 536.98 last 17.14) - open/close 10996 (time 1781242350.47 total 547.21 last 21.88) - open/close 11243 (time 1781242360.47 total 557.21 last 24.70) - open/close 11393 (time 1781242371.37 total 568.12 last 13.76) - open/close 11535 (time 1781242382.10 total 578.84 last 13.24) - open/close 11733 (time 1781242392.27 total 589.01 last 19.47) - open/close 11852 (time 1781242403.22 total 599.97 last 10.86) - open/close 12030 (time 1781242413.23 total 609.97 last 17.79) - open/close 12144 (time 1781242423.94 total 620.68 last 10.65) - open/close 12262 (time 1781242434.72 total 631.46 last 10.94) - open/close 12492 (time 1781242444.75 total 641.49 last 22.94) - open/close 12698 (time 1781242454.75 total 651.49 last 20.59) - open/close 12856 (time 1781242465.63 total 662.37 last 14.53) - open/close 13017 (time 1781242477.53 total 674.28 last 13.52) - open/close 13237 (time 1781242487.54 total 684.29 last 21.97) - open/close 13421 (time 1781242497.56 total 694.30 last 18.37) - open/close 13640 (time 1781242508.11 total 704.85 last 20.77) - open/close 13915 (time 1781242518.11 total 714.85 last 27.49) - open/close 14111 (time 1781242528.11 total 724.85 last 19.60) - open/close 14336 (time 1781242538.19 total 734.93 last 22.32) - open/close 14444 (time 1781242548.20 total 744.95 last 10.79) - open/close 14653 (time 1781242558.21 total 754.96 last 20.88) - open/close 14777 (time 1781242569.37 total 766.12 last 11.11) - open/close 14887 (time 1781242580.11 total 776.86 last 10.24) - open/close 15062 (time 1781242590.13 total 786.88 last 17.47) - open/close 15342 (time 1781242601.03 total 797.78 last 25.68) - open/close 15638 (time 1781242611.88 total 808.63 last 27.28) - open/close 15905 (time 1781242621.91 total 818.65 last 26.64) - open/close 16161 (time 1781242632.66 total 829.41 last 23.80) - open/close 16404 (time 1781242643.02 total 839.77 last 23.46) - open/close 16655 (time 1781242653.67 total 850.42 last 23.57) - open/close 16852 (time 1781242663.76 total 860.50 last 19.54) - open/close 17080 (time 1781242674.41 total 871.16 last 21.39) - open/close 17276 (time 1781242684.43 total 881.18 last 19.56) - open/close 17332 (time 1781242695.06 total 891.80 last 5.27) - open/close 17468 (time 1781242705.37 total 902.11 last 13.19) - open/close 17659 (time 1781242715.39 total 912.13 last 19.07) - open/close 17777 (time 1781242726.22 total 922.96 last 10.89) - open/close 18016 (time 1781242736.24 total 932.98 last 23.86) - open/close 18148 (time 1781242746.25 total 942.99 last 13.18) - open/close 18342 (time 1781242756.40 total 953.14 last 19.12) - open/close 18499 (time 1781242766.87 total 963.61 last 14.99) - open/close 19155 (time 1781242776.87 total 973.61 last 65.59) - open/close 19973 (time 1781242786.88 total 983.62 last 81.74) - open/close 20828 (time 1781242796.88 total 993.62 last 85.50) - open/close 21614 (time 1781242806.89 total 1003.64 last 78.48) - open/close 22440 (time 1781242816.90 total 1013.64 last 82.59) - open/close 23186 (time 1781242826.90 total 1023.64 last 74.58) - open/close 23896 (time 1781242836.91 total 1033.66 last 70.90) - open/close 24729 (time 1781242846.91 total 1043.66 last 83.30) - open/close 25630 (time 1781242856.91 total 1053.66 last 90.07) - open/close 26515 (time 1781242866.95 total 1063.69 last 88.22) - open/close 27281 (time 1781242876.96 total 1073.70 last 76.51) - open/close 28014 (time 1781242886.96 total 1083.71 last 73.26) - open/close 28785 (time 1781242896.98 total 1093.73 last 76.94) - open/close 29566 (time 1781242906.99 total 1103.73 last 78.08) - open/close 30000 (time 1781242913.81 total 1110.56 last 63.57) - open/close 30841 (time 1781242923.81 total 1120.56 last 84.10) - open/close 31635 (time 1781242933.82 total 1130.56 last 79.39) - open/close 32424 (time 1781242943.82 total 1140.56 last 78.87) - open/close 33287 (time 1781242953.82 total 1150.56 last 86.30) - open/close 34156 (time 1781242963.84 total 1160.58 last 86.70) - open/close 34880 (time 1781242973.85 total 1170.59 last 72.36) - open/close 35662 (time 1781242983.85 total 1180.60 last 78.16) - open/close 36416 (time 1781242993.86 total 1190.61 last 75.33) - open/close 37142 (time 1781243003.86 total 1200.61 last 72.59) - open/close 37969 (time 1781243013.87 total 1210.61 last 82.64) - open/close 38780 (time 1781243023.88 total 1220.62 last 81.06) - open/close 39587 (time 1781243033.88 total 1230.62 last 80.67) - open/close 40000 (time 1781243038.92 total 1235.66 last 81.92) - open/close 40738 (time 1781243048.95 total 1245.69 last 73.62) - open/close 41498 (time 1781243058.96 total 1255.70 last 75.88) - open/close 42344 (time 1781243068.96 total 1265.70 last 84.60) - open/close 43260 (time 1781243078.97 total 1275.71 last 91.55) - open/close 44160 (time 1781243088.97 total 1285.72 last 89.94) - open/close 44960 (time 1781243098.99 total 1295.73 last 79.90) - open/close 45788 (time 1781243108.99 total 1305.73 last 82.76) - open/close 46626 (time 1781243119.00 total 1315.74 last 83.71) - open/close 47441 (time 1781243129.00 total 1325.75 last 81.49) - open/close 48244 (time 1781243139.00 total 1335.75 last 80.28) - open/close 49076 (time 1781243149.01 total 1345.75 last 83.19) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1352.00 seconds: 36.71 ops/second - unlinked 0 (time 1781243158 ; total 0 ; last 0) - unlinked 10000 (time 1781243266 ; total 108 ; last 108) - unlinked 20000 (time 1781243358 ; total 200 ; last 92) - unlinked 30000 (time 1781243419 ; total 261 ; last 61) - unlinked 40000 (time 1781243479 ; total 321 ; last 60) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 374 seconds: 132.703201 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg210-server unloading modules via unload_modules_local on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing unload_modules_local oleg210-server: modules unloaded. pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2