== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:23:20 (1784942600) start mds service on oleg315-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing load_modules_local oleg315-server: Loading modules from /home/green/git/lustre-release/lustre oleg315-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg315-server: detected 4 online CPUs by sysfs oleg315-server: MODOPTS_LIBCFS= oleg315-server: Force libcfs to create 2 CPU partitions oleg315-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg315-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg315-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg315-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg315-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0000 oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 169 (time 1784942666.51 total 11.04 last 15.31) - open/close 371 (time 1784942676.75 total 21.28 last 19.72) - open/close 695 (time 1784942686.76 total 31.29 last 32.37) - open/close 918 (time 1784942697.05 total 41.57 last 21.68) - open/close 1176 (time 1784942707.41 total 51.93 last 24.90) - open/close 1427 (time 1784942717.41 total 61.93 last 25.10) - open/close 1653 (time 1784942728.14 total 72.67 last 21.06) - open/close 1882 (time 1784942739.08 total 83.61 last 20.92) - open/close 2088 (time 1784942749.64 total 94.17 last 19.51) - open/close 2409 (time 1784942760.14 total 104.67 last 30.58) - open/close 2729 (time 1784942771.09 total 115.62 last 29.22) - open/close 3065 (time 1784942782.03 total 126.56 last 30.73) - open/close 3371 (time 1784942792.84 total 137.37 last 28.30) - open/close 3670 (time 1784942803.34 total 147.87 last 28.48) - open/close 3934 (time 1784942813.34 total 157.87 last 26.39) - open/close 4144 (time 1784942824.21 total 168.73 last 19.33) - open/close 4398 (time 1784942834.51 total 179.04 last 24.65) - open/close 4666 (time 1784942844.75 total 189.28 last 26.16) - open/close 5002 (time 1784942855.56 total 200.09 last 31.08) - open/close 5273 (time 1784942866.46 total 210.99 last 24.87) - open/close 5411 (time 1784942876.46 total 220.99 last 13.80) - open/close 5529 (time 1784942886.46 total 230.99 last 11.80) - open/close 5645 (time 1784942896.47 total 240.99 last 11.60) - open/close 5809 (time 1784942906.58 total 251.11 last 16.21) - open/close 5895 (time 1784942917.29 total 261.82 last 8.03) - open/close 6032 (time 1784942928.29 total 272.82 last 12.46) - open/close 6193 (time 1784942939.10 total 283.62 last 14.90) - open/close 6378 (time 1784942949.50 total 294.03 last 17.79) - open/close 6511 (time 1784942959.51 total 304.03 last 13.29) - open/close 6724 (time 1784942969.52 total 314.04 last 21.28) - open/close 6964 (time 1784942979.93 total 324.46 last 23.05) - open/close 7214 (time 1784942990.29 total 334.82 last 24.13) - open/close 7411 (time 1784943001.23 total 345.76 last 18.00) - open/close 7645 (time 1784943011.99 total 356.51 last 21.76) - open/close 7913 (time 1784943021.99 total 366.52 last 26.79) - open/close 8129 (time 1784943032.40 total 376.93 last 20.74) - open/close 8364 (time 1784943042.40 total 386.93 last 23.50) - open/close 8575 (time 1784943052.95 total 397.48 last 20.01) - open/close 8809 (time 1784943063.58 total 408.10 last 22.02) - open/close 9077 (time 1784943074.20 total 418.72 last 25.24) - open/close 9296 (time 1784943084.96 total 429.49 last 20.35) - open/close 9519 (time 1784943095.82 total 440.35 last 20.53) - open/close 9749 (time 1784943105.83 total 450.36 last 22.98) - open/close 9990 (time 1784943116.69 total 461.22 last 22.19) - open/close 10269 (time 1784943126.70 total 471.22 last 27.90) - open/close 10529 (time 1784943137.56 total 482.09 last 23.92) - open/close 10782 (time 1784943147.57 total 492.10 last 25.29) - open/close 11039 (time 1784943157.57 total 502.10 last 25.69) - open/close 11268 (time 1784943168.23 total 512.75 last 21.50) - open/close 11509 (time 1784943178.32 total 522.85 last 23.87) - open/close 11756 (time 1784943188.33 total 532.86 last 24.67) - open/close 11980 (time 1784943198.33 total 542.86 last 22.40) - open/close 12194 (time 1784943208.35 total 552.88 last 21.36) - open/close 12422 (time 1784943218.83 total 563.35 last 21.76) - open/close 12706 (time 1784943228.85 total 573.38 last 28.33) - open/close 12943 (time 1784943239.12 total 583.65 last 23.08) - open/close 13194 (time 1784943249.13 total 593.66 last 25.07) - open/close 13421 (time 1784943259.14 total 603.67 last 22.68) - open/close 13657 (time 1784943269.15 total 613.67 last 23.59) - open/close 13886 (time 1784943280.03 total 624.55 last 21.05) - open/close 14132 (time 1784943290.09 total 634.62 last 24.44) - open/close 14351 (time 1784943300.70 total 645.23 last 20.64) - open/close 14579 (time 1784943311.19 total 655.71 last 21.75) - open/close 14814 (time 1784943321.20 total 665.73 last 23.47) - open/close 15595 (time 1784943331.22 total 675.75 last 77.93) - open/close 16306 (time 1784943341.22 total 685.75 last 71.09) - open/close 17075 (time 1784943351.22 total 695.75 last 76.87) - open/close 17794 (time 1784943361.23 total 705.76 last 71.86) - open/close 18506 (time 1784943371.23 total 715.76 last 71.17) - open/close 19215 (time 1784943381.24 total 725.76 last 70.88) - open/close 19918 (time 1784943391.24 total 735.77 last 70.26) - open/close 20642 (time 1784943401.25 total 745.78 last 72.36) - open/close 21408 (time 1784943411.26 total 755.78 last 76.54) - open/close 22159 (time 1784943421.26 total 765.79 last 75.05) - open/close 22827 (time 1784943431.27 total 775.80 last 66.74) - open/close 23566 (time 1784943441.28 total 785.81 last 73.85) - open/close 24290 (time 1784943451.29 total 795.82 last 72.35) - open/close 24999 (time 1784943461.30 total 805.83 last 70.83) - open/close 25704 (time 1784943471.31 total 815.83 last 70.44) - open/close 26444 (time 1784943481.31 total 825.84 last 73.97) - open/close 27178 (time 1784943491.31 total 835.84 last 73.38) - open/close 27957 (time 1784943501.32 total 845.84 last 77.87) - open/close 28734 (time 1784943511.32 total 855.85 last 77.67) - open/close 29489 (time 1784943521.32 total 865.85 last 75.49) - open/close 30000 (time 1784943527.70 total 872.23 last 80.15) - open/close 30797 (time 1784943537.71 total 882.24 last 79.62) - open/close 31543 (time 1784943547.71 total 892.24 last 74.55) - open/close 32287 (time 1784943557.73 total 902.25 last 74.32) - open/close 33123 (time 1784943567.73 total 912.26 last 83.59) - open/close 33932 (time 1784943577.74 total 922.27 last 80.78) - open/close 34726 (time 1784943587.75 total 932.28 last 79.33) - open/close 35520 (time 1784943597.75 total 942.28 last 79.38) - open/close 36338 (time 1784943607.76 total 952.28 last 81.79) - open/close 37161 (time 1784943617.76 total 962.29 last 82.23) - open/close 37921 (time 1784943627.77 total 972.29 last 75.98) - open/close 38731 (time 1784943637.77 total 982.30 last 80.93) - open/close 39623 (time 1784943647.79 total 992.32 last 89.02) - open/close 40000 (time 1784943652.54 total 997.06 last 79.52) - open/close 40846 (time 1784943662.55 total 1007.07 last 84.52) - open/close 41620 (time 1784943672.55 total 1017.07 last 77.40) - open/close 42406 (time 1784943682.55 total 1027.08 last 78.54) - open/close 43153 (time 1784943692.56 total 1037.08 last 74.68) - open/close 43900 (time 1784943702.56 total 1047.09 last 74.65) - open/close 44717 (time 1784943712.57 total 1057.10 last 81.62) - open/close 45530 (time 1784943722.58 total 1067.10 last 81.28) - open/close 46246 (time 1784943732.58 total 1077.11 last 71.55) - open/close 47033 (time 1784943742.59 total 1087.12 last 78.67) - open/close 47846 (time 1784943752.59 total 1097.12 last 81.30) - open/close 48593 (time 1784943762.59 total 1107.12 last 74.69) - open/close 49373 (time 1784943772.61 total 1117.13 last 77.88) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1120.74 seconds: 44.28 ops/second - unlinked 0 (time 1784943780 ; total 1 ; last 1) - unlinked 10000 (time 1784943900 ; total 121 ; last 120) - unlinked 20000 (time 1784944012 ; total 233 ; last 112) - unlinked 30000 (time 1784944101 ; total 322 ; last 89) - unlinked 40000 (time 1784944160 ; total 381 ; last 59) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 432 seconds: 114.886574 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg315-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg315-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg315-server unloading modules via unload_modules_local on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing unload_modules_local oleg315-server: modules unloaded. pdsh@oleg315-client: oleg315-server: ssh exited with exit code 2 pdsh@oleg315-client: oleg315-client: ssh exited with exit code 2 pdsh@oleg315-client: oleg315-client: ssh exited with exit code 2 pdsh@oleg315-client: oleg315-server: ssh exited with exit code 2