== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:28:52 (1782484132) start mds service on oleg425-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing load_modules_local oleg425-server: Loading modules from /home/green/git/lustre-release/lustre oleg425-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg425-server: detected 4 online CPUs by sysfs oleg425-server: MODOPTS_LIBCFS= oleg425-server: Force libcfs to create 2 CPU partitions oleg425-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg425-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg425-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg425-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg425-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-OST0000 oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 239 (time 1782484196.16 total 10.00 last 23.89) - open/close 506 (time 1782484206.18 total 20.02 last 26.65) - open/close 736 (time 1782484216.19 total 30.03 last 22.98) - open/close 957 (time 1782484226.20 total 40.04 last 22.08) - open/close 1031 (time 1782484236.21 total 50.05 last 7.39) - open/close 1386 (time 1782484246.55 total 60.39 last 34.33) - open/close 1614 (time 1782484256.66 total 70.50 last 22.53) - open/close 1916 (time 1782484266.90 total 80.74 last 29.51) - open/close 2234 (time 1782484277.14 total 90.97 last 31.06) - open/close 2551 (time 1782484287.44 total 101.28 last 30.75) - open/close 2801 (time 1782484298.06 total 111.90 last 23.55) - open/close 3190 (time 1782484309.07 total 122.91 last 35.34) - open/close 3469 (time 1782484319.44 total 133.28 last 26.89) - open/close 3792 (time 1782484331.30 total 145.13 last 27.25) - open/close 4010 (time 1782484341.53 total 155.37 last 21.29) - open/close 4346 (time 1782484351.54 total 165.38 last 33.57) - open/close 4460 (time 1782484361.69 total 175.53 last 11.24) - open/close 4630 (time 1782484371.69 total 185.53 last 17.00) - open/close 4830 (time 1782484382.68 total 196.52 last 18.20) - open/close 5005 (time 1782484393.37 total 207.21 last 16.37) - open/close 5148 (time 1782484403.48 total 217.32 last 14.14) - open/close 5327 (time 1782484416.31 total 230.15 last 13.96) - open/close 5545 (time 1782484426.33 total 240.17 last 21.75) - open/close 5749 (time 1782484436.63 total 250.47 last 19.81) - open/close 5875 (time 1782484446.63 total 260.47 last 12.59) - open/close 6045 (time 1782484456.93 total 270.77 last 16.51) - open/close 6337 (time 1782484466.94 total 280.78 last 29.17) - open/close 6576 (time 1782484477.66 total 291.50 last 22.28) - open/close 6849 (time 1782484487.67 total 301.51 last 27.28) - open/close 7090 (time 1782484497.77 total 311.60 last 23.87) - open/close 7315 (time 1782484508.50 total 322.34 last 20.96) - open/close 7545 (time 1782484519.38 total 333.22 last 21.14) - open/close 7787 (time 1782484530.40 total 344.23 last 21.97) - open/close 7945 (time 1782484540.40 total 354.24 last 15.79) - open/close 8263 (time 1782484550.42 total 364.26 last 31.75) - open/close 8520 (time 1782484560.98 total 374.82 last 24.33) - open/close 8721 (time 1782484571.61 total 385.45 last 18.91) - open/close 8952 (time 1782484582.24 total 396.07 last 21.74) - open/close 9203 (time 1782484593.05 total 406.88 last 23.22) - open/close 9419 (time 1782484603.30 total 417.14 last 21.06) - open/close 9646 (time 1782484613.73 total 427.57 last 21.78) - open/close 9868 (time 1782484624.15 total 437.99 last 21.29) - open/close 10000 (time 1782484629.26 total 443.10 last 25.84) - open/close 10202 (time 1782484639.45 total 453.29 last 19.83) - open/close 10441 (time 1782484650.08 total 463.92 last 22.47) - open/close 10684 (time 1782484660.88 total 474.72 last 22.50) - open/close 10912 (time 1782484671.51 total 485.35 last 21.46) - open/close 11116 (time 1782484681.90 total 495.74 last 19.64) - open/close 11360 (time 1782484692.26 total 506.10 last 23.55) - open/close 11565 (time 1782484702.26 total 516.10 last 20.49) - open/close 11735 (time 1782484712.72 total 526.56 last 16.25) - open/close 11899 (time 1782484722.73 total 536.57 last 16.38) - open/close 12041 (time 1782484733.59 total 547.43 last 13.08) - open/close 12236 (time 1782484743.77 total 557.61 last 19.16) - open/close 12460 (time 1782484753.93 total 567.77 last 22.03) - open/close 12809 (time 1782484764.75 total 578.59 last 32.26) - open/close 13079 (time 1782484775.63 total 589.47 last 24.83) - open/close 13165 (time 1782484786.00 total 599.84 last 8.29) - open/close 13281 (time 1782484797.73 total 611.57 last 9.89) - open/close 13509 (time 1782484810.26 total 624.10 last 18.20) - open/close 13688 (time 1782484820.64 total 634.48 last 17.26) - open/close 13870 (time 1782484830.67 total 644.51 last 18.13) - open/close 14117 (time 1782484840.68 total 654.52 last 24.68) - open/close 15012 (time 1782484850.68 total 664.52 last 89.48) - open/close 16533 (time 1782484860.68 total 674.52 last 152.05) - open/close 17817 (time 1782484870.69 total 684.52 last 128.38) - open/close 18999 (time 1782484880.69 total 694.53 last 118.13) - open/close 20000 (time 1782484888.39 total 702.23 last 129.96) - open/close 20822 (time 1782484898.40 total 712.24 last 82.13) - open/close 21519 (time 1782484908.40 total 722.24 last 69.70) - open/close 22160 (time 1782484918.42 total 732.25 last 64.02) - open/close 22925 (time 1782484928.42 total 742.26 last 76.48) - open/close 23590 (time 1782484938.44 total 752.28 last 66.38) - open/close 24117 (time 1782484948.44 total 762.28 last 52.67) - open/close 24771 (time 1782484958.45 total 772.29 last 65.37) - open/close 25600 (time 1782484968.45 total 782.29 last 82.87) - open/close 26366 (time 1782484978.45 total 792.29 last 76.60) - open/close 27226 (time 1782484988.45 total 802.29 last 85.99) - open/close 28087 (time 1782484998.46 total 812.30 last 86.06) - open/close 29073 (time 1782485008.46 total 822.30 last 98.59) - open/close 30000 (time 1782485017.97 total 831.81 last 97.45) - open/close 30913 (time 1782485027.97 total 841.81 last 91.29) - open/close 31789 (time 1782485037.98 total 851.82 last 87.52) - open/close 32680 (time 1782485047.99 total 861.83 last 89.00) - open/close 33604 (time 1782485057.99 total 871.83 last 92.40) - open/close 34517 (time 1782485068.00 total 881.84 last 91.24) - open/close 35482 (time 1782485078.01 total 891.85 last 96.43) - open/close 36395 (time 1782485088.02 total 901.86 last 91.21) - open/close 37329 (time 1782485098.02 total 911.86 last 93.40) - open/close 38269 (time 1782485108.03 total 921.87 last 93.88) - open/close 39164 (time 1782485118.03 total 931.87 last 89.49) - open/close 40000 (time 1782485126.22 total 940.06 last 102.06) - open/close 40965 (time 1782485136.22 total 950.06 last 96.49) - open/close 41901 (time 1782485146.23 total 960.07 last 93.56) - open/close 42884 (time 1782485156.23 total 970.07 last 98.28) - open/close 44018 (time 1782485166.23 total 980.07 last 113.39) - open/close 45140 (time 1782485176.23 total 990.07 last 112.17) - open/close 46192 (time 1782485186.24 total 1000.08 last 105.14) - open/close 47208 (time 1782485196.24 total 1010.08 last 101.53) - open/close 48198 (time 1782485206.26 total 1020.09 last 98.89) - open/close 49123 (time 1782485216.26 total 1030.10 last 92.45) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1035.26 seconds: 47.94 ops/second - unlinked 0 (time 1782485224 ; total 0 ; last 0) - unlinked 10000 (time 1782485325 ; total 101 ; last 101) - unlinked 20000 (time 1782485380 ; total 156 ; last 55) - unlinked 30000 (time 1782485430 ; total 206 ; last 50) - unlinked 40000 (time 1782485480 ; total 256 ; last 50) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 300 seconds: 165.436661 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg425-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg425-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg425-server stop mds service on oleg425-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg425-server stop mds service on oleg425-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg425-server unloading modules via unload_modules_local on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing unload_modules_local oleg425-server: modules unloaded. pdsh@oleg425-client: oleg425-client: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-server: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-client: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-server: ssh exited with exit code 2