== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:11:44 (1781665904) start mds service on oleg318-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions oleg318-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg318-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg318-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg318-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg318-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 354 (time 1781665956.02 total 10.49 last 33.74) - open/close 697 (time 1781665966.20 total 20.67 last 33.69) - open/close 1067 (time 1781665976.70 total 31.17 last 35.25) - open/close 1228 (time 1781665986.70 total 41.17 last 16.09) - open/close 1392 (time 1781665996.90 total 51.37 last 16.08) - open/close 1541 (time 1781666006.93 total 61.40 last 14.86) - open/close 1674 (time 1781666017.02 total 71.49 last 13.18) - open/close 1842 (time 1781666027.05 total 81.52 last 16.76) - open/close 1973 (time 1781666037.45 total 91.92 last 12.59) - open/close 2107 (time 1781666047.72 total 102.20 last 13.05) - open/close 2329 (time 1781666058.44 total 112.91 last 20.73) - open/close 2513 (time 1781666068.71 total 123.18 last 17.90) - open/close 2644 (time 1781666078.90 total 133.37 last 12.87) - open/close 2780 (time 1781666089.48 total 143.95 last 12.85) - open/close 2990 (time 1781666099.51 total 153.98 last 20.92) - open/close 3190 (time 1781666112.07 total 166.54 last 15.93) - open/close 3399 (time 1781666122.25 total 176.73 last 20.52) - open/close 3679 (time 1781666133.31 total 187.78 last 25.32) - open/close 3928 (time 1781666143.32 total 197.79 last 24.88) - open/close 4173 (time 1781666154.27 total 208.74 last 22.38) - open/close 4443 (time 1781666164.68 total 219.15 last 25.94) - open/close 4716 (time 1781666175.51 total 229.98 last 25.21) - open/close 4987 (time 1781666186.05 total 240.52 last 25.70) - open/close 5246 (time 1781666196.06 total 250.53 last 25.89) - open/close 5495 (time 1781666206.06 total 260.53 last 24.89) - open/close 5764 (time 1781666216.07 total 270.54 last 26.87) - open/close 6023 (time 1781666226.08 total 280.55 last 25.89) - open/close 6285 (time 1781666237.11 total 291.58 last 23.74) - open/close 6540 (time 1781666247.12 total 301.59 last 25.47) - open/close 6786 (time 1781666257.16 total 311.63 last 24.49) - open/close 7099 (time 1781666267.16 total 321.64 last 31.30) - open/close 7368 (time 1781666277.69 total 332.16 last 25.57) - open/close 7674 (time 1781666288.00 total 342.47 last 29.68) - open/close 7956 (time 1781666298.60 total 353.07 last 26.59) - open/close 8122 (time 1781666309.47 total 363.94 last 15.28) - open/close 8384 (time 1781666319.47 total 373.94 last 26.19) - open/close 8521 (time 1781666329.80 total 384.27 last 13.26) - open/close 8615 (time 1781666340.22 total 394.69 last 9.02) - open/close 8793 (time 1781666351.03 total 405.50 last 16.46) - open/close 8955 (time 1781666361.17 total 415.64 last 15.98) - open/close 9152 (time 1781666371.45 total 425.92 last 19.17) - open/close 9293 (time 1781666381.46 total 435.93 last 14.08) - open/close 9498 (time 1781666391.47 total 445.94 last 20.48) - open/close 9764 (time 1781666401.49 total 455.96 last 26.56) - open/close 9999 (time 1781666412.22 total 466.69 last 21.89) - open/close 10196 (time 1781666424.07 total 478.54 last 16.63) - open/close 10410 (time 1781666434.08 total 488.55 last 21.39) - open/close 10507 (time 1781666444.37 total 498.84 last 9.42) - open/close 10744 (time 1781666454.53 total 509.00 last 23.32) - open/close 10999 (time 1781666465.59 total 520.06 last 23.05) - open/close 11102 (time 1781666476.09 total 530.56 last 9.81) - open/close 11329 (time 1781666486.46 total 540.93 last 21.90) - open/close 11415 (time 1781666496.64 total 551.11 last 8.44) - open/close 11697 (time 1781666509.44 total 563.91 last 22.04) - open/close 12044 (time 1781666519.44 total 573.91 last 34.68) - open/close 12387 (time 1781666529.59 total 584.06 last 33.80) - open/close 12677 (time 1781666540.55 total 595.02 last 26.47) - open/close 12908 (time 1781666550.78 total 605.25 last 22.58) - open/close 13180 (time 1781666561.65 total 616.12 last 25.01) - open/close 13416 (time 1781666571.71 total 626.18 last 23.47) - open/close 13659 (time 1781666584.12 total 638.59 last 19.58) - open/close 13964 (time 1781666594.49 total 648.96 last 29.42) - open/close 14292 (time 1781666604.92 total 659.39 last 31.45) - open/close 14632 (time 1781666615.16 total 669.63 last 33.19) - open/close 14997 (time 1781666625.27 total 679.74 last 36.11) - open/close 15331 (time 1781666636.21 total 690.68 last 30.52) - open/close 15666 (time 1781666647.03 total 701.50 last 30.97) - open/close 15976 (time 1781666657.85 total 712.32 last 28.64) - open/close 16288 (time 1781666668.60 total 723.07 last 29.04) - open/close 16610 (time 1781666678.61 total 733.08 last 32.18) - open/close 16933 (time 1781666688.61 total 743.08 last 32.29) - open/close 18328 (time 1781666698.61 total 753.08 last 139.46) - open/close 20000 (time 1781666708.03 total 762.50 last 177.56) - open/close 22482 (time 1781666718.03 total 772.50 last 248.16) - open/close 25538 (time 1781666728.03 total 782.50 last 305.56) - open/close 28716 (time 1781666738.03 total 792.50 last 317.78) - open/close 30000 (time 1781666741.54 total 796.01 last 366.51) - open/close 33311 (time 1781666751.54 total 806.01 last 331.09) - open/close 35494 (time 1781666761.54 total 816.01 last 218.27) - open/close 36465 (time 1781666771.54 total 826.01 last 97.05) - open/close 37481 (time 1781666781.54 total 836.01 last 101.60) - open/close 38618 (time 1781666791.54 total 846.01 last 113.68) - open/close 39641 (time 1781666801.55 total 856.02 last 102.23) - open/close 40000 (time 1781666804.20 total 858.67 last 135.59) - open/close 41046 (time 1781666814.20 total 868.67 last 104.58) - open/close 42065 (time 1781666824.20 total 878.67 last 101.89) - open/close 43095 (time 1781666834.20 total 888.68 last 102.96) - open/close 44132 (time 1781666844.21 total 898.68 last 103.66) - open/close 45209 (time 1781666854.21 total 908.68 last 107.69) - open/close 46542 (time 1781666864.21 total 918.68 last 133.27) - open/close 47835 (time 1781666874.22 total 928.69 last 129.23) - open/close 49031 (time 1781666884.22 total 938.69 last 119.52) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 943.61 seconds: 52.60 ops/second - unlinked 0 (time 1781666891 ; total 0 ; last 0) - unlinked 10000 (time 1781666964 ; total 73 ; last 73) - unlinked 20000 (time 1781667003 ; total 112 ; last 39) - unlinked 30000 (time 1781667043 ; total 152 ; last 40) - unlinked 40000 (time 1781667085 ; total 194 ; last 42) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 229 seconds: 216.729263 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg318-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg318-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg318-server stop mds service on oleg318-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg318-server stop mds service on oleg318-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg318-server unloading modules via unload_modules_local on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing unload_modules_local oleg318-server: modules unloaded. pdsh@oleg318-client: oleg318-client: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-client: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 2