== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:27:50 (1788276470) start mds service on oleg453-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg453-server' oleg453-server: oleg453-server.virtnet: executing load_modules_local oleg453-server: Loading modules from /home/green/git/lustre-release/lustre oleg453-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg453-server: detected 4 online CPUs by sysfs oleg453-server: MODOPTS_LIBCFS= oleg453-server: Force libcfs to create 2 CPU partitions oleg453-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg453-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg453-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg453-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg453-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-OST0000 oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg453-client.virtnet: -o user_xattr,flock 192.168.204.153@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 135 (time 1788276601.40 total 10.06 last 13.42) - open/close 236 (time 1788276611.41 total 20.07 last 10.09) - open/close 327 (time 1788276622.09 total 30.74 last 8.52) - open/close 492 (time 1788276632.44 total 41.09 last 15.94) - open/close 679 (time 1788276647.04 total 55.70 last 12.81) - open/close 780 (time 1788276657.87 total 66.52 last 9.33) - open/close 978 (time 1788276667.89 total 76.55 last 19.75) - open/close 1176 (time 1788276678.21 total 86.86 last 19.19) - open/close 1426 (time 1788276688.22 total 96.88 last 24.97) - open/close 1674 (time 1788276698.73 total 107.39 last 23.59) - open/close 1851 (time 1788276709.44 total 118.10 last 16.53) - open/close 2099 (time 1788276720.34 total 128.99 last 22.76) - open/close 2367 (time 1788276731.20 total 139.86 last 24.67) - open/close 2634 (time 1788276741.20 total 149.86 last 26.70) - open/close 2843 (time 1788276751.56 total 160.22 last 20.17) - open/close 3117 (time 1788276761.57 total 170.22 last 27.40) - open/close 3379 (time 1788276771.57 total 180.22 last 26.20) - open/close 3551 (time 1788276784.25 total 192.91 last 13.56) - open/close 3732 (time 1788276794.31 total 202.96 last 18.00) - open/close 3977 (time 1788276804.31 total 212.97 last 24.50) - open/close 4163 (time 1788276814.41 total 223.06 last 18.42) - open/close 4432 (time 1788276824.41 total 233.07 last 26.89) - open/close 4672 (time 1788276834.42 total 243.08 last 23.97) - open/close 4858 (time 1788276844.43 total 253.08 last 18.59) - open/close 5058 (time 1788276854.45 total 263.10 last 19.96) - open/close 5197 (time 1788276864.45 total 273.11 last 13.89) - open/close 5320 (time 1788276874.47 total 283.13 last 12.27) - open/close 5417 (time 1788276885.20 total 293.85 last 9.05) - open/close 5574 (time 1788276895.32 total 303.97 last 15.52) - open/close 5741 (time 1788276905.87 total 314.52 last 15.83) - open/close 5829 (time 1788276915.91 total 324.57 last 8.76) - open/close 5962 (time 1788276926.78 total 335.43 last 12.25) - open/close 6109 (time 1788276937.03 total 345.69 last 14.33) - open/close 6204 (time 1788276947.64 total 356.29 last 8.96) - open/close 6383 (time 1788276957.64 total 366.30 last 17.89) - open/close 6537 (time 1788276968.13 total 376.78 last 14.68) - open/close 6699 (time 1788276979.11 total 387.77 last 14.75) - open/close 6883 (time 1788276989.14 total 397.79 last 18.35) - open/close 7001 (time 1788276999.14 total 407.79 last 11.80) - open/close 7132 (time 1788277009.31 total 417.96 last 12.88) - open/close 7321 (time 1788277019.33 total 427.98 last 18.87) - open/close 7451 (time 1788277029.33 total 437.99 last 12.99) - open/close 7624 (time 1788277042.96 total 451.61 last 12.70) - open/close 7654 (time 1788277054.07 total 462.73 last 2.70) - open/close 7802 (time 1788277064.52 total 473.17 last 14.17) - open/close 7990 (time 1788277075.47 total 484.13 last 17.16) - open/close 8219 (time 1788277087.49 total 496.14 last 19.06) - open/close 8396 (time 1788277097.50 total 506.15 last 17.68) - open/close 8566 (time 1788277110.47 total 519.13 last 13.10) - open/close 8695 (time 1788277121.27 total 529.93 last 11.94) - open/close 8857 (time 1788277131.58 total 540.24 last 15.71) - open/close 8930 (time 1788277144.59 total 553.25 last 5.61) - open/close 9067 (time 1788277154.61 total 563.27 last 13.67) - open/close 9305 (time 1788277164.61 total 573.27 last 23.79) - open/close 9439 (time 1788277175.48 total 584.13 last 12.34) - open/close 9528 (time 1788277186.00 total 594.65 last 8.46) - open/close 9704 (time 1788277196.61 total 605.27 last 16.59) - open/close 9966 (time 1788277207.48 total 616.13 last 24.10) - open/close 10243 (time 1788277218.31 total 626.96 last 25.58) - open/close 10474 (time 1788277228.67 total 637.33 last 22.28) - open/close 10674 (time 1788277238.68 total 647.34 last 19.99) - open/close 10813 (time 1788277249.67 total 658.32 last 12.65) - open/close 10900 (time 1788277260.49 total 669.15 last 8.04) - open/close 10973 (time 1788277270.53 total 679.19 last 7.27) - open/close 11150 (time 1788277280.56 total 689.21 last 17.65) - open/close 11307 (time 1788277290.56 total 699.21 last 15.70) - open/close 11493 (time 1788277300.57 total 709.22 last 18.59) - open/close 11623 (time 1788277311.62 total 720.28 last 11.76) - open/close 11860 (time 1788277321.63 total 730.29 last 23.68) - open/close 12099 (time 1788277332.37 total 741.02 last 22.26) - open/close 12366 (time 1788277343.29 total 751.95 last 24.44) - open/close 12617 (time 1788277353.40 total 762.06 last 24.83) - open/close 12886 (time 1788277364.22 total 772.88 last 24.86) - open/close 13125 (time 1788277374.71 total 783.36 last 22.80) - open/close 13370 (time 1788277385.41 total 794.06 last 22.90) - open/close 13642 (time 1788277395.78 total 804.43 last 26.23) - open/close 13840 (time 1788277405.83 total 814.48 last 19.70) - open/close 14093 (time 1788277415.85 total 824.50 last 25.25) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-14225) error: No space left on device total: 14225 open/close in 829.17 seconds: 17.16 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9017: kill: (114812) - No such process - unlinked 0 (time 1788277427 ; total 0 ; last 0) - unlinked 10000 (time 1788277543 ; total 116 ; last 116) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-14225) error: No such file or directory total: 14225 unlinks in 152 seconds: 93.585526 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg453-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg453-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg453-server stop mds service on oleg453-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg453-server stop mds service on oleg453-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg453-server unloading modules via unload_modules_local on: 'oleg453-server' oleg453-server: oleg453-server.virtnet: executing unload_modules_local oleg453-server: modules unloaded. pdsh@oleg453-client: oleg453-client: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-client: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 2