== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:30:34 (1780335034) start mds service on oleg359-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg359-server' oleg359-server: oleg359-server.virtnet: executing load_modules_local oleg359-server: Loading modules from /home/green/git/lustre-release/lustre oleg359-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg359-server: detected 4 online CPUs by sysfs oleg359-server: MODOPTS_LIBCFS= oleg359-server: Force libcfs to create 2 CPU partitions oleg359-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg359-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg359-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg359-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg359-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0000 oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg359-client.virtnet: -o user_xattr,flock 192.168.203.159@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 267 (time 1780335127.43 total 10.01 last 26.67) - open/close 560 (time 1780335137.43 total 20.02 last 29.28) - open/close 819 (time 1780335147.99 total 30.57 last 24.54) - open/close 1124 (time 1780335158.35 total 40.93 last 29.43) - open/close 1412 (time 1780335169.38 total 51.96 last 26.11) - open/close 1584 (time 1780335179.48 total 62.06 last 17.03) - open/close 1899 (time 1780335189.49 total 72.08 last 31.45) - open/close 2140 (time 1780335199.91 total 82.49 last 23.14) - open/close 2441 (time 1780335210.90 total 93.48 last 27.40) - open/close 2619 (time 1780335221.42 total 104.01 last 16.91) - open/close 2890 (time 1780335232.48 total 115.07 last 24.50) - open/close 3052 (time 1780335242.51 total 125.09 last 16.17) - open/close 3286 (time 1780335252.51 total 135.10 last 23.39) - open/close 3482 (time 1780335262.67 total 145.26 last 19.28) - open/close 3737 (time 1780335272.67 total 155.26 last 25.50) - open/close 3959 (time 1780335282.77 total 165.35 last 22.00) - open/close 4212 (time 1780335293.67 total 176.26 last 23.20) - open/close 4477 (time 1780335304.12 total 186.70 last 25.37) - open/close 4579 (time 1780335314.92 total 197.50 last 9.45) - open/close 4772 (time 1780335324.92 total 207.51 last 19.28) - open/close 4984 (time 1780335335.11 total 217.69 last 20.81) - open/close 5212 (time 1780335345.36 total 227.94 last 22.25) - open/close 5482 (time 1780335355.36 total 237.94 last 26.99) - open/close 5731 (time 1780335365.37 total 247.96 last 24.86) - open/close 6035 (time 1780335375.39 total 257.97 last 30.36) - open/close 6310 (time 1780335385.75 total 268.33 last 26.55) - open/close 6442 (time 1780335396.24 total 278.83 last 12.58) - open/close 6612 (time 1780335406.26 total 288.84 last 16.97) - open/close 6856 (time 1780335416.28 total 298.86 last 24.35) - open/close 7089 (time 1780335426.28 total 308.86 last 23.30) - open/close 7340 (time 1780335436.29 total 318.87 last 25.07) - open/close 7608 (time 1780335447.25 total 329.83 last 24.45) - open/close 7865 (time 1780335457.50 total 340.08 last 25.07) - open/close 8153 (time 1780335467.87 total 350.45 last 27.77) - open/close 8413 (time 1780335477.88 total 360.46 last 25.98) - open/close 8728 (time 1780335488.79 total 371.37 last 28.86) - open/close 9011 (time 1780335498.98 total 381.56 last 27.78) - open/close 9256 (time 1780335509.39 total 391.98 last 23.52) - open/close 9511 (time 1780335519.40 total 401.98 last 25.49) - open/close 9796 (time 1780335529.94 total 412.53 last 27.02) - open/close 9991 (time 1780335540.37 total 422.95 last 18.70) - open/close 10242 (time 1780335550.38 total 432.96 last 25.08) - open/close 10510 (time 1780335560.38 total 442.96 last 26.80) - open/close 10783 (time 1780335570.38 total 452.97 last 27.29) - open/close 11094 (time 1780335580.39 total 462.97 last 31.08) - open/close 11361 (time 1780335590.50 total 473.09 last 26.40) - open/close 11649 (time 1780335600.59 total 483.18 last 28.54) - open/close 11913 (time 1780335611.55 total 494.14 last 24.09) - open/close 12188 (time 1780335622.11 total 504.70 last 26.04) - open/close 12461 (time 1780335632.92 total 515.51 last 25.25) - open/close 12710 (time 1780335642.96 total 525.54 last 24.82) - open/close 12987 (time 1780335653.39 total 535.98 last 26.54) - open/close 13287 (time 1780335664.09 total 546.67 last 28.05) - open/close 13494 (time 1780335674.95 total 557.54 last 19.05) - open/close 13755 (time 1780335685.65 total 568.23 last 24.41) - open/close 14068 (time 1780335696.53 total 579.11 last 28.77) - open/close 14358 (time 1780335706.54 total 589.12 last 28.97) - open/close 14546 (time 1780335719.20 total 601.79 last 14.84) - open/close 14808 (time 1780335729.76 total 612.34 last 24.83) - open/close 15021 (time 1780335739.77 total 622.36 last 21.27) - open/close 15274 (time 1780335749.78 total 632.37 last 25.27) - open/close 15578 (time 1780335759.79 total 642.37 last 30.40) - open/close 15819 (time 1780335769.79 total 652.37 last 24.10) - open/close 16088 (time 1780335780.34 total 662.92 last 25.49) - open/close 16362 (time 1780335791.13 total 673.72 last 25.38) - open/close 16621 (time 1780335801.57 total 684.16 last 24.82) - open/close 16913 (time 1780335812.27 total 694.86 last 27.29) - open/close 17188 (time 1780335822.55 total 705.14 last 26.75) - open/close 17493 (time 1780335833.18 total 715.77 last 28.70) - open/close 17768 (time 1780335843.20 total 725.78 last 27.46) - open/close 17984 (time 1780335853.41 total 736.00 last 21.14) - open/close 18289 (time 1780335863.41 total 746.00 last 30.49) - open/close 18545 (time 1780335873.82 total 756.40 last 24.60) - open/close 18838 (time 1780335884.63 total 767.22 last 27.09) - open/close 19124 (time 1780335894.64 total 777.22 last 28.59) - open/close 19308 (time 1780335904.85 total 787.43 last 18.02) - open/close 19489 (time 1780335914.96 total 797.55 last 17.89) - open/close 19733 (time 1780335925.22 total 807.81 last 23.79) - open/close 19982 (time 1780335935.23 total 817.81 last 24.88) - open/close 20265 (time 1780335945.70 total 828.28 last 27.04) - open/close 20525 (time 1780335956.49 total 839.07 last 24.09) - open/close 20766 (time 1780335966.50 total 849.08 last 24.07) - open/close 21162 (time 1780335976.51 total 859.09 last 39.57) - open/close 22068 (time 1780335986.52 total 869.10 last 90.52) - open/close 22889 (time 1780335996.52 total 879.10 last 82.07) - open/close 23802 (time 1780336006.52 total 889.11 last 91.28) - open/close 24689 (time 1780336016.53 total 899.12 last 88.59) - open/close 25700 (time 1780336026.54 total 909.12 last 101.03) - open/close 26641 (time 1780336036.55 total 919.14 last 93.98) - open/close 27545 (time 1780336046.56 total 929.15 last 90.31) - open/close 28469 (time 1780336056.57 total 939.15 last 92.36) - open/close 29402 (time 1780336066.58 total 949.16 last 93.22) - open/close 30000 (time 1780336072.69 total 955.27 last 97.81) - open/close 30923 (time 1780336082.70 total 965.29 last 92.19) - open/close 31875 (time 1780336092.71 total 975.29 last 95.16) - open/close 32726 (time 1780336102.73 total 985.32 last 84.88) - open/close 33661 (time 1780336112.73 total 995.32 last 93.49) - open/close 34587 (time 1780336122.75 total 1005.33 last 92.50) - open/close 35517 (time 1780336132.76 total 1015.34 last 92.88) - open/close 36405 (time 1780336142.76 total 1025.34 last 88.80) - open/close 37309 (time 1780336152.76 total 1035.35 last 90.36) - open/close 38275 (time 1780336162.77 total 1045.35 last 96.58) - open/close 39179 (time 1780336172.77 total 1055.36 last 90.34) - open/close 40000 (time 1780336181.56 total 1064.14 last 93.47) - open/close 40751 (time 1780336191.56 total 1074.14 last 75.09) - open/close 41719 (time 1780336201.56 total 1084.15 last 96.76) - open/close 42689 (time 1780336211.56 total 1094.15 last 96.97) - open/close 43651 (time 1780336221.57 total 1104.15 last 96.17) - open/close 44612 (time 1780336231.57 total 1114.16 last 96.07) - open/close 45554 (time 1780336241.57 total 1124.16 last 94.19) - open/close 46516 (time 1780336251.58 total 1134.16 last 96.13) - open/close 47505 (time 1780336261.58 total 1144.17 last 98.89) - open/close 48435 (time 1780336271.60 total 1154.19 last 92.79) - open/close 49308 (time 1780336281.60 total 1164.19 last 87.29) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1167.86 seconds: 42.50 ops/second - unlinked 0 (time 1780336289 ; total 0 ; last 0) - unlinked 10000 (time 1780336400 ; total 111 ; last 111) - unlinked 20000 (time 1780336515 ; total 226 ; last 115) - unlinked 30000 (time 1780336585 ; total 296 ; last 70) - unlinked 40000 (time 1780336647 ; total 358 ; last 62) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 418 seconds: 118.734451 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg359-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg359-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg359-server stop mds service on oleg359-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg359-server stop mds service on oleg359-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg359-server unloading modules via unload_modules_local on: 'oleg359-server' oleg359-server: oleg359-server.virtnet: executing unload_modules_local oleg359-server: modules unloaded. pdsh@oleg359-client: oleg359-client: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-client: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2