== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:23:59 (1781277839) start mds service on oleg135-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions oleg135-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg135-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg135-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg135-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg135-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 244 (time 1781277954.43 total 11.47 last 21.27) - open/close 538 (time 1781277964.55 total 21.59 last 29.04) - open/close 768 (time 1781277974.87 total 31.91 last 22.29) - open/close 1009 (time 1781277984.98 total 42.01 last 23.85) - open/close 1289 (time 1781277995.63 total 52.67 last 26.27) - open/close 1548 (time 1781278005.64 total 62.68 last 25.88) - open/close 1731 (time 1781278015.94 total 72.98 last 17.76) - open/close 1981 (time 1781278026.52 total 83.56 last 23.64) - open/close 2221 (time 1781278036.83 total 93.87 last 23.28) - open/close 2473 (time 1781278047.31 total 104.35 last 24.03) - open/close 2751 (time 1781278057.33 total 114.36 last 27.76) - open/close 2903 (time 1781278067.55 total 124.59 last 14.86) - open/close 3113 (time 1781278077.57 total 134.61 last 20.97) - open/close 3264 (time 1781278088.42 total 145.45 last 13.92) - open/close 3446 (time 1781278098.43 total 155.47 last 18.17) - open/close 3573 (time 1781278110.99 total 168.03 last 10.11) - open/close 3715 (time 1781278122.02 total 179.06 last 12.88) - open/close 3904 (time 1781278132.24 total 189.28 last 18.48) - open/close 4176 (time 1781278142.53 total 199.57 last 26.44) - open/close 4445 (time 1781278153.16 total 210.20 last 25.30) - open/close 4714 (time 1781278163.45 total 220.49 last 26.16) - open/close 4944 (time 1781278175.52 total 232.56 last 19.05) - open/close 5133 (time 1781278185.52 total 242.56 last 18.90) - open/close 5385 (time 1781278195.53 total 252.57 last 25.19) - open/close 5598 (time 1781278207.38 total 264.42 last 17.98) - open/close 5875 (time 1781278217.38 total 274.42 last 27.69) - open/close 6135 (time 1781278229.70 total 286.74 last 21.10) - open/close 6394 (time 1781278239.71 total 296.75 last 25.87) - open/close 6665 (time 1781278250.10 total 307.14 last 26.10) - open/close 6862 (time 1781278260.12 total 317.16 last 19.66) - open/close 7049 (time 1781278270.47 total 327.51 last 18.06) - open/close 7304 (time 1781278280.66 total 337.70 last 25.04) - open/close 7550 (time 1781278291.52 total 348.56 last 22.64) - open/close 7684 (time 1781278301.65 total 358.69 last 13.23) - open/close 7845 (time 1781278311.66 total 368.70 last 16.08) - open/close 8053 (time 1781278322.01 total 379.05 last 20.11) - open/close 8268 (time 1781278332.63 total 389.67 last 20.25) - open/close 8518 (time 1781278344.14 total 401.18 last 21.71) - open/close 8789 (time 1781278355.17 total 412.21 last 24.58) - open/close 9034 (time 1781278365.18 total 422.22 last 24.47) - open/close 9291 (time 1781278378.18 total 435.22 last 19.77) - open/close 9539 (time 1781278388.95 total 445.99 last 23.03) - open/close 9792 (time 1781278398.95 total 455.99 last 25.30) - open/close 10000 (time 1781278407.79 total 464.83 last 23.53) - open/close 10180 (time 1781278418.60 total 475.64 last 16.65) - open/close 10376 (time 1781278429.11 total 486.14 last 18.65) - open/close 10646 (time 1781278439.12 total 496.15 last 26.97) - open/close 10871 (time 1781278449.31 total 506.35 last 22.06) - open/close 11150 (time 1781278460.32 total 517.35 last 25.36) - open/close 11430 (time 1781278470.34 total 527.38 last 27.94) - open/close 11726 (time 1781278480.35 total 537.39 last 29.55) - open/close 11910 (time 1781278492.88 total 549.92 last 14.69) - open/close 12186 (time 1781278502.93 total 559.97 last 27.46) - open/close 12354 (time 1781278513.51 total 570.55 last 15.87) - open/close 12636 (time 1781278524.35 total 581.39 last 26.02) - open/close 12807 (time 1781278534.38 total 591.42 last 17.05) - open/close 13053 (time 1781278544.38 total 601.42 last 24.60) - open/close 13290 (time 1781278554.41 total 611.45 last 23.64) - open/close 13531 (time 1781278565.08 total 622.12 last 22.59) - open/close 13774 (time 1781278575.08 total 632.12 last 24.30) - open/close 13910 (time 1781278585.87 total 642.91 last 12.61) - open/close 14030 (time 1781278596.05 total 653.08 last 11.79) - open/close 14193 (time 1781278607.52 total 664.56 last 14.21) - open/close 14393 (time 1781278617.55 total 674.58 last 19.95) - open/close 14613 (time 1781278627.71 total 684.74 last 21.65) - open/close 14850 (time 1781278638.12 total 695.16 last 22.76) - open/close 15111 (time 1781278648.98 total 706.02 last 24.03) - open/close 15388 (time 1781278659.84 total 716.88 last 25.50) - open/close 15582 (time 1781278669.84 total 726.88 last 19.40) - open/close 15845 (time 1781278680.47 total 737.51 last 24.74) - open/close 16095 (time 1781278690.47 total 747.51 last 25.00) - open/close 16341 (time 1781278701.13 total 758.16 last 23.10) - open/close 16451 (time 1781278711.13 total 768.17 last 11.00) - open/close 16563 (time 1781278721.87 total 778.90 last 10.43) - open/close 16800 (time 1781278732.95 total 789.99 last 21.38) - open/close 17093 (time 1781278743.95 total 800.99 last 26.64) - open/close 17361 (time 1781278754.49 total 811.53 last 25.42) - open/close 17517 (time 1781278765.08 total 822.12 last 14.73) - open/close 17708 (time 1781278775.51 total 832.55 last 18.31) - open/close 17943 (time 1781278786.07 total 843.11 last 22.27) - open/close 18169 (time 1781278796.09 total 853.13 last 22.54) - open/close 18372 (time 1781278806.11 total 863.14 last 20.27) - open/close 18567 (time 1781278816.77 total 873.81 last 18.28) - open/close 18759 (time 1781278827.56 total 884.60 last 17.80) - open/close 19041 (time 1781278838.49 total 895.53 last 25.82) - open/close 19311 (time 1781278849.26 total 906.29 last 25.07) - open/close 19600 (time 1781278859.27 total 916.31 last 28.85) - open/close 19803 (time 1781278870.64 total 927.67 last 17.86) - open/close 20000 (time 1781278877.61 total 934.65 last 28.26) - open/close 21004 (time 1781278887.61 total 944.65 last 100.33) - open/close 21884 (time 1781278897.61 total 954.65 last 88.00) - open/close 22868 (time 1781278907.63 total 964.66 last 98.29) - open/close 23926 (time 1781278917.63 total 974.66 last 105.80) - open/close 24842 (time 1781278927.63 total 984.67 last 91.58) - open/close 25809 (time 1781278937.63 total 994.67 last 96.65) - open/close 26808 (time 1781278947.64 total 1004.68 last 99.79) - open/close 27649 (time 1781278957.65 total 1014.69 last 84.06) - open/close 28250 (time 1781278967.68 total 1024.72 last 59.92) - open/close 28907 (time 1781278977.69 total 1034.73 last 65.61) - open/close 29889 (time 1781278987.70 total 1044.74 last 98.15) - open/close 30894 (time 1781278997.71 total 1054.75 last 100.39) - open/close 32030 (time 1781279007.72 total 1064.76 last 113.50) - open/close 33072 (time 1781279017.72 total 1074.76 last 104.18) - open/close 34073 (time 1781279027.74 total 1084.78 last 99.89) - open/close 35097 (time 1781279037.75 total 1094.79 last 102.32) - open/close 36222 (time 1781279047.75 total 1104.79 last 112.44) - open/close 37162 (time 1781279057.76 total 1114.80 last 93.93) - open/close 38167 (time 1781279067.76 total 1124.80 last 100.47) - open/close 39204 (time 1781279077.77 total 1134.81 last 103.66) - open/close 40000 (time 1781279085.42 total 1142.46 last 104.02) - open/close 41016 (time 1781279095.42 total 1152.46 last 101.56) - open/close 42061 (time 1781279105.43 total 1162.47 last 104.43) - open/close 43024 (time 1781279115.44 total 1172.48 last 96.25) - open/close 43939 (time 1781279125.44 total 1182.48 last 91.44) - open/close 44991 (time 1781279135.46 total 1192.49 last 105.08) - open/close 46019 (time 1781279145.46 total 1202.50 last 102.74) - open/close 46916 (time 1781279155.47 total 1212.51 last 89.60) - open/close 47836 (time 1781279165.48 total 1222.52 last 91.96) - open/close 48741 (time 1781279175.48 total 1232.52 last 90.49) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1240.77 seconds: 40.00 ops/second - unlinked 0 (time 1781279187 ; total 0 ; last 0) - unlinked 10000 (time 1781279293 ; total 106 ; last 106) - unlinked 20000 (time 1781279401 ; total 214 ; last 108) - unlinked 30000 (time 1781279461 ; total 274 ; last 60) - unlinked 40000 (time 1781279528 ; total 341 ; last 67) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 414 seconds: 119.881645 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg135-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg135-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server stop mds service on oleg135-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg135-server stop mds service on oleg135-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg135-server unloading modules via unload_modules_local on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing unload_modules_local oleg135-server: modules unloaded. pdsh@oleg135-client: oleg135-client: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-client: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2