== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:54:03 (1784235243) start mds service on oleg438-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg438-server: detected 4 online CPUs by sysfs oleg438-server: MODOPTS_LIBCFS= oleg438-server: Force libcfs to create 2 CPU partitions oleg438-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg438-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg438-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg438-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg438-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 806 (time 1784235297.66 total 10.00 last 80.59) - open/close 1377 (time 1784235308.17 total 20.51 last 54.35) - open/close 2042 (time 1784235318.98 total 31.32 last 61.49) - open/close 2632 (time 1784235329.54 total 41.88 last 55.87) - open/close 3073 (time 1784235339.72 total 52.06 last 43.34) - open/close 3677 (time 1784235350.28 total 62.62 last 57.19) - open/close 4385 (time 1784235361.23 total 73.57 last 64.69) - open/close 5128 (time 1784235372.04 total 84.38 last 68.71) - open/close 5736 (time 1784235382.86 total 95.20 last 56.20) - open/close 6093 (time 1784235393.55 total 105.89 last 33.39) - open/close 6435 (time 1784235403.56 total 115.90 last 34.18) - open/close 6627 (time 1784235413.72 total 126.06 last 18.89) - open/close 6775 (time 1784235424.40 total 136.74 last 13.86) - open/close 6972 (time 1784235434.51 total 146.85 last 19.48) - open/close 7129 (time 1784235445.52 total 157.86 last 14.26) - open/close 7345 (time 1784235455.95 total 168.29 last 20.71) - open/close 7613 (time 1784235466.06 total 178.40 last 26.51) - open/close 7817 (time 1784235476.43 total 188.77 last 19.68) - open/close 7948 (time 1784235486.47 total 198.81 last 13.04) - open/close 8223 (time 1784235497.48 total 209.82 last 24.98) - open/close 8435 (time 1784235508.51 total 220.85 last 19.22) - open/close 8665 (time 1784235520.81 total 233.15 last 18.70) - open/close 8711 (time 1784235534.35 total 246.69 last 3.40) - open/close 8919 (time 1784235544.79 total 257.13 last 19.93) - open/close 9025 (time 1784235555.62 total 267.96 last 9.79) - open/close 9493 (time 1784235566.09 total 278.43 last 44.69) - open/close 10000 (time 1784235573.35 total 285.69 last 69.80) - open/close 10582 (time 1784235584.14 total 296.48 last 53.97) - open/close 10914 (time 1784235594.35 total 306.69 last 32.51) - open/close 11203 (time 1784235604.36 total 316.70 last 28.87) - open/close 11375 (time 1784235614.49 total 326.83 last 16.98) - open/close 11544 (time 1784235624.74 total 337.08 last 16.49) - open/close 11809 (time 1784235634.83 total 347.17 last 26.26) - open/close 12009 (time 1784235645.84 total 358.18 last 18.17) - open/close 12209 (time 1784235656.09 total 368.43 last 19.51) - open/close 12415 (time 1784235666.46 total 378.80 last 19.86) - open/close 12640 (time 1784235677.15 total 389.49 last 21.06) - open/close 12862 (time 1784235688.03 total 400.37 last 20.40) - open/close 13123 (time 1784235699.04 total 411.38 last 23.70) - open/close 13397 (time 1784235709.98 total 422.31 last 25.05) - open/close 13649 (time 1784235720.92 total 433.26 last 23.02) - open/close 13818 (time 1784235731.48 total 443.82 last 16.02) - open/close 13963 (time 1784235741.49 total 453.83 last 14.48) - open/close 14161 (time 1784235751.70 total 464.04 last 19.39) - open/close 14341 (time 1784235762.39 total 474.73 last 16.83) - open/close 14547 (time 1784235772.40 total 484.74 last 20.58) - open/close 14808 (time 1784235782.41 total 494.75 last 26.09) - open/close 15064 (time 1784235793.31 total 505.65 last 23.48) - open/close 15272 (time 1784235803.60 total 515.94 last 20.21) - open/close 15793 (time 1784235813.61 total 525.95 last 52.04) - open/close 16614 (time 1784235823.63 total 535.97 last 81.96) - open/close 17451 (time 1784235833.63 total 545.97 last 83.66) - open/close 18285 (time 1784235843.63 total 555.97 last 83.40) - open/close 19139 (time 1784235853.65 total 565.99 last 85.26) - open/close 20000 (time 1784235863.41 total 575.75 last 88.22) - open/close 20880 (time 1784235873.41 total 585.75 last 87.99) - open/close 21744 (time 1784235883.42 total 595.75 last 86.36) - open/close 22564 (time 1784235893.42 total 605.76 last 81.97) - open/close 23434 (time 1784235903.43 total 615.77 last 86.95) - open/close 24290 (time 1784235913.43 total 625.77 last 85.54) - open/close 25249 (time 1784235923.44 total 635.78 last 95.84) - open/close 26872 (time 1784235933.44 total 645.78 last 162.30) - open/close 27882 (time 1784235943.44 total 655.78 last 100.98) - open/close 28715 (time 1784235953.45 total 665.78 last 83.26) - open/close 29438 (time 1784235963.45 total 675.79 last 72.27) - open/close 30000 (time 1784235971.80 total 684.14 last 67.26) - open/close 30891 (time 1784235981.81 total 694.15 last 89.05) - open/close 31844 (time 1784235991.81 total 704.15 last 95.30) - open/close 32783 (time 1784236001.81 total 714.15 last 93.86) - open/close 33713 (time 1784236011.82 total 724.16 last 92.97) - open/close 34556 (time 1784236021.82 total 734.16 last 84.26) - open/close 35059 (time 1784236031.82 total 744.16 last 50.29) - open/close 35604 (time 1784236041.83 total 754.17 last 54.46) - open/close 36168 (time 1784236051.84 total 764.18 last 56.36) - open/close 36720 (time 1784236061.84 total 774.18 last 55.19) - open/close 37442 (time 1784236071.86 total 784.20 last 72.06) - open/close 38199 (time 1784236081.87 total 794.21 last 75.65) - open/close 38997 (time 1784236091.88 total 804.22 last 79.72) - open/close 39804 (time 1784236101.88 total 814.22 last 80.68) - open/close 40000 (time 1784236104.14 total 816.48 last 86.63) - open/close 40814 (time 1784236114.15 total 826.49 last 81.38) - open/close 41591 (time 1784236124.16 total 836.50 last 77.62) - open/close 42324 (time 1784236134.17 total 846.51 last 73.22) - open/close 43077 (time 1784236144.17 total 856.51 last 75.28) - open/close 43844 (time 1784236154.17 total 866.51 last 76.69) - open/close 44671 (time 1784236164.20 total 876.54 last 82.49) - open/close 45429 (time 1784236174.21 total 886.54 last 75.74) - open/close 46218 (time 1784236184.22 total 896.56 last 78.81) - open/close 46972 (time 1784236194.23 total 906.57 last 75.29) - open/close 47719 (time 1784236204.24 total 916.58 last 74.65) - open/close 48487 (time 1784236214.25 total 926.59 last 76.71) - open/close 49289 (time 1784236224.25 total 936.59 last 80.17) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 941.36 seconds: 52.72 ops/second - unlinked 0 (time 1784236233 ; total 0 ; last 0) - unlinked 10000 (time 1784236369 ; total 136 ; last 136) - unlinked 20000 (time 1784236502 ; total 269 ; last 133) - unlinked 30000 (time 1784236635 ; total 402 ; last 133) - unlinked 40000 (time 1784236766 ; total 533 ; last 131) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 683 seconds: 72.666176 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg438-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg438-server unloading modules via unload_modules_local on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing unload_modules_local oleg438-server: modules unloaded. pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2