== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 06:35:05 (1778927705) start mds service on oleg410-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg410-server' oleg410-server: oleg410-server.virtnet: executing load_modules_local oleg410-server: Loading modules from /home/green/git/lustre-release/lustre oleg410-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg410-server: detected 4 online CPUs by sysfs oleg410-server: MODOPTS_LIBCFS= oleg410-server: Force libcfs to create 2 CPU partitions oleg410-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg410-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg410-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg410-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg410-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-OST0000 oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 143 (time 1778927770.93 total 10.00 last 14.30) - open/close 296 (time 1778927781.80 total 20.87 last 14.07) - open/close 457 (time 1778927791.81 total 30.88 last 16.09) - open/close 666 (time 1778927802.42 total 41.49 last 19.70) - open/close 815 (time 1778927813.49 total 52.56 last 13.46) - open/close 1012 (time 1778927824.38 total 63.45 last 18.10) - open/close 1144 (time 1778927835.31 total 74.38 last 12.07) - open/close 1261 (time 1778927846.13 total 85.20 last 10.82) - open/close 1458 (time 1778927856.56 total 95.63 last 18.88) - open/close 1573 (time 1778927867.25 total 106.32 last 10.76) - open/close 1729 (time 1778927877.25 total 116.32 last 15.60) - open/close 1891 (time 1778927887.53 total 126.61 last 15.75) - open/close 1981 (time 1778927898.09 total 137.17 last 8.52) - open/close 2064 (time 1778927908.42 total 147.49 last 8.04) - open/close 2195 (time 1778927919.28 total 158.36 last 12.06) - open/close 2318 (time 1778927930.99 total 170.06 last 10.51) - open/close 2443 (time 1778927941.10 total 180.18 last 12.35) - open/close 2591 (time 1778927951.54 total 190.61 last 14.18) - open/close 2723 (time 1778927961.65 total 200.72 last 13.06) - open/close 2877 (time 1778927971.95 total 211.02 last 14.95) - open/close 3045 (time 1778927982.13 total 221.20 last 16.51) - open/close 3237 (time 1778927992.17 total 231.24 last 19.12) - open/close 3365 (time 1778928003.13 total 242.20 last 11.68) - open/close 3502 (time 1778928013.13 total 252.21 last 13.69) - open/close 3669 (time 1778928023.28 total 262.35 last 16.46) - open/close 3944 (time 1778928034.24 total 273.31 last 25.09) - open/close 4106 (time 1778928044.85 total 283.92 last 15.27) - open/close 4252 (time 1778928054.85 total 293.93 last 14.59) - open/close 4421 (time 1778928064.86 total 303.93 last 16.89) - open/close 4516 (time 1778928075.52 total 314.59 last 8.91) - open/close 4711 (time 1778928087.98 total 327.06 last 15.64) - open/close 4957 (time 1778928098.86 total 337.93 last 22.62) - open/close 5066 (time 1778928109.23 total 348.30 last 10.51) - open/close 5154 (time 1778928119.73 total 358.81 last 8.38) - open/close 5224 (time 1778928130.81 total 369.88 last 6.32) - open/close 5301 (time 1778928140.98 total 380.05 last 7.57) - open/close 5495 (time 1778928151.35 total 390.42 last 18.70) - open/close 5664 (time 1778928161.36 total 400.43 last 16.89) - open/close 5931 (time 1778928171.95 total 411.02 last 25.21) - open/close 6039 (time 1778928182.45 total 421.52 last 10.29) - open/close 6378 (time 1778928192.74 total 431.82 last 32.92) - open/close 6717 (time 1778928202.75 total 441.82 last 33.88) - open/close 6939 (time 1778928212.97 total 452.04 last 21.72) - open/close 7258 (time 1778928223.03 total 462.10 last 31.73) - open/close 7517 (time 1778928233.51 total 472.59 last 24.70) - open/close 7819 (time 1778928244.20 total 483.28 last 28.24) - open/close 8124 (time 1778928254.21 total 493.28 last 30.48) - open/close 8472 (time 1778928264.21 total 503.29 last 34.78) - open/close 8646 (time 1778928276.27 total 515.34 last 14.44) - open/close 8949 (time 1778928286.32 total 525.39 last 30.15) - open/close 9137 (time 1778928296.43 total 535.50 last 18.59) - open/close 9475 (time 1778928306.43 total 545.50 last 33.79) - open/close 10000 (time 1778928310.72 total 549.79 last 122.41) - open/close 11251 (time 1778928320.73 total 559.80 last 125.00) - open/close 12474 (time 1778928330.73 total 569.80 last 122.28) - open/close 13584 (time 1778928340.73 total 579.80 last 110.99) - open/close 14744 (time 1778928350.73 total 589.81 last 115.95) - open/close 15880 (time 1778928360.74 total 599.81 last 113.55) - open/close 16929 (time 1778928370.74 total 609.82 last 104.83) - open/close 18067 (time 1778928380.75 total 619.83 last 113.71) - open/close 19233 (time 1778928390.76 total 629.83 last 116.53) - open/close 20000 (time 1778928397.75 total 636.82 last 109.70) - open/close 21611 (time 1778928407.75 total 646.83 last 161.03) - open/close 23020 (time 1778928417.76 total 656.83 last 140.85) - open/close 24686 (time 1778928427.76 total 666.83 last 166.56) - open/close 26270 (time 1778928437.76 total 676.84 last 158.37) - open/close 27411 (time 1778928447.77 total 686.84 last 114.06) - open/close 28500 (time 1778928457.77 total 696.84 last 108.88) - open/close 29770 (time 1778928467.77 total 706.85 last 126.96) - open/close 31118 (time 1778928477.78 total 716.85 last 134.75) - open/close 32315 (time 1778928487.78 total 726.86 last 119.62) - open/close 33471 (time 1778928497.79 total 736.86 last 115.51) - open/close 34948 (time 1778928507.79 total 746.87 last 147.66) - open/close 36370 (time 1778928517.79 total 756.87 last 142.18) - open/close 37594 (time 1778928527.80 total 766.87 last 122.37) - open/close 39020 (time 1778928537.80 total 776.88 last 142.48) - open/close 40000 (time 1778928546.92 total 785.99 last 107.50) - open/close 41148 (time 1778928556.92 total 796.00 last 114.75) - open/close 42445 (time 1778928566.93 total 806.00 last 129.63) - open/close 43672 (time 1778928576.94 total 816.01 last 122.64) - open/close 44926 (time 1778928586.94 total 826.02 last 125.31) - open/close 46105 (time 1778928596.94 total 836.02 last 117.88) - open/close 47610 (time 1778928606.94 total 846.02 last 150.49) - open/close 49186 (time 1778928616.95 total 856.02 last 157.56) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 859.04 seconds: 57.77 ops/second - unlinked 0 (time 1778928622 ; total 0 ; last 0) - unlinked 10000 (time 1778928676 ; total 54 ; last 54) - unlinked 20000 (time 1778928714 ; total 92 ; last 38) - unlinked 30000 (time 1778928754 ; total 132 ; last 40) - unlinked 40000 (time 1778928797 ; total 175 ; last 43) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 210 seconds: 236.338089 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg410-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg410-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg410-server stop mds service on oleg410-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg410-server stop mds service on oleg410-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg410-server unloading modules via unload_modules_local on: 'oleg410-server' oleg410-server: oleg410-server.virtnet: executing unload_modules_local oleg410-server: modules unloaded. pdsh@oleg410-client: oleg410-client: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-server: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-server: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-client: ssh exited with exit code 2