== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:32:08 (1786483928) start mds service on oleg204-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing load_modules_local oleg204-server: Loading modules from /home/green/git/lustre-release/lustre oleg204-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg204-server: detected 4 online CPUs by sysfs oleg204-server: MODOPTS_LIBCFS= oleg204-server: Force libcfs to create 2 CPU partitions oleg204-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg204-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg204-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg204-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg204-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 271 (time 1786484013.43 total 10.00 last 27.09) - open/close 483 (time 1786484023.46 total 20.03 last 21.14) - open/close 665 (time 1786484033.47 total 30.04 last 18.19) - open/close 772 (time 1786484044.47 total 41.04 last 9.72) - open/close 863 (time 1786484055.28 total 51.85 last 8.42) - open/close 979 (time 1786484066.10 total 62.67 last 10.73) - open/close 1102 (time 1786484077.09 total 73.66 last 11.19) - open/close 1236 (time 1786484087.91 total 84.48 last 12.38) - open/close 1347 (time 1786484098.72 total 95.29 last 10.27) - open/close 1578 (time 1786484108.73 total 105.30 last 23.08) - open/close 1758 (time 1786484118.74 total 115.31 last 17.98) - open/close 1928 (time 1786484129.28 total 125.85 last 16.14) - open/close 2061 (time 1786484139.33 total 135.90 last 13.23) - open/close 2207 (time 1786484150.26 total 146.83 last 13.36) - open/close 2332 (time 1786484160.27 total 156.84 last 12.48) - open/close 2586 (time 1786484170.28 total 166.85 last 25.39) - open/close 2756 (time 1786484180.94 total 177.51 last 15.95) - open/close 3017 (time 1786484191.67 total 188.25 last 24.31) - open/close 3330 (time 1786484201.68 total 198.25 last 31.29) - open/close 3445 (time 1786484212.00 total 208.57 last 11.14) - open/close 3683 (time 1786484222.01 total 218.58 last 23.78) - open/close 3896 (time 1786484233.77 total 230.34 last 18.12) - open/close 4019 (time 1786484244.71 total 241.29 last 11.24) - open/close 4234 (time 1786484257.32 total 253.89 last 17.05) - open/close 4392 (time 1786484267.32 total 263.90 last 15.79) - open/close 4487 (time 1786484278.12 total 274.69 last 8.80) - open/close 4656 (time 1786484289.09 total 285.66 last 15.40) - open/close 4812 (time 1786484299.39 total 295.96 last 15.15) - open/close 5009 (time 1786484309.40 total 305.97 last 19.68) - open/close 5175 (time 1786484319.77 total 316.34 last 16.00) - open/close 5413 (time 1786484330.53 total 327.10 last 22.13) - open/close 5536 (time 1786484340.59 total 337.17 last 12.22) - open/close 5775 (time 1786484350.60 total 347.17 last 23.89) - open/close 5997 (time 1786484361.41 total 357.98 last 20.53) - open/close 6117 (time 1786484372.45 total 369.02 last 10.88) - open/close 6346 (time 1786484382.58 total 379.15 last 22.60) - open/close 6597 (time 1786484392.59 total 389.16 last 25.08) - open/close 6767 (time 1786484403.42 total 399.99 last 15.69) - open/close 6940 (time 1786484413.98 total 410.55 last 16.38) - open/close 7139 (time 1786484424.00 total 420.57 last 19.86) - open/close 7305 (time 1786484434.20 total 430.78 last 16.27) - open/close 7515 (time 1786484444.44 total 441.01 last 20.52) - open/close 7730 (time 1786484454.76 total 451.33 last 20.83) - open/close 7878 (time 1786484465.44 total 462.01 last 13.86) - open/close 8068 (time 1786484475.45 total 472.02 last 18.99) - open/close 8355 (time 1786484485.46 total 482.03 last 28.66) - open/close 8592 (time 1786484495.97 total 492.54 last 22.55) - open/close 8733 (time 1786484506.16 total 502.73 last 13.84) - open/close 8892 (time 1786484516.59 total 513.16 last 15.25) - open/close 9203 (time 1786484526.60 total 523.18 last 31.05) - open/close 9419 (time 1786484537.63 total 534.20 last 19.60) - open/close 9544 (time 1786484547.63 total 544.20 last 12.50) - open/close 9699 (time 1786484558.05 total 554.63 last 14.87) - open/close 9837 (time 1786484568.06 total 564.63 last 13.80) - open/close 10000 (time 1786484573.68 total 570.25 last 28.98) - open/close 10192 (time 1786484583.76 total 580.34 last 19.04) - open/close 10459 (time 1786484594.58 total 591.15 last 24.69) - open/close 10685 (time 1786484605.23 total 601.80 last 21.22) - open/close 10921 (time 1786484615.84 total 612.41 last 22.24) - open/close 11232 (time 1786484626.65 total 623.22 last 28.77) - open/close 11445 (time 1786484637.01 total 633.59 last 20.55) - open/close 11584 (time 1786484647.32 total 643.89 last 13.49) - open/close 11781 (time 1786484657.32 total 653.90 last 19.69) - open/close 12210 (time 1786484667.42 total 663.99 last 42.50) - open/close 12605 (time 1786484678.42 total 674.99 last 35.90) - open/close 13103 (time 1786484689.24 total 685.81 last 46.04) - open/close 13829 (time 1786484699.48 total 696.05 last 70.87) - open/close 14564 (time 1786484709.48 total 706.05 last 73.49) - open/close 19446 (time 1786484719.48 total 716.05 last 488.18) - open/close 24278 (time 1786484729.48 total 726.06 last 483.14) - open/close 28488 (time 1786484739.49 total 736.06 last 420.91) - open/close 30000 (time 1786484743.01 total 739.58 last 428.76) - open/close 33944 (time 1786484753.01 total 749.58 last 394.39) - open/close 38314 (time 1786484763.01 total 759.58 last 437.00) - open/close 40000 (time 1786484767.04 total 763.61 last 419.05) - open/close 44086 (time 1786484777.04 total 773.61 last 408.55) - open/close 48084 (time 1786484787.04 total 783.61 last 399.80) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 787.60 seconds: 63.02 ops/second - unlinked 0 (time 1786484792 ; total 0 ; last 0) - unlinked 10000 (time 1786484810 ; total 18 ; last 18) - unlinked 20000 (time 1786484823 ; total 31 ; last 13) - unlinked 30000 (time 1786484836 ; total 44 ; last 13) - unlinked 40000 (time 1786484848 ; total 56 ; last 12) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 69 seconds: 719.289856 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg204-server unloading modules via unload_modules_local on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing unload_modules_local oleg204-server: modules unloaded. pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2