== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:27:27 (1776922047) start mds service on oleg228-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions oleg228-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg228-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg228-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg228-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg228-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0000 oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 105 (time 1776922108.05 total 10.34 last 10.15) - open/close 392 (time 1776922118.73 total 21.02 last 26.88) - open/close 721 (time 1776922129.62 total 31.91 last 30.22) - open/close 1020 (time 1776922140.24 total 42.53 last 28.16) - open/close 1363 (time 1776922151.17 total 53.46 last 31.36) - open/close 1541 (time 1776922161.61 total 63.90 last 17.06) - open/close 1838 (time 1776922171.91 total 74.20 last 28.81) - open/close 2096 (time 1776922182.86 total 85.15 last 23.57) - open/close 2283 (time 1776922193.28 total 95.57 last 17.94) - open/close 2581 (time 1776922203.29 total 105.58 last 29.79) - open/close 2839 (time 1776922215.88 total 118.17 last 20.48) - open/close 3092 (time 1776922225.99 total 128.28 last 25.03) - open/close 3423 (time 1776922236.04 total 138.33 last 32.94) - open/close 3827 (time 1776922246.09 total 148.38 last 40.21) - open/close 4104 (time 1776922256.26 total 158.55 last 27.23) - open/close 4360 (time 1776922266.26 total 168.55 last 25.59) - open/close 4567 (time 1776922277.70 total 179.99 last 18.09) - open/close 4924 (time 1776922288.52 total 190.81 last 33.02) - open/close 5087 (time 1776922298.90 total 201.19 last 15.70) - open/close 5315 (time 1776922309.19 total 211.48 last 22.16) - open/close 5510 (time 1776922319.56 total 221.85 last 18.80) - open/close 5828 (time 1776922329.73 total 232.02 last 31.26) - open/close 6200 (time 1776922339.73 total 242.02 last 37.20) - open/close 6458 (time 1776922349.74 total 252.03 last 25.79) - open/close 6639 (time 1776922360.14 total 262.43 last 17.40) - open/close 6933 (time 1776922370.15 total 272.44 last 29.39) - open/close 7141 (time 1776922380.42 total 282.71 last 20.24) - open/close 7366 (time 1776922390.79 total 293.08 last 21.71) - open/close 7525 (time 1776922400.97 total 303.26 last 15.62) - open/close 7736 (time 1776922411.46 total 313.75 last 20.11) - open/close 8045 (time 1776922422.41 total 324.70 last 28.22) - open/close 8244 (time 1776922432.72 total 335.01 last 19.31) - open/close 8438 (time 1776922443.21 total 345.50 last 18.50) - open/close 8790 (time 1776922454.22 total 356.51 last 31.97) - open/close 9337 (time 1776922464.27 total 366.56 last 54.42) - open/close 9866 (time 1776922474.64 total 376.93 last 50.99) - open/close 10151 (time 1776922485.06 total 387.35 last 27.35) - open/close 10768 (time 1776922495.07 total 397.36 last 61.67) - open/close 13546 (time 1776922505.07 total 407.36 last 277.77) - open/close 17337 (time 1776922515.07 total 417.36 last 379.05) - open/close 20000 (time 1776922520.15 total 422.44 last 524.63) - open/close 25138 (time 1776922530.15 total 432.44 last 513.71) - open/close 30000 (time 1776922540.07 total 442.36 last 489.79) - open/close 34735 (time 1776922550.07 total 452.36 last 473.49) - open/close 39843 (time 1776922560.08 total 462.37 last 510.71) - open/close 44344 (time 1776922570.08 total 472.37 last 450.05) - open/close 49053 (time 1776922580.08 total 482.37 last 470.85) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 483.66 seconds: 102.62 ops/second - unlinked 0 (time 1776922582 ; total 0 ; last 0) - unlinked 10000 (time 1776922594 ; total 12 ; last 12) - unlinked 20000 (time 1776922605 ; total 23 ; last 11) - unlinked 30000 (time 1776922615 ; total 33 ; last 10) - unlinked 40000 (time 1776922627 ; total 45 ; last 12) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 57 seconds: 870.736816 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg228-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg228-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg228-server stop mds service on oleg228-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg228-server stop mds service on oleg228-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg228-server unloading modules via unload_modules_local on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing unload_modules_local oleg228-server: modules unloaded. pdsh@oleg228-client: oleg228-client: ssh exited with exit code 2 pdsh@oleg228-client: oleg228-server: ssh exited with exit code 2 pdsh@oleg228-client: oleg228-client: ssh exited with exit code 2 pdsh@oleg228-client: oleg228-server: ssh exited with exit code 2