== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:19:58 (1781219998) start mds service on oleg607-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg607-server' oleg607-server: oleg607-server.virtnet: executing load_modules_local oleg607-server: Loading modules from /home/green/git/lustre-release/lustre oleg607-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg607-server: detected 4 online CPUs by sysfs oleg607-server: MODOPTS_LIBCFS= oleg607-server: Force libcfs to create 2 CPU partitions oleg607-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg607-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg607-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg607-server: oleg607-server.virtnet: executing set_default_debug -1 all pdsh@oleg607-client: oleg607-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg607-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg607-server: oleg607-server.virtnet: executing set_default_debug -1 all pdsh@oleg607-client: oleg607-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg607-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg607-server: oleg607-server.virtnet: executing set_default_debug -1 all pdsh@oleg607-client: oleg607-server: ssh exited with exit code 1 Started lustre-OST0000 oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg607-client.virtnet: -o user_xattr,flock 192.168.206.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 294 (time 1781220055.33 total 10.58 last 27.78) - open/close 587 (time 1781220065.82 total 21.07 last 27.93) - open/close 821 (time 1781220076.83 total 32.08 last 21.26) - open/close 1075 (time 1781220087.84 total 43.09 last 23.07) - open/close 1390 (time 1781220097.84 total 53.09 last 31.50) - open/close 1694 (time 1781220108.51 total 63.76 last 28.50) - open/close 1945 (time 1781220119.26 total 74.51 last 23.35) - open/close 2246 (time 1781220129.63 total 84.88 last 29.03) - open/close 2518 (time 1781220139.81 total 95.06 last 26.71) - open/close 2790 (time 1781220150.37 total 105.62 last 25.77) - open/close 2958 (time 1781220160.73 total 115.98 last 16.21) - open/close 3197 (time 1781220171.55 total 126.80 last 22.09) - open/close 3459 (time 1781220181.56 total 136.81 last 26.19) - open/close 3746 (time 1781220191.78 total 147.03 last 28.08) - open/close 4033 (time 1781220201.95 total 157.20 last 28.20) - open/close 4317 (time 1781220212.38 total 167.63 last 27.23) - open/close 4660 (time 1781220222.43 total 177.68 last 34.14) - open/close 4873 (time 1781220232.43 total 187.68 last 21.30) - open/close 5157 (time 1781220243.48 total 198.74 last 25.69) - open/close 5378 (time 1781220254.18 total 209.43 last 20.67) - open/close 5640 (time 1781220264.22 total 219.47 last 26.07) - open/close 5845 (time 1781220274.23 total 229.48 last 20.49) - open/close 6091 (time 1781220285.15 total 240.40 last 22.53) - open/close 6402 (time 1781220295.45 total 250.70 last 30.19) - open/close 6663 (time 1781220305.46 total 260.71 last 26.09) - open/close 6934 (time 1781220316.51 total 271.76 last 24.52) - open/close 7234 (time 1781220326.62 total 281.87 last 29.65) - open/close 7559 (time 1781220337.12 total 292.37 last 30.97) - open/close 7913 (time 1781220347.12 total 302.37 last 35.39) - open/close 8169 (time 1781220357.79 total 313.04 last 23.99) - open/close 8525 (time 1781220368.55 total 323.80 last 33.09) - open/close 8856 (time 1781220379.51 total 334.76 last 30.21) - open/close 9207 (time 1781220389.51 total 344.76 last 35.08) - open/close 9452 (time 1781220399.65 total 354.90 last 24.17) - open/close 9744 (time 1781220409.95 total 365.20 last 28.33) - open/close 10000 (time 1781220418.71 total 373.96 last 29.24) - open/close 10270 (time 1781220429.66 total 384.91 last 24.65) - open/close 10571 (time 1781220439.67 total 394.92 last 30.08) - open/close 10816 (time 1781220450.08 total 405.33 last 23.53) - open/close 11095 (time 1781220461.02 total 416.27 last 25.50) - open/close 11438 (time 1781220471.03 total 426.28 last 34.28) - open/close 11792 (time 1781220481.03 total 436.28 last 35.39) - open/close 13286 (time 1781220491.04 total 446.29 last 149.24) - open/close 14567 (time 1781220501.05 total 456.30 last 128.04) - open/close 15943 (time 1781220511.05 total 466.30 last 137.48) - open/close 17491 (time 1781220521.06 total 476.31 last 154.75) - open/close 18997 (time 1781220531.07 total 486.32 last 150.42) - open/close 20000 (time 1781220539.73 total 494.98 last 115.75) - open/close 21342 (time 1781220549.74 total 504.99 last 134.16) - open/close 22508 (time 1781220559.75 total 515.00 last 116.50) - open/close 23653 (time 1781220569.75 total 525.00 last 114.44) - open/close 24840 (time 1781220579.75 total 535.00 last 118.69) - open/close 26171 (time 1781220589.75 total 545.00 last 133.10) - open/close 27451 (time 1781220599.75 total 555.00 last 128.00) - open/close 28795 (time 1781220609.76 total 565.01 last 134.36) - open/close 30000 (time 1781220618.57 total 573.82 last 136.65) - open/close 31709 (time 1781220628.58 total 583.83 last 170.77) - open/close 33363 (time 1781220638.59 total 593.84 last 165.32) - open/close 34756 (time 1781220648.60 total 603.85 last 139.17) - open/close 35786 (time 1781220658.60 total 613.85 last 102.96) - open/close 36816 (time 1781220668.60 total 623.85 last 102.97) - open/close 37917 (time 1781220678.61 total 633.86 last 110.03) - open/close 38946 (time 1781220688.61 total 643.86 last 102.90) - open/close 40000 (time 1781220698.61 total 653.86 last 105.42) - open/close 41029 (time 1781220708.61 total 663.86 last 102.87) - open/close 41964 (time 1781220718.62 total 673.87 last 93.41) - open/close 42916 (time 1781220728.62 total 683.87 last 95.18) - open/close 43899 (time 1781220738.63 total 693.88 last 98.23) - open/close 44870 (time 1781220748.64 total 703.89 last 97.00) - open/close 45879 (time 1781220758.64 total 713.89 last 100.86) - open/close 47016 (time 1781220768.65 total 723.90 last 113.65) - open/close 48277 (time 1781220778.65 total 733.90 last 126.02) - open/close 49348 (time 1781220788.66 total 743.91 last 107.02) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 746.48 seconds: 66.49 ops/second - unlinked 0 (time 1781220793 ; total 0 ; last 0) - unlinked 10000 (time 1781220854 ; total 61 ; last 61) - unlinked 20000 (time 1781220894 ; total 101 ; last 40) - unlinked 30000 (time 1781220935 ; total 142 ; last 41) - unlinked 40000 (time 1781220970 ; total 177 ; last 35) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 221 seconds: 224.574661 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg607-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg607-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg607-server stop mds service on oleg607-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg607-server stop mds service on oleg607-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg607-server unloading modules via unload_modules_local on: 'oleg607-server' oleg607-server: oleg607-server.virtnet: executing unload_modules_local oleg607-server: modules unloaded. pdsh@oleg607-client: oleg607-server: ssh exited with exit code 2 pdsh@oleg607-client: oleg607-client: ssh exited with exit code 2 pdsh@oleg607-client: oleg607-client: ssh exited with exit code 2 pdsh@oleg607-client: oleg607-server: ssh exited with exit code 2