== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:14:47 (1786569287) start mds service on oleg212-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg212-server' oleg212-server: oleg212-server.virtnet: executing load_modules_local oleg212-server: Loading modules from /home/green/git/lustre-release/lustre oleg212-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg212-server: detected 4 online CPUs by sysfs oleg212-server: MODOPTS_LIBCFS= oleg212-server: Force libcfs to create 2 CPU partitions oleg212-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg212-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg212-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg212-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg212-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-OST0000 oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg212-client.virtnet: -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 174 (time 1786569373.81 total 10.47 last 16.61) - open/close 298 (time 1786569384.62 total 21.28 last 11.47) - open/close 489 (time 1786569394.62 total 31.28 last 19.10) - open/close 629 (time 1786569404.78 total 41.45 last 13.78) - open/close 749 (time 1786569414.78 total 51.45 last 12.00) - open/close 910 (time 1786569425.00 total 61.67 last 15.75) - open/close 1062 (time 1786569435.70 total 72.37 last 14.21) - open/close 1401 (time 1786569446.69 total 83.36 last 30.85) - open/close 1532 (time 1786569456.70 total 93.36 last 13.10) - open/close 1681 (time 1786569466.70 total 103.37 last 14.89) - open/close 1811 (time 1786569477.41 total 114.08 last 12.14) - open/close 1997 (time 1786569488.50 total 125.16 last 16.78) - open/close 2172 (time 1786569498.51 total 135.17 last 17.48) - open/close 2408 (time 1786569509.10 total 145.76 last 22.29) - open/close 2607 (time 1786569519.11 total 155.77 last 19.88) - open/close 3024 (time 1786569529.58 total 166.24 last 39.83) - open/close 3220 (time 1786569539.59 total 176.25 last 19.58) - open/close 3335 (time 1786569550.12 total 186.79 last 10.91) - open/close 3424 (time 1786569560.30 total 196.96 last 8.75) - open/close 3613 (time 1786569570.47 total 207.14 last 18.58) - open/close 3800 (time 1786569580.47 total 217.14 last 18.69) - open/close 4120 (time 1786569590.48 total 227.14 last 32.00) - open/close 4254 (time 1786569601.38 total 238.05 last 12.29) - open/close 4501 (time 1786569611.89 total 248.55 last 23.51) - open/close 4792 (time 1786569622.89 total 259.56 last 26.44) - open/close 5036 (time 1786569633.71 total 270.37 last 22.56) - open/close 5324 (time 1786569644.58 total 281.24 last 26.49) - open/close 5479 (time 1786569654.59 total 291.25 last 15.49) - open/close 5793 (time 1786569664.94 total 301.61 last 30.33) - open/close 5925 (time 1786569675.75 total 312.42 last 12.21) - open/close 6043 (time 1786569686.12 total 322.79 last 11.38) - open/close 6234 (time 1786569696.30 total 332.96 last 18.77) - open/close 6311 (time 1786569706.41 total 343.08 last 7.61) - open/close 6490 (time 1786569717.29 total 353.96 last 16.45) - open/close 6762 (time 1786569727.29 total 363.96 last 27.20) - open/close 7064 (time 1786569737.32 total 373.98 last 30.12) - open/close 7348 (time 1786569748.21 total 384.87 last 26.08) - open/close 7619 (time 1786569758.77 total 395.43 last 25.67) - open/close 7939 (time 1786569769.52 total 406.19 last 29.75) - open/close 8087 (time 1786569781.43 total 418.10 last 12.43) - open/close 8447 (time 1786569792.49 total 429.15 last 32.55) - open/close 8810 (time 1786569802.53 total 439.20 last 36.14) - open/close 9096 (time 1786569813.04 total 449.70 last 27.23) - open/close 9365 (time 1786569823.35 total 460.02 last 26.07) - open/close 9671 (time 1786569833.35 total 470.02 last 30.59) - open/close 9860 (time 1786569843.82 total 480.48 last 18.07) - open/close 10000 (time 1786569847.39 total 484.05 last 39.21) - open/close 10190 (time 1786569857.57 total 494.24 last 18.65) - open/close 10485 (time 1786569867.57 total 504.24 last 29.50) - open/close 10576 (time 1786569877.75 total 514.41 last 8.94) - open/close 10713 (time 1786569888.75 total 525.41 last 12.46) - open/close 10813 (time 1786569899.06 total 535.73 last 9.70) - open/close 11690 (time 1786569909.07 total 545.74 last 87.59) - open/close 13008 (time 1786569919.08 total 555.74 last 131.77) - open/close 14253 (time 1786569929.08 total 565.75 last 124.44) - open/close 15522 (time 1786569939.09 total 575.75 last 126.82) - open/close 16762 (time 1786569949.09 total 585.75 last 124.00) - open/close 18002 (time 1786569959.09 total 595.75 last 123.97) - open/close 19264 (time 1786569969.09 total 605.76 last 126.14) - open/close 20000 (time 1786569976.33 total 612.99 last 101.73) - open/close 21208 (time 1786569986.33 total 623.00 last 120.78) - open/close 22420 (time 1786569996.33 total 633.00 last 121.19) - open/close 23594 (time 1786570006.33 total 643.00 last 117.37) - open/close 24837 (time 1786570016.34 total 653.01 last 124.22) - open/close 25988 (time 1786570026.34 total 663.01 last 115.08) - open/close 27186 (time 1786570036.34 total 673.01 last 119.78) - open/close 28347 (time 1786570046.34 total 683.01 last 116.10) - open/close 29543 (time 1786570056.35 total 693.01 last 119.57) - open/close 30000 (time 1786570059.85 total 696.52 last 130.33) - open/close 31279 (time 1786570069.86 total 706.53 last 127.76) - open/close 32452 (time 1786570079.88 total 716.55 last 117.10) - open/close 33579 (time 1786570089.89 total 726.55 last 112.61) - open/close 34743 (time 1786570099.89 total 736.56 last 116.39) - open/close 35929 (time 1786570109.89 total 746.56 last 118.59) - open/close 37176 (time 1786570119.90 total 756.56 last 124.64) - open/close 38219 (time 1786570129.90 total 766.57 last 104.23) - open/close 39279 (time 1786570139.91 total 776.58 last 105.91) - open/close 40000 (time 1786570146.12 total 782.79 last 116.08) - open/close 41122 (time 1786570156.13 total 792.79 last 112.12) - open/close 42148 (time 1786570166.13 total 802.80 last 102.57) - open/close 43241 (time 1786570176.13 total 812.80 last 109.28) - open/close 44455 (time 1786570186.14 total 822.80 last 121.35) - open/close 45604 (time 1786570196.14 total 832.81 last 114.85) - open/close 46667 (time 1786570206.15 total 842.81 last 106.23) - open/close 47942 (time 1786570216.16 total 852.83 last 127.34) - open/close 49337 (time 1786570226.16 total 862.83 last 139.50) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 865.37 seconds: 57.35 ops/second - unlinked 0 (time 1786570231 ; total 0 ; last 0) - unlinked 10000 (time 1786570298 ; total 67 ; last 67) - unlinked 20000 (time 1786570342 ; total 111 ; last 44) - unlinked 30000 (time 1786570380 ; total 149 ; last 38) - unlinked 40000 (time 1786570412 ; total 181 ; last 32) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 218 seconds: 227.665131 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg212-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg212-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg212-server stop mds service on oleg212-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg212-server stop mds service on oleg212-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg212-server unloading modules via unload_modules_local on: 'oleg212-server' oleg212-server: oleg212-server.virtnet: executing unload_modules_local oleg212-server: modules unloaded. pdsh@oleg212-client: oleg212-client: ssh exited with exit code 2 pdsh@oleg212-client: oleg212-server: ssh exited with exit code 2 pdsh@oleg212-client: oleg212-server: ssh exited with exit code 2 pdsh@oleg212-client: oleg212-client: ssh exited with exit code 2