== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:45:02 (1787712302) start mds service on oleg316-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg316-server' oleg316-server: oleg316-server.virtnet: executing load_modules_local oleg316-server: Loading modules from /home/green/git/lustre-release/lustre oleg316-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg316-server: detected 4 online CPUs by sysfs oleg316-server: MODOPTS_LIBCFS= oleg316-server: Force libcfs to create 2 CPU partitions oleg316-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg316-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg316-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg316-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg316-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0000 oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 201 (time 1787712405.32 total 10.42 last 19.29) - open/close 484 (time 1787712416.13 total 21.24 last 26.17) - open/close 664 (time 1787712426.65 total 31.75 last 17.12) - open/close 771 (time 1787712437.00 total 42.11 last 10.33) - open/close 906 (time 1787712447.02 total 52.12 last 13.48) - open/close 1071 (time 1787712457.64 total 62.74 last 15.54) - open/close 1321 (time 1787712467.65 total 72.76 last 24.96) - open/close 1561 (time 1787712480.15 total 85.25 last 19.21) - open/close 1844 (time 1787712490.86 total 95.97 last 26.40) - open/close 2128 (time 1787712500.87 total 105.97 last 28.39) - open/close 2379 (time 1787712511.38 total 116.49 last 23.87) - open/close 2659 (time 1787712521.39 total 126.49 last 27.99) - open/close 2900 (time 1787712531.92 total 137.03 last 22.88) - open/close 3038 (time 1787712541.94 total 147.04 last 13.78) - open/close 3218 (time 1787712552.46 total 157.56 last 17.11) - open/close 3335 (time 1787712562.77 total 167.87 last 11.35) - open/close 3632 (time 1787712572.78 total 177.88 last 29.67) - open/close 3959 (time 1787712583.83 total 188.93 last 29.60) - open/close 4209 (time 1787712594.21 total 199.31 last 24.08) - open/close 4450 (time 1787712604.22 total 209.33 last 24.06) - open/close 4732 (time 1787712614.86 total 219.96 last 26.52) - open/close 5008 (time 1787712625.42 total 230.52 last 26.13) - open/close 5275 (time 1787712635.72 total 240.83 last 25.91) - open/close 5555 (time 1787712645.73 total 250.83 last 27.99) - open/close 5818 (time 1787712656.46 total 261.57 last 24.50) - open/close 6116 (time 1787712667.31 total 272.42 last 27.47) - open/close 6363 (time 1787712677.77 total 282.88 last 23.62) - open/close 6591 (time 1787712687.78 total 292.88 last 22.78) - open/close 6810 (time 1787712698.25 total 303.36 last 20.91) - open/close 7107 (time 1787712709.20 total 314.30 last 27.13) - open/close 7416 (time 1787712720.14 total 325.25 last 28.23) - open/close 7721 (time 1787712731.10 total 336.20 last 27.85) - open/close 7973 (time 1787712741.11 total 346.22 last 25.15) - open/close 8208 (time 1787712751.13 total 356.23 last 23.47) - open/close 8427 (time 1787712762.12 total 367.23 last 19.91) - open/close 8700 (time 1787712772.13 total 377.24 last 27.27) - open/close 8909 (time 1787712782.15 total 387.25 last 20.87) - open/close 9163 (time 1787712792.59 total 397.70 last 24.32) - open/close 9443 (time 1787712803.24 total 408.34 last 26.30) - open/close 9663 (time 1787712814.22 total 419.33 last 20.03) - open/close 9919 (time 1787712824.23 total 429.33 last 25.58) - open/close 10000 (time 1787712827.54 total 432.64 last 24.47) - open/close 10220 (time 1787712837.54 total 442.65 last 22.00) - open/close 10459 (time 1787712847.56 total 452.66 last 23.86) - open/close 10646 (time 1787712858.58 total 463.69 last 16.96) - open/close 10923 (time 1787712868.60 total 473.71 last 27.64) - open/close 11129 (time 1787712878.61 total 483.71 last 20.60) - open/close 11388 (time 1787712888.62 total 493.72 last 25.87) - open/close 11604 (time 1787712898.66 total 503.76 last 21.51) - open/close 11863 (time 1787712909.33 total 514.43 last 24.28) - open/close 12152 (time 1787712920.21 total 525.32 last 26.54) - open/close 12378 (time 1787712930.24 total 535.34 last 22.55) - open/close 12486 (time 1787712940.25 total 545.36 last 10.78) - open/close 12618 (time 1787712953.32 total 558.42 last 10.10) - open/close 12860 (time 1787712965.80 total 570.91 last 19.39) - open/close 13099 (time 1787712976.53 total 581.63 last 22.29) - open/close 13217 (time 1787712986.54 total 591.64 last 11.79) - open/close 13297 (time 1787712996.64 total 601.74 last 7.92) - open/close 13477 (time 1787713006.66 total 611.77 last 17.95) - open/close 13681 (time 1787713016.70 total 621.81 last 20.32) - open/close 13903 (time 1787713028.60 total 633.71 last 18.65) - open/close 14156 (time 1787713039.13 total 644.23 last 24.04) - open/close 14311 (time 1787713049.23 total 654.34 last 15.34) - open/close 14551 (time 1787713059.86 total 664.96 last 22.59) - open/close 14763 (time 1787713070.41 total 675.51 last 20.10) - open/close 15011 (time 1787713080.42 total 685.52 last 24.77) - open/close 15154 (time 1787713090.43 total 695.54 last 14.28) - open/close 15406 (time 1787713100.89 total 705.99 last 24.10) - open/close 15625 (time 1787713110.89 total 715.99 last 21.90) - open/close 15894 (time 1787713120.91 total 726.02 last 26.84) - open/close 16108 (time 1787713133.20 total 738.30 last 17.42) - open/close 16258 (time 1787713143.23 total 748.33 last 14.96) - open/close 16480 (time 1787713153.57 total 758.67 last 21.47) - open/close 16694 (time 1787713163.69 total 768.80 last 21.14) - open/close 16881 (time 1787713173.71 total 778.81 last 18.67) - open/close 17036 (time 1787713183.88 total 788.99 last 15.23) - open/close 17274 (time 1787713194.38 total 799.49 last 22.67) - open/close 17537 (time 1787713204.38 total 809.49 last 26.30) - open/close 17836 (time 1787713215.26 total 820.36 last 27.49) - open/close 18057 (time 1787713225.32 total 830.42 last 21.97) - open/close 18246 (time 1787713236.17 total 841.27 last 17.42) - open/close 18446 (time 1787713246.17 total 851.27 last 20.00) - open/close 18666 (time 1787713256.18 total 861.29 last 21.97) - open/close 18902 (time 1787713267.00 total 872.11 last 21.82) - open/close 19139 (time 1787713277.64 total 882.75 last 22.27) - open/close 19385 (time 1787713287.64 total 892.75 last 24.60) - open/close 19623 (time 1787713297.65 total 902.75 last 23.79) - open/close 19888 (time 1787713307.69 total 912.79 last 26.39) - open/close 20000 (time 1787713315.94 total 921.04 last 13.58) - open/close 20235 (time 1787713325.98 total 931.08 last 23.40) - open/close 20568 (time 1787713335.98 total 941.09 last 33.28) - open/close 20804 (time 1787713345.99 total 951.10 last 23.57) - open/close 21154 (time 1787713356.00 total 961.10 last 35.00) - open/close 21994 (time 1787713366.01 total 971.11 last 83.90) - open/close 22936 (time 1787713376.01 total 981.12 last 94.17) - open/close 23696 (time 1787713386.02 total 991.13 last 75.92) - open/close 24573 (time 1787713396.04 total 1001.14 last 87.57) - open/close 25418 (time 1787713406.05 total 1011.15 last 84.38) - open/close 26172 (time 1787713416.06 total 1021.16 last 75.34) - open/close 27063 (time 1787713426.06 total 1031.17 last 89.07) - open/close 27984 (time 1787713436.07 total 1041.17 last 92.05) - open/close 28853 (time 1787713446.07 total 1051.18 last 86.85) - open/close 29725 (time 1787713456.08 total 1061.18 last 87.15) - open/close 30000 (time 1787713458.55 total 1063.65 last 111.31) - open/close 30782 (time 1787713468.55 total 1073.66 last 78.16) - open/close 31406 (time 1787713478.56 total 1083.66 last 62.36) - open/close 32274 (time 1787713488.57 total 1093.67 last 86.76) - open/close 33078 (time 1787713498.57 total 1103.67 last 80.38) - open/close 33831 (time 1787713508.57 total 1113.68 last 75.25) - open/close 34505 (time 1787713518.58 total 1123.68 last 67.38) - open/close 35195 (time 1787713528.59 total 1133.69 last 68.93) - open/close 35887 (time 1787713538.61 total 1143.71 last 69.07) - open/close 36484 (time 1787713548.61 total 1153.71 last 59.69) - open/close 37310 (time 1787713558.61 total 1163.71 last 82.57) - open/close 38204 (time 1787713568.63 total 1173.73 last 89.27) - open/close 38955 (time 1787713578.63 total 1183.74 last 75.05) - open/close 39860 (time 1787713588.64 total 1193.75 last 90.41) - open/close 40723 (time 1787713598.65 total 1203.75 last 86.25) - open/close 41541 (time 1787713608.65 total 1213.75 last 81.79) - open/close 42337 (time 1787713618.66 total 1223.76 last 79.52) - open/close 43134 (time 1787713628.66 total 1233.77 last 79.68) - open/close 43924 (time 1787713638.67 total 1243.77 last 78.94) - open/close 44789 (time 1787713648.67 total 1253.78 last 86.47) - open/close 45600 (time 1787713658.69 total 1263.79 last 81.01) - open/close 46472 (time 1787713668.69 total 1273.79 last 87.18) - open/close 47386 (time 1787713678.69 total 1283.80 last 91.35) - open/close 48228 (time 1787713688.70 total 1293.80 last 84.15) - open/close 49016 (time 1787713698.70 total 1303.81 last 78.79) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1312.49 seconds: 37.81 ops/second - unlinked 0 (time 1787713711 ; total 0 ; last 0) - unlinked 10000 (time 1787713835 ; total 124 ; last 124) - unlinked 20000 (time 1787713951 ; total 240 ; last 116) - unlinked 30000 (time 1787714072 ; total 361 ; last 121) - unlinked 40000 (time 1787714192 ; total 481 ; last 120) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 561 seconds: 88.468803 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg316-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg316-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg316-server stop mds service on oleg316-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg316-server stop mds service on oleg316-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg316-server unloading modules via unload_modules_local on: 'oleg316-server' oleg316-server: oleg316-server.virtnet: executing unload_modules_local oleg316-server: modules unloaded. pdsh@oleg316-client: oleg316-client: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-server: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-client: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-server: ssh exited with exit code 2