== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:44:40 (1787870680) start mds service on oleg121-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg121-server' oleg121-server: oleg121-server.virtnet: executing load_modules_local oleg121-server: Loading modules from /home/green/git/lustre-release/lustre oleg121-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg121-server: detected 4 online CPUs by sysfs oleg121-server: MODOPTS_LIBCFS= oleg121-server: Force libcfs to create 2 CPU partitions oleg121-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg121-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg121-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg121-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg121-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-OST0000 oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 150 (time 1787870795.93 total 10.02 last 14.97) - open/close 359 (time 1787870807.83 total 21.92 last 17.56) - open/close 612 (time 1787870818.58 total 32.67 last 23.53) - open/close 878 (time 1787870829.22 total 43.31 last 25.00) - open/close 1170 (time 1787870839.66 total 53.75 last 27.97) - open/close 1432 (time 1787870850.28 total 64.37 last 24.67) - open/close 1631 (time 1787870860.44 total 74.53 last 19.58) - open/close 1866 (time 1787870870.45 total 84.54 last 23.48) - open/close 2120 (time 1787870881.29 total 95.38 last 23.43) - open/close 2329 (time 1787870891.99 total 106.08 last 19.54) - open/close 2516 (time 1787870902.00 total 116.09 last 18.68) - open/close 2620 (time 1787870912.15 total 126.24 last 10.24) - open/close 2881 (time 1787870922.99 total 137.08 last 24.08) - open/close 3116 (time 1787870933.03 total 147.12 last 23.41) - open/close 3386 (time 1787870943.95 total 158.04 last 24.73) - open/close 3642 (time 1787870954.01 total 168.10 last 25.45) - open/close 3886 (time 1787870964.23 total 178.32 last 23.88) - open/close 4128 (time 1787870974.75 total 188.84 last 23.00) - open/close 4359 (time 1787870984.76 total 198.85 last 23.08) - open/close 4561 (time 1787870994.78 total 208.87 last 20.16) - open/close 4814 (time 1787871005.65 total 219.74 last 23.28) - open/close 5014 (time 1787871016.53 total 230.62 last 18.38) - open/close 5159 (time 1787871026.55 total 240.64 last 14.48) - open/close 5454 (time 1787871037.32 total 251.42 last 27.37) - open/close 5748 (time 1787871047.33 total 261.42 last 29.39) - open/close 5987 (time 1787871057.51 total 271.60 last 23.48) - open/close 6204 (time 1787871068.46 total 282.55 last 19.81) - open/close 6494 (time 1787871079.51 total 293.60 last 26.25) - open/close 6775 (time 1787871090.45 total 304.54 last 25.67) - open/close 6978 (time 1787871100.46 total 314.55 last 20.29) - open/close 7131 (time 1787871110.46 total 324.55 last 15.29) - open/close 7362 (time 1787871120.48 total 334.58 last 23.05) - open/close 7648 (time 1787871130.49 total 344.58 last 28.58) - open/close 7898 (time 1787871140.51 total 354.60 last 24.96) - open/close 8056 (time 1787871151.07 total 365.16 last 14.96) - open/close 8126 (time 1787871161.10 total 375.19 last 6.98) - open/close 8399 (time 1787871171.11 total 385.20 last 27.26) - open/close 8629 (time 1787871181.46 total 395.55 last 22.22) - open/close 8922 (time 1787871191.47 total 405.56 last 29.28) - open/close 9070 (time 1787871204.12 total 418.22 last 11.70) - open/close 9341 (time 1787871215.08 total 429.17 last 24.73) - open/close 9576 (time 1787871225.10 total 439.20 last 23.45) - open/close 9717 (time 1787871236.13 total 450.22 last 12.79) - open/close 9996 (time 1787871247.00 total 461.09 last 25.66) - open/close 10258 (time 1787871257.00 total 471.09 last 26.20) - open/close 10482 (time 1787871267.01 total 481.10 last 22.39) - open/close 10753 (time 1787871277.93 total 492.03 last 24.80) - open/close 11048 (time 1787871287.94 total 502.03 last 29.48) - open/close 11276 (time 1787871299.88 total 513.97 last 19.10) - open/close 11509 (time 1787871310.36 total 524.45 last 22.24) - open/close 11739 (time 1787871320.36 total 534.45 last 22.99) - open/close 12001 (time 1787871330.37 total 544.46 last 26.17) - open/close 12197 (time 1787871340.38 total 554.47 last 19.59) - open/close 12475 (time 1787871351.18 total 565.27 last 25.74) - open/close 12750 (time 1787871361.19 total 575.28 last 27.47) - open/close 13024 (time 1787871371.25 total 585.34 last 27.25) - open/close 13181 (time 1787871381.55 total 595.64 last 15.23) - open/close 13306 (time 1787871393.62 total 607.71 last 10.36) - open/close 13566 (time 1787871406.24 total 620.33 last 20.60) - open/close 13697 (time 1787871416.25 total 630.34 last 13.09) - open/close 13940 (time 1787871426.26 total 640.35 last 24.27) - open/close 14242 (time 1787871436.26 total 650.36 last 30.17) - open/close 14524 (time 1787871446.27 total 660.36 last 28.19) - open/close 14696 (time 1787871456.78 total 670.87 last 16.37) - open/close 14988 (time 1787871466.78 total 680.87 last 29.20) - open/close 15144 (time 1787871477.99 total 692.08 last 13.92) - open/close 15381 (time 1787871488.47 total 702.56 last 22.61) - open/close 15642 (time 1787871498.48 total 712.57 last 26.08) - open/close 15881 (time 1787871508.72 total 722.81 last 23.35) - open/close 16089 (time 1787871519.64 total 733.74 last 19.03) - open/close 16302 (time 1787871529.67 total 743.76 last 21.25) - open/close 16557 (time 1787871540.05 total 754.14 last 24.57) - open/close 16745 (time 1787871550.56 total 764.65 last 17.89) - open/close 17030 (time 1787871560.58 total 774.67 last 28.44) - open/close 17316 (time 1787871570.60 total 784.69 last 28.55) - open/close 17520 (time 1787871582.29 total 796.38 last 17.44) - open/close 17775 (time 1787871592.92 total 807.01 last 24.00) - open/close 18021 (time 1787871604.17 total 818.27 last 21.85) - open/close 18293 (time 1787871614.87 total 828.96 last 25.44) - open/close 18583 (time 1787871624.88 total 838.97 last 28.96) - open/close 18861 (time 1787871635.35 total 849.44 last 26.55) - open/close 19152 (time 1787871645.98 total 860.07 last 27.38) - open/close 19450 (time 1787871656.28 total 870.37 last 28.93) - open/close 19630 (time 1787871666.92 total 881.01 last 16.91) - open/close 19867 (time 1787871676.98 total 891.08 last 23.56) - open/close 20000 (time 1787871685.90 total 899.99 last 14.92) - open/close 20234 (time 1787871696.54 total 910.63 last 21.98) - open/close 20581 (time 1787871706.60 total 920.69 last 34.49) - open/close 20888 (time 1787871717.42 total 931.51 last 28.37) - open/close 21042 (time 1787871727.43 total 941.52 last 15.39) - open/close 21265 (time 1787871737.45 total 951.54 last 22.25) - open/close 21846 (time 1787871747.45 total 961.54 last 58.07) - open/close 22633 (time 1787871757.47 total 971.57 last 78.54) - open/close 23454 (time 1787871767.49 total 981.58 last 81.96) - open/close 24437 (time 1787871777.51 total 991.60 last 98.12) - open/close 25287 (time 1787871787.52 total 1001.61 last 84.95) - open/close 26180 (time 1787871797.52 total 1011.61 last 89.24) - open/close 27240 (time 1787871807.53 total 1021.62 last 105.89) - open/close 28044 (time 1787871817.54 total 1031.63 last 80.35) - open/close 28879 (time 1787871827.54 total 1041.64 last 83.46) - open/close 29855 (time 1787871837.54 total 1051.64 last 97.59) - open/close 30653 (time 1787871847.55 total 1061.64 last 79.77) - open/close 31510 (time 1787871857.56 total 1071.65 last 85.64) - open/close 32242 (time 1787871867.56 total 1081.65 last 73.18) - open/close 33128 (time 1787871877.56 total 1091.65 last 88.58) - open/close 34133 (time 1787871887.56 total 1101.65 last 100.50) - open/close 34967 (time 1787871897.57 total 1111.66 last 83.35) - open/close 35867 (time 1787871907.57 total 1121.67 last 89.94) - open/close 36851 (time 1787871917.58 total 1131.67 last 98.35) - open/close 37846 (time 1787871927.58 total 1141.67 last 99.46) - open/close 38941 (time 1787871937.59 total 1151.68 last 109.48) - open/close 40000 (time 1787871947.53 total 1161.62 last 106.49) - open/close 40797 (time 1787871957.54 total 1171.63 last 79.63) - open/close 41467 (time 1787871967.54 total 1181.63 last 66.99) - open/close 42507 (time 1787871977.54 total 1191.63 last 103.97) - open/close 43403 (time 1787871987.55 total 1201.64 last 89.53) - open/close 44272 (time 1787871997.55 total 1211.64 last 86.89) - open/close 45142 (time 1787872007.56 total 1221.65 last 86.91) - open/close 46053 (time 1787872017.57 total 1231.66 last 91.03) - open/close 46975 (time 1787872027.57 total 1241.66 last 92.18) - open/close 47794 (time 1787872037.59 total 1251.68 last 81.73) - open/close 48624 (time 1787872047.61 total 1261.70 last 82.85) - open/close 49449 (time 1787872057.61 total 1271.71 last 82.47) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49630) error: No space left on device total: 49630 open/close in 1274.15 seconds: 38.95 ops/second - unlinked 0 (time 1787872064 ; total 0 ; last 0) - unlinked 10000 (time 1787872177 ; total 113 ; last 113) - unlinked 20000 (time 1787872290 ; total 226 ; last 113) - unlinked 30000 (time 1787872355 ; total 291 ; last 65) - unlinked 40000 (time 1787872417 ; total 353 ; last 62) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49630) error: No such file or directory total: 49630 unlinks in 415 seconds: 119.590363 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg121-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg121-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg121-server stop mds service on oleg121-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg121-server stop mds service on oleg121-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg121-server unloading modules via unload_modules_local on: 'oleg121-server' oleg121-server: oleg121-server.virtnet: executing unload_modules_local oleg121-server: modules unloaded. pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-client: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-client: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2