== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:06:31 (1781229991) start mds service on oleg136-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg136-server' oleg136-server: oleg136-server.virtnet: executing load_modules_local oleg136-server: Loading modules from /home/green/git/lustre-release/lustre oleg136-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg136-server: detected 4 online CPUs by sysfs oleg136-server: MODOPTS_LIBCFS= oleg136-server: Force libcfs to create 2 CPU partitions oleg136-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg136-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg136-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg136-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg136-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-OST0000 oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 110 (time 1781230095.08 total 10.28 last 10.70) - open/close 305 (time 1781230105.09 total 20.29 last 19.48) - open/close 413 (time 1781230115.36 total 30.57 last 10.51) - open/close 532 (time 1781230125.37 total 40.57 last 11.89) - open/close 696 (time 1781230135.83 total 51.03 last 15.68) - open/close 823 (time 1781230145.84 total 61.04 last 12.69) - open/close 1119 (time 1781230156.43 total 71.63 last 27.96) - open/close 1381 (time 1781230166.44 total 81.65 last 26.15) - open/close 1575 (time 1781230176.47 total 91.68 last 19.35) - open/close 1793 (time 1781230186.91 total 102.11 last 20.89) - open/close 1985 (time 1781230197.91 total 113.12 last 17.45) - open/close 2083 (time 1781230208.60 total 123.80 last 9.17) - open/close 2336 (time 1781230218.60 total 133.81 last 25.30) - open/close 2598 (time 1781230228.61 total 143.82 last 26.17) - open/close 2749 (time 1781230238.63 total 153.84 last 15.07) - open/close 3031 (time 1781230248.92 total 164.12 last 27.42) - open/close 3226 (time 1781230258.97 total 174.18 last 19.39) - open/close 3395 (time 1781230268.97 total 184.18 last 16.90) - open/close 3665 (time 1781230278.98 total 194.18 last 26.99) - open/close 3815 (time 1781230289.94 total 205.15 last 13.68) - open/close 4029 (time 1781230302.17 total 217.38 last 17.50) - open/close 4140 (time 1781230312.99 total 228.19 last 10.26) - open/close 4223 (time 1781230323.75 total 238.96 last 7.71) - open/close 4389 (time 1781230333.78 total 248.98 last 16.56) - open/close 4577 (time 1781230344.23 total 259.43 last 17.99) - open/close 4782 (time 1781230354.27 total 269.48 last 20.41) - open/close 5017 (time 1781230365.21 total 280.42 last 21.49) - open/close 5296 (time 1781230375.21 total 290.42 last 27.90) - open/close 5554 (time 1781230385.45 total 300.65 last 25.21) - open/close 5832 (time 1781230396.33 total 311.54 last 25.54) - open/close 6028 (time 1781230406.80 total 322.01 last 18.72) - open/close 6284 (time 1781230416.81 total 332.01 last 25.59) - open/close 6580 (time 1781230426.84 total 342.04 last 29.52) - open/close 6807 (time 1781230437.69 total 352.89 last 20.92) - open/close 7018 (time 1781230448.24 total 363.44 last 20.00) - open/close 7175 (time 1781230458.75 total 373.95 last 14.94) - open/close 7404 (time 1781230469.54 total 384.75 last 21.21) - open/close 7650 (time 1781230480.25 total 395.45 last 22.98) - open/close 7849 (time 1781230490.34 total 405.55 last 19.71) - open/close 8088 (time 1781230500.85 total 416.05 last 22.75) - open/close 8364 (time 1781230511.72 total 426.93 last 25.38) - open/close 8655 (time 1781230521.73 total 436.94 last 29.07) - open/close 8845 (time 1781230531.75 total 446.96 last 18.97) - open/close 8961 (time 1781230543.18 total 458.39 last 10.15) - open/close 9179 (time 1781230553.38 total 468.59 last 21.37) - open/close 9392 (time 1781230563.39 total 478.59 last 21.29) - open/close 9558 (time 1781230573.40 total 488.61 last 16.57) - open/close 9768 (time 1781230584.43 total 499.63 last 19.05) - open/close 9907 (time 1781230595.74 total 510.95 last 12.29) - open/close 10000 (time 1781230598.31 total 513.51 last 36.25) - open/close 10248 (time 1781230609.24 total 524.45 last 22.68) - open/close 10418 (time 1781230619.82 total 535.02 last 16.07) - open/close 10625 (time 1781230630.62 total 545.83 last 19.16) - open/close 10770 (time 1781230640.64 total 555.85 last 14.47) - open/close 11003 (time 1781230653.24 total 568.45 last 18.49) - open/close 11223 (time 1781230663.26 total 578.46 last 21.97) - open/close 11422 (time 1781230673.26 total 588.47 last 19.88) - open/close 11679 (time 1781230683.27 total 598.47 last 25.69) - open/close 11862 (time 1781230693.42 total 608.62 last 18.03) - open/close 12159 (time 1781230704.15 total 619.35 last 27.68) - open/close 12446 (time 1781230714.16 total 629.37 last 28.66) - open/close 12664 (time 1781230724.20 total 639.40 last 21.72) - open/close 12943 (time 1781230734.38 total 649.58 last 27.40) - open/close 13157 (time 1781230744.41 total 659.61 last 21.34) - open/close 13414 (time 1781230754.42 total 669.63 last 25.66) - open/close 13670 (time 1781230764.73 total 679.94 last 24.84) - open/close 13746 (time 1781230774.77 total 689.97 last 7.57) - open/close 13993 (time 1781230784.77 total 699.98 last 24.69) - open/close 14135 (time 1781230797.49 total 712.69 last 11.17) - open/close 14389 (time 1781230808.35 total 723.56 last 23.38) - open/close 14657 (time 1781230818.36 total 733.56 last 26.79) - open/close 14912 (time 1781230828.94 total 744.14 last 24.10) - open/close 15202 (time 1781230839.67 total 754.88 last 27.02) - open/close 15399 (time 1781230849.87 total 765.08 last 19.31) - open/close 15443 (time 1781230860.57 total 775.78 last 4.11) - open/close 15476 (time 1781230871.91 total 787.11 last 2.91) - open/close 15602 (time 1781230882.09 total 797.30 last 12.37) - open/close 15702 (time 1781230892.11 total 807.32 last 9.98) - open/close 15784 (time 1781230903.84 total 819.05 last 6.99) - open/close 15907 (time 1781230914.91 total 830.12 last 11.11) - open/close 16042 (time 1781230925.39 total 840.59 last 12.89) - open/close 16260 (time 1781230935.41 total 850.62 last 21.74) - open/close 16438 (time 1781230945.42 total 860.63 last 17.78) - open/close 16705 (time 1781230955.45 total 870.65 last 26.63) - open/close 16851 (time 1781230965.45 total 880.66 last 14.60) - open/close 17005 (time 1781230975.47 total 890.67 last 15.38) - open/close 17156 (time 1781230985.70 total 900.90 last 14.75) - open/close 17316 (time 1781230996.38 total 911.58 last 14.98) - open/close 17507 (time 1781231006.38 total 921.59 last 19.09) - open/close 17701 (time 1781231016.39 total 931.59 last 19.39) - open/close 17884 (time 1781231026.84 total 942.04 last 17.51) - open/close 18081 (time 1781231037.33 total 952.54 last 18.77) - open/close 18324 (time 1781231048.66 total 963.86 last 21.46) - open/close 18487 (time 1781231058.78 total 973.98 last 16.11) - open/close 18726 (time 1781231069.25 total 984.46 last 22.81) - open/close 18968 (time 1781231079.48 total 994.69 last 23.65) - open/close 19161 (time 1781231089.49 total 1004.70 last 19.28) - open/close 19323 (time 1781231099.52 total 1014.72 last 16.16) - open/close 19511 (time 1781231109.52 total 1024.73 last 18.79) - open/close 20000 (time 1781231119.32 total 1034.53 last 49.92) - open/close 20879 (time 1781231129.32 total 1044.53 last 87.87) - open/close 21778 (time 1781231139.32 total 1054.53 last 89.90) - open/close 22655 (time 1781231149.34 total 1064.54 last 87.60) - open/close 23464 (time 1781231159.34 total 1074.54 last 80.88) - open/close 24460 (time 1781231169.35 total 1084.55 last 99.52) - open/close 25426 (time 1781231179.35 total 1094.55 last 96.59) - open/close 26229 (time 1781231189.36 total 1104.56 last 80.21) - open/close 27016 (time 1781231199.36 total 1114.57 last 78.65) - open/close 27848 (time 1781231209.37 total 1124.58 last 83.14) - open/close 28596 (time 1781231219.38 total 1134.58 last 74.74) - open/close 29408 (time 1781231229.38 total 1144.59 last 81.16) - open/close 30000 (time 1781231235.51 total 1150.71 last 96.71) - open/close 30905 (time 1781231245.51 total 1160.71 last 90.47) - open/close 31580 (time 1781231255.51 total 1170.71 last 67.50) - open/close 32177 (time 1781231265.52 total 1180.73 last 59.61) - open/close 32724 (time 1781231275.53 total 1190.74 last 54.64) - open/close 33415 (time 1781231285.54 total 1200.75 last 69.04) - open/close 34058 (time 1781231295.55 total 1210.75 last 64.28) - open/close 34792 (time 1781231305.57 total 1220.77 last 73.24) - open/close 35277 (time 1781231315.57 total 1230.78 last 48.47) - open/close 36016 (time 1781231325.58 total 1240.79 last 73.86) - open/close 36702 (time 1781231335.58 total 1250.79 last 68.60) - open/close 37692 (time 1781231345.58 total 1260.79 last 98.97) - open/close 38522 (time 1781231355.59 total 1270.80 last 82.93) - open/close 39340 (time 1781231365.60 total 1280.80 last 81.74) - open/close 40000 (time 1781231373.36 total 1288.56 last 85.05) - open/close 40950 (time 1781231383.37 total 1298.57 last 94.93) - open/close 41865 (time 1781231393.37 total 1308.57 last 91.47) - open/close 42866 (time 1781231403.38 total 1318.58 last 100.02) - open/close 43776 (time 1781231413.39 total 1328.60 last 90.88) - open/close 44504 (time 1781231423.40 total 1338.60 last 72.76) - open/close 45193 (time 1781231433.40 total 1348.61 last 68.85) - open/close 46021 (time 1781231443.41 total 1358.61 last 82.76) - open/close 46959 (time 1781231453.41 total 1368.62 last 93.74) - open/close 47808 (time 1781231463.42 total 1378.63 last 84.85) - open/close 48645 (time 1781231473.42 total 1388.63 last 83.69) - open/close 49474 (time 1781231483.43 total 1398.63 last 82.84) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1400.49 seconds: 35.44 ops/second - unlinked 0 (time 1781231490 ; total 0 ; last 0) - unlinked 10000 (time 1781231599 ; total 109 ; last 109) - unlinked 20000 (time 1781231710 ; total 220 ; last 111) - unlinked 30000 (time 1781231780 ; total 290 ; last 70) - unlinked 40000 (time 1781231852 ; total 362 ; last 72) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 426 seconds: 116.504692 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg136-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg136-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server stop mds service on oleg136-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server stop mds service on oleg136-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server unloading modules via unload_modules_local on: 'oleg136-server' oleg136-server: oleg136-server.virtnet: executing unload_modules_local oleg136-server: modules unloaded. pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2