== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:48:29 (1778618909) start mds service on oleg410-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg410-server' oleg410-server: oleg410-server.virtnet: executing load_modules_local oleg410-server: Loading modules from /home/green/git/lustre-release/lustre oleg410-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg410-server: detected 4 online CPUs by sysfs oleg410-server: MODOPTS_LIBCFS= oleg410-server: Force libcfs to create 2 CPU partitions oleg410-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg410-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg410-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg410-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-OST0000 oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 131 (time 1778618997.02 total 11.54 last 11.35) - open/close 361 (time 1778619007.70 total 22.23 last 21.53) - open/close 595 (time 1778619018.08 total 32.60 last 22.55) - open/close 776 (time 1778619028.08 total 42.61 last 18.09) - open/close 1003 (time 1778619039.08 total 53.60 last 20.65) - open/close 1157 (time 1778619049.10 total 63.62 last 15.37) - open/close 1295 (time 1778619059.60 total 74.13 last 13.14) - open/close 1367 (time 1778619069.61 total 84.13 last 7.20) - open/close 1564 (time 1778619079.90 total 94.42 last 19.15) - open/close 1758 (time 1778619089.90 total 104.42 last 19.40) - open/close 2004 (time 1778619100.19 total 114.72 last 23.90) - open/close 2185 (time 1778619111.20 total 125.72 last 16.45) - open/close 2358 (time 1778619121.83 total 136.36 last 16.27) - open/close 2515 (time 1778619132.75 total 147.28 last 14.38) - open/close 2617 (time 1778619142.82 total 157.34 last 10.13) - open/close 2803 (time 1778619153.37 total 167.90 last 17.62) - open/close 2992 (time 1778619163.49 total 178.01 last 18.69) - open/close 3125 (time 1778619173.49 total 188.02 last 13.30) - open/close 3361 (time 1778619185.58 total 200.11 last 19.52) - open/close 3522 (time 1778619196.19 total 210.71 last 15.18) - open/close 3700 (time 1778619206.20 total 220.73 last 17.78) - open/close 3922 (time 1778619216.22 total 230.74 last 22.16) - open/close 4036 (time 1778619226.97 total 241.50 last 10.60) - open/close 4208 (time 1778619237.02 total 251.54 last 17.12) - open/close 4392 (time 1778619247.02 total 261.55 last 18.39) - open/close 4613 (time 1778619257.03 total 271.55 last 22.09) - open/close 4868 (time 1778619267.10 total 281.62 last 25.32) - open/close 5088 (time 1778619277.11 total 291.64 last 21.97) - open/close 5240 (time 1778619287.58 total 302.10 last 14.52) - open/close 5338 (time 1778619297.87 total 312.40 last 9.52) - open/close 5591 (time 1778619307.89 total 322.42 last 25.25) - open/close 5789 (time 1778619317.91 total 332.43 last 19.77) - open/close 5982 (time 1778619327.91 total 342.44 last 19.30) - open/close 6246 (time 1778619338.80 total 353.33 last 24.25) - open/close 6505 (time 1778619348.83 total 363.36 last 25.82) - open/close 6663 (time 1778619359.49 total 374.01 last 14.82) - open/close 6840 (time 1778619369.50 total 384.03 last 17.68) - open/close 6918 (time 1778619379.50 total 394.03 last 7.80) - open/close 7120 (time 1778619389.71 total 404.24 last 19.79) - open/close 7347 (time 1778619399.73 total 414.26 last 22.65) - open/close 7624 (time 1778619410.25 total 424.78 last 26.33) - open/close 7767 (time 1778619420.57 total 435.10 last 13.86) - open/close 7999 (time 1778619430.57 total 445.10 last 23.20) - open/close 8183 (time 1778619441.49 total 456.02 last 16.85) - open/close 8409 (time 1778619451.52 total 466.05 last 22.53) - open/close 8645 (time 1778619462.17 total 476.70 last 22.16) - open/close 8837 (time 1778619475.03 total 489.55 last 14.94) - open/close 9052 (time 1778619485.05 total 499.57 last 21.45) - open/close 9300 (time 1778619495.84 total 510.36 last 22.99) - open/close 9513 (time 1778619506.89 total 521.41 last 19.28) - open/close 9766 (time 1778619517.65 total 532.17 last 23.51) - open/close 10000 (time 1778619526.62 total 541.15 last 26.07) - open/close 10197 (time 1778619536.98 total 551.51 last 19.02) - open/close 10268 (time 1778619546.98 total 561.51 last 7.10) - open/close 10461 (time 1778619557.27 total 571.80 last 18.77) - open/close 10656 (time 1778619567.45 total 581.98 last 19.15) - open/close 10779 (time 1778619578.45 total 592.97 last 11.19) - open/close 10955 (time 1778619588.82 total 603.34 last 16.97) - open/close 11174 (time 1778619599.13 total 613.65 last 21.24) - open/close 11428 (time 1778619609.14 total 623.66 last 25.38) - open/close 11658 (time 1778619619.61 total 634.14 last 21.95) - open/close 11853 (time 1778619630.49 total 645.02 last 17.92) - open/close 12104 (time 1778619641.19 total 655.71 last 23.48) - open/close 12328 (time 1778619651.20 total 665.72 last 22.37) - open/close 12543 (time 1778619661.59 total 676.11 last 20.70) - open/close 12693 (time 1778619671.60 total 686.12 last 14.98) - open/close 12868 (time 1778619681.62 total 696.14 last 17.46) - open/close 13003 (time 1778619692.24 total 706.76 last 12.71) - open/close 13229 (time 1778619702.88 total 717.41 last 21.23) - open/close 13377 (time 1778619712.89 total 727.42 last 14.78) - open/close 13568 (time 1778619723.46 total 737.98 last 18.08) - open/close 13717 (time 1778619733.48 total 748.01 last 14.86) - open/close 13893 (time 1778619743.48 total 758.01 last 17.59) - open/close 14135 (time 1778619753.49 total 768.01 last 24.20) - open/close 14320 (time 1778619764.06 total 778.59 last 17.50) - open/close 14555 (time 1778619775.06 total 789.58 last 21.37) - open/close 14786 (time 1778619786.10 total 800.62 last 20.93) - open/close 14997 (time 1778619796.12 total 810.64 last 21.05) - open/close 15231 (time 1778619806.47 total 821.00 last 22.60) - open/close 15351 (time 1778619816.49 total 831.01 last 11.98) - open/close 15466 (time 1778619826.49 total 841.02 last 11.50) - open/close 15542 (time 1778619836.84 total 851.37 last 7.34) - open/close 15677 (time 1778619846.84 total 861.37 last 13.50) - open/close 15888 (time 1778619856.85 total 871.38 last 21.08) - open/close 16001 (time 1778619867.95 total 882.48 last 10.18) - open/close 16229 (time 1778619878.16 total 892.68 last 22.34) - open/close 16511 (time 1778619888.29 total 902.81 last 27.84) - open/close 16730 (time 1778619898.40 total 912.93 last 21.66) - open/close 16971 (time 1778619908.89 total 923.42 last 22.97) - open/close 17124 (time 1778619919.00 total 933.52 last 15.14) - open/close 17320 (time 1778619929.00 total 943.53 last 19.59) - open/close 17455 (time 1778619939.85 total 954.38 last 12.44) - open/close 17726 (time 1778619950.49 total 965.02 last 25.46) - open/close 17915 (time 1778619960.86 total 975.38 last 18.23) - open/close 18023 (time 1778619971.28 total 985.81 last 10.36) - open/close 18469 (time 1778619981.29 total 995.81 last 44.57) - open/close 19130 (time 1778619991.29 total 1005.82 last 66.05) - open/close 19735 (time 1778620001.29 total 1015.82 last 60.50) - open/close 20000 (time 1778620005.13 total 1019.65 last 69.10) - open/close 20641 (time 1778620015.14 total 1029.66 last 64.04) - open/close 21287 (time 1778620025.14 total 1039.67 last 64.57) - open/close 21964 (time 1778620035.16 total 1049.69 last 67.59) - open/close 22576 (time 1778620045.16 total 1059.69 last 61.19) - open/close 23248 (time 1778620055.16 total 1069.69 last 67.17) - open/close 23908 (time 1778620065.17 total 1079.70 last 65.95) - open/close 24580 (time 1778620075.17 total 1089.70 last 67.19) - open/close 25234 (time 1778620085.17 total 1099.70 last 65.39) - open/close 25861 (time 1778620095.18 total 1109.71 last 62.67) - open/close 26452 (time 1778620105.19 total 1119.72 last 59.04) - open/close 27060 (time 1778620115.20 total 1129.72 last 60.74) - open/close 27647 (time 1778620125.20 total 1139.72 last 58.70) - open/close 28281 (time 1778620135.21 total 1149.74 last 63.33) - open/close 28812 (time 1778620145.21 total 1159.74 last 53.09) - open/close 29440 (time 1778620155.22 total 1169.74 last 62.76) - open/close 30000 (time 1778620163.58 total 1178.11 last 66.97) - open/close 30628 (time 1778620173.59 total 1188.12 last 62.74) - open/close 31247 (time 1778620183.60 total 1198.13 last 61.82) - open/close 31899 (time 1778620193.61 total 1208.13 last 65.18) - open/close 32525 (time 1778620203.61 total 1218.14 last 62.58) - open/close 33182 (time 1778620213.62 total 1228.15 last 65.63) - open/close 33821 (time 1778620223.64 total 1238.16 last 63.79) - open/close 34452 (time 1778620233.64 total 1248.17 last 63.09) - open/close 35132 (time 1778620243.64 total 1258.17 last 67.99) - open/close 35734 (time 1778620253.65 total 1268.18 last 60.12) - open/close 36363 (time 1778620263.66 total 1278.18 last 62.87) - open/close 37051 (time 1778620273.66 total 1288.19 last 68.77) - open/close 37685 (time 1778620283.73 total 1298.25 last 62.99) - open/close 38325 (time 1778620293.74 total 1308.26 last 63.94) - open/close 38880 (time 1778620303.76 total 1318.29 last 55.37) - open/close 39474 (time 1778620313.77 total 1328.30 last 59.32) - open/close 40000 (time 1778620322.58 total 1337.11 last 59.71) - open/close 40678 (time 1778620332.60 total 1347.12 last 67.69) - open/close 41329 (time 1778620342.60 total 1357.13 last 65.06) - open/close 41949 (time 1778620352.61 total 1367.14 last 61.97) - open/close 42547 (time 1778620362.61 total 1377.14 last 59.78) - open/close 43117 (time 1778620372.61 total 1387.14 last 56.99) - open/close 43781 (time 1778620382.61 total 1397.14 last 66.40) - open/close 44362 (time 1778620392.64 total 1407.17 last 57.92) - open/close 44942 (time 1778620402.66 total 1417.19 last 57.90) - open/close 45521 (time 1778620412.67 total 1427.20 last 57.84) - open/close 46125 (time 1778620422.69 total 1437.22 last 60.27) - open/close 46771 (time 1778620432.70 total 1447.23 last 64.56) - open/close 47424 (time 1778620442.71 total 1457.24 last 65.23) - open/close 48094 (time 1778620452.75 total 1467.28 last 66.72) - open/close 48694 (time 1778620462.77 total 1477.30 last 59.88) - open/close 49376 (time 1778620472.80 total 1487.32 last 68.04) total: 50000 open/close in 1496.27 seconds: 33.42 ops/second - unlinked 0 (time 1778620486 ; total 0 ; last 0) - unlinked 10000 (time 1778620653 ; total 167 ; last 167) - unlinked 20000 (time 1778620820 ; total 334 ; last 167) - unlinked 30000 (time 1778620988 ; total 502 ; last 168) - unlinked 40000 (time 1778621155 ; total 669 ; last 167) total: 50000 unlinks in 834 seconds: 59.952038 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg410-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg410-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg410-server stop mds service on oleg410-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg410-server unloading modules via unload_modules_local on: 'oleg410-server' oleg410-server: oleg410-server.virtnet: executing unload_modules_local oleg410-server: modules unloaded. pdsh@oleg410-client: oleg410-client: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-server: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-client: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-server: ssh exited with exit code 2