== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:52:23 (1787140343) start mds service on oleg451-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing load_modules_local oleg451-server: Loading modules from /home/green/git/lustre-release/lustre oleg451-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg451-server: detected 4 online CPUs by sysfs oleg451-server: MODOPTS_LIBCFS= oleg451-server: Force libcfs to create 2 CPU partitions oleg451-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg451-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg451-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg451-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 97 (time 1787140435.81 total 10.54 last 9.21) - open/close 219 (time 1787140446.51 total 21.24 last 11.40) - open/close 413 (time 1787140456.72 total 31.45 last 19.00) - open/close 630 (time 1787140466.73 total 41.45 last 21.68) - open/close 900 (time 1787140476.74 total 51.47 last 26.96) - open/close 1097 (time 1787140487.05 total 61.77 last 19.12) - open/close 1295 (time 1787140497.57 total 72.29 last 18.82) - open/close 1384 (time 1787140508.20 total 82.93 last 8.37) - open/close 1499 (time 1787140518.20 total 92.93 last 11.50) - open/close 1642 (time 1787140529.16 total 103.89 last 13.05) - open/close 1713 (time 1787140539.17 total 113.90 last 7.09) - open/close 1814 (time 1787140550.13 total 124.86 last 9.22) - open/close 1962 (time 1787140560.14 total 134.87 last 14.78) - open/close 2051 (time 1787140570.17 total 144.89 last 8.88) - open/close 2269 (time 1787140580.18 total 154.90 last 21.78) - open/close 2510 (time 1787140590.54 total 165.27 last 23.24) - open/close 2650 (time 1787140600.80 total 175.53 last 13.65) - open/close 2779 (time 1787140611.08 total 185.81 last 12.55) - open/close 2880 (time 1787140621.71 total 196.43 last 9.51) - open/close 2956 (time 1787140634.06 total 208.79 last 6.15) - open/close 3098 (time 1787140644.83 total 219.55 last 13.19) - open/close 3256 (time 1787140654.83 total 229.55 last 15.80) - open/close 3391 (time 1787140666.88 total 241.61 last 11.20) - open/close 3530 (time 1787140677.44 total 252.17 last 13.17) - open/close 3637 (time 1787140688.07 total 262.80 last 10.06) - open/close 3793 (time 1787140698.09 total 272.81 last 15.58) - open/close 3946 (time 1787140708.11 total 282.84 last 15.26) - open/close 4021 (time 1787140718.12 total 292.85 last 7.49) - open/close 4206 (time 1787140728.63 total 303.36 last 17.60) - open/close 4323 (time 1787140738.64 total 313.37 last 11.69) - open/close 4544 (time 1787140749.32 total 324.05 last 20.69) - open/close 4768 (time 1787140760.34 total 335.07 last 20.33) - open/close 4989 (time 1787140770.35 total 345.07 last 22.09) - open/close 5107 (time 1787140785.16 total 359.89 last 7.97) - open/close 5240 (time 1787140795.17 total 369.89 last 13.29) - open/close 5440 (time 1787140805.92 total 380.65 last 18.60) - open/close 5664 (time 1787140815.94 total 390.66 last 22.38) - open/close 5866 (time 1787140826.50 total 401.23 last 19.12) - open/close 5966 (time 1787140839.14 total 413.87 last 7.91) - open/close 6234 (time 1787140849.15 total 423.87 last 26.80) - open/close 6351 (time 1787140859.17 total 433.89 last 11.67) - open/close 6446 (time 1787140869.21 total 443.94 last 9.46) - open/close 6542 (time 1787140879.91 total 454.63 last 8.98) - open/close 6658 (time 1787140890.84 total 465.56 last 10.61) - open/close 6836 (time 1787140901.57 total 476.30 last 16.58) - open/close 6948 (time 1787140911.60 total 486.32 last 11.17) - open/close 7063 (time 1787140922.52 total 497.24 last 10.53) - open/close 7189 (time 1787140932.53 total 507.25 last 12.59) - open/close 7255 (time 1787140943.28 total 518.00 last 6.14) - open/close 7392 (time 1787140954.20 total 528.93 last 12.54) - open/close 7553 (time 1787140965.19 total 539.91 last 14.66) - open/close 7621 (time 1787140975.45 total 550.18 last 6.62) - open/close 7812 (time 1787140985.46 total 560.19 last 19.08) - open/close 7937 (time 1787140996.91 total 571.64 last 10.92) - open/close 8063 (time 1787141007.11 total 581.83 last 12.36) - open/close 8170 (time 1787141017.12 total 591.84 last 10.69) - open/close 8260 (time 1787141027.72 total 602.44 last 8.49) - open/close 8320 (time 1787141037.74 total 612.46 last 5.99) - open/close 8508 (time 1787141049.20 total 623.93 last 16.40) - open/close 8686 (time 1787141060.43 total 635.15 last 15.86) - open/close 8938 (time 1787141070.80 total 645.53 last 24.29) - open/close 9082 (time 1787141080.99 total 655.72 last 14.13) - open/close 9267 (time 1787141090.99 total 665.72 last 18.49) - open/close 9445 (time 1787141101.60 total 676.32 last 16.78) - open/close 9733 (time 1787141111.77 total 686.49 last 28.32) - open/close 9968 (time 1787141122.01 total 696.73 last 22.96) - open/close 10000 (time 1787141128.90 total 703.63 last 4.64) - open/close 10115 (time 1787141138.91 total 713.64 last 11.49) - open/close 10304 (time 1787141149.97 total 724.69 last 17.09) - open/close 10522 (time 1787141160.20 total 734.92 last 21.31) - open/close 10769 (time 1787141171.21 total 745.93 last 22.43) - open/close 11024 (time 1787141181.65 total 756.38 last 24.42) - open/close 11185 (time 1787141192.40 total 767.12 last 14.98) - open/close 11445 (time 1787141202.76 total 777.49 last 25.08) - open/close 11642 (time 1787141213.39 total 788.11 last 18.54) - open/close 11855 (time 1787141223.39 total 798.12 last 21.30) - open/close 12092 (time 1787141233.82 total 808.54 last 22.73) - open/close 12351 (time 1787141243.84 total 818.56 last 25.84) - open/close 12521 (time 1787141253.85 total 828.57 last 16.99) - open/close 12743 (time 1787141264.72 total 839.44 last 20.42) - open/close 12974 (time 1787141274.73 total 849.46 last 23.07) - open/close 13169 (time 1787141285.02 total 859.75 last 18.95) - open/close 13413 (time 1787141295.75 total 870.48 last 22.74) - open/close 13607 (time 1787141305.80 total 880.53 last 19.31) - open/close 13889 (time 1787141316.26 total 890.98 last 26.97) - open/close 14148 (time 1787141326.74 total 901.47 last 24.70) - open/close 14367 (time 1787141337.48 total 912.20 last 20.40) - open/close 14505 (time 1787141347.73 total 922.45 last 13.46) - open/close 14623 (time 1787141358.30 total 933.03 last 11.16) - open/close 14713 (time 1787141368.33 total 943.05 last 8.98) - open/close 14894 (time 1787141378.77 total 953.49 last 17.33) - open/close 15066 (time 1787141391.56 total 966.28 last 13.45) - open/close 15178 (time 1787141401.59 total 976.31 last 11.17) - open/close 15275 (time 1787141412.24 total 986.97 last 9.10) - open/close 15358 (time 1787141422.44 total 997.16 last 8.14) - open/close 15585 (time 1787141432.46 total 1007.18 last 22.66) - open/close 15809 (time 1787141443.41 total 1018.13 last 20.45) - open/close 16006 (time 1787141453.90 total 1028.63 last 18.77) - open/close 16179 (time 1787141463.95 total 1038.67 last 17.23) - open/close 16582 (time 1787141473.95 total 1048.68 last 40.27) - open/close 17238 (time 1787141483.96 total 1058.68 last 65.58) - open/close 17850 (time 1787141493.97 total 1068.70 last 61.11) - open/close 18450 (time 1787141503.97 total 1078.70 last 59.99) - open/close 19046 (time 1787141513.99 total 1088.71 last 59.52) - open/close 19704 (time 1787141523.99 total 1098.71 last 65.80) - open/close 20000 (time 1787141528.75 total 1103.47 last 62.17) - open/close 20694 (time 1787141538.77 total 1113.50 last 69.22) - open/close 21347 (time 1787141548.78 total 1123.50 last 65.26) - open/close 21991 (time 1787141558.78 total 1133.51 last 64.37) - open/close 22624 (time 1787141568.79 total 1143.51 last 63.29) - open/close 23251 (time 1787141578.79 total 1153.51 last 62.70) - open/close 23894 (time 1787141588.81 total 1163.53 last 64.18) - open/close 24475 (time 1787141598.81 total 1173.54 last 58.06) - open/close 25038 (time 1787141608.84 total 1183.57 last 56.13) - open/close 25616 (time 1787141618.84 total 1193.57 last 57.80) - open/close 26227 (time 1787141628.85 total 1203.57 last 61.08) - open/close 26898 (time 1787141638.85 total 1213.58 last 67.05) - open/close 27520 (time 1787141648.87 total 1223.60 last 62.07) - open/close 28071 (time 1787141658.89 total 1233.61 last 55.02) - open/close 28641 (time 1787141668.90 total 1243.62 last 56.94) - open/close 29209 (time 1787141678.94 total 1253.67 last 56.56) - open/close 29798 (time 1787141688.95 total 1263.67 last 58.85) - open/close 30000 (time 1787141692.02 total 1266.75 last 65.77) - open/close 30556 (time 1787141702.03 total 1276.75 last 55.56) - open/close 31139 (time 1787141712.03 total 1286.75 last 58.29) - open/close 31774 (time 1787141722.04 total 1296.76 last 63.44) - open/close 32278 (time 1787141732.05 total 1306.77 last 50.36) - open/close 32893 (time 1787141742.05 total 1316.78 last 61.45) - open/close 33514 (time 1787141752.06 total 1326.78 last 62.08) - open/close 34059 (time 1787141762.06 total 1336.78 last 54.49) - open/close 34672 (time 1787141772.07 total 1346.80 last 61.21) - open/close 35284 (time 1787141782.08 total 1356.80 last 61.18) - open/close 35877 (time 1787141792.09 total 1366.81 last 59.23) - open/close 36457 (time 1787141802.11 total 1376.83 last 57.90) - open/close 37041 (time 1787141812.11 total 1386.84 last 58.35) - open/close 37643 (time 1787141822.13 total 1396.85 last 60.13) - open/close 38262 (time 1787141832.13 total 1406.86 last 61.86) - open/close 38877 (time 1787141842.16 total 1416.89 last 61.32) - open/close 39467 (time 1787141852.19 total 1426.91 last 58.83) - open/close 40000 (time 1787141861.95 total 1436.68 last 54.58) - open/close 40531 (time 1787141871.96 total 1446.68 last 53.09) - open/close 41089 (time 1787141881.97 total 1456.69 last 55.74) - open/close 41689 (time 1787141891.98 total 1466.70 last 59.94) - open/close 42327 (time 1787141902.00 total 1476.72 last 63.67) - open/close 42986 (time 1787141912.00 total 1486.73 last 65.88) - open/close 43600 (time 1787141922.01 total 1496.73 last 61.37) - open/close 44222 (time 1787141932.01 total 1506.73 last 62.20) - open/close 44840 (time 1787141942.02 total 1516.75 last 61.72) - open/close 45462 (time 1787141952.04 total 1526.76 last 62.10) - open/close 46015 (time 1787141962.04 total 1536.77 last 55.26) - open/close 46576 (time 1787141972.05 total 1546.77 last 56.07) - open/close 47213 (time 1787141982.08 total 1556.80 last 63.53) - open/close 47841 (time 1787141992.10 total 1566.82 last 62.66) - open/close 48400 (time 1787142002.10 total 1576.83 last 55.88) - open/close 49032 (time 1787142012.11 total 1586.83 last 63.17) - open/close 49593 (time 1787142022.11 total 1596.83 last 56.09) total: 50000 open/close in 1604.35 seconds: 31.17 ops/second - unlinked 0 (time 1787142034 ; total 0 ; last 0) - unlinked 10000 (time 1787142206 ; total 172 ; last 172) - unlinked 20000 (time 1787142380 ; total 346 ; last 174) - unlinked 30000 (time 1787142562 ; total 528 ; last 182) - unlinked 40000 (time 1787142743 ; total 709 ; last 181) total: 50000 unlinks in 878 seconds: 56.947609 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server unloading modules via unload_modules_local on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing unload_modules_local oleg451-server: modules unloaded. pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2