== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:48:00 (1778924880) start mds service on oleg229-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg229-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg229-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg229-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 199 (time 1778924952.49 total 10.26 last 19.39) - open/close 399 (time 1778924962.49 total 20.26 last 20.00) - open/close 627 (time 1778924972.52 total 30.29 last 22.75) - open/close 909 (time 1778924982.53 total 40.30 last 28.16) - open/close 1113 (time 1778924992.61 total 50.38 last 20.23) - open/close 1378 (time 1778925002.61 total 60.39 last 26.49) - open/close 1584 (time 1778925012.70 total 70.47 last 20.43) - open/close 1811 (time 1778925022.72 total 80.49 last 22.64) - open/close 2018 (time 1778925032.75 total 90.52 last 20.64) - open/close 2259 (time 1778925043.03 total 100.80 last 23.44) - open/close 2506 (time 1778925053.61 total 111.38 last 23.35) - open/close 2751 (time 1778925063.62 total 121.39 last 24.49) - open/close 2952 (time 1778925074.22 total 131.99 last 18.96) - open/close 3045 (time 1778925084.69 total 142.46 last 8.88) - open/close 3200 (time 1778925095.71 total 153.48 last 14.07) - open/close 3380 (time 1778925106.72 total 164.49 last 16.35) - open/close 3620 (time 1778925116.73 total 174.50 last 23.98) - open/close 3855 (time 1778925126.73 total 184.50 last 23.49) - open/close 4076 (time 1778925137.17 total 194.94 last 21.17) - open/close 4306 (time 1778925147.99 total 205.76 last 21.26) - open/close 4446 (time 1778925160.16 total 217.93 last 11.51) - open/close 4635 (time 1778925170.81 total 228.58 last 17.75) - open/close 4675 (time 1778925181.21 total 238.98 last 3.84) - open/close 4931 (time 1778925191.77 total 249.54 last 24.25) - open/close 5158 (time 1778925202.15 total 259.92 last 21.87) - open/close 5401 (time 1778925212.17 total 269.94 last 24.25) - open/close 5616 (time 1778925222.19 total 279.96 last 21.45) - open/close 5877 (time 1778925232.20 total 289.97 last 26.08) - open/close 6089 (time 1778925242.78 total 300.55 last 20.03) - open/close 6341 (time 1778925252.81 total 310.58 last 25.13) - open/close 6555 (time 1778925262.82 total 320.59 last 21.39) - open/close 6788 (time 1778925273.06 total 330.83 last 22.74) - open/close 7004 (time 1778925283.07 total 340.84 last 21.59) - open/close 7243 (time 1778925293.41 total 351.18 last 23.11) - open/close 7474 (time 1778925303.45 total 361.22 last 23.00) - open/close 7720 (time 1778925314.08 total 371.85 last 23.14) - open/close 7943 (time 1778925324.64 total 382.41 last 21.13) - open/close 8188 (time 1778925334.65 total 392.42 last 24.47) - open/close 8400 (time 1778925345.24 total 403.01 last 20.01) - open/close 8651 (time 1778925355.76 total 413.53 last 23.87) - open/close 8883 (time 1778925365.78 total 423.55 last 23.14) - open/close 9114 (time 1778925375.80 total 433.57 last 23.06) - open/close 9283 (time 1778925386.26 total 444.03 last 16.15) - open/close 9478 (time 1778925396.29 total 454.06 last 19.45) - open/close 9719 (time 1778925406.30 total 464.07 last 24.07) - open/close 9892 (time 1778925416.31 total 474.08 last 17.28) - open/close 10000 (time 1778925424.03 total 481.80 last 13.99) - open/close 10136 (time 1778925434.06 total 491.83 last 13.57) - open/close 10371 (time 1778925444.90 total 502.68 last 21.66) - open/close 10601 (time 1778925455.49 total 513.26 last 21.73) - open/close 10800 (time 1778925466.52 total 524.29 last 18.03) - open/close 10954 (time 1778925476.77 total 534.55 last 15.02) - open/close 11081 (time 1778925487.25 total 545.02 last 12.13) - open/close 11256 (time 1778925498.08 total 555.85 last 16.15) - open/close 11443 (time 1778925508.09 total 565.86 last 18.69) - open/close 11666 (time 1778925518.94 total 576.71 last 20.55) - open/close 11899 (time 1778925528.95 total 586.72 last 23.28) - open/close 12124 (time 1778925539.22 total 596.99 last 21.91) - open/close 12279 (time 1778925549.38 total 607.15 last 15.25) - open/close 12413 (time 1778925559.44 total 617.21 last 13.33) - open/close 12523 (time 1778925570.17 total 627.94 last 10.25) - open/close 12778 (time 1778925580.20 total 637.97 last 25.42) - open/close 13024 (time 1778925590.75 total 648.52 last 23.33) - open/close 13198 (time 1778925600.78 total 658.55 last 17.36) - open/close 13430 (time 1778925613.63 total 671.40 last 18.04) - open/close 13696 (time 1778925624.42 total 682.20 last 24.65) - open/close 13924 (time 1778925634.43 total 692.21 last 22.78) - open/close 14069 (time 1778925646.70 total 704.47 last 11.82) - open/close 14251 (time 1778925657.15 total 714.92 last 17.42) - open/close 14499 (time 1778925667.22 total 724.99 last 24.62) - open/close 14746 (time 1778925677.24 total 735.02 last 24.64) - open/close 14964 (time 1778925688.06 total 745.83 last 20.15) - open/close 15109 (time 1778925698.36 total 756.13 last 14.08) - open/close 15317 (time 1778925708.37 total 766.14 last 20.79) - open/close 15519 (time 1778925718.90 total 776.67 last 19.18) - open/close 15734 (time 1778925728.92 total 786.69 last 21.46) - open/close 15968 (time 1778925739.73 total 797.50 last 21.65) - open/close 16195 (time 1778925749.73 total 807.50 last 22.70) - open/close 16426 (time 1778925759.74 total 817.51 last 23.09) - open/close 16664 (time 1778925770.40 total 828.17 last 22.31) - open/close 16860 (time 1778925780.42 total 838.19 last 19.58) - open/close 17106 (time 1778925790.42 total 848.19 last 24.60) - open/close 17314 (time 1778925801.56 total 859.33 last 18.67) - open/close 17484 (time 1778925811.81 total 869.58 last 16.58) - open/close 17607 (time 1778925823.60 total 881.37 last 10.44) - open/close 17679 (time 1778925833.83 total 891.60 last 7.04) - open/close 17796 (time 1778925843.83 total 901.61 last 11.70) - open/close 17936 (time 1778925854.56 total 912.34 last 13.05) - open/close 18126 (time 1778925864.59 total 922.36 last 18.96) - open/close 18367 (time 1778925874.60 total 932.37 last 24.08) - open/close 18531 (time 1778925884.61 total 942.38 last 16.38) - open/close 19006 (time 1778925894.62 total 952.39 last 47.43) - open/close 19674 (time 1778925904.63 total 962.40 last 66.73) - open/close 20000 (time 1778925909.87 total 967.64 last 62.23) - open/close 20717 (time 1778925919.88 total 977.65 last 71.65) - open/close 21457 (time 1778925929.90 total 987.67 last 73.84) - open/close 22139 (time 1778925939.90 total 997.67 last 68.18) - open/close 22860 (time 1778925949.90 total 1007.67 last 72.09) - open/close 23654 (time 1778925959.91 total 1017.68 last 79.32) - open/close 24267 (time 1778925969.92 total 1027.69 last 61.28) - open/close 24966 (time 1778925979.92 total 1037.69 last 69.89) - open/close 25667 (time 1778925989.92 total 1047.69 last 70.09) - open/close 26489 (time 1778925999.94 total 1057.71 last 82.01) - open/close 27077 (time 1778926009.95 total 1067.72 last 58.74) - open/close 27661 (time 1778926019.96 total 1077.73 last 58.35) - open/close 28338 (time 1778926029.97 total 1087.74 last 67.68) - open/close 29083 (time 1778926039.97 total 1097.74 last 74.48) - open/close 29897 (time 1778926049.97 total 1107.74 last 81.39) - open/close 30650 (time 1778926059.98 total 1117.75 last 75.25) - open/close 31402 (time 1778926069.99 total 1127.76 last 75.13) - open/close 32091 (time 1778926080.01 total 1137.78 last 68.75) - open/close 32836 (time 1778926090.01 total 1147.78 last 74.49) - open/close 33586 (time 1778926100.05 total 1157.82 last 74.72) - open/close 34388 (time 1778926110.05 total 1167.82 last 80.15) - open/close 35254 (time 1778926120.06 total 1177.83 last 86.53) - open/close 35865 (time 1778926130.06 total 1187.83 last 61.08) - open/close 36576 (time 1778926140.06 total 1197.83 last 71.09) - open/close 37317 (time 1778926150.08 total 1207.85 last 73.99) - open/close 38043 (time 1778926160.08 total 1217.85 last 72.60) - open/close 38799 (time 1778926170.09 total 1227.86 last 75.54) - open/close 39334 (time 1778926180.10 total 1237.88 last 53.40) - open/close 39881 (time 1778926190.11 total 1247.88 last 54.69) - open/close 40576 (time 1778926200.11 total 1257.88 last 69.49) - open/close 41258 (time 1778926210.11 total 1267.89 last 68.15) - open/close 41848 (time 1778926220.12 total 1277.89 last 58.99) - open/close 42682 (time 1778926230.12 total 1287.89 last 83.36) - open/close 43321 (time 1778926240.12 total 1297.89 last 63.88) - open/close 44012 (time 1778926250.12 total 1307.89 last 69.10) - open/close 44757 (time 1778926260.13 total 1317.90 last 74.45) - open/close 45375 (time 1778926270.15 total 1327.92 last 61.70) - open/close 46034 (time 1778926280.16 total 1337.93 last 65.84) - open/close 46747 (time 1778926290.17 total 1347.94 last 71.18) - open/close 47428 (time 1778926300.17 total 1357.94 last 68.10) - open/close 48138 (time 1778926310.18 total 1367.95 last 70.96) - open/close 48776 (time 1778926320.18 total 1377.95 last 63.80) - open/close 49486 (time 1778926330.18 total 1387.95 last 70.99) total: 50000 open/close in 1395.37 seconds: 35.83 ops/second - unlinked 0 (time 1778926343 ; total 0 ; last 0) - unlinked 10000 (time 1778926504 ; total 161 ; last 161) - unlinked 20000 (time 1778926648 ; total 305 ; last 144) - unlinked 30000 (time 1778926790 ; total 447 ; last 142) - unlinked 40000 (time 1778926927 ; total 584 ; last 137) total: 50000 unlinks in 721 seconds: 69.348129 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server unloading modules via unload_modules_local on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing unload_modules_local oleg229-server: modules unloaded. pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2