== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:21:18 (1775848878) start mds service on oleg253-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg253-server' oleg253-server: oleg253-server.virtnet: executing load_modules_local oleg253-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg253-server: Loading modules from /home/green/git/lustre-release/lustre oleg253-server: detected 4 online CPUs by sysfs oleg253-server: MODOPTS_LIBCFS= oleg253-server: Force libcfs to create 2 CPU partitions oleg253-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg253-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg253-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg253-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-OST0000 oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg253-client.virtnet: -o user_xattr,flock 192.168.202.153@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 148 (time 1775848960.28 total 12.73 last 11.63) - open/close 268 (time 1775848971.29 total 23.73 last 10.90) - open/close 434 (time 1775848981.29 total 33.74 last 16.59) - open/close 629 (time 1775848991.38 total 43.82 last 19.34) - open/close 890 (time 1775849001.38 total 53.83 last 26.08) - open/close 1120 (time 1775849011.39 total 63.84 last 22.98) - open/close 1348 (time 1775849021.41 total 73.85 last 22.77) - open/close 1573 (time 1775849031.42 total 83.86 last 22.47) - open/close 1822 (time 1775849041.43 total 93.88 last 24.87) - open/close 1996 (time 1775849051.64 total 104.09 last 17.04) - open/close 2265 (time 1775849061.66 total 114.10 last 26.85) - open/close 2532 (time 1775849071.66 total 124.10 last 26.70) - open/close 2737 (time 1775849082.28 total 134.72 last 19.31) - open/close 2955 (time 1775849092.29 total 144.73 last 21.78) - open/close 3104 (time 1775849102.83 total 155.27 last 14.14) - open/close 3294 (time 1775849112.84 total 165.29 last 18.97) - open/close 3513 (time 1775849123.37 total 175.81 last 20.81) - open/close 3761 (time 1775849133.37 total 185.81 last 24.79) - open/close 3960 (time 1775849143.84 total 196.28 last 19.01) - open/close 4178 (time 1775849154.58 total 207.02 last 20.30) - open/close 4349 (time 1775849165.43 total 217.87 last 15.76) - open/close 4617 (time 1775849175.43 total 227.87 last 26.80) - open/close 4858 (time 1775849186.09 total 238.54 last 22.59) - open/close 4953 (time 1775849196.30 total 248.74 last 9.31) - open/close 5170 (time 1775849206.32 total 258.76 last 21.66) - open/close 5387 (time 1775849216.33 total 268.77 last 21.68) - open/close 5619 (time 1775849227.15 total 279.59 last 21.45) - open/close 5905 (time 1775849237.19 total 289.63 last 28.47) - open/close 6017 (time 1775849248.27 total 300.71 last 10.11) - open/close 6170 (time 1775849258.39 total 310.83 last 15.12) - open/close 6330 (time 1775849268.43 total 320.87 last 15.95) - open/close 6435 (time 1775849278.87 total 331.32 last 10.05) - open/close 6581 (time 1775849289.62 total 342.06 last 13.59) - open/close 6752 (time 1775849299.75 total 352.19 last 16.89) - open/close 6934 (time 1775849310.68 total 363.12 last 16.64) - open/close 7139 (time 1775849320.69 total 373.14 last 20.48) - open/close 7338 (time 1775849330.70 total 383.15 last 19.88) - open/close 7536 (time 1775849340.82 total 393.26 last 19.57) - open/close 7757 (time 1775849351.53 total 403.97 last 20.64) - open/close 8000 (time 1775849361.54 total 413.98 last 24.26) - open/close 8196 (time 1775849371.56 total 424.00 last 19.57) - open/close 8377 (time 1775849381.57 total 434.01 last 18.08) - open/close 8622 (time 1775849391.89 total 444.33 last 23.74) - open/close 8833 (time 1775849401.90 total 454.34 last 21.09) - open/close 9065 (time 1775849411.91 total 464.36 last 23.16) - open/close 9253 (time 1775849424.40 total 476.85 last 15.05) - open/close 9460 (time 1775849434.41 total 486.86 last 20.68) - open/close 9659 (time 1775849445.13 total 497.58 last 18.56) - open/close 9925 (time 1775849456.16 total 508.60 last 24.13) - open/close 10000 (time 1775849461.84 total 514.28 last 13.20) - open/close 10163 (time 1775849471.87 total 524.31 last 16.26) - open/close 10375 (time 1775849481.87 total 534.31 last 21.19) - open/close 10538 (time 1775849491.88 total 544.32 last 16.28) - open/close 10742 (time 1775849501.89 total 554.33 last 20.38) - open/close 10844 (time 1775849513.61 total 566.05 last 8.71) - open/close 11081 (time 1775849524.48 total 576.92 last 21.80) - open/close 11193 (time 1775849535.13 total 587.57 last 10.51) - open/close 11433 (time 1775849545.13 total 597.58 last 23.99) - open/close 11637 (time 1775849555.28 total 607.73 last 20.10) - open/close 11899 (time 1775849565.52 total 617.96 last 25.60) - open/close 12112 (time 1775849575.52 total 627.96 last 21.29) - open/close 12265 (time 1775849588.18 total 640.63 last 12.08) - open/close 12499 (time 1775849598.19 total 650.63 last 23.39) - open/close 12721 (time 1775849608.27 total 660.71 last 22.02) - open/close 12956 (time 1775849618.38 total 670.82 last 23.25) - open/close 13176 (time 1775849628.81 total 681.26 last 21.08) - open/close 13403 (time 1775849639.45 total 691.89 last 21.35) - open/close 13532 (time 1775849650.46 total 702.90 last 11.72) - open/close 13765 (time 1775849661.27 total 713.71 last 21.55) - open/close 13920 (time 1775849671.31 total 723.76 last 15.43) - open/close 14170 (time 1775849681.63 total 734.07 last 24.23) - open/close 14409 (time 1775849692.63 total 745.07 last 21.73) - open/close 14496 (time 1775849703.43 total 755.87 last 8.06) - open/close 14747 (time 1775849713.43 total 765.88 last 25.08) - open/close 14953 (time 1775849723.45 total 775.90 last 20.56) - open/close 15168 (time 1775849733.46 total 785.90 last 21.49) - open/close 15377 (time 1775849743.46 total 795.91 last 20.89) - open/close 15531 (time 1775849753.50 total 805.94 last 15.34) - open/close 15747 (time 1775849763.52 total 815.96 last 21.57) - open/close 15941 (time 1775849774.57 total 827.01 last 17.56) - open/close 16148 (time 1775849784.57 total 837.01 last 20.69) - open/close 16331 (time 1775849795.36 total 847.81 last 16.96) - open/close 16536 (time 1775849805.37 total 857.81 last 20.49) - open/close 16736 (time 1775849815.51 total 867.95 last 19.72) - open/close 16988 (time 1775849825.52 total 877.96 last 25.19) - open/close 17093 (time 1775849835.52 total 887.97 last 10.49) - open/close 17251 (time 1775849846.56 total 899.00 last 14.31) - open/close 17302 (time 1775849856.58 total 909.02 last 5.09) - open/close 17498 (time 1775849867.11 total 919.55 last 18.61) - open/close 17733 (time 1775849877.26 total 929.70 last 23.15) - open/close 17986 (time 1775849888.34 total 940.78 last 22.84) - open/close 18199 (time 1775849899.03 total 951.47 last 19.92) - open/close 18466 (time 1775849909.03 total 961.47 last 26.69) - open/close 19314 (time 1775849919.04 total 971.49 last 84.71) - open/close 20000 (time 1775849927.17 total 979.61 last 84.40) - open/close 20833 (time 1775849937.17 total 989.61 last 83.29) - open/close 21615 (time 1775849947.18 total 999.62 last 78.18) - open/close 22513 (time 1775849957.18 total 1009.62 last 89.75) - open/close 23327 (time 1775849967.20 total 1019.64 last 81.24) - open/close 24072 (time 1775849977.20 total 1029.65 last 74.48) - open/close 24699 (time 1775849987.21 total 1039.65 last 62.68) - open/close 25579 (time 1775849997.22 total 1049.66 last 87.89) - open/close 26350 (time 1775850007.23 total 1059.67 last 77.05) - open/close 27116 (time 1775850017.23 total 1069.68 last 76.53) - open/close 27900 (time 1775850027.25 total 1079.70 last 78.24) - open/close 28651 (time 1775850037.26 total 1089.71 last 75.03) - open/close 29398 (time 1775850047.27 total 1099.71 last 74.64) - open/close 30000 (time 1775850054.52 total 1106.96 last 83.07) - open/close 30791 (time 1775850064.53 total 1116.97 last 79.01) - open/close 31557 (time 1775850074.56 total 1127.00 last 76.41) - open/close 32348 (time 1775850084.56 total 1137.00 last 79.10) - open/close 33195 (time 1775850094.57 total 1147.01 last 84.60) - open/close 34026 (time 1775850104.57 total 1157.01 last 83.06) - open/close 34793 (time 1775850114.58 total 1167.02 last 76.67) - open/close 35493 (time 1775850124.58 total 1177.02 last 69.96) - open/close 36314 (time 1775850134.59 total 1187.03 last 82.07) - open/close 37187 (time 1775850144.59 total 1197.03 last 87.24) - open/close 38012 (time 1775850154.60 total 1207.04 last 82.46) - open/close 38734 (time 1775850164.61 total 1217.05 last 72.09) - open/close 39562 (time 1775850174.61 total 1227.05 last 82.79) - open/close 40000 (time 1775850180.71 total 1233.15 last 71.82) - open/close 40746 (time 1775850190.72 total 1243.16 last 74.54) - open/close 41558 (time 1775850200.73 total 1253.17 last 81.11) - open/close 42430 (time 1775850210.73 total 1263.17 last 87.19) - open/close 43146 (time 1775850220.74 total 1273.18 last 71.56) - open/close 43890 (time 1775850230.75 total 1283.19 last 74.33) - open/close 44621 (time 1775850240.76 total 1293.21 last 72.97) - open/close 45344 (time 1775850250.76 total 1303.21 last 72.30) - open/close 46140 (time 1775850260.78 total 1313.22 last 79.45) - open/close 46892 (time 1775850270.78 total 1323.23 last 75.19) - open/close 47680 (time 1775850280.79 total 1333.23 last 78.73) - open/close 48476 (time 1775850290.80 total 1343.24 last 79.53) - open/close 49259 (time 1775850300.82 total 1353.26 last 78.13) total: 50000 open/close in 1362.63 seconds: 36.69 ops/second - unlinked 0 (time 1775850314 ; total 0 ; last 0) - unlinked 10000 (time 1775850453 ; total 139 ; last 139) - unlinked 20000 (time 1775850593 ; total 279 ; last 140) - unlinked 30000 (time 1775850733 ; total 419 ; last 140) - unlinked 40000 (time 1775850874 ; total 560 ; last 141) total: 50000 unlinks in 706 seconds: 70.821533 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg253-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg253-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg253-server stop mds service on oleg253-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg253-server unloading modules via unload_modules_local on: 'oleg253-server' oleg253-server: oleg253-server.virtnet: executing unload_modules_local oleg253-server: modules unloaded. pdsh@oleg253-client: oleg253-server: ssh exited with exit code 2 pdsh@oleg253-client: oleg253-client: ssh exited with exit code 2 pdsh@oleg253-client: oleg253-client: ssh exited with exit code 2 pdsh@oleg253-client: oleg253-server: ssh exited with exit code 2