== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:47:44 (1788155264) start mds service on oleg460-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing load_modules_local oleg460-server: Loading modules from /home/green/git/lustre-release/lustre oleg460-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg460-server: detected 4 online CPUs by sysfs oleg460-server: MODOPTS_LIBCFS= oleg460-server: Force libcfs to create 2 CPU partitions oleg460-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg460-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg460-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg460-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 110 (time 1788155352.35 total 10.37 last 10.60) - open/close 251 (time 1788155362.44 total 20.46 last 13.98) - open/close 439 (time 1788155372.93 total 30.96 last 17.91) - open/close 647 (time 1788155382.95 total 40.97 last 20.77) - open/close 777 (time 1788155393.27 total 51.29 last 12.60) - open/close 857 (time 1788155403.28 total 61.30 last 7.99) - open/close 1041 (time 1788155414.09 total 72.11 last 17.03) - open/close 1132 (time 1788155425.99 total 84.01 last 7.65) - open/close 1326 (time 1788155436.21 total 94.23 last 18.98) - open/close 1564 (time 1788155447.07 total 105.09 last 21.90) - open/close 1782 (time 1788155458.42 total 116.44 last 19.22) - open/close 2027 (time 1788155469.30 total 127.32 last 22.51) - open/close 2299 (time 1788155479.31 total 137.33 last 27.18) - open/close 2432 (time 1788155489.33 total 147.35 last 13.28) - open/close 2638 (time 1788155499.39 total 157.41 last 20.47) - open/close 2851 (time 1788155509.71 total 167.73 last 20.63) - open/close 3051 (time 1788155519.72 total 177.74 last 19.99) - open/close 3173 (time 1788155530.01 total 188.03 last 11.86) - open/close 3348 (time 1788155540.41 total 198.43 last 16.82) - open/close 3589 (time 1788155550.42 total 208.44 last 24.09) - open/close 3752 (time 1788155562.82 total 220.84 last 13.15) - open/close 4017 (time 1788155572.82 total 230.84 last 26.49) - open/close 4229 (time 1788155583.30 total 241.32 last 20.24) - open/close 4462 (time 1788155593.40 total 251.42 last 23.07) - open/close 4674 (time 1788155603.40 total 261.42 last 21.20) - open/close 4900 (time 1788155613.75 total 271.77 last 21.84) - open/close 5116 (time 1788155623.79 total 281.81 last 21.52) - open/close 5350 (time 1788155634.29 total 292.31 last 22.28) - open/close 5583 (time 1788155644.82 total 302.84 last 22.12) - open/close 5805 (time 1788155654.82 total 312.84 last 22.20) - open/close 5939 (time 1788155667.37 total 325.39 last 10.68) - open/close 6079 (time 1788155677.80 total 335.82 last 13.41) - open/close 6317 (time 1788155687.82 total 345.84 last 23.76) - open/close 6508 (time 1788155697.84 total 355.86 last 19.06) - open/close 6717 (time 1788155707.86 total 365.88 last 20.85) - open/close 7020 (time 1788155717.89 total 375.92 last 30.20) - open/close 7180 (time 1788155728.27 total 386.29 last 15.43) - open/close 7297 (time 1788155738.28 total 396.30 last 11.68) - open/close 7453 (time 1788155749.12 total 407.14 last 14.38) - open/close 7605 (time 1788155759.14 total 417.16 last 15.18) - open/close 7708 (time 1788155769.41 total 427.43 last 10.03) - open/close 7932 (time 1788155779.41 total 437.44 last 22.38) - open/close 8163 (time 1788155790.14 total 448.16 last 21.54) - open/close 8396 (time 1788155800.16 total 458.18 last 23.26) - open/close 8620 (time 1788155810.94 total 468.96 last 20.78) - open/close 8861 (time 1788155821.95 total 479.97 last 21.89) - open/close 9113 (time 1788155832.59 total 490.61 last 23.68) - open/close 9292 (time 1788155843.58 total 501.60 last 16.29) - open/close 9469 (time 1788155853.81 total 511.83 last 17.31) - open/close 9690 (time 1788155863.81 total 521.83 last 22.09) - open/close 9828 (time 1788155873.82 total 531.84 last 13.78) - open/close 10000 (time 1788155883.63 total 541.65 last 17.54) - open/close 10184 (time 1788155893.65 total 551.67 last 18.35) - open/close 10404 (time 1788155904.50 total 562.52 last 20.28) - open/close 10637 (time 1788155914.51 total 572.53 last 23.29) - open/close 10727 (time 1788155924.61 total 582.63 last 8.91) - open/close 10923 (time 1788155934.78 total 592.80 last 19.28) - open/close 11029 (time 1788155947.28 total 605.30 last 8.48) - open/close 11287 (time 1788155957.44 total 615.46 last 25.39) - open/close 11463 (time 1788155967.45 total 625.47 last 17.59) - open/close 11660 (time 1788155977.93 total 635.95 last 18.78) - open/close 11907 (time 1788155988.99 total 647.01 last 22.34) - open/close 12120 (time 1788155999.49 total 657.51 last 20.28) - open/close 12291 (time 1788156009.50 total 667.52 last 17.09) - open/close 12519 (time 1788156020.08 total 678.10 last 21.55) - open/close 12802 (time 1788156032.50 total 690.52 last 22.80) - open/close 13043 (time 1788156045.32 total 703.34 last 18.80) - open/close 13170 (time 1788156055.37 total 713.39 last 12.64) - open/close 13359 (time 1788156066.29 total 724.31 last 17.30) - open/close 13600 (time 1788156076.87 total 734.89 last 22.78) - open/close 13879 (time 1788156087.47 total 745.49 last 26.32) - open/close 14106 (time 1788156097.50 total 755.52 last 22.63) - open/close 14340 (time 1788156108.20 total 766.23 last 21.87) - open/close 14464 (time 1788156119.29 total 777.32 last 11.18) - open/close 14638 (time 1788156131.78 total 789.80 last 13.94) - open/close 14849 (time 1788156141.80 total 799.82 last 21.05) - open/close 15057 (time 1788156153.81 total 811.83 last 17.32) - open/close 15160 (time 1788156164.49 total 822.51 last 9.64) - open/close 15337 (time 1788156174.50 total 832.52 last 17.70) - open/close 15525 (time 1788156185.41 total 843.43 last 17.23) - open/close 15735 (time 1788156195.73 total 853.75 last 20.34) - open/close 15900 (time 1788156205.73 total 863.76 last 16.49) - open/close 16067 (time 1788156216.39 total 874.41 last 15.67) - open/close 16309 (time 1788156227.32 total 885.34 last 22.15) - open/close 16485 (time 1788156238.07 total 896.09 last 16.37) - open/close 16741 (time 1788156248.07 total 906.09 last 25.59) - open/close 16976 (time 1788156258.08 total 916.10 last 23.49) - open/close 17187 (time 1788156268.08 total 926.10 last 21.10) - open/close 17310 (time 1788156281.03 total 939.05 last 9.50) - open/close 17420 (time 1788156291.03 total 949.05 last 10.99) - open/close 17490 (time 1788156301.25 total 959.27 last 6.85) - open/close 17699 (time 1788156311.89 total 969.91 last 19.64) - open/close 17834 (time 1788156322.19 total 980.21 last 13.11) - open/close 18153 (time 1788156332.19 total 990.21 last 31.90) - open/close 18832 (time 1788156342.21 total 1000.23 last 67.82) - open/close 19458 (time 1788156352.21 total 1010.23 last 62.55) - open/close 20000 (time 1788156360.56 total 1018.58 last 64.94) - open/close 20640 (time 1788156370.58 total 1028.60 last 63.86) - open/close 21223 (time 1788156380.59 total 1038.61 last 58.28) - open/close 21828 (time 1788156390.60 total 1048.62 last 60.42) - open/close 22463 (time 1788156400.60 total 1058.62 last 63.48) - open/close 23085 (time 1788156410.62 total 1068.64 last 62.11) - open/close 23737 (time 1788156420.64 total 1078.66 last 65.08) - open/close 24382 (time 1788156430.65 total 1088.67 last 64.41) - open/close 24989 (time 1788156440.66 total 1098.68 last 60.67) - open/close 25610 (time 1788156450.67 total 1108.69 last 62.04) - open/close 26206 (time 1788156460.67 total 1118.69 last 59.59) - open/close 26876 (time 1788156470.68 total 1128.70 last 66.91) - open/close 27567 (time 1788156480.69 total 1138.71 last 69.02) - open/close 28234 (time 1788156490.70 total 1148.72 last 66.61) - open/close 28848 (time 1788156500.71 total 1158.73 last 61.38) - open/close 29459 (time 1788156510.71 total 1168.73 last 61.08) - open/close 30000 (time 1788156519.61 total 1177.63 last 60.81) - open/close 30601 (time 1788156529.61 total 1187.63 last 60.09) - open/close 31116 (time 1788156539.64 total 1197.66 last 51.34) - open/close 31686 (time 1788156549.64 total 1207.66 last 56.98) - open/close 32163 (time 1788156559.65 total 1217.67 last 47.67) - open/close 32789 (time 1788156569.66 total 1227.68 last 62.52) - open/close 33424 (time 1788156579.66 total 1237.68 last 63.50) - open/close 34021 (time 1788156589.66 total 1247.68 last 59.70) - open/close 34666 (time 1788156599.68 total 1257.70 last 64.38) - open/close 35260 (time 1788156609.69 total 1267.71 last 59.39) - open/close 35850 (time 1788156619.70 total 1277.72 last 58.91) - open/close 36488 (time 1788156629.71 total 1287.73 last 63.75) - open/close 37101 (time 1788156639.72 total 1297.74 last 61.24) - open/close 37765 (time 1788156649.73 total 1307.75 last 66.30) - open/close 38399 (time 1788156659.74 total 1317.76 last 63.34) - open/close 38992 (time 1788156669.76 total 1327.78 last 59.21) - open/close 39602 (time 1788156679.76 total 1337.78 last 60.98) - open/close 40000 (time 1788156687.03 total 1345.05 last 54.76) - open/close 40563 (time 1788156697.05 total 1355.07 last 56.18) - open/close 41087 (time 1788156707.05 total 1365.08 last 52.38) - open/close 41666 (time 1788156717.06 total 1375.08 last 57.88) - open/close 42297 (time 1788156727.06 total 1385.08 last 63.08) - open/close 42853 (time 1788156737.07 total 1395.09 last 55.57) - open/close 43386 (time 1788156747.07 total 1405.09 last 53.27) - open/close 43943 (time 1788156757.14 total 1415.16 last 55.35) - open/close 44492 (time 1788156767.15 total 1425.17 last 54.84) - open/close 45130 (time 1788156777.15 total 1435.17 last 63.76) - open/close 45755 (time 1788156787.16 total 1445.18 last 62.44) - open/close 46370 (time 1788156797.17 total 1455.19 last 61.47) - open/close 46896 (time 1788156807.19 total 1465.21 last 52.48) - open/close 47436 (time 1788156817.20 total 1475.22 last 53.97) - open/close 48024 (time 1788156827.21 total 1485.23 last 58.71) - open/close 48660 (time 1788156837.22 total 1495.24 last 63.54) - open/close 49198 (time 1788156847.23 total 1505.25 last 53.74) - open/close 49762 (time 1788156857.24 total 1515.26 last 56.35) total: 50000 open/close in 1519.75 seconds: 32.90 ops/second - unlinked 0 (time 1788156867 ; total 0 ; last 0) - unlinked 10000 (time 1788157056 ; total 189 ; last 189) - unlinked 20000 (time 1788157248 ; total 381 ; last 192) - unlinked 30000 (time 1788157436 ; total 569 ; last 188) - unlinked 40000 (time 1788157617 ; total 750 ; last 181) total: 50000 unlinks in 928 seconds: 53.879311 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg460-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg460-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server stop mds service on oleg460-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg460-server unloading modules via unload_modules_local on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing unload_modules_local oleg460-server: modules unloaded. pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2