== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:37:08 (1786750628) start mds service on oleg442-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing load_modules_local oleg442-server: Loading modules from /home/green/git/lustre-release/lustre oleg442-server: detected 4 online CPUs by sysfs oleg442-server: MODOPTS_LIBCFS= oleg442-server: Force libcfs to create 2 CPU partitions oleg442-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg442-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg442-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg442-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg442-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0000 oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 157 (time 1786750724.60 total 10.01 last 15.68) - open/close 291 (time 1786750734.68 total 20.09 last 13.30) - open/close 468 (time 1786750744.80 total 30.21 last 17.50) - open/close 644 (time 1786750756.83 total 42.24 last 14.63) - open/close 777 (time 1786750766.85 total 52.26 last 13.28) - open/close 908 (time 1786750777.63 total 63.04 last 12.16) - open/close 974 (time 1786750787.74 total 73.15 last 6.53) - open/close 1153 (time 1786750797.77 total 83.18 last 17.84) - open/close 1413 (time 1786750807.79 total 93.19 last 25.96) - open/close 1592 (time 1786750817.79 total 103.20 last 17.89) - open/close 1706 (time 1786750827.81 total 113.22 last 11.38) - open/close 1802 (time 1786750838.05 total 123.46 last 9.37) - open/close 2002 (time 1786750849.07 total 134.48 last 18.15) - open/close 2157 (time 1786750859.63 total 145.04 last 14.68) - open/close 2376 (time 1786750870.50 total 155.91 last 20.15) - open/close 2603 (time 1786750880.51 total 165.92 last 22.68) - open/close 2816 (time 1786750891.22 total 176.63 last 19.88) - open/close 3006 (time 1786750901.23 total 186.64 last 18.99) - open/close 3230 (time 1786750912.23 total 197.64 last 20.36) - open/close 3360 (time 1786750922.40 total 207.81 last 12.79) - open/close 3504 (time 1786750933.05 total 218.46 last 13.52) - open/close 3694 (time 1786750943.39 total 228.80 last 18.37) - open/close 3868 (time 1786750953.71 total 239.12 last 16.87) - open/close 3982 (time 1786750963.75 total 249.16 last 11.35) - open/close 4050 (time 1786750974.69 total 260.10 last 6.22) - open/close 4200 (time 1786750984.89 total 270.30 last 14.70) - open/close 4350 (time 1786750994.91 total 280.32 last 14.97) - open/close 4565 (time 1786751004.94 total 290.35 last 21.44) - open/close 4677 (time 1786751015.09 total 300.50 last 11.03) - open/close 4826 (time 1786751025.95 total 311.36 last 13.72) - open/close 4970 (time 1786751036.65 total 322.06 last 13.45) - open/close 5030 (time 1786751047.59 total 333.00 last 5.49) - open/close 5154 (time 1786751057.65 total 343.06 last 12.32) - open/close 5225 (time 1786751067.71 total 353.12 last 7.06) - open/close 5358 (time 1786751078.31 total 363.72 last 12.55) - open/close 5395 (time 1786751088.80 total 374.21 last 3.53) - open/close 5511 (time 1786751098.84 total 384.25 last 11.56) - open/close 5679 (time 1786751108.85 total 394.26 last 16.78) - open/close 5798 (time 1786751118.86 total 404.27 last 11.89) - open/close 5940 (time 1786751128.86 total 414.27 last 14.19) - open/close 6044 (time 1786751138.91 total 424.31 last 10.36) - open/close 6162 (time 1786751151.15 total 436.56 last 9.64) - open/close 6273 (time 1786751161.85 total 447.26 last 10.37) - open/close 6423 (time 1786751172.35 total 457.76 last 14.28) - open/close 6526 (time 1786751182.36 total 467.77 last 10.29) - open/close 6647 (time 1786751192.66 total 478.07 last 11.75) - open/close 6738 (time 1786751203.51 total 488.92 last 8.39) - open/close 6865 (time 1786751213.52 total 498.93 last 12.68) - open/close 7018 (time 1786751224.11 total 509.52 last 14.44) - open/close 7157 (time 1786751235.17 total 520.58 last 12.58) - open/close 7277 (time 1786751245.19 total 530.60 last 11.97) - open/close 7416 (time 1786751255.20 total 540.61 last 13.89) - open/close 7551 (time 1786751266.21 total 551.62 last 12.26) - open/close 7637 (time 1786751276.23 total 561.64 last 8.58) - open/close 7738 (time 1786751286.38 total 571.79 last 9.95) - open/close 7848 (time 1786751297.85 total 583.26 last 9.59) - open/close 7914 (time 1786751308.26 total 593.67 last 6.34) - open/close 8005 (time 1786751319.12 total 604.53 last 8.38) - open/close 8077 (time 1786751330.66 total 616.06 last 6.24) - open/close 8230 (time 1786751340.71 total 626.12 last 15.22) - open/close 8349 (time 1786751351.58 total 636.99 last 10.94) - open/close 8463 (time 1786751362.10 total 647.51 last 10.84) - open/close 8544 (time 1786751372.46 total 657.86 last 7.82) - open/close 8696 (time 1786751382.94 total 668.35 last 14.50) - open/close 8860 (time 1786751393.86 total 679.27 last 15.02) - open/close 8914 (time 1786751404.07 total 689.48 last 5.29) - open/close 9015 (time 1786751414.11 total 699.52 last 10.06) - open/close 9071 (time 1786751426.26 total 711.67 last 4.61) - open/close 9198 (time 1786751436.29 total 721.70 last 12.67) - open/close 9359 (time 1786751449.18 total 734.59 last 12.48) - open/close 9534 (time 1786751461.59 total 747.00 last 14.11) - open/close 9703 (time 1786751472.47 total 757.88 last 15.53) - open/close 9824 (time 1786751482.90 total 768.31 last 11.60) - open/close 9954 (time 1786751493.28 total 778.69 last 12.53) - open/close 10000 (time 1786751499.69 total 785.09 last 7.18) - open/close 10189 (time 1786751510.37 total 795.78 last 17.69) - open/close 10288 (time 1786751521.06 total 806.47 last 9.26) - open/close 10466 (time 1786751531.56 total 816.97 last 16.95) - open/close 10548 (time 1786751541.73 total 827.14 last 8.06) - open/close 10706 (time 1786751551.76 total 837.16 last 15.76) - open/close 10910 (time 1786751563.94 total 849.35 last 16.74) - open/close 11044 (time 1786751574.18 total 859.59 last 13.08) - open/close 11186 (time 1786751584.20 total 869.61 last 14.18) - open/close 11339 (time 1786751596.45 total 881.86 last 12.49) - open/close 11453 (time 1786751607.08 total 892.49 last 10.72) - open/close 11537 (time 1786751617.59 total 903.00 last 7.99) - open/close 11648 (time 1786751627.61 total 913.02 last 11.08) - open/close 11763 (time 1786751638.62 total 924.03 last 10.44) - open/close 11919 (time 1786751648.80 total 934.21 last 15.32) - open/close 12066 (time 1786751659.33 total 944.74 last 13.96) - open/close 12178 (time 1786751669.33 total 954.74 last 11.19) - open/close 12337 (time 1786751680.37 total 965.77 last 14.41) - open/close 12468 (time 1786751690.40 total 975.81 last 13.06) - open/close 12629 (time 1786751701.35 total 986.76 last 14.70) - open/close 12782 (time 1786751712.04 total 997.45 last 14.31) - open/close 12973 (time 1786751722.85 total 1008.26 last 17.66) - open/close 13100 (time 1786751732.89 total 1018.30 last 12.65) - open/close 13269 (time 1786751743.22 total 1028.63 last 16.36) - open/close 13385 (time 1786751753.56 total 1038.97 last 11.22) - open/close 13498 (time 1786751764.58 total 1049.99 last 10.25) - open/close 13654 (time 1786751774.60 total 1060.01 last 15.56) - open/close 13860 (time 1786751784.61 total 1070.02 last 20.60) - open/close 13959 (time 1786751795.96 total 1081.37 last 8.72) - open/close 14075 (time 1786751805.97 total 1091.38 last 11.58) - open/close 14295 (time 1786751816.37 total 1101.78 last 21.16) - open/close 14467 (time 1786751826.38 total 1111.79 last 17.18) - open/close 14630 (time 1786751836.39 total 1121.80 last 16.28) - open/close 14794 (time 1786751846.40 total 1131.81 last 16.39) - open/close 15343 (time 1786751856.40 total 1141.81 last 54.87) - open/close 15916 (time 1786751866.41 total 1151.82 last 57.27) - open/close 16548 (time 1786751876.41 total 1161.82 last 63.17) - open/close 17160 (time 1786751886.42 total 1171.83 last 61.14) - open/close 17749 (time 1786751896.43 total 1181.84 last 58.88) - open/close 18251 (time 1786751906.43 total 1191.84 last 50.16) - open/close 18737 (time 1786751916.44 total 1201.85 last 48.56) - open/close 19215 (time 1786751926.46 total 1211.87 last 47.72) - open/close 19819 (time 1786751936.47 total 1221.88 last 60.35) - open/close 20000 (time 1786751940.52 total 1225.93 last 44.64) - open/close 20617 (time 1786751950.53 total 1235.94 last 61.65) - open/close 21249 (time 1786751960.54 total 1245.95 last 63.14) - open/close 21900 (time 1786751970.54 total 1255.95 last 65.08) - open/close 22572 (time 1786751980.55 total 1265.96 last 67.18) - open/close 23204 (time 1786751990.55 total 1275.96 last 63.16) - open/close 23851 (time 1786752000.55 total 1285.96 last 64.70) - open/close 24525 (time 1786752010.56 total 1295.97 last 67.38) - open/close 25095 (time 1786752020.57 total 1305.98 last 56.90) - open/close 25647 (time 1786752030.58 total 1315.99 last 55.18) - open/close 26195 (time 1786752040.60 total 1326.01 last 54.70) - open/close 26680 (time 1786752050.61 total 1336.02 last 48.43) - open/close 27170 (time 1786752060.62 total 1346.03 last 48.96) - open/close 27803 (time 1786752070.63 total 1356.04 last 63.25) - open/close 28329 (time 1786752080.63 total 1366.04 last 52.57) - open/close 28902 (time 1786752090.64 total 1376.04 last 57.27) - open/close 29576 (time 1786752100.65 total 1386.06 last 67.30) - open/close 30000 (time 1786752106.86 total 1392.26 last 68.33) - open/close 30633 (time 1786752116.86 total 1402.27 last 63.26) - open/close 31214 (time 1786752126.87 total 1412.28 last 58.07) - open/close 31800 (time 1786752136.88 total 1422.29 last 58.51) - open/close 32360 (time 1786752146.88 total 1432.29 last 55.99) - open/close 32985 (time 1786752156.90 total 1442.31 last 62.40) - open/close 33531 (time 1786752166.90 total 1452.31 last 54.57) - open/close 34186 (time 1786752176.91 total 1462.32 last 65.46) - open/close 34702 (time 1786752186.92 total 1472.32 last 51.57) - open/close 35222 (time 1786752196.93 total 1482.34 last 51.94) - open/close 35829 (time 1786752206.94 total 1492.35 last 60.64) - open/close 36411 (time 1786752216.95 total 1502.36 last 58.14) - open/close 36982 (time 1786752226.96 total 1512.37 last 57.04) - open/close 37540 (time 1786752236.96 total 1522.37 last 55.76) - open/close 38032 (time 1786752246.97 total 1532.38 last 49.16) - open/close 38577 (time 1786752256.99 total 1542.40 last 54.39) - open/close 39088 (time 1786752267.02 total 1552.43 last 50.96) - open/close 39595 (time 1786752277.03 total 1562.44 last 50.66) - open/close 40000 (time 1786752283.56 total 1568.97 last 62.02) - open/close 40548 (time 1786752293.56 total 1578.97 last 54.77) - open/close 41222 (time 1786752303.57 total 1588.98 last 67.36) - open/close 41775 (time 1786752313.58 total 1598.99 last 55.21) - open/close 42323 (time 1786752323.60 total 1609.01 last 54.72) - open/close 42917 (time 1786752333.62 total 1619.03 last 59.29) - open/close 43510 (time 1786752343.62 total 1629.03 last 59.26) - open/close 44022 (time 1786752353.63 total 1639.04 last 51.17) - open/close 44508 (time 1786752363.65 total 1649.06 last 48.52) - open/close 44975 (time 1786752373.66 total 1659.07 last 46.66) - open/close 45539 (time 1786752383.66 total 1669.07 last 56.35) - open/close 46079 (time 1786752393.67 total 1679.08 last 53.98) - open/close 46645 (time 1786752403.68 total 1689.09 last 56.53) - open/close 47272 (time 1786752413.69 total 1699.10 last 62.65) - open/close 47838 (time 1786752423.69 total 1709.10 last 56.59) - open/close 48414 (time 1786752433.72 total 1719.13 last 57.40) - open/close 48916 (time 1786752443.73 total 1729.14 last 50.16) - open/close 49480 (time 1786752453.74 total 1739.15 last 56.36) total: 50000 open/close in 1748.04 seconds: 28.60 ops/second - unlinked 0 (time 1786752467 ; total 0 ; last 0) - unlinked 10000 (time 1786752645 ; total 178 ; last 178) - unlinked 20000 (time 1786752829 ; total 362 ; last 184) - unlinked 30000 (time 1786753032 ; total 565 ; last 203) - unlinked 40000 (time 1786753215 ; total 748 ; last 183) total: 50000 unlinks in 920 seconds: 54.347828 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg442-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg442-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg442-server stop mds service on oleg442-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg442-server unloading modules via unload_modules_local on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing unload_modules_local oleg442-server: modules unloaded. pdsh@oleg442-client: oleg442-client: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-server: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-client: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-server: ssh exited with exit code 2