== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:58:44 (1787846324) start mds service on oleg332-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing load_modules_local oleg332-server: Loading modules from /home/green/git/lustre-release/lustre oleg332-server: detected 4 online CPUs by sysfs oleg332-server: MODOPTS_LIBCFS= oleg332-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg332-server: Force libcfs to create 2 CPU partitions oleg332-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg332-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg332-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg332-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg332-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 268 (time 1787846435.15 total 10.02 last 26.74) - open/close 470 (time 1787846445.15 total 20.02 last 20.20) - open/close 719 (time 1787846455.95 total 30.82 last 23.05) - open/close 983 (time 1787846465.95 total 40.82 last 26.40) - open/close 1236 (time 1787846476.76 total 51.64 last 23.40) - open/close 1441 (time 1787846486.76 total 61.64 last 20.50) - open/close 1710 (time 1787846497.69 total 72.56 last 24.62) - open/close 1975 (time 1787846508.71 total 83.58 last 24.04) - open/close 2224 (time 1787846519.26 total 94.14 last 23.60) - open/close 2460 (time 1787846529.46 total 104.33 last 23.14) - open/close 2607 (time 1787846539.47 total 114.35 last 14.68) - open/close 2740 (time 1787846549.48 total 124.35 last 13.29) - open/close 3007 (time 1787846559.49 total 134.36 last 26.67) - open/close 3259 (time 1787846569.55 total 144.42 last 25.05) - open/close 3395 (time 1787846580.44 total 155.31 last 12.49) - open/close 3504 (time 1787846590.48 total 165.35 last 10.85) - open/close 3583 (time 1787846600.86 total 175.73 last 7.61) - open/close 3754 (time 1787846610.86 total 185.74 last 17.10) - open/close 3987 (time 1787846620.87 total 195.75 last 23.27) - open/close 4167 (time 1787846630.88 total 205.76 last 17.98) - open/close 4354 (time 1787846640.89 total 215.77 last 18.68) - open/close 4510 (time 1787846651.92 total 226.79 last 14.15) - open/close 4663 (time 1787846662.23 total 237.11 last 14.83) - open/close 4839 (time 1787846672.91 total 247.79 last 16.48) - open/close 5087 (time 1787846683.03 total 257.90 last 24.52) - open/close 5349 (time 1787846693.90 total 268.77 last 24.10) - open/close 5455 (time 1787846706.34 total 281.22 last 8.52) - open/close 5633 (time 1787846717.07 total 291.94 last 16.60) - open/close 5820 (time 1787846730.08 total 304.95 last 14.37) - open/close 5981 (time 1787846741.07 total 315.95 last 14.64) - open/close 6258 (time 1787846751.08 total 325.96 last 27.67) - open/close 6358 (time 1787846761.95 total 336.83 last 9.20) - open/close 6593 (time 1787846772.64 total 347.51 last 21.99) - open/close 6832 (time 1787846782.91 total 357.79 last 23.26) - open/close 7090 (time 1787846793.93 total 368.80 last 23.42) - open/close 7215 (time 1787846803.93 total 378.81 last 12.50) - open/close 7505 (time 1787846814.67 total 389.55 last 27.01) - open/close 7699 (time 1787846825.65 total 400.52 last 17.68) - open/close 7952 (time 1787846835.65 total 410.53 last 25.29) - open/close 8166 (time 1787846846.20 total 421.07 last 20.29) - open/close 8326 (time 1787846856.88 total 431.75 last 14.98) - open/close 8433 (time 1787846866.88 total 441.75 last 10.70) - open/close 8519 (time 1787846877.07 total 451.95 last 8.44) - open/close 8641 (time 1787846887.07 total 461.95 last 12.20) - open/close 8689 (time 1787846897.91 total 472.79 last 4.43) - open/close 8792 (time 1787846907.92 total 482.80 last 10.29) - open/close 8863 (time 1787846918.24 total 493.12 last 6.88) - open/close 9051 (time 1787846928.24 total 503.12 last 18.80) - open/close 9258 (time 1787846938.54 total 513.42 last 20.10) - open/close 9345 (time 1787846948.55 total 523.43 last 8.69) - open/close 9424 (time 1787846958.59 total 533.46 last 7.87) - open/close 9683 (time 1787846968.60 total 543.47 last 25.88) - open/close 9863 (time 1787846979.16 total 554.04 last 17.04) - open/close 10000 (time 1787846983.83 total 558.70 last 29.39) - open/close 10192 (time 1787846994.78 total 569.65 last 17.53) - open/close 10385 (time 1787847005.22 total 580.09 last 18.49) - open/close 10535 (time 1787847017.49 total 592.36 last 12.22) - open/close 10577 (time 1787847028.07 total 602.94 last 3.97) - open/close 10704 (time 1787847038.07 total 612.94 last 12.70) - open/close 10814 (time 1787847048.07 total 622.95 last 10.99) - open/close 10942 (time 1787847058.10 total 632.97 last 12.77) - open/close 11072 (time 1787847068.14 total 643.01 last 12.95) - open/close 11174 (time 1787847078.63 total 653.50 last 9.72) - open/close 11235 (time 1787847088.63 total 663.50 last 6.10) - open/close 11303 (time 1787847100.91 total 675.79 last 5.54) - open/close 11547 (time 1787847113.96 total 688.83 last 18.71) - open/close 11810 (time 1787847123.96 total 698.83 last 26.29) - open/close 11899 (time 1787847133.97 total 708.84 last 8.89) - open/close 12149 (time 1787847144.46 total 719.34 last 23.82) - open/close 12398 (time 1787847154.72 total 729.60 last 24.27) - open/close 12589 (time 1787847164.73 total 739.60 last 19.10) - open/close 12749 (time 1787847174.73 total 749.60 last 15.99) - open/close 12870 (time 1787847185.07 total 759.95 last 11.70) - open/close 13006 (time 1787847195.22 total 770.09 last 13.41) - open/close 13202 (time 1787847205.28 total 780.16 last 19.47) - open/close 13390 (time 1787847215.30 total 790.18 last 18.76) - open/close 13579 (time 1787847226.34 total 801.21 last 17.13) - open/close 13818 (time 1787847237.08 total 811.95 last 22.25) - open/close 13959 (time 1787847250.28 total 825.15 last 10.68) - open/close 14082 (time 1787847260.89 total 835.76 last 11.59) - open/close 14275 (time 1787847272.94 total 847.81 last 16.02) - open/close 14448 (time 1787847283.44 total 858.31 last 16.48) - open/close 14576 (time 1787847293.45 total 868.32 last 12.78) - open/close 14692 (time 1787847304.53 total 879.40 last 10.47) - open/close 14907 (time 1787847314.53 total 889.41 last 21.49) - open/close 14965 (time 1787847324.56 total 899.43 last 5.79) - open/close 15096 (time 1787847334.56 total 909.43 last 13.10) - open/close 15135 (time 1787847345.11 total 919.99 last 3.70) - open/close 15415 (time 1787847355.13 total 930.00 last 27.96) - open/close 15600 (time 1787847371.30 total 946.18 last 11.44) - open/close 15676 (time 1787847382.29 total 957.17 last 6.91) - open/close 15809 (time 1787847394.44 total 969.32 last 10.95) - open/close 15942 (time 1787847404.45 total 979.32 last 13.29) - open/close 16127 (time 1787847414.48 total 989.35 last 18.44) - open/close 16227 (time 1787847424.50 total 999.37 last 9.98) - open/close 16338 (time 1787847434.50 total 1009.37 last 11.10) - open/close 16490 (time 1787847444.70 total 1019.57 last 14.90) - open/close 16576 (time 1787847457.20 total 1032.07 last 6.88) - open/close 16619 (time 1787847468.58 total 1043.46 last 3.78) - open/close 16757 (time 1787847478.59 total 1053.46 last 13.79) - open/close 17471 (time 1787847488.60 total 1063.48 last 71.31) - open/close 18170 (time 1787847498.61 total 1073.48 last 69.87) - open/close 18809 (time 1787847508.62 total 1083.50 last 63.81) - open/close 19389 (time 1787847518.65 total 1093.52 last 57.85) - open/close 19986 (time 1787847528.65 total 1103.53 last 59.66) - open/close 20726 (time 1787847538.66 total 1113.53 last 73.97) - open/close 21435 (time 1787847548.67 total 1123.54 last 70.82) - open/close 22200 (time 1787847558.68 total 1133.55 last 76.44) - open/close 22893 (time 1787847568.70 total 1143.57 last 69.16) - open/close 23638 (time 1787847578.70 total 1153.58 last 74.45) - open/close 24280 (time 1787847588.72 total 1163.59 last 64.12) - open/close 24926 (time 1787847598.72 total 1173.59 last 64.57) - open/close 25592 (time 1787847608.73 total 1183.60 last 66.54) - open/close 26230 (time 1787847618.73 total 1193.61 last 63.77) - open/close 26911 (time 1787847628.74 total 1203.61 last 68.08) - open/close 27586 (time 1787847638.74 total 1213.61 last 67.48) - open/close 28301 (time 1787847648.74 total 1223.62 last 71.49) - open/close 29073 (time 1787847658.75 total 1233.62 last 77.13) - open/close 29724 (time 1787847668.76 total 1243.64 last 65.01) - open/close 30000 (time 1787847673.36 total 1248.24 last 60.04) - open/close 30623 (time 1787847683.36 total 1258.24 last 62.29) - open/close 31376 (time 1787847693.37 total 1268.24 last 75.28) - open/close 32097 (time 1787847703.37 total 1278.25 last 72.04) - open/close 32840 (time 1787847713.38 total 1288.25 last 74.29) - open/close 33511 (time 1787847723.38 total 1298.25 last 67.10) - open/close 34277 (time 1787847733.38 total 1308.25 last 76.59) - open/close 35064 (time 1787847743.38 total 1318.25 last 78.68) - open/close 35814 (time 1787847753.39 total 1328.26 last 74.93) - open/close 36561 (time 1787847763.39 total 1338.27 last 74.69) - open/close 37183 (time 1787847773.40 total 1348.27 last 62.15) - open/close 37781 (time 1787847783.40 total 1358.27 last 59.80) - open/close 38392 (time 1787847793.41 total 1368.29 last 61.03) - open/close 38967 (time 1787847803.42 total 1378.29 last 57.47) - open/close 39612 (time 1787847813.43 total 1388.30 last 64.41) - open/close 40000 (time 1787847821.39 total 1396.27 last 48.73) - open/close 40655 (time 1787847831.40 total 1406.27 last 65.44) - open/close 41344 (time 1787847841.41 total 1416.28 last 68.85) - open/close 42080 (time 1787847851.41 total 1426.29 last 73.57) - open/close 42822 (time 1787847861.42 total 1436.30 last 74.12) - open/close 43551 (time 1787847871.43 total 1446.30 last 72.86) - open/close 44291 (time 1787847881.43 total 1456.30 last 74.00) - open/close 45006 (time 1787847891.45 total 1466.32 last 71.35) - open/close 45711 (time 1787847901.46 total 1476.33 last 70.44) - open/close 46372 (time 1787847911.48 total 1486.35 last 65.98) - open/close 47051 (time 1787847921.48 total 1496.35 last 67.88) - open/close 47822 (time 1787847931.48 total 1506.36 last 77.08) - open/close 48540 (time 1787847941.49 total 1516.36 last 71.77) - open/close 49268 (time 1787847951.49 total 1526.36 last 72.77) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1531.77 seconds: 32.40 ops/second - unlinked 0 (time 1787847961 ; total 0 ; last 0) - unlinked 10000 (time 1787848102 ; total 141 ; last 141) - unlinked 20000 (time 1787848239 ; total 278 ; last 137) - unlinked 30000 (time 1787848383 ; total 422 ; last 144) - unlinked 40000 (time 1787848516 ; total 555 ; last 133) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 686 seconds: 72.348396 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg332-server unloading modules via unload_modules_local on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing unload_modules_local oleg332-server: modules unloaded. pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2