== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:14:01 (1768551241) start mds service on oleg259-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg259-server' oleg259-server: oleg259-server.virtnet: executing load_modules_local oleg259-server: Loading modules from /home/green/git/lustre-release/lustre oleg259-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg259-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg259-server: detected 4 online CPUs by sysfs oleg259-server: MODOPTS_LIBCFS= oleg259-server: Force libcfs to create 2 CPU partitions oleg259-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg259-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg259-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg259-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 269 (time 1768551349.61 total 10.81 last 24.87) - open/close 499 (time 1768551359.61 total 20.82 last 22.99) - open/close 732 (time 1768551369.62 total 30.83 last 23.28) - open/close 1002 (time 1768551380.60 total 41.80 last 24.61) - open/close 1077 (time 1768551390.63 total 51.83 last 7.48) - open/close 1197 (time 1768551402.02 total 63.22 last 10.54) - open/close 1357 (time 1768551414.13 total 75.33 last 13.21) - open/close 1517 (time 1768551424.58 total 85.78 last 15.31) - open/close 1759 (time 1768551434.59 total 95.79 last 24.18) - open/close 1998 (time 1768551444.95 total 106.15 last 23.07) - open/close 2274 (time 1768551455.79 total 116.99 last 25.45) - open/close 2542 (time 1768551465.81 total 127.01 last 26.76) - open/close 2834 (time 1768551476.65 total 137.85 last 26.93) - open/close 3103 (time 1768551486.65 total 147.85 last 26.90) - open/close 3307 (time 1768551496.81 total 158.02 last 20.08) - open/close 3402 (time 1768551509.51 total 170.71 last 7.48) - open/close 3495 (time 1768551520.47 total 181.67 last 8.49) - open/close 3603 (time 1768551530.86 total 192.07 last 10.39) - open/close 3709 (time 1768551541.43 total 202.63 last 10.03) - open/close 3828 (time 1768551552.31 total 213.52 last 10.93) - open/close 4018 (time 1768551562.56 total 223.77 last 18.54) - open/close 4094 (time 1768551572.79 total 233.99 last 7.43) - open/close 4321 (time 1768551582.80 total 244.01 last 22.67) - open/close 4526 (time 1768551593.12 total 254.33 last 19.86) - open/close 4801 (time 1768551603.94 total 265.15 last 25.42) - open/close 5056 (time 1768551614.57 total 275.78 last 23.98) - open/close 5268 (time 1768551624.59 total 285.79 last 21.17) - open/close 5504 (time 1768551635.27 total 296.47 last 22.09) - open/close 5660 (time 1768551645.38 total 306.58 last 15.43) - open/close 5930 (time 1768551656.32 total 317.52 last 24.67) - open/close 6195 (time 1768551667.24 total 328.45 last 24.26) - open/close 6386 (time 1768551677.80 total 339.01 last 18.09) - open/close 6666 (time 1768551688.48 total 349.69 last 26.22) - open/close 6917 (time 1768551698.49 total 359.69 last 25.08) - open/close 7125 (time 1768551708.72 total 369.93 last 20.32) - open/close 7265 (time 1768551721.41 total 382.61 last 11.04) - open/close 7419 (time 1768551731.41 total 392.62 last 15.39) - open/close 7555 (time 1768551741.43 total 402.63 last 13.58) - open/close 7762 (time 1768551752.20 total 413.40 last 19.22) - open/close 8044 (time 1768551763.18 total 424.39 last 25.66) - open/close 8291 (time 1768551775.97 total 437.18 last 19.31) - open/close 8563 (time 1768551785.98 total 447.19 last 27.17) - open/close 8788 (time 1768551796.07 total 457.28 last 22.30) - open/close 9049 (time 1768551806.79 total 467.99 last 24.35) - open/close 9341 (time 1768551816.79 total 478.00 last 29.19) - open/close 9506 (time 1768551827.05 total 488.26 last 16.09) - open/close 9704 (time 1768551837.06 total 498.26 last 19.79) - open/close 9920 (time 1768551847.22 total 508.43 last 21.25) - open/close 10000 (time 1768551852.76 total 513.97 last 14.44) - open/close 10289 (time 1768551862.78 total 523.98 last 28.86) - open/close 10513 (time 1768551873.33 total 534.53 last 21.23) - open/close 10722 (time 1768551883.56 total 544.76 last 20.43) - open/close 10902 (time 1768551893.60 total 554.80 last 17.93) - open/close 11113 (time 1768551903.60 total 564.81 last 21.09) - open/close 11423 (time 1768551913.61 total 574.81 last 30.97) - open/close 11679 (time 1768551923.62 total 584.82 last 25.58) - open/close 11887 (time 1768551933.63 total 594.84 last 20.77) - open/close 12124 (time 1768551943.64 total 604.84 last 23.69) - open/close 12374 (time 1768551954.22 total 615.42 last 23.63) - open/close 12630 (time 1768551964.23 total 625.44 last 25.56) - open/close 12906 (time 1768551974.24 total 635.44 last 27.59) - open/close 13140 (time 1768551984.50 total 645.70 last 22.80) - open/close 13399 (time 1768551994.99 total 656.19 last 24.69) - open/close 13690 (time 1768552005.42 total 666.62 last 27.90) - open/close 13893 (time 1768552015.92 total 677.12 last 19.33) - open/close 14154 (time 1768552025.92 total 687.12 last 26.10) - open/close 14403 (time 1768552036.50 total 697.70 last 23.54) - open/close 14494 (time 1768552046.89 total 708.10 last 8.75) - open/close 14738 (time 1768552057.53 total 718.74 last 22.93) - open/close 14993 (time 1768552067.55 total 728.75 last 25.46) - open/close 15256 (time 1768552077.55 total 738.76 last 26.29) - open/close 15521 (time 1768552087.90 total 749.11 last 25.61) - open/close 15799 (time 1768552097.91 total 759.11 last 27.79) - open/close 16055 (time 1768552108.90 total 770.11 last 23.28) - open/close 16343 (time 1768552119.72 total 780.92 last 26.64) - open/close 16617 (time 1768552130.60 total 791.80 last 25.18) - open/close 16868 (time 1768552140.91 total 802.12 last 24.34) - open/close 17130 (time 1768552151.89 total 813.09 last 23.87) - open/close 17279 (time 1768552161.90 total 823.10 last 14.88) - open/close 17442 (time 1768552171.92 total 833.13 last 16.26) - open/close 17672 (time 1768552182.32 total 843.52 last 22.13) - open/close 17934 (time 1768552193.67 total 854.87 last 23.08) - open/close 18003 (time 1768552203.69 total 864.89 last 6.89) - open/close 18256 (time 1768552214.07 total 875.28 last 24.36) - open/close 18463 (time 1768552226.87 total 888.07 last 16.18) - open/close 18638 (time 1768552237.16 total 898.37 last 17.00) - open/close 18758 (time 1768552247.80 total 909.00 last 11.28) - open/close 18830 (time 1768552258.22 total 919.42 last 6.91) - open/close 19004 (time 1768552268.66 total 929.87 last 16.67) - open/close 19199 (time 1768552278.66 total 939.87 last 19.49) - open/close 19465 (time 1768552288.68 total 949.89 last 26.55) - open/close 20000 (time 1768552295.77 total 956.97 last 75.52) - open/close 20901 (time 1768552305.78 total 966.98 last 90.01) - open/close 21834 (time 1768552315.78 total 976.98 last 93.28) - open/close 22691 (time 1768552325.78 total 986.98 last 85.69) - open/close 23663 (time 1768552335.79 total 996.99 last 97.14) - open/close 24619 (time 1768552345.81 total 1007.02 last 95.35) - open/close 25693 (time 1768552355.82 total 1017.03 last 107.31) - open/close 26807 (time 1768552365.83 total 1027.03 last 111.33) - open/close 27577 (time 1768552375.83 total 1037.03 last 76.99) - open/close 28473 (time 1768552385.84 total 1047.04 last 89.54) - open/close 29348 (time 1768552395.84 total 1057.04 last 87.48) - open/close 30000 (time 1768552402.38 total 1063.59 last 99.63) - open/close 31004 (time 1768552412.39 total 1073.59 last 100.38) - open/close 31874 (time 1768552422.40 total 1083.60 last 86.89) - open/close 32925 (time 1768552432.40 total 1093.61 last 105.04) - open/close 33714 (time 1768552442.41 total 1103.61 last 78.88) - open/close 34631 (time 1768552452.41 total 1113.62 last 91.66) - open/close 35359 (time 1768552462.43 total 1123.63 last 72.67) - open/close 35863 (time 1768552472.44 total 1133.64 last 50.36) - open/close 36634 (time 1768552482.44 total 1143.64 last 77.08) - open/close 37486 (time 1768552492.45 total 1153.66 last 85.11) - open/close 38265 (time 1768552502.45 total 1163.66 last 77.88) - open/close 39146 (time 1768552512.46 total 1173.67 last 88.04) - open/close 39892 (time 1768552522.48 total 1183.69 last 74.45) - open/close 40752 (time 1768552532.48 total 1193.69 last 85.98) - open/close 41588 (time 1768552542.48 total 1203.69 last 83.60) - open/close 42416 (time 1768552552.49 total 1213.69 last 82.78) - open/close 43227 (time 1768552562.49 total 1223.70 last 81.03) - open/close 44036 (time 1768552572.50 total 1233.70 last 80.86) - open/close 44878 (time 1768552582.51 total 1243.71 last 84.11) - open/close 45976 (time 1768552592.52 total 1253.72 last 109.73) - open/close 46988 (time 1768552602.53 total 1263.73 last 101.10) - open/close 48047 (time 1768552612.53 total 1273.74 last 105.84) - open/close 48700 (time 1768552622.54 total 1283.74 last 65.27) - open/close 49594 (time 1768552632.55 total 1293.75 last 89.28) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1294.25 seconds: 38.35 ops/second - unlinked 0 (time 1768552638 ; total 0 ; last 0) - unlinked 10000 (time 1768552766 ; total 128 ; last 128) - unlinked 20000 (time 1768552876 ; total 238 ; last 110) - unlinked 30000 (time 1768552946 ; total 308 ; last 70) - unlinked 40000 (time 1768553015 ; total 377 ; last 69) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 452 seconds: 109.805313 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg259-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg259-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg259-server stop mds service on oleg259-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg259-server stop mds service on oleg259-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg259-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg259-server' oleg259-server: oleg259-server.virtnet: executing unload_modules_local oleg259-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg259-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg259-client: oleg259-client: ssh exited with exit code 2 pdsh@oleg259-client: oleg259-server: ssh exited with exit code 2 pdsh@oleg259-client: oleg259-client: ssh exited with exit code 2 pdsh@oleg259-client: oleg259-server: ssh exited with exit code 2