== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:51:33 (1786182693) start mds service on oleg228-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions oleg228-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg228-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg228-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg228-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg228-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0000 oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 182 (time 1786182815.95 total 14.16 last 12.85) - open/close 366 (time 1786182825.95 total 24.16 last 18.40) - open/close 489 (time 1786182837.01 total 35.22 last 11.12) - open/close 726 (time 1786182850.01 total 48.21 last 18.24) - open/close 1011 (time 1786182860.01 total 58.22 last 28.49) - open/close 1254 (time 1786182870.81 total 69.02 last 22.49) - open/close 1404 (time 1786182881.58 total 79.78 last 13.94) - open/close 1571 (time 1786182892.06 total 90.26 last 15.93) - open/close 1763 (time 1786182902.08 total 100.29 last 19.15) - open/close 2063 (time 1786182912.10 total 110.30 last 29.96) - open/close 2257 (time 1786182922.10 total 120.31 last 19.38) - open/close 2438 (time 1786182933.91 total 132.12 last 15.33) - open/close 2656 (time 1786182944.68 total 142.89 last 20.24) - open/close 2757 (time 1786182955.20 total 153.40 last 9.61) - open/close 2902 (time 1786182965.58 total 163.79 last 13.96) - open/close 3049 (time 1786182975.59 total 173.80 last 14.69) - open/close 3262 (time 1786182986.19 total 184.40 last 20.10) - open/close 3438 (time 1786182996.20 total 194.41 last 17.58) - open/close 3512 (time 1786183007.96 total 206.17 last 6.29) - open/close 3721 (time 1786183017.97 total 216.17 last 20.89) - open/close 3932 (time 1786183028.45 total 226.66 last 20.13) - open/close 4126 (time 1786183038.87 total 237.08 last 18.62) - open/close 4316 (time 1786183048.88 total 247.09 last 18.98) - open/close 4451 (time 1786183059.31 total 257.52 last 12.94) - open/close 4687 (time 1786183070.86 total 269.07 last 20.43) - open/close 4865 (time 1786183080.87 total 279.08 last 17.78) - open/close 5104 (time 1786183090.89 total 289.09 last 23.87) - open/close 5342 (time 1786183100.91 total 299.12 last 23.75) - open/close 5595 (time 1786183110.92 total 309.12 last 25.29) - open/close 5746 (time 1786183121.23 total 319.43 last 14.64) - open/close 5959 (time 1786183131.69 total 329.90 last 20.36) - open/close 6163 (time 1786183142.11 total 340.32 last 19.58) - open/close 6346 (time 1786183152.66 total 350.86 last 17.35) - open/close 6426 (time 1786183162.67 total 360.87 last 7.99) - open/close 6623 (time 1786183172.95 total 371.16 last 19.16) - open/close 6785 (time 1786183183.78 total 381.98 last 14.96) - open/close 6939 (time 1786183194.68 total 392.88 last 14.13) - open/close 7064 (time 1786183205.41 total 403.61 last 11.65) - open/close 7236 (time 1786183215.71 total 413.92 last 16.69) - open/close 7370 (time 1786183225.73 total 423.93 last 13.38) - open/close 7615 (time 1786183235.73 total 433.94 last 24.48) - open/close 7826 (time 1786183245.74 total 443.95 last 21.09) - open/close 8003 (time 1786183256.18 total 454.38 last 16.96) - open/close 8217 (time 1786183266.19 total 464.40 last 21.37) - open/close 8409 (time 1786183277.27 total 475.48 last 17.32) - open/close 8519 (time 1786183288.03 total 486.23 last 10.23) - open/close 8767 (time 1786183298.04 total 496.25 last 24.76) - open/close 8986 (time 1786183310.74 total 508.95 last 17.25) - open/close 9143 (time 1786183322.07 total 520.28 last 13.86) - open/close 9284 (time 1786183332.67 total 530.88 last 13.30) - open/close 9509 (time 1786183344.34 total 542.55 last 19.28) - open/close 9598 (time 1786183354.81 total 553.02 last 8.50) - open/close 9751 (time 1786183365.34 total 563.54 last 14.54) - open/close 10000 (time 1786183374.92 total 573.13 last 25.97) - open/close 10210 (time 1786183385.85 total 584.06 last 19.22) - open/close 10408 (time 1786183396.39 total 594.60 last 18.78) - open/close 10614 (time 1786183407.15 total 605.35 last 19.16) - open/close 10748 (time 1786183417.16 total 615.36 last 13.38) - open/close 10854 (time 1786183427.16 total 625.37 last 10.60) - open/close 10947 (time 1786183437.17 total 635.37 last 9.30) - open/close 11091 (time 1786183447.34 total 645.54 last 14.16) - open/close 11283 (time 1786183457.87 total 656.08 last 18.23) - open/close 11444 (time 1786183468.83 total 667.03 last 14.69) - open/close 11671 (time 1786183479.57 total 677.78 last 21.12) - open/close 11874 (time 1786183490.00 total 688.21 last 19.47) - open/close 12038 (time 1786183500.02 total 698.22 last 16.38) - open/close 12188 (time 1786183510.49 total 708.69 last 14.33) - open/close 12403 (time 1786183520.82 total 719.03 last 20.80) - open/close 12629 (time 1786183531.36 total 729.57 last 21.44) - open/close 12843 (time 1786183542.00 total 740.21 last 20.12) - open/close 13050 (time 1786183552.40 total 750.61 last 19.89) - open/close 13124 (time 1786183562.86 total 761.06 last 7.08) - open/close 13231 (time 1786183575.10 total 773.30 last 8.74) - open/close 13452 (time 1786183585.96 total 784.16 last 20.34) - open/close 13670 (time 1786183595.96 total 794.17 last 21.79) - open/close 13873 (time 1786183606.24 total 804.45 last 19.75) - open/close 14013 (time 1786183616.34 total 814.54 last 13.87) - open/close 14264 (time 1786183626.36 total 824.56 last 25.05) - open/close 14464 (time 1786183636.87 total 835.07 last 19.03) - open/close 14643 (time 1786183647.67 total 845.87 last 16.57) - open/close 14854 (time 1786183657.79 total 855.99 last 20.84) - open/close 15026 (time 1786183668.00 total 866.20 last 16.85) - open/close 15177 (time 1786183679.26 total 877.47 last 13.41) - open/close 15299 (time 1786183689.49 total 887.70 last 11.92) - open/close 15611 (time 1786183699.50 total 897.71 last 31.18) - open/close 15860 (time 1786183709.78 total 907.99 last 24.22) - open/close 16085 (time 1786183720.66 total 918.86 last 20.69) - open/close 16323 (time 1786183730.66 total 928.87 last 23.79) - open/close 16511 (time 1786183741.53 total 939.74 last 17.30) - open/close 16754 (time 1786183751.54 total 949.75 last 24.28) - open/close 16973 (time 1786183762.02 total 960.23 last 20.90) - open/close 17176 (time 1786183772.12 total 970.33 last 20.10) - open/close 17236 (time 1786183783.14 total 981.35 last 5.45) - open/close 17435 (time 1786183794.59 total 992.80 last 17.37) - open/close 17520 (time 1786183806.89 total 1005.10 last 6.91) - open/close 17726 (time 1786183817.26 total 1015.46 last 19.88) - open/close 17922 (time 1786183827.26 total 1025.47 last 19.59) - open/close 18124 (time 1786183839.85 total 1038.05 last 16.05) - open/close 18743 (time 1786183849.86 total 1048.07 last 61.81) - open/close 19433 (time 1786183859.88 total 1058.08 last 68.89) - open/close 20000 (time 1786183867.28 total 1065.49 last 76.59) - open/close 20768 (time 1786183877.28 total 1075.49 last 76.78) - open/close 21436 (time 1786183887.29 total 1085.50 last 66.73) - open/close 22158 (time 1786183897.30 total 1095.51 last 72.13) - open/close 22831 (time 1786183907.33 total 1105.53 last 67.14) - open/close 23574 (time 1786183917.33 total 1115.54 last 74.28) - open/close 24337 (time 1786183927.36 total 1125.56 last 76.10) - open/close 25094 (time 1786183937.36 total 1135.56 last 75.68) - open/close 25798 (time 1786183947.37 total 1145.57 last 70.34) - open/close 26452 (time 1786183957.38 total 1155.58 last 65.35) - open/close 27092 (time 1786183967.38 total 1165.59 last 63.95) - open/close 27893 (time 1786183977.41 total 1175.61 last 79.90) - open/close 28540 (time 1786183987.43 total 1185.63 last 64.58) - open/close 29348 (time 1786183997.43 total 1195.64 last 80.74) - open/close 30000 (time 1786184004.86 total 1203.07 last 87.77) - open/close 30786 (time 1786184014.87 total 1213.07 last 78.57) - open/close 31477 (time 1786184024.88 total 1223.08 last 69.01) - open/close 32088 (time 1786184034.88 total 1233.09 last 61.08) - open/close 32701 (time 1786184044.90 total 1243.11 last 61.18) - open/close 33418 (time 1786184054.91 total 1253.12 last 71.64) - open/close 34084 (time 1786184064.91 total 1263.12 last 66.59) - open/close 34667 (time 1786184074.92 total 1273.12 last 58.26) - open/close 35475 (time 1786184084.93 total 1283.13 last 80.73) - open/close 36312 (time 1786184094.93 total 1293.13 last 83.69) - open/close 37052 (time 1786184104.94 total 1303.15 last 73.91) - open/close 37801 (time 1786184114.95 total 1313.15 last 74.84) - open/close 38652 (time 1786184124.96 total 1323.16 last 85.01) - open/close 39395 (time 1786184134.96 total 1333.17 last 74.28) - open/close 40000 (time 1786184142.82 total 1341.02 last 77.02) - open/close 40665 (time 1786184152.82 total 1351.03 last 66.45) - open/close 41493 (time 1786184162.83 total 1361.04 last 82.74) - open/close 42197 (time 1786184172.84 total 1371.05 last 70.32) - open/close 43121 (time 1786184182.84 total 1381.05 last 92.39) - open/close 43925 (time 1786184192.85 total 1391.06 last 80.32) - open/close 44693 (time 1786184202.87 total 1401.07 last 76.68) - open/close 45360 (time 1786184212.87 total 1411.08 last 66.69) - open/close 45994 (time 1786184222.88 total 1421.08 last 63.36) - open/close 46605 (time 1786184232.89 total 1431.10 last 61.01) - open/close 47157 (time 1786184242.90 total 1441.10 last 55.17) - open/close 47962 (time 1786184252.91 total 1451.12 last 80.40) - open/close 48503 (time 1786184262.91 total 1461.12 last 54.08) - open/close 49083 (time 1786184272.92 total 1471.13 last 57.95) - open/close 49623 (time 1786184282.94 total 1481.14 last 53.93) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1481.22 seconds: 33.51 ops/second - unlinked 0 (time 1786184288 ; total 0 ; last 0) - unlinked 10000 (time 1786184423 ; total 135 ; last 135) - unlinked 20000 (time 1786184559 ; total 271 ; last 136) - unlinked 30000 (time 1786184688 ; total 400 ; last 129) - unlinked 40000 (time 1786184821 ; total 533 ; last 133) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 661 seconds: 75.084717 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg228-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg228-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg228-server stop mds service on oleg228-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg228-server stop mds service on oleg228-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg228-server unloading modules via unload_modules_local on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing unload_modules_local oleg228-server: modules unloaded. pdsh@oleg228-client: oleg228-client: ssh exited with exit code 2 pdsh@oleg228-client: oleg228-server: ssh exited with exit code 2 pdsh@oleg228-client: oleg228-client: ssh exited with exit code 2 pdsh@oleg228-client: oleg228-server: ssh exited with exit code 2