== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:09:01 (1761286141) start mds service on oleg102-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg102-server' oleg102-server: oleg102-server.virtnet: executing load_modules_local oleg102-server: Loading modules from /home/green/git/lustre-release/lustre oleg102-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg102-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg102-server: detected 4 online CPUs by sysfs oleg102-server: MODOPTS_LIBCFS= oleg102-server: Force libcfs to create 2 CPU partitions oleg102-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg102-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg102-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg102-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-OST0000 oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg102-client.virtnet: -o user_xattr,flock 192.168.201.102@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 127 (time 1761286229.01 total 10.00 last 12.70) - open/close 283 (time 1761286239.52 total 20.51 last 14.85) - open/close 503 (time 1761286249.54 total 30.53 last 21.96) - open/close 744 (time 1761286260.58 total 41.57 last 21.83) - open/close 971 (time 1761286270.59 total 51.58 last 22.68) - open/close 1133 (time 1761286280.60 total 61.59 last 16.18) - open/close 1350 (time 1761286291.48 total 72.47 last 19.95) - open/close 1558 (time 1761286301.56 total 82.55 last 20.64) - open/close 1819 (time 1761286311.58 total 92.57 last 26.06) - open/close 2015 (time 1761286321.58 total 102.57 last 19.59) - open/close 2254 (time 1761286332.39 total 113.38 last 22.11) - open/close 2510 (time 1761286342.39 total 123.38 last 25.59) - open/close 2773 (time 1761286352.42 total 133.41 last 26.24) - open/close 2943 (time 1761286362.43 total 143.42 last 16.97) - open/close 3189 (time 1761286373.29 total 154.28 last 22.66) - open/close 3406 (time 1761286384.22 total 165.21 last 19.85) - open/close 3616 (time 1761286394.24 total 175.23 last 20.95) - open/close 3786 (time 1761286404.51 total 185.50 last 16.55) - open/close 4024 (time 1761286414.52 total 195.51 last 23.80) - open/close 4259 (time 1761286425.26 total 206.25 last 21.88) - open/close 4447 (time 1761286436.11 total 217.10 last 17.32) - open/close 4664 (time 1761286446.13 total 227.12 last 21.67) - open/close 4882 (time 1761286456.14 total 237.13 last 21.77) - open/close 5089 (time 1761286466.14 total 247.13 last 20.69) - open/close 5269 (time 1761286476.45 total 257.44 last 17.47) - open/close 5458 (time 1761286486.69 total 267.68 last 18.46) - open/close 5664 (time 1761286496.71 total 277.70 last 20.56) - open/close 5853 (time 1761286507.11 total 288.10 last 18.17) - open/close 6092 (time 1761286517.80 total 298.79 last 22.34) - open/close 6315 (time 1761286528.69 total 309.68 last 20.49) - open/close 6505 (time 1761286538.99 total 319.98 last 18.44) - open/close 6706 (time 1761286548.99 total 329.98 last 20.09) - open/close 6912 (time 1761286559.73 total 340.72 last 19.19) - open/close 7082 (time 1761286569.78 total 350.77 last 16.93) - open/close 7243 (time 1761286580.54 total 361.53 last 14.96) - open/close 7407 (time 1761286591.08 total 372.07 last 15.56) - open/close 7594 (time 1761286601.08 total 382.07 last 18.69) - open/close 7758 (time 1761286611.09 total 392.08 last 16.38) - open/close 7942 (time 1761286621.98 total 402.97 last 16.89) - open/close 8161 (time 1761286632.54 total 413.53 last 20.73) - open/close 8349 (time 1761286642.55 total 423.54 last 18.79) - open/close 8574 (time 1761286652.58 total 433.57 last 22.42) - open/close 8736 (time 1761286663.58 total 444.57 last 14.73) - open/close 8834 (time 1761286675.49 total 456.48 last 8.23) - open/close 8930 (time 1761286686.18 total 467.17 last 8.98) - open/close 9096 (time 1761286697.20 total 478.19 last 15.07) - open/close 9297 (time 1761286707.69 total 488.68 last 19.15) - open/close 9458 (time 1761286717.71 total 498.70 last 16.08) - open/close 9672 (time 1761286727.72 total 508.71 last 21.38) - open/close 9804 (time 1761286738.21 total 519.20 last 12.58) - open/close 9999 (time 1761286748.90 total 529.89 last 18.23) - open/close 10184 (time 1761286759.96 total 540.95 last 16.73) - open/close 10345 (time 1761286770.65 total 551.64 last 15.06) - open/close 10489 (time 1761286780.83 total 561.82 last 14.15) - open/close 10659 (time 1761286791.00 total 571.99 last 16.71) - open/close 10794 (time 1761286801.63 total 582.62 last 12.70) - open/close 10960 (time 1761286811.63 total 592.62 last 16.60) - open/close 11156 (time 1761286821.86 total 602.85 last 19.16) - open/close 11279 (time 1761286832.17 total 613.16 last 11.93) - open/close 11487 (time 1761286842.92 total 623.91 last 19.36) - open/close 11685 (time 1761286853.42 total 634.41 last 18.85) - open/close 11879 (time 1761286864.44 total 645.43 last 17.61) - open/close 12087 (time 1761286874.45 total 655.44 last 20.77) - open/close 12314 (time 1761286884.46 total 665.45 last 22.68) - open/close 12478 (time 1761286895.22 total 676.21 last 15.25) - open/close 12660 (time 1761286906.30 total 687.29 last 16.43) - open/close 12848 (time 1761286916.32 total 697.31 last 18.75) - open/close 13055 (time 1761286926.34 total 707.33 last 20.67) - open/close 13286 (time 1761286936.68 total 717.67 last 22.34) - open/close 13491 (time 1761286946.69 total 727.68 last 20.48) - open/close 13724 (time 1761286957.49 total 738.48 last 21.57) - open/close 13941 (time 1761286967.49 total 748.48 last 21.69) - open/close 14620 (time 1761286977.50 total 758.49 last 67.83) - open/close 15323 (time 1761286987.51 total 768.50 last 70.22) - open/close 16034 (time 1761286997.53 total 778.52 last 70.98) - open/close 16718 (time 1761287007.53 total 788.52 last 68.39) - open/close 17455 (time 1761287017.54 total 798.53 last 73.67) - open/close 18172 (time 1761287027.54 total 808.53 last 71.67) - open/close 18889 (time 1761287037.55 total 818.54 last 71.60) - open/close 19603 (time 1761287047.57 total 828.56 last 71.28) - open/close 20000 (time 1761287052.61 total 833.60 last 78.84) - open/close 20730 (time 1761287062.62 total 843.61 last 72.93) - open/close 21333 (time 1761287072.62 total 853.61 last 60.28) - open/close 22045 (time 1761287082.63 total 863.62 last 71.16) - open/close 22764 (time 1761287092.63 total 873.62 last 71.90) - open/close 23527 (time 1761287102.64 total 883.63 last 76.23) - open/close 24272 (time 1761287112.64 total 893.63 last 74.43) - open/close 24945 (time 1761287122.65 total 903.64 last 67.29) - open/close 25600 (time 1761287132.65 total 913.64 last 65.46) - open/close 26253 (time 1761287142.65 total 923.65 last 65.28) - open/close 26904 (time 1761287152.66 total 933.65 last 65.08) - open/close 27563 (time 1761287162.66 total 943.65 last 65.87) - open/close 28262 (time 1761287172.69 total 953.68 last 69.74) - open/close 28920 (time 1761287182.69 total 963.68 last 65.75) - open/close 29600 (time 1761287192.70 total 973.69 last 67.98) - open/close 30000 (time 1761287198.48 total 979.47 last 69.19) - open/close 30734 (time 1761287208.48 total 989.47 last 73.39) - open/close 31458 (time 1761287218.48 total 999.47 last 72.38) - open/close 32186 (time 1761287228.48 total 1009.47 last 72.79) - open/close 32865 (time 1761287238.49 total 1019.48 last 67.86) - open/close 33611 (time 1761287248.50 total 1029.49 last 74.49) - open/close 34355 (time 1761287258.51 total 1039.50 last 74.36) - open/close 35196 (time 1761287268.52 total 1049.51 last 84.02) - open/close 35929 (time 1761287278.52 total 1059.51 last 73.27) - open/close 36547 (time 1761287288.53 total 1069.52 last 61.76) - open/close 37115 (time 1761287298.54 total 1079.53 last 56.77) - open/close 37737 (time 1761287308.54 total 1089.53 last 62.16) - open/close 38448 (time 1761287318.55 total 1099.54 last 71.05) - open/close 39149 (time 1761287328.56 total 1109.55 last 70.00) - open/close 39820 (time 1761287338.57 total 1119.56 last 67.06) - open/close 40000 (time 1761287341.68 total 1122.67 last 57.88) - open/close 40622 (time 1761287351.69 total 1132.68 last 62.17) - open/close 41302 (time 1761287361.69 total 1142.68 last 67.96) - open/close 42009 (time 1761287371.71 total 1152.70 last 70.60) - open/close 42691 (time 1761287381.71 total 1162.70 last 68.16) - open/close 43390 (time 1761287391.72 total 1172.71 last 69.87) - open/close 44077 (time 1761287401.73 total 1182.72 last 68.62) - open/close 44803 (time 1761287411.73 total 1192.72 last 72.59) - open/close 45478 (time 1761287421.73 total 1202.72 last 67.49) - open/close 46170 (time 1761287431.74 total 1212.73 last 69.16) - open/close 46923 (time 1761287441.74 total 1222.73 last 75.29) - open/close 47631 (time 1761287451.75 total 1232.74 last 70.75) - open/close 48353 (time 1761287461.76 total 1242.75 last 72.13) - open/close 49224 (time 1761287471.76 total 1252.75 last 87.06) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1258.39 seconds: 39.44 ops/second - unlinked 0 (time 1761287481 ; total 0 ; last 0) - unlinked 10000 (time 1761287617 ; total 136 ; last 136) - unlinked 20000 (time 1761287757 ; total 276 ; last 140) - unlinked 30000 (time 1761287899 ; total 418 ; last 142) - unlinked 40000 (time 1761288041 ; total 560 ; last 142) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 690 seconds: 71.930435 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg102-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg102-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg102-server stop mds service on oleg102-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg102-server stop mds service on oleg102-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg102-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg102-server' oleg102-server: oleg102-server.virtnet: executing unload_modules_local oleg102-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg102-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg102-client: oleg102-client: ssh exited with exit code 2 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 2 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 2 pdsh@oleg102-client: oleg102-client: ssh exited with exit code 2