== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:52:37 (1769079157) start mds service on oleg141-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing load_modules_local oleg141-server: Loading modules from /home/green/git/lustre-release/lustre oleg141-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg141-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg141-server: detected 4 online CPUs by sysfs oleg141-server: MODOPTS_LIBCFS= oleg141-server: Force libcfs to create 2 CPU partitions oleg141-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg141-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg141-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg141-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-OST0000 oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg141-client.virtnet: -o user_xattr,flock 192.168.201.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 170 (time 1769079267.89 total 10.84 last 15.68) - open/close 389 (time 1769079277.90 total 20.86 last 21.87) - open/close 576 (time 1769079290.88 total 33.83 last 14.42) - open/close 772 (time 1769079300.89 total 43.84 last 19.57) - open/close 852 (time 1769079310.97 total 53.93 last 7.93) - open/close 1073 (time 1769079321.24 total 64.19 last 21.53) - open/close 1264 (time 1769079331.24 total 74.19 last 19.10) - open/close 1377 (time 1769079341.84 total 84.79 last 10.66) - open/close 1606 (time 1769079352.31 total 95.26 last 21.87) - open/close 1863 (time 1769079362.33 total 105.28 last 25.64) - open/close 2010 (time 1769079372.59 total 115.54 last 14.33) - open/close 2241 (time 1769079382.61 total 125.57 last 23.05) - open/close 2482 (time 1769079392.62 total 135.58 last 24.08) - open/close 2762 (time 1769079402.62 total 145.58 last 28.00) - open/close 2986 (time 1769079412.63 total 155.58 last 22.40) - open/close 3154 (time 1769079423.23 total 166.18 last 15.84) - open/close 3411 (time 1769079433.23 total 176.18 last 25.70) - open/close 3644 (time 1769079444.28 total 187.23 last 21.09) - open/close 3913 (time 1769079455.04 total 197.99 last 24.99) - open/close 4170 (time 1769079465.27 total 208.23 last 25.12) - open/close 4444 (time 1769079475.29 total 218.25 last 27.35) - open/close 4610 (time 1769079485.29 total 228.25 last 16.60) - open/close 4898 (time 1769079495.85 total 238.80 last 27.28) - open/close 5132 (time 1769079505.92 total 248.87 last 23.24) - open/close 5419 (time 1769079515.92 total 258.88 last 28.69) - open/close 5671 (time 1769079525.92 total 268.88 last 25.20) - open/close 5957 (time 1769079536.38 total 279.34 last 27.34) - open/close 6196 (time 1769079547.77 total 290.72 last 20.99) - open/close 6428 (time 1769079557.93 total 300.88 last 22.83) - open/close 6689 (time 1769079567.98 total 310.94 last 25.96) - open/close 6962 (time 1769079578.87 total 321.82 last 25.08) - open/close 7216 (time 1769079589.96 total 332.92 last 22.89) - open/close 7539 (time 1769079600.94 total 343.90 last 29.42) - open/close 7740 (time 1769079611.40 total 354.35 last 19.23) - open/close 7995 (time 1769079621.75 total 364.70 last 24.62) - open/close 8304 (time 1769079632.75 total 375.70 last 28.09) - open/close 8517 (time 1769079643.37 total 386.32 last 20.06) - open/close 8794 (time 1769079654.08 total 397.04 last 25.85) - open/close 9061 (time 1769079664.09 total 407.04 last 26.69) - open/close 9305 (time 1769079674.29 total 417.25 last 23.90) - open/close 9539 (time 1769079684.29 total 427.25 last 23.40) - open/close 9796 (time 1769079694.30 total 437.25 last 25.68) - open/close 10000 (time 1769079703.81 total 446.76 last 21.45) - open/close 10250 (time 1769079714.19 total 457.14 last 24.10) - open/close 10499 (time 1769079724.21 total 467.17 last 24.83) - open/close 10771 (time 1769079734.77 total 477.72 last 25.77) - open/close 11006 (time 1769079745.52 total 488.47 last 21.86) - open/close 11255 (time 1769079755.62 total 498.58 last 24.64) - open/close 11518 (time 1769079765.63 total 508.58 last 26.28) - open/close 11654 (time 1769079775.63 total 518.58 last 13.60) - open/close 11893 (time 1769079785.64 total 528.60 last 23.87) - open/close 12085 (time 1769079795.65 total 538.61 last 19.18) - open/close 12370 (time 1769079805.66 total 548.61 last 28.49) - open/close 12613 (time 1769079815.66 total 558.61 last 24.29) - open/close 12868 (time 1769079826.17 total 569.12 last 24.27) - open/close 13155 (time 1769079836.19 total 579.14 last 28.64) - open/close 13360 (time 1769079846.22 total 589.18 last 20.43) - open/close 13617 (time 1769079856.76 total 599.71 last 24.39) - open/close 13885 (time 1769079867.12 total 610.07 last 25.88) - open/close 14194 (time 1769079877.80 total 620.76 last 28.92) - open/close 14429 (time 1769079887.81 total 630.76 last 23.49) - open/close 14658 (time 1769079897.82 total 640.78 last 22.87) - open/close 14903 (time 1769079907.89 total 650.84 last 24.35) - open/close 15191 (time 1769079918.77 total 661.72 last 26.46) - open/close 15476 (time 1769079929.01 total 671.96 last 27.84) - open/close 15629 (time 1769079939.12 total 682.08 last 15.12) - open/close 15885 (time 1769079949.69 total 692.64 last 24.23) - open/close 16113 (time 1769079959.69 total 702.64 last 22.80) - open/close 16359 (time 1769079969.83 total 712.79 last 24.25) - open/close 16644 (time 1769079979.88 total 722.84 last 28.36) - open/close 16886 (time 1769079990.83 total 733.78 last 22.12) - open/close 17170 (time 1769080001.40 total 744.36 last 26.85) - open/close 17409 (time 1769080012.46 total 755.42 last 21.61) - open/close 17630 (time 1769080022.47 total 765.43 last 22.08) - open/close 17872 (time 1769080032.49 total 775.44 last 24.16) - open/close 18106 (time 1769080042.92 total 785.88 last 22.43) - open/close 18323 (time 1769080053.03 total 795.99 last 21.46) - open/close 18519 (time 1769080063.28 total 806.23 last 19.13) - open/close 18730 (time 1769080073.33 total 816.29 last 20.98) - open/close 18975 (time 1769080083.88 total 826.83 last 23.23) - open/close 19279 (time 1769080094.39 total 837.35 last 28.92) - open/close 19507 (time 1769080104.62 total 847.57 last 22.29) - open/close 19735 (time 1769080115.25 total 858.20 last 21.46) - open/close 20000 (time 1769080125.11 total 868.06 last 26.87) - open/close 20922 (time 1769080135.11 total 878.07 last 92.13) - open/close 21987 (time 1769080145.12 total 888.07 last 106.46) - open/close 23082 (time 1769080155.13 total 898.08 last 109.38) - open/close 24094 (time 1769080165.13 total 908.09 last 101.17) - open/close 25183 (time 1769080175.15 total 918.10 last 108.76) - open/close 26208 (time 1769080185.15 total 928.10 last 102.45) - open/close 27230 (time 1769080195.15 total 938.11 last 102.18) - open/close 28253 (time 1769080205.16 total 948.11 last 102.22) - open/close 29339 (time 1769080215.16 total 958.12 last 108.56) - open/close 30000 (time 1769080222.10 total 965.05 last 95.34) - open/close 31021 (time 1769080232.10 total 975.06 last 102.05) - open/close 32096 (time 1769080242.11 total 985.06 last 107.44) - open/close 33140 (time 1769080252.11 total 995.07 last 104.33) - open/close 34250 (time 1769080262.12 total 1005.07 last 110.97) - open/close 35297 (time 1769080272.12 total 1015.08 last 104.65) - open/close 36288 (time 1769080282.13 total 1025.08 last 99.06) - open/close 37177 (time 1769080292.13 total 1035.09 last 88.82) - open/close 38229 (time 1769080302.14 total 1045.09 last 105.14) - open/close 39224 (time 1769080312.15 total 1055.10 last 99.45) - open/close 40000 (time 1769080319.29 total 1062.24 last 108.63) - open/close 41090 (time 1769080329.29 total 1072.24 last 108.97) - open/close 42055 (time 1769080339.30 total 1082.25 last 96.42) - open/close 43078 (time 1769080349.31 total 1092.26 last 102.23) - open/close 44147 (time 1769080359.31 total 1102.26 last 106.87) - open/close 45195 (time 1769080369.32 total 1112.27 last 104.74) - open/close 46093 (time 1769080379.32 total 1122.27 last 89.80) - open/close 47087 (time 1769080389.32 total 1132.27 last 99.40) - open/close 48121 (time 1769080399.32 total 1142.28 last 103.32) - open/close 49122 (time 1769080409.33 total 1152.28 last 100.05) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1157.55 seconds: 42.88 ops/second - unlinked 0 (time 1769080417 ; total 0 ; last 0) - unlinked 10000 (time 1769080510 ; total 93 ; last 93) - unlinked 20000 (time 1769080582 ; total 165 ; last 72) - unlinked 30000 (time 1769080628 ; total 211 ; last 46) - unlinked 40000 (time 1769080673 ; total 256 ; last 45) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 302 seconds: 164.344376 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg141-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg141-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg141-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing unload_modules_local oleg141-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg141-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2