== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:58:12 (1785081492) start mds service on oleg343-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg343-server' oleg343-server: oleg343-server.virtnet: executing load_modules_local oleg343-server: Loading modules from /home/green/git/lustre-release/lustre oleg343-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg343-server: detected 4 online CPUs by sysfs oleg343-server: MODOPTS_LIBCFS= oleg343-server: Force libcfs to create 2 CPU partitions oleg343-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg343-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg343-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg343-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg343-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-OST0000 oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 131 (time 1785081600.51 total 10.80 last 12.13) - open/close 434 (time 1785081611.32 total 21.62 last 28.02) - open/close 676 (time 1785081622.33 total 32.62 last 21.99) - open/close 952 (time 1785081632.97 total 43.26 last 25.93) - open/close 1179 (time 1785081643.21 total 53.50 last 22.17) - open/close 1431 (time 1785081653.55 total 63.85 last 24.36) - open/close 1726 (time 1785081663.56 total 73.86 last 29.48) - open/close 1939 (time 1785081673.57 total 83.86 last 21.29) - open/close 2196 (time 1785081683.57 total 93.87 last 25.69) - open/close 2444 (time 1785081693.59 total 103.88 last 24.77) - open/close 2677 (time 1785081703.79 total 114.08 last 22.83) - open/close 2951 (time 1785081714.09 total 124.39 last 26.59) - open/close 3097 (time 1785081724.73 total 135.03 last 13.72) - open/close 3265 (time 1785081734.74 total 145.03 last 16.79) - open/close 3475 (time 1785081745.52 total 155.81 last 19.48) - open/close 3705 (time 1785081755.58 total 165.87 last 22.86) - open/close 3801 (time 1785081765.62 total 175.91 last 9.57) - open/close 4027 (time 1785081775.63 total 185.92 last 22.58) - open/close 4284 (time 1785081785.64 total 195.93 last 25.67) - open/close 4515 (time 1785081795.64 total 205.94 last 23.08) - open/close 4712 (time 1785081805.66 total 215.95 last 19.67) - open/close 4885 (time 1785081815.67 total 225.97 last 17.28) - open/close 5095 (time 1785081825.90 total 236.20 last 20.53) - open/close 5389 (time 1785081836.74 total 247.03 last 27.14) - open/close 5647 (time 1785081846.80 total 257.09 last 25.65) - open/close 5838 (time 1785081856.80 total 267.09 last 19.09) - open/close 6093 (time 1785081867.32 total 277.61 last 24.25) - open/close 6327 (time 1785081878.07 total 288.36 last 21.77) - open/close 6595 (time 1785081888.08 total 298.38 last 26.76) - open/close 6823 (time 1785081898.30 total 308.60 last 22.31) - open/close 7050 (time 1785081908.32 total 318.61 last 22.67) - open/close 7340 (time 1785081919.22 total 329.51 last 26.61) - open/close 7553 (time 1785081929.22 total 339.52 last 21.29) - open/close 7670 (time 1785081939.77 total 350.06 last 11.09) - open/close 7952 (time 1785081950.79 total 361.08 last 25.59) - open/close 8190 (time 1785081961.41 total 371.70 last 22.41) - open/close 8377 (time 1785081971.42 total 381.72 last 18.67) - open/close 8628 (time 1785081981.74 total 392.04 last 24.32) - open/close 8873 (time 1785081991.75 total 402.04 last 24.49) - open/close 9154 (time 1785082002.29 total 412.58 last 26.65) - open/close 9408 (time 1785082012.34 total 422.63 last 25.27) - open/close 9689 (time 1785082022.92 total 433.22 last 26.56) - open/close 9931 (time 1785082033.13 total 443.43 last 23.70) - open/close 10000 (time 1785082035.36 total 445.65 last 31.00) - open/close 10271 (time 1785082046.07 total 456.37 last 25.29) - open/close 10560 (time 1785082056.09 total 466.38 last 28.85) - open/close 10804 (time 1785082067.07 total 477.36 last 22.22) - open/close 11060 (time 1785082077.71 total 488.00 last 24.06) - open/close 11303 (time 1785082087.71 total 498.00 last 24.30) - open/close 11541 (time 1785082098.61 total 508.90 last 21.84) - open/close 11715 (time 1785082109.96 total 520.25 last 15.33) - open/close 11917 (time 1785082119.97 total 530.26 last 20.17) - open/close 12129 (time 1785082129.98 total 540.27 last 21.19) - open/close 12400 (time 1785082139.98 total 550.27 last 27.10) - open/close 12603 (time 1785082152.19 total 562.48 last 16.62) - open/close 12764 (time 1785082162.36 total 572.66 last 15.83) - open/close 12938 (time 1785082172.37 total 582.66 last 17.40) - open/close 13094 (time 1785082182.39 total 592.68 last 15.56) - open/close 13349 (time 1785082193.15 total 603.44 last 23.70) - open/close 13555 (time 1785082203.15 total 613.45 last 20.59) - open/close 13692 (time 1785082213.64 total 623.93 last 13.06) - open/close 13816 (time 1785082224.05 total 634.34 last 11.92) - open/close 14057 (time 1785082234.05 total 644.34 last 24.10) - open/close 14201 (time 1785082244.84 total 655.13 last 13.34) - open/close 14298 (time 1785082255.64 total 665.93 last 8.98) - open/close 14507 (time 1785082266.57 total 676.86 last 19.12) - open/close 14712 (time 1785082276.57 total 686.87 last 20.50) - open/close 14896 (time 1785082286.57 total 696.87 last 18.40) - open/close 15172 (time 1785082297.60 total 707.89 last 25.04) - open/close 15305 (time 1785082307.60 total 717.90 last 13.29) - open/close 15506 (time 1785082318.39 total 728.68 last 18.63) - open/close 15683 (time 1785082328.45 total 738.75 last 17.59) - open/close 15847 (time 1785082338.70 total 748.99 last 16.01) - open/close 16130 (time 1785082349.44 total 759.74 last 26.33) - open/close 16325 (time 1785082360.06 total 770.35 last 18.37) - open/close 16568 (time 1785082371.70 total 781.99 last 20.88) - open/close 16788 (time 1785082381.70 total 792.00 last 22.00) - open/close 16934 (time 1785082392.03 total 802.32 last 14.14) - open/close 17065 (time 1785082403.08 total 813.38 last 11.85) - open/close 17219 (time 1785082413.32 total 823.61 last 15.05) - open/close 17479 (time 1785082423.37 total 833.66 last 25.86) - open/close 17585 (time 1785082433.73 total 844.02 last 10.23) - open/close 17861 (time 1785082443.74 total 854.04 last 27.56) - open/close 18041 (time 1785082453.77 total 864.06 last 17.95) - open/close 18303 (time 1785082464.05 total 874.34 last 25.50) - open/close 18498 (time 1785082474.48 total 884.77 last 18.70) - open/close 18789 (time 1785082484.68 total 894.97 last 28.53) - open/close 18878 (time 1785082495.24 total 905.53 last 8.43) - open/close 19013 (time 1785082505.55 total 915.84 last 13.10) - open/close 19274 (time 1785082515.90 total 926.19 last 25.21) - open/close 19522 (time 1785082525.90 total 936.20 last 24.78) - open/close 19791 (time 1785082535.92 total 946.21 last 26.87) - open/close 20000 (time 1785082540.75 total 951.05 last 43.21) - open/close 20873 (time 1785082550.75 total 961.05 last 87.28) - open/close 21786 (time 1785082560.76 total 971.06 last 91.21) - open/close 22530 (time 1785082570.79 total 981.08 last 74.21) - open/close 23181 (time 1785082580.80 total 991.10 last 65.01) - open/close 23978 (time 1785082590.81 total 1001.10 last 79.69) - open/close 24818 (time 1785082600.82 total 1011.11 last 83.92) - open/close 25743 (time 1785082610.83 total 1021.13 last 92.35) - open/close 26596 (time 1785082620.84 total 1031.13 last 85.26) - open/close 27354 (time 1785082630.84 total 1041.13 last 75.79) - open/close 28207 (time 1785082640.84 total 1051.14 last 85.25) - open/close 28881 (time 1785082650.85 total 1061.15 last 67.33) - open/close 29693 (time 1785082660.86 total 1071.15 last 81.18) - open/close 30000 (time 1785082664.92 total 1075.22 last 75.48) - open/close 30905 (time 1785082674.93 total 1085.23 last 90.40) - open/close 31719 (time 1785082684.94 total 1095.23 last 81.34) - open/close 32552 (time 1785082694.95 total 1105.24 last 83.27) - open/close 33432 (time 1785082704.95 total 1115.24 last 87.96) - open/close 34235 (time 1785082714.96 total 1125.26 last 80.19) - open/close 34841 (time 1785082724.97 total 1135.26 last 60.58) - open/close 35634 (time 1785082734.99 total 1145.29 last 79.10) - open/close 36486 (time 1785082745.00 total 1155.29 last 85.13) - open/close 37325 (time 1785082755.01 total 1165.30 last 83.86) - open/close 38267 (time 1785082765.01 total 1175.30 last 94.19) - open/close 39162 (time 1785082775.02 total 1185.31 last 89.39) - open/close 40000 (time 1785082784.60 total 1194.90 last 87.45) - open/close 40855 (time 1785082794.61 total 1204.90 last 85.46) - open/close 41652 (time 1785082804.63 total 1214.92 last 79.53) - open/close 42436 (time 1785082814.63 total 1224.93 last 78.36) - open/close 43287 (time 1785082824.65 total 1234.95 last 84.92) - open/close 44176 (time 1785082834.66 total 1244.95 last 88.88) - open/close 45029 (time 1785082844.66 total 1254.96 last 85.25) - open/close 45972 (time 1785082854.66 total 1264.96 last 94.28) - open/close 46849 (time 1785082864.66 total 1274.96 last 87.69) - open/close 47757 (time 1785082874.67 total 1284.96 last 90.78) - open/close 48614 (time 1785082884.67 total 1294.96 last 85.68) - open/close 49426 (time 1785082894.67 total 1304.96 last 81.19) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1307.21 seconds: 37.97 ops/second - unlinked 0 (time 1785082901 ; total 0 ; last 0) - unlinked 10000 (time 1785083022 ; total 121 ; last 121) - unlinked 20000 (time 1785083136 ; total 235 ; last 114) - unlinked 30000 (time 1785083202 ; total 301 ; last 66) - unlinked 40000 (time 1785083274 ; total 373 ; last 72) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 436 seconds: 113.832565 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg343-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg343-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg343-server stop mds service on oleg343-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg343-server stop mds service on oleg343-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg343-server unloading modules via unload_modules_local on: 'oleg343-server' oleg343-server: oleg343-server.virtnet: executing unload_modules_local oleg343-server: modules unloaded. pdsh@oleg343-client: oleg343-client: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-server: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-client: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-server: ssh exited with exit code 2