== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:45:20 (1786715120) start mds service on oleg625-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg625-server' oleg625-server: oleg625-server.virtnet: executing load_modules_local oleg625-server: Loading modules from /home/green/git/lustre-release/lustre oleg625-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg625-server: detected 4 online CPUs by sysfs oleg625-server: MODOPTS_LIBCFS= oleg625-server: Force libcfs to create 2 CPU partitions oleg625-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg625-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg625-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg625-server: oleg625-server.virtnet: executing set_default_debug -1 all pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg625-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg625-server: oleg625-server.virtnet: executing set_default_debug -1 all pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg625-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg625-server: oleg625-server.virtnet: executing set_default_debug -1 all pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1 Started lustre-OST0000 oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg625-client.virtnet: -o user_xattr,flock 192.168.206.125@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 265 (time 1786715233.10 total 10.11 last 26.21) - open/close 507 (time 1786715243.27 total 20.28 last 23.79) - open/close 759 (time 1786715253.29 total 30.30 last 25.16) - open/close 1021 (time 1786715263.37 total 40.38 last 25.98) - open/close 1276 (time 1786715273.40 total 50.41 last 25.44) - open/close 1505 (time 1786715283.40 total 60.41 last 22.89) - open/close 1703 (time 1786715294.31 total 71.32 last 18.16) - open/close 1996 (time 1786715304.31 total 81.32 last 29.29) - open/close 2237 (time 1786715314.65 total 91.66 last 23.32) - open/close 2429 (time 1786715325.32 total 102.33 last 17.99) - open/close 2613 (time 1786715335.82 total 112.84 last 17.51) - open/close 2814 (time 1786715346.12 total 123.13 last 19.52) - open/close 3017 (time 1786715356.95 total 133.96 last 18.75) - open/close 3286 (time 1786715367.85 total 144.86 last 24.68) - open/close 3562 (time 1786715378.37 total 155.38 last 26.23) - open/close 3896 (time 1786715388.39 total 165.40 last 33.32) - open/close 4173 (time 1786715398.41 total 175.43 last 27.64) - open/close 4447 (time 1786715408.44 total 185.45 last 27.33) - open/close 4609 (time 1786715418.44 total 195.45 last 16.20) - open/close 4873 (time 1786715428.89 total 205.91 last 25.26) - open/close 5161 (time 1786715439.31 total 216.32 last 27.65) - open/close 5444 (time 1786715449.33 total 226.34 last 28.26) - open/close 5591 (time 1786715460.12 total 237.13 last 13.62) - open/close 5826 (time 1786715471.50 total 248.52 last 20.64) - open/close 6063 (time 1786715482.00 total 259.01 last 22.59) - open/close 6358 (time 1786715492.00 total 269.01 last 29.48) - open/close 6609 (time 1786715502.01 total 279.03 last 25.07) - open/close 6884 (time 1786715512.03 total 289.04 last 27.46) - open/close 7127 (time 1786715522.25 total 299.27 last 23.76) - open/close 7395 (time 1786715532.26 total 309.27 last 26.78) - open/close 7636 (time 1786715542.26 total 319.28 last 24.10) - open/close 7904 (time 1786715552.93 total 329.94 last 25.14) - open/close 8223 (time 1786715562.93 total 339.94 last 31.89) - open/close 8529 (time 1786715573.19 total 350.20 last 29.82) - open/close 8791 (time 1786715583.20 total 360.21 last 26.18) - open/close 9096 (time 1786715593.21 total 370.22 last 30.48) - open/close 9346 (time 1786715603.22 total 380.23 last 24.97) - open/close 9611 (time 1786715613.22 total 390.23 last 26.49) - open/close 9847 (time 1786715623.24 total 400.25 last 23.57) - open/close 10000 (time 1786715629.21 total 406.22 last 25.62) - open/close 10247 (time 1786715639.21 total 416.23 last 24.68) - open/close 10513 (time 1786715649.36 total 426.37 last 26.23) - open/close 10631 (time 1786715661.58 total 438.60 last 9.65) - open/close 10737 (time 1786715671.60 total 448.61 last 10.58) - open/close 10997 (time 1786715681.62 total 458.63 last 25.95) - open/close 11266 (time 1786715692.32 total 469.33 last 25.13) - open/close 11569 (time 1786715702.94 total 479.96 last 28.53) - open/close 11816 (time 1786715712.96 total 489.97 last 24.67) - open/close 12095 (time 1786715723.27 total 500.28 last 27.05) - open/close 12377 (time 1786715733.28 total 510.29 last 28.18) - open/close 12675 (time 1786715743.84 total 520.86 last 28.20) - open/close 12877 (time 1786715754.07 total 531.08 last 19.76) - open/close 13157 (time 1786715764.99 total 542.01 last 25.63) - open/close 13494 (time 1786715775.06 total 552.08 last 33.47) - open/close 13758 (time 1786715785.07 total 562.08 last 26.39) - open/close 14025 (time 1786715795.72 total 572.73 last 25.06) - open/close 14207 (time 1786715806.12 total 583.13 last 17.50) - open/close 14295 (time 1786715816.32 total 593.33 last 8.63) - open/close 14533 (time 1786715826.58 total 603.59 last 23.21) - open/close 14800 (time 1786715837.33 total 614.34 last 24.83) - open/close 15031 (time 1786715847.33 total 624.35 last 23.09) - open/close 15266 (time 1786715857.45 total 634.47 last 23.22) - open/close 15511 (time 1786715868.43 total 645.44 last 22.32) - open/close 15654 (time 1786715878.44 total 655.45 last 14.29) - open/close 15915 (time 1786715888.79 total 665.80 last 25.22) - open/close 16174 (time 1786715898.79 total 675.80 last 25.90) - open/close 16402 (time 1786715909.33 total 686.35 last 21.62) - open/close 16614 (time 1786715919.34 total 696.35 last 21.19) - open/close 16858 (time 1786715930.26 total 707.27 last 22.34) - open/close 17124 (time 1786715940.28 total 717.29 last 26.55) - open/close 17380 (time 1786715950.99 total 728.00 last 23.90) - open/close 17625 (time 1786715961.68 total 738.69 last 22.92) - open/close 17918 (time 1786715971.68 total 748.70 last 29.30) - open/close 18179 (time 1786715981.68 total 758.70 last 26.10) - open/close 18421 (time 1786715991.70 total 768.71 last 24.16) - open/close 18628 (time 1786716001.71 total 778.72 last 20.69) - open/close 18871 (time 1786716012.03 total 789.05 last 23.53) - open/close 19106 (time 1786716022.48 total 799.50 last 22.49) - open/close 19359 (time 1786716032.50 total 809.51 last 25.26) - open/close 19620 (time 1786716042.51 total 819.52 last 26.08) - open/close 19894 (time 1786716053.00 total 830.01 last 26.11) - open/close 20000 (time 1786716056.41 total 833.43 last 31.05) - open/close 20282 (time 1786716066.42 total 843.43 last 28.18) - open/close 20475 (time 1786716076.43 total 853.44 last 19.28) - open/close 20679 (time 1786716086.43 total 863.45 last 20.39) - open/close 20828 (time 1786716096.85 total 873.86 last 14.31) - open/close 21187 (time 1786716106.86 total 883.87 last 35.86) - open/close 22158 (time 1786716116.86 total 893.87 last 97.06) - open/close 23121 (time 1786716126.87 total 903.88 last 96.22) - open/close 24117 (time 1786716136.87 total 913.88 last 99.59) - open/close 25097 (time 1786716146.87 total 923.89 last 97.97) - open/close 25922 (time 1786716156.88 total 933.89 last 82.45) - open/close 26848 (time 1786716166.89 total 943.90 last 92.52) - open/close 27831 (time 1786716176.89 total 953.90 last 98.27) - open/close 28791 (time 1786716186.90 total 963.91 last 95.97) - open/close 29805 (time 1786716196.90 total 973.91 last 101.37) - open/close 30791 (time 1786716206.90 total 983.92 last 98.54) - open/close 31664 (time 1786716216.91 total 993.93 last 87.22) - open/close 32624 (time 1786716226.92 total 1003.93 last 95.98) - open/close 33617 (time 1786716236.92 total 1013.93 last 99.29) - open/close 34572 (time 1786716246.93 total 1023.94 last 95.41) - open/close 35508 (time 1786716256.93 total 1033.95 last 93.54) - open/close 36459 (time 1786716266.94 total 1043.95 last 95.05) - open/close 37384 (time 1786716276.94 total 1053.95 last 92.47) - open/close 38308 (time 1786716286.94 total 1063.96 last 92.39) - open/close 39298 (time 1786716296.95 total 1073.96 last 98.97) - open/close 40000 (time 1786716304.54 total 1081.55 last 92.45) - open/close 40927 (time 1786716314.54 total 1091.55 last 92.68) - open/close 41793 (time 1786716324.55 total 1101.56 last 86.54) - open/close 42709 (time 1786716334.56 total 1111.57 last 91.51) - open/close 43539 (time 1786716344.59 total 1121.60 last 82.78) - open/close 44498 (time 1786716354.59 total 1131.60 last 95.85) - open/close 45440 (time 1786716364.59 total 1141.61 last 94.18) - open/close 46457 (time 1786716374.60 total 1151.61 last 101.68) - open/close 47360 (time 1786716384.62 total 1161.63 last 90.11) - open/close 48295 (time 1786716394.62 total 1171.63 last 93.47) - open/close 49186 (time 1786716404.62 total 1181.64 last 89.07) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1186.19 seconds: 41.84 ops/second - unlinked 0 (time 1786716413 ; total 0 ; last 0) - unlinked 10000 (time 1786716537 ; total 124 ; last 124) - unlinked 20000 (time 1786716647 ; total 234 ; last 110) - unlinked 30000 (time 1786716711 ; total 298 ; last 64) - unlinked 40000 (time 1786716780 ; total 367 ; last 69) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 429 seconds: 115.689980 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg625-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg625-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg625-server stop mds service on oleg625-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg625-server stop mds service on oleg625-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg625-server unloading modules via unload_modules_local on: 'oleg625-server' oleg625-server: oleg625-server.virtnet: executing unload_modules_local oleg625-server: modules unloaded. pdsh@oleg625-client: oleg625-server: ssh exited with exit code 2 pdsh@oleg625-client: oleg625-client: ssh exited with exit code 2 pdsh@oleg625-client: oleg625-server: ssh exited with exit code 2 pdsh@oleg625-client: oleg625-client: ssh exited with exit code 2