== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:51:00 (1769759460) start mds service on oleg148-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions oleg148-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg148-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg148-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg148-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 211 (time 1769759530.11 total 12.77 last 16.52) - open/close 366 (time 1769759540.16 total 22.82 last 15.42) - open/close 657 (time 1769759551.09 total 33.75 last 26.63) - open/close 976 (time 1769759561.10 total 43.76 last 31.87) - open/close 1143 (time 1769759571.79 total 54.45 last 15.62) - open/close 1275 (time 1769759582.46 total 65.12 last 12.37) - open/close 1393 (time 1769759595.25 total 77.91 last 9.22) - open/close 1546 (time 1769759605.25 total 87.91 last 15.30) - open/close 1692 (time 1769759615.42 total 98.08 last 14.36) - open/close 1819 (time 1769759626.17 total 108.83 last 11.82) - open/close 1970 (time 1769759636.41 total 119.07 last 14.74) - open/close 2150 (time 1769759647.03 total 129.69 last 16.96) - open/close 2399 (time 1769759657.04 total 139.69 last 24.88) - open/close 2599 (time 1769759667.83 total 150.49 last 18.53) - open/close 2693 (time 1769759678.48 total 161.14 last 8.82) - open/close 2880 (time 1769759688.52 total 171.18 last 18.63) - open/close 3045 (time 1769759698.53 total 181.19 last 16.47) - open/close 3200 (time 1769759708.60 total 191.26 last 15.39) - open/close 3430 (time 1769759718.64 total 201.30 last 22.91) - open/close 3700 (time 1769759728.83 total 211.49 last 26.49) - open/close 3960 (time 1769759739.39 total 222.05 last 24.63) - open/close 4376 (time 1769759749.39 total 232.05 last 41.59) - open/close 4652 (time 1769759760.12 total 242.77 last 25.74) - open/close 4955 (time 1769759770.75 total 253.41 last 28.48) - open/close 5294 (time 1769759780.75 total 263.41 last 33.89) - open/close 5520 (time 1769759791.24 total 273.90 last 21.56) - open/close 5908 (time 1769759801.24 total 283.90 last 38.77) - open/close 6162 (time 1769759811.38 total 294.04 last 25.07) - open/close 6408 (time 1769759822.32 total 304.98 last 22.48) - open/close 6681 (time 1769759832.95 total 315.61 last 25.69) - open/close 6929 (time 1769759843.83 total 326.49 last 22.79) - open/close 7155 (time 1769759853.84 total 336.50 last 22.57) - open/close 7485 (time 1769759863.85 total 346.51 last 32.98) - open/close 7807 (time 1769759873.85 total 356.51 last 32.19) - open/close 8037 (time 1769759884.09 total 366.75 last 22.46) - open/close 8228 (time 1769759894.09 total 376.75 last 19.10) - open/close 8455 (time 1769759904.10 total 386.76 last 22.69) - open/close 8761 (time 1769759914.74 total 397.40 last 28.75) - open/close 9086 (time 1769759925.75 total 408.41 last 29.52) - open/close 9408 (time 1769759935.80 total 418.46 last 32.03) - open/close 9633 (time 1769759946.30 total 428.96 last 21.43) - open/close 9882 (time 1769759957.11 total 439.77 last 23.03) - open/close 10000 (time 1769759961.27 total 443.93 last 28.38) - open/close 10247 (time 1769759971.83 total 454.49 last 23.39) - open/close 10546 (time 1769759981.84 total 464.49 last 29.89) - open/close 10665 (time 1769759992.32 total 474.98 last 11.35) - open/close 11010 (time 1769760002.42 total 485.08 last 34.17) - open/close 11306 (time 1769760012.85 total 495.51 last 28.39) - open/close 11839 (time 1769760022.85 total 505.51 last 53.29) - open/close 13368 (time 1769760032.85 total 515.51 last 152.87) - open/close 14751 (time 1769760042.85 total 525.51 last 138.28) - open/close 16138 (time 1769760052.86 total 535.52 last 138.65) - open/close 17470 (time 1769760062.86 total 545.52 last 133.11) - open/close 18864 (time 1769760072.87 total 555.53 last 139.26) - open/close 20000 (time 1769760080.75 total 563.41 last 144.14) - open/close 21424 (time 1769760090.76 total 573.42 last 142.29) - open/close 22855 (time 1769760100.77 total 583.43 last 143.03) - open/close 24365 (time 1769760110.77 total 593.43 last 150.93) - open/close 25813 (time 1769760120.78 total 603.44 last 144.76) - open/close 27617 (time 1769760130.78 total 613.44 last 180.38) - open/close 29174 (time 1769760140.78 total 623.44 last 155.58) - open/close 30000 (time 1769760145.25 total 627.91 last 185.04) - open/close 31825 (time 1769760155.25 total 637.91 last 182.45) - open/close 33752 (time 1769760165.26 total 647.92 last 192.58) - open/close 35970 (time 1769760175.26 total 657.92 last 221.79) - open/close 38303 (time 1769760185.26 total 667.92 last 233.26) - open/close 40000 (time 1769760191.59 total 674.25 last 267.92) - open/close 42820 (time 1769760201.60 total 684.25 last 281.96) - open/close 44961 (time 1769760211.60 total 694.26 last 214.05) - open/close 46900 (time 1769760221.60 total 704.26 last 193.79) - open/close 48931 (time 1769760231.61 total 714.27 last 202.98) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 718.21 seconds: 69.11 ops/second - unlinked 0 (time 1769760237 ; total 0 ; last 0) - unlinked 10000 (time 1769760270 ; total 33 ; last 33) - unlinked 20000 (time 1769760296 ; total 59 ; last 26) - unlinked 30000 (time 1769760322 ; total 85 ; last 26) - unlinked 40000 (time 1769760365 ; total 128 ; last 43) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 168 seconds: 295.428558 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg148-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg148-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg148-server stop mds service on oleg148-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg148-server stop mds service on oleg148-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg148-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing unload_modules_local oleg148-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg148-client: oleg148-client: ssh exited with exit code 2 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 2 pdsh@oleg148-client: oleg148-client: ssh exited with exit code 2 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 2