== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:31:39 (1760128299) start mds service on oleg101-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg101-server' oleg101-server: oleg101-server.virtnet: executing load_modules_local oleg101-server: Loading modules from /home/green/git/lustre-release/lustre oleg101-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg101-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg101-server: detected 4 online CPUs by sysfs oleg101-server: MODOPTS_LIBCFS= oleg101-server: Force libcfs to create 2 CPU partitions oleg101-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg101-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg101-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-OST0000 oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg101-client.virtnet: -o user_xattr,flock 192.168.201.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 185 (time 1760128379.49 total 10.01 last 18.48) - open/close 319 (time 1760128389.51 total 20.04 last 13.37) - open/close 538 (time 1760128399.97 total 30.50 last 20.94) - open/close 769 (time 1760128414.78 total 45.31 last 15.59) - open/close 993 (time 1760128426.39 total 56.92 last 19.30) - open/close 1240 (time 1760128437.40 total 67.92 last 22.44) - open/close 1494 (time 1760128447.41 total 77.94 last 25.36) - open/close 1716 (time 1760128457.80 total 88.33 last 21.37) - open/close 1991 (time 1760128467.81 total 98.33 last 27.48) - open/close 2207 (time 1760128477.81 total 108.33 last 21.60) - open/close 2453 (time 1760128488.59 total 119.12 last 22.80) - open/close 2658 (time 1760128498.84 total 129.36 last 20.02) - open/close 2854 (time 1760128509.32 total 139.85 last 18.69) - open/close 3042 (time 1760128519.32 total 149.85 last 18.79) - open/close 3235 (time 1760128529.33 total 159.85 last 19.29) - open/close 3422 (time 1760128539.39 total 169.92 last 18.58) - open/close 3673 (time 1760128550.28 total 180.81 last 23.05) - open/close 3870 (time 1760128562.71 total 193.24 last 15.85) - open/close 4067 (time 1760128572.73 total 203.25 last 19.67) - open/close 4282 (time 1760128582.79 total 213.32 last 21.36) - open/close 4515 (time 1760128592.85 total 223.38 last 23.15) - open/close 4765 (time 1760128602.86 total 233.38 last 24.99) - open/close 4925 (time 1760128612.94 total 243.47 last 15.87) - open/close 5172 (time 1760128623.49 total 254.02 last 23.41) - open/close 5383 (time 1760128634.08 total 264.60 last 19.93) - open/close 5594 (time 1760128645.00 total 275.52 last 19.32) - open/close 5840 (time 1760128655.77 total 286.30 last 22.83) - open/close 6076 (time 1760128666.18 total 296.71 last 22.67) - open/close 6323 (time 1760128677.25 total 307.78 last 22.32) - open/close 6550 (time 1760128687.84 total 318.37 last 21.43) - open/close 6792 (time 1760128698.24 total 328.77 last 23.27) - open/close 7008 (time 1760128708.25 total 338.78 last 21.57) - open/close 7223 (time 1760128718.49 total 349.01 last 21.01) - open/close 7458 (time 1760128728.84 total 359.36 last 22.71) - open/close 7689 (time 1760128739.58 total 370.11 last 21.50) - open/close 7914 (time 1760128749.58 total 380.11 last 22.50) - open/close 8100 (time 1760128759.87 total 390.40 last 18.08) - open/close 8339 (time 1760128769.89 total 400.42 last 23.86) - open/close 8535 (time 1760128781.06 total 411.58 last 17.55) - open/close 8780 (time 1760128792.09 total 422.61 last 22.21) - open/close 9026 (time 1760128802.09 total 432.61 last 24.59) - open/close 9246 (time 1760128812.94 total 443.47 last 20.27) - open/close 9469 (time 1760128823.76 total 454.28 last 20.62) - open/close 9714 (time 1760128834.69 total 465.21 last 22.41) - open/close 9941 (time 1760128844.69 total 475.21 last 22.70) - open/close 10000 (time 1760128846.79 total 477.32 last 28.01) - open/close 10191 (time 1760128856.79 total 487.32 last 19.10) - open/close 10398 (time 1760128872.54 total 503.06 last 13.15) - open/close 10591 (time 1760128882.84 total 513.37 last 18.73) - open/close 10789 (time 1760128892.84 total 523.37 last 19.79) - open/close 11057 (time 1760128903.77 total 534.29 last 24.54) - open/close 11223 (time 1760128914.69 total 545.22 last 15.19) - open/close 11422 (time 1760128924.94 total 555.47 last 19.42) - open/close 11666 (time 1760128934.95 total 565.48 last 24.38) - open/close 11772 (time 1760128948.19 total 578.72 last 8.01) - open/close 11982 (time 1760128958.53 total 589.06 last 20.31) - open/close 12221 (time 1760128968.53 total 599.06 last 23.89) - open/close 12365 (time 1760128978.96 total 609.49 last 13.81) - open/close 12514 (time 1760128989.64 total 620.16 last 13.96) - open/close 12727 (time 1760129000.07 total 630.60 last 20.42) - open/close 12840 (time 1760129010.71 total 641.24 last 10.62) - open/close 13094 (time 1760129023.79 total 654.32 last 19.41) - open/close 13346 (time 1760129034.05 total 664.58 last 24.56) - open/close 13563 (time 1760129044.30 total 674.83 last 21.17) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-13745) error: No space left on device total: 13745 open/close in 681.91 seconds: 20.16 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 8861: kill: (74078) - No such process - unlinked 0 (time 1760129060 ; total 0 ; last 0) - unlinked 10000 (time 1760129161 ; total 101 ; last 101) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-13745) error: No such file or directory total: 13745 unlinks in 131 seconds: 104.923668 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg101-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg101-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg101-server stop mds service on oleg101-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg101-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg101-server' oleg101-server: oleg101-server.virtnet: executing unload_modules_local oleg101-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg101-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg101-client: oleg101-client: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-server: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-client: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-server: ssh exited with exit code 2