== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:16:45 (1761142605) start mds service on oleg357-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg357-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg357-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg357-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 712 (time 1761142646.65 total 10.04 last 70.91) - open/close 1427 (time 1761142657.66 total 21.05 last 64.94) - open/close 2210 (time 1761142667.77 total 31.16 last 77.44) - open/close 2961 (time 1761142677.81 total 41.21 last 74.75) - open/close 3610 (time 1761142688.63 total 52.03 last 60.00) - open/close 4407 (time 1761142698.94 total 62.33 last 77.34) - open/close 5267 (time 1761142709.11 total 72.51 last 84.50) - open/close 6047 (time 1761142719.11 total 82.51 last 77.99) - open/close 6717 (time 1761142730.04 total 93.43 last 61.32) - open/close 7465 (time 1761142741.05 total 104.44 last 67.95) - open/close 8264 (time 1761142751.05 total 114.45 last 79.88) - open/close 8966 (time 1761142761.91 total 125.31 last 64.63) - open/close 9804 (time 1761142772.22 total 135.61 last 81.32) - open/close 10518 (time 1761142782.22 total 145.61 last 71.39) - open/close 11277 (time 1761142792.22 total 155.61 last 75.89) - open/close 11932 (time 1761142803.00 total 166.39 last 60.76) - open/close 12657 (time 1761142813.00 total 176.40 last 72.49) - open/close 13359 (time 1761142823.93 total 187.32 last 64.26) - open/close 14023 (time 1761142834.87 total 198.27 last 60.65) - open/close 14790 (time 1761142844.88 total 208.27 last 76.69) - open/close 15440 (time 1761142855.73 total 219.13 last 59.86) - open/close 16194 (time 1761142865.85 total 229.25 last 74.54) - open/close 16946 (time 1761142875.96 total 239.36 last 74.38) - open/close 17693 (time 1761142885.96 total 249.36 last 74.68) - open/close 18416 (time 1761142895.96 total 259.36 last 72.29) - open/close 19085 (time 1761142906.87 total 270.27 last 61.33) - open/close 19825 (time 1761142916.87 total 280.27 last 73.99) - open/close 20000 (time 1761142919.34 total 282.74 last 70.97) - open/close 20652 (time 1761142930.17 total 293.56 last 60.23) - open/close 21294 (time 1761142941.05 total 304.44 last 59.00) - open/close 23914 (time 1761142951.05 total 314.44 last 261.98) - open/close 28643 (time 1761142961.05 total 324.44 last 472.84) - open/close 30000 (time 1761142963.86 total 327.25 last 482.96) - open/close 34641 (time 1761142973.86 total 337.25 last 464.10) - open/close 39290 (time 1761142983.86 total 347.26 last 464.87) - open/close 43619 (time 1761142993.86 total 357.26 last 432.89) - open/close 48210 (time 1761143003.86 total 367.26 last 459.07) total: 50000 open/close in 371.13 seconds: 134.72 ops/second - unlinked 0 (time 1761143009 ; total 0 ; last 0) - unlinked 10000 (time 1761143022 ; total 13 ; last 13) - unlinked 20000 (time 1761143035 ; total 26 ; last 13) - unlinked 30000 (time 1761143049 ; total 40 ; last 14) - unlinked 40000 (time 1761143064 ; total 55 ; last 15) total: 50000 unlinks in 68 seconds: 735.294128 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg357-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing unload_modules_local oleg357-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg357-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2