== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:26:18 (1754011578) start mds service on oleg334-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing load_modules_local oleg334-server: Loading modules from /home/green/git/lustre-release/lustre oleg334-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg334-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg334-server: detected 4 online CPUs by sysfs oleg334-server: MODOPTS_LIBCFS= oleg334-server: Force libcfs to create 2 CPU partitions oleg334-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg334-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg334-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg334-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-OST0000 oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 323 (time 1754011633.75 total 10.41 last 31.03) - open/close 544 (time 1754011644.37 total 21.02 last 20.82) - open/close 734 (time 1754011654.81 total 31.46 last 18.21) - open/close 913 (time 1754011665.49 total 42.15 last 16.75) - open/close 1173 (time 1754011676.12 total 52.77 last 24.47) - open/close 1399 (time 1754011686.74 total 63.40 last 21.27) - open/close 1579 (time 1754011696.92 total 73.58 last 17.68) - open/close 1732 (time 1754011707.03 total 83.68 last 15.14) - open/close 1928 (time 1754011717.27 total 93.92 last 19.15) - open/close 2136 (time 1754011728.27 total 104.93 last 18.89) - open/close 2588 (time 1754011740.13 total 116.79 last 38.12) - open/close 2891 (time 1754011750.13 total 126.79 last 30.30) - open/close 3029 (time 1754011761.18 total 137.83 last 12.49) - open/close 3258 (time 1754011771.42 total 148.08 last 22.36) - open/close 3529 (time 1754011781.91 total 158.57 last 25.83) - open/close 3801 (time 1754011792.09 total 168.75 last 26.72) - open/close 4054 (time 1754011803.10 total 179.75 last 22.99) - open/close 4261 (time 1754011813.78 total 190.44 last 19.37) - open/close 4461 (time 1754011824.21 total 200.87 last 19.18) - open/close 4577 (time 1754011835.16 total 211.82 last 10.60) - open/close 4674 (time 1754011845.97 total 222.63 last 8.97) - open/close 4786 (time 1754011856.79 total 233.44 last 10.36) - open/close 4912 (time 1754011867.80 total 244.45 last 11.44) - open/close 5112 (time 1754011878.04 total 254.69 last 19.53) - open/close 5284 (time 1754011888.34 total 264.99 last 16.69) - open/close 5562 (time 1754011898.84 total 275.49 last 26.49) - open/close 5809 (time 1754011909.33 total 285.99 last 23.53) - open/close 6298 (time 1754011919.33 total 295.99 last 48.89) - open/close 6484 (time 1754011929.43 total 306.08 last 18.43) - open/close 6661 (time 1754011939.54 total 316.20 last 17.50) - open/close 6983 (time 1754011950.10 total 326.76 last 30.49) - open/close 7427 (time 1754011960.22 total 336.87 last 43.88) - open/close 7557 (time 1754011970.22 total 346.88 last 13.00) - open/close 7768 (time 1754011980.69 total 357.35 last 20.15) - open/close 7972 (time 1754011990.69 total 367.35 last 20.40) - open/close 8762 (time 1754012000.70 total 377.35 last 78.97) - open/close 10000 (time 1754012007.17 total 383.82 last 191.31) - open/close 12725 (time 1754012017.17 total 393.83 last 272.39) - open/close 14992 (time 1754012027.17 total 403.83 last 226.64) - open/close 17201 (time 1754012037.18 total 413.83 last 220.86) - open/close 19790 (time 1754012047.18 total 423.83 last 258.85) - open/close 22580 (time 1754012057.18 total 433.83 last 278.95) - open/close 25720 (time 1754012067.18 total 443.83 last 313.98) - open/close 28283 (time 1754012077.18 total 453.84 last 256.23) - open/close 30000 (time 1754012083.62 total 460.27 last 266.73) - open/close 32760 (time 1754012093.62 total 470.28 last 275.98) - open/close 35700 (time 1754012103.62 total 480.28 last 293.91) - open/close 38200 (time 1754012113.63 total 490.28 last 249.93) - open/close 40000 (time 1754012120.21 total 496.86 last 273.46) - open/close 43071 (time 1754012130.21 total 506.87 last 306.98) - open/close 46303 (time 1754012140.21 total 516.87 last 323.14) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 526.84 seconds: 94.21 ops/second - unlinked 0 (time 1754012151 ; total 0 ; last 0) - unlinked 10000 (time 1754012169 ; total 18 ; last 18) - unlinked 20000 (time 1754012186 ; total 35 ; last 17) - unlinked 30000 (time 1754012205 ; total 54 ; last 19) - unlinked 40000 (time 1754012223 ; total 72 ; last 18) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 90 seconds: 551.466675 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg334-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg334-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg334-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing unload_modules_local oleg334-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg334-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2