== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:39:07 (1752511147) start mds service on oleg407-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing load_modules_local oleg407-server: Loading modules from /home/green/git/lustre-release/lustre oleg407-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg407-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg407-server: detected 4 online CPUs by sysfs oleg407-server: MODOPTS_LIBCFS= oleg407-server: Force libcfs to create 2 CPU partitions oleg407-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg407-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg407-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-OST0000 oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 783 (time 1752511183.55 total 10.34 last 75.72) - open/close 1799 (time 1752511194.36 total 21.15 last 93.96) - open/close 2712 (time 1752511204.92 total 31.71 last 86.45) - open/close 3196 (time 1752511215.42 total 42.21 last 46.11) - open/close 4012 (time 1752511225.78 total 52.58 last 78.70) - open/close 4657 (time 1752511238.71 total 65.51 last 49.88) - open/close 5329 (time 1752511248.72 total 75.51 last 67.20) - open/close 6010 (time 1752511258.72 total 85.51 last 68.10) - open/close 6680 (time 1752511269.50 total 96.29 last 62.14) - open/close 7584 (time 1752511280.12 total 106.92 last 85.08) - open/close 8273 (time 1752511290.12 total 116.92 last 68.90) - open/close 9084 (time 1752511300.66 total 127.46 last 76.95) - open/close 9775 (time 1752511310.66 total 137.46 last 69.09) - open/close 10000 (time 1752511313.40 total 140.19 last 82.31) - open/close 10730 (time 1752511323.64 total 150.44 last 71.25) - open/close 11523 (time 1752511333.88 total 160.68 last 77.44) - open/close 12188 (time 1752511344.83 total 171.62 last 60.77) - open/close 12939 (time 1752511355.83 total 182.63 last 68.25) - open/close 13848 (time 1752511366.33 total 193.12 last 86.58) - open/close 14624 (time 1752511376.64 total 203.43 last 75.30) - open/close 15264 (time 1752511386.64 total 213.43 last 63.99) - open/close 15869 (time 1752511397.58 total 224.38 last 55.28) - open/close 16093 (time 1752511408.58 total 235.37 last 20.37) - open/close 16189 (time 1752511419.13 total 245.93 last 9.10) - open/close 16349 (time 1752511429.25 total 256.05 last 15.81) - open/close 16528 (time 1752511441.29 total 268.08 last 14.88) - open/close 16841 (time 1752511451.84 total 278.63 last 29.67) - open/close 17126 (time 1752511462.66 total 289.45 last 26.33) - open/close 17240 (time 1752511472.66 total 299.46 last 11.40) - open/close 17410 (time 1752511482.94 total 309.73 last 16.55) - open/close 17733 (time 1752511493.51 total 320.30 last 30.55) - open/close 17884 (time 1752511503.62 total 330.41 last 14.94) - open/close 18084 (time 1752511513.78 total 340.58 last 19.67) - open/close 18453 (time 1752511524.35 total 351.14 last 34.93) - open/close 19498 (time 1752511534.35 total 361.14 last 104.48) - open/close 22019 (time 1752511544.35 total 371.15 last 252.02) - open/close 23868 (time 1752511554.35 total 381.15 last 184.86) - open/close 26030 (time 1752511564.35 total 391.15 last 216.18) - open/close 28478 (time 1752511574.36 total 401.15 last 244.79) - open/close 30000 (time 1752511580.81 total 407.61 last 235.65) - open/close 32830 (time 1752511590.81 total 417.61 last 282.99) - open/close 35280 (time 1752511600.82 total 427.61 last 244.89) - open/close 37484 (time 1752511610.82 total 437.61 last 220.39) - open/close 39851 (time 1752511620.82 total 447.62 last 236.65) - open/close 42275 (time 1752511630.83 total 457.62 last 242.26) - open/close 44946 (time 1752511640.83 total 467.63 last 267.04) - open/close 47728 (time 1752511650.83 total 477.63 last 278.06) - open/close 49939 (time 1752511660.84 total 487.63 last 221.08) total: 50000 open/close in 487.87 seconds: 102.49 ops/second - unlinked 0 (time 1752511662 ; total 0 ; last 0) - unlinked 10000 (time 1752511691 ; total 29 ; last 29) - unlinked 20000 (time 1752511715 ; total 53 ; last 24) - unlinked 30000 (time 1752511735 ; total 73 ; last 20) - unlinked 40000 (time 1752511755 ; total 93 ; last 20) total: 50000 unlinks in 113 seconds: 442.477875 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg407-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg407-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server stop mds service on oleg407-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg407-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing unload_modules_local oleg407-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg407-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2