== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:21:22 (1764793282) start mds service on oleg133-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing load_modules_local oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg133-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg133-server: detected 4 online CPUs by sysfs oleg133-server: MODOPTS_LIBCFS= oleg133-server: Force libcfs to create 2 CPU partitions oleg133-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg133-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg133-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0000 oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 667 (time 1764793323.93 total 10.32 last 64.60) - open/close 1376 (time 1764793334.11 total 20.50 last 69.67) - open/close 2025 (time 1764793344.35 total 30.74 last 63.38) - open/close 2667 (time 1764793354.39 total 40.79 last 63.89) - open/close 3387 (time 1764793364.70 total 51.10 last 69.85) - open/close 3998 (time 1764793374.70 total 61.10 last 61.09) - open/close 4632 (time 1764793384.71 total 71.10 last 63.39) - open/close 5139 (time 1764793395.68 total 82.07 last 46.21) - open/close 5847 (time 1764793406.43 total 92.82 last 65.86) - open/close 6442 (time 1764793416.43 total 102.82 last 59.50) - open/close 7037 (time 1764793426.43 total 112.82 last 59.49) - open/close 7679 (time 1764793436.64 total 123.03 last 62.88) - open/close 8333 (time 1764793449.19 total 135.58 last 52.12) - open/close 8861 (time 1764793460.45 total 146.84 last 46.89) - open/close 9281 (time 1764793473.37 total 159.77 last 32.49) - open/close 9558 (time 1764793483.80 total 170.20 last 26.56) - open/close 9638 (time 1764793494.63 total 181.02 last 7.39) - open/close 9835 (time 1764793504.63 total 191.02 last 19.70) - open/close 10000 (time 1764793513.53 total 199.92 last 18.54) - open/close 10720 (time 1764793524.12 total 210.52 last 67.94) - open/close 11291 (time 1764793534.17 total 220.57 last 56.84) - open/close 12001 (time 1764793544.67 total 231.06 last 67.65) - open/close 12443 (time 1764793555.23 total 241.62 last 41.84) - open/close 13166 (time 1764793565.34 total 251.74 last 71.50) - open/close 13854 (time 1764793575.52 total 261.91 last 67.60) - open/close 14549 (time 1764793585.88 total 272.28 last 67.06) - open/close 15327 (time 1764793596.12 total 282.52 last 75.98) - open/close 16041 (time 1764793606.23 total 292.63 last 70.61) - open/close 16721 (time 1764793616.24 total 302.63 last 67.99) - open/close 17311 (time 1764793626.27 total 312.66 last 58.81) - open/close 17989 (time 1764793636.27 total 322.66 last 67.80) - open/close 19661 (time 1764793646.27 total 332.67 last 167.16) - open/close 23600 (time 1764793656.27 total 342.67 last 393.89) - open/close 27612 (time 1764793666.27 total 352.67 last 401.16) - open/close 30000 (time 1764793671.79 total 358.19 last 432.56) - open/close 33873 (time 1764793681.79 total 368.19 last 387.26) - open/close 37180 (time 1764793691.80 total 378.19 last 330.66) - open/close 40000 (time 1764793700.29 total 386.69 last 331.90) - open/close 43788 (time 1764793710.29 total 396.69 last 378.74) - open/close 47489 (time 1764793720.29 total 406.69 last 370.09) total: 50000 open/close in 412.59 seconds: 121.19 ops/second - unlinked 0 (time 1764793727 ; total 0 ; last 0) - unlinked 10000 (time 1764793743 ; total 16 ; last 16) - unlinked 20000 (time 1764793758 ; total 31 ; last 15) - unlinked 30000 (time 1764793772 ; total 45 ; last 14) - unlinked 40000 (time 1764793785 ; total 58 ; last 13) total: 50000 unlinks in 72 seconds: 694.444458 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg133-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg133-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server stop mds service on oleg133-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg133-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg133-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2