== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:22:06 (1759814526) start mds service on oleg458-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg458-server' oleg458-server: oleg458-server.virtnet: executing load_modules_local oleg458-server: Loading modules from /home/green/git/lustre-release/lustre oleg458-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg458-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg458-server: detected 4 online CPUs by sysfs oleg458-server: MODOPTS_LIBCFS= oleg458-server: Force libcfs to create 2 CPU partitions oleg458-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg458-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg458-server: oleg458-server.virtnet: executing set_default_debug -1 all pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg458-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg458-server: oleg458-server.virtnet: executing set_default_debug -1 all pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Started lustre-OST0000 oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg458-client.virtnet: -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 650 (time 1759814567.58 total 10.89 last 59.71) - open/close 1284 (time 1759814578.59 total 21.90 last 57.59) - open/close 1981 (time 1759814588.63 total 31.94 last 69.38) - open/close 2635 (time 1759814599.51 total 42.82 last 60.10) - open/close 3377 (time 1759814609.52 total 52.83 last 74.19) - open/close 3969 (time 1759814619.54 total 62.85 last 59.04) - open/close 4620 (time 1759814629.60 total 72.91 last 64.75) - open/close 5341 (time 1759814639.83 total 83.14 last 70.45) - open/close 5994 (time 1759814649.83 total 93.14 last 65.29) - open/close 6613 (time 1759814660.76 total 104.07 last 56.65) - open/close 7322 (time 1759814670.76 total 114.07 last 70.89) - open/close 8026 (time 1759814681.69 total 125.00 last 64.42) - open/close 8762 (time 1759814692.70 total 136.01 last 66.86) - open/close 9428 (time 1759814702.70 total 146.01 last 66.59) - open/close 10000 (time 1759814711.51 total 154.82 last 64.92) - open/close 10557 (time 1759814722.07 total 165.38 last 52.73) - open/close 11116 (time 1759814732.70 total 176.01 last 52.59) - open/close 11866 (time 1759814742.70 total 186.01 last 74.99) - open/close 12662 (time 1759814753.05 total 196.36 last 76.91) - open/close 13338 (time 1759814763.05 total 206.36 last 67.59) - open/close 14062 (time 1759814773.28 total 216.58 last 70.82) - open/close 14683 (time 1759814784.22 total 227.53 last 56.75) - open/close 15370 (time 1759814795.10 total 238.41 last 63.16) - open/close 15948 (time 1759814805.60 total 248.90 last 55.05) - open/close 16458 (time 1759814816.28 total 259.59 last 47.73) - open/close 17039 (time 1759814826.97 total 270.28 last 54.34) - open/close 17681 (time 1759814837.59 total 280.90 last 60.44) - open/close 18172 (time 1759814848.09 total 291.40 last 46.79) - open/close 18571 (time 1759814858.27 total 301.58 last 39.20) - open/close 20000 (time 1759814864.18 total 307.49 last 241.47) - open/close 24612 (time 1759814874.19 total 317.50 last 461.13) - open/close 29022 (time 1759814884.19 total 327.50 last 440.91) - open/close 30000 (time 1759814886.32 total 329.63 last 458.05) - open/close 34965 (time 1759814896.40 total 339.71 last 492.63) - open/close 40000 (time 1759814906.00 total 349.31 last 524.70) - open/close 44416 (time 1759814916.00 total 359.31 last 441.57) - open/close 49098 (time 1759814926.00 total 369.31 last 468.17) total: 50000 open/close in 371.25 seconds: 134.68 ops/second - unlinked 0 (time 1759814929 ; total 0 ; last 0) - unlinked 10000 (time 1759814944 ; total 15 ; last 15) - unlinked 20000 (time 1759814957 ; total 28 ; last 13) - unlinked 30000 (time 1759814974 ; total 45 ; last 17) - unlinked 40000 (time 1759814990 ; total 61 ; last 16) total: 50000 unlinks in 75 seconds: 666.666687 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg458-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg458-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg458-server stop mds service on oleg458-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg458-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg458-server' oleg458-server: oleg458-server.virtnet: executing unload_modules_local oleg458-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg458-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg458-client: oleg458-client: ssh exited with exit code 2 pdsh@oleg458-client: oleg458-server: ssh exited with exit code 2 pdsh@oleg458-client: oleg458-client: ssh exited with exit code 2 pdsh@oleg458-client: oleg458-server: ssh exited with exit code 2