== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:16:38 (1754406998) start mds service on oleg335-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing load_modules_local oleg335-server: Loading modules from /home/green/git/lustre-release/lustre oleg335-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg335-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg335-server: detected 4 online CPUs by sysfs oleg335-server: MODOPTS_LIBCFS= oleg335-server: Force libcfs to create 2 CPU partitions oleg335-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg335-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg335-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 477 (time 1754407036.07 total 10.00 last 47.69) - open/close 1070 (time 1754407046.07 total 20.00 last 59.29) - open/close 1172 (time 1754407056.88 total 30.82 last 9.43) - open/close 1352 (time 1754407067.31 total 41.25 last 17.26) - open/close 1566 (time 1754407077.93 total 51.87 last 20.14) - open/close 1717 (time 1754407088.05 total 61.98 last 14.93) - open/close 1877 (time 1754407098.22 total 72.16 last 15.73) - open/close 2489 (time 1754407108.66 total 82.59 last 58.65) - open/close 3005 (time 1754407118.77 total 92.70 last 51.03) - open/close 3382 (time 1754407128.88 total 102.82 last 37.27) - open/close 3733 (time 1754407139.50 total 113.44 last 33.05) - open/close 3786 (time 1754407149.94 total 123.87 last 5.08) - open/close 4145 (time 1754407159.94 total 133.88 last 35.89) - open/close 4543 (time 1754407170.03 total 143.96 last 39.45) - open/close 4747 (time 1754407180.40 total 154.34 last 19.67) - open/close 5104 (time 1754407191.36 total 165.29 last 32.58) - open/close 5446 (time 1754407202.10 total 176.03 last 31.84) - open/close 5606 (time 1754407212.27 total 186.21 last 15.73) - open/close 5782 (time 1754407223.35 total 197.28 last 15.89) - open/close 5990 (time 1754407233.52 total 207.45 last 20.45) - open/close 6453 (time 1754407243.63 total 217.57 last 45.78) - open/close 6969 (time 1754407254.51 total 228.44 last 47.44) - open/close 7283 (time 1754407265.14 total 239.07 last 29.55) - open/close 7899 (time 1754407275.50 total 249.44 last 59.41) - open/close 8178 (time 1754407286.06 total 260.00 last 26.43) - open/close 8584 (time 1754407296.94 total 270.88 last 37.32) - open/close 8770 (time 1754407308.21 total 282.14 last 16.51) - open/close 9304 (time 1754407318.21 total 292.14 last 53.40) - open/close 9894 (time 1754407328.24 total 302.17 last 58.82) - open/close 10546 (time 1754407338.35 total 312.29 last 64.47) - open/close 11063 (time 1754407350.96 total 324.90 last 41.00) - open/close 11574 (time 1754407361.26 total 335.20 last 49.59) - open/close 12092 (time 1754407371.38 total 345.31 last 51.23) - open/close 13158 (time 1754407381.38 total 355.31 last 106.59) - open/close 16457 (time 1754407391.38 total 365.31 last 329.88) - open/close 19807 (time 1754407401.38 total 375.31 last 334.99) - open/close 23684 (time 1754407411.38 total 385.31 last 387.61) - open/close 27423 (time 1754407421.38 total 395.32 last 373.80) - open/close 30000 (time 1754407429.31 total 403.25 last 325.00) - open/close 34121 (time 1754407439.31 total 413.25 last 412.04) - open/close 37735 (time 1754407449.31 total 423.25 last 361.34) - open/close 40000 (time 1754407454.73 total 428.67 last 417.91) - open/close 44026 (time 1754407464.74 total 438.67 last 402.40) - open/close 46540 (time 1754407474.74 total 448.68 last 251.37) - open/close 49199 (time 1754407484.74 total 458.68 last 265.86) total: 50000 open/close in 461.96 seconds: 108.23 ops/second - unlinked 0 (time 1754407489 ; total 0 ; last 0) - unlinked 10000 (time 1754407508 ; total 19 ; last 19) - unlinked 20000 (time 1754407526 ; total 37 ; last 18) - unlinked 30000 (time 1754407542 ; total 53 ; last 16) - unlinked 40000 (time 1754407560 ; total 71 ; last 18) total: 50000 unlinks in 88 seconds: 568.181824 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg335-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server stop mds service on oleg335-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg335-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing unload_modules_local oleg335-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg335-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg335-client: oleg335-client: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-client: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2