== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:34:02 (1709055242) start mds service on oleg227-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing load_modules_local oleg227-server: Loading modules from /home/green/git/lustre-release/lustre oleg227-server: detected 4 online CPUs by sysfs oleg227-server: Force libcfs to create 2 CPU partitions oleg227-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg227-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg227-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-OST0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg227-client.virtnet: -o user_xattr,flock oleg227-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 221 (time 1709055280.06 total 10.01 last 22.09) - open/close 353 (time 1709055290.07 total 20.01 last 13.19) - open/close 647 (time 1709055300.21 total 30.16 last 28.98) - open/close 1003 (time 1709055310.59 total 40.54 last 34.29) - open/close 1251 (time 1709055321.43 total 51.37 last 22.90) - open/close 1523 (time 1709055331.50 total 61.44 last 27.01) - open/close 1676 (time 1709055341.86 total 71.81 last 14.76) - open/close 2003 (time 1709055352.78 total 82.73 last 29.94) - open/close 2290 (time 1709055362.79 total 92.73 last 28.69) - open/close 2565 (time 1709055373.36 total 103.30 last 26.01) - open/close 2772 (time 1709055384.27 total 114.21 last 18.97) - open/close 2976 (time 1709055394.27 total 124.22 last 20.40) - open/close 3241 (time 1709055404.81 total 134.75 last 25.16) - open/close 3551 (time 1709055415.07 total 145.01 last 30.20) - open/close 3821 (time 1709055425.81 total 155.75 last 25.14) - open/close 4206 (time 1709055435.81 total 165.75 last 38.50) - open/close 4360 (time 1709055446.32 total 176.26 last 14.66) - open/close 4659 (time 1709055456.75 total 186.69 last 28.67) - open/close 4876 (time 1709055466.93 total 196.87 last 21.32) - open/close 5027 (time 1709055477.59 total 207.53 last 14.16) - open/close 5338 (time 1709055487.60 total 217.54 last 31.06) - open/close 5561 (time 1709055498.27 total 228.21 last 20.91) - open/close 5890 (time 1709055508.75 total 238.69 last 31.37) - open/close 6103 (time 1709055518.85 total 248.79 last 21.09) - open/close 6308 (time 1709055528.86 total 258.80 last 20.49) - open/close 6595 (time 1709055538.90 total 268.84 last 28.57) - open/close 6837 (time 1709055549.24 total 279.18 last 23.42) - open/close 6967 (time 1709055559.70 total 289.64 last 12.42) - open/close 7186 (time 1709055572.08 total 302.02 last 17.69) - open/close 7505 (time 1709055582.34 total 312.29 last 31.08) - open/close 7754 (time 1709055593.03 total 322.97 last 23.31) - open/close 8027 (time 1709055603.68 total 333.62 last 25.63) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-8081) error: No space left on device total: 8081 open/close in 338.14 seconds: 23.90 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 8267: kill: (10593) - No such process - unlinked 0 (time 1709055611 ; total 0 ; last 0) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-8081) error: No such file or directory total: 8081 unlinks in 34 seconds: 237.676468 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg227-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg227-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server unloading modules on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg227-client: oleg227-client: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-client: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2