-----============= acceptance-small: conf-sanity ============----- Tue Dec 24 22:54:55 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg442-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 110 Stopping clients: oleg442-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg442-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg442-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg442-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg442-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg442-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg442-server oleg442-server: oleg442-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing load_modules_local oleg442-server: Loading modules from /home/green/git/lustre-release/lustre oleg442-server: detected 4 online CPUs by sysfs oleg442-server: MODOPTS_LIBCFS= oleg442-server: Force libcfs to create 2 CPU partitions oleg442-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg442-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg442-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg442-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg442-server: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg442-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg442-server: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg442-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg442-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg442-server stop mds service on oleg442-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg442-server stop mds service on oleg442-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg442-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 22:56:31 (1735098991) Stopping clients: oleg442-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg442-client.virtnet /mnt/lustre2 (opts:-f) oleg442-server: oleg442-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing load_modules_local oleg442-server: Loading modules from /home/green/git/lustre-release/lustre oleg442-server: detected 4 online CPUs by sysfs oleg442-server: MODOPTS_LIBCFS= oleg442-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg442-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg442-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg442-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3380 (time 1735099036.58 total 10.00 last 338.00) - open/close 6921 (time 1735099046.58 total 20.00 last 354.06) - open/close 9720 (time 1735099056.58 total 30.00 last 279.83) - open/close 12860 (time 1735099066.58 total 40.00 last 313.99) - open/close 17255 (time 1735099076.58 total 50.00 last 439.47) - open/close 20000 (time 1735099083.18 total 56.60 last 416.27) - open/close 23742 (time 1735099093.18 total 66.60 last 374.19) - open/close 26812 (time 1735099103.18 total 76.60 last 306.97) - open/close 29849 (time 1735099113.18 total 86.60 last 303.64) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 94.61 seconds: 344.17 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7801 (time 1735099132.16 total 10.00 last 780.10) - /unlink 10000 (time 1735099134.64 total 12.48 last 886.47) - /unlink 18704 (time 1735099144.64 total 22.48 last 870.39) - /unlink 26922 (time 1735099154.64 total 32.48 last 821.75) - /unlink 30000 (time 1735099158.85 total 36.70 last 730.33) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 39.54 seconds: 823.57 ops/second Waiting for MDT destroys to complete oleg442-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg442-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 - open/close 3002 (time 1735099197.50 total 10.00 last 300.09) - open/close 6526 (time 1735099207.51 total 20.01 last 352.26) - open/close 10000 (time 1735099217.30 total 29.80 last 354.91) - open/close 13573 (time 1735099227.30 total 39.80 last 357.25) - open/close 17183 (time 1735099237.30 total 49.80 last 360.91) - open/close 20000 (time 1735099245.33 total 57.83 last 350.99) - open/close 23528 (time 1735099255.33 total 67.83 last 352.74) - open/close 27220 (time 1735099265.33 total 77.83 last 369.08) - open/close 30000 (time 1735099273.61 total 86.11 last 335.61) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 94.00 seconds: 346.38 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 6758 (time 1735099292.57 total 10.00 last 675.52) - /unlink 10000 (time 1735099297.25 total 14.69 last 691.67) - /unlink 17200 (time 1735099307.25 total 24.69 last 719.93) - /unlink 20000 (time 1735099310.98 total 28.41 last 752.37) - /unlink 26819 (time 1735099320.98 total 38.41 last 681.90) - /unlink 30000 (time 1735099326.72 total 44.16 last 553.99) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 48.77 seconds: 667.58 ops/second Waiting for MDT destroys to complete oleg442-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg442-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 - open/close 1749 (time 1735099375.49 total 10.01 last 174.81) - open/close 3708 (time 1735099385.49 total 20.01 last 195.84) - open/close 5518 (time 1735099395.49 total 30.01 last 180.96) - open/close 7277 (time 1735099405.50 total 40.01 last 175.85) - open/close 9215 (time 1735099415.50 total 50.02 last 193.74) - open/close 10000 (time 1735099419.06 total 53.57 last 220.58) - open/close 12903 (time 1735099429.06 total 63.58 last 290.26) - open/close 16392 (time 1735099439.06 total 73.58 last 348.83) - open/close 18976 (time 1735099449.06 total 83.58 last 258.35) - open/close 20000 (time 1735099452.30 total 86.82 last 316.41) - open/close 22376 (time 1735099462.30 total 96.82 last 237.56) - open/close 24896 (time 1735099472.30 total 106.82 last 251.97) - open/close 27449 (time 1735099482.30 total 116.82 last 255.29) - open/close 29804 (time 1735099492.31 total 126.82 last 235.46) - open/close 32141 (time 1735099502.31 total 136.82 last 233.63) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 138.71 seconds: 234.74 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4405 (time 1735099515.24 total 10.00 last 440.29) - /unlink 8955 (time 1735099525.24 total 20.01 last 454.97) - /unlink 10000 (time 1735099527.66 total 22.43 last 430.79) - /unlink 14273 (time 1735099537.66 total 32.43 last 427.28) - /unlink 18908 (time 1735099547.67 total 42.44 last 463.30) - /unlink 20000 (time 1735099550.18 total 44.95 last 435.15) - /unlink 24660 (time 1735099560.18 total 54.95 last 465.91) - /unlink 28645 (time 1735099570.18 total 64.95 last 398.46) - /unlink 30000 (time 1735099573.57 total 68.34 last 399.86) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 74.89 seconds: 434.76 ops/second Waiting for MDT destroys to complete oleg442-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg442-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 - open/close 2154 (time 1735099598.70 total 10.00 last 215.36) - open/close 4197 (time 1735099608.70 total 20.00 last 204.29) - open/close 6413 (time 1735099618.70 total 30.00 last 221.57) - open/close 8339 (time 1735099628.70 total 40.00 last 192.57) - open/close 10000 (time 1735099637.11 total 48.41 last 197.53) - open/close 11907 (time 1735099647.12 total 58.42 last 190.65) - open/close 14023 (time 1735099657.12 total 68.42 last 211.55) - open/close 16255 (time 1735099667.12 total 78.42 last 223.10) - open/close 18218 (time 1735099677.13 total 88.43 last 196.19) - open/close 20000 (time 1735099687.10 total 98.40 last 178.64) - open/close 21897 (time 1735099697.11 total 108.41 last 189.61) - open/close 23857 (time 1735099707.11 total 118.41 last 195.93) - open/close 25737 (time 1735099717.12 total 128.42 last 187.94) - open/close 27583 (time 1735099727.12 total 138.42 last 184.54) - open/close 29571 (time 1735099737.12 total 148.42 last 198.74) - open/close 30000 (time 1735099739.20 total 150.50 last 206.24) - open/close 32059 (time 1735099749.20 total 160.50 last 205.85) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 162.89 seconds: 199.89 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4770 (time 1735099762.83 total 10.00 last 476.97) - /unlink 9558 (time 1735099772.84 total 20.00 last 478.71) - /unlink 13756 (time 1735099782.84 total 30.00 last 419.75) - /unlink 18143 (time 1735099792.84 total 40.00 last 438.65) - /unlink 20000 (time 1735099796.12 total 43.29 last 565.32) - /unlink 25216 (time 1735099806.12 total 53.29 last 521.54) - /unlink 29690 (time 1735099816.13 total 63.29 last 447.34) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 69.04 seconds: 471.60 ops/second Waiting for MDT destroys to complete oleg442-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg442-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 - open/close 2991 (time 1735099840.06 total 10.00 last 299.08) - open/close 5727 (time 1735099850.07 total 20.00 last 273.55) - open/close 8432 (time 1735099860.07 total 30.01 last 270.39) - open/close 10000 (time 1735099866.15 total 36.09 last 257.97) - open/close 12613 (time 1735099876.15 total 46.09 last 261.22) - open/close 15136 (time 1735099886.15 total 56.09 last 252.29) - open/close 17613 (time 1735099896.16 total 66.09 last 247.55) - open/close 19811 (time 1735099906.16 total 76.09 last 219.79) - open/close 22041 (time 1735099916.16 total 86.10 last 222.95) - open/close 23978 (time 1735099926.16 total 96.10 last 193.66) - open/close 26441 (time 1735099936.16 total 106.10 last 246.27) - open/close 28457 (time 1735099946.17 total 116.10 last 201.56) - open/close 30000 (time 1735099955.64 total 125.58 last 162.81) - open/close 31687 (time 1735099965.64 total 135.58 last 168.68) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 140.88 seconds: 231.12 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4332 (time 1735099982.27 total 10.00 last 433.16) - /unlink 8292 (time 1735099992.27 total 20.00 last 395.97) - /unlink 10000 (time 1735100001.91 total 29.64 last 177.19) - /unlink 11841 (time 1735100011.92 total 39.65 last 183.97) - /unlink 14001 (time 1735100021.93 total 49.66 last 215.76) - /unlink 16627 (time 1735100031.93 total 59.66 last 262.58) - /unlink 19363 (time 1735100041.93 total 69.66 last 273.55) - /unlink 20000 (time 1735100044.14 total 71.87 last 288.49) - /unlink 23149 (time 1735100054.14 total 81.87 last 314.82) - /unlink 26202 (time 1735100064.14 total 91.87 last 305.28) - /unlink 29101 (time 1735100074.14 total 101.87 last 289.89) - /unlink 30000 (time 1735100077.25 total 104.98 last 289.12) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 114.22 seconds: 285.06 ops/second Waiting for MDT destroys to complete oleg442-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg442-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg442-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg442-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg442-server stop mds service on oleg442-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg442-server stop mds service on oleg442-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg442-server unloading modules via unload_modules_local on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg442-client: oleg442-client: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-server: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-client: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-server: ssh exited with exit code 2 PASS 106 (1130s) Stopping clients: oleg442-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg442-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg442-client: oleg442-server: ssh exited with exit code 2 oleg442-server: oleg442-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing load_modules_local oleg442-server: Loading modules from /home/green/git/lustre-release/lustre oleg442-server: detected 4 online CPUs by sysfs oleg442-server: MODOPTS_LIBCFS= oleg442-server: Force libcfs to create 2 CPU partitions oleg442-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg442-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1250 sec ========== 23:15:46 (1735100146) === conf-sanity: start cleanup 23:15:46 (1735100146) === === conf-sanity: finish cleanup 23:15:47 (1735100147) ===