== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:50:14 (1786751414) Stopping clients: oleg234-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg234-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg234-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg234-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg234-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg234-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg234-server oleg234-server: oleg234-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing load_modules_local oleg234-server: Loading modules from /home/green/git/lustre-release/lustre oleg234-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg234-server: detected 4 online CPUs by sysfs oleg234-server: MODOPTS_LIBCFS= oleg234-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg234-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg234-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg234-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 793 (time 1786751616.84 total 10.00 last 79.26) - open/close 1567 (time 1786751626.85 total 20.01 last 77.34) - open/close 2318 (time 1786751636.85 total 30.02 last 75.08) - open/close 3137 (time 1786751646.87 total 40.03 last 81.77) - open/close 3883 (time 1786751656.88 total 50.04 last 74.53) total: 4500 open/close in 58.06 seconds: 77.51 ops/second - /unlink 1420 (time 1786751679.36 total 10.01 last 141.82) - /unlink 2663 (time 1786751689.36 total 20.02 last 124.26) - /unlink 4215 (time 1786751699.37 total 30.02 last 155.07) total: 4500 /unlink in 32.59 seconds: 138.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 756 (time 1786751716.34 total 10.01 last 75.51) - open/close 1552 (time 1786751726.35 total 20.02 last 79.56) - open/close 2394 (time 1786751736.35 total 30.02 last 84.15) - open/close 3254 (time 1786751746.36 total 40.03 last 85.97) - open/close 4078 (time 1786751756.36 total 50.03 last 82.36) total: 4500 open/close in 55.18 seconds: 81.55 ops/second - /unlink 1521 (time 1786751776.60 total 10.00 last 152.03) - /unlink 2844 (time 1786751786.61 total 20.01 last 132.26) - /unlink 4167 (time 1786751796.61 total 30.01 last 132.21) total: 4500 /unlink in 32.89 seconds: 136.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 708 (time 1786751814.63 total 10.01 last 70.75) - open/close 1259 (time 1786751824.63 total 20.01 last 55.07) - open/close 1836 (time 1786751834.64 total 30.02 last 57.66) - open/close 2473 (time 1786751844.65 total 40.03 last 63.65) - open/close 3210 (time 1786751854.65 total 50.03 last 73.66) - open/close 4100 (time 1786751864.66 total 60.04 last 88.92) total: 4500 open/close in 66.02 seconds: 68.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1434 (time 1786751885.12 total 10.01 last 143.33) - /unlink 2634 (time 1786751895.13 total 20.01 last 119.89) - /unlink 4024 (time 1786751905.14 total 30.02 last 138.91) total: 4500 /unlink in 34.66 seconds: 129.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 770 (time 1786751925.73 total 10.02 last 76.88) - open/close 1444 (time 1786751935.74 total 20.02 last 67.34) - open/close 2021 (time 1786751945.74 total 30.03 last 57.66) - open/close 2833 (time 1786751955.74 total 40.03 last 81.20) - open/close 3571 (time 1786751965.75 total 50.04 last 73.73) - open/close 4446 (time 1786751975.75 total 60.04 last 87.49) total: 4500 open/close in 60.61 seconds: 74.24 ops/second - /unlink 1537 (time 1786751990.92 total 10.01 last 153.54) - /unlink 2907 (time 1786752000.92 total 20.01 last 136.98) - /unlink 4303 (time 1786752010.92 total 30.01 last 139.56) total: 4500 /unlink in 31.47 seconds: 143.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 682 (time 1786752026.93 total 10.00 last 68.18) - open/close 1482 (time 1786752036.95 total 20.01 last 79.90) - open/close 2254 (time 1786752046.97 total 30.04 last 76.99) - open/close 2944 (time 1786752056.98 total 40.05 last 68.98) - open/close 3453 (time 1786752066.98 total 50.05 last 50.89) - open/close 4128 (time 1786752077.00 total 60.07 last 67.36) total: 4500 open/close in 64.68 seconds: 69.57 ops/second - /unlink 1405 (time 1786752095.63 total 10.00 last 140.46) - /unlink 2753 (time 1786752105.63 total 20.01 last 134.77) - /unlink 4092 (time 1786752115.63 total 30.01 last 133.84) total: 4500 /unlink in 33.16 seconds: 135.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 629 (time 1786752134.36 total 10.01 last 62.83) - open/close 1218 (time 1786752144.37 total 20.02 last 58.86) - open/close 1941 (time 1786752154.38 total 30.02 last 72.26) - open/close 2763 (time 1786752164.38 total 40.03 last 82.12) - open/close 3499 (time 1786752174.39 total 50.03 last 73.59) - open/close 4172 (time 1786752184.40 total 60.05 last 67.22) total: 4500 open/close in 63.67 seconds: 70.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1400 (time 1786752203.01 total 10.01 last 139.89) - /unlink 2790 (time 1786752213.01 total 20.01 last 138.99) - /unlink 4016 (time 1786752223.01 total 30.01 last 122.59) total: 4500 /unlink in 35.06 seconds: 128.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 709 (time 1786752243.29 total 10.01 last 70.82) - open/close 1467 (time 1786752253.30 total 20.02 last 75.74) - open/close 2306 (time 1786752263.30 total 30.02 last 83.86) - open/close 3094 (time 1786752273.31 total 40.03 last 78.75) - open/close 3869 (time 1786752283.31 total 50.03 last 77.47) total: 4500 open/close in 59.48 seconds: 75.65 ops/second - /unlink 1332 (time 1786752307.56 total 10.01 last 133.10) - /unlink 2366 (time 1786752317.56 total 20.01 last 103.37) - /unlink 3924 (time 1786752327.56 total 30.01 last 155.74) total: 4500 /unlink in 33.73 seconds: 133.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 680 (time 1786752345.89 total 10.01 last 67.93) - open/close 1320 (time 1786752355.89 total 20.01 last 63.98) - open/close 1934 (time 1786752365.90 total 30.03 last 61.32) - open/close 2688 (time 1786752375.91 total 40.03 last 75.37) - open/close 3361 (time 1786752385.91 total 50.03 last 67.29) - open/close 4074 (time 1786752395.91 total 60.04 last 71.26) total: 4500 open/close in 66.30 seconds: 67.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1171 (time 1786752418.18 total 10.00 last 117.04) - /unlink 2500 (time 1786752428.19 total 20.01 last 132.81) - /unlink 3787 (time 1786752438.19 total 30.01 last 128.69) total: 4500 /unlink in 34.81 seconds: 129.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 622 (time 1786752457.39 total 10.00 last 62.17) - open/close 1321 (time 1786752467.40 total 20.01 last 69.86) - open/close 1936 (time 1786752477.42 total 30.04 last 61.33) - open/close 2505 (time 1786752487.44 total 40.05 last 56.81) - open/close 3270 (time 1786752497.44 total 50.06 last 76.47) - open/close 4165 (time 1786752507.46 total 60.07 last 89.40) total: 4500 open/close in 63.67 seconds: 70.68 ops/second - /unlink 1463 (time 1786752525.77 total 10.00 last 146.28) - /unlink 2580 (time 1786752535.77 total 20.00 last 111.67) - /unlink 3953 (time 1786752545.78 total 30.01 last 137.17) total: 4500 /unlink in 34.01 seconds: 132.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 801 (time 1786752566.32 total 10.00 last 80.06) - open/close 1652 (time 1786752576.33 total 20.02 last 84.97) - open/close 2343 (time 1786752586.34 total 30.03 last 69.05) - open/close 3041 (time 1786752596.36 total 40.04 last 69.70) - open/close 3729 (time 1786752606.36 total 50.05 last 68.78) - open/close 4366 (time 1786752616.39 total 60.07 last 63.52) total: 4500 open/close in 62.19 seconds: 72.36 ops/second - /unlink 1547 (time 1786752632.79 total 10.00 last 154.66) - /unlink 2851 (time 1786752642.80 total 20.01 last 130.36) - /unlink 4204 (time 1786752652.81 total 30.02 last 135.14) total: 4500 /unlink in 32.80 seconds: 137.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 737 (time 1786752670.84 total 10.02 last 73.59) - open/close 1481 (time 1786752680.85 total 20.03 last 74.30) - open/close 2170 (time 1786752690.85 total 30.03 last 68.87) - open/close 2921 (time 1786752700.86 total 40.04 last 75.07) - open/close 3408 (time 1786752710.87 total 50.05 last 48.63) - open/close 3885 (time 1786752720.88 total 60.06 last 47.65) - open/close 4480 (time 1786752730.89 total 70.07 last 59.47) total: 4500 open/close in 70.27 seconds: 64.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1233 (time 1786752745.69 total 10.01 last 123.23) - /unlink 2522 (time 1786752755.69 total 20.01 last 128.90) - /unlink 3697 (time 1786752765.70 total 30.01 last 117.48) total: 4500 /unlink in 36.52 seconds: 123.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 768 (time 1786752788.33 total 10.01 last 76.72) - open/close 1429 (time 1786752798.34 total 20.02 last 66.03) - open/close 2216 (time 1786752808.35 total 30.03 last 78.64) - open/close 2967 (time 1786752818.36 total 40.04 last 75.05) - open/close 3769 (time 1786752828.36 total 50.04 last 80.16) total: 4500 open/close in 58.15 seconds: 77.38 ops/second - /unlink 1323 (time 1786752851.05 total 10.00 last 132.27) - /unlink 2799 (time 1786752861.05 total 20.01 last 147.54) - /unlink 3872 (time 1786752871.07 total 30.02 last 107.15) total: 4500 /unlink in 35.42 seconds: 127.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 648 (time 1786752891.43 total 10.01 last 64.77) - open/close 1428 (time 1786752901.44 total 20.01 last 77.94) - open/close 2147 (time 1786752911.44 total 30.02 last 71.84) - open/close 2869 (time 1786752921.45 total 40.03 last 72.14) - open/close 3573 (time 1786752931.46 total 50.04 last 70.34) - open/close 4350 (time 1786752941.47 total 60.04 last 77.65) total: 4500 open/close in 61.98 seconds: 72.60 ops/second - /unlink 1298 (time 1786752959.46 total 10.01 last 129.72) - /unlink 2773 (time 1786752969.46 total 20.01 last 147.39) - /unlink 3815 (time 1786752979.47 total 30.02 last 104.17) total: 4500 /unlink in 35.67 seconds: 126.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 843 (time 1786752999.60 total 10.00 last 84.27) - open/close 1432 (time 1786753009.60 total 20.01 last 58.89) - open/close 2113 (time 1786753019.61 total 30.01 last 68.05) - open/close 2767 (time 1786753029.61 total 40.01 last 65.39) - open/close 3524 (time 1786753039.62 total 50.02 last 75.66) - open/close 4364 (time 1786753049.63 total 60.03 last 83.91) total: 4500 open/close in 62.17 seconds: 72.38 ops/second - /unlink 1511 (time 1786753066.86 total 10.00 last 151.08) - /unlink 2753 (time 1786753076.87 total 20.01 last 124.11) - /unlink 4050 (time 1786753086.87 total 30.01 last 129.70) total: 4500 /unlink in 33.15 seconds: 135.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 654 (time 1786753105.89 total 10.01 last 65.33) - open/close 1415 (time 1786753115.90 total 20.02 last 76.05) - open/close 2157 (time 1786753125.90 total 30.02 last 74.19) - open/close 2815 (time 1786753135.91 total 40.03 last 65.73) - open/close 3648 (time 1786753145.92 total 50.04 last 83.20) - open/close 4393 (time 1786753155.94 total 60.06 last 74.38) total: 4500 open/close in 62.02 seconds: 72.55 ops/second - /unlink 1604 (time 1786753172.12 total 10.01 last 160.32) - /unlink 3032 (time 1786753182.12 total 20.01 last 142.74) - /unlink 4308 (time 1786753192.12 total 30.01 last 127.60) total: 4500 /unlink in 31.25 seconds: 143.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 880 (time 1786753208.96 total 10.01 last 87.92) - open/close 1603 (time 1786753218.97 total 20.02 last 72.21) - open/close 2175 (time 1786753228.99 total 30.04 last 57.10) - open/close 3014 (time 1786753238.99 total 40.04 last 83.89) - open/close 3804 (time 1786753249.01 total 50.06 last 78.88) - open/close 4449 (time 1786753259.02 total 60.07 last 64.42) total: 4500 open/close in 61.11 seconds: 73.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1443 (time 1786753277.51 total 10.01 last 144.14) - /unlink 2782 (time 1786753287.52 total 20.02 last 133.79) - /unlink 4295 (time 1786753297.52 total 30.02 last 151.30) total: 4500 /unlink in 31.55 seconds: 142.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 732 (time 1786753314.15 total 10.01 last 73.09) - open/close 1546 (time 1786753324.16 total 20.02 last 81.32) - open/close 2352 (time 1786753334.17 total 30.03 last 80.56) - open/close 3174 (time 1786753344.18 total 40.04 last 82.07) - open/close 3932 (time 1786753354.19 total 50.05 last 75.76) total: 4500 open/close in 56.84 seconds: 79.17 ops/second - /unlink 1252 (time 1786753375.97 total 10.01 last 125.10) - /unlink 2378 (time 1786753385.97 total 20.01 last 112.56) - /unlink 3602 (time 1786753395.97 total 30.01 last 122.39) total: 4500 /unlink in 37.00 seconds: 121.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 722 (time 1786753418.01 total 10.01 last 72.15) - open/close 1559 (time 1786753428.03 total 20.03 last 83.52) - open/close 2272 (time 1786753438.04 total 30.04 last 71.23) - open/close 2983 (time 1786753448.05 total 40.05 last 71.03) - open/close 3644 (time 1786753458.07 total 50.06 last 66.00) - open/close 4330 (time 1786753468.08 total 60.07 last 68.55) total: 4500 open/close in 62.87 seconds: 71.58 ops/second - /unlink 1086 (time 1786753486.91 total 10.01 last 108.50) - /unlink 2333 (time 1786753496.91 total 20.01 last 124.67) - /unlink 3656 (time 1786753506.91 total 30.01 last 132.27) total: 4500 /unlink in 37.20 seconds: 120.95 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95349 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg234-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg234-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg234-server unloading modules via unload_modules_local on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing unload_modules_local oleg234-server: modules unloaded.