== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:18:40 (1787797120) Stopping clients: oleg219-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg219-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg219-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg219-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg219-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg219-server oleg219-server: oleg219-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing load_modules_local oleg219-server: Loading modules from /home/green/git/lustre-release/lustre oleg219-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg219-server: detected 4 online CPUs by sysfs oleg219-server: MODOPTS_LIBCFS= oleg219-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg219-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg219-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg219-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 724 (time 1787797331.15 total 10.01 last 72.33) - open/close 1502 (time 1787797341.16 total 20.01 last 77.76) - open/close 2254 (time 1787797351.16 total 30.02 last 75.20) - open/close 3014 (time 1787797361.17 total 40.03 last 75.90) - open/close 3903 (time 1787797371.17 total 50.03 last 88.90) total: 4500 open/close in 59.14 seconds: 76.09 ops/second - /unlink 1811 (time 1787797394.80 total 10.01 last 180.96) - /unlink 2930 (time 1787797404.80 total 20.01 last 111.85) - /unlink 4164 (time 1787797414.81 total 30.02 last 123.29) total: 4500 /unlink in 32.55 seconds: 138.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 794 (time 1787797432.47 total 10.00 last 79.39) - open/close 1572 (time 1787797442.47 total 20.01 last 77.76) - open/close 2410 (time 1787797452.49 total 30.02 last 83.69) - open/close 3166 (time 1787797462.49 total 40.02 last 75.59) - open/close 4125 (time 1787797472.50 total 50.03 last 95.81) total: 4500 open/close in 54.41 seconds: 82.70 ops/second - /unlink 1515 (time 1787797491.89 total 10.00 last 151.44) - /unlink 2937 (time 1787797501.89 total 20.01 last 142.15) total: 4500 /unlink in 29.72 seconds: 151.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 795 (time 1787797526.56 total 10.01 last 79.46) - open/close 1612 (time 1787797536.56 total 20.01 last 81.70) - open/close 2420 (time 1787797546.57 total 30.02 last 80.72) - open/close 3252 (time 1787797556.58 total 40.02 last 83.16) - open/close 4155 (time 1787797566.59 total 50.03 last 90.18) total: 4500 open/close in 54.53 seconds: 82.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1430 (time 1787797586.85 total 10.00 last 142.96) - /unlink 2903 (time 1787797596.86 total 20.00 last 147.29) - /unlink 4385 (time 1787797606.86 total 30.01 last 148.10) total: 4500 /unlink in 30.81 seconds: 146.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 815 (time 1787797623.59 total 10.00 last 81.49) - open/close 1813 (time 1787797633.60 total 20.01 last 99.71) - open/close 2839 (time 1787797643.61 total 30.01 last 102.57) - open/close 3626 (time 1787797653.62 total 40.03 last 78.60) - open/close 4353 (time 1787797663.62 total 50.03 last 72.68) total: 4500 open/close in 51.93 seconds: 86.65 ops/second - /unlink 1540 (time 1787797680.37 total 10.00 last 153.94) - /unlink 2799 (time 1787797690.38 total 20.01 last 125.78) - /unlink 4302 (time 1787797700.38 total 30.01 last 150.29) total: 4500 /unlink in 31.44 seconds: 143.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 629 (time 1787797716.93 total 10.03 last 62.72) - open/close 1566 (time 1787797726.94 total 20.04 last 93.63) - open/close 2550 (time 1787797736.96 total 30.06 last 98.20) - open/close 3379 (time 1787797746.96 total 40.06 last 82.89) - open/close 4208 (time 1787797756.97 total 50.07 last 82.81) total: 4500 open/close in 53.32 seconds: 84.39 ops/second - /unlink 1610 (time 1787797774.97 total 10.00 last 160.97) - /unlink 3233 (time 1787797784.98 total 20.01 last 162.25) total: 4500 /unlink in 28.02 seconds: 160.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 784 (time 1787797807.21 total 10.00 last 78.36) - open/close 1540 (time 1787797817.22 total 20.01 last 75.55) - open/close 2446 (time 1787797827.24 total 30.04 last 90.37) - open/close 3380 (time 1787797837.25 total 40.04 last 93.36) - open/close 4228 (time 1787797847.25 total 50.04 last 84.77) total: 4500 open/close in 53.03 seconds: 84.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1421 (time 1787797864.47 total 10.00 last 142.08) - /unlink 2837 (time 1787797874.47 total 20.01 last 141.55) - /unlink 4368 (time 1787797884.48 total 30.01 last 153.06) total: 4500 /unlink in 31.10 seconds: 144.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 989 (time 1787797900.39 total 10.02 last 98.75) - open/close 1925 (time 1787797910.39 total 20.02 last 93.54) - open/close 2678 (time 1787797920.40 total 30.03 last 75.26) - open/close 3390 (time 1787797930.41 total 40.04 last 71.11) - open/close 4282 (time 1787797940.42 total 50.05 last 89.09) total: 4500 open/close in 52.53 seconds: 85.66 ops/second - /unlink 1902 (time 1787797958.05 total 10.02 last 189.87) - /unlink 3614 (time 1787797968.05 total 20.02 last 171.15) total: 4500 /unlink in 26.46 seconds: 170.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 974 (time 1787797989.59 total 10.01 last 97.32) - open/close 1771 (time 1787797999.60 total 20.02 last 79.63) - open/close 2651 (time 1787798009.60 total 30.02 last 87.96) - open/close 3524 (time 1787798019.62 total 40.04 last 87.17) - open/close 4319 (time 1787798029.62 total 50.04 last 79.45) total: 4500 open/close in 52.25 seconds: 86.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1431 (time 1787798046.06 total 10.00 last 143.10) - /unlink 2923 (time 1787798056.06 total 20.00 last 149.14) total: 4500 /unlink in 29.46 seconds: 152.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 814 (time 1787798079.40 total 10.01 last 81.33) - open/close 1645 (time 1787798089.40 total 20.01 last 83.10) - open/close 2449 (time 1787798099.41 total 30.02 last 80.34) - open/close 3304 (time 1787798109.41 total 40.02 last 85.46) - open/close 4102 (time 1787798119.42 total 50.03 last 79.72) total: 4500 open/close in 55.23 seconds: 81.48 ops/second - /unlink 1540 (time 1787798139.23 total 10.00 last 153.93) - /unlink 3062 (time 1787798149.24 total 20.01 last 152.14) - /unlink 4484 (time 1787798159.24 total 30.02 last 142.10) total: 4500 /unlink in 30.24 seconds: 148.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 755 (time 1787798174.36 total 10.00 last 75.48) - open/close 1621 (time 1787798184.37 total 20.01 last 86.54) - open/close 2417 (time 1787798194.38 total 30.02 last 79.50) - open/close 3240 (time 1787798204.39 total 40.03 last 82.25) - open/close 4081 (time 1787798214.39 total 50.03 last 84.07) total: 4500 open/close in 55.22 seconds: 81.50 ops/second - /unlink 1466 (time 1787798234.67 total 10.00 last 146.59) - /unlink 2746 (time 1787798244.67 total 20.00 last 127.95) - /unlink 4259 (time 1787798254.68 total 30.01 last 151.25) total: 4500 /unlink in 31.53 seconds: 142.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 701 (time 1787798272.40 total 10.01 last 70.03) - open/close 1509 (time 1787798282.42 total 20.03 last 80.66) - open/close 2377 (time 1787798292.43 total 30.03 last 86.74) - open/close 3119 (time 1787798302.44 total 40.05 last 74.10) - open/close 4001 (time 1787798312.45 total 50.06 last 88.09) total: 4500 open/close in 55.71 seconds: 80.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1613 (time 1787798332.53 total 10.00 last 161.25) - /unlink 3103 (time 1787798342.53 total 20.01 last 148.94) total: 4500 /unlink in 28.70 seconds: 156.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 715 (time 1787798366.73 total 10.01 last 71.40) - open/close 1518 (time 1787798376.74 total 20.02 last 80.27) - open/close 2305 (time 1787798386.74 total 30.02 last 78.70) - open/close 3122 (time 1787798396.74 total 40.02 last 81.68) - open/close 4005 (time 1787798406.75 total 50.03 last 88.25) total: 4500 open/close in 56.17 seconds: 80.12 ops/second - /unlink 1353 (time 1787798427.32 total 10.00 last 135.29) - /unlink 2956 (time 1787798437.32 total 20.01 last 160.19) - /unlink 4496 (time 1787798447.33 total 30.02 last 153.88) total: 4500 /unlink in 30.07 seconds: 149.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 631 (time 1787798462.73 total 10.00 last 63.08) - open/close 1241 (time 1787798472.74 total 20.01 last 60.97) - open/close 1965 (time 1787798482.74 total 30.01 last 72.38) - open/close 2610 (time 1787798492.74 total 40.01 last 64.49) - open/close 3447 (time 1787798502.75 total 50.02 last 83.65) - open/close 4389 (time 1787798512.75 total 60.02 last 94.14) total: 4500 open/close in 61.52 seconds: 73.15 ops/second - /unlink 1441 (time 1787798528.60 total 10.00 last 144.08) - /unlink 3067 (time 1787798538.60 total 20.00 last 162.57) total: 4500 /unlink in 29.35 seconds: 153.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 839 (time 1787798562.70 total 10.00 last 83.87) - open/close 1762 (time 1787798572.71 total 20.02 last 92.18) - open/close 2611 (time 1787798582.72 total 30.02 last 84.84) - open/close 3438 (time 1787798592.72 total 40.03 last 82.68) - open/close 4177 (time 1787798602.72 total 50.03 last 73.90) total: 4500 open/close in 54.31 seconds: 82.86 ops/second - /unlink 1747 (time 1787798620.55 total 10.00 last 174.66) - /unlink 3292 (time 1787798630.55 total 20.00 last 154.46) total: 4500 /unlink in 26.31 seconds: 171.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 858 (time 1787798651.38 total 10.00 last 85.78) - open/close 1589 (time 1787798661.40 total 20.02 last 73.00) - open/close 2287 (time 1787798671.43 total 30.05 last 69.55) - open/close 2996 (time 1787798681.44 total 40.06 last 70.85) - open/close 3837 (time 1787798691.45 total 50.07 last 84.05) total: 4500 open/close in 57.84 seconds: 77.80 ops/second - /unlink 1589 (time 1787798714.50 total 10.00 last 158.83) - /unlink 3174 (time 1787798724.50 total 20.01 last 158.46) total: 4500 /unlink in 27.61 seconds: 162.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 970 (time 1787798745.32 total 10.00 last 96.96) - open/close 1714 (time 1787798755.35 total 20.03 last 74.22) - open/close 2679 (time 1787798765.37 total 30.05 last 96.30) - open/close 3528 (time 1787798775.37 total 40.06 last 84.85) - open/close 4214 (time 1787798785.39 total 50.07 last 68.50) total: 4500 open/close in 54.23 seconds: 82.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1505 (time 1787798805.41 total 10.01 last 150.40) - /unlink 2828 (time 1787798815.41 total 20.01 last 132.24) - /unlink 4137 (time 1787798825.42 total 30.01 last 130.89) total: 4500 /unlink in 32.37 seconds: 139.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 801 (time 1787798843.42 total 10.01 last 80.03) - open/close 1731 (time 1787798853.42 total 20.01 last 92.97) - open/close 2557 (time 1787798863.43 total 30.02 last 82.53) - open/close 3516 (time 1787798873.45 total 40.03 last 95.77) - open/close 4329 (time 1787798883.45 total 50.04 last 81.26) total: 4500 open/close in 52.28 seconds: 86.07 ops/second - /unlink 1344 (time 1787798900.11 total 10.00 last 134.40) - /unlink 2778 (time 1787798910.12 total 20.01 last 143.25) - /unlink 4417 (time 1787798920.13 total 30.01 last 163.87) total: 4500 /unlink in 30.68 seconds: 146.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 826 (time 1787798936.11 total 10.02 last 82.45) - open/close 1571 (time 1787798946.12 total 20.03 last 74.42) - open/close 2353 (time 1787798956.15 total 30.05 last 78.02) - open/close 3260 (time 1787798966.15 total 40.05 last 90.69) - open/close 4123 (time 1787798976.15 total 50.06 last 86.25) total: 4500 open/close in 54.11 seconds: 83.17 ops/second - /unlink 1511 (time 1787798994.59 total 10.00 last 151.09) - /unlink 3069 (time 1787799004.59 total 20.00 last 155.76) total: 4500 /unlink in 29.82 seconds: 150.90 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95270 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg219-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg219-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server stop mds service on oleg219-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg219-server stop mds service on oleg219-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg219-server unloading modules via unload_modules_local on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing unload_modules_local oleg219-server: modules unloaded.