== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:18:49 (1786645129) Stopping clients: oleg118-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg118-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg118-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg118-server oleg118-server: oleg118-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing load_modules_local oleg118-server: Loading modules from /home/green/git/lustre-release/lustre oleg118-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg118-server: detected 4 online CPUs by sysfs oleg118-server: MODOPTS_LIBCFS= oleg118-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg118-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg118-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg118-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 742 (time 1786645350.09 total 10.01 last 74.14) - open/close 1404 (time 1786645360.10 total 20.02 last 66.13) - open/close 2008 (time 1786645370.10 total 30.03 last 60.35) - open/close 2722 (time 1786645380.11 total 40.03 last 71.34) - open/close 3525 (time 1786645390.12 total 50.04 last 80.26) - open/close 4244 (time 1786645400.13 total 60.05 last 71.84) total: 4500 open/close in 63.76 seconds: 70.57 ops/second - /unlink 1296 (time 1786645419.17 total 10.00 last 129.57) - /unlink 2540 (time 1786645429.17 total 20.01 last 124.31) - /unlink 4061 (time 1786645439.18 total 30.01 last 152.09) total: 4500 /unlink in 33.05 seconds: 136.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 661 (time 1786645457.88 total 10.01 last 66.07) - open/close 1182 (time 1786645467.89 total 20.02 last 52.03) - open/close 1712 (time 1786645477.90 total 30.03 last 52.95) - open/close 2355 (time 1786645487.91 total 40.03 last 64.26) - open/close 3105 (time 1786645497.91 total 50.04 last 74.96) - open/close 3841 (time 1786645507.92 total 60.05 last 73.53) total: 4500 open/close in 67.86 seconds: 66.32 ops/second - /unlink 1470 (time 1786645529.97 total 10.00 last 146.99) - /unlink 2829 (time 1786645539.99 total 20.02 last 135.69) - /unlink 4185 (time 1786645549.99 total 30.02 last 135.57) total: 4500 /unlink in 33.37 seconds: 134.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 428 (time 1786645569.66 total 10.03 last 42.68) - open/close 818 (time 1786645579.68 total 20.04 last 38.94) - open/close 1284 (time 1786645589.69 total 30.05 last 46.57) - open/close 1813 (time 1786645599.69 total 40.05 last 52.88) - open/close 2492 (time 1786645609.70 total 50.07 last 67.81) - open/close 3320 (time 1786645619.70 total 60.07 last 82.80) - open/close 3919 (time 1786645629.71 total 70.08 last 59.83) total: 4500 open/close in 79.26 seconds: 56.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1240 (time 1786645653.50 total 10.01 last 123.87) - /unlink 2364 (time 1786645663.50 total 20.01 last 112.37) - /unlink 3697 (time 1786645673.50 total 30.02 last 133.22) total: 4500 /unlink in 36.66 seconds: 122.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 721 (time 1786645695.10 total 10.00 last 72.08) - open/close 1334 (time 1786645705.10 total 20.00 last 61.29) - open/close 2030 (time 1786645715.11 total 30.01 last 69.55) - open/close 2767 (time 1786645725.11 total 40.02 last 73.63) - open/close 3448 (time 1786645735.12 total 50.02 last 68.07) - open/close 4188 (time 1786645745.12 total 60.03 last 73.98) total: 4500 open/close in 64.76 seconds: 69.49 ops/second - /unlink 916 (time 1786645766.69 total 10.00 last 91.57) - /unlink 2002 (time 1786645776.69 total 20.01 last 108.53) - /unlink 3157 (time 1786645786.70 total 30.02 last 115.39) - /unlink 4437 (time 1786645796.71 total 40.03 last 127.92) total: 4500 /unlink in 40.71 seconds: 110.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 584 (time 1786645813.07 total 10.01 last 58.31) - open/close 1240 (time 1786645823.08 total 20.02 last 65.55) - open/close 1702 (time 1786645833.10 total 30.05 last 46.10) - open/close 2439 (time 1786645843.14 total 40.08 last 73.43) - open/close 3067 (time 1786645853.14 total 50.09 last 62.78) - open/close 3660 (time 1786645863.15 total 60.09 last 59.28) - open/close 4279 (time 1786645873.15 total 70.09 last 61.90) total: 4500 open/close in 73.09 seconds: 61.57 ops/second - /unlink 1280 (time 1786645891.14 total 10.01 last 127.82) - /unlink 2700 (time 1786645901.15 total 20.02 last 141.94) - /unlink 3836 (time 1786645911.15 total 30.02 last 113.58) total: 4500 /unlink in 36.29 seconds: 123.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 556 (time 1786645934.14 total 10.00 last 55.59) - open/close 1244 (time 1786645944.16 total 20.02 last 68.70) - open/close 1995 (time 1786645954.16 total 30.02 last 75.08) - open/close 2703 (time 1786645964.17 total 40.03 last 70.74) - open/close 3369 (time 1786645974.17 total 50.03 last 66.59) - open/close 3892 (time 1786645984.18 total 60.05 last 52.22) - open/close 4297 (time 1786645994.19 total 70.05 last 40.49) total: 4500 open/close in 74.48 seconds: 60.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1233 (time 1786646014.95 total 10.00 last 123.25) - /unlink 2518 (time 1786646024.95 total 20.01 last 128.48) - /unlink 3649 (time 1786646034.95 total 30.01 last 113.05) total: 4500 /unlink in 37.19 seconds: 121.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 561 (time 1786646057.82 total 10.02 last 55.98) - open/close 1184 (time 1786646067.83 total 20.03 last 62.21) - open/close 1930 (time 1786646077.85 total 30.05 last 74.48) - open/close 2599 (time 1786646087.85 total 40.05 last 66.88) - open/close 3323 (time 1786646097.87 total 50.07 last 72.28) - open/close 3826 (time 1786646107.87 total 60.08 last 50.27) - open/close 4339 (time 1786646117.88 total 70.08 last 51.26) total: 4500 open/close in 73.15 seconds: 61.52 ops/second - /unlink 1299 (time 1786646136.36 total 10.00 last 129.89) - /unlink 2610 (time 1786646146.36 total 20.01 last 131.04) - /unlink 3953 (time 1786646156.37 total 30.01 last 134.26) total: 4500 /unlink in 34.32 seconds: 131.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 609 (time 1786646176.01 total 10.01 last 60.82) - open/close 1034 (time 1786646186.04 total 20.04 last 42.39) - open/close 1461 (time 1786646196.05 total 30.05 last 42.67) - open/close 1880 (time 1786646206.30 total 40.30 last 40.85) - open/close 2365 (time 1786646216.31 total 50.31 last 48.45) - open/close 2943 (time 1786646226.32 total 60.32 last 57.76) - open/close 3552 (time 1786646236.32 total 70.32 last 60.90) - open/close 4100 (time 1786646246.34 total 80.34 last 54.71) total: 4500 open/close in 87.11 seconds: 51.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1187 (time 1786646268.78 total 10.00 last 118.65) - /unlink 2279 (time 1786646278.78 total 20.00 last 109.20) - /unlink 3428 (time 1786646288.78 total 30.01 last 114.89) total: 4500 /unlink in 38.97 seconds: 115.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 675 (time 1786646313.71 total 10.02 last 67.34) - open/close 1338 (time 1786646323.72 total 20.03 last 66.23) - open/close 2012 (time 1786646333.73 total 30.05 last 67.31) - open/close 2719 (time 1786646343.74 total 40.06 last 70.60) - open/close 3418 (time 1786646353.76 total 50.08 last 69.78) - open/close 4205 (time 1786646363.77 total 60.09 last 78.63) total: 4500 open/close in 64.25 seconds: 70.04 ops/second - /unlink 1093 (time 1786646383.17 total 10.00 last 109.26) - /unlink 2116 (time 1786646393.18 total 20.01 last 102.24) - /unlink 3287 (time 1786646403.18 total 30.01 last 117.10) - /unlink 4252 (time 1786646413.19 total 40.02 last 96.44) total: 4500 /unlink in 43.75 seconds: 102.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 508 (time 1786646434.57 total 10.04 last 50.62) - open/close 1191 (time 1786646444.58 total 20.04 last 68.24) - open/close 1858 (time 1786646454.59 total 30.05 last 66.68) - open/close 2547 (time 1786646464.60 total 40.06 last 68.80) - open/close 3313 (time 1786646474.60 total 50.06 last 76.58) - open/close 4038 (time 1786646484.60 total 60.07 last 72.49) total: 4500 open/close in 65.98 seconds: 68.20 ops/second - /unlink 1338 (time 1786646505.37 total 10.00 last 133.74) - /unlink 2522 (time 1786646515.37 total 20.01 last 118.31) - /unlink 3820 (time 1786646525.38 total 30.02 last 129.69) total: 4500 /unlink in 35.38 seconds: 127.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 804 (time 1786646546.01 total 10.00 last 80.38) - open/close 1528 (time 1786646556.01 total 20.00 last 72.40) - open/close 2233 (time 1786646566.01 total 30.01 last 70.47) - open/close 2883 (time 1786646576.01 total 40.01 last 64.99) - open/close 3548 (time 1786646586.02 total 50.01 last 66.47) - open/close 4120 (time 1786646596.04 total 60.03 last 57.08) total: 4500 open/close in 65.05 seconds: 69.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1330 (time 1786646617.15 total 10.00 last 132.97) - /unlink 2277 (time 1786646627.15 total 20.01 last 94.66) - /unlink 3535 (time 1786646637.15 total 30.01 last 125.77) total: 4500 /unlink in 37.69 seconds: 119.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 652 (time 1786646661.60 total 10.01 last 65.15) - open/close 1287 (time 1786646671.60 total 20.01 last 63.48) - open/close 1813 (time 1786646681.62 total 30.03 last 52.51) - open/close 2387 (time 1786646691.62 total 40.03 last 57.39) - open/close 3058 (time 1786646701.63 total 50.04 last 67.04) - open/close 3607 (time 1786646711.64 total 60.04 last 54.86) - open/close 4069 (time 1786646721.66 total 70.07 last 46.09) total: 4500 open/close in 76.47 seconds: 58.85 ops/second - /unlink 931 (time 1786646742.86 total 10.00 last 93.06) - /unlink 1935 (time 1786646752.86 total 20.01 last 100.39) - /unlink 3349 (time 1786646762.86 total 30.01 last 141.31) total: 4500 /unlink in 39.27 seconds: 114.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 566 (time 1786646788.25 total 10.04 last 56.38) - open/close 1200 (time 1786646798.25 total 20.04 last 63.39) - open/close 1810 (time 1786646808.29 total 30.08 last 60.74) - open/close 2453 (time 1786646818.30 total 40.09 last 64.25) - open/close 3279 (time 1786646828.31 total 50.10 last 82.51) - open/close 4025 (time 1786646838.33 total 60.12 last 74.50) total: 4500 open/close in 67.60 seconds: 66.57 ops/second - /unlink 1343 (time 1786646860.85 total 10.01 last 134.17) - /unlink 2490 (time 1786646870.86 total 20.01 last 114.65) - /unlink 3626 (time 1786646880.86 total 30.02 last 113.55) total: 4500 /unlink in 36.50 seconds: 123.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 576 (time 1786646901.69 total 10.02 last 57.51) - open/close 1246 (time 1786646911.70 total 20.03 last 66.91) - open/close 1865 (time 1786646921.70 total 30.03 last 61.89) - open/close 2563 (time 1786646931.72 total 40.05 last 69.69) - open/close 3311 (time 1786646941.72 total 50.05 last 74.79) - open/close 3965 (time 1786646951.72 total 60.05 last 65.38) total: 4500 open/close in 68.31 seconds: 65.88 ops/second - /unlink 1277 (time 1786646974.57 total 10.01 last 127.54) - /unlink 2273 (time 1786646984.59 total 20.03 last 99.45) - /unlink 3598 (time 1786646994.59 total 30.03 last 132.50) total: 4500 /unlink in 36.83 seconds: 122.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 675 (time 1786647016.42 total 10.01 last 67.45) - open/close 1381 (time 1786647026.44 total 20.03 last 70.46) - open/close 1986 (time 1786647036.45 total 30.03 last 60.45) - open/close 2676 (time 1786647046.46 total 40.04 last 68.97) - open/close 3382 (time 1786647056.46 total 50.04 last 70.56) - open/close 4014 (time 1786647066.47 total 60.05 last 63.14) total: 4500 open/close in 67.48 seconds: 66.68 ops/second - /unlink 1269 (time 1786647087.70 total 10.00 last 126.85) - /unlink 2471 (time 1786647097.71 total 20.01 last 120.15) - /unlink 3823 (time 1786647107.71 total 30.02 last 135.09) total: 4500 /unlink in 34.04 seconds: 132.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 604 (time 1786647127.34 total 10.00 last 60.38) - open/close 1308 (time 1786647137.35 total 20.01 last 70.34) - open/close 1942 (time 1786647147.37 total 30.03 last 63.26) - open/close 2586 (time 1786647157.37 total 40.04 last 64.38) - open/close 3233 (time 1786647167.38 total 50.04 last 64.68) - open/close 3938 (time 1786647177.39 total 60.06 last 70.38) total: 4500 open/close in 69.76 seconds: 64.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 884 (time 1786647202.50 total 10.02 last 88.26) - /unlink 1599 (time 1786647212.50 total 20.02 last 71.47) - /unlink 2777 (time 1786647222.53 total 30.05 last 117.43) - /unlink 4009 (time 1786647232.54 total 40.05 last 123.17) total: 4500 /unlink in 44.81 seconds: 100.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 580 (time 1786647252.76 total 10.00 last 57.97) - open/close 1262 (time 1786647262.76 total 20.01 last 68.18) - open/close 1920 (time 1786647272.77 total 30.01 last 65.76) - open/close 2585 (time 1786647282.79 total 40.03 last 66.39) - open/close 3174 (time 1786647292.79 total 50.04 last 58.86) - open/close 3804 (time 1786647302.79 total 60.04 last 62.99) - open/close 4367 (time 1786647312.80 total 70.04 last 56.28) total: 4500 open/close in 73.49 seconds: 61.23 ops/second - /unlink 717 (time 1786647335.67 total 10.01 last 71.64) - /unlink 1782 (time 1786647345.67 total 20.01 last 106.46) - /unlink 2653 (time 1786647355.67 total 30.01 last 87.10) - /unlink 3963 (time 1786647365.67 total 40.01 last 130.97) total: 4500 /unlink in 44.11 seconds: 102.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 808 (time 1786647384.49 total 10.01 last 80.76) - open/close 1400 (time 1786647394.50 total 20.02 last 59.13) - open/close 1973 (time 1786647404.52 total 30.03 last 57.24) - open/close 2555 (time 1786647414.53 total 40.05 last 58.10) - open/close 3201 (time 1786647424.54 total 50.05 last 64.54) - open/close 3868 (time 1786647434.55 total 60.06 last 66.64) total: 4500 open/close in 68.89 seconds: 65.33 ops/second - /unlink 799 (time 1786647459.84 total 10.00 last 79.87) - /unlink 1569 (time 1786647469.84 total 20.00 last 76.99) - /unlink 2268 (time 1786647479.84 total 30.01 last 69.88) - /unlink 3631 (time 1786647489.85 total 40.01 last 136.24) total: 4500 /unlink in 46.95 seconds: 95.85 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95325 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg118-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg118-server unloading modules via unload_modules_local on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing unload_modules_local oleg118-server: modules unloaded.