== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:04:21 (1787342661)
Stopping clients: oleg224-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg224-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg224-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg224-server
oleg224-server: oleg224-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing load_modules_local
oleg224-server: Loading modules from /home/green/git/lustre-release/lustre
oleg224-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg224-server: detected 4 online CPUs by sysfs
oleg224-server: MODOPTS_LIBCFS=
oleg224-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg224-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg224-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg224-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg224-client.virtnet:  -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 587 (time 1787342884.80 total 10.00 last 58.68)
 - open/close 1286 (time 1787342894.81 total 20.01 last 69.87)
 - open/close 2022 (time 1787342904.83 total 30.03 last 73.44)
 - open/close 2752 (time 1787342914.84 total 40.04 last 72.94)
 - open/close 3518 (time 1787342924.84 total 50.04 last 76.60)
 - open/close 4248 (time 1787342934.85 total 60.04 last 72.95)
total: 4500 open/close in 63.53 seconds: 70.83 ops/second
 - /unlink 1321 (time 1787342953.48 total 10.01 last 131.98)
 - /unlink 2849 (time 1787342963.48 total 20.01 last 152.79)
 - /unlink 4339 (time 1787342973.48 total 30.01 last 148.99)
total: 4500 /unlink in 30.95 seconds: 145.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 745 (time 1787342988.99 total 10.01 last 74.45)
 - open/close 1526 (time 1787342998.99 total 20.01 last 78.09)
 - open/close 2304 (time 1787343009.00 total 30.02 last 77.72)
 - open/close 3050 (time 1787343019.01 total 40.03 last 74.54)
 - open/close 3755 (time 1787343029.02 total 50.03 last 70.45)
 - open/close 4489 (time 1787343039.02 total 60.04 last 73.39)
total: 4500 open/close in 60.25 seconds: 74.69 ops/second
 - /unlink 1425 (time 1787343053.75 total 10.00 last 142.48)
 - /unlink 2881 (time 1787343063.76 total 20.01 last 145.45)
 - /unlink 4398 (time 1787343073.77 total 30.02 last 151.62)
total: 4500 /unlink in 30.68 seconds: 146.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 747 (time 1787343088.85 total 10.00 last 74.69)
 - open/close 1436 (time 1787343098.85 total 20.00 last 68.88)
 - open/close 2169 (time 1787343108.86 total 30.01 last 73.23)
 - open/close 2900 (time 1787343118.87 total 40.03 last 73.02)
 - open/close 3690 (time 1787343128.88 total 50.03 last 78.95)
total: 4500 open/close in 59.91 seconds: 75.11 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1521 (time 1787343153.26 total 10.00 last 152.06)
 - /unlink 3019 (time 1787343163.26 total 20.01 last 149.71)
total: 4500 /unlink in 29.81 seconds: 150.96 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 752 (time 1787343187.90 total 10.01 last 75.12)
 - open/close 1362 (time 1787343197.92 total 20.03 last 60.85)
 - open/close 2090 (time 1787343207.95 total 30.06 last 72.63)
 - open/close 2874 (time 1787343218.00 total 40.11 last 77.99)
 - open/close 3561 (time 1787343228.01 total 50.13 last 68.59)
 - open/close 4354 (time 1787343238.02 total 60.13 last 79.24)
total: 4500 open/close in 61.91 seconds: 72.69 ops/second
 - /unlink 1435 (time 1787343254.44 total 10.01 last 143.42)
 - /unlink 2684 (time 1787343264.46 total 20.02 last 124.67)
 - /unlink 3953 (time 1787343274.47 total 30.03 last 126.81)
total: 4500 /unlink in 34.67 seconds: 129.80 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 727 (time 1787343294.01 total 10.00 last 72.70)
 - open/close 1445 (time 1787343304.02 total 20.01 last 71.71)
 - open/close 2218 (time 1787343314.03 total 30.02 last 77.25)
 - open/close 2959 (time 1787343324.04 total 40.03 last 74.01)
 - open/close 3624 (time 1787343334.04 total 50.03 last 66.50)
 - open/close 4320 (time 1787343344.05 total 60.04 last 69.57)
total: 4500 open/close in 62.94 seconds: 71.50 ops/second
 - /unlink 1388 (time 1787343364.30 total 10.00 last 138.76)
 - /unlink 2507 (time 1787343374.31 total 20.01 last 111.82)
 - /unlink 3808 (time 1787343384.31 total 30.01 last 130.09)
total: 4500 /unlink in 34.98 seconds: 128.66 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 702 (time 1787343403.90 total 10.01 last 70.16)
 - open/close 1361 (time 1787343413.91 total 20.01 last 65.84)
 - open/close 2079 (time 1787343423.92 total 30.03 last 71.71)
 - open/close 2874 (time 1787343433.93 total 40.03 last 79.45)
 - open/close 3653 (time 1787343443.94 total 50.04 last 77.84)
 - open/close 4473 (time 1787343453.94 total 60.05 last 81.94)
total: 4500 open/close in 60.33 seconds: 74.59 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1352 (time 1787343468.31 total 10.00 last 135.19)
 - /unlink 2703 (time 1787343478.31 total 20.01 last 135.04)
 - /unlink 3753 (time 1787343488.32 total 30.01 last 104.96)
total: 4500 /unlink in 37.31 seconds: 120.62 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 842 (time 1787343510.57 total 10.00 last 84.16)
 - open/close 1472 (time 1787343520.59 total 20.02 last 62.90)
 - open/close 2180 (time 1787343530.60 total 30.03 last 70.75)
 - open/close 3017 (time 1787343540.62 total 40.05 last 83.50)
 - open/close 3848 (time 1787343550.63 total 50.06 last 83.04)
total: 4500 open/close in 58.66 seconds: 76.71 ops/second
 - /unlink 1462 (time 1787343573.82 total 10.02 last 145.95)
 - /unlink 2909 (time 1787343583.83 total 20.02 last 144.61)
 - /unlink 4288 (time 1787343593.84 total 30.03 last 137.81)
total: 4500 /unlink in 31.44 seconds: 143.13 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 824 (time 1787343610.48 total 10.01 last 82.32)
 - open/close 1609 (time 1787343620.50 total 20.03 last 78.37)
 - open/close 2298 (time 1787343630.51 total 30.03 last 68.85)
 - open/close 2931 (time 1787343640.54 total 40.06 last 63.11)
 - open/close 3615 (time 1787343650.54 total 50.07 last 68.36)
 - open/close 4305 (time 1787343660.55 total 60.07 last 68.96)
total: 4500 open/close in 63.17 seconds: 71.23 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1271 (time 1787343679.14 total 10.00 last 127.08)
 - /unlink 2684 (time 1787343689.14 total 20.01 last 141.24)
 - /unlink 4223 (time 1787343699.14 total 30.01 last 153.90)
total: 4500 /unlink in 32.48 seconds: 138.55 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 709 (time 1787343717.08 total 10.00 last 70.89)
 - open/close 1371 (time 1787343727.08 total 20.01 last 66.17)
 - open/close 2189 (time 1787343737.09 total 30.01 last 81.74)
 - open/close 2771 (time 1787343747.09 total 40.01 last 58.18)
 - open/close 3520 (time 1787343757.10 total 50.02 last 74.86)
 - open/close 3974 (time 1787343767.10 total 60.02 last 45.38)
total: 4500 open/close in 68.16 seconds: 66.02 ops/second
 - /unlink 1506 (time 1787343790.21 total 10.00 last 150.56)
 - /unlink 3178 (time 1787343800.21 total 20.00 last 167.18)
total: 4500 /unlink in 28.53 seconds: 157.74 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 592 (time 1787343823.91 total 10.01 last 59.14)
 - open/close 1221 (time 1787343833.91 total 20.02 last 62.86)
 - open/close 2028 (time 1787343843.92 total 30.02 last 80.65)
 - open/close 2735 (time 1787343853.93 total 40.03 last 70.65)
 - open/close 3391 (time 1787343863.93 total 50.03 last 65.57)
 - open/close 4141 (time 1787343873.95 total 60.05 last 74.89)
total: 4500 open/close in 64.99 seconds: 69.24 ops/second
 - /unlink 1494 (time 1787343893.87 total 10.00 last 149.37)
 - /unlink 2972 (time 1787343903.87 total 20.01 last 147.70)
 - /unlink 4480 (time 1787343913.88 total 30.01 last 150.77)
total: 4500 /unlink in 30.30 seconds: 148.51 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 745 (time 1787343929.73 total 10.01 last 74.44)
 - open/close 1425 (time 1787343939.74 total 20.01 last 67.96)
 - open/close 2047 (time 1787343949.75 total 30.03 last 62.12)
 - open/close 2725 (time 1787343959.76 total 40.03 last 67.74)
 - open/close 3289 (time 1787343969.77 total 50.04 last 56.37)
 - open/close 4053 (time 1787343979.77 total 60.04 last 76.38)
total: 4500 open/close in 65.53 seconds: 68.67 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1396 (time 1787343999.32 total 10.00 last 139.56)
 - /unlink 2348 (time 1787344009.32 total 20.01 last 95.18)
 - /unlink 3473 (time 1787344019.32 total 30.01 last 112.46)
total: 4500 /unlink in 39.00 seconds: 115.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 734 (time 1787344042.79 total 10.00 last 73.40)
 - open/close 1582 (time 1787344052.79 total 20.00 last 84.77)
 - open/close 2200 (time 1787344062.81 total 30.03 last 61.66)
 - open/close 3037 (time 1787344072.81 total 40.03 last 83.67)
 - open/close 3901 (time 1787344082.82 total 50.03 last 86.36)
total: 4500 open/close in 57.08 seconds: 78.83 ops/second
 - /unlink 1582 (time 1787344103.74 total 10.01 last 158.09)
 - /unlink 2941 (time 1787344113.75 total 20.01 last 135.82)
 - /unlink 4476 (time 1787344123.75 total 30.02 last 153.43)
total: 4500 /unlink in 30.19 seconds: 149.05 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 769 (time 1787344138.06 total 10.00 last 76.87)
 - open/close 1555 (time 1787344148.06 total 20.00 last 78.60)
 - open/close 2453 (time 1787344158.07 total 30.02 last 89.70)
 - open/close 3322 (time 1787344168.07 total 40.02 last 86.89)
 - open/close 4058 (time 1787344178.08 total 50.03 last 73.52)
total: 4500 open/close in 55.40 seconds: 81.23 ops/second
 - /unlink 1496 (time 1787344197.99 total 10.01 last 149.48)
 - /unlink 2642 (time 1787344207.99 total 20.01 last 114.57)
 - /unlink 3893 (time 1787344218.00 total 30.02 last 125.03)
total: 4500 /unlink in 33.92 seconds: 132.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 678 (time 1787344236.70 total 10.00 last 67.78)
 - open/close 1462 (time 1787344246.73 total 20.03 last 78.16)
 - open/close 2119 (time 1787344256.73 total 30.04 last 65.68)
 - open/close 2913 (time 1787344266.74 total 40.04 last 79.35)
 - open/close 3625 (time 1787344276.75 total 50.05 last 71.15)
 - open/close 4286 (time 1787344286.75 total 60.05 last 66.07)
total: 4500 open/close in 62.98 seconds: 71.45 ops/second
 - /unlink 1407 (time 1787344304.06 total 10.01 last 140.63)
 - /unlink 2843 (time 1787344314.06 total 20.01 last 143.53)
 - /unlink 4102 (time 1787344324.07 total 30.01 last 125.84)
total: 4500 /unlink in 33.73 seconds: 133.40 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 694 (time 1787344344.03 total 10.01 last 69.32)
 - open/close 1391 (time 1787344354.04 total 20.02 last 69.63)
 - open/close 2131 (time 1787344364.05 total 30.03 last 73.92)
 - open/close 2851 (time 1787344374.05 total 40.04 last 71.98)
 - open/close 3588 (time 1787344384.07 total 50.05 last 73.61)
 - open/close 4366 (time 1787344394.08 total 60.06 last 77.72)
total: 4500 open/close in 61.86 seconds: 72.74 ops/second
 - /unlink 1393 (time 1787344410.76 total 10.01 last 139.21)
 - /unlink 2555 (time 1787344420.76 total 20.01 last 116.16)
 - /unlink 3856 (time 1787344430.76 total 30.01 last 130.07)
total: 4500 /unlink in 36.84 seconds: 122.14 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 767 (time 1787344453.49 total 10.01 last 76.61)
 - open/close 1525 (time 1787344463.51 total 20.03 last 75.67)
 - open/close 2338 (time 1787344473.51 total 30.03 last 81.29)
 - open/close 3216 (time 1787344483.51 total 40.03 last 87.78)
 - open/close 3990 (time 1787344493.51 total 50.03 last 77.39)
total: 4500 open/close in 58.17 seconds: 77.36 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1055 (time 1787344516.77 total 10.00 last 105.49)
 - /unlink 2328 (time 1787344526.77 total 20.01 last 127.24)
 - /unlink 3460 (time 1787344536.77 total 30.01 last 113.17)
total: 4500 /unlink in 37.64 seconds: 119.56 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 714 (time 1787344561.53 total 10.02 last 71.28)
 - open/close 1370 (time 1787344571.53 total 20.02 last 65.55)
 - open/close 2156 (time 1787344581.54 total 30.03 last 78.57)
 - open/close 2887 (time 1787344591.54 total 40.03 last 73.07)
 - open/close 3629 (time 1787344601.54 total 50.04 last 74.17)
 - open/close 4263 (time 1787344611.55 total 60.04 last 63.39)
total: 4500 open/close in 63.05 seconds: 71.37 ops/second
 - /unlink 1372 (time 1787344629.29 total 10.00 last 137.17)
 - /unlink 2673 (time 1787344639.30 total 20.01 last 130.05)
 - /unlink 3894 (time 1787344649.30 total 30.01 last 122.05)
total: 4500 /unlink in 34.39 seconds: 130.86 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 868 (time 1787344667.94 total 10.01 last 86.71)
 - open/close 1535 (time 1787344677.97 total 20.05 last 66.46)
 - open/close 2321 (time 1787344687.98 total 30.06 last 78.53)
 - open/close 3109 (time 1787344697.98 total 40.06 last 78.79)
 - open/close 3799 (time 1787344707.99 total 50.06 last 68.99)
 - open/close 4468 (time 1787344717.99 total 60.06 last 66.87)
total: 4500 open/close in 60.45 seconds: 74.44 ops/second
 - /unlink 1360 (time 1787344733.46 total 10.01 last 135.90)
 - /unlink 2600 (time 1787344743.46 total 20.01 last 123.96)
 - /unlink 3907 (time 1787344753.46 total 30.01 last 130.68)
total: 4500 /unlink in 34.68 seconds: 129.75 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95311 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg224-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
stop mds service on oleg224-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server
stop mds service on oleg224-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg224-server

unloading modules via unload_modules_local on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing unload_modules_local
oleg224-server: 
modules unloaded.