== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 12:39:01 (1786034341) Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg238-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server oleg238-server: oleg238-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing load_modules_local oleg238-server: Loading modules from /home/green/git/lustre-release/lustre oleg238-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg238-server: detected 4 online CPUs by sysfs oleg238-server: MODOPTS_LIBCFS= oleg238-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg238-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg238-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg238-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 871 (time 1786034558.85 total 10.00 last 87.08) - open/close 1570 (time 1786034568.85 total 20.00 last 69.89) - open/close 2177 (time 1786034578.86 total 30.01 last 60.66) - open/close 2763 (time 1786034588.86 total 40.01 last 58.60) - open/close 3433 (time 1786034598.87 total 50.02 last 66.95) - open/close 3992 (time 1786034608.89 total 60.04 last 55.77) total: 4500 open/close in 68.25 seconds: 65.94 ops/second - /unlink 1086 (time 1786034633.74 total 10.00 last 108.57) - /unlink 2422 (time 1786034643.74 total 20.01 last 133.52) - /unlink 3669 (time 1786034653.75 total 30.01 last 124.63) total: 4500 /unlink in 37.38 seconds: 120.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 708 (time 1786034676.17 total 10.01 last 70.76) - open/close 1520 (time 1786034686.18 total 20.01 last 81.14) - open/close 2299 (time 1786034696.19 total 30.02 last 77.84) - open/close 3046 (time 1786034706.20 total 40.03 last 74.64) - open/close 3701 (time 1786034716.20 total 50.04 last 65.45) - open/close 4354 (time 1786034726.21 total 60.04 last 65.27) total: 4500 open/close in 62.18 seconds: 72.37 ops/second - /unlink 1289 (time 1786034742.96 total 10.00 last 128.86) - /unlink 2531 (time 1786034752.97 total 20.01 last 124.10) - /unlink 4057 (time 1786034762.97 total 30.01 last 152.58) total: 4500 /unlink in 33.24 seconds: 135.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 714 (time 1786034782.27 total 10.00 last 71.38) - open/close 1353 (time 1786034792.28 total 20.01 last 63.82) - open/close 2113 (time 1786034802.29 total 30.02 last 75.96) - open/close 2823 (time 1786034812.30 total 40.04 last 70.89) - open/close 3488 (time 1786034822.30 total 50.04 last 66.49) - open/close 4172 (time 1786034832.31 total 60.04 last 68.35) total: 4500 open/close in 64.69 seconds: 69.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1400 (time 1786034851.66 total 10.00 last 139.99) - /unlink 2718 (time 1786034861.67 total 20.00 last 131.78) - /unlink 3877 (time 1786034871.67 total 30.00 last 115.89) total: 4500 /unlink in 34.46 seconds: 130.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 596 (time 1786034891.48 total 10.00 last 59.59) - open/close 1183 (time 1786034901.49 total 20.01 last 58.67) - open/close 1923 (time 1786034911.50 total 30.01 last 73.94) - open/close 2450 (time 1786034921.50 total 40.02 last 52.65) - open/close 2963 (time 1786034931.52 total 50.04 last 51.20) - open/close 3655 (time 1786034941.54 total 60.06 last 69.09) - open/close 4353 (time 1786034951.54 total 70.06 last 69.76) total: 4500 open/close in 72.77 seconds: 61.84 ops/second - /unlink 1466 (time 1786034969.37 total 10.01 last 146.49) - /unlink 2545 (time 1786034979.38 total 20.01 last 107.89) - /unlink 3643 (time 1786034989.38 total 30.01 last 109.75) total: 4500 /unlink in 37.53 seconds: 119.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 845 (time 1786035012.00 total 10.01 last 84.40) - open/close 1567 (time 1786035022.01 total 20.02 last 72.13) - open/close 2211 (time 1786035032.01 total 30.02 last 64.38) - open/close 3049 (time 1786035042.01 total 40.03 last 83.76) - open/close 3864 (time 1786035052.02 total 50.04 last 81.44) - open/close 4434 (time 1786035062.03 total 60.04 last 56.97) total: 4500 open/close in 61.19 seconds: 73.54 ops/second - /unlink 1383 (time 1786035078.59 total 10.01 last 138.20) - /unlink 2482 (time 1786035088.61 total 20.03 last 109.60) - /unlink 3628 (time 1786035098.62 total 30.04 last 114.52) total: 4500 /unlink in 37.69 seconds: 119.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 615 (time 1786035123.01 total 10.01 last 61.46) - open/close 1265 (time 1786035133.02 total 20.02 last 64.94) - open/close 2028 (time 1786035143.02 total 30.02 last 76.28) - open/close 2844 (time 1786035153.03 total 40.02 last 81.55) - open/close 3556 (time 1786035163.05 total 50.05 last 71.03) - open/close 4383 (time 1786035173.06 total 60.06 last 82.63) total: 4500 open/close in 61.27 seconds: 73.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1021 (time 1786035189.56 total 10.01 last 102.04) - /unlink 2285 (time 1786035199.56 total 20.01 last 126.33) - /unlink 3482 (time 1786035209.57 total 30.02 last 119.62) total: 4500 /unlink in 39.66 seconds: 113.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 638 (time 1786035235.52 total 10.00 last 63.79) - open/close 1277 (time 1786035245.53 total 20.02 last 63.81) - open/close 1836 (time 1786035255.53 total 30.02 last 55.89) - open/close 2500 (time 1786035265.54 total 40.02 last 66.39) - open/close 3038 (time 1786035275.55 total 50.04 last 53.71) - open/close 3582 (time 1786035285.58 total 60.07 last 54.22) - open/close 4249 (time 1786035295.60 total 70.08 last 66.62) total: 4500 open/close in 75.20 seconds: 59.84 ops/second - /unlink 1241 (time 1786035316.41 total 10.03 last 123.77) - /unlink 2195 (time 1786035326.41 total 20.03 last 95.34) - /unlink 3262 (time 1786035336.42 total 30.04 last 106.63) total: 4500 /unlink in 39.55 seconds: 113.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 702 (time 1786035362.19 total 10.04 last 69.95) - open/close 1461 (time 1786035372.20 total 20.04 last 75.87) - open/close 2186 (time 1786035382.20 total 30.04 last 72.49) - open/close 2891 (time 1786035392.22 total 40.06 last 70.38) - open/close 3561 (time 1786035402.22 total 50.06 last 66.98) - open/close 4344 (time 1786035412.23 total 60.07 last 78.24) total: 4500 open/close in 62.16 seconds: 72.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1159 (time 1786035429.51 total 10.00 last 115.86) - /unlink 2379 (time 1786035439.51 total 20.01 last 121.95) - /unlink 3414 (time 1786035449.51 total 30.01 last 103.46) total: 4500 /unlink in 39.06 seconds: 115.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 595 (time 1786035474.41 total 10.01 last 59.46) - open/close 1181 (time 1786035484.42 total 20.02 last 58.55) - open/close 1833 (time 1786035494.43 total 30.03 last 65.14) - open/close 2612 (time 1786035504.44 total 40.04 last 77.81) - open/close 3332 (time 1786035514.46 total 50.06 last 71.84) - open/close 4052 (time 1786035524.47 total 60.07 last 71.94) total: 4500 open/close in 65.60 seconds: 68.59 ops/second - /unlink 1191 (time 1786035544.94 total 10.01 last 119.02) - /unlink 2168 (time 1786035554.94 total 20.01 last 97.65) - /unlink 3354 (time 1786035564.95 total 30.02 last 118.47) total: 4500 /unlink in 39.47 seconds: 114.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 664 (time 1786035589.82 total 10.03 last 66.21) - open/close 1233 (time 1786035599.83 total 20.04 last 56.83) - open/close 1838 (time 1786035609.83 total 30.04 last 60.48) - open/close 2551 (time 1786035619.83 total 40.04 last 71.30) - open/close 3367 (time 1786035629.83 total 50.04 last 81.60) - open/close 4102 (time 1786035639.84 total 60.05 last 73.48) total: 4500 open/close in 65.89 seconds: 68.30 ops/second - /unlink 953 (time 1786035661.07 total 10.00 last 95.26) - /unlink 2075 (time 1786035671.08 total 20.02 last 112.05) - /unlink 3217 (time 1786035681.08 total 30.02 last 114.20) - /unlink 4422 (time 1786035691.09 total 40.03 last 120.40) total: 4500 /unlink in 40.81 seconds: 110.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 660 (time 1786035708.10 total 10.00 last 65.99) - open/close 1323 (time 1786035718.10 total 20.01 last 66.26) - open/close 2049 (time 1786035728.12 total 30.02 last 72.48) - open/close 2853 (time 1786035738.13 total 40.04 last 80.29) - open/close 3535 (time 1786035748.14 total 50.04 last 68.15) - open/close 4061 (time 1786035758.15 total 60.06 last 52.54) total: 4500 open/close in 67.99 seconds: 66.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1210 (time 1786035782.33 total 10.00 last 121.00) - /unlink 2488 (time 1786035792.34 total 20.00 last 127.76) - /unlink 3755 (time 1786035802.34 total 30.00 last 126.68) total: 4500 /unlink in 35.33 seconds: 127.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 594 (time 1786035823.20 total 10.01 last 59.35) - open/close 1219 (time 1786035833.21 total 20.01 last 62.47) - open/close 1918 (time 1786035843.22 total 30.02 last 69.83) - open/close 2677 (time 1786035853.22 total 40.02 last 75.89) - open/close 3228 (time 1786035863.23 total 50.03 last 55.05) - open/close 3903 (time 1786035873.23 total 60.04 last 67.49) total: 4500 open/close in 68.81 seconds: 65.40 ops/second - /unlink 1062 (time 1786035897.31 total 10.00 last 106.15) - /unlink 2193 (time 1786035907.33 total 20.02 last 112.89) - /unlink 3479 (time 1786035917.33 total 30.03 last 128.56) - /unlink 4465 (time 1786035927.34 total 40.03 last 98.55) total: 4500 /unlink in 40.43 seconds: 111.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 587 (time 1786035944.05 total 10.01 last 58.66) - open/close 1126 (time 1786035954.05 total 20.01 last 53.89) - open/close 1591 (time 1786035964.06 total 30.02 last 46.43) - open/close 2107 (time 1786035974.08 total 40.04 last 51.53) - open/close 2602 (time 1786035984.08 total 50.04 last 49.49) - open/close 3161 (time 1786035994.09 total 60.05 last 55.83) - open/close 3692 (time 1786036004.10 total 70.06 last 53.06) - open/close 4272 (time 1786036014.11 total 80.07 last 57.94) total: 4500 open/close in 83.25 seconds: 54.05 ops/second - /unlink 1057 (time 1786036033.37 total 10.00 last 105.69) - /unlink 2186 (time 1786036043.37 total 20.00 last 112.87) - /unlink 3357 (time 1786036053.38 total 30.01 last 117.03) total: 4500 /unlink in 39.29 seconds: 114.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 579 (time 1786036079.22 total 10.01 last 57.84) - open/close 1118 (time 1786036089.25 total 20.04 last 53.76) - open/close 1717 (time 1786036099.26 total 30.05 last 59.85) - open/close 2270 (time 1786036109.27 total 40.05 last 55.26) - open/close 2801 (time 1786036119.30 total 50.08 last 52.94) - open/close 3586 (time 1786036129.30 total 60.09 last 78.46) - open/close 4358 (time 1786036139.31 total 70.10 last 77.13) total: 4500 open/close in 71.68 seconds: 62.78 ops/second - /unlink 1566 (time 1786036155.22 total 10.00 last 156.54) - /unlink 2979 (time 1786036165.23 total 20.01 last 141.16) - /unlink 4183 (time 1786036175.23 total 30.02 last 120.36) total: 4500 /unlink in 32.32 seconds: 139.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 724 (time 1786036192.76 total 10.00 last 72.39) - open/close 1452 (time 1786036202.77 total 20.01 last 72.74) - open/close 2136 (time 1786036212.78 total 30.03 last 68.29) - open/close 2882 (time 1786036222.80 total 40.04 last 74.50) - open/close 3596 (time 1786036232.80 total 50.05 last 71.35) - open/close 4186 (time 1786036242.81 total 60.05 last 58.97) total: 4500 open/close in 64.79 seconds: 69.46 ops/second - /unlink 1087 (time 1786036264.56 total 10.00 last 108.68) - /unlink 2104 (time 1786036274.56 total 20.01 last 101.65) - /unlink 2960 (time 1786036284.57 total 30.01 last 85.54) - /unlink 3928 (time 1786036294.60 total 40.04 last 96.55) total: 4500 /unlink in 45.92 seconds: 97.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 582 (time 1786036317.12 total 10.01 last 58.14) - open/close 1078 (time 1786036327.14 total 20.03 last 49.52) - open/close 1641 (time 1786036337.14 total 30.03 last 56.26) - open/close 2282 (time 1786036347.15 total 40.04 last 64.04) - open/close 2884 (time 1786036357.16 total 50.05 last 60.15) - open/close 3560 (time 1786036367.16 total 60.05 last 67.60) - open/close 4110 (time 1786036377.18 total 70.06 last 54.93) total: 4500 open/close in 77.46 seconds: 58.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1160 (time 1786036400.09 total 10.01 last 115.92) - /unlink 2350 (time 1786036410.10 total 20.02 last 118.89) - /unlink 3455 (time 1786036420.10 total 30.02 last 110.50) total: 4500 /unlink in 38.69 seconds: 116.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 690 (time 1786036444.66 total 10.01 last 68.93) - open/close 1360 (time 1786036454.67 total 20.03 last 66.90) - open/close 2091 (time 1786036464.67 total 30.03 last 73.07) - open/close 2729 (time 1786036474.68 total 40.03 last 63.79) - open/close 3419 (time 1786036484.69 total 50.04 last 68.93) - open/close 4027 (time 1786036494.70 total 60.05 last 60.74) total: 4500 open/close in 68.45 seconds: 65.74 ops/second - /unlink 1075 (time 1786036519.19 total 10.01 last 107.40) - /unlink 2147 (time 1786036529.19 total 20.01 last 107.20) - /unlink 3445 (time 1786036539.20 total 30.01 last 129.76) total: 4500 /unlink in 36.81 seconds: 122.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 790 (time 1786036561.58 total 10.03 last 78.80) - open/close 1563 (time 1786036571.59 total 20.03 last 77.26) - open/close 2382 (time 1786036581.59 total 30.03 last 81.89) - open/close 3112 (time 1786036591.60 total 40.04 last 72.96) - open/close 3796 (time 1786036601.60 total 50.04 last 68.36) total: 4500 open/close in 59.81 seconds: 75.23 ops/second - /unlink 1158 (time 1786036626.53 total 10.01 last 115.67) - /unlink 2353 (time 1786036636.53 total 20.01 last 119.47) - /unlink 3598 (time 1786036646.54 total 30.02 last 124.48) total: 4500 /unlink in 37.51 seconds: 119.98 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95246 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server stop mds service on oleg238-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server stop mds service on oleg238-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server unloading modules via unload_modules_local on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing unload_modules_local oleg238-server: modules unloaded.