Quantcast

How to clean up huge .index?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

How to clean up huge .index?

Peter Palmreuther
Hello,

I have a Nexus-OSE installation, version 1.9.2.2, running on my FreeBSD 8.2 using OpenJDK 6b23.
It's providing 13 proxied repositories, 3 hosted (3rd party, Snapshots, Releases), 2 virtual (M1 -> M2 shadow) and two groups (public and public-snapshots).

Within cached and deployed artifacts and data I have a total of 211MiB (du -sch ${sonatype_work}/storage/*/*).
But within indexes (du -sch ${sonatype_work}/storage/*/.index) there's an overall of 16 GiB consuming my disk space.

The biggest amount is occupied by a java.net proxy repository index.
Having a look at http://download.java.net/maven/2/.index/ there's one 21MiB index file.
Looking at my own .index directory for this repository's cache there're 823 index files (.gz), each being 14-16MiB.
In total it's 12GiB being occupied.

I see it's due to incremental indexing, but wonder if there's a way to reduce the number of kept incremental index files, as 12GiB index for 3.5MiB cached artifact is a pretty devastating ratio ...

There's an "Optimize Repository Index" task running every week on "All Repositories" but even executing manually doesn't seem to change anything regarding the space consumed.

Additionally I do have a pretty huge ${sonatype_work}/indexer directory (3.5GiB) and would like to know if there's anything I can do about it's size?

Any hint or help appreciated,
and thanks in advance,

Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to clean up huge .index?

Peter Palmreuther
Hi,

On 05.09.2011 at 09:05 Peter Palmreuther wrote:
>
> [...]
> I [...] wonder if there's a way to reduce the number of kept incremental index files, as 12GiB index for 3.5MiB cached artifact is a pretty devastating ratio ...
> [...]

Anyone?
--
Best regards,

Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to clean up huge .index?

Tamás Cservenák
Well, not quite an answer, but you should drop your java.net proxies,
all that is in central now:
http://www.sonatype.com/people/2011/08/java-net-moves-to-central/

And about "resetting" the incremental index: Repair Index task should
"reset" the incremental baseline too, and will it delete all the GZ
files.

But, in general, simplest is to shut nexus down, remove the folders in
question (paired, so storage/repoID/.index AND indexer/repoID), and
restart nexus. Then run (probably manually, this should be a "once to
be done" task) Repair Index on repositories that have index folders
removed.


Thanks,
~t~

On Fri, Sep 9, 2011 at 7:36 AM, Peter Palmreuther
<[hidden email]> wrote:

> Hi,
>
> On 05.09.2011 at 09:05 Peter Palmreuther wrote:
>>
>> [...]
>> I [...] wonder if there's a way to reduce the number of kept incremental index files, as 12GiB index for 3.5MiB cached artifact is a pretty devastating ratio ...
>> [...]
>
> Anyone?
> --
> Best regards,
>
> Peter
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to clean up huge .index?

Peter Palmreuther
Hi,

On 09.09.2011 at 09:59 Tamás Cservenák wrote:
>
> Well, not quite an answer, but you should drop your java.net proxies,
> all that is in central now:
> http://www.sonatype.com/people/2011/08/java-net-moves-to-central/

Thanks for pointing towards this.

> And about "resetting" the incremental index: Repair Index task should
> "reset" the incremental baseline too, and will it delete all the GZ
> files.

And thanks for slapping me with the obvious. I completely forgot about the repair index option.
It seems to have solved my primary issue. Supplemented by subsequent deletion (subsequent because I wanted to test if repair would help if java.net would not have been merged into central) I now have significant more free space.

> But, in general, simplest is to shut nexus down, remove the folders in
> question (paired, so storage/repoID/.index AND indexer/repoID), and
> restart nexus. Then run (probably manually, this should be a "once to
> be done" task) Repair Index on repositories that have index folders
> removed.

Thanks for information about relation between .index and indexer-subdirectories.
As "repair index" worked I'm happy I didn't had to shut down and delete manually.
I somehow prefer the idea of keeping the system running and/or repair by "work as designed".
But good to know what to do in case this doesn't work out, without recreating everything.
--
Best regards,

Peter

> On Fri, Sep 9, 2011 at 7:36 AM, Peter Palmreuther
> <[hidden email]> wrote:
>> Hi,
>>
>> On 05.09.2011 at 09:05 Peter Palmreuther wrote:
>>>
>>> [...]
>>> I [...] wonder if there's a way to reduce the number of kept incremental index files, as 12GiB index for 3.5MiB cached artifact is a pretty devastating ratio ...
>>> [...]
>>
>> Anyone?

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to clean up huge .index?

Tamás Cservenák
There's more to it:

And sorry for late response!


Thanks,
~t~

On Sat, Sep 10, 2011 at 12:06 AM, Peter Palmreuther <[hidden email]> wrote:
Hi,

On 09.09.2011 at 09:59 Tamás Cservenák wrote:
>
> Well, not quite an answer, but you should drop your java.net proxies,
> all that is in central now:
> http://www.sonatype.com/people/2011/08/java-net-moves-to-central/

Thanks for pointing towards this.

> And about "resetting" the incremental index: Repair Index task should
> "reset" the incremental baseline too, and will it delete all the GZ
> files.

And thanks for slapping me with the obvious. I completely forgot about the repair index option.
It seems to have solved my primary issue. Supplemented by subsequent deletion (subsequent because I wanted to test if repair would help if java.net would not have been merged into central) I now have significant more free space.

> But, in general, simplest is to shut nexus down, remove the folders in
> question (paired, so storage/repoID/.index AND indexer/repoID), and
> restart nexus. Then run (probably manually, this should be a "once to
> be done" task) Repair Index on repositories that have index folders
> removed.

Thanks for information about relation between .index and indexer-subdirectories.
As "repair index" worked I'm happy I didn't had to shut down and delete manually.
I somehow prefer the idea of keeping the system running and/or repair by "work as designed".
But good to know what to do in case this doesn't work out, without recreating everything.
--
Best regards,

Peter

> On Fri, Sep 9, 2011 at 7:36 AM, Peter Palmreuther
> <[hidden email]> wrote:
>> Hi,
>>
>> On 05.09.2011 at 09:05 Peter Palmreuther wrote:
>>>
>>> [...]
>>> I [...] wonder if there's a way to reduce the number of kept incremental index files, as 12GiB index for 3.5MiB cached artifact is a pretty devastating ratio ...
>>> [...]
>>
>> Anyone?

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to clean up huge .index?

Peter Palmreuther
On 10.09.2011 at 00:17 Tamás Cservenák wrote:
There's more to it:

Thanks again. Albeit this seems to be "work in progress", 'cause a quick check revealed there're significant parts still missing (at least in the index).
To name only one example: 'org/jboss/as'.

So I think I'll leave my JBoss proxy "In Service", but check 'central' on a regular basis, when to switch.

And sorry for late response!

No problem, thanks for responding at all.
-- 
Regards,

Peter
Loading...