MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath

Falko Modler
Hi everyone,

I'd like to raise awareness for the MavenProject concurrency problem
that is causing MNG-6843 "Parallel build fails due to missing JAR
artifacts in compilePath" [1] and probably others [2] [3] [4].

Almost a month ago, I created a ThreadLocal-based fix for this [5]
(after another, older cloning-based approach had raised some concerns by
Robert Scholte [6]).

Michael Osipov was the only one so far having a look (thanks!) and he
suggested that more Maven team members should review this.

So, before I take a stab at the not so trivial integration test that
Michael proposed [7], I'd like to get an approval for the general
aproach (or a declination in case someone has a better idea).

Thanks for your attention and feedback!

Cheers,

Falko


[1] https://issues.apache.org/jira/browse/MNG-6843

[2] https://issues.apache.org/jira/browse/MNG-4996

[3] https://issues.apache.org/jira/browse/MNG-5750

[4] https://issues.apache.org/jira/browse/MNG-5960

[5] https://github.com/apache/maven/pull/413

[6] https://github.com/apache/maven/pull/310#issuecomment-571317501

[7] https://github.com/apache/maven/pull/413#issuecomment-754661032


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath

Tibor Digana
I commented on one issue regarding the NULL JAR file in Artifact a few days
ago.
The thing that some data is "missing" in some large object structures in
the environment with multiple threads may not necessarily have to do with
concurrent access.
There may not be any writes to MavenProject or MavenSession causing
"missed" data, and the answer why this happens is Memory Model.

It's the fact that non-concurrent or non-immutable objects may lose some
references very easily!
This has all to do with JMM and not the happens-before relationship.

Suppose that we have thread T1 creating ArrayList and adding elements into
this collection.
artifacts = new ArrayList();
artifacts.add(new DefaultArtifact(...));

Suppose thread T2 reads the artifacts from the collection right after
"artifacts.add()".
Artifact a = artifacts.get(0);

In practice the following happens:
artifacts.size() returns 1
but artifacts.get(0) returns NULL

Let;s explain why it happens.
The implementation of ArrayList is not native. It is a pure Java
implementation which has two variables inside:
+ count:int
+ array:Object[]
These two variables always appear in a critical section and they do not
have proper treatments in ArrayList.
Technically, the things are complicated on the CPU level and more
complicated than happens-before theorems.
T1 contains pointers and data in CPU registers or CPU cache. No Thread has
a direct access to a stack of another Thread, and of course it does not
operate on main memory.
The CPU uses memory barriers (assembler instructions) and a cache to
operate with RAM and memory coherency.
These instructions are used via Java keywords: final, volatile and
synchronized.
T2 may not see all elements completely from the ArrayList because there are
no safety mechanisms in the implementation of ArrayList to make this happen.
Thus the T2 may see the values in the Java variable "count" *but it may not
see the values in* "array", or vice versa.

The results are NPE, or missing JAR artifacts or the issues with Maven
Resolver, as we can see in https://github.com/apache/maven/pull/310

The solution with ThreadLocal would eat too much memory.
Reimplementing the POJO classes in Maven and making them thread safe would
solve many issues in the Core and Resolver.
Considering my examples with ArrayList, the thread safety should continue
deeper with the implementation of DefaultArtifact, etc.
In my experience, it's worth using the collection which appears in the
package "java.util.concurrent".
For instance, I use ConcurrentLinkedDequeue for simple iterators with small
amounts of elements. Alternatively use COWAL for large data and reordering
of elements by adding or removing them somewhere inside.

Cheers
Tibor17



On Sat, Jan 16, 2021 at 10:21 PM Dan Tran <[hidden email]> wrote:

> we are facing the same issue at work (300+ modules), classpath
> empty randomly empty
>
> Love to see some resolution, will help to test it
>
> Thanks
>
> -D
>
> On Fri, Jan 15, 2021 at 1:51 PM Falko Modler <[hidden email]> wrote:
>
> > Hi everyone,
> >
> > I'd like to raise awareness for the MavenProject concurrency problem
> > that is causing MNG-6843 "Parallel build fails due to missing JAR
> > artifacts in compilePath" [1] and probably others [2] [3] [4].
> >
> > Almost a month ago, I created a ThreadLocal-based fix for this [5]
> > (after another, older cloning-based approach had raised some concerns by
> > Robert Scholte [6]).
> >
> > Michael Osipov was the only one so far having a look (thanks!) and he
> > suggested that more Maven team members should review this.
> >
> > So, before I take a stab at the not so trivial integration test that
> > Michael proposed [7], I'd like to get an approval for the general
> > aproach (or a declination in case someone has a better idea).
> >
> > Thanks for your attention and feedback!
> >
> > Cheers,
> >
> > Falko
> >
> >
> > [1] https://issues.apache.org/jira/browse/MNG-6843
> >
> > [2] https://issues.apache.org/jira/browse/MNG-4996
> >
> > [3] https://issues.apache.org/jira/browse/MNG-5750
> >
> > [4] https://issues.apache.org/jira/browse/MNG-5960
> >
> > [5] https://github.com/apache/maven/pull/413
> >
> > [6] https://github.com/apache/maven/pull/310#issuecomment-571317501
> >
> > [7] https://github.com/apache/maven/pull/413#issuecomment-754661032
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>