exclusive execution scheduling

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

exclusive execution scheduling

Delany
We use the exec-maven-plugin (a thread-safe plugin) to call msbuild for
dotnet builds within Maven.
We have 8 dotnet projects with 3 of them depending on shared libraries in
another directory in the repo (a total of 500 projects).

With multi-threading on (4 threads), these dotnet projects always fail
because the msbuild instances are all trying to lock the files in the
shared libraries and can't open them in the other instances. This is on
Windows 10 with NTFS. With only one thread, they always build successfully.

Note that I don't use the multi-threading capabilities of msbuild because
that would be silly.

To get this reactor working I can either

   - package up the shared libraries as part of the build, and unpack them
   into every dotnet project that needs them (using
   https://maven.apache.org/plugins/maven-remote-resources-plugin/ )

Or ensure that no 2 instances of msbuild ever run at the same time, by
either

   - chaining the dotnet projects together as one sequence of dependent
   projects (ridiculous)
   - building the dotnet projects separately with their own maven invocation
   - turning off multi-threading for the whole reactor

Are there other options?

Would it be feasible to consider adding a further thread-safe check in the
plugin architecture whereby a configuration could be set to enforce that no
two executions with the same id will ever run simultaneously?

Thanks,
Delany
Reply | Threaded
Open this post in threaded view
|

Re: exclusive execution scheduling

Tommy Svensson
I would say this is a windows problem rather than a maven problem. I have had 
similar problems trying to build simpler things where only solution is to reboot
windows to unlock NTFS (NoT a File System). 

I'm very seriously considering getting VMWare and installing a Linux Mint 20 and 
work in Linux instead. It seems like the only serious way to develop Java on a
Windows 10 machine. Working on Windows directly costs to much time in 
constant reboots to resolve locks. 

I'm having a very good 4K display, which the free virtualbox does not support
full size, thereby VmWare for me. If you don't have my display requirement, test
with VirtualBox.  

Do note however that a virtual machine takes more memory!  Personally I have
32 GB memory in my machine, but if you are at 16 GB or lower it can be 
problematic. 

And no, this wasn't exactly what you were asking. But since I'm having NTFS
locks commonly even without multi threaded builds I don't believe the problem
lies in maven. 

I have to admit that Windows 10 is far better than previous windows 
versions. It however seem to still be using same old, NTFS. 

Cheers,
Tommy 


Från: Delany <[hidden email]>
Svara: Maven Users List <[hidden email]>
Datum: 4 februari 2021 at 19:01:18
Till: Maven Users List <[hidden email]>
Ämne:  exclusive execution scheduling  

We use the exec-maven-plugin (a thread-safe plugin) to call msbuild for  
dotnet builds within Maven.  
We have 8 dotnet projects with 3 of them depending on shared libraries in  
another directory in the repo (a total of 500 projects).  

With multi-threading on (4 threads), these dotnet projects always fail  
because the msbuild instances are all trying to lock the files in the  
shared libraries and can't open them in the other instances. This is on  
Windows 10 with NTFS. With only one thread, they always build successfully.  

Note that I don't use the multi-threading capabilities of msbuild because  
that would be silly.  

To get this reactor working I can either  

- package up the shared libraries as part of the build, and unpack them  
into every dotnet project that needs them (using  
https://maven.apache.org/plugins/maven-remote-resources-plugin/ )  

Or ensure that no 2 instances of msbuild ever run at the same time, by  
either  

- chaining the dotnet projects together as one sequence of dependent  
projects (ridiculous)  
- building the dotnet projects separately with their own maven invocation  
- turning off multi-threading for the whole reactor  

Are there other options?  

Would it be feasible to consider adding a further thread-safe check in the  
plugin architecture whereby a configuration could be set to enforce that no  
two executions with the same id will ever run simultaneously?  

Thanks,  
Delany  
Reply | Threaded
Open this post in threaded view
|

Re: exclusive execution scheduling

Delany
I completely get what you're saying, in fact it hasn't been 48hrs since
I've had to reboot an NTFS machine to unlock a file.
But here the problem as it manifests is with msbuild itself (if you're
interested https://github.com/dotnet/sdk/issues/9585)

That might be where we leave it. But I think something else can be done,
and without much effort. The exec-maven-plugin claims to be thread-safe -
whatever that means (internally thread-safe?) But if it's going to call
some ad-hoc binary, it's an empty claim.

Making a simple configuration available to the model could simply and
easily prevent this situation, i.e. maintain a list of active executions in
the session, and delay the start of an execution if it was configured to
run exclusively. Does anyone else think it's worth a ticket?

Thanks,
Delany

On Fri, 5 Feb 2021 at 12:56, Tommy Svensson <[hidden email]> wrote:

> I would say this is a windows problem rather than a maven problem. I have
> had
> similar problems trying to build simpler things where only solution is to
> reboot
> windows to unlock NTFS (NoT a File System).
>
> I'm very seriously considering getting VMWare and installing a Linux Mint
> 20 and
> work in Linux instead. It seems like the only serious way to develop Java
> on a
> Windows 10 machine. Working on Windows directly costs to much time in
> constant reboots to resolve locks.
>
> I'm having a very good 4K display, which the free virtualbox does not
> support
> full size, thereby VmWare for me. If you don't have my display
> requirement, test
> with VirtualBox.
>
> Do note however that a virtual machine takes more memory!  Personally I
> have
> 32 GB memory in my machine, but if you are at 16 GB or lower it can be
> problematic.
>
> And no, this wasn't exactly what you were asking. But since I'm having NTFS
> locks commonly even without multi threaded builds I don't believe the
> problem
> lies in maven.
>
> I have to admit that Windows 10 is far better than previous windows
> versions. It however seem to still be using same old, NTFS.
>
> Cheers,
> Tommy
>
>
> Från: Delany <[hidden email]> <[hidden email]>
> Svara: Maven Users List <[hidden email]> <[hidden email]>
> Datum: 4 februari 2021 at 19:01:18
> Till: Maven Users List <[hidden email]> <[hidden email]>
> Ämne:  exclusive execution scheduling
>
> We use the exec-maven-plugin (a thread-safe plugin) to call msbuild for
> dotnet builds within Maven.
> We have 8 dotnet projects with 3 of them depending on shared libraries in
> another directory in the repo (a total of 500 projects).
>
> With multi-threading on (4 threads), these dotnet projects always fail
> because the msbuild instances are all trying to lock the files in the
> shared libraries and can't open them in the other instances. This is on
> Windows 10 with NTFS. With only one thread, they always build
> successfully.
>
> Note that I don't use the multi-threading capabilities of msbuild because
> that would be silly.
>
> To get this reactor working I can either
>
> - package up the shared libraries as part of the build, and unpack them
> into every dotnet project that needs them (using
> https://maven.apache.org/plugins/maven-remote-resources-plugin/ )
>
> Or ensure that no 2 instances of msbuild ever run at the same time, by
> either
>
> - chaining the dotnet projects together as one sequence of dependent
> projects (ridiculous)
> - building the dotnet projects separately with their own maven invocation
> - turning off multi-threading for the whole reactor
>
> Are there other options?
>
> Would it be feasible to consider adding a further thread-safe check in the
> plugin architecture whereby a configuration could be set to enforce that
> no
> two executions with the same id will ever run simultaneously?
>
> Thanks,
> Delany
>
>
Reply | Threaded
Open this post in threaded view
|

Re: exclusive execution scheduling

Lasse Lindqvist
Hi. It should already be possible for you to build a dependency graph or
execution order the way you need for multi-threaded builds.
Example: A and B are required executions for C and D can then be executed
after all, A, B and C. So just package every execution into a separate
Maven submodule and declare dependecies from D to C and from C to A and B.
In this case A and B might be executed at the same time, but C waits for A
and B and D waits for C.

pe 5. helmik. 2021 klo 19.18 Delany ([hidden email]) kirjoitti:

> I completely get what you're saying, in fact it hasn't been 48hrs since
> I've had to reboot an NTFS machine to unlock a file.
> But here the problem as it manifests is with msbuild itself (if you're
> interested https://github.com/dotnet/sdk/issues/9585)
>
> That might be where we leave it. But I think something else can be done,
> and without much effort. The exec-maven-plugin claims to be thread-safe -
> whatever that means (internally thread-safe?) But if it's going to call
> some ad-hoc binary, it's an empty claim.
>
> Making a simple configuration available to the model could simply and
> easily prevent this situation, i.e. maintain a list of active executions in
> the session, and delay the start of an execution if it was configured to
> run exclusively. Does anyone else think it's worth a ticket?
>
> Thanks,
> Delany
>
> On Fri, 5 Feb 2021 at 12:56, Tommy Svensson <[hidden email]> wrote:
>
> > I would say this is a windows problem rather than a maven problem. I have
> > had
> > similar problems trying to build simpler things where only solution is to
> > reboot
> > windows to unlock NTFS (NoT a File System).
> >
> > I'm very seriously considering getting VMWare and installing a Linux Mint
> > 20 and
> > work in Linux instead. It seems like the only serious way to develop Java
> > on a
> > Windows 10 machine. Working on Windows directly costs to much time in
> > constant reboots to resolve locks.
> >
> > I'm having a very good 4K display, which the free virtualbox does not
> > support
> > full size, thereby VmWare for me. If you don't have my display
> > requirement, test
> > with VirtualBox.
> >
> > Do note however that a virtual machine takes more memory!  Personally I
> > have
> > 32 GB memory in my machine, but if you are at 16 GB or lower it can be
> > problematic.
> >
> > And no, this wasn't exactly what you were asking. But since I'm having
> NTFS
> > locks commonly even without multi threaded builds I don't believe the
> > problem
> > lies in maven.
> >
> > I have to admit that Windows 10 is far better than previous windows
> > versions. It however seem to still be using same old, NTFS.
> >
> > Cheers,
> > Tommy
> >
> >
> > Från: Delany <[hidden email]> <[hidden email]>
> > Svara: Maven Users List <[hidden email]> <[hidden email]
> >
> > Datum: 4 februari 2021 at 19:01:18
> > Till: Maven Users List <[hidden email]> <[hidden email]>
> > Ämne:  exclusive execution scheduling
> >
> > We use the exec-maven-plugin (a thread-safe plugin) to call msbuild for
> > dotnet builds within Maven.
> > We have 8 dotnet projects with 3 of them depending on shared libraries in
> > another directory in the repo (a total of 500 projects).
> >
> > With multi-threading on (4 threads), these dotnet projects always fail
> > because the msbuild instances are all trying to lock the files in the
> > shared libraries and can't open them in the other instances. This is on
> > Windows 10 with NTFS. With only one thread, they always build
> > successfully.
> >
> > Note that I don't use the multi-threading capabilities of msbuild because
> > that would be silly.
> >
> > To get this reactor working I can either
> >
> > - package up the shared libraries as part of the build, and unpack them
> > into every dotnet project that needs them (using
> > https://maven.apache.org/plugins/maven-remote-resources-plugin/ )
> >
> > Or ensure that no 2 instances of msbuild ever run at the same time, by
> > either
> >
> > - chaining the dotnet projects together as one sequence of dependent
> > projects (ridiculous)
> > - building the dotnet projects separately with their own maven invocation
> > - turning off multi-threading for the whole reactor
> >
> > Are there other options?
> >
> > Would it be feasible to consider adding a further thread-safe check in
> the
> > plugin architecture whereby a configuration could be set to enforce that
> > no
> > two executions with the same id will ever run simultaneously?
> >
> > Thanks,
> > Delany
> >
> >
>