Re: rewrite StatelessXmlReporter fo JAXB

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rewrite StatelessXmlReporter fo JAXB

Michael Osipov-2
Am 2017-04-16 um 14:21 schrieb Tibor Digana:

> Hi,
>
> I want to first talk with you about XML marshaller StatelessXmlReporter [1]
> in Surefire which is built on the top of SharedUtils' PrettyPrintXMLWriter.
>
> I found out issues after a contributor opened this issue [2].
>
> The problem is that we are using two streams, OutputStream and Writer, to
> create XML in file system for one reason. We write CDATA directly to the
> stream apart of xml facility. We have bug with escaping illegal characters.
>
> Instead, my proposal is to create XSD and Jaxb stubs and marshal it to XML
> file and we do need to care about low level XML and escaping characters.
> The code will be easier to understand. The elements system-out and
> system-err can be large but we have the stream already in temp files
> Utf8RecodingDeferredFileOutputStream which means that the Jaxb mashaller
> will not keep String in memory too long - only while Marshaller needs the
> string.
>
> What is your opinion?

Looking at the current code -- a lot of boilerplate. I am quite
experienced with JAXB, XJC and the Maven JAXB2 Plugin [1].

Consider that JAXB always holds the model in memory. If you intend to
maintain streams within an XML you can resort fragments and other stuff
MOXy supports. But let me tell you that it won't be that easy as you
think if you want everything as streaming with little memory usage.

Michael

BTW: this should go to dev@

[1] https://github.com/highsource/maven-jaxb2-plugin

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rewrite StatelessXmlReporter fo JAXB

Tibor Digana
Since we have XSD [1] we can generate classes and call setters in
particular places in StatelessXmlReporter.java.
Regarding memory consumption system-out may consume really a lot of memory.
Therefore this is my proposal:

@XmlJavaTypeAdapter(CdataSystemAdapter.class)
public Utf8RecodingDeferredFileOutputStream getSystemOut()

class CdataSystemAdapter
WeakReference with   utf8RecodingDeferredFileOutputStream.writeTo( stream )
return "<![CDATA[" + new String( stream, UTF_8 ) + "]]>"

[1]
https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd




On Sun, Apr 16, 2017 at 2:43 PM, michaelo [via Maven] <
[hidden email]> wrote:

> Am 2017-04-16 um 14:21 schrieb Tibor Digana:
>
> > Hi,
> >
> > I want to first talk with you about XML marshaller StatelessXmlReporter
> [1]
> > in Surefire which is built on the top of SharedUtils'
> PrettyPrintXMLWriter.
> >
> > I found out issues after a contributor opened this issue [2].
> >
> > The problem is that we are using two streams, OutputStream and Writer,
> to
> > create XML in file system for one reason. We write CDATA directly to the
> > stream apart of xml facility. We have bug with escaping illegal
> characters.
> >
> > Instead, my proposal is to create XSD and Jaxb stubs and marshal it to
> XML
> > file and we do need to care about low level XML and escaping characters.
> > The code will be easier to understand. The elements system-out and
> > system-err can be large but we have the stream already in temp files
> > Utf8RecodingDeferredFileOutputStream which means that the Jaxb
> mashaller
> > will not keep String in memory too long - only while Marshaller needs
> the
> > string.
> >
> > What is your opinion?
>
> Looking at the current code -- a lot of boilerplate. I am quite
> experienced with JAXB, XJC and the Maven JAXB2 Plugin [1].
>
> Consider that JAXB always holds the model in memory. If you intend to
> maintain streams within an XML you can resort fragments and other stuff
> MOXy supports. But let me tell you that it won't be that easy as you
> think if you want everything as streaming with little memory usage.
>
> Michael
>
> BTW: this should go to dev@
>
> [1] https://github.com/highsource/maven-jaxb2-plugin
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> <http:///user/SendEmail.jtp?type=node&node=5906482&i=0>
> For additional commands, e-mail: [hidden email]
> <http:///user/SendEmail.jtp?type=node&node=5906482&i=1>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://maven.40175.n5.nabble.com/Re-rewrite-StatelessXmlReporter-fo-JAXB-
> tp5906482.html
> To start a new topic under Maven Developers, email
> [hidden email]
> To unsubscribe from Maven Developers, click here
> <
> .
> NAML
> <
http://maven.40175.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://maven.40175.n5.nabble.com/Re-rewrite-StatelessXmlReporter-fo-JAXB-tp5906482p5906521.html
Sent from the Maven Developers mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rewrite StatelessXmlReporter fo JAXB

Michael Osipov-2
Am 2017-04-16 um 16:15 schrieb Tibor Digana:

> Since we have XSD [1] we can generate classes and call setters in
> particular places in StatelessXmlReporter.java.
> Regarding memory consumption system-out may consume really a lot of memory.
> Therefore this is my proposal:
>
> @XmlJavaTypeAdapter(CdataSystemAdapter.class)
> public Utf8RecodingDeferredFileOutputStream getSystemOut()
>
> class CdataSystemAdapter
> WeakReference with   utf8RecodingDeferredFileOutputStream.writeTo( stream )
> return "<![CDATA[" + new String( stream, UTF_8 ) + "]]>"

That would still create a string in memory, won't it?

> [1]
> https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd

This needs a rewrite, too much duplication, no typing but xs:string only.

> On Sun, Apr 16, 2017 at 2:43 PM, michaelo [via Maven] <
> [hidden email]> wrote:
>
>> Am 2017-04-16 um 14:21 schrieb Tibor Digana:
>>
>>> Hi,
>>>
>>> I want to first talk with you about XML marshaller StatelessXmlReporter
>> [1]
>>> in Surefire which is built on the top of SharedUtils'
>> PrettyPrintXMLWriter.
>>>
>>> I found out issues after a contributor opened this issue [2].
>>>
>>> The problem is that we are using two streams, OutputStream and Writer,
>> to
>>> create XML in file system for one reason. We write CDATA directly to the
>>> stream apart of xml facility. We have bug with escaping illegal
>> characters.
>>>
>>> Instead, my proposal is to create XSD and Jaxb stubs and marshal it to
>> XML
>>> file and we do need to care about low level XML and escaping characters.
>>> The code will be easier to understand. The elements system-out and
>>> system-err can be large but we have the stream already in temp files
>>> Utf8RecodingDeferredFileOutputStream which means that the Jaxb
>> mashaller
>>> will not keep String in memory too long - only while Marshaller needs
>> the
>>> string.
>>>
>>> What is your opinion?
>>
>> Looking at the current code -- a lot of boilerplate. I am quite
>> experienced with JAXB, XJC and the Maven JAXB2 Plugin [1].
>>
>> Consider that JAXB always holds the model in memory. If you intend to
>> maintain streams within an XML you can resort fragments and other stuff
>> MOXy supports. But let me tell you that it won't be that easy as you
>> think if you want everything as streaming with little memory usage.
>>
>> Michael
>>
>> BTW: this should go to dev@
>>
>> [1] https://github.com/highsource/maven-jaxb2-plugin
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=5906482&i=0>
>> For additional commands, e-mail: [hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=5906482&i=1>
>>
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://maven.40175.n5.nabble.com/Re-rewrite-StatelessXmlReporter-fo-JAXB-
>> tp5906482.html
>> To start a new topic under Maven Developers, email
>> [hidden email]
>> To unsubscribe from Maven Developers, click here
>> <
>> .
>> NAML
>> <
http://maven.40175.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
>
>
> --
> View this message in context: http://maven.40175.n5.nabble.com/Re-rewrite-StatelessXmlReporter-fo-JAXB-tp5906482p5906521.html
> Sent from the Maven Developers mailing list archive at Nabble.com.
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rewrite StatelessXmlReporter fo JAXB

Tibor Digana-2
>> That would still create a string in memory, won't it?
It would still leak in memory while getter is called.
Streaming to XML is better in this case.

Do we have any alternative?

On Sun, Apr 16, 2017 at 4:40 PM, Michael Osipov <[hidden email]> wrote:

> Am 2017-04-16 um 16:15 schrieb Tibor Digana:
>
>> Since we have XSD [1] we can generate classes and call setters in
>> particular places in StatelessXmlReporter.java.
>> Regarding memory consumption system-out may consume really a lot of
>> memory.
>> Therefore this is my proposal:
>>
>> @XmlJavaTypeAdapter(CdataSystemAdapter.class)
>> public Utf8RecodingDeferredFileOutputStream getSystemOut()
>>
>> class CdataSystemAdapter
>> WeakReference with   utf8RecodingDeferredFileOutputStream.writeTo(
>> stream )
>> return "<![CDATA[" + new String( stream, UTF_8 ) + "]]>"
>>
>
> That would still create a string in memory, won't it?
>
> [1]
>> https://maven.apache.org/surefire/maven-surefire-plugin/xsd/
>> surefire-test-report.xsd
>>
>
> This needs a rewrite, too much duplication, no typing but xs:string only.
>
> On Sun, Apr 16, 2017 at 2:43 PM, michaelo [via Maven] <
>> [hidden email]> wrote:
>>
>> Am 2017-04-16 um 14:21 schrieb Tibor Digana:
>>>
>>> Hi,
>>>>
>>>> I want to first talk with you about XML marshaller StatelessXmlReporter
>>>>
>>> [1]
>>>
>>>> in Surefire which is built on the top of SharedUtils'
>>>>
>>> PrettyPrintXMLWriter.
>>>
>>>>
>>>> I found out issues after a contributor opened this issue [2].
>>>>
>>>> The problem is that we are using two streams, OutputStream and Writer,
>>>>
>>> to
>>>
>>>> create XML in file system for one reason. We write CDATA directly to the
>>>> stream apart of xml facility. We have bug with escaping illegal
>>>>
>>> characters.
>>>
>>>>
>>>> Instead, my proposal is to create XSD and Jaxb stubs and marshal it to
>>>>
>>> XML
>>>
>>>> file and we do need to care about low level XML and escaping characters.
>>>> The code will be easier to understand. The elements system-out and
>>>> system-err can be large but we have the stream already in temp files
>>>> Utf8RecodingDeferredFileOutputStream which means that the Jaxb
>>>>
>>> mashaller
>>>
>>>> will not keep String in memory too long - only while Marshaller needs
>>>>
>>> the
>>>
>>>> string.
>>>>
>>>> What is your opinion?
>>>>
>>>
>>> Looking at the current code -- a lot of boilerplate. I am quite
>>> experienced with JAXB, XJC and the Maven JAXB2 Plugin [1].
>>>
>>> Consider that JAXB always holds the model in memory. If you intend to
>>> maintain streams within an XML you can resort fragments and other stuff
>>> MOXy supports. But let me tell you that it won't be that easy as you
>>> think if you want everything as streaming with little memory usage.
>>>
>>> Michael
>>>
>>> BTW: this should go to dev@
>>>
>>> [1] https://github.com/highsource/maven-jaxb2-plugin
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=5906482&i=0>
>>> For additional commands, e-mail: [hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=5906482&i=1>
>>>
>>>
>>>
>>> ------------------------------
>>> If you reply to this email, your message will be added to the discussion
>>> below:
>>> http://maven.40175.n5.nabble.com/Re-rewrite-StatelessXmlRepo
>>> rter-fo-JAXB-
>>> tp5906482.html
>>> To start a new topic under Maven Developers, email
>>> [hidden email]
>>> To unsubscribe from Maven Developers, click here
>>> <http://maven.40175.n5.nabble.com/template/NamlServlet.jtp?m
>>> acro=unsubscribe_by_code&node=142166&code=dGlib3JkaWdhbmFAYX
>>> BhY2hlLm9yZ3wxNDIxNjZ8LTI4OTQ5MjEwMg==>
>>> .
>>> NAML
>>> <http://maven.40175.n5.nabble.com/template/NamlServlet.jtp?m
>>> acro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base
>>> =nabble.naml.namespaces.BasicNamespace-nabble.view.web.
>>> template.NabbleNamespace-nabble.view.web.template.NodeNamesp
>>> ace&breadcrumbs=notify_subscribers%21nabble%3Aemail.
>>> naml-instant_emails%21nabble%3Aemail.naml-send_instant_
>>> email%21nabble%3Aemail.naml>
>>>
>>>
>>
>>
>>
>> --
>> View this message in context: http://maven.40175.n5.nabble.c
>> om/Re-rewrite-StatelessXmlReporter-fo-JAXB-tp5906482p5906521.html
>> Sent from the Maven Developers mailing list archive at Nabble.com.
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
Cheers
Tibor
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rewrite StatelessXmlReporter fo JAXB

Michael Osipov-2
Am 2017-04-16 um 16:52 schrieb Tibor Digana:
>>> That would still create a string in memory, won't it?
> It would still leak in memory while getter is called.
> Streaming to XML is better in this case.
>
> Do we have any alternative?

I have to think about it and understand the whole system.
But why not use the regular XMLStreamWriter paired with this:
http://stackoverflow.com/a/20000514/696632.

You'll be able to stream whatever you want.

> On Sun, Apr 16, 2017 at 4:40 PM, Michael Osipov <[hidden email]> wrote:
>
>> Am 2017-04-16 um 16:15 schrieb Tibor Digana:
>>
>>> Since we have XSD [1] we can generate classes and call setters in
>>> particular places in StatelessXmlReporter.java.
>>> Regarding memory consumption system-out may consume really a lot of
>>> memory.
>>> Therefore this is my proposal:
>>>
>>> @XmlJavaTypeAdapter(CdataSystemAdapter.class)
>>> public Utf8RecodingDeferredFileOutputStream getSystemOut()
>>>
>>> class CdataSystemAdapter
>>> WeakReference with   utf8RecodingDeferredFileOutputStream.writeTo(
>>> stream )
>>> return "<![CDATA[" + new String( stream, UTF_8 ) + "]]>"
>>>
>>
>> That would still create a string in memory, won't it?
>>
>> [1]
>>> https://maven.apache.org/surefire/maven-surefire-plugin/xsd/
>>> surefire-test-report.xsd
>>>
>>
>> This needs a rewrite, too much duplication, no typing but xs:string only.
>>
>> On Sun, Apr 16, 2017 at 2:43 PM, michaelo [via Maven] <
>>> [hidden email]> wrote:
>>>
>>> Am 2017-04-16 um 14:21 schrieb Tibor Digana:
>>>>
>>>> Hi,
>>>>>
>>>>> I want to first talk with you about XML marshaller StatelessXmlReporter
>>>>>
>>>> [1]
>>>>
>>>>> in Surefire which is built on the top of SharedUtils'
>>>>>
>>>> PrettyPrintXMLWriter.
>>>>
>>>>>
>>>>> I found out issues after a contributor opened this issue [2].
>>>>>
>>>>> The problem is that we are using two streams, OutputStream and Writer,
>>>>>
>>>> to
>>>>
>>>>> create XML in file system for one reason. We write CDATA directly to the
>>>>> stream apart of xml facility. We have bug with escaping illegal
>>>>>
>>>> characters.
>>>>
>>>>>
>>>>> Instead, my proposal is to create XSD and Jaxb stubs and marshal it to
>>>>>
>>>> XML
>>>>
>>>>> file and we do need to care about low level XML and escaping characters.
>>>>> The code will be easier to understand. The elements system-out and
>>>>> system-err can be large but we have the stream already in temp files
>>>>> Utf8RecodingDeferredFileOutputStream which means that the Jaxb
>>>>>
>>>> mashaller
>>>>
>>>>> will not keep String in memory too long - only while Marshaller needs
>>>>>
>>>> the
>>>>
>>>>> string.
>>>>>
>>>>> What is your opinion?
>>>>>
>>>>
>>>> Looking at the current code -- a lot of boilerplate. I am quite
>>>> experienced with JAXB, XJC and the Maven JAXB2 Plugin [1].
>>>>
>>>> Consider that JAXB always holds the model in memory. If you intend to
>>>> maintain streams within an XML you can resort fragments and other stuff
>>>> MOXy supports. But let me tell you that it won't be that easy as you
>>>> think if you want everything as streaming with little memory usage.
>>>>
>>>> Michael
>>>>
>>>> BTW: this should go to dev@
>>>>
>>>> [1] https://github.com/highsource/maven-jaxb2-plugin
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> <http:///user/SendEmail.jtp?type=node&node=5906482&i=0>
>>>> For additional commands, e-mail: [hidden email]
>>>> <http:///user/SendEmail.jtp?type=node&node=5906482&i=1>
>>>>
>>>>
>>>>
>>>> ------------------------------
>>>> If you reply to this email, your message will be added to the discussion
>>>> below:
>>>> http://maven.40175.n5.nabble.com/Re-rewrite-StatelessXmlRepo
>>>> rter-fo-JAXB-
>>>> tp5906482.html
>>>> To start a new topic under Maven Developers, email
>>>> [hidden email]
>>>> To unsubscribe from Maven Developers, click here
>>>> <http://maven.40175.n5.nabble.com/template/NamlServlet.jtp?m
>>>> acro=unsubscribe_by_code&node=142166&code=dGlib3JkaWdhbmFAYX
>>>> BhY2hlLm9yZ3wxNDIxNjZ8LTI4OTQ5MjEwMg==>
>>>> .
>>>> NAML
>>>> <http://maven.40175.n5.nabble.com/template/NamlServlet.jtp?m
>>>> acro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base
>>>> =nabble.naml.namespaces.BasicNamespace-nabble.view.web.
>>>> template.NabbleNamespace-nabble.view.web.template.NodeNamesp
>>>> ace&breadcrumbs=notify_subscribers%21nabble%3Aemail.
>>>> naml-instant_emails%21nabble%3Aemail.naml-send_instant_
>>>> email%21nabble%3Aemail.naml>
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> View this message in context: http://maven.40175.n5.nabble.c
>>> om/Re-rewrite-StatelessXmlReporter-fo-JAXB-tp5906482p5906521.html
>>> Sent from the Maven Developers mailing list archive at Nabble.com.
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rewrite StatelessXmlReporter fo JAXB

Andreas Gudian
What ever you do, make sure you never try to put unbounded sysout/syserr of
a test case into memory, ever. Not even for a short time, as there may be
users with a ton of output that is currently buffered in the
overflow-files. Changing this can cause serious performance issues (gc
pauses, swapping) or just break the current behavior.

Writing some XML is no rocket science, especially as our schema is very
simple and barely changes anyway. The only thing to take care is encoding
and closing all streams at the right time - but that won't be much simpler
using JAXB.

Just my 2 cent ;-)


Michael Osipov <[hidden email]> schrieb am So. 16. Apr. 2017 um 17:12:

> Am 2017-04-16 um 16:52 schrieb Tibor Digana:
> >>> That would still create a string in memory, won't it?
> > It would still leak in memory while getter is called.
> > Streaming to XML is better in this case.
> >
> > Do we have any alternative?
>
> I have to think about it and understand the whole system.
> But why not use the regular XMLStreamWriter paired with this:
> http://stackoverflow.com/a/20000514/696632.
>
> You'll be able to stream whatever you want.
>
> > On Sun, Apr 16, 2017 at 4:40 PM, Michael Osipov <[hidden email]>
> wrote:
> >
> >> Am 2017-04-16 um 16:15 schrieb Tibor Digana:
> >>
> >>> Since we have XSD [1] we can generate classes and call setters in
> >>> particular places in StatelessXmlReporter.java.
> >>> Regarding memory consumption system-out may consume really a lot of
> >>> memory.
> >>> Therefore this is my proposal:
> >>>
> >>> @XmlJavaTypeAdapter(CdataSystemAdapter.class)
> >>> public Utf8RecodingDeferredFileOutputStream getSystemOut()
> >>>
> >>> class CdataSystemAdapter
> >>> WeakReference with   utf8RecodingDeferredFileOutputStream.writeTo(
> >>> stream )
> >>> return "<![CDATA[" + new String( stream, UTF_8 ) + "]]>"
> >>>
> >>
> >> That would still create a string in memory, won't it?
> >>
> >> [1]
> >>> https://maven.apache.org/surefire/maven-surefire-plugin/xsd/
> >>> surefire-test-report.xsd
> >>>
> >>
> >> This needs a rewrite, too much duplication, no typing but xs:string
> only.
> >>
> >> On Sun, Apr 16, 2017 at 2:43 PM, michaelo [via Maven] <
> >>> [hidden email]> wrote:
> >>>
> >>> Am 2017-04-16 um 14:21 schrieb Tibor Digana:
> >>>>
> >>>> Hi,
> >>>>>
> >>>>> I want to first talk with you about XML marshaller
> StatelessXmlReporter
> >>>>>
> >>>> [1]
> >>>>
> >>>>> in Surefire which is built on the top of SharedUtils'
> >>>>>
> >>>> PrettyPrintXMLWriter.
> >>>>
> >>>>>
> >>>>> I found out issues after a contributor opened this issue [2].
> >>>>>
> >>>>> The problem is that we are using two streams, OutputStream and
> Writer,
> >>>>>
> >>>> to
> >>>>
> >>>>> create XML in file system for one reason. We write CDATA directly to
> the
> >>>>> stream apart of xml facility. We have bug with escaping illegal
> >>>>>
> >>>> characters.
> >>>>
> >>>>>
> >>>>> Instead, my proposal is to create XSD and Jaxb stubs and marshal it
> to
> >>>>>
> >>>> XML
> >>>>
> >>>>> file and we do need to care about low level XML and escaping
> characters.
> >>>>> The code will be easier to understand. The elements system-out and
> >>>>> system-err can be large but we have the stream already in temp files
> >>>>> Utf8RecodingDeferredFileOutputStream which means that the Jaxb
> >>>>>
> >>>> mashaller
> >>>>
> >>>>> will not keep String in memory too long - only while Marshaller needs
> >>>>>
> >>>> the
> >>>>
> >>>>> string.
> >>>>>
> >>>>> What is your opinion?
> >>>>>
> >>>>
> >>>> Looking at the current code -- a lot of boilerplate. I am quite
> >>>> experienced with JAXB, XJC and the Maven JAXB2 Plugin [1].
> >>>>
> >>>> Consider that JAXB always holds the model in memory. If you intend to
> >>>> maintain streams within an XML you can resort fragments and other
> stuff
> >>>> MOXy supports. But let me tell you that it won't be that easy as you
> >>>> think if you want everything as streaming with little memory usage.
> >>>>
> >>>> Michael
> >>>>
> >>>> BTW: this should go to dev@
> >>>>
> >>>> [1] https://github.com/highsource/maven-jaxb2-plugin
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: [hidden email]
> >>>> <http:///user/SendEmail.jtp?type=node&node=5906482&i=0>
> >>>> For additional commands, e-mail: [hidden email]
> >>>> <http:///user/SendEmail.jtp?type=node&node=5906482&i=1>
> >>>>
> >>>>
> >>>>
> >>>> ------------------------------
> >>>> If you reply to this email, your message will be added to the
> discussion
> >>>> below:
> >>>> http://maven.40175.n5.nabble.com/Re-rewrite-StatelessXmlRepo
> >>>> rter-fo-JAXB-
> >>>> tp5906482.html
> >>>> To start a new topic under Maven Developers, email
> >>>> [hidden email]
> >>>> To unsubscribe from Maven Developers, click here
> >>>> <http://maven.40175.n5.nabble.com/template/NamlServlet.jtp?m
> >>>> acro=unsubscribe_by_code&node=142166&code=dGlib3JkaWdhbmFAYX
> >>>> BhY2hlLm9yZ3wxNDIxNjZ8LTI4OTQ5MjEwMg==>
> >>>> .
> >>>> NAML
> >>>> <http://maven.40175.n5.nabble.com/template/NamlServlet.jtp?m
> >>>> acro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base
> >>>> =nabble.naml.namespaces.BasicNamespace-nabble.view.web.
> >>>> template.NabbleNamespace-nabble.view.web.template.NodeNamesp
> >>>> ace&breadcrumbs=notify_subscribers%21nabble%3Aemail.
> >>>> naml-instant_emails%21nabble%3Aemail.naml-send_instant_
> >>>> email%21nabble%3Aemail.naml>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> View this message in context: http://maven.40175.n5.nabble.c
> >>> om/Re-rewrite-StatelessXmlReporter-fo-JAXB-tp5906482p5906521.html
> >>> Sent from the Maven Developers mailing list archive at Nabble.com.
> >>>
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rewrite StatelessXmlReporter fo JAXB

Tibor Digana-2
Hi Andreas,
I will not change the reporter. The reason is the risk with memory
consumption. That's why we opened this discussion. I will always prevent
from growing Surefire memory consumption.

I do not want to change encoding because there is no way to do and does not
make sense. Maybe additional documentation would help the users.

Closing streams is on my list. I would like to finish populating existing
changes made so far and then communicate with all in ML.

In 2.20.1 I want to support JDK 9 and I would like to have another format
of stream we are sending from forked JVM to Maven. The stream format is in
good progress and should fix SUREFIRE-1222 and maybe those blocker issues
we have now.


On Mon, Apr 17, 2017 at 8:33 PM, Andreas Gudian <[hidden email]>
wrote:

> What ever you do, make sure you never try to put unbounded sysout/syserr of
> a test case into memory, ever. Not even for a short time, as there may be
> users with a ton of output that is currently buffered in the
> overflow-files. Changing this can cause serious performance issues (gc
> pauses, swapping) or just break the current behavior.
>
> Writing some XML is no rocket science, especially as our schema is very
> simple and barely changes anyway. The only thing to take care is encoding
> and closing all streams at the right time - but that won't be much simpler
> using JAXB.
>
> Just my 2 cent ;-)
>
>
> Michael Osipov <[hidden email]> schrieb am So. 16. Apr. 2017 um
> 17:12:
>
> > Am 2017-04-16 um 16:52 schrieb Tibor Digana:
> > >>> That would still create a string in memory, won't it?
> > > It would still leak in memory while getter is called.
> > > Streaming to XML is better in this case.
> > >
> > > Do we have any alternative?
> >
> > I have to think about it and understand the whole system.
> > But why not use the regular XMLStreamWriter paired with this:
> > http://stackoverflow.com/a/20000514/696632.
> >
> > You'll be able to stream whatever you want.
> >
> > > On Sun, Apr 16, 2017 at 4:40 PM, Michael Osipov <[hidden email]>
> > wrote:
> > >
> > >> Am 2017-04-16 um 16:15 schrieb Tibor Digana:
> > >>
> > >>> Since we have XSD [1] we can generate classes and call setters in
> > >>> particular places in StatelessXmlReporter.java.
> > >>> Regarding memory consumption system-out may consume really a lot of
> > >>> memory.
> > >>> Therefore this is my proposal:
> > >>>
> > >>> @XmlJavaTypeAdapter(CdataSystemAdapter.class)
> > >>> public Utf8RecodingDeferredFileOutputStream getSystemOut()
> > >>>
> > >>> class CdataSystemAdapter
> > >>> WeakReference with   utf8RecodingDeferredFileOutputStream.writeTo(
> > >>> stream )
> > >>> return "<![CDATA[" + new String( stream, UTF_8 ) + "]]>"
> > >>>
> > >>
> > >> That would still create a string in memory, won't it?
> > >>
> > >> [1]
> > >>> https://maven.apache.org/surefire/maven-surefire-plugin/xsd/
> > >>> surefire-test-report.xsd
> > >>>
> > >>
> > >> This needs a rewrite, too much duplication, no typing but xs:string
> > only.
> > >>
> > >> On Sun, Apr 16, 2017 at 2:43 PM, michaelo [via Maven] <
> > >>> [hidden email]> wrote:
> > >>>
> > >>> Am 2017-04-16 um 14:21 schrieb Tibor Digana:
> > >>>>
> > >>>> Hi,
> > >>>>>
> > >>>>> I want to first talk with you about XML marshaller
> > StatelessXmlReporter
> > >>>>>
> > >>>> [1]
> > >>>>
> > >>>>> in Surefire which is built on the top of SharedUtils'
> > >>>>>
> > >>>> PrettyPrintXMLWriter.
> > >>>>
> > >>>>>
> > >>>>> I found out issues after a contributor opened this issue [2].
> > >>>>>
> > >>>>> The problem is that we are using two streams, OutputStream and
> > Writer,
> > >>>>>
> > >>>> to
> > >>>>
> > >>>>> create XML in file system for one reason. We write CDATA directly
> to
> > the
> > >>>>> stream apart of xml facility. We have bug with escaping illegal
> > >>>>>
> > >>>> characters.
> > >>>>
> > >>>>>
> > >>>>> Instead, my proposal is to create XSD and Jaxb stubs and marshal it
> > to
> > >>>>>
> > >>>> XML
> > >>>>
> > >>>>> file and we do need to care about low level XML and escaping
> > characters.
> > >>>>> The code will be easier to understand. The elements system-out and
> > >>>>> system-err can be large but we have the stream already in temp
> files
> > >>>>> Utf8RecodingDeferredFileOutputStream which means that the Jaxb
> > >>>>>
> > >>>> mashaller
> > >>>>
> > >>>>> will not keep String in memory too long - only while Marshaller
> needs
> > >>>>>
> > >>>> the
> > >>>>
> > >>>>> string.
> > >>>>>
> > >>>>> What is your opinion?
> > >>>>>
> > >>>>
> > >>>> Looking at the current code -- a lot of boilerplate. I am quite
> > >>>> experienced with JAXB, XJC and the Maven JAXB2 Plugin [1].
> > >>>>
> > >>>> Consider that JAXB always holds the model in memory. If you intend
> to
> > >>>> maintain streams within an XML you can resort fragments and other
> > stuff
> > >>>> MOXy supports. But let me tell you that it won't be that easy as you
> > >>>> think if you want everything as streaming with little memory usage.
> > >>>>
> > >>>> Michael
> > >>>>
> > >>>> BTW: this should go to dev@
> > >>>>
> > >>>> [1] https://github.com/highsource/maven-jaxb2-plugin
> > >>>>
> > >>>> ------------------------------------------------------------
> ---------
> > >>>> To unsubscribe, e-mail: [hidden email]
> > >>>> <http:///user/SendEmail.jtp?type=node&node=5906482&i=0>
> > >>>> For additional commands, e-mail: [hidden email]
> > >>>> <http:///user/SendEmail.jtp?type=node&node=5906482&i=1>
> > >>>>
> > >>>>
> > >>>>
> > >>>> ------------------------------
> > >>>> If you reply to this email, your message will be added to the
> > discussion
> > >>>> below:
> > >>>> http://maven.40175.n5.nabble.com/Re-rewrite-StatelessXmlRepo
> > >>>> rter-fo-JAXB-
> > >>>> tp5906482.html
> > >>>> To start a new topic under Maven Developers, email
> > >>>> [hidden email]
> > >>>> To unsubscribe from Maven Developers, click here
> > >>>> <http://maven.40175.n5.nabble.com/template/NamlServlet.jtp?m
> > >>>> acro=unsubscribe_by_code&node=142166&code=dGlib3JkaWdhbmFAYX
> > >>>> BhY2hlLm9yZ3wxNDIxNjZ8LTI4OTQ5MjEwMg==>
> > >>>> .
> > >>>> NAML
> > >>>> <http://maven.40175.n5.nabble.com/template/NamlServlet.jtp?m
> > >>>> acro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base
> > >>>> =nabble.naml.namespaces.BasicNamespace-nabble.view.web.
> > >>>> template.NabbleNamespace-nabble.view.web.template.NodeNamesp
> > >>>> ace&breadcrumbs=notify_subscribers%21nabble%3Aemail.
> > >>>> naml-instant_emails%21nabble%3Aemail.naml-send_instant_
> > >>>> email%21nabble%3Aemail.naml>
> > >>>>
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> View this message in context: http://maven.40175.n5.nabble.c
> > >>> om/Re-rewrite-StatelessXmlReporter-fo-JAXB-tp5906482p5906521.html
> > >>> Sent from the Maven Developers mailing list archive at Nabble.com.
> > >>>
> > >>>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: [hidden email]
> > >> For additional commands, e-mail: [hidden email]
> > >>
> > >>
> > >
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>



--
Cheers
Tibor
Loading...