ssconvert: long running time (hours) - is there a diagnostic signal?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

ssconvert: long running time (hours) - is there a diagnostic signal?

Peter Brooks
I'm running ssconvert, and it's been running for 15 hours. It's
clearly got into some sort of trouble.

This is the run string:

      ssconvert --merge-to=view_all_records.xlsx
view_AT_No_cancer_L.csv view_AT_No_cancer_R.csv view_CC_cancer_L.csv
view_CC_cancer_R.csv view_CCID_cancer_R.csv view_CCID_No_cancer_L.csv
view_CCID_No_cancer_R.csv view_CC_No_cancer_L.csv
view_CC_No_cancer_R.csv view_CV_No_cancer_L.csv
view_CV_No_cancer_R.csv view_LM_No_cancer_L.csv
view_LM_No_cancer_R.csv view_LMO_No_cancer_L.csv
view_LMO_No_cancer_R.csv view_ML_No_cancer_L.csv
view_ML_No_cancer_R.csv view_MLO_cancer_L.csv view_MLO_cancer_R.csv
view_MLOID_cancer_R.csv view_MLOID_No_cancer_L.csv
view_MLOID_No_cancer_R.csv view_MLO_No_cancer_L.csv
view_MLO_No_cancer_R.csv view_XCCL_cancer_L.csv view_XCCL_cancer_R.csv
view_XCCL_No_cancer_L.csv view_XCCL_No_cancer_R.csv
view_XCCM_No_cancer_L.csv view_XCCM_No_cancer_R.csv

Can ssconvert catch any signals and give diagnostic information?

I'd like to send it a signal to either get it to dump what it has so
far, or give an indication of where it is. Can this be done?

It's running on:

      Linux 4.4.0-59-generic #80-Ubuntu SMP Fri Jan 6 17:47:47 UTC
2017 x86_64 x86_64 x86_64 GNU/Linux

The version is:

      ssconvert version '1.12.28'

I don't know if it helps, but the only .csv file it has open at the
moment ( from lsof -p {pid} ) is:

      ssconvert 24522 peterbrooks  mem       REG   0,45 11660818
1263 /media/sf_mammogram/isolate_extremes/view_CC_No_cancer_R.csv

As you can see, that's the 9th file in the list. All the files have
been generated by awk, so it's the same format as the others, which
is:

"
Field_1,Field_2,Field_3,Field_4,Field_5,Field_6,Field_7,Field_8,Field_9,Field_10,Field_11,Field_12,Field_13,Field_14,Field_15,Field_16,Field_17,Field_18,Field_19,Field_20,Field_21,Field_22,Field_23,Field_24,Field_25,Field_26,Field_27,Field_28,Field_29,Field_30,Field_31,Field_32,Field_33,Field_34,Field_35,Field_36,Field_37,Field_38,Field_39,Field_40,Field_41,Field_42,Field_43,Field_44,Field_45,Field_46,Field_47,Field_48,Field_49,Field_50,Field_51,Field_52,Field_53,Field_54,Field_55,Field_56,Field_57,Field_58,
Image,Patient,View,Laterality,Cancer,
Stats,,,,Min,=min(f10:f50613),
Stats,,Cancer:,,Max,=max(f10:f50613),
Stats,,View:,,Mean,=average(f10:g50613),
Stats,,Laterality:,,Median,=median(f10:g50613),
Stats,,,,STD,=stdev(f10:f50613),
Stats,,,,Median - 2.5 *STD,=f6-2.5*f7,
Stats,,,,Median + 2.5 * STD,=f6+2.5*f7,
000127,19,CC,R,0,0.00157771707226,6.46075141091,1.71799998937,1.0,0.0,1.5131442631,0.869322106816,0.869322106816,2.36342017425,1.62347086736,,1.721785563616257,145.2976810847176,8.78271637341,13.0534197754,0.425333149379,1.50443704098,8.78271637341,0.425333149379,1.50443704098,1.09450649523,6.23365530718,6.23365530718,,232.83740111899138,5.963275296580617,7.480778387254143,-2.673423701633893,,79561907344.7,5.7372355245694,,-3.4156414976895983,55.691497132046386,8.93892840263,2.73881470782,0.968385123524,13.2078152934,8.93892840263,0.968385123524,13.2078152934,1.99828819462,5.78550138489,5.78550138489,,55.885750775325334,5.636237254410181,1.3493215236951794,-2.5241607854411345,359587665206.0,18.53947763416856,
,"=AVERAGE(F9:BF9)","=STDEV(F10:bf10)","=SUMPRODUCT(--(F9:BF9<F:BF))","=SUMPRODUCT(--(F9:BF9<F:BF))"
"

Any ideas or suggestions?

--
Peter Brooks


Mobile: +27 82 717 6404
Direct:  +27 21 447 9752
Skype:  Fustbariclation
Twitter: Fustbariclation
Google+: Fustbariclation
Author Page: amazon.com/author/peter_brooks
_______________________________________________
gnumeric-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gnumeric-list
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ssconvert: long running time (hours) - is there a diagnostic signal?

Peter Brooks
I'm replying to my own post, in case anybody else has this problem in
future. I've resolved it the resolution was:

1. To remove the calculation from every row. I've just put the
calculation in the first row, and I'll copy and paste it in the
spreadsheet program (MacOS Numbers, or OpenOffice).
2. To use .ods instead of the old-fashioned .xlxs. I have a feeling
that there are some limitations, perhaps not documented, to what the
old, proprietary format can do.

In future, I'll always use .ods.

I'll also remember that, for some reason, ssconvert seems to calculate
the formulae for all cells. I had thought that it would just copy it
in.

I hope this helps others..

[for Prakhar - you have subscribed to this list, if you want to
unsubscribe, you have to read the lines at the bottom of each message,
and follow the instructions]

On 19 January 2017 at 04:44, Peter Brooks <[hidden email]> wrote:

> I'm running ssconvert, and it's been running for 15 hours. It's
> clearly got into some sort of trouble.
>
> This is the run string:
>
>       ssconvert --merge-to=view_all_records.xlsx
> view_AT_No_cancer_L.csv view_AT_No_cancer_R.csv view_CC_cancer_L.csv
> view_CC_cancer_R.csv view_CCID_cancer_R.csv view_CCID_No_cancer_L.csv
> view_CCID_No_cancer_R.csv view_CC_No_cancer_L.csv
> view_CC_No_cancer_R.csv view_CV_No_cancer_L.csv
> view_CV_No_cancer_R.csv view_LM_No_cancer_L.csv
> view_LM_No_cancer_R.csv view_LMO_No_cancer_L.csv
> view_LMO_No_cancer_R.csv view_ML_No_cancer_L.csv
> view_ML_No_cancer_R.csv view_MLO_cancer_L.csv view_MLO_cancer_R.csv
> view_MLOID_cancer_R.csv view_MLOID_No_cancer_L.csv
> view_MLOID_No_cancer_R.csv view_MLO_No_cancer_L.csv
> view_MLO_No_cancer_R.csv view_XCCL_cancer_L.csv view_XCCL_cancer_R.csv
> view_XCCL_No_cancer_L.csv view_XCCL_No_cancer_R.csv
> view_XCCM_No_cancer_L.csv view_XCCM_No_cancer_R.csv
>
> Can ssconvert catch any signals and give diagnostic information?
>
> I'd like to send it a signal to either get it to dump what it has so
> far, or give an indication of where it is. Can this be done?
>
> It's running on:
>
>       Linux 4.4.0-59-generic #80-Ubuntu SMP Fri Jan 6 17:47:47 UTC
> 2017 x86_64 x86_64 x86_64 GNU/Linux
>
> The version is:
>
>       ssconvert version '1.12.28'
>
> I don't know if it helps, but the only .csv file it has open at the
> moment ( from lsof -p {pid} ) is:
>
>       ssconvert 24522 peterbrooks  mem       REG   0,45 11660818
> 1263 /media/sf_mammogram/isolate_extremes/view_CC_No_cancer_R.csv
>
> As you can see, that's the 9th file in the list. All the files have
> been generated by awk, so it's the same format as the others, which
> is:
>
> "
> Field_1,Field_2,Field_3,Field_4,Field_5,Field_6,Field_7,Field_8,Field_9,Field_10,Field_11,Field_12,Field_13,Field_14,Field_15,Field_16,Field_17,Field_18,Field_19,Field_20,Field_21,Field_22,Field_23,Field_24,Field_25,Field_26,Field_27,Field_28,Field_29,Field_30,Field_31,Field_32,Field_33,Field_34,Field_35,Field_36,Field_37,Field_38,Field_39,Field_40,Field_41,Field_42,Field_43,Field_44,Field_45,Field_46,Field_47,Field_48,Field_49,Field_50,Field_51,Field_52,Field_53,Field_54,Field_55,Field_56,Field_57,Field_58,
> Image,Patient,View,Laterality,Cancer,
> Stats,,,,Min,=min(f10:f50613),
> Stats,,Cancer:,,Max,=max(f10:f50613),
> Stats,,View:,,Mean,=average(f10:g50613),
> Stats,,Laterality:,,Median,=median(f10:g50613),
> Stats,,,,STD,=stdev(f10:f50613),
> Stats,,,,Median - 2.5 *STD,=f6-2.5*f7,
> Stats,,,,Median + 2.5 * STD,=f6+2.5*f7,
> 000127,19,CC,R,0,0.00157771707226,6.46075141091,1.71799998937,1.0,0.0,1.5131442631,0.869322106816,0.869322106816,2.36342017425,1.62347086736,,1.721785563616257,145.2976810847176,8.78271637341,13.0534197754,0.425333149379,1.50443704098,8.78271637341,0.425333149379,1.50443704098,1.09450649523,6.23365530718,6.23365530718,,232.83740111899138,5.963275296580617,7.480778387254143,-2.673423701633893,,79561907344.7,5.7372355245694,,-3.4156414976895983,55.691497132046386,8.93892840263,2.73881470782,0.968385123524,13.2078152934,8.93892840263,0.968385123524,13.2078152934,1.99828819462,5.78550138489,5.78550138489,,55.885750775325334,5.636237254410181,1.3493215236951794,-2.5241607854411345,359587665206.0,18.53947763416856,
> ,"=AVERAGE(F9:BF9)","=STDEV(F10:bf10)","=SUMPRODUCT(--(F9:BF9<F:BF))","=SUMPRODUCT(--(F9:BF9<F:BF))"
> "
>
> Any ideas or suggestions?
>
> --
> Peter Brooks
>
>
> Mobile: +27 82 717 6404
> Direct:  +27 21 447 9752
> Skype:  Fustbariclation
> Twitter: Fustbariclation
> Google+: Fustbariclation
> Author Page: amazon.com/author/peter_brooks



--
Peter Brooks


Mobile: +27 82 717 6404
Direct:  +27 21 447 9752
Skype:  Fustbariclation
Twitter: Fustbariclation
Google+: Fustbariclation
Author Page: amazon.com/author/peter_brooks
_______________________________________________
gnumeric-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gnumeric-list
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ssconvert: long running time (hours) - is there a diagnostic signal?

Morten Welinder-2
I would be interested in the dataset that caused this to happen.  Can
you share it?

There really shouldn't be anything in xlsx causing such problems.

Morten


On Thu, Jan 19, 2017 at 7:08 AM, Peter Brooks
<[hidden email]> wrote:

> I'm replying to my own post, in case anybody else has this problem in
> future. I've resolved it the resolution was:
>
> 1. To remove the calculation from every row. I've just put the
> calculation in the first row, and I'll copy and paste it in the
> spreadsheet program (MacOS Numbers, or OpenOffice).
> 2. To use .ods instead of the old-fashioned .xlxs. I have a feeling
> that there are some limitations, perhaps not documented, to what the
> old, proprietary format can do.
>
> In future, I'll always use .ods.
>
> I'll also remember that, for some reason, ssconvert seems to calculate
> the formulae for all cells. I had thought that it would just copy it
> in.
>
> I hope this helps others..
>
> [for Prakhar - you have subscribed to this list, if you want to
> unsubscribe, you have to read the lines at the bottom of each message,
> and follow the instructions]
>
> On 19 January 2017 at 04:44, Peter Brooks <[hidden email]> wrote:
>> I'm running ssconvert, and it's been running for 15 hours. It's
>> clearly got into some sort of trouble.
>>
>> This is the run string:
>>
>>       ssconvert --merge-to=view_all_records.xlsx
>> view_AT_No_cancer_L.csv view_AT_No_cancer_R.csv view_CC_cancer_L.csv
>> view_CC_cancer_R.csv view_CCID_cancer_R.csv view_CCID_No_cancer_L.csv
>> view_CCID_No_cancer_R.csv view_CC_No_cancer_L.csv
>> view_CC_No_cancer_R.csv view_CV_No_cancer_L.csv
>> view_CV_No_cancer_R.csv view_LM_No_cancer_L.csv
>> view_LM_No_cancer_R.csv view_LMO_No_cancer_L.csv
>> view_LMO_No_cancer_R.csv view_ML_No_cancer_L.csv
>> view_ML_No_cancer_R.csv view_MLO_cancer_L.csv view_MLO_cancer_R.csv
>> view_MLOID_cancer_R.csv view_MLOID_No_cancer_L.csv
>> view_MLOID_No_cancer_R.csv view_MLO_No_cancer_L.csv
>> view_MLO_No_cancer_R.csv view_XCCL_cancer_L.csv view_XCCL_cancer_R.csv
>> view_XCCL_No_cancer_L.csv view_XCCL_No_cancer_R.csv
>> view_XCCM_No_cancer_L.csv view_XCCM_No_cancer_R.csv
>>
>> Can ssconvert catch any signals and give diagnostic information?
>>
>> I'd like to send it a signal to either get it to dump what it has so
>> far, or give an indication of where it is. Can this be done?
>>
>> It's running on:
>>
>>       Linux 4.4.0-59-generic #80-Ubuntu SMP Fri Jan 6 17:47:47 UTC
>> 2017 x86_64 x86_64 x86_64 GNU/Linux
>>
>> The version is:
>>
>>       ssconvert version '1.12.28'
>>
>> I don't know if it helps, but the only .csv file it has open at the
>> moment ( from lsof -p {pid} ) is:
>>
>>       ssconvert 24522 peterbrooks  mem       REG   0,45 11660818
>> 1263 /media/sf_mammogram/isolate_extremes/view_CC_No_cancer_R.csv
>>
>> As you can see, that's the 9th file in the list. All the files have
>> been generated by awk, so it's the same format as the others, which
>> is:
>>
>> "
>> Field_1,Field_2,Field_3,Field_4,Field_5,Field_6,Field_7,Field_8,Field_9,Field_10,Field_11,Field_12,Field_13,Field_14,Field_15,Field_16,Field_17,Field_18,Field_19,Field_20,Field_21,Field_22,Field_23,Field_24,Field_25,Field_26,Field_27,Field_28,Field_29,Field_30,Field_31,Field_32,Field_33,Field_34,Field_35,Field_36,Field_37,Field_38,Field_39,Field_40,Field_41,Field_42,Field_43,Field_44,Field_45,Field_46,Field_47,Field_48,Field_49,Field_50,Field_51,Field_52,Field_53,Field_54,Field_55,Field_56,Field_57,Field_58,
>> Image,Patient,View,Laterality,Cancer,
>> Stats,,,,Min,=min(f10:f50613),
>> Stats,,Cancer:,,Max,=max(f10:f50613),
>> Stats,,View:,,Mean,=average(f10:g50613),
>> Stats,,Laterality:,,Median,=median(f10:g50613),
>> Stats,,,,STD,=stdev(f10:f50613),
>> Stats,,,,Median - 2.5 *STD,=f6-2.5*f7,
>> Stats,,,,Median + 2.5 * STD,=f6+2.5*f7,
>> 000127,19,CC,R,0,0.00157771707226,6.46075141091,1.71799998937,1.0,0.0,1.5131442631,0.869322106816,0.869322106816,2.36342017425,1.62347086736,,1.721785563616257,145.2976810847176,8.78271637341,13.0534197754,0.425333149379,1.50443704098,8.78271637341,0.425333149379,1.50443704098,1.09450649523,6.23365530718,6.23365530718,,232.83740111899138,5.963275296580617,7.480778387254143,-2.673423701633893,,79561907344.7,5.7372355245694,,-3.4156414976895983,55.691497132046386,8.93892840263,2.73881470782,0.968385123524,13.2078152934,8.93892840263,0.968385123524,13.2078152934,1.99828819462,5.78550138489,5.78550138489,,55.885750775325334,5.636237254410181,1.3493215236951794,-2.5241607854411345,359587665206.0,18.53947763416856,
>> ,"=AVERAGE(F9:BF9)","=STDEV(F10:bf10)","=SUMPRODUCT(--(F9:BF9<F:BF))","=SUMPRODUCT(--(F9:BF9<F:BF))"
>> "
>>
>> Any ideas or suggestions?
>>
>> --
>> Peter Brooks
>>
>>
>> Mobile: +27 82 717 6404
>> Direct:  +27 21 447 9752
>> Skype:  Fustbariclation
>> Twitter: Fustbariclation
>> Google+: Fustbariclation
>> Author Page: amazon.com/author/peter_brooks
>
>
>
> --
> Peter Brooks
>
>
> Mobile: +27 82 717 6404
> Direct:  +27 21 447 9752
> Skype:  Fustbariclation
> Twitter: Fustbariclation
> Google+: Fustbariclation
> Author Page: amazon.com/author/peter_brooks
> _______________________________________________
> gnumeric-list mailing list
> [hidden email]
> https://mail.gnome.org/mailman/listinfo/gnumeric-list
_______________________________________________
gnumeric-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gnumeric-list
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ssconvert: long running time (hours) - is there a diagnostic signal?

Stéphane Goujet
Hi,

On Thu, 19 Jan 2017, Morten Welinder wrote:

> I would be interested in the dataset that caused this to happen.  Can
> you share it?
> There really shouldn't be anything in xlsx causing such problems.

   I too have a problematic file. It's a .gnumeric file I stopped using
because it began taking several minutes to load, and then, once opened,
some operations took forever too, so I started another one from scratch.

   I just timed it on my computer: it took about 7 minutes @ 100% CPU to
open that file. The file is not huge (500ko, a couple hundred lines, a
dozen columns), it is home made, no fancy stuff except some conditional
formatting. It became slow at some point and only got worse after that.

   Do you want that I send it to you in private?


Faithfully yours,
   Stéphane.
_______________________________________________
gnumeric-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gnumeric-list
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ssconvert: long running time (hours) - is there a diagnostic signal?

Morten Welinder-2
> Do you want that I send it to you in private?

That's probably unrelated to Peter's problem, whatever that might be,
but go ahead
and send it.  I'll have a look then.

Morten


On Thu, Jan 19, 2017 at 3:24 PM, Stéphane Goujet
<[hidden email]> wrote:

> Hi,
>
> On Thu, 19 Jan 2017, Morten Welinder wrote:
>
>> I would be interested in the dataset that caused this to happen.  Can
>> you share it?
>> There really shouldn't be anything in xlsx causing such problems.
>
>
>   I too have a problematic file. It's a .gnumeric file I stopped using
> because it began taking several minutes to load, and then, once opened, some
> operations took forever too, so I started another one from scratch.
>
>   I just timed it on my computer: it took about 7 minutes @ 100% CPU to open
> that file. The file is not huge (500ko, a couple hundred lines, a dozen
> columns), it is home made, no fancy stuff except some conditional
> formatting. It became slow at some point and only got worse after that.
>
>   Do you want that I send it to you in private?
>
>
> Faithfully yours,
>   Stéphane.
> _______________________________________________
> gnumeric-list mailing list
> [hidden email]
> https://mail.gnome.org/mailman/listinfo/gnumeric-list
>
_______________________________________________
gnumeric-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gnumeric-list
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ssconvert: long running time (hours) - is there a diagnostic signal?

Stéphane Goujet
On Thu, 19 Jan 2017, Morten Welinder wrote:

>> Do you want that I send it to you in private?
>
> That's probably unrelated to Peter's problem, whatever that might be,
> but go ahead and send it.  I'll have a look then.

   Oh yes, I confess I highjacked the thread :-) for I remembered
trying to use ssconvert on that file and killing the process after many
minutes and no final result yet.

   I send you that file.


Goodbye,
   Stéphane
_______________________________________________
gnumeric-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gnumeric-list
Loading...