[gentoo-user] Trying to automate HTML ---> pdf

[gentoo-user] Trying to automate HTML ---> pdf

[felix]

I am trying to automate converting a URL into a pdf file.  These web
pages include javascript and fancy formatting, so the simple minded
converters just don't cut the ice.  My next plan was to hack up a real
browser so it would take two command line args, the URL and the print
file, render the page, print it to the pdf file, and exit.  From what
I know of some of them, they would have to be configured in advance,
and invocation would have to be strictly controlled so only one
instance runs at a time, at least per user.  I could probably create
several firefox user sessions and have each of them running
simultaneously, but multiple real users works for me too.

Firefox doesn't print to pdf, however.  But konqueror does.  By using
the DCOP interface, I can even pass it commands to load a URL and
print the page, altho I have to settle for the configured print file
name.  But since I have to run individual sessions anyway, that's no
big deal.  The commands look like this:

    dcop konqueror-6352 'konqueror-mainwindow#1' openURL 'http://slashdot.org'
    dcop konqueror-6352 html-widget2 print true

There's a bit more than that, since widget names change, but a simple
perl program handles it easily (so far!).

However, there's a problem.  The "openURL" command returns without
waiting for the web page to finish loading, and the "print" command
does not wait for it to finish loading.  The "print" command does wait
for printing to finish before returning, which is nice.

This means I have to put in some arbitrary "sleep 30" or so between
"openURL" and "print" to have a good chance of a complete printed
page, and even then, there is no guarantee it actually will be
complete.  We have to send these pdf files to a bank, and it would not
be good to send them incomplete pages, even if only one out of 100 or
even 1000.  There will be at least hundreds of these every day.

I started to look at sources but there is no "konqueror-3.5.8.tar.gz"
or anything similar.  No doubt most of the code is handled by Qt
widgets and KDE libs.

Here are my quests:

0.  Is there a better place to ask this?  I tried a KDE mailing list
    and got no responses; there weren't even many views.

1.  Is there either a DCOP command to wait for a URL to be loaded or a
    DCOP command like openURL which waits?

2.  Is there a source file for konqueror which I could hack to take
    command line parameters without changing libraries or other code
    which would affect the rest of KDE?  I don't have any problem with
    a hacked and renamed konqueror command.

3.  Is there some other way of converting complicated web pages into
    pdf?  If they don't understand javascript and style sheets and
    everything else that a real browser does, they are useless to me.

4.  Are there other ways to do this that I haven't thought of?

-- 
            ... _._. ._ ._. . _._. ._. ___ .__ ._. . .__. ._ .. ._.
     Felix Finch: scarecrow repairman & rocket surgeon / felix@crowfix.com
  GPG = E987 4493 C860 246C 3B1E  6477 7838 76E9 182E 8151 ITAR license #4933
I've found a solution to Fermat's Last Theorem but I see I've run out of room o
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Trying to automate HTML ---> pdf

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 804 bytes --]

On Sun, 27 Jan 2008 09:06:15 -0800, felix@crowfix.com wrote:

> 1.  Is there either a DCOP command to wait for a URL to be loaded or a
>     DCOP command like openURL which waits?

I can't see one, but it sounds like it would be useful enough to file a
bug report requesting one. A DCOP command to tell whether the page has
finished lading would be suitable.

> 2.  Is there a source file for konqueror which I could hack to take
>     command line parameters without changing libraries or other code
>     which would affect the rest of KDE?  I don't have any problem with
>     a hacked and renamed konqueror command.

Konqueror is part of kdebase, so you'll find the source somewhere in
there.


-- 
Neil Bothwick

Where the system is concerned, you're not allowed to ask `Why?'

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Trying to automate HTML ---> pdf

[Etaoin Shrdlu]

On Sunday 27 January 2008, felix@crowfix.com wrote:

>  dcop konqueror-6352 'konqueror-mainwindow#1' openURL 
> 'http://slashdot.org' 
>  dcop konqueror-6352 html-widget2 print true 
>
> There's a bit more than that, since widget names change, but a simple
> perl program handles it easily (so far!).
>
> However, there's a problem.  The "openURL" command returns without
> waiting for the web page to finish loading, and the "print" command
> does not wait for it to finish loading.  The "print" command does wait
> for printing to finish before returning, which is nice.
>[cut]
> 1.  Is there either a DCOP command to wait for a URL to be loaded or a
>     DCOP command like openURL which waits?

I know of no direct method, and I can't answer your other questions 
either.
However, the following (admittedly *really* kludgy and quick-and-dirty) 
method *seems* to work:

dcop konqueror-6352 'konqueror-mainwindow#1' openURL 'http://my.url'

while true; do

  # check if the "stop" button is clickable
  stat=`dcop konqueror-6352 konqueror-mainwindow#1 actionIsEnabled stop`

  if [ "$stat" == "true" ]; then
    # "stop" button is active, so page is still loading
    sleep 5    
  else
    # "stop" button is not active, page has loaded
    break
  fi
done

# do what you want here

As I said above, I did some tests and this seems to work. However, I'm 
not claiming that it's the solution to your problem, nor that it will 
always work as expected. Therefore, I strongly suggest you test it 
thoroughly before using it.

Hope that helped.
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Trying to automate HTML ---> pdf

[felix]

On Sun, Jan 27, 2008 at 06:56:59PM +0100, Etaoin Shrdlu wrote:

> However, the following (admittedly *really* kludgy and quick-and-dirty) 
> method *seems* to work:

Oh geez, I LOVE it!  I will play with it, it just might do the trick.
It's sure not what I had been expecting, but if it works reliably, it
is just the ticket.

Sheesh.  A bloomin' genius is what you are :-)

-- 
            ... _._. ._ ._. . _._. ._. ___ .__ ._. . .__. ._ .. ._.
     Felix Finch: scarecrow repairman & rocket surgeon / felix@crowfix.com
  GPG = E987 4493 C860 246C 3B1E  6477 7838 76E9 182E 8151 ITAR license #4933
I've found a solution to Fermat's Last Theorem but I see I've run out of room o
-- 
gentoo-user@lists.gentoo.org mailing list

[gentoo-user] Re: Trying to automate HTML ---> pdf

[Grant Edwards]

On 2008-01-27, felix@crowfix.com <felix@crowfix.com> wrote:

> Firefox doesn't print to pdf, however.

It'll print to postscript, and ps2pdf has always worked well
for me.

-- 
Grant Edwards                   grante             Yow! I represent a
                                  at               sardine!!
                               visi.com            

-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Re: Trying to automate HTML ---> pdf

[felix]

On Sun, Jan 27, 2008 at 09:26:33PM +0000, Grant Edwards wrote:
> On 2008-01-27, felix@crowfix.com <felix@crowfix.com> wrote:
> 
> > Firefox doesn't print to pdf, however.
> 
> It'll print to postscript, and ps2pdf has always worked well
> for me.

I had some problems with some of the more complex web sites, so I
didn't explore this much.  One thing I had tried was taking snapshots
to jpeg pictures, but I had no way of telling when the page had
finished loading.  Now that I have a working DCOP method of telling
this, I will have to try the jpeg business again.  One problem with it
is setting up a phony X enviornment that persists.  This is sure an
odd project.

-- 
            ... _._. ._ ._. . _._. ._. ___ .__ ._. . .__. ._ .. ._.
     Felix Finch: scarecrow repairman & rocket surgeon / felix@crowfix.com
  GPG = E987 4493 C860 246C 3B1E  6477 7838 76E9 182E 8151 ITAR license #4933
I've found a solution to Fermat's Last Theorem but I see I've run out of room o
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Trying to automate HTML ---> pdf

[Etaoin Shrdlu]

On Sunday 27 January 2008, felix@crowfix.com wrote:

> Oh geez, I LOVE it!  I will play with it, it just might do the trick.
> It's sure not what I had been expecting, but if it works reliably, it
> is just the ticket.

Java applets and flash animations could possibly cause problems, since 
they might need a few seconds to initialize even after the page is fully 
loaded (and thus the "stop" button is already inactive). Of course, if 
the pages you load don't use java/flash this is not a problem; but there 
might be other pitfalls. For example, I've noticed that konqueror loads 
some complex pages in two or more "stages", with a brief pause (and 
the "stop" button inactive) between one stage and the next.

You can check that by running something like

while true; do 
 dcop konqueror-8364 konqueror-mainwindow#1 actionIsEnabled stop; 
done

ie, continuously checking the status of the "stop" button, and you'll see 
something like

...
true
true
true
true
true
true
false
false
false
false
false
false
false
false
false
false
true
true
true
true
true
true
true
...
true
false

before the page is fully loaded and the status eventually settles 
to "false". So, if the script runs the test during the short "false" 
interval, it might be fooled into thinking that the page has loaded. I 
have not investigated further the cause of this behavior (perhaps 
multiple-frame pages?), but these few facts alone should be enough to 
deserve extra attention and thorough testing before using the kludge.

> Sheesh.  A bloomin' genius is what you are :-)

Thanks, glad you have at least a slightly better solution than before!
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Trying to automate HTML ---> pdf

[felix]

On Mon, Jan 28, 2008 at 04:01:12PM +0100, Etaoin Shrdlu wrote:

> You can check that by running something like
> 
> while true; do 
>  dcop konqueror-8364 konqueror-mainwindow#1 actionIsEnabled stop; 
> done

That will bear protecting against.  I have the basic program working,
but it does need some fine tuning, and I will make it insist on having
the status stay good for a couple of seconds after.  One of the screwy
things is that the widget names change as URLs are loaded.  I mainly
add a lot of checks and put out an alert for manual intervention when
something off happens.

So far none of the web sites use flash or Java applets.  That would be
a real mess for printing alone.

-- 
            ... _._. ._ ._. . _._. ._. ___ .__ ._. . .__. ._ .. ._.
     Felix Finch: scarecrow repairman & rocket surgeon / felix@crowfix.com
  GPG = E987 4493 C860 246C 3B1E  6477 7838 76E9 182E 8151 ITAR license #4933
I've found a solution to Fermat's Last Theorem but I see I've run out of room o
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Re: Trying to automate HTML ---> pdf

[Justin Findlay]

On AD 2008 January 27 Sunday 01:43:26 PM -0800, felix@crowfix.com wrote:
> I had some problems with some of the more complex web sites, so I
> didn't explore this much.  One thing I had tried was taking snapshots
> to jpeg pictures, but I had no way of telling when the page had
> finished loading.  Now that I have a working DCOP method of telling
> this, I will have to try the jpeg business again.  One problem with it
> is setting up a phony X enviornment that persists.  This is sure an
> odd project.

I don't think so.  What I think is odd is that you can't just say

$ konqueror http://url.com/page.html --output page.pdf

It seems like a common enough thing to want to do.


Justin
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Re: Trying to automate HTML ---> pdf

[felix]

On Mon, Jan 28, 2008 at 08:44:03PM -0700, Justin Findlay wrote:
> 
> I don't think so.  What I think is odd is that you can't just say
> 
> $ konqueror http://url.com/page.html --output page.pdf
> 
> It seems like a common enough thing to want to do.

I was astounded to not find such an option in either firefox or
konqueror.  The html to pdf utilities are only good for the most basic
pages.  It does seem like an incredibly simple common idea, and one
that shouldn't need an X environment either.  Tell it pdf or ps, the
page size (A4, letter, legal, ...), color or b/w ... not much else.

Such a simple idea, and no way to do it.  It really surprised me.

-- 
            ... _._. ._ ._. . _._. ._. ___ .__ ._. . .__. ._ .. ._.
     Felix Finch: scarecrow repairman & rocket surgeon / felix@crowfix.com
  GPG = E987 4493 C860 246C 3B1E  6477 7838 76E9 182E 8151 ITAR license #4933
I've found a solution to Fermat's Last Theorem but I see I've run out of room o
-- 
gentoo-user@lists.gentoo.org mailing list

[gentoo-user] Re: Trying to automate HTML ---> pdf

[Grant Edwards]

On 2008-01-29, felix@crowfix.com <felix@crowfix.com> wrote:

>> I don't think so.  What I think is odd is that you can't just say
>> 
>> $ konqueror http://url.com/page.html --output page.pdf
>> 
>> It seems like a common enough thing to want to do.

It's not that uncommon amon users.  It appears it's just not
something the _devs_ want to do.

> I was astounded to not find such an option in either firefox or
> konqueror.

Many people have been requesting that feature for _8_years_ in
the case of mozilla.  There are literally dozens of bugs filed
requesting this feature. It gets tossed back and forth from one
dev to another for a while, then somebody sets to the lowest
possible priority (IOW, put on the "no, we're not going to do
it, but we won't admit it" list).  After that, everybody just
sort of ignores it until somebody files yet another dup bug
requesting it. Apparenty the developers don't have any need for
the feature, so the requests just get ignored.

> Such a simple idea, and no way to do it.  It really surprised
> me.

Here's your chance.  Add the feature to Firefox. ;)

Or pay somebody else to...

-- 
Grant Edwards                   grante             Yow!  .. Everything
                                  at               is....FLIPPING AROUND!!
                               visi.com            

-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Re: Trying to automate HTML ---> pdf

[felix]

On Tue, Jan 29, 2008 at 05:49:19AM +0000, Grant Edwards wrote:

> Here's your chance.  Add the feature to Firefox. ;)

I took a look at firefox; it looks like the GUI environment is a
pretty big part of things, and it wasn't at all obvious how to get
around that.  But then again, I only spent a half hour or so staring
at a lot of strange code.

Konqueror would be even harder, since the binary itself is less than
5K and all the functionality presumably comes from the libs.

-- 
            ... _._. ._ ._. . _._. ._. ___ .__ ._. . .__. ._ .. ._.
     Felix Finch: scarecrow repairman & rocket surgeon / felix@crowfix.com
  GPG = E987 4493 C860 246C 3B1E  6477 7838 76E9 182E 8151 ITAR license #4933
I've found a solution to Fermat's Last Theorem but I see I've run out of room o
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Re: Trying to automate HTML ---> pdf

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 654 bytes --]

On Mon, 28 Jan 2008 21:24:37 -0800, felix@crowfix.com wrote:

> > $ konqueror http://url.com/page.html --output page.pdf
> > 
> > It seems like a common enough thing to want to do.  
> 
> I was astounded to not find such an option in either firefox or
> konqueror.

I think it would be more appropriate to include it in kprinter, but I
tried "kprinter someurl" and it couldn't handle HTML.


-- 
Neil Bothwick

There are some micro-organisms that exhibit characteristics of both
plants and animals.  When exposed to light they undergo photosynthesis;
and when the lights go out, they turn into animals.  But then again,
don't we all?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Re: Trying to automate HTML ---> pdf

[William Kenworthy]

On Tue, 2008-01-29 at 08:42 +0000, Neil Bothwick wrote:
> On Mon, 28 Jan 2008 21:24:37 -0800, felix@crowfix.com wrote:
> 
> > > $ konqueror http://url.com/page.html --output page.pdf
> > > 
> > > It seems like a common enough thing to want to do.  
> > 
> > I was astounded to not find such an option in either firefox or
> > konqueror.
> 
> I think it would be more appropriate to include it in kprinter, but I
> tried "kprinter someurl" and it couldn't handle HTML.
> 
> 

Why not a pdf printer:

net-print/cups-pdf

This used to work well for me, though it was about 12 months ago.  A bit
clunky and the pdf files were dumped in a non-obvious place - hopefully
fixed now.

BillK



-- 
William Kenworthy <billk@iinet.net.au>
Home in Perth!
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Re: Trying to automate HTML ---> pdf

[Stroller]

On 29 Jan 2008, at 07:24, felix@crowfix.com wrote:
> On Tue, Jan 29, 2008 at 05:49:19AM +0000, Grant Edwards wrote:
>
>> Here's your chance.  Add the feature to Firefox. ;)
> ...
> Konqueror would be even harder, since the binary itself is less than
> 5K and all the functionality presumably comes from the libs.

That makes me imagine it might actually be _easier_, but I'm not a  
practised programmer.

Stroller.
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Re: Trying to automate HTML ---> pdf

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 491 bytes --]

On Tue, 29 Jan 2008 21:31:28 +0900, William Kenworthy wrote:

> > I think it would be more appropriate to include it in kprinter, but I
> > tried "kprinter someurl" and it couldn't handle HTML.

> Why not a pdf printer:
> 
> net-print/cups-pdf


Because you would still need something to render the web page before
printing. Kprinter should be able to do this using the same libraries as
Konqueror uses.


-- 
Neil Bothwick

You shall know the truth, and you shall freak.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Re: Trying to automate HTML ---> pdf

[felix]

On Tue, Jan 29, 2008 at 12:45:41PM +0000, Stroller wrote:
> On 29 Jan 2008, at 07:24, felix@crowfix.com wrote:
>> Konqueror would be even harder, since the binary itself is less than
>> 5K and all the functionality presumably comes from the libs.
>
> That makes me imagine it might actually be _easier_, but I'm not a 
> practised programmer.

Except that if you change the libraries, everything which used those
libraries would also change.  I am trying to avoid that :-)

-- 
            ... _._. ._ ._. . _._. ._. ___ .__ ._. . .__. ._ .. ._.
     Felix Finch: scarecrow repairman & rocket surgeon / felix@crowfix.com
  GPG = E987 4493 C860 246C 3B1E  6477 7838 76E9 182E 8151 ITAR license #4933
I've found a solution to Fermat's Last Theorem but I see I've run out of room o
-- 
gentoo-user@lists.gentoo.org mailing list