General: Using Windows PowerShell with a text file of URLs as input?

[dwids]

So glad I found this great script, thanks very much. I have a website, where the subpages (the content I'm after) are all links on the front page. I'm using PowerShell on Windows, and just stuck at the last bit (no xargs in PowerShell). I can use Python (BeautifulSoup) to scrape my pages and build valid URLs from the relative links. They are in a text file, say list.txt and I want to merge them into a single epub:

https://mysiteblah.info/intro.html
https://mysiteblah.info/chapter01.html
https://mysiteblah.info/chapter02.html

But I can't work out how to pass this file - or it's contents - to percollate . I've tried:

$list =  (get-content -path list.txt) 
percollate epub --output="test01.epub" $list

Also $list = @((get-content -path list.txt) -join " ") and other variations.

It doesn't 'parse' them as such, and appears to be treating all 3 as one big long single URL which looks ok (has spaces)
https://mysiteblah.info/intro.html https://mysiteblah.info/chapter01.html https://mysiteblah.info/chapter02.html but then builds a .epub with a single page with '404 Not Found' and the above single URL listed.

Any hints? Thanks!

[danburzo]

Hi @dwids, I don’t have experience with PowerShell, so I can only be of limited help. This page suggests two approaches:

1. using splatting, it sounds like you should be able to just percollate epub @(Get-Content -Path list.txt)
2. Alternatively, using the -ArgumentList option

If there’s a way to confirm that either of these produce a valid list of command-line arguments, it means we then need to investigate at the Node.js / percollate level.

[dwids]

G'day @danburzo. Thanks very much. Firstly I'm going to re-try using the Linux-under-Windows (WSL2) later today, so I'll focus on that.

Lower priority: As for Windows, pe se, I had an idea that it could be the way PowerShell passes (even parses) things - like it's $variables - when talking to Node.js / percollate. Two simple tests seems to imply an issue (?)

# Base case; go 'manual' ; enter the 4 URLs individually
percollate pdf --output="Blog Test 01.pdf"  https://sidwell.id.au/2020/05/11/hanging-rock-and-me/  https://sidwell.id.au/2020/06/04/first-australian-stories-myths-or-not/ https://sidwell.id.au/2020/05/30/its-all-relative-mummy-c-2004/ https://sidwell.id.au/2020/05/29/computer-things-going-wrong-01/

That worked; a valid PDF with 4 'chapters' (including a valid TOC)

Now just load the 4 urls into a PowerShell variable ($urls) and drop that into the command

$urls = ' https://sidwell.id.au/2020/05/11/hanging-rock-and-me/  https://sidwell.id.au/2020/06/04/first-australian-stories-myths-or-not/ https://sidwell.id.au/2020/05/30/its-all-relative-mummy-c-2004/ https://sidwell.id.au/2020/05/29/computer-things-going-wrong-01/'
percollate pdf --output="Blog Test 02.pdf"  $urls

And it seems to see the 4 URLs as 'one':

Fetching:  https://sidwell.id.au/2020/05/11/hanging-rock-and-me/  https://sidwell.id.au/2020/06/04/first-australian-stories-myths-or-not/ https://sidwell.id.au/2020/05/30/its-all-relative-mummy-c-2004/ https://sidwell.id.au/2020/05/29/computer-things-going-wrong-01/ ✓
Enhancing web page:  https://sidwell.id.au/2020/05/11/hanging-rock-and-me/  https://sidwell.id.au/2020/06/04/first-australian-stories-myths-or-not/ https://sidwell.id.au/2020/05/30/its-all-relative-mummy-c-2004/ https://sidwell.id.au/2020/05/29/computer-things-going-wrong-01/ ✓
Saved PDF: Blog Test 02.pdf

Blog Test02.pdf only had the content from the final URL computer-things-going-wrong-01 but listed the source as being from the 4 of them:

[danburzo]

I learned that Powershell is available on macOS as well, and having installed it, this worked for me:

percollate epub -o book.epub @(Get-Content -Path urls.txt)

[dwids]

Thanks! I re-tried that exact command on Windows 11

percollate epub -o book.epub @(Get-Content -Path urls.txt)

And it repeated the above; only added the final URL's contents and listed the '4 in 1' Source inside the epub. But I've got better news on WSL2. Will document here next

[dwids]

tl;dr :-) Percollate under Windows Subsystem for Linux 2 (WSL2) worked first time. I used the standard Windows Terminal. WSL2 is running Ubuntu 22.04.4 LTS

Summary:

1. I installed node.js and nvm via the numbered steps here
2. Note: you should restart the terminal session after installing nvm (not in the instructions, it seems)
3. I installed percollate as per your standard instructions

Jumped to my Windows D: drive folder - from earlier test - with my url list text files. It worked first time

cd /mnt/d/temp/toNAS/ctemp/percollate
cat urls.txt | xargs percollate epub --output=myblogtest01.epub

The epub has the correct 4 'chapters'; one per blog page. Very happy! Thanks again.

[danburzo]

Great to hear it works with WSL2!