post: soften the data collection wording in the series
All checks were successful
Deploy to OVH VPS / deploy (push) Successful in 1m15s
All checks were successful
Deploy to OVH VPS / deploy (push) Successful in 1m15s
This commit is contained in:
parent
d69d38e708
commit
2251a3e938
2 changed files with 4 additions and 4 deletions
|
|
@ -52,7 +52,7 @@ The notice board gives you a title and a PDF. The interesting part is inside the
|
|||
|
||||
So the pipeline got longer:
|
||||
|
||||
1. download the PDF (4.6 million so far)
|
||||
1. download the PDF
|
||||
2. extract the text, with OCR when it's a scan (and it's a scan more often than you think)
|
||||
3. pull out the fields: amounts, suppliers, dates, offices
|
||||
|
||||
|
|
@ -66,14 +66,14 @@ These acts are full of personal data. Names, addresses, health situations of peo
|
|||
|
||||
The law says they must stay public for 15 days. That doesn't mean "republish forever on a search engine".
|
||||
<br/>
|
||||
So nothing goes online before a privacy filter checks it. Whole categories of acts are never shown at all. Crawling everything is fine. Showing everything is not.
|
||||
So nothing goes online before a privacy filter checks it. Whole categories of acts are never shown at all. Personal data is the part to handle with the most care, not the least.
|
||||
|
||||
## What I learned
|
||||
|
||||
- **There is no standard, only vendors.** If you want to read the public sector, you must understand who sells software to it.
|
||||
- **A 200 is not a proof of life.** Check what the page says.
|
||||
- **Regex first, LLM after.** It's not sexy, but it pays the bills.
|
||||
- **Collect everything, publish carefully.** Two different decisions.
|
||||
- **Privacy is a design choice.** It goes in the pipeline from day one, not in a disclaimer at the end.
|
||||
|
||||
Next time: the festivals hidden inside all these acts. 🎉
|
||||
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ The plan was very small: learn Rust by scraping the public notice board of **one
|
|||
Six months later the numbers are these:
|
||||
|
||||
- **7,896** Italian municipalities, all 20 regions, plus unions of towns, health agencies, parks, provinces... **9,617** public bodies in total
|
||||
- more than **5.1 million** administrative acts, 4.6 million of them with the PDF downloaded
|
||||
- more than **5.1 million** administrative acts
|
||||
- **17** Rust crates in the workspace
|
||||
- **2,161** commits
|
||||
- three products and one new job title
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue