Lately I was involved in some sort of web scraping thing, and it was very exciting. One can do so many things automatically with a simple, yet powerful script. In particular what python with lxml can offer.
So, in the city of my living I came across the service, which provides You with the current information of Your public transportation card. It’s very-very not user friendly and if I catch myself on the run, I need to go to their site and log in and then I only then I can see the critical information I need — how much money is there in my public transportation card. No app is provided on any platform. So the only possibility is to do some hacking of the system and write an app for myself.
This only means, that the data from the inputs is send directly to the server with XHR in background and then everything goes straight where it belongs to be. So the next step — what does it send?
All the form elements, plus some empty variables (?) and something randomly generated, looking like key-ish thing, which looks like no-changeable thing, because after some testing, I was still able to make a POST request to the server, and he was perfectly fine with this random thing. This is basically it, now the only thing left is the code.
The python library lies @github.com. Fell free to fork it and make something from it. And here’s the screenshot of the text-content of the returned html.
After working with scrappers, I’ve realized how insecure the web is. The only thing, which can make some changes in the field is the captcha-ish things included in the forms.